Python Bytes - #297 I AM the documentation

Episode Date: August 16, 2022

Topics covered in this episode: SQLCodeGen The death of setup.py*, long live pyproject.toml aiocache Hatch: a modern, extensible Python project manager Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/297

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 297, recorded August 16, 2022, and I am Brian Ocken. I'm Michael Kennedy. It's good to be back and be with you, Michael. Yeah, it's great to be back. Not in the usual location today. You may hear some nature sounds from my end. I apologize if animals go crazy. But there's construction which is guaranteed to be a problem in my office so i'm sitting in the backyard we'll see how that goes

yeah for people that um uh listening at the podcast the live stream people get to see his lovely view from his backyard some nice trees yeah big trees it's all it's all oregon backyard here nice but also i also want to say this episode is brought to you by IRL podcast from Mozilla. So we'll tell you more about that later, but thanks to Mozilla and the IRL podcast for spreading the show. Thank you. Let's talk about writing code by not writing code. Does that sound like a good idea? Well, okay. Not in the low code, no code sense. There's actually some pretty cool tools in that space, but that's not what I'm talking about.

Imagine, Brian, somebody says, we used to be a.NET shop and we have this huge database and all of our code to talk to it is in some other language or it's Ruby or it's Java or whatever. It's not Python. And you decide the best way to talk to this database

is with some flavor of SQL Alchemy, SQL alchemy straight. Or if you prefer, you could have it with data classes, or you could even mix in a little sprinkling of SQL model if you're doing async and fast API and Pydantic and whatever choice you want to make from that perspective. If you're looking at a database with 150 tables and all sorts of gnarly relationships, you might say, well, I'm going to have to spend the next week planning out how to model those out. So I get them to exactly match the database. Is it a VAR char? Is that a VAR char with a limit like VAR char 10 or, you know, what, that doesn't sound fun, does it? No, no, not, not, I mean, at least for me, maybe some people that's a special kind of fun for

them but josh thurston sent over this project called sequel a for sequel alchemy code gen so i'm taking that he doesn't think it sounds like fun i certainly don't think that it does either so people should check this out it's it looks really cool and what you do is it's an automatic model generator for SQL Alchemy. What does it generate it from? So what you do is you go through and you point it at some database. Okay. Just say SQLACodeGen and you give it your connection string. For example, Postgres SQL colon triple slash some database connection string.

And then magic happens, and you have a whole bunch of Python classes that are attempted to look handwritten. So instead of taking all your time, like a week, to model out the database and the relationships and all that stuff in Python, you run this one command line thing, and then you have all the classes, and then you can tweak them a tiny bit if you see fit.

Okay, so this is, okay, you probably said this and I missed it, and then you have all the classes and then you can tweak them a tiny bit if you see fit. Okay, so this is, okay, you probably said this and I missed it, but so you already have data in a database and you're trying to hook up an application to it or something. Yeah, yeah, which is why one of my theoretical example, I have a database, I have code that talks to it, but it's not Python. And so there's not really something to work from,

but I've got a really complicated database that's been around for a while. It doesn't have to be complicated, but the more gnarly the database, the more you will appreciate this tool making, you know, doing it for you. I think I have this situation. I totally want to use this. Yeah. Yeah. Yeah. Yeah. So it does a bunch of neat things. It's written to read the structure of an existing database and generate the appropriate SQL alchemy model code using the declarative style if possible. So derived from SQL alchemy declarative base. It's also, there was some other tool called SQL autocode, which apparently has some limitations

such as, for example, it doesn't support Python 3 or recent versions of SQL alchemy. That seems like a pretty large limitation, but whatever. So this supports the newest version. It produces PEP8 compliant code, tries to make it look like you're writing code by hand so it doesn't look auto-generated. It automatically detects join table inheritance and all kinds of things. So if I scroll down here, it's got these different generators. So you can generate table objects for people who don't want to use the ORM because SQL Alchemy has these two flavors, like a low level, slightly above just raw SQL, and then the ORM, which is the one that I use all the time. By default, it uses the ORM one, but you can also use, like I said, data classes, which is pretty

excellent instead of, you know, in case that's what you want your code to look like, or even better than that, you can use SQL model models or using SQL model, which is the project, I'm sure we've discussed it before, by Sebastian. It's based on Pydantic and Async, but then it's built really on top of SQL Alchemy. So if you're looking to do the newer version of that, you'll get this. By the way, just thinking about it while I'm looking at this, maybe I have actually a SQL Alchemy generated database, but it's written in the older style of SQL Alchemy and it's not using SQL model and I want to just upgrade to SQL model.

You might be able to use the database to just go rewrite it for me again, but in this flavor, you know, or use it to generate the data class version. That actually looks pretty cool. Or do all of them and look at it and see which one looks like it's more fun to maintain. Yeah, exactly.

So there's a whole bunch of options and stuff that you can use, but you can basically pass a bunch of command line arguments and stuff to change how it works. Cool. Like change how it names objects, or change how it names fields, et cetera, et cetera.

People, if they really want to look into it and use it, I think they all got the idea from this. Okay, cool. What do you think? Sound cool? Yeah. Looks really cool. Yeah. Anytime you've got a database, they're so hard to model because you've got to get the SQL Alchemy code to match it just right, or it won't work at all. And yet, you know, do you really want to do that by hand? No, I don't. And so SQLACodeGen, thanks, Josh. Well, I'm going to talk about packaging. I've been, my headspace is in packaging lately because I'm preparing a talk for next month and it's going to have some packaging stuff in it.

So this is really exciting with, I heard from Juan Luis Cano Rodriguez. Rodriguez, sorry. Setup tool 664.00 is out, and it ships with PEP 660 editable installs. So the big headline is not that, although that's really cool. It's that most projects don't need a setup.py or a setup.cfg anymore um those those

can be gone so and not that setup.py was evil but it kind of was evil because what it does is it runs python runs python while you install something like when used normally it's fine when but it has this tremendous gaping hole for abusing things at different levels. Yeah. Okay, so the caveat on this is the reason why it has that is sometimes it goes out and compiles stuff if it's not just pure Python stuff. But you don't need that for that. Most projects don't need that anymore because they're not really compiling stuff there on during pip install. They're they're compiling stuff ahead of time and they have their separate wheels for different architectures.

So I like that model better. So anyway, the I'm like pretty excited about this. So there's a so yeah, congrats to the pipe for getting that done. We've got there's an article called Development Mode Editable Installs. So here you've got pip install editable. That works with setup tools without a setup.py. Used to have to have a shim. So everything can be in pyproject.toml now.

And so one of the cool things, and I actually discovered this also at the same time I was researching packaging stuff. The PyPA has this really cool guide packaging Python projects. And what's neat is they keep it up to date fairly well. And this whole tutorial, it's a simple. So you got a simple Python project and you want to try to learn how to package. They have this this page here it's nice there's no mention of setup.py or setup.cfg it's all pyproject.tom oh wow that's awesome here's how you do it and you don't even talk about the older more troublesome way yeah

and since setup tools and pip are not part of python proper they're separate things well i mean you get pip when you when you download python but you get a version and you usually upgrade anyway but setup tools is separate uh so they can move at a faster pace than uh python itself so all right okay excellent well this is great news uh anything that makes the supply chain side of Python stronger is good. Yep. Indeed. Well, before we move on to the next thing, Brian, let me tell you about our sponsor. All right.

So this episode of Python Bytes is brought to you by the IRL podcast, an original podcast from Mozilla. If you're like me and Brian, we care about ideas behind technology, not just the tech itself. So we know that tech has an enormous influence on society. Many of these effects are hugely beneficial. Just think about carrying all of the world's information in our pockets sort of thing. But other tech influences can have negative effects. And I really appreciate that Mozilla is always on the lookout for and working to mitigate negative influences of tech on all of us, all the tracking stuff they're doing, but a bunch of awareness things as well. And so if these ideas resonate with you, you should definitely check out the IRL podcast.

It's hosted by Bridget Todd. And this season is very much in the focus of Python. It's AI in real life. So who can AI help? Who can it harm? The show features fascinating conversations with people who are working on building more trustworthy AI. For example, there's an episode about how our world is mapped with AI. And it's the data that's missing from those maps that tells as much of the story as the maps themselves.

Another one's about gig workers and how they're pushing back on algorithms to create better working style. And for political junkies, there's an episode about how the role of AI plays when it comes to spreading disinformation about elections. Obviously, huge concern just across the world for all the democracies. I also just listened to The Tech We Won't Build, which explores when developers and data scientists should consider pushing back on projects that can be harmful to society, even though the machine learning can easily be turned on them. If this sounds interesting, try an episode for yourself.

Just check out, just search for IRL on your podcast player or visit pythonbytes.fm slash IRL. The link is in your podcast player show notes. Thank you so much to IRL and Mozilla for supporting the show. I've been listening to it. It's a really great show too.

Yeah, yeah. I enjoy it as well. It's not super, super technical where it's all about APIs and stuff. You can kind of just kick back and enjoy it. Not like this podcast where we're super technical. Exactly.

Yeah, we talk about a bunch of technical things, sometimes not too deeply, huh? Oh, I like it. I like our level. Yeah, I do, too. I do, too. Before we get on to the next topic, I just want to do a quick audience comment from Anna here.

It says, hello from London, UK. SQL Code Gen sounds like it could save a lot of headaches. Yeah, I think it's going to be great. Yeah. And feedback for the tutorial that you highlighted, Brian, on Python.org. Yep. Henry Schreiner says, it took around six months

for my rewrite of that page to get accepted well thank you for all the hard work henry that's awesome it's great job it's it looks great so yeah very very cool previously i had talked about async cache remember that where it's like the func tools lru cache but a little bit more however you can apply to async methods owen lamont said you may also be interested in AIO cache. What this one does is this lets you use proper distributed backends for caching. So for example, if you're on a web app, you might have five, six, 10, maybe many more worker processes, either on one machine using the supervisor mode of like Unicorn, or it could be even across different computers.

If you're using that in-memory version of cache, every time the request goes to a different part of your site or different runtime, different process running your site, you've got to recompute it, right? Well, this one also supports Redis and Memcached. Oh, sweet. Or Memcached. And it has a common API across all of them,

which is pretty fantastic. And they're all async and awaitable, which is cool. Oh, sweet. say, give me the values corresponding to these four IDs, or does this thing exist or not? Delete, clear, even increment a value, like how many people viewed this page? Increment that in the cache. And it's shared across, like I said, 10 worker processes across machines instantly. How cool is that? That's pretty cool.

Yeah, so super easy to work with. You can install it, but then you should also reference probably the specialization that you're using or the backend that you're using. So for example, you can say pip install AIO cache, but if you want to use Redis, it's bracket Redis. If you want to use Redis and memcached, you might say bracket Redis memcached. They have message pack and different formatting.

So depending on how you're using it, you might have to install some dependencies. The optional install mechanism of pip is pretty cool, though. I like that. It is. It is pretty cool. So you just import asyncIO, easy, and then you import AOCache, and you've got to just basically run your loop somewhere. Or if you're using something like FastAPI,

just that thing's generating or managing the loop for you, so you don't have to worry about it. So you can just say await cache.set key comma value, await cache.get key. Sounds pretty straightforward, right? You can even use it as a decorator. So if you put at cache on a function, you can give it time to live,

the target, which is Redis, the key to use for that particular thing and so on. And then off it goes. The serializer, you have pickles or you have message pack or JSON. And then off it goes. The serializer, you have pickles or you have message pack or JSON. And then there you go. Pretty cool, huh? So does it, okay, so for a function,

does it cache the input and output of that function then? I think it caches the output. Okay. But it doesn't look like it varies by argument. At least in this example, it's not very high. At least, yeah, and it's not very fine. Yeah, and I don't see how you would, like this key is the lookup value, right?

So you might call that cache call key result or something. I don't see how you dynamically do that. It's got to be like a void. But, you know, a lot of times that's, you just like show me all the products in this database or whatever, right? Yeah. Cool. Cool.

Yeah, pretty neat, huh? Very neat. Yeah, Cool. Cool. Yeah, pretty neat, huh? Very neat. Yeah, and then you have different three basic ideas to think about. You have these backends. So you have Redis backed by AIO Redis. You have Memcache backed by AIO Memcache. And then serializers, like you can serialize just string,

pickle, JSON, message pack, but you can also build your own. And you can also plug in, there's a bunch of examples and documentation people can check out. So this looks really neat to me. Yeah. It's not quite something that I need,

but if I did need it, I would definitely go. I know. And I'd be all over it. Yeah. Regarding the level of detail we have in our podcast, SE Steve says,

I only listen to podcasts about APIs. Nice. Nice, nice. What's your last one, Brian? My last one is, well, the Python packaging project. Packaging Python project. Same thing. No, I have a new one, but I got it from this.

So when I was reading this article or this tutorial again, I came across something I wasn't familiar with. So I had to go check it out. So down with creating a PyProject.toml, one of the options is hatchling. Hatchling? Yeah. I've heard of hatch. Is this somehow related to hatch? Yeah, it is hatch.

So have we already covered hatch? No, I don't think we have here, but i love the idea of it okay so so hatch is a modern extensible python project manager with a whole bunch of cool features like a standardized build systems build system and reproducible builds by default um and environment management which you know okay so so i'm not sure if this is similar to poetry's environment manager or not. I haven't played with it much. But you don't actually have to care, which is nice. Because Poetry, you have to care about it.

Because that's part of the whole thing. Anyway, publishing is easy to PyPI and other sources. Version management, project generation with sane defaults, which I haven't tried that. So I want to try that. And supposedly a responsive CLI that's two to three times faster than equivalent tools so this i definitely need to try so this is uh i would think it's similar on the line of flit uh i think um uh but with some extra things thrown in because and one of the reasons why i love flit now is because even though setup tools does now

support pi project homo properly completely flits like twice as fast for building stuff but uh but so i definitely want to try out hatch and try this i did try one little small project just converting a flip project to hatch and it took me like five minutes just using the the documentation here is great so the excellent documentation here about how to use the different pieces of it. So it's pretty neat. Have you tried it? I have not tried it. I've looked at it and it looks neat to me.

I don't make many packages. I more build applications and web apps and stuff. So I'm less in the what's the right tool to build packages properly, you know, and I know you're doing that a little bit more. Yeah. I guess good mix. I do more packages and less applications and you do more applications. Yeah.

Some live feedback. Henry Schreiner says, you can use any 621 backend, Hatchling, PDM, Flitcore, and so on with Hatch or with PDM2, one of the fantastic results of standardization. And that hatchling does a much better job of getting source files right than FlitCore.

Okay, interesting. Yeah, there was a lot of cool options with the hatch that you could specify exactly which modules and packages to pick up if you need to. That's one of the things that's a little bit mysterious with Flit, how to figure it out, because it just sort of knows somehow. I think it's the stuff that's in Git, but it's interesting.

So the main thing is, is I really like, with the standardization, that Hatch is possible, that Flit's possible, that PDM is possible, that we can do new things uh and they're not that different like uh the it's kind of the back end of packaging and the front end is the pipe project.com uh that's a better world to be in so yeah that separation lets let there be a lot more yeah exploration a lot more variation yep all right well that's it for our items, isn't it? I think so. Yeah. You got any extras? No, but hopefully I will soon. So I've got something I'm working on.

Oh, right on. Yeah. Yes, I know. I think I know what you're alluding to. Very exciting stuff coming quite soon. Yeah. How about you? I have one extra.

This one's quick, but quite cool. So I'm sitting here on my MacBook Pro with the M1, you know, MacBook Pro, the M1 Max. And until recently, I wasn't able to use PyPy. Now, PyPy is the JIT compiled, often faster version of Python. Sometimes you'll hear people say PyPy when they are referring to PyPI, but all the people who work on PyPI pronounce it that way. And it leaves space for PyPi to be pronounced like it should. So PyPi is the fast JIT-compiled version of Python. And the big news is a couple weeks ago,

they announced support for M1, which is pretty cool. So if you're on Apple Silicon, you can now use PyPi. That's very cool. Natively. Yeah. It was done by Fajal and supported by contributions from the Open Collective, which is pretty cool. And it's based on support for

AARCH64, which is ARM64 and Linux with some variations on how this works. So they've got 3.8 and 3.9 working on macOS ARM64 platform, which is pretty cool. So if you're using that and you've been waiting for this,

it should make your code run faster, maybe use less memory, that kind of thing. Very cool. If people are interested in PyPy, testing code on episode 190, I interviewed Carl Frederick Bowles about testing PyPy. Oh, cool. Yeah, there's a lot of testing. I mean, it's the entire Python runtime, basically. It's in much of the standard library.

That's a lot of work. Yeah, it's an interesting story. Would you say that testing and documentation are often really good things to add to your project that go along together in some ways? Well, hopefully you're doing it at the same time, but yes. Yes. You know, we've all worked with different types of team dynamics,

the sort of flat hierarchy. People would just take over the projects, the parts of the projects that they seem best suited for, and there might be more hierarchical versions. So our joke this week is about a somewhat dominating senior developer here, and there's a junior developer just hired onto the team. This is a picture. People can check it out. Just follow the link in the show notes. The junior asked, where's the documentation? And a very stern face,

the team says, I am the documentation. Yes. Hopefully you're not currently working in this situation but it's pretty funny you know there's always bits there's always pieces of the system that like well how does this work oh you've got to ask that guy he's not even on our team before anymore yeah but he's the one that wrote it uh and luckily he's still with the company so go talk to him exactly no one understands that we don't touch it anymore. It seems to still work. Yeah. Exactly. All right.

Well, what seems to still be working is our podcast, Brian. It does. Yeah. Thank you. Thank you. Yeah. Thanks for everyone to listen. See y'all later.

Bye.