Python Bytes - #106 Fluent query APIs on Python collections

Episode Date: December 1, 2018

Topics covered in this episode: [play:0:49] Dependency Management through a DevOps Lens [play:5:25] Plugins made simple with pluginlib [play:8:00] How to Test Your Django App with Selenium and pytest [play:12:40]* Fluent collection APIs (flupy and asq)* [play:16:41] Guido blogging again [play:21:33]* Web apps in pure Python apps with Anvil* Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/106

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to earbuds. This is episode 106, recorded November 30th, 2018. I'm Michael Kennedy. And I'm Brian Atkin. Hey, Brian. How you doing? I'm good. It's early in the morning, man. Yeah, good morning, man. We're recording a little earlier than our average lunchtime get-together. But, you know, this is what we do for our audience. We get up when it's dark, and we talk about Python. Yes, we do. Yeah. So before we get on to the topics,

just want to say thank you to DigitalOcean. They're sponsoring the show like they did for much of this year. So check them out at pythonbytes.fm slash DigitalOcean, get $100 free credit for new users. Speaking of cloud hosting and whatnot, one of the main innovations that's been coming along here the last couple years is this whole idea of DevOps, right? I don't know. DevOps has been around for a while and really important for a lot of people. But I think Python is getting involved more and more with DevOps. We've also talked about packaging tools and dependency management a lot on this podcast and our others as well. We've talked about Poetry and PIPenv. I don't know if we've covered PIPtools before.

I don't think we have. PIPtools is a package that combines, it has a couple commands, PIPcompile and PIPsync, which are ways to work with PIP-based, usage-based packages. And I'm not going to really get into that too much, but what I found was interesting was there's an article recently written by Henick that is called Python Application Dependency Management in 2018.

So basically, he's got a use case. He's working in DevOps or devops problem space right here and he wants the requirements he's got for doing packaging is he wants to be able to specify immediate dependencies like a requirement.txt or something but then and then be able to have a tool resolve the dependency tree so i know i need django, but if Django needs a bunch of stuff, I don't want to have to specify all that. And then hopefully have hashes like the lock files, and then integrate these with talks so that you can run tests.

And then also the last one is in deploying servers, I don't want to actually have to activate a virtual environment and then install everything. I want to be able to install dependencies, an application and a dependency into a virtual environment without having to actually go in there and do it. Those all seem reasonable, but so far he's tried PIPMV, he's tried Poetry, and right now he's sticking with pip tools based solution like using the pip tools thing itself or just pip a combination of them because being able to you

can't just use pip alone for a lot of this stuff so like for instance pip tools is one of the things that you can create hashes with you can create like a the lock files essentially and working with multiple requirements files like a requirements text or like a requirements he has a work model which i haven't seen before which is sticking multiple requirements files in a requirements directory so you've got a main oh and i see a requirement slash main.txt and then a dev.txt it's for development requirements and then you also have to be able to update them. So occasionally you've got pinned requirements,

but you want to be able to update them to the newest and test that. So all these things, it requires like a work model that should – these are reasonable requests. One of the things I really liked about this was it was just – basically we have an environment now that we're we want to be able to I think it's a good thing we want to be able to not slam people for like make fun of or or bash on different projects too much but you also want to be able to be honest and say hey I tried this thing and it didn't work for me. And so one of the things I liked about this is I think it's a good example

article of, of saying, Hey, this was my use case. I tried this thing. This thing didn't quite work for me. And here's why, but also don't slam the people who made that thing because I,

I think it's cool that they're working on the problem. It just didn't work quite right for me yet. Yeah. This whole package management, package tooling stuff, dependency tooling, it feels a little bit like the editor wars. Like, don't tell me to use Vim. I want to use Emacs or I want to use PyCharm or whatever. Not quite, but it seems something like people get into their workflow

and they really like their workflow and they kind of want to stick with it. Yeah. I think there's also some limitations to some of these tools. They're built to address different problems than the one you're trying to solve. Right. They're not one-size-fits-all things. And I found it refreshing because I actually really like the idea of poetry. Actually, I like the idea of pip-am-vam poetry.

And I don't have any specifics right now, but I jumped in and tried to use it in some of my workflows. Like Kinect, some of the things I tried just didn't quite work. So I have to go back to old ways. Yeah, I mean, I guess that makes sense. The more that those tools try to do for you all at once, the more likely they are to not exactly fit your workflow. Yeah. So one way that's really cool to extend applications would be to have the ability for someone to just drop a file into a directory, accessing a common API and have that just become part of your app

or part of the capabilities of your app, right? Like a plugin, right? Yeah. That'd be cool. So there's a cool library called Plugin Lib. Plugin Lib. P-L-U-G-I-N Lib. Yeah. So really nice. And it addresses this problem in a couple ways. And I first heard about this. I'm like, well, can't you just put, you know, like a file in a directory and be good with it? You know, just try to load it. But the more I looked at what this library is

doing, it's actually solving a lot of problems for us. For one, it's an interesting example of meta classes and programming with meta classes, but you don't really have to deal with it because you just use the API and it works. But one of the things, plugins are validated when they're loaded instead of when they're used, right? Right away they're validated. Make sure everything fits the API. They can be loaded through either file paths, through entry points, through module names, and so on. You could have multiple versions of the same plugin.

Oh, interesting. Yeah, pretty interesting. You can blacklist them. So if somebody puts some kind of plugin there, you're like, no, we're not going to allow you to load this. Even, even if we find it, you know, it could figure something like that. You can group them. You can have conditionals like this plugin works on Mac OS, but not on Linux, for example, all sorts of cool stuff like that. So if, if you were thinking about having this extensible plugin model, you know, give plugin them a look. It looks like it handles a pretty simple problem, but it has a lot of support for you while it's doing it.

Yeah, this is pretty neat. Let's check this out. Yeah, I just don't work on any apps that really makes a lot of sense for me. But if I did, I would definitely want to have a look at that. Yeah, it looks like it's partly based on abstract methods of class hierarchies. Yeah. And I think I remember looking at this and this doesn't scare me off, but the particular application I was working with, we didn't have any classes uses anywhere and I didn't really want to introduce them for this. You're like, let's throw in some abstract based classesbased classes and do a thing that doesn't even use classes.

Yeah. Yeah, that makes sense. But at the same time, writing a plugin for an application is a very specific need. And I think it's fine for it didn't match the model of the rest of your system. So something to look into. Yeah, for sure. So what do you got for us next? For some reason, I'm a little interested in testing sometimes.

Really? Yeah. Yeah. I'm surprised. One of the PyBytes guys, Bob Belder little interested in testing sometimes. Really? Yeah. That's surprising. One of the PyBytes guys, Bob Belderboss, he wrote an article called How to Test Your Django App with Selenium and PyTest. And yeah, PyTest is one of my favorite things, too. It's a really nice write-up of, I'm going to quote the beginning of it. It says, in this article, I will show you how to test a Django app with PyTest and Selenium.

One of the things I was intrigued by is that they don't use a toy app. They use their Code Challenges platform and compare the logged out homepage versus the logged in dashboard. And I think that's really cool and brave of them to publish it. This is how we're testing our own stuff. Publicly, on the internet. Yeah, no, it's really cool. Yeah of them to publish it. This is how we're testing our own stuff. Publicly, on the internet. Yeah, no, it's really cool. Yeah, and it talks through, I probably wouldn't have done it,

part of the project setup, they're using the activate and deactivate of a virtual environment to load environmental variables. I don't have specific links to it, but I know there's a ways to do this within PyTest itself as well using something else. Yeah, I know. It's kind of cool to do that, but at the same time it makes it hard to... it makes it a little bit hard

to... I think it's kind of hidden in the whole setup, right? There's nowhere in the code that you see that, and you basically accomplish that by editing the activate script to set some environment variables when you activate the environment. But if you go to another machine, you check it out against some other location, that kind of stuff is not there. So it's hard to store passwords and secrets. You don't want them on. They definitely don't want to put them into this refo either. So what are you going to do? But it is tricky. Yeah. And you wouldn't be able to use Tox with this because Tox is going to create its own virtual environment.

Yeah, so maybe I'll have to write up an example of how I would tackle that. Yeah, that actually would be interesting. But the rest of it, even though they're not using mocking out a database itself, they do show how to do that, how to get the database set up.

And then, yeah, anyway, it's just kind of a cool example. I love working with Selenium, and this is a neat write-up. That's cool. Maybe not everyone knows about Selenium. They maybe know about web scraping and maybe trying to test it that way. They also maybe know about testing inside of apps.

Maybe just quickly tell folks who are not totally familiar, what is this like? Is this like requests, or what is this like like is this like request or what is this what selenium does is it really just opens up while it's running you don't have to interact with uh like uh what you see is what you get or anything you're you're using python code to control a web browser and you can use it in a couple ways you can have like a web browser that you can actually see so while it's, it pops up on your screen. Or like, for instance, Chrome and I believe Firefox also has a headless version where it can operate without actually, they just call it headless because it doesn't pop anything on your screen.

But other than that, it interacts just like it, as if it was open and you can you know navigate the dom and you can select things and activate things and and one of the benefits of going through this using selenium is that it all the javascript stuff is loaded also you're loading everything you can interact with it as if it were just literally in your browser because it's literally in a hidden browser in the headless mode so yeah it's definitely an interesting way to do it like his example they're like let's take these things and put it in the username put it in the password and then click this button you know find this button and click it the login button and stuff like that is it's really unique okay so before we get on to flu pi which is the next thing i want to

talk about let's talk about a digital issue real. So you probably built a lot of apps, have a lot of cloud resources running so far this year, but which one goes with which project? That's always been something that's driven me crazy about other places, like I go to AWS and there's just tons of virtual machines running and stuff. You're like, what is this? Do we need these? I don't even know anymore. So it would be really nice if we could group them together. And at DigitalOcean, they create this idea called projects where you can group droplets and load balancers and IPs and all that into application categories that they work with. Like you could have, you know, your app, you could have a your app sandbox testing Q&A environment, and you could have, you know, multiple ones of

those and so on. So really nice, super cool way to organize your environment in DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. And like I said, for people who are new users, you get $100 free credit to play around. So that's also really nice to have. Very nice. So I'm a big fan of fluent APIs in general.

And maybe not everyone knows the term. It's like when you take a thing and you call a function on it, and that function itself returns back the same object. So instead of saying I've got a variable and I first initialize it to a list, and then I do some operation on that list and it changes it. And I do another operation. Maybe it turns it into a set. Then I pass that set somewhere else.

I could just say, you know, like, let's say customers or let's say numbers. That's an example. It's easier. So say numbers.map.filter.skip.take.orderby, right? Just like one after another, all in one line. That's the Fluent API in general. So the thing that I want to point out is actually two libraries. So you can take your flavor of this Fluent API and apply it to collections

because there's often more than one thing you might want to do to a collection. You might want to change its values. You might want to filter it. You might want to only take a couple of them or put them into groups and things like that. So there's one called, I originally wanted to pronounce it as Fluffy, but FluPy. FluPy is in Fluent Python. And it lets you do, well, exactly what I was describing on collections. So given any collection, you can upgrade it to a Fluent collection by just calling Flu

and passing the collection. And then it has all these operations, like you say dot map and pass it a lambda. And then on that result, you say dot filter, pass another lambda dot chunk to break it into like pages, I believe, and dot take will like give you only three chunks of five back, for example. So you can do that all in effectively one line. Now maybe you want to wrap it for readability, but you don't have to use a bunch of intermediate variables or intermediate collections or anything like that. This is nice.

Yeah. It's slick, right? And the thing that's really cool is it's all based on yield and generators. Really? Yeah. Yeah. So every step in this chain is some generator type of operation, which means you could pass it a billion items and it's not going to like filter a billion items, a map a billion items and all that.

It's just going to take them one at a time through this sort of data pipeline that you build. Nice. Yeah. So the example that I put in the show notes uses count, like just all of it, but you know, it ends with a break it into like pages of five and take three pages right so it only takes 15 items out okay even though you pass it in the way that that works of course with generators because you can just pull through till like the last part stops pulling and then it basically stops iterating this infinite collection really nice in the little example you give you do like a four item in pipeline at the end too.

So you can use the result as a generator also. Yeah, exactly. Because it's a generator, like that whole sequence of operations doesn't actually execute until you iterate it, until you pass it to something that itself does, like pass it to a list or to a dictionary or something like that.

Yeah. Or use it in another one of these pipelines. So it's really cool to build up these data pipeline, data transformation things using FluPy. Yeah, and they also have a CLI. So you can do this kind of stuff from the CLI for apparently this guy's data science team. Not everyone necessarily is great with Linux, but they would like to do a little bit more

with these tools and with these ideas, these Python data pipelines. So you can actually execute these sorts of operations on the command line as well. I like it. And then do you want to cover Ask? I do. I want to cover Ask. So ASQ. So this is another one.

I don't think we've covered it, but I guess it's possible that we have. There's kind of two philosophies here. They both do the same general thing, Flupy and Ask, ASQ. But the operations that you get to apply these fluent collections changes or varies. So the Flupy is like map and filter and so on. And the other idea is if we work with this link operation, this link API, which comes from the C Sharp space and is actually, I think, one of the better parts of C Sharp. It really maps very closely to the SQL query language with where, select, order by, and so on.

So ask is very, very similar to this for Python. It's exactly the same thing, a fluid API on tops of collections. But here's a collection of words like 0123 as written out. And you say query of words dot order by Len, then by and you pass it just here is just ordering alphabetically, but you could pass something else and take five select string dot upper to list and what you get back is a list of the first five and that ordered by length one so the the short ones like one six ten and so on to me this the database like operators of ask is a little bit nicer than the map filter reduce syntax of the other one, but they're both valuable and they both have the same idea. So I kind of thought it'd be fun to

just cover them, cover them both here. I think it's good. There's both of them because it kind of depends on where, what sort of mindset you're thinking about. It's a different way to think of similar problems. Yeah. But they both solve similar problems in really interesting ways you know i mean you could say like a list comprehension okay well list comprehension could do a filter and it could do a map but it doesn't i wish there was just a little bit more on those types of things like for example wouldn't it be great if there was a order by clause in a list comprehension instead of having to do the list comprehension and then do a sorted on top of the result, you know, things like that. It's just, it's like you could almost not do these things with list comprehensions, but not quite. You can't do a paging and stuff.

So it's like, it just adds that little bit extra that often you really need in practice. Yeah. Yeah. True. Nice. So there's a pretty exciting blog. Let's kick him back into gear. Yeah, it's a blog called NeoPythonic, and it's Guido's blog. And the last time we saw him, it looks like the last blog post before November was in July of 2016. So one of the benefits of him stepping down as a BDFL is that he can blog now. So this is good. Yeah, I think it's, and you know, the one from July was just him turning off comments

because people are spamming us. So it was even further back than that. It's great to see him getting back into action. So we have this, we have him doing some more interviews with like the MIT podcast video series and whatnot. So yeah, it's really great. And a couple of questions. So he's just answering listener questions

or student questions, really. And a couple of questions are whether to choose a nine-to-five job or be an entrepreneur. And then the other one is, will AI make human software developers redundant? And of course, the answer to the second one is no, because software development

is not boring and ai is to replace boring tasks and like driving a car yeah but the nine to five versus entrepreneur is interesting because um as we know he's worked for other people for a while so yeah some really interesting places right like google and Dropbox and stuff like that. Yeah. I mean, they kind of tied together because at the end of the nine to five question, he said, just be sure to not take a job that can be immediately replaced by a computer. Yeah, for sure. Did you ever see the posters called demotivators? I think that's what it's called. No, it sounds great. So, you know, those posters that like corporations will hang up, they just like make you cringe it'll be like a an eagle flying over like ocean with a sunset and it'll say we all soar together or we can also hire like something you're just like whoa

no come on who's that supposed to act so demotivators have pictures like that but the caption isn't something super positive. One example would be there's a hang glider flying over an ocean into a sunset. And the caption is something to the effect of, it's so great to dream big and to soar towards your dream. It's even better to sit on the shore and watch people sink into the ocean. It's just stuff to sit on the shore and watch people sink into the ocean when they, you know, it's just like stuff like this, right? So there's one about what you're talking about here.

And it says, see, do you have a boring, it's got some picture of like people working together. It says, your job is boring. So boring that it'll probably be replaced by robots in 10 years. You should find a new job, but it's got like this beautiful picture.

Maybe I'll try to find it and link to it, but yeah, they're, they're really good. It's dark. Nice. Yeah. It's great. Great. Great humor. All right. So the last thing I want to talk about today is something called Anvil. And I'm surprised I haven't talked about on the show. I've had Meredith on TalkPython who created this. And I think there's several interesting aspects. In and of itself, it's interesting, but then also how it works, pretty unique and maybe inspires some other folks as well. So Anvil is a visual web building tool.

So what you do is you go there and you log in and it gives you kind of this design surface, right? You can go and you've got buttons and text boxes and combo boxes and stuff like that. And you can drop them down and interact with them, set some properties. And you've got this visual bit and you've got like this sort of code part that runs as you interact with it. So you've got a visual piece you can work with and a code behind or code view, they call it. And what's really interesting is all of that stuff happens in Python. So if you've got like a combo box, you want to hook the change event, you don't go and write JavaScript. You like double click the little combo box and it takes you to a method that's Python,

you know, def combo box dot click or underscore click. And then a little comment that says write your code here oh nice that code is python code and it runs in your browser that's cool so you know we've talked a lot about web assembly and some of these other things about how do we get python in your browser here is like a full web development framework where the majority of your logic runs on the client side written in python but then behind the scenes, you know, translated to JavaScript. Okay, well, this would be fun to play with. Yeah, it's really interesting. And then they also have like a back end database and back end services where you can write some

code that runs in like a Docker container or something. So you have a server side bit and a front client side bit, but most of the code you write by default would probably land up in, you know, if you don't restructure things, probably land up as Python running the browser. And it's really quite nice. So here's a cool way for people who don't really know that much about web apps, but they know Python, a really nice way to quickly get up to speed with sort of what I think of as like forms over data type web apps, right? Here's a grid. Here's a bunch of input boxes. I need to enter that data, save it, put it into a list and show it to you. Like really quite common types of apps for sort of intranet type apps.

So check that out. It's pretty cool. And it does have a paid version, but there's also a free version. So maybe it's interesting to people, but I think it's just interesting as a case study of how they're using Python in the browser as well. Yeah. Cool. Cool. Yeah. All right. Well, that's it for our six items this week. Brian, you got anything else you want to share with everyone?

Yeah. I want to share. I've got big news, big, big news. Anyway, not huge news, but I wrote a book called Python Testing with PyTest. Did you know that? Yes, I did. It's awesome. Yeah, so we talked about it a lot. For the first year we were doing this. But the news related to that is there's a second printing out. So I updated it, ran everything with Python 3.7 and the modern version of PyTest at the time,

and then also fixed a whole bunch of the errata of people that pointed out, just little typos and things like that. And updated the source code, stuff like that. Nice. Congratulations. So it's available. P2 is available. P2 must be second printing P2. How about you? Any news? No, nothing. Nothing right now.

I'm in just massive podcast recording mode, trying to get ready for the holidays, both so that I can take a week or two off. It's also hard to get guests to come on over the holidays because obviously they want to do just chill out stuff. It's been all recording all the time. I have cool things I've recorded, but they're not out yet. We've even done one together, right? That'll totally yeah we have here and yeah and then i had you

on mine too because i have the same problem you do that's right yeah we got a lot of fun stuff coming out on testing code and talk python and of course as always on python vice so we're not taking a break over christmas break we're going to try to go just go all the way through that's right we may physically be taking a break but due to the We're going to try to go, just go all the way through. That's right. We may physically be taking a break, but due to the magic of time shifting and podcasting, the podcast will not be taking a break. Yes.

Wonderful. Yeah. Awesome. Well, thanks for talking to me this morning. Yeah, you bet. Bye. Bye.

Thank you for listening to Python Bites. Follow the show on Twitter via at Python Bites. That's Python Bites as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm if you have a news item you want featured just visit pythonbytes.fm and send it our way we're always on the lookout for sharing something cool on behalf of myself and Brian Ocken

this is Michael Kennedy thank you for listening and sharing this podcast with your friends and colleagues