Python Bytes - #114 What should be in the Python standard library?

Episode Date: January 26, 2019

Topics covered in this episode: [play:0:37] What should be in the Python standard library? [play:9:00] Data Science portal for Home Assistant launched [play:12:17] What's the future of the pandas library? [play:16:17] PyOxidizer [play:19:31] Working With Files in Python [play:22:06] $ python == $ python3? Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/114

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 114, recorded January 23rd, 2019. I'm Michael Kennedy. And I'm Brian Eichen. And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. Brian, how are you doing? I am doing good.

That's great to hear. I am as well. And I hear often people praise Python because it is a batteries included language or technology. And I think that's really important, but it's not a settled debate, is it? No, this was interesting. We actually, there's a article we're going to link to called what should be in the Python standard library. So with the batteries included, we often think of that as, well, I don't know. I used to think of it as the standard library. So with the batteries included, we often think of that as, well, I don't know. I used to think of it as the standard library.

Now I kind of think of it as the standard library plus some of the recommended PyPI packages and stuff. Exactly. Like there's, it's almost like an onion. You have the language, you have the standard library and you have pip install anti-gravity like wrapping around it. I agree. When I think of batteries included, I think of it as like that whole whole spectrum then there's even stuff that you couldn't can't easily install

even through pi pi things like that are in distributions like the scientific packages and stuff yep indeed we were going to talk about it last week but i couldn't find an article that didn't say stuff like this is internal but now i found a an article onWN.net, and I'm probably not the first one to say that this is a really ugly website. This is technically true. It could certainly use a little bootstrap action theme or something. But it's by Jake Edge, and Jake Edge does a lot of good coverage of the core developer meeting. So well done, Jake. No, yeah, definitely great information.

This will be hard to summarize. But anyway, we've got somebody, I think Jonathan Underwood proposed adding LZ4 compression to standard library because there's already some other compression. Right. You can do zip files and tar files. Why not LZ4 or RAR

or some random other type of compression? Right. So, and then Brett Cannon proposed making something similar to hash lib. If there's a whole bunch of other kinds of compression algorithms, maybe we could have like a compressed lib or something that had a bunch of different algorithms in it. Then can of worms opens. So, there's an argument against it. Basically, we don't't actually standard lib doesn't need lz4 maybe people do but standard lib doesn't it already has bz2 there which i'm not sure what

bz2 it i think it's a zip thing but it's um kind of late to remove it since people rely on it and then that then kind of a discussion ensued, like if standard lib is the batteries included, shouldn't we have some way to add new batteries? Even things that aren't needed by standard library, but other people might need? And then there's a discussion as well of you can't just rely on PyPI because some people don't have easy access to it or can't install things. Right. Maybe it's like REPL, some kind of online REPL, and you can type whatever they've installed, but that's it. That's true. There's online REPL

stuff. Then there's also things like people working behind a firewall that have to have proxies to get out and use PyPI, and you have to apply for a proxy password and stuff like that. So there's issues. And then it gets bigger as to to, well, if we're not going to add a whole bunch of other stuff, maybe we should look at the stuff we already have and maybe kick some of it out. Who would decide what gets kicked out? And it falls along with a discussion of, well, maybe we don't really think of it as just the standard library as our batteries included, but maybe we could have a standard distribution that had like some set of recommended pi pi packages right because the initial discussion was can we make standard lib bigger and some of the action

the responses were actually can we make it smaller can we have less stuff not more which i thought was pretty interesting one of the and which has led to let's make it a lot smaller, but let's make much of what is standard lib, like a standard distribution, like Anaconda versus Miniconda type of thing. Right. With more or less, but you can choose that sort of thing, right? Yep. There really isn't a solution here. It's just an interesting discussion going on. So I thought I'd bring it up. Anything that would happen would end up happening through PEPs, and the PEP process is sort of stalled right now.

So we've got to figure out that first. Yeah, it's interesting. I certainly see some of the drawbacks, right? Like one of the complaints or one of the cautionary flags that was raised when they said, oh, can we add this? It was like, you're trying to give us more puppies. We have a kennel full of puppies and we're really busy taking care of them. Like you're trying to give us more of these modules that we have to care for indefinitely, right? It's almost impossible to take something out of the standard lib. So maybe we shouldn't do that so much because releasing the next version of Python means all

the standard lib modules are up to date, got all their patches and any additional features. It was just a bigger software project to corral. And so Steve Dower suggested maybe like a smaller standard library that has a standard distribution that comes with the stuff that kind of got pushed out or something to that effect. And maybe those comes with even more batteries, like a bunch of pypi modules curated by core developers and i thought that's a pretty cool idea like you have a mini python and then you've got oh and it comes with requests and it comes with i don't know pandas or like whatever you name it right it comes with some web framework included i don't know right but it's it's a cool idea and then you could update it independently. One of the interesting arguments for taking things out

is that there are things that are in the standard library that once you get into Python a little bit, you realize that's not the right way to do it. The standard way to do certain things is to use requests, and requests isn't in there. Yeah, almost nobody uses the built-in ACP library unless they are explicitly trying to go without dependencies. Yeah, and there are people that try to do things just without dependencies.

And yeah, it's an interesting thing. Like, for instance, unit test. One of the reasons why a lot of people recommend or say they'd rather use unit test over PyTest is because unit test is in the standard library and PyTest is not. And why is the unit test in the standard library? Because the standard library needs something to test itself with. Right, and that predates everything.

I mean, it's very, like, JUnit-esque, right? It's got some old-school style to it, but, you know, that's the way it was. It can't come out. Like, for instance, if we had the standard distribution, we could possibly put, like, unit test out and into the standard distribution. Right. And then PyTest could be one of those curated modules that comes with the distribution, right? And those are just a couple examples.

I kind of like this idea of a standard distribution thing. I do, too. I mean, because if there's an improvement to one of the modules there, you can pip install, upgrade it, whereas you've got to wait 18 months for it to come out with part of CPython. Yeah. And making another thing is if we make the standard distribution smaller, it makes the process for releasing new versions of Python a little easier because there's less stuff to test. Yeah, I'm not sure if this is a good idea or not. My first impression is I see a lot of benefits to it. I can certainly see some drawbacks, right? Like, oh, you got the wrong distribution. Not only did you have to have Python 3.6 and not 3.5,

but you also need to have the super-duper core. But at the same time, there's a lot of benefits to it, right? Like, the standard Python experience could get better if it came with a bunch of awesome curated PyPI packages. Yeah, so I'll just leave this as everybody involved with this, I'd like to have people just remember to listen to each other and not talk past each other, talk at each other. Yeah, yeah, yeah.

And finally, I guess maybe just a data point, if people look at what Microsoft did with.NET, they had a huge, huge distribution and base class library, and they decided that this is not, doesn't make sense to maintain all the stuff if we're going to have a cross-platform.NET, which they created. So they made a lot of the.NET standard library or base class library basically PIP installable. You get this core thing and then you install the little other bits. So it's not that different than what Steve is proposing. How is that received by the.NET community? I think it's not that different than what Steve is proposing. How was that received by

the.NET community? I think it's mixed, honestly. Like it makes life a little bit more hard, right? Because you can't just use the stuff. You're like, oh, you got to have these dependencies and install it. But it did make it possible for them to have a Linux and Mac OS version that they can now use when before it was stuck in Windows. So they're like, this part only belongs in Windows. It can't go somewhere else. So now it's going to be part of this external thing that you can only get by installing that dependency on Windows. So, I don't know. There's some benefits, but also, you know,

packaging is already challenging in Python, so I'm not sure I want to poke the bear. It's pretty cool. Speaking of pretty cool, something I've wanted to install, but I don't have enough devices to justify is this thing called Home Assistant. Do you know this? Well, we've talked about it a couple times, but I haven't have enough devices to justify is this thing called Home Assistant. Do you know this?

Well, we've talked about it a couple of times, but I haven't tried it yet. Yeah, neither have I, because I think I might have one smart light bulb I bought on accident. I technically have a Nest. My car is electric, so it has a charger, but the charger doesn't integrate with Home Assistant, so that doesn't help me. So I would really love to have it, but I just don't have enough. My home is too old school and too analog for it. Anyway, Home Assistant is this cool web server that integrates all your smart home appliances

and things and collects data about them. It can run as a web server on a Raspberry Pi, and you just leave that in your house, and it does magic, which is cool. Very cool. So Paul Cutler, one of our listeners, let us know that they have now launched a data science portal to process and work with your data that comes from your home backed by Home Assistant. That's pretty cool, right? Oh, yeah. Nice.

You can have like temperature sensors and stuff. Yeah, exactly. And it's already collecting all that data on just because what Home Assistant is. But now they've set up a special way to work with it. So they said, look, one of the core ideas of Home Assistant is all of your data lives on your Raspberry Pi on an SD card, not somewhere else. So they set up this Home Assistant data science website to help you work with that data and analyze it. And then they went so far to create a new add-on for their iOS that runs Home Assistant called JupyterLab Lite. It just runs JupyterLab,

like right on your Raspberry Pi, hosting Home Assistant. So on the same thing that is your Home Assistant, now you have a JupyterLab running there to analyze your data in place. Oh, cool. Yeah, and they also created a Python library called Haas, that's their operating system,

Data Detective, that's based on things like pandas. It lets you start get going quick. So anyway, it's pretty cool. Yeah, very nice. So if people have a smart home and they want to do some data science-y stuff about it, or maybe even build a product for other people

so that they have more data control over their home. Yeah. Yeah, so that's pretty cool. And then finally, I think I finally decided my first IoT project. We'll see if I can actually make it happen. But I've always wanted to build an IoT thing, and I'm always like, but I don't really have any use for one. But I think I have a use for one now.

Oh, tell me about it. Yeah, so you might sympathize with this. I know a lot of people who do things like we do would. So I have a separate office of my garage where I can record quietly away from the kids. But especially in the summer, kids are home, they come over, they want to talk to me or my wife's like, hey, I got to ask you something. And I might be in the middle of recording a course or talking to you, right, recording a podcast. So I'm going to get a big fat button and I can press the button, recording, not recording, and then down by the door before people even get to here, it's going to say

recording, not recording. Oh, that'd be cool. people even get to here, it's going to say recording, not recording. Oh, that'd be cool. Wouldn't that be fun? It's super easy. Yeah. I wonder if you could hook it up to the Skype connection status. Oh, yeah. Maybe so.

Do you have Camtasia running? Are you on a Skype call? Yeah. Oh, if it could be automatic, that'd be sweet. Yeah. Why not, right? Zoom? A couple of those things?

Yeah. Yeah. All right, though. There we go. It just went up a notch. Pretty cool. It started easy. Now it's hard. Now, all of a sudden, I have a machine learning problem. No, all right, though. There we go. It just went up a notch. Pretty cool. It started easy. Now it's hard.

Now, all of a sudden, I have a machine learning problem. No, just kidding. A bunch of APIs I've got to learn. That's cool. All right, speaking of machine learning in the future, what do you got next? Kevin Markham over at DataSchool.io. I don't know if we've talked about him before, but DataSchool.io is pretty cool. Yeah, Kevin's doing cool stuff.

This isn't a lab forum. It's just he's got a lot of cool resources. But he wrote an article called What is the Future of the Pandas Library? And I didn't realize Pandas is one of those zero-ver projects, which is odd considering everybody uses it. But they're considering going to a 1.0 release early this year, and there's an article describing some of the stuff that is coming new and coming with some of the new versions of Pandas. One of the things is we already know method chaining is becoming more popular with functional programming and people used to that.

You can already do that with a lot of stuff in Pandas. What's new really is just that they're going to try to take that further and make more methods that support chaining. All you have to do to support chaining is to return the object that you're operating on as a return for a function call so that you can chain a bunch of function calls. And a lot of times they probably return nothing. So they might as well just return the thing again so you can have this fluent API, which would be nice, right? Yeah. There's also people in the know might know what Apache Arrow is. I really don't know what Apache Arrow is,

but apparently it's something that can help the back end of pandas become a little bit more efficient, and so they're going to try to push that out. It'll probably be after 1.0 because it doesn't change the interface at all. Yeah, it's cool. I don't know much about Apache Arrow, but it sounds cool. They talk about things like working with data that's larger than what you can fit into memory and things like that. So instead of loading everything just into kind of a data frame or something, it's like it'll stream the data off disk and all sorts of stuff.

It's pretty cool. Yep. And I know a lot of people listening to this use Pandas every day. I'm starting to use it more. So the rest of my spiel will be stuff that I don't really know about, but you might. One of the things is apparently it's hard to do custom data types because of some of the limitations. So you kind of have to jump through hoops, but there is going to be extension arrays that make it easier to create custom data types for using with pandas, which

that sounds neat. And then some things that are going away. So some deprecations that have been proposed in place parameter, it doesn't really work as it's supposed to, and it mucks up chaining. So they're going to try to deprecate that. The IX accessor, which is going away or it's deprecating, use loke or iloke instead. The panel data structure, apparently you should use multi-index instead. And the sparse data frame never really worked as it was supposed to, so they're going to just support data frame. And last but not least, in the 1.0 release, they will stop support for 2.7. Woohoo! Viva modern Python, right? Yeah. That's right. Living on. So no more legacy Python. Super.

Okay. I think these things Python. Super. Okay. I think these things are all great. I don't use Pandas enough to really say a whole lot, but I do think, you know, they talk, Kevin talks a little bit about like the zero-ver impression of it not being ready and so on. And it feels like these deprecations are kind of like, all right, these are a few of the rough edges of the API that we wish we could get rid of,

and we're going to call it 1.0, and we're going to drop the few things that we don't really like, including Python 2. Yeah. Yeah. Pretty sweet. All right. Also sweet is DigitalOcean. So if you want to do anything with containers, you got to orchestrate them.

You got to get them to talk together. So check out Kubernetes and DigitalOcean just launched their public Kubernetes service, DOK8. Super simple, managed Kubernetes service. So you can deploy faster, configure your Kubernetes cluster in seconds, and provision and access your cluster in a few minutes. You can scale reliably based on incoming traffic and everything stored in block storage and behind load balancers, stuff like that.

And people are seeing a 2.4 times better price to performance ratio compared to other providers. So if you want to do all that cool stuff or more with a free $100 credit for new users, check them out at pythonbytes.fm slash digital ocean. Now, I'd love to talk about something that we haven't, I don't think we've touched on it very much, but maybe maybe packaging up Python apps. Have we talked about that? Yes, we have. I do remember that now, that three-week stint. And we will more. We will more. This one is, at least it promises to be pretty excellent. So let us count the ways there's more than this. We have pex, which is a way to make a zip file

of Python code executable with its dependencies. We have pyinstaller, which will take a Python environment plus its dependencies and turn it into kind of an embedded Python interpreter plus like a zip file or something to that effect of its dependencies and its source files, and then run that as like an exe or a dot app.'s pyto app there's cx freeze there's many of these right yes the new kid on the block is py oxidizer okay so when you take the py element and you combine it with iron and oxygen through the rust compiler you get the py oxidizer outcome now so py oxidizer is a set of rust crates libraries i'm I'm guessing, is the way to put it, that facilitate building libraries and binaries containing Python interpreters.

Okay, interesting. So CX Freeze, PyInstaller, you're like, okay, great. Well, somebody loves Rust, and they're just like doing it again, but this time with Rust, because it's amazing. But this one has some special capabilities that maybe are better. So it makes a single executable file in exe or.app or something. And all the dependencies and all the resources like pyc files are embedded inside the executable. So like with PyInstaller, you get an exe and then a bunch of loose files and like zip files and directories. And somehow all that stuff gets like put back together to run.

Here you get a single AXE or executable that takes those and puts them inside the binary as a resource and then runs it. That's cool. That's cool, right? Yeah. So it says, of course, the oxidizer part comes from a Rust. And these are compiled from Rust.

And basically, it becomes like a Rust executable, right? And the Rust executable code is responsible for managing and running the embedded Python interpreter in all of its operations. So it's like totally self-contained. Okay, this is kind of cool. It's pretty cool, right? So it's like all these others that I mentioned, except for the produced executables contain embedded,

statically linked Python interpreters, so no dependencies. They have very little runtime dependencies on the OS it runs on. And everything is run from memory rather than extracting temporary Python files to a directory and trying to run them from there with weird paths and stuff. So I'm pretty excited about this.

I haven't got a chance to try it, but I want to. Yeah, me too. It looks really promising. It looks like this is the way it probably should be. So I'm pretty excited if it works the way they're promising. And you have to work with no files on the operating system. You get one file, so it's simple. But not all file systems are simple, right?

Right. And actually, I love it when I can work on a project where I don't have to deal with the file system at all because sometimes it's just kind of a pain. But anybody that's using tools can go, i can automate i want to automate some part of my job and often that involves dealing with the file system yeah and real python just recently put out a an article called working with files in python and at first i i'm like oh cool another another file system thing but it's a pretty nice nice article. It's a very comprehensive write-up. And they cover both legacy ways, like the OS and Sys versions, to do some of these things that I'll cover in just a second. But they also use Pathlib. I'm trying to use Pathlib more

and more, but Pathlib is for more recent versions of Python, and you might not be there. However, it might be that you're used to doing, you've done file system stuff in the past and you want to try PathLab also. So having the examples right next to each other is kind of nice to be able to say, hey, I used to do this in OS and now I'm going to use the PathLab version here. So that's cool. Yeah, yeah, quite cool. I'm not going to like read the article, but a lot of the stuff you might have to do is get a directory listing, what all's in a directory, looking at file attributes, creating directories, doing pattern matching on file names, traversing directories and doing stuff with the files that are there, creating temporary directories and files, deleting, copying, moving, renaming. They include in this article how to deal with zip and tar files, including reading the contents of those. So there's quite a bit of stuff here. Yeah, it's really a really nice and standard comprehensive real Python article. So well done on that. One of the things

that I like about this is it doesn't go, here's a way to read zip files. Here's a way to create directories. It goes, here are the ways, all the ways in the standard library to do this and when you would choose this over that and why that's better and so on. So for example, like with Path from Pathlib, you can say, I want to create the directories, but normally there'd be an exception if it already exists. You can say, it's okay if it exists.

I just need it to be there. Like this is a idempotent type of thing. I'm trying to do stuff like that right or i want to create multiple directories in a chain like how do i create the intermediate ones without loops and other annoying checks so yeah pretty nice yeah it's good to have all that stuff in one place too so yeah it's definitely a good reference thing right like you probably don't need if you if you've got to do all the things listed here at once i don't know what you're doing you've got something going on it's a here at once, I don't know what you're doing. You've got something going on.

It's a little crazy. Like I've got to zip and tar stuff and create directories and get the file attributes, right? But it's certainly good to like have as a reference for when you've got to do one of them. Yeah, but you usually have to do like one or like two of these things. You might have to like to create multiple directories and then read the file in there or something. Yeah, for sure. All right, so this last one that I picked first, Brian, this is a little bit motivated by a conversation we had before.

We had talked about having Python, the command you type on a terminal or a command prompt, being converted from meaning Python 2 to meaning Python 3. That's part of this whole transition, right? Yes. I think what Red Hat was doing was basically saying, there shall be no Python. You have to type Python 2 or type Python 3 on the new Red Hat Enterprise Linux.

And there was some debate about that. So David Furphy sent a really cool thread from pep394. And it says, look, this thing that you sensibly suggested or at least debated on the show recently, it's been tried and it didn't go super well. So Homebrew tried it. Homebrew said, you know what? Python equals Python 3, yes.

And there was a bunch of knock-on effects and they said, we're really sorry. We kind of broke some stuff. We'll put it back. So Homebrew tried it and they had to actually roll it back and there's a link to that conversation. Also on the Pp 394 there's some interesting conversation over on github around it so this pep 394 is allow the python command not to be installed basically other minor changes

so basically requiring you to type python 2 or to type python 3 so there's no way to just type python which to me doesn't feel like a great fix like we want to move to the next version of python requiring you to type Python 2 or to type Python 3. So there's no way to just type Python, which to me doesn't feel like a great fix. Like we want to move to the next version of Python. So when Python 4 comes out, it's going to be like, well, everyone's using Python 3 in their tutorials, and they keep breaking. It doesn't seem very scalable, but nonetheless,

that's what the thing says. So I want to read you a couple of thoughts that Gita Van Rossum had about this okay so somebody said python doesn't exist as a command on mac os so it's solved right so he's like no no python 2 doesn't exist as a built-in command but python definitely does i said however i'm still unhappy with basically people are saying like, if you type Python and that means Python 2, what that is saying is the core developers prefer Python 2 over Python 3. Because if you type the simplest statement to run Python, it does old Python, not new Python. Okay. So there's an endorsement to say, let's not encourage Python 2, legacy Python. Let's have Python point to Python 3. It's what we've been saying, right? So Gita said, I'm still

unhappy with any kind of endorsement of Python pointing to Python 3. When the user gets bitten by this, they're going to be really unhappy. Regardless of what macOS does, I think I would be happier in the future if Python as a command does not exist, and you have to say Python 2 or Python 3.

Right? So anyway, that's just a bit of a follow-up to this Python equals Python 3 discussion we had. Yeah. I'm not super happy with it because I feel like, well, what happens when the next version of Python comes out? Then it just gets complicated in the same way.

But still, it's okay. I think that's our children's problem. That's right. It's like global warming. That's someone else's problem. We're just going to kick that down the road. Let them deal with what happens when you type python and it goes to python 3 not 4 well how many other tools have like do i which grep do you want do you want to grep to i mean exactly you want to run homebrew there's seven choices which version of homebrew you have installed do you want to run like no, no, I just want to run Homebrew. Preferably the latest. Yeah.

Anyway, yeah. I guess that summarizes our thoughts, doesn't it? Yeah. Cool. Well, I do have one quick extra thing. You got any? No, not right now. So I'll just throw this thing out here because it wouldn't be a show if we didn't mention Anthony Shaw somehow, right?

Yeah. He's a good friend of the show. That's right so he wrote a letter to the python community in africa which is a pretty interesting summation of the state of the python community throughout the different regions of africa and really highlighting a lot of cool stuff that's happening over there that's cool yeah so he says look if you look at what they're doing there's actually a lot of stuff that the broader python community can learn for what people are doing there and i'll let you all read the article it's really long and it's just this is an extra right not not a deep analysis or whatever but just to give you

a sense like the attendance in terms of gender of pycon na was 50 male and 50 female like think of other tech conferences where that's the case. Yeah, I've never been to one. Neither have I. Neither have I. It wouldn't be terrible, would it? That looks like the population. Hey, wouldn't it be cool if your tech community looked like the population in general? Anyway, he covers a bunch of stuff like that, and there's a lot of interesting things going on. So I thought I'd give him a shout out there. I know we've laughed a lot on the show, but I don't think we're done. No, we're not. This is what I was waiting for, actually. No, it was all good content, but it was good. So we have our joke.

Are you telling it or am I telling it? You can tell it. Okay. So this comes to us from Luke Russell, who sent in a joke, which is great because it helps us to have jokes if people send them to us. Yeah. All right. So this is a knock-knock joke.

You ready? Yeah. Knock-knock. Who's there? Java. Takes a while to get started. Hold on. We're good now.

We're running. I love that. I told this a couple times at work, and instead of the pause, I counted it on my fingers. Well, as long as they can't see, it's fine. So actually, this has brought back memories of a really funny doesn't directly involve python but it involves open source and if it were made in a later date it probably would be involving python but instead it involves java video which is really funny yeah called java forever and this this is an amazing video so people watch this it's like high production

value i hadn't seen this before and And it's really, really hilarious. Yeah. So there's this family. They love.NET, Microsoft, and they will never stray from them. Like a very authoritative father. But there's a rebellious 18-year-old who loves open source and Java and all sorts of crazy mayhem ensues. And we can't do it justice here, so I'll just link to the YouTube. It's pretty funny.

Check it out. Yeah. All right. All right. Thanks for the joke, Brian. It's pretty funny. Check it out. Yeah. All right. All right. Thanks for the joke, Brian. Thank you. Yeah, catch you later. Bye.

Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.