Python Bytes - #96 Python Language Summit 2018

Episode Date: September 22, 2018

Topics covered in this episode: Plumbum: Shell Combinators and More Windows 10 Linux subsystem for Python developers Type hints cheat sheet (Python 3) Python driving new languages asyncio documentation rewritten from scratch The 2018 Python Language Summit Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/96

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 96, recorded September 19th, 2018. I'm Michael Kennedy. And I'm Brian Ocken. Hey, Brian, how you doing? I'm doing really good. How are you? I'm doing great. There's always something to talk about in the Python space. It never gets boring. Yeah, no. Definitely doesn't. And we're here bringing this to you today in part because of Digital

Ocean sponsoring this episode. So check them out at pythonbytes.fm slash digitalocean and get $100 credit for some server goodies. Tell you more about that later. Brian, you found an interesting one. I don't even know what to make of it here. Tell us about it. I can't even remember where I came across this. I guess it doesn't matter. There's a Python library called Plumbum, and that's spelled P-L--u-m-b-u-m i have no idea where they

came up with that name but it's a toolbox of goodies to do shell like things in python it says shell combinators and more i'm not sure what a combinator is oh there was that y combinator on uh back to the future other than that i don't know what a combinator is. Is it like the pipes between things? Maybe. Perhaps. Anyway, the motto of the library, and this is a direct quote, is never write shell scripts again. And thus, it attempts to mimic the shell syntax. Oh, shell combinator is shell syntax. Where it makes sense while keeping all the Pythonic and cross-platform goodiness of Python.

And it's kind of fun. And it's kind of cheating, actually, I think. But it's cool. We have an example in the show notes, but there's a bunch of examples on their website, on the readme and stuff. You can do things like chaining commands with pipes. And there's a built-in grep and word count and head counts within this library. So if you've got a bash cell script and somebody says, hey, you should convert that to Python, this would take you like five minutes to convert it to Python using this.

And that's not necessarily a bad thing. If you know kind of how to transform something here, you can convert it right away. And then maybe if you want to go figure out how to do that with other ways you could but for a lot of quick and dirty things why not do something like this it looks really cool like certainly writing bash stuff more tests you have to do and the like looping over things it's just like how do you do that in bash again you know it's just it's not so nice but there's certainly things you can do in the shell that are like painful in Python. Like show me the files here.

It's like, okay, well, fine. We're going to import OS and we're going to like do a lister and loop over that, right? Like LS, right? That's super easy. And this kind of brings those things together. It's really nice.

Yeah. So I've got a couple of things I'm going to throw this out right away as we're converting from on in Jenkins, you can do shell commands within like the little command steps of Jenkins. But if you're doing pipelines on Windows, you can't do bash for some reason,

or I can't figure out a way to do bash. But I can do Python. So I could write a little Python script to do the thing that I wanted to do in bash and do it right there. So we'll try that. Yeah, nice. So this is a really cool project.

And certainly if you're thinking about some kind of Bash script or something like that, this is a really nice option. Maybe I should just try to use the Linux subsystem on Windows 10 instead of... Yeah, so if you have a problem with getting Bash to work, it doesn't work on Windows, right? But you can definitely bring in the Windows 10 Linux subsystem.

And obviously that's Windows 10. I'm sure that it's available on the Windows server equivalent of Windows 10, which I honestly don't even know what version that is. But there's probably a server equivalent that has the Linux subsystem as well. People can throw that in the show comments if they want. But there's an article by Marcus Sherman called the Windows 10 Linux subsystem for Python developers. First of all, have you heard of this Windows 10 Linux subsystem? Do you know what this is? Yeah, I've been trying to keep an eye on it. And as they make fixes to it

and add features, I've been trying some of these things out. Yeah, it's interesting. So it's basically Linux embedded inside of Windows. And you've always been able to run like in Windows, like a VMware virtual machine or something and run, say, Ubuntu. But that's not what this is. It's not like a separate, entirely separate running VM, but it's like a little tiny micro VM type thing that is the command line shell for Ubuntu. So you can basically have Ubuntu commands and Ubuntu terminal in your Windows. The idea is somebody says, hey, here is some library you want to use.

And the way you get it is you compile it from source and run it. Or you need to have this other weird utility or type these commands, right? Like how often have you seen a tutorial that says, okay, Brian, the next step is to type python3-mvenv.env, and then you say source.env, then activate. That's awesome, right? It works fine on your Mac. It works fine on my Mac.

It works fine on my Linux server. Does it work fine on Windows? No, you have to use the scripts directory. Scripts directory. There is no Python 3. There is no source. What is all this stuff, right? Why is it different?

And so the motivation that marcus had for writing this is like hey if if you're going to learn python and you're on windows you might consider using the ubuntu subsystem because then all those commands that everybody tells you to do they just work because that's a good idea you could use say anaconda and you get all the tools but you still can't type python but you still can't type Python 3. You still can't type source. Like all these things don't work unless you run Ubuntu for Windows. So it's a really nice step-by-step way to go through it specifically for setting up for Python development on Windows.

Okay, cool. Neat. Yeah, pretty sweet, right? So you might say it's a little cheat sheet for getting Ubuntu working there, right? Yeah. Hey, we're doing really good on the whole transition thing. Yeah, it's working. We've got a cheat sheet on my next item. I've been trying to do type hints a little bit more, especially

so we've got a lot of people at work and myself are using PyCharm and the little pop-up dialog, I don't know what that's called. The autocomplete drop-down when you type dot? Yeah, all that stuff works really great. And it also works better if you have type hints in your API. It's just two things.

So I've been trying to do, at least for the external functions, I've been trying to do type hints in there. But I forget some things. So I ran across this type hints cheat sheet. And it's actually on the MyPy documentation site. But the things that, I mean, I remember some of the easy things, like how do you represent string and int and stuff like that. Coal and int.

What's the problem? But what about duck type stuff? Like if I have a thing that acts like an array, but it might be a tuple or it might be a list or it might be something else that just, I just know that I need to be able to use the bracket index operator and find the length. Well, that's a sequence for type hints, but I don't always remember that if that's an iterator, iterable or sequence or things like that. So those duck type things are what I'm using it for the most, but I have other people

that I work with that are just sort of learning this type in thing. So all of it's useful. Yeah, and it's not that long. You know, a scroll bar estimation is probably seven or eight pages and flip through that once and you pretty much have mastered the type ins. There's a couple interesting things in here that I thought was was nice talking about adding type ins for asynchronous function. So async def, the one that I had never seen was class level variables. So class var looks pretty interesting like it's not on the instance but it's on the type oh interesting

right you can use class var annotation to declare a class variable i'd never done that i'm just you know and that's pretty interesting and then also um specifying the type for say a generator or something that yields some kind of iterator. It's pretty interesting. There's also a generator, actual generator type decoration that takes three arguments that's not covered in here. So that's also potentially worth noting. But yeah, very cool stuff.

And I think this is a nice find. Cool. Nice. Now, before we get on to the next thing, I want to tell you quickly about DigitalOcean. Like I said, they're sponsoring the show, and they have lots of good reasons to go check them out. Something that they just announced that's coming soon and is available for early access is Kubernetes. So if you have some kind

of site that you would like to run on a Kubernetes cluster, you can now go first visit pythonbytes.fm slash DigitalOcean, create an account so you get the free 100 credit and then go to digital ocean dot com slash products slash kubernetes and you can sign up for early access to check that out so then you can submit containers to run on the kubernetes cluster and have simple zero downtime deployments that's actually very very cool yeah i'm pretty excited about uh that coming along that's that's to be fun. So every now and then we cover these sort of higher level topics, a little softer. Here's one that tries to look a little bit farther in the future. And I'm just going to put these out there and then I'll let

you give me your, you react to this article. Okay. So the overall theme here is that Python is driving the creation and foundation of new languages. Okay. So this was an article and it says, here are five predictions for what programming will look like in 10 years. Programming will be, okay. So number one, programming will be more abstract. When I first read that, I'm like, oh, please don't tell me it's drag and drop again or something like this. Right. But no, they say that trends like serverless technology containers, we just talked about Kubernetes, and low-code platforms suggest that developers may be able to work at a higher level of abstractions in the future, not thinking about servers, for example.

And that sounds reasonable. Yeah, to some extent, yeah. Yeah, potentially. AI will become a part of every developer's toolkit, but it won't replace them. What do you think? I don't buy that, actually, but okay. I get some comment I could say to some of the voice assistants, but I'm pretty sure I'll set off like thousands of them, so I won't replace them. What do you think? I don't buy that actually, but okay. You get some comment I could say to some of the voice assistants, but I'm pretty sure I'll set off like thousands of them, so I won't do it.

All right. A universal programming language will arise. Well, Python. It's already arisen. So yeah. So the reason this whole article was pretty interesting is it has a pretty strong Python influence. So for example, at the beginning it says,

look, just in the last few years, some languages have risen and some have fallen. And like the have risen links to an interesting article about Python, the have fallen links to an interesting one about Ruby, for example. And then under this section, there's a nice quote that says,

Python may be remembered as being the great, great, great grandmother of languages, of the languages of the future, which underneath the hood may look like English, but are actually far easier to use. Okay, yeah. Okay. You ready for my reaction yet, or you got more?

I'm ready, bring it. I think some of the things are, one of the jokes about Python sometimes is that it's runnable pseudocode. I don't like it, I don't actually want it to get closer to English, because English has so many subtleties and weird things that you can do that have implied meaning that the closer we get to the actual English language, actually the harder it is to actually debug something. So having it not be that close, I think, is better because it's less up for interpretation.

Right. Please read that sentence. The people over there have already read the red sign. You know, things like that. I think is better because it's less up for interpretation. Right. Please read that sentence. The people over there have already read the red sign. You know, things like that. Yeah. I'm not so sure that a universal language will arise either. There's always something you may want to take most advantage of.

And by doing so, you make it less good for other things. For example, look at like, say, the data science tool set. It's better working with sets of data than individual things. I don't know. It focuses on one thing, and so then other things become less easy with it. One of the changes, which is nice, is

that the languages are not being developed in the vacuum that they used to be. They're learning from each other. Yeah, that's for sure. And we're getting the mega cool feature in one language that we would like to have in another. We're already seeing that. Python's taking some features from other languages and pulling it in.

Other features are taking cool languages. And now I've heard from a lot of people that hardly any other languages have good testing tools like PyTest. Specifically, they don't have PyTest for other languages. And that would be cool if we could have some better testing tools in other languages. Yeah, I mean, there's certainly examples of that, like Swift. Apple's Swift is very much Python-inspired. You see the optional, the null-coalescing operator being moved towards python and 3a potentially things like that so yeah i agree i definitely see it as a uh a blending i don't know if there'll be a single one so all right final two every developer will need to work with

data i think that they're doing that now right yeah i think that's already we always have yeah and then programming will become a core tenant of the education system like reading and writing i believe that i do too it education system, like reading and writing. I believe that. I do too. It isn't necessarily like reading and writing. I think it's more like learning biology and basic chemistry. It's not that you actually need it every day, but it's good that most people kind of understand how it sort of works. Yeah, I'm totally with you.

And the thing I think it will eject from the education system is geometry. I feel like the reason that geometry is taught in middle school and high school is so that critical thinking and proof type of logical thought can be taught. Like you don't really need to prove this thing about isosceles triangles. Like that doesn't help you in life. But the thought process does. And I feel like programming, you have the same type of thought processes, but you get a skill, not just a grade. Anyway, that's my prediction. That's interesting. Yeah. Yeah. We'll see about that. All right. So another one on asyncio, something I've been digging into a ton lately.

Yeah. I wasn't even aware that this was a thing going on, but apparently the standard library asyncio documentation was completely rewritten from scratch recently. That's awesome. That's big news. Yeah, it even says it in the tweet announcement. This is big news with like two flames on the side. So I haven't actually read through this a whole bunch, but it does look like there's quite a bit of documentation there. So that's neat.

That's really cool. And it says they're not done. So there's a comment that this is just the beginning, that they're going to add some tutorials for high-level concepts and APIs, and then they're also going to keep going because apparently there's some new things planned for 3.8. Yeah, that's awesome. Yeah, you highlight some of the things coming for AsyncIO itself,

like the library and the capabilities Async and Await are getting better, right? Yeah, and I think this sort of talks to the importance of async in the future. If not only are people still working on making, improving the async story within Python, but they're also trying to improve the documentation to make sure that it's usable, which I'm glad to see that sort of documentation going side by side with new concepts.

That's a good thing. But that kind of documentation going side by side with new concepts. That's a good thing. But that kind of highlights the importance of async going forward. Yeah, I think async is just becoming increasingly important. And I'm really glad to see it really gaining some momentum. Because if you look at apps written with and without async in a library that can take advantage of it, it's like night and day. It's ridiculous. I think that, like you said, again, it's a different concept sort of thing.

You can think in higher level concepts and break problems up into more logical breaking points. Right. You don't have to think about threads and processes and coordination and all that. You just say, I'm going to do this part, and then I'll wait, then I'll do that part, then I'll wait, then I'll do that part.

And then, you know, that's sort of that higher level abstraction. Yeah. You want to get caught up on all the places that python is headed all the directions all the threads yeah definitely so there was the 2018 python language summit okay that's where all the core developers get together and they talk about what they're doing they they do proposals to each other like hey i think we should focus on this or we need need to address that, and so on. And there's some really interesting items. On LWN.net, which is a pretty interesting site, they've got a lot of great stuff, mostly

focused on Linux. It's kind of like the Craigslist of Linux news or something. I don't know. It's got an interesting design. But basically, this is a meta article about the Python Language Summit this year. And it lists off a whole bunch of sessions, each of which are covered in detail in another article. So what I'm going to do is cover the high-level bits of each one of these main sessions, okay? Okay. So the first one is, could we have sub-interpreter support for Python?

So we were just talking about async, and stuff one of the main problems i don't know if i'd call it really a problem because there's a lot of ways around it actually but conceptions is that python's not great for threading specifically around like taking advantage of multiple cores for cpu stuff because of the gill right it could have 100 threads and only one will run at a time if it's pure python because of the gill so the suggestion is could we take this feature which is apparently already in python called sub interpreter support to create like semi like lightweight sub processes that run within the same process so sub interpreters will allow multiple python interpreters per process and there's potential for zero copy data sharing between them and things like that which is pretty cool that's interesting it might be interesting even for versioning, right?

Like this library requires this version where that library requires that version. Well, you can run like maybe multiple sub-interpreters. I don't know. That's a whole different angle. And it wasn't part of this presentation, but they say the problem is that sub-interpreters currently share the GIL.

So the potential work is, could we make them not share the GIL so they actually can take advantage of multiple cores? Yeah. Right? Like, it's becoming a thing. So, like, my MacBook, my new MacBook here has 12 cores as far as the OS sees it. And if I write Python code and it takes full advantage of all the processor that it can in single-threaded, it's only 8.3% of the capacity of the CPU.

That's a lot less than, you know, 100%. So having features like this to let you do more computation would be great. All right, so I don't want to go too deep in these because it'll make this an hour-long show, so I'll try to go a little quicker. So the next one is modifying the Python object model, looking at CPython's structure.

So Carl Shapiro from Instagram talked about how they might rework the way the interpreter interacts with classes for better performance and memory and stuff because Instagram is all about that. It's one of the interesting things in this article is that there was some controversy around like the presentation and some other stuff. So I'll let you read about it. I won't bring it up here, but it's worth looking at. We just talked about the GIL for parallelism. And Larry Hastings has been on and off working on this thing called the Gilectomy, the removal of the GIL. And the GIL basically is a thread protection around allocation in Python, which means you don't have to take a bunch of fine-grained locks when you're allocating new objects. But because of that, it means only one instruction can run at a time.

So could we take that out and then have a Python that does like true parallelism? Basically, the update is not much has happened. So for better or worse, Marietta talked about using GitHub issues for Python. I didn't realize this wasn't already happening, but apparently it's not. Apparently there's another site that's currently being used

and trying to move over to GitHub issues. That's pretty cool. Lucas talked about shortening the Python release cycle. there's another site that's currently being used and trying to move over to github issues that's pretty cool lucas talked about shortening the python release cycle so right now it's on an 18 month release cycle so in 18 months from whenever three seven was out three eight will be out can we make that yearly it's possibly the conversation so you can read that one that you might find interesting is unplugging old batteries so unplug unplugging old batteries, there's encoding libraries for formats that predate MIME and other weird type of things that are in there that nobody likely uses anymore. So what is the process for taking stuff out of the standard library?

There's a process for it getting in, but how do you get it out? Oh, that's interesting. Yeah, a lot of things could probably be not just completely eliminated, but moved to like just a pip package or something. Exactly. Like this is now out, but if you need it back, you can pip install it separately, but your job is to maintain it. Like we're not chipping it anymore. Something to that effect. It's kind of cool. Yeah. They talked about Linux distributions and Python two with Python two coming to end of life. What is the process for like basically getting those out of the Linux systems that they ship with? And how do we do that? You talked about Python's static typing.

So there's an update on that, mostly focused on stub files. Have you heard of stub files? Yeah. It's like the dot, dot, dot, right? I've looked at them and they basically are the method signature and then dot, dot, dot. And they're not actually the implementation. They're just saying here is the actual shape of this method. And so what you do is they get imported alongside of your code by things like MyPy. Okay. So they're kept in this thing called the type shed, which is a place for third-party libraries to be in there. So you can look through there and see things like what type of stuff,

like the context variable in what is its type definition, for example. So there's a bunch of things with dot, dot, dots. They're just like empty, not implemented sort of structures. And so this is where a lot of the third-party stuff and the standard library goes. But they're saying, look, a lot of packages are not putting things here, like namely things like Django and SQL Alchemy. So there's a proposal of finding a way to bind the type information to the package and not to the central location so that package maintainers can ship those stubs along with their libraries.

Oh, yeah, that would make more sense. Or at least it would make some sense. Yeah, like it'd be easier because then you don't have to say, like, do a PR to some other repo. Like you just put it along with your library and it's in pip. Yeah. It's in PyPI. Okay, so Python virtual environments,

there's some debate about whether they're useful and what we might do instead, primarily around, like if you're trying to teach Python, you shouldn't have to start out with a 20-minute conversation of how to do virtual environments, both on Linux, Windows, and macOS. And then you can actually talk about Hello World, but just how do we just do that? So an interesting debate there. There was some conversation around the whole new decision-making process now that Guido stepped down. Another one about getting along in the Python community, trying to make things like the dev mailer list be more interesting and welcoming as well.

Also finally, mentoring and diversity for Python with Marietta and Victor Stinner as well, two separate presentations in that general area. So there's every single one of these things I've talked about, there's an article about it and more references. And so if you want to dive into it, it's like it'll keep you busy for an afternoon. Okay.

One of the things that you wrote down but we didn't talk about was TKinter. Oh, so the one about getting along and being more civil on dev was somebody complaining that TKinter is so badly out of date that it should be removed from Python, but it's shipped with Python, which kind of brings us back to how do we unplug old batteries? Because if people don't maintain it and it doesn't work for anyone, then why would TK Enter be there? So we might get back on another gooey kick if we go down that too far. Okay. Anyway, that's what that was about. The sort of rudeness on the mailing list was focused around TK Enter and whether it needs

to be upgraded, removed, whatever. All right. Well, that is it for the Python Language Summit. But I thought that was super interesting. And there's just so many looks at where people are putting their effort. I mean, this is where the core developers are focusing their time. So I think it's definitely worth paying attention to. Yeah, definitely. Cool. All right. You got it. Got anything else you want to share with folks while we're here? Yeah, one thing. I noticed this from, I'm going to get his name wrong, Anthony Saudil, Anthony Saudil, Code with Anthony on Twitter, mentioned that pre-commit had a huge jump in downloads on July 30th. And I first assumed that it was because we covered it or something, but apparently it didn't line up with us covering it.

We should have covered it earlier. Could have gotten credit, but apparently it didn't line up with us covering it. We should have covered it earlier. Could have gotten credit, but no. But I looked at other packages and there is a weird jump in almost all package downloads starting at the end of July. I don't have any news around that. That's weird and I wonder what's going on. You know, I wonder if that has to do with when the new PyPI launched. Oh yeah, I wonder if that has to do with when the new PyPI launched.

Oh, yeah, I wonder. And it's almost coincident with the PIP-18 as well. The new PyPI launched in April. So, yeah, it could be with PIP-18. Who knows? I bet it's some low-level infrastructure thing, like caching got expired and everything that had cached it everywhere got a new one or something. Yeah, maybe they changed the way they were caching it or something i do i do remember seeing that from pip that it every now and then like if you run a sufficiently old pip it'll say that it's like it's offline cache is not

compatible and it'll re-download so i'm wondering if in the reverse direction when a new pip came out that may happen so i bring that up so that if anybody knows what's going on they can get a hold of us and we can find out yeah throw in the show notes the show notes. Pythonbytes.fm slash 96. Yeah. And that's one of the things that I love about this is because we can talk about things that we don't quite know about and people will correct us. They're generally pretty nice about it too. So thank you for that. Yeah. So there's one of the things that somebody mentioned, we've talked about PyCuda. KudPy. Did I get that right? I think it was Kudpai. Yeah. Cudpai. Yeah, Cudpai. Cudpai. We were guessing that maybe if you had a video card on your laptop, maybe you could use it,

but we were corrected that it would have to be an NVIDIA card because Cuda only runs on NVIDIA. Yeah. So apparently Cuda is a NVIDIA thing, which I didn't realize. I thought it was just a graphics card standard, but now it's an NVIDIA thing. So this is all triggered by your question, could I do this on my MacBook or on my laptop? If you have like a Windows or Linux laptop that has an NVIDIA card, yes. Otherwise, no, apparently is the answer. So a couple of people wrote us and communicated that to us. So thank you all for that. I guess if you're on a MacBook, you could get an external. On the new ones, you can do external GPUs and you could get an N. On the new ones, you can do external GPUs, and you could get an in-video one there.

But even the one sold by Apple is actually a Radeon, so it doesn't help you. Okay. So anyway, just thanks for the clarification. Yeah. Do you have anything else to add? Nothing else. Not right now.

I have some good news next week I'm going to share, but right now, nothing to say. All right. Well, thanks a lot. Just working away on projects. Yeah, thank you. Great to chat with you as always. Yeah. Thank you for listening to Python Bytes.

Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy.

Thank you for listening and sharing this podcast with your friends and colleagues.