Python Bytes - #92 Will your Python be compiled?

Episode Date: August 25, 2018

Topics covered in this episode: IEEE Survey Ranks Programming Languages MyPyC Beyond Interactive: Notebook Innovation at Netflix How to create a Windows Service in Python An Overview of Packaging for Python PEP 505 -- None-aware operators Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/92

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 92, recorded August 23rd, 2018. I'm Michael Kennedy. And I'm Brian Akin. Hey Brian, how you doing? I'm doing great. Yeah, same here. I'm excited to be back together and I can't wait to talk about some of these things. We have some like amazing drop the mic sort of stuff coming up, so pretty excited to get into that. And also pretty excited

that DigitalOcean is sponsoring this episode. So thank you, DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. Get $100 credit for new users. Right now, have we covered the fact that Python is popular? Yeah, but I wanted to cover it again. Why not? So Python is pretty popular. And for the second year in a row, it came out as the number one popular programming language in the IEEE survey. So every year, IEEE puts out a popular programming survey. And that at the top isn't really news. What's really kind of cool about it, though, and we'll have a link to the results, it's an interactive results thing. I think we covered it last year. But this time, Python is number one, even in embedded. And that's the thing that stands out this year. Yeah. And it's worth pointing out that this survey is for electrical engineers. It's not like the same as, say, the Stack Overflow survey, which is maybe more developer-focused. So it's like you saw another area where Python is sort of taking over, which is awesome. Embedded, like you say.

The article that we're going to link to actually talks about whether or not this is real. And some people are doubting that Python really is that popular in embedded. Because, I mean, there's a lot of day-to-day job are embedded. The metadata work I do is in C++. There's a big range of what it means to be embedded. So there's not, they don't really define that. You have to kind of define that yourself. I define it mostly as you're developing software on a platform that's not the target platform. And the target platform doesn't really look like a computer. Like a Raspberry Pi or maybe like something embedded in your car or something like that? Yeah, definitely. But even it's a broad definition. So the embedded software I do

actually runs on a Windows machine, but it doesn't look like a Windows machine. It looks like it's a testing equipment. It looks like a stereo component with dials and stuff we don't run python we run c++ however like you said like raspberry pi i think there's been a big push lately i think that's where this is coming from there's been a big push lately of trying to get people involved in more circuits and more real interactive uh programming with the electronics through the use of micro python andython on like Raspberry Pis and Arduinos and micro bits and all that sort of stuff. And including the Mew editor that really tries to teach people right at the get go. So there is some doubt of whether or not really

Python's the top for embedded, but I don't doubt it at all. I think it's going to stay. Yeah, I think it's cool. I mean, you could have the debate of, well, it's on professional shipping embedded devices that are for sale versus embedded in general, which like you say, could include $5 Adafruit type things that people play with or they're using in school, but it's not for sale per se. Right. There's a lot of people taking those Adafruit things and turning them into kits for people or products for people to wear. And then the people that are learning on that are going to go into and might go into other embedded jobs. And I think since Python is faster to develop in, there's a big incentive, I think, for people to use a higher level language in embedded if they can, even in a production level. Yeah. And if they are, it's got to be Python, right? Like what else would it be?

Right. Yeah. Yeah. Maybe JavaScript, but that's a little shaky. Very interesting. All right. So are you ready for something incredible? Yeah. Is this drop the mic thing you're talking about? Yeah. Unfortunately, it's the second one, so I have to pick it back up. But let me work my way into this the way that it develops. So we have the mailing list, Python dev, okay? And that's where a lot of the core developers talk. Okay, on this list, someone was talking about finding a better way to test the C APIs

instead of writing some C tests, some Python tests, and those kinds of things, right? You can be sympathetic with this, I'm sure. And they were saying, look, we need to test the C code inside CPython. And one way to do this would maybe we could use something like Cython. And we talked previously about how Cython lets you pretty easily integrate with C and Python at the same time. So there was some debate saying, you know, it might be a good idea, it might not be a good idea. Like on one hand, Cython is changing a lot. It's a pretty big dependency to take as part of the core development component of Python itself, things

like that. But then something pretty incredible came along here. And I don't know how far along this is or whatever. But this guy named Yuri steps in and drops a bit of a bombshell. And he says, hey, you guys, the Cython thing is kind of cool, and I don't know if you want to do it. But over at Dropbox, we're working on a new compiler for Python called MyPyC. And it just says, well, this MyPyC will compile type-annotated Python to an optimized C code, which ultimately compiles to machine instructions. That's pretty cool. How about that? So it's like Cython, right?

And we've seen that where Cython can make things like 20 or 50 times faster in certain circumstances because it's compiled. Really interesting stuff there. But the problem with Cython is it's a superset of Python. You have like the C def and like you basically write in a separate language that if you squint, you're like, wait a minute, is that Python? No, no, no. There's a lot of weird otherness, other stuff going on around here. This is not Python, but it's Python like, and it doesn't even use the same type annotations

because it pre it sort of predated that. So Scython is good, but it has like this, you got to write in Scython, not Python. This mypyc thing they're talking about, this is straight Python 3.6 code. It runs on CPython, but you can compile it to machine instructions like Cython. It's not all of Python. It's a subset, right? It's a subset, exactly. And so I think, you know, the main thing is you have to type annotate everything so it can actually compile it.

That's more or less the difference with Cython is you type annotate everything, but it's a different type. It's using cython types right but this is using just normal type annotations uh that's it's pretty incredible it says look this is standard python it's a subset of python and you can easily integrate with c libraries because of cffi you can even run it on pi pi because it's a subset of python so uh it's pretty cool and if dropbox releases a full-on compiler for python that would shock the world i think yeah definitely like even with this with a subset to be able to develop it in python and then um do some little extra work to go to the to make sure you're able to compile it within this this would be great and the extra work is not much it's like put type annotations on it so maybe you have

like three or four functions that you call that are like really called frequently or really a major portion of your time is spent there and if you wrote it in like c it would be way faster well if you just put type annotations on it you know you could probably hit it with this thing so a pretty interesting uh little thread that got picked up there we'll have to watch this and make update people if we see any more progress yeah what i did not see is here's a link to the git repo or here's the announcement or anything like that so we'll have to see if anything comes from it but i think it's really interesting you know that that we've sort of gotten to this place because from my understanding is guido is not a huge fan of compiling python code and didn't see a need for this but he must be involved in this because it's from dropbox and the main work he's doing at dropbox is on my pi and it's all about this so i suspect he's involved

somehow but i have no confirmation of that. Yeah. Okay, cool. Boom, mic drop. Okay, I'll pick you back up. Let's talk about Netflix. They are doing such cool stuff with Python. Yeah, so it's Python and more. So there's a kind of a really cool article

called Beyond Interactive Notebook Innovation at Netflix. And as we've all known, Jupyter notebooks have, and using the Jupyter system has kind of revolutionized a lot of data analysis and machine learning and quite a bit of that interactive, interacting with data environment. And that's true at Netflix as well. And that's what kind of this article is talking about, is a lot of the cool things that they've done. And it's over the last few years, they've seen a larger and larger growth. And they've also said, basically, this is our supplied data chain. So there's, I never

really thought about all the different types of people that interact with data, but they listed, they have a little like diagram, they listed like nine or ten different job roles that interact with data a little bit differently, but they're all interacting with it. Data experts or data analysts or algorithm engineers or data scientists, there's all these different things. I'm not quite sure what they all do different, but they're all interacting with these huge data sets. And in their effort to use notebooks and Jupyter to kind of streamline and make things easier for people, in the process, they've came up with some really cool extra projects that they've

open sourced or developing in the open. And one of them is, I'm going to run through a few of them. One of them is called Interact, and it's an alternative user interface that has some cool things like inline cell toolbars and drag and drop cells. I didn't know that you couldn't do that in Jupyter, but that's neat. And a built-in data explorer. And then if you've got a Jupyter notebook all set up with some data, well, wouldn't it be cool if you could have some of your states parameterized so you can run the whole notebook with different parameters? This is almost like making a notebook like a function you can call and pass it inputs and get outputs.

Exactly. And that's what Paper Mill is partly. So there's a project called Paper Mill that parameterizes notebooks. And then it also does things like if you've parameterized a bunch, you can analyze the different set of parameterized runs and do some analysis on the set. And then there's, yeah, it's kind of making a notebook kind of, like you said, like a function that you can call.

And then if you're using that and you want to share it with other people with the data and with the results, they've put together a thing called Commuter that lets people share them with each other. And then also scheduling. So there's a separate article that's attached at the end about how if you've got this function with different parameters and stuff, maybe you want to test different live data, so you want to schedule it regularly or at different times. They've got a notebook scheduling

system in place. So all of this is, it's not like a super easy thing to jump into, but they're trying to make it easier for people. And this is actually some pretty neat things that they've shared with everybody. This is incredible. And if you look at this big infrastructure diagram they have, it's like, it's just this entire architecture and cloud platform for running their style of executable notebooks using this new UI on containers you can configure and schedule. It's really crazy. And you can even come down and say, I'd like to run on some container with four CPUs and 30 gigs of RAM, go. Right? That's crazy.

It's kind of amazing. And I'm trying to start exploring a little bit more of what people are doing with Jupyter notebooks and other parts of the Jupyter system because it is sort of opening up a lot of different data analysis into the open, which is neat. Yeah, that is really cool. Something else I just see in here, and I don't know quite where this shows up, but in that same organization that released all of these

projects on GitHub, you know, they're grouped into a bunch of projects under an org and the GitHub org style. There's a thing called VDOM for Python. What does it say? It says a virtual HTML DOM for Python python which is pretty interesting for creating basically html and and whatnot like divs and paragraphs and stuff so that's also in here somewhere for whatever people want to do with that interesting yeah so i feel like this one is massive too like this is a really major thing and you know organizations that depend on this notebook style like this could really be a big deal and i like that they split it out into different projects like uh the parameterization

idea is is powerful and maybe you can use that by itself and the um if you don't quite like the user interface and uh so they've got the interact there that you can use that instead these are cool that they've split them out a bit i I don't know how much they're tied together. I expect things like Titus and Commuter might depend on other pieces more than others, but it's neat. It's really neat. So another thing that is pretty cool

that I want to talk about is DigitalOcean. And I've told you a bunch of things about it last time we talked about projects, and that's pretty awesome. Brian, you've heard of Heroku, right? Yeah. Do you know what Doku is? Daku? I think it's Daku actually. No, I don't. So Daku is apparently a miniature Heroku powered by Docker. So if you want to have your own Heroku, you can spin up a Linux machine or machines and install this thing called Doku and then treat it like Heroku for deploying and managing your code oh that's neat that's neat so over at digital ocean they have a one-click deploy for creating

doku servers so you should go over there create a new droplet and you just choose i want a doku thing all set up click boom you're ready to make your own little version of heroku wow that's really cool yeah it's really cool so they have all these cool like one-click things to create different types of environments. And I just thought I'd highlight this Doku one because I know people use Heroku a lot. And if you want some flexibility, this is a really awesome way to do it. So anyway, check them out at pythonbytes.fm

slash digitalocean. Get $100 credit if you're a new user to play with Doku or whatever else you want to play with. All right, so remember a while ago we talked about running a python script as a systemd service yes that was a while ago it was it was quite a while ago i'll link to that uh when we talked about that as well that episode but the idea was i would like to take some python code and make it you know turn on when i boot my linux system the idea was to take a Python script and make it just boot when

Linux boots and run in the background. It's kind of like if you did a cron job, but you have more control, right? Like your app, your Python code, it's constantly running. If it crashes, it'll restart if you configure it that way and stuff. Well, that's great for Linux, but what about for Windows, right? So the equivalent infrastructure in Windows is called a Windows service. It starts when Windows starts, if you want it that way, it can run when you're logged out, or it can run as a different user, it can run as a restricted account, all these things. So if you want your app to run basically, as Windows runs, right, then you have to create this Windows service. Well, it turns out it's quite easy to do this in Python

if you use the right libraries. So there's a cool article that just says how to create a Windows service in Python, and it's based on this project called PyWin32, which seems to have no documentation. Maybe I just missed it, but it doesn't seem at all obvious how to use. But it is on GitHub and pretty popular. But this article talks about how to create a Windows service.

And the idea is you just derive from a certain class and you override a stop, a start, and a main method. And boom, you have Python code that runs as if it were part of Windows. Actually, this is awesome. I'll use this right away. Yeah, yeah. It's really cool. I don't have a good – I've written a couple of Windows services back in the day. I haven't had a chance or reason to do so recently.

I did take that system D idea and do some really cool stuff around my courses for automation there. But if people are on Windows or they have to deploy to Windows, I think this is a great little example. So if that appeals to you, if that sounds like something you want to try to do,

this is a really accessible way to do it. That's cool. Yeah, indeed. I'm glad that this came out because I thought about doing this before. I'm like, oh man, I'm a smart guy. I can figure this out. I don't even know where to start looking.

There's like weird registration stuff you have to do, but it's cool. You just say like the name of the service is this, the description of the service is this, the way it starts is like on system start or delayed or whatever. And then you just like run a command to install it. It's great.

Yeah. You probably want to package it up though when you send it out yeah you should package it have we talked about packaging before i think so not enough it's one of the things that should be more uh more accessible okay well let's talk about it again so the mamu dashemi a while i don't remember how long ago this was i don't know if we covered it or not. But anyway, he wrote an article about different ways to package different levels of Python projects. And then he did a talk on it. And then now it's been, all this information has been shared with others and edited. And it's part of the PyPA, the Python Package Authority documentation. And it's called an overview of packaging for Python. And it's called an Overview of Packaging for Python. And I kind of like how it's broken out

because when you want to share Python code, there's different levels. So if you just want to share a module with somebody, there's that. And then so, I mean, it starts there and talks about how to just share a simple module with somebody else. But then you quickly get into packages

with source distributions and wheels and possibly binary distributions and how to do that. But that's complete. Packages are different than if you want to share an application. So that's the part where I get fuzzy. But actually, this is a good starting point that hopefully they'll keep current that talks about some of the different ways that you can share applications because you know sharing web applications going to be different and if you share it on heroku it's going to be different than sharing it a desktop application or a command line application or something so and then also whether or not you want to assume people have python or whether you want to package Python with it. And what do you do with the dependencies and all that?

So we've talked about all this before and there's a lot of different solutions. And so this is a good, a good starting point to take a look at if you want to share some code with somebody else, start here and then get lost. There you go. I like it. It brings a lot of the stuff we've talked about together. Like for example, in the bring your own Python executable section, it's like, here's a whole bunch of ways to do this. You could use PyInstaller, CXFreeze, Constructor, which apparently I'd never heard of. OSnap, OSnap, another one. Like there's a bunch of things in here and I actually have only

heard about half of them. So it's pretty cool. And since applications kind of range, everything else is fairly consistent. There's like one obvious right way to do it up through packages. And then it's with applications where it starts going off and feathering out into different solutions. I think it's because there's a lot of different requirements for applications, different than just sharing source code. So I don't think it's a bad thing that we have a lot of different solutions right now. We've talked about how it'd be really nice if there was one obvious way. We're just not there yet.

So here's a good list of some of them. Yep, that's really cool. Well done, Mahmoud. So there was a little bit of drama a few weeks ago around PEP572 about changing the Python language for an assignment in place assignment operator. Yeah. And that was actually, that was part of the, in the Python language for an assignment in place assignment operator. Yeah.

And that was actually, that was part of the, one of the sort of straws that caused Guido to say, all right, I've had it with this. Tired of fighting over these things. I'm out. Well, here's a new pep that I think proposes a bigger change to the Python language. This one is not accepted. It's in draft mode. So people can respond to it. However, I actually think the value this one brings is massive.

It's a big deal though. It's a really big deal. So the idea is there's PEP 505 now, which is for none aware operators. And the idea is there are several languages that are proven. There's some nice design patterns or language patterns that short circuit working with none or null or whatever it is in the Swift. I forgot about what they call it there, but nothing, right? So there's basically two ways in which you work with this. One of these are called null coalescing operators, which lets you substitute a value if you have a none object. And null aware operators, which lets

you chain operations regardless of whether with their null. So the two languages that are most popular that come to mind have to be C sharp and Swift. And both of these have deep support for this concept. Swift takes it to a whole nother level to say, you can't even assign the equivalent of none to a variable unless you mark it as nullable explicitly, which is way far out there. So the idea is like Python could benefit from this. So there's these two cases, the null coalescing and the null aware. So I'll give you the null coalescing, right? It's if you're going to do some kind of weird test, like you'd say something like value, if value is not none,

else missing, right? Like this big, complicated thing could just be value, question mark, question mark missing. It's either going to be the value, but if value is none, then it's the other value missing. What do you think of that, Brian? I'm still on the fence. Okay, this one to me actually doesn't offer a huge amount of value. It's okay. The next one to me is like pretty killer, and more with the fluent style of calling functions chained together so the null aware member access operator it's the same basic pep and the same basic structure but it's used in a different case it lets you chain these fluent interfaces together without testing for none so suppose i have a user the user has orders the orders you can call a first operation

but if it's there's no orders then it returns. But if it does return one, the order has a name, right? So you would say like, if user is none, return none, user dot orders at first, if the first order is none, then we kind of bail out. But if you make it through all those tests, you can get first order dot name. So that's a lot of code to be spoken over over the air. But you now, with this new proposal, say user question mark dot orders dot first question mark dot name. And it would mean exactly the same thing. Okay. If you look at those two lines of code on digital paper, the amount of space on one and the other is ridiculous, right? Yeah.

Sorry, I didn't mean to cut you off. First reading of the non-aware operators, I was thinking it's just going to make the language more complicated to teach because when you're reading it, it isn't obvious what these things are doing. However, it does make the language smaller in that you can be more expressive in less code. And that's a good thing. I think it would make people more willing to effectively test for none when it could be none without cluttering the code, right? Like these two examples I gave you, it's like eight lines versus one line. You're willing to just go, oh, I just think it's there. We'll just go through it. And then you'll get none type does not have attributes such and such, right? Whatever the attribute is, right? You get that error all the,

it's one of the most common errors in Python. But if you just put the question marks in there, that totally goes away. That's the thing where if you don't think you need that problem just think about how many times you see that like you know none none does not have the index operator yes exactly exactly and this basically provides a much nicer way to deal with it so my vote is for this feature especially the null aware member access yeah but i would be okay if Python never had it, but I do think the languages that do have it really make pretty good use of it. So it's in draft mode.

People can give feedback before some sort of battle erupts over it. Yeah, I think it's good. Yeah, I do too. Cool. It'll let you have a second edition or third edition of your testing book, right? Because there's probably somewhere

you got to put a question mark in there. You're just adding work for me. Thanks. Anyway, it's something for people to check out. Quite, quite good. I don't have anything extra to share with the folks. You, Brian? I've got a couple of things. I just noticed the other day that the Pi Cascades, that's going to be in February, February 23rd and 24th of 2019. And it's going to be in Seattle this time. It's the call for proposals is now open. Oh, that's sweet. You got a long time too. You got it now through October 21st to get in your proposal. Oh, I'm super excited about that. The opening, the inaugural Pi Cascades in

Vancouver, BC last year was excellent. And I'm really glad I went. It was much smaller and it was a really intimate event. You got to meet a lot of the people who were there. And I'm looking forward to have it in Seattle. And it's coming to Portland next year. So that'll be even better. I'll try to go and I'll also try to submit something. That'll be good. The other thing I wanted to highlight is I just wanted to brag because I got a test and code episode out a week or so ago.

That was with David Heinemeyer Hansen and it really went well. And, um, he's not a Python person, but it's a interesting information. So that was fun. All right. So for people who don't know who David Heinemeyer Hansen is, what's his claim to fame? Well, he's got lots, but he's the guy that, that made up Ruby on rails and he's one of the co-founders of base camp. Yeah, just that. So that's pretty awesome. Yeah, it's really cool that he was on your show and I'm looking forward to checking that out myself. It's definitely applicable to Python people as well because we talk about language agnostic things like just testing and stuff like that. Very cool. Well, thanks for the

call out on PyCascades. Sure call it pycast kids but i also i was trying to i was catching up on your podcast talk python and it was uh they had a couple episodes that were people that learned python after the didn't get um degree like cs degrees and learned python later programming later and just the whole concept of that and listening to everybody's stories was very interesting i like those yeah thank you I did a two-part series on that. And that's, I think it really connected to a lot of people because so many people find their way into our world of programming in Python without going through the traditional CS degree path. And I think it's just an example of a lot of people doing really great in alternate ways in.

So, yeah, very cool. Well, good talking to you. And we'll talk next week. Yep, you as well. Catch you later. Bye, everyone. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S.

And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.