Python Bytes - #50 Bundling , shipping, and protecting Python applications

Episode Date: November 2, 2017

Topics covered in this episode: Think Like a Pythonista Serpent.AI - Game Agent Framework MkDocs PyInstaller 3.3 released PEX: A library and tool for generating .pex (Python EXecutable) files Using Cython to protect a Python codebase Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/50

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 50, recorded November 1st, 2017. I'm Michael Kennedy. And I'm Brian Ocken. And like always, we have some pretty awesome stuff lined up for you, but I want to say thanks. Thanks to DigitalOcean. So DigitalOcean is sponsoring this episode, and they're going to share this new thing they've launched with you guys called Spaces. It's like S3, but way, way better.

I'll tell you more about that later. Right now, I want to be a little philosophical and think like a Pythonista. Yeah, think like a Pythonista. We're going to highlight a video from Luciano Romalo, and he did a talk at PyBay 2017 called Think Like a Pythonista. And one of the things, actually,

when most of my programming i don't really create new types that much in python i take advantage of i don't really think like i'm creating new types like i'll use um named tuples or something but even so i take advantage of all the built-in types and that's kind of what part of his talk is about it's he create, walks through talking about creating another container type called a deck of cards. So he's got a cards type and a deck type. And the deck has to do, is a container, and it has to do things like indexing and shuffling and stuff. And he walks through how to do that so that you can use the built-in iterate for loops and

accessors it's just a really kind of a fascinating look at how how little code it takes to make something look like another any other python type using some of the dunders and stuff yeah for sure i think he did a super job with this and this comes from his book book, Fluent Python, which is absolutely one of the best books, once you know a little bit of the language, to really get the zen of it. And he's really big into the Python data model, the Dunder methods, and all that kind of stuff. And so I'm sure this is a really great way to see how that comes out in a custom type, right? Yeah, it is. And also, I think it's a good talk for somebody to watch, even if you don't think you're going to ever create a container type yourself, because you learn about a lot of the things that your built-in types do for you that you don't even really think about.

There's quite a bit of information in there that's worth watching, even if you don't feel like doing it. Plus, he does the entire talk, or most of the talk, with a Jupyter notebook and even changes some of the code and fixes a bug and does monkey patching with live code. It's pretty cool. Yeah, that's pretty cool. I think definitely this concept of Jupyter notebooks, the whole scientific computing stuff is bleeding over into all sorts of presentations. I think it's great. Near the end of the talk, he does an example of why operator overloading is awesome and he shows a little numpy a little tiny script that can graph the trajectory of

a launched item you know like throwing a ball or something and it's so clean in such little code because the numpy library is allowed to take their data types and overload operators so that multiplication and addition work on these large objects. Yeah, that's awesome. It's pretty great. Yeah, Luciano definitely has some special sauce he sprinkles on what he does. It makes it really smooth.

So check that one out. That's a really great one. That's a live presentation from basically derived from his book. A link to this, of course, but there's also a link to the rest of the PyBay 2017 playlist. Yeah, that's cool, for sure. So we talked about how neither of us are really gamers last time, right, Brian?

Right. Yeah, but one of the really interesting possibilities, you know, Python is probably the biggest language for machine learning and AI. And one of the big possibilities would be like, what if you could take the AI you're trying to teach about a world and like immerse it into a game

and just make it be in the game, right? So if you wanted to have like a self-driving car, like could you stick it into like a car simulator game and make it race around and try to win? Or, you know, you want to have a puzzle solving one, stick it in the middle of a Go game and make it try to play Go. So the thing I want to feature this week is something called Serpent.ai. So that's both the middle of a Go game and try to make it try to pay Go. So the thing I want to

feature this week is something called serpent.ai. So that's both the name of the product and the website. And it's a game agent framework. So this is pretty insane. What you can do is you can take any video game, and you can plug in says any, that's kind of intense, any video game, and you can plug in AI and bot programming with python what do you think about that wow i think that looks fun i know it looks super fun oh my gosh and they have a the guy who created it i'm sorry i couldn't find his name on here he runs a twitch channel so you can drop in on his twitch channel which is linked to from serpent.ai and actually pulled up one particular video that he recorded on Twitch,

which is like streaming gaming stuff. And he'll do like six hour coding sessions where he's like, we're going to take this game, we're going to plug in AI into it and solve it in real time. And it's pretty intense. Wow, that's cool. If you want to spend like a day

watching somebody like plugging in AI to a game, you can do it. That sounds neat. So I do know that, I mean, Twitch is being used a lot by a lot of game, people watching gamers and stuff. But do do know that, I mean, Twitch is being used a lot by a lot of game, people watching gamers and stuff. But do you know if anybody's doing or if it's catching on much with like live coding examples and stuff?

I was thinking about that as I was watching. He does do live coding. He's got like PyCharm and Jupiter and the game going and like console stuff in the video that I linked to. You know, I honestly don't know, but I feel like there should be if people aren't doing it. Probably is. Probably just not in my circles. Yeah, yeah, exactly. We don't do enough gaming, right? So some of the things they talk about in this, it says like there's really no conventions based on this. You can just plug in whatever you want. So if you

want to use like the latest reinforcement deep learning algorithm from Google, try that. If you want to use computer vision, we talked about one recently, Luminoth, you could plug that in there, for example, to do like image processing. And it's even if you want to just randomly press buttons, you could try that as well. So I'm guessing this game, it has to be something that you can like get at with Python library or something, right? Or I mean, I mean, I can't play my Xbox, for instance. No, I think it's got to run on a computer, but I think it simulates arrow keys and stuff like that. I'm not entirely sure how it

determines whether it wins, because you've got to give positive or negative feedback to the deep learning models, but something's happening in there. It looks pretty cool. Maybe you just have to hang out with a microphone and say, good job. Yeah, just pat it on the back. Well done, AI, well done. So you've been working with Cookie Cutter lately, right? I have. Most of my work has been making some plugins for work, but I'm utilizing the PyTest Cookie Cutter plugin. And that caused me to, I hadn't really, one of the options was to generate some project documentation with MakeDocs, and I never really used them before. I don't know if it's pronounced MakeDocs. That's how I've been pronouncing it. I think there's a little bit of debate, but I think that that totally works.

Okay, so it's MKDocs, and I think it stands for Markdown Docs. And a lot of, I know about Sphinx, and I know that a lot of the Python documentation is with restructured text, but I just don't really like restructured text. So anyway, I just wanted to highlight MakeDocs. I thought it was neat, and I've been documenting some of my projects with it, and especially with CookieCutter, it just pops it in there, or at least this PyTest plugin, CookieCutter, it pops it in and creates a YAML file for you. And really, it's kind of neat because you you have this configuration of like what your theme is and

all that. But your writing is just a directory with a bunch of markdown files. Markdown is a really great invention. I'm really glad it exists. The one thing I was worried about is I do kind of want to keep some of the like the library API documentation with the code, not somewhere else. And I do want to document that somehow. So I did stumble across one of the friends of our show, Christian Medina. He wrote a Medium article called How to Write Your Own Python Document Generator. And essentially, it's using MKDocs also, but walking through your code and pulling out the doc strings.

So I'm going to give that a try next week and see how that goes. Oh, yeah. You'll have to give us a report how it came out. Yeah. Awesome. All right. This audio file that you're listening to right now came to you through Digital Ocean Spaces. And like I said at the beginning, they're sponsoring this episode, as they are a number of them.

And you may know of Amazon S3. It's a place where you can put your files both for people to consume, browsers and stuff, but also for your app to integrate with. Well, DigitalOcean decided to take it to another level of spaces and create something similar but with a much better business model. You pay five bucks a month,

you get 250 gigs of storage, a terabyte of outbound bandwidth, which by the way is like $93 just there on its own in S3. So five bucks, that is a super good deal. And inbound data is always, always free. So it's way more predictable. You can save tons of money and it has the same API as S3. So all your tools and stuff still work as well.

It's definitely recommended. I'm using it more and more every day. And like I said, using it to deliver all the audio for this podcast. That's really cool. And so, like you said, they're sponsoring this episode, but they also save us money on every episode since we're... That's right. They're kind of sponsoring everyone in a little bit of a sense, right?

Save us some good money. So check them out at do.co slash python. Now, you and I, I think we were kind of breathing the same air. Something was in the air this week here in Portland because I chose an item that has to do with taking your Python applications and packaging them up for reuse in one way. And you chose the very next item, packaging up your Python applications in slightly a different way. So let's do some two interesting takes on this. Neat. So there's a couple of programs or libraries that you can use to build proper applications

out of your Python code. I feel like deployment to end users is one of the things that's really tricky for Python. This is one of its weak points. You've got a bunch of scripts. You're going to send those out. They have dependencies. You can't give a regular user a Python thing and say, oh, yeah, run this pip install this to like configure it, right?

That's not going to work. You have to go, here's an exe. Here's a dot app and so on. So you can use CX free. There's a few others. Pyto app, Pyto exe. But one that's really nice is called PyInstaller. And they just released version 3.3.

So this works on all the operating systems, Windows, Linux, Mac, even FreeBSD, Solaris, AIX, a bunch of them. And the idea is that its primary goal is to do what I described, but also be compatible with third-party packages out of the box. So a lot of the weaknesses of the other ones that I've mentioned, they'll run into trouble trying to use Matplotlib or numpy or pyqt, for example, which is super painful. And it'll take that and package all those libraries up for you. And they just released the 3.3 version, which has Python 3.6 support. Oh, that's cool.

Do you see yourself maybe using this anymore? I definitely could see using this, especially, like you said, if you're sharing an application with somebody that isn't a developer it's good right yeah if you're going to ship some end product you want to run it in python that's pretty awesome and you can use the the one the dash dash one file option and dash dash no console and it can just be like a gui and you wouldn't really know that it's not like built in c++ or dot net if you run it on windows or you know coco on, things like that. It's really, really nice. Nice.

Yeah. So what's your take on this? On this one, I think I definitely have to try it. But the item that I brought up I was thinking about is Pex. And it says it's a library and tool for generating Python executable files, Pex files. And the files are, yeah, they're self-contained executables, but there's executable virtual environments. Right, whereas the one that I was thinking of

kind of packages up all the Python things you need to run into an AXE. This is like, can we just package the virtual environment and somehow teach it to run, right? I'd have to look, but I'm almost positive there is no, it doesn't package the Python with it. So if you have everybody already having python installed but you want to share things around with share python programs this might be

a really cool thing and it looks like i just got introduced to it the other day but it it looks like yeah it came from twitter and it used to be part of the twitter comments package which looks like there's some cool stuff in there i'm gonna check that out also yeah i was when I was checking out the video that you're about to talk about, I was like, whoa, this Twitter comment thing. It looks awesome. There's way more to be learning here. Yeah, but the it's now a standalone thing. And there's this great we linked to a video called WTF is pecs. And it's by Brian Wickman. And it's a little 16-minute video where he just pretty much talks about what it is. And the insight is Python can import from a zip file. Even if you corrupt the zip file.

Even if you corrupt it. So they corrupt the zip file by putting a main at the end and the header on there to tell the rest of the operating system that's a Python program. And it just tries, your environment just tries to run it like Python and it imports it. It's all of its dependencies from itself. And, you know, I kind of got lost as to exactly how that works, but it looks fun also. Yeah, it definitely looks cool. So if your goal is more to share executable things with developers, it seems like PECS is really awesome. Or maybe even with infrastructure, like I want to deploy this thing to the server. And I just want to drop it in, have it go and not have to deal with dependencies and virtual environments, all that stuff looks super cool. And I get the sense that they're using this in production at Twitter, even though I don't remember them exactly saying that. Or using it just even for tools, because some of the examples were just to, you've got a little project, it's got a virtual environment,

or it's got some requirements for it and some dependencies. You just kind of make that into an executable, and then you can just drop the whole thing into your bin directory or something and not worry about it. That's really nice. That's pretty great.

Super cool. All right, so I want to keep with this theme of we're going to package up some not worry about it. That's really nice. That's pretty great. Super cool. All right. So I want to keep with this theme of we're going to package up some code and run it. And even with either of these, like it's great that PECS can zip up your source files or that PI installer can zip up your source files, but you're still sending your source code like straight to people. And that can be bad potentially. Maybe you've got some secret algorithm.

Maybe you've got licensing that you don't want to just be able to like, well, we'll just comment out the license check, it won't be good, right? Something like that. So there's this interesting article that came by this week called using Cython to protect a Python code base. So Cython is a fully to machine instruction compiled variant of Python, right? We've got Python, we've got PyPy, we've got Cython, we've got IronPython, all these variations. And Cython is one that came from the SageMath guys way, way back in the day to compile this to faster machine instructions. But it's fully compiled code out of.py files, which is pretty interesting. And so that makes it way harder to mess with. So this article walks you through building a Python package

and all the little extra tricks and tweaks that you've got to do so that you can compile your Python code with Cython and then ship it as a wheel, which is the current standard for PIP installing a thing, right? So you can create a binary distributed wheel that has zero source code that is compiled for a particular architecture. Wow.

Isn't that cool? So if you want to, yeah, and then you could combine this with PI installer and make it an EXE and off you go, right? Yeah. Since the algorithm's open, is it reversible then? Well, it's like compiling C code or C++ or something. So it is technically reversible

and you can get it back, but it's way, way harder, right? Yeah. So they do say there are certain ways to like hack the runtime. If you import it, you might be able to get it to give up some of its secrets,

but it's definitely better than actually shipping the source code. This reminds me of Obfuscators. I can't remember if it was in C Sharp or Java. Do you remember Obfuscators? Yeah, I totally remember those. That was such, that was even such a weird word to say when I first learned about it.

I remember that actually. I think it first came out in Java to sort of scramble up bytecode, Java bytecode. And.NET definitely has it as well because they have IL, which both Java bytecode and.NET definitely has it as well because they have IL which both java bytecode and .NET IL are super reversible like it's almost exactly what you wrote gets pulled back other than maybe like where you put your curly braces on the new line on the top of the if line you know

things like that it's like really really reversible so I think this protecting with scython actually would be even better than obfuscating although i haven't looked at all the different angles it's really an interesting idea though and it doesn't have to be everything right like you could take your core algorithm that you want to protect or whatever scythonize it and then just import it into the rest of your your code yeah that's a really great idea because like for instance um a bunch of the stuff like you're reading config files or your user interface or something, you might not care about that part.

Right, exactly. It can be reverse engineered pretty easily just by looking at it. But your core algorithms and the real reason why people are buying your product, that's not a bad idea. So this is cool. Yep. I've never tried it, but this is a pretty cool idea. And it's a step-by-step sort of thing.

So I thought people might find that useful out there. I think so. Yeah, awesome. Well, that's our six items. That's awesome. Went really fast. Like always. So I don't have anything extra to add. Do you have anything you'd like to talk about? I have a little bit of a survey I'd like people to consider filling out, really only if you're like super passionate about this. One of the projects that I'm working on is I want to create a course to make working with Python on Windows better and delightful, not frustrating and challenging. So I'm trying to gather a bunch of information on what people find hard with working with Python on Windows. I've got a bunch of people filled out a

form, like about 200 people filled this out already, but I put a link here to a Google Docs form. If you work with Python on Windows and you've had struggles with it and you have some sort of feedback you can give, so you could help me maybe create a class or something, some sort of resource to help people be more successful with Python on Windows, fill out the form at the bottom of the show notes. That'd be awesome. Oh, great. I'll definitely fill this out and pass it around because there's a ton of people that I work with that are kind of new to Python and new to Python on Windows. Oh, that'd be perfect. Yeah, that'd be awesome.

I mean, a lot of people are like, well, just use Linux or whatever. But when you show up at a tutorial or to give a training class to a corporate group and they all have their corporate Windows 7 laptops, it doesn't matter whether you recommend Linux or Mac OS, they have to make it work on their environments. And so my goal is to try to make that like a much smoother process. Great. Cool. Awesome. Well, thanks again for meeting up and pulling all these ideas together. It's always fun. Oh, yeah. Thank you. Talk to you next week. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news

item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.