Python Bytes - #9 Walking with async coroutines, diving deep into requests, and a universe of options (for AIs)

Episode Date: January 17, 2017

Topics covered in this episode: - Talk CPython Internals and Learning Python with pythontutor.com - Talk Write an Excellent Programming Blog Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/9

This is Python Bytes. Python headlines and news delivered directly to your earbuds. It's episode 9, recorded Tuesday, January 17th, 2016. This is one of your co-hosts, Michael Kennedy. I'm here with my other co-host, Brian Aukin. Hey, Brian. How's it going, man? It's going very good. How are you? I'm doing well. I always look forward to our weekly chat about cool stuff going on in the Python ecosystem. And I see that you found something quite cool for our first item.

Actually, it's a series of things. The link that we have is to Python Asynchronous I.O. Walkthrough. And here's the story. So in July, there was an open source book published called 500 Lines or Less. The Architecture of Open Source Applications is the website.

But they do various open source free books, which is pretty cool. Anyway, the 500 Lines or Less had one chapter in it from Jesse Jero Davis and Guido van Rossum called A Web Crawler with AsyncIO Coroutines. And it's a pretty cool chapter about explaining asynchronous networking, showing how non-blocking sockets work, and how Python 3 coroutines improve asynchronous network programs.

Yeah, that's really cool. I think that that book is amazing. It's not just this particular chapter. There's many chapters. It's also not just Python, but it has a heavy lot of pieces of pieces which are python and basically it's can you show me something amazing that really i thought was really hard but can be explained or written in 500 lines or less and then discuss the architecture and there's like 20 different projects or something that was very cool yeah and the the series is pretty neat so this this is 500 lines or less there's like 20 different projects or something. It was very cool.

Yeah, and the series is pretty neat. So this is 500 lines or less. There's previous books that are about architecture and about performance issues. And it's a neat thing they're doing. But this particular chapter for me, since I'm kind of new to the asynchronous world, it still was a little bit heavy to get into. And so Philip Guo, who I think he interviewed you once. Yeah, he's been on my show on Talk Python to Me on episode 22. And I also was on his show. He started a podcast recently. It's got like 25 episodes or something. And yeah, I'll be sure to link to that. That's a really cool one as well. He interviews people just kind of generally in the programming space. and then also going through and doing some code examples. And he split, it's about 90 minutes total to get through this,

but he split it up into eight parts. So each one is really not that hard to get through. And it's a pretty cool idea, and I really appreciate it. I think it's a good way to learn, to see somebody doing it, to hear them, to use all your senses. I really like that. I think this is great work. Nice work, Philip. Way to go. He's done stuff like that before. Like I had him on, the reason I had him on episode 22 of Talk Python was he did this thing called CPython, a 10 hour code walk, which was like a, an exploration through the

CPython internals, which I found really enlightening. And we actually, I talked to Jesse about his article on episode 69 as well, when we talked about writing as a developer. He's got some really cool thinking around like design patterns on technical writing and stuff. So this is really cool. I think seeing a problem unfold before your eyes in code is much different than seeing the finished product discussed.

And I find I learn really well that way. So I'm happy this is here. I haven't had a chance to go through it, but I'm looking forward to it. And also both Jesse and Philip are doing some cool things to help the entire community, which I appreciate. Jesse's been supportive of a lot of people, including me. Yeah, absolutely.

Jesse's doing great stuff. Do you want to think about the future of Python a little bit? I thought I was thinking about the future of Python with switching to 3.6 recently. But what do you think the future has in store? We were living right on the edge of the front wave, but there's an article I thought was pretty interesting. I don't necessarily agree with everything,

but I felt that it was certainly thought-provoking. A guy named Sardar Yagulap, I'm sorry, I'm sure I messed up your name, wrote an article and it's called Four Likely Future Twists for Python. And I thought that was pretty cool. So let's go through the four twists. The first one does not make me happy, but is certainly something that I can see coming into existence as a thing. So we'll see. But the first one that they put out there is that Python 2x may live on. Yeah, I actually can see that this might happen. I mean, it's an open source project, so there's nobody stopping you from doing it. The thing that's going to make the difference is what do the core developers do, right? So right now, the core developers are all focused on making sure that we make Python 3 as great as possible.

And they're putting their energy into it and so on. So I think Python 3 is certainly the right place for people to be. But I can see like large banks. You know, you think of places like Bank of America. They've got, gosh, I don't remember the exact stats, but I was talking to Mahmoud Hashemi from one of the Enterprise Python episodes I did. And it's something like the number of lines of Python to, I don't even think it's two seven, it's like two six or two five code is in the millions, like five million lines of codes or something crazy like that. I can see a company like that forking Python

and just making some kind of ongoing effort. Supporting your own Python, that just doesn't sound fun. It sounds like a really bad idea. I would much rather people just upgrade and we can all work together and live together and do great stuff. Here's one of the predictions that there'll probably be somebody that forks Python 2 and tries to keep it going. Whether that turns out to be beneficial or, you know, trading one set of problems for a worse set of problems, we'll see. Yeah. The other one was, this one I had no visibility into this. I didn't see this coming.

And I read the title and I said, requirements.txt may be replaced with something better. What? Okay, well, maybe. It turns out the guys over at the Python Packaging Authority, that's the PyPI and PIP and all those projects. So there's an organization, a group of people that actually work on that. They're working on this thing called PIP file. And it's like a proposed new file format that's supposed to replace requirements.txt.

That's interesting. Isn't it? Yeah. And so the idea is that pip file is going to be better or superior to requirements and that it has a Toml syntax, which is, if I recall correctly, it's a little like YAML. There's one pip file that can have multiple configurations in it, so you don't have like a dev requirements, a test requirements, a prod requirements and so on. Yeah, it's cool. You can do like a PIP file freeze on it and it'll generate something that is a much richer

document instead of just a TXT file with a package name like a version. So there's a lot of configuration and stuff in there. Like for example, one of the things you can do in these pip files is you can actually set the source of the packages, like right in your requirements file. So you can say, I want to use this version of Python. I want to use it against this source, like pipi.org. Or you could create your own dev pi server thing and run it internally, and that would be part of your requirements file.

So lots of cool enhancements there. I'm interested to see where it goes, but it's out there on GitHub. You can actually check it out. Yeah, I'd like to make sure it sort of goes along the lines with other packaging issues like the setup file and stuff. Hopefully we have one thing that replaces everything. Yeah, definitely. Okay, so number three is Python could get more enterprise additions.

So we've got things like Anaconda and InThought. And Intel actually has their own Python implementation that is sort of a repackaging and tweaking of the Anaconda distribution. But just talking about how there may be more variations of Python distributions focused on enterprise needs. Maybe. I certainly see it happening in data science. I could see it happening elsewhere. I could see it, especially since we have to, I mean, everybody that's relying on Anaconda or something kind of has to wait for them to release something when they're waiting for a new version. And the last one is Python's software repository. So PyPI basically could get enterprise-friendly package management.

So Christian Medina wrote a really interesting article as a follow-up to my episode 84 on the problems with PyPI and open source infrastructure funding and stuff and put together a pretty interesting article talking about something called the Python Trusted Registry or Trusted Index or something like that as a way to both fund PyPI and make it more enterprise friendly. I'm actually thinking about this problem as well. And I'm probably going to release something like an article and a proof of concept, but I'm not ready to talk about it yet. Okay. Sounds interesting. Well, so that was four predictions. Are we done with the future? That's the future. Let's go back to the present. Okay. So in the present, NumPy is used for a lot of data science and really just science

and engineering. A lot of people use NumPy. There's a new book, an open source book, called From Python to NumPy. And it's from, I think this is a French person, Nicolas Rogier. The book is pretty interesting. It takes the assumption that you already know Python. You're an intermediate developer. And you just need to get your understanding of NumPy to apply it to whatever problem you're on. And it looks well. I haven't read a ton of it, but it looks really well formatted. I like the topics that are in there.

Just sounds interesting. Yeah, it sounds really cool. And if you're looking for scientific computation and you're thinking of like, am I going to shell out $2,000 for MATLAB? Well, you know, maybe look at some of the Python tools, right? It's definitely growing. Yeah. You know, it's cool for us to think about the future, but what if like machines could think

about the future? Machines, yeah. I really feel like 2016 and 2017 is the year where AI became not something that's always five years in the future, but actually something that's starting to take shape in the present. Yeah. And it now affects everybody with all of their speaking devices. Yeah, and we could be mean and we could say, hey, Amazon thing, order us a... But we won't do that.

The item I want to talk about for number four here is something called Universe. And it's a way to take games, basically, all sorts of games, and make them available as ways for AIs to experience the world. So here's the thing they put together where you can build an AI, put it into a Docker image, give it to their environment, and then they will let it play literally thousands of video games. And it can learn from playing the games so you can play like flash-based browser games other types of games a whole bunch like i said over a thousand environments and the way it plays it is the so you don't have to integrate with like all these different games

your docker image just is presented the visual screen that the humans look at, and then they send mouse and keyboard events to the games. Yeah. So they basically just play as if they were a person playing the game. So they have the same experience of the game as you, but here's all these environments in which you can train and test AIs. What do you think? It sounds like tic-tac-toe in global thermonuclear war to me. Hello, Dr. Falcon. Would you like to play a game? Well, so what's the – so if you have an AI that has learned a whole bunch of stuff about games, like why? Why would you care about this, right?

Like, well, maybe you're working on a robot and you want to teach the robot to walk around and interact with the world. But you don't have the robot finished. You could take the smarts of the robot and let it play Counter-Strike or something for until it gets really good at like not bumping into stuff, like making it through paths. And then you could just plug that into your robot. I don't know, something like, I mean, this is me totally speculating on the spot. I don't know. Yeah. Just don't give your robot a gun. Yes, exactly. Because it's going to be really good at it at that point. Yeah.

All right. So another thing that I've been doing a lot with lately is working with APIs and HTTP services and stuff. And your next item is a pretty interesting deep dive into literally the most popular Python package request, which for those of you who don't know, is downloaded 7 million times a month. Yeah, and it's titled Python Requests Deep Dive, and it came from Anthony Shaw,

and he put it on Medium. And Anthony's been... I don't even know why I know Anthony. I've just known about him for a long time. But he's... My first thought was it's going to be just teaching people how to use requests, but it's not. It's a really cool article.

What happened was he was on a project to convert a large project from HTTP lib, so he wasn't even using URL lib 2 or anything http lib into requests and this project was apache lib cloud and with i was handling stuff like http https rest json xml whole bunch of bizarre apis with over he lists over 80 client libraries to handle different cloud services and had one connection class that did the encoding and decoding of everything and converted this all. And this article is about all the stuff he had to deal with, like dealing with sessions, dealing with authentication, how he tests it, doing prepared requests, streams. It's just a chock full of stuff. And it's not even that long

of an article. So I think he did a good job. Yeah, that sounds really cool. I haven't had a chance to read it, but I definitely bookmarked it. Anthony Shaw is a great guy. He's doing cool stuff. He talked about LibCloud on Podcast Init. I'm afraid I don't remember the number, but you can listen to him there. And that's really cool. It's a cool project. And this sounds like a pretty serious real world upgrade from one library to another sort of example. And I think there's probably a lot of lessons people can get from this.

And one of the neat things he has in here is he uses a tool called RequestsMock. And I looked through the API for this and it really is pretty cool. So my philosophy on mocks is to really only mock what you absolutely have to have to do. Not everything, but some of your testing should be using mocks when you're dealing with outside resources. And an HTTP request clearly is dealing with an outside resource. But the API for this is very, very simple and very powerful.

And I got to try it out. Looks neat. I hadn't heard of request mock, requests mock. And that sounds like super useful because you're right. That's one of the places if you're doing testing, you really don't want to have to set up the service and configure it in test mode or whatever. Now that was dealing with APIs on the internet. How about desktop applications? Yeah. So let's talk about Python and desktop applications. I certainly,

you can build interesting desktop applications. You know, Dropbox is a real world example of, you know, probably the most widely deployed Python desktop app in the world because that client thing that I see spitting right now in my little notification bar, that is all Python. Okay, Python two, seven, I believe they're working on upgrading to Python three, but right now it's two, seven. And that's interesting. You can use a lot of different technologies, but I kind of feel like there's just something missing to make Python a first-class environment for building desktop applications. So we've had TKender for a long time, and that's sort of built in. But I always feel like that looks, I don't know, super out of place, especially on Mac or Windows.

It looks much more like it belongs in sort of a Unix, Linux space. But even there, it doesn't look super fabulous. It looks like the TK stuff that I was using with TCL when I was in college a long time ago. Yes, exactly. And it totally works and you can do it and so on. So the thing I want to highlight is a Reddit conversation. It's not a link. It's just a back and forth on Reddit.

What's the community's favorite for developing OS X applications with Python? So there's a couple couple people that weighed in. By far, I think PyQt came in as the most highly voted up. And have you done anything with Qt? I did a small project with it a long time ago. Yeah. The thing I like about Qt is the stuff that comes out of it looks native.

It's based on C++, so it starts super fast. It feels like an app that belongs wherever it's running. And I think that's cool. The challenge with this is you've got PyQt, the package, and you've got PySide. And these have been sort of competing, not entirely compatible ways of writing this code. I think PySide is the better licensing model, but it's, if I recall correctly, it only supports Qt 4 and Qt is on Qt 5.

And so there's just all this sort of like, it almost works. It's kind of hard to deploy Qt on the various machines. You got to, it's sort of a heavyweight install and so on. This should be getting better. I think Qt, the company that runs qt is bringing piside back in and gonna modernize it but that's not coming out until the end of this year probably so one vote was for qt that was cool uh another one is for tk simple avoids dependencies

but like we already talked about that kivy kivy definitely got a vote but kivy is much more it's not sort of a desktop app feeling. It's more like, almost like games. I think its origins were in games. So what you get is it doesn't really have a native feel, right? So that's not that great. I think some people are using it for kiosks and stuff. Yeah, yeah.

And that's not bad. I know it does a great thing, great stuff on Movable as well. There's some debate whether that was the right choice. There's Toga, which is really cool, Toga from the Beware guys. But it's also Toga is like real early, doesn't have support for all the APIs and stuff. So I kind of want to point this out to just say like, you know, to keep raising the issue like, hey, Python community, like it would be awesome if there was something that was really clearly fantastic for building apps where, where people say, of course you should use this. It's as good as Swift for

OS 10 apps. It's as good as WPF for windows apps or whatever. It's just, it feels native. It's really all coherent. And I just don't see that existing yet. I think the beware guys are the closest to it. It was an interesting conversation. And you've brought it up before that it would, I think you brought it up on Twitter, that it'd be cool if we had something kind of like Electron. Yeah. For those of you who don't know what Electron is, Electron.js is a way to write apps that ship entirely bundled. There's just a single thing you run. It's an exe or a binary on Linux or dot app on Mac. There's zero dependencies and it runs Node.js as the sort of smarts and it embeds

its own copy of Chromium, the UI. So it's basically you use HTML, JavaScript and CSS to build your UI and you can use frameworks that make it look native. And then you've got Node as the backend to keep this all going. There's a few projects that almost do this in Python. Like there's one about Django that will let you like ship Django sort of backends that will create gooey looking things where you're doing it with, with basically Django running locally. You know, I don't really want my app to say to, you know, do you want this app to act as a server when I try to run it and be like, this is not supposed to be a server app. Like what kind of weird thing have I got a hold of here? Right. I want something that feels like it drops in and

it's basically indistinguishable from a native app and how we can dream one, one step. And that's a good, good step in that direction is CX Freeze, which lets you take Python code plus its dependencies and the Python implementation and everything and package it into a redistributable. And that just released version 501 in January. So that's cool to see that coming along. So that's one of the important building blocks at least. Okay, that's cool.

Someday I dream of a future where, I don't know, there's some whizzy UI framework with a visual designer for Python. It's interesting that I don't know, there's some whizzy UI framework with a visual designer for Python. It's interesting that I don't really think about that much, but you spend so much time interacting with elements on a desktop, even

though we don't develop them very often. Not to say it can't be done, I just think it could go up a couple levels and it would make a really big difference in the adoption of Python. Definitely. It's already growing super well, but if you could take over the desktop market with it as well,

then who knows where it would go. Maybe you should do that, Michael. You could take it on. Yeah, I have so much time. That seems like a great idea. Yeah, okay. So if you don't have time for that, what are you doing? I'm working on my classes.

I'm actually working on some stuff that has these uh these things we were talking about in mind it's sort of at the top of my mind but i'm not ready to announce anything yet but cool cool projects that i'm working on and i'm i'm working on a special episode of talk python to me we've got episode 100 coming up so i have a special guest i'm not going to announce until i release it but that one's going to be big. I can't believe it's been 100 already. I know. Two years. Amazing.

Yeah. Speaking of podcasts, I've got a couple coming up hopefully pretty soon. They just need edited. Talk to some really great people. And then I was interviewed on a podcast called Reflection as a Service. And that was really fun to talk with somebody else about test-driven development. And I think I talked about the testing column that I think I made up on the fly while I was talking with you one day. Yeah, we're going to make that a thing.

So yeah, and then I'm still writing. So that's what I'm up to. Awesome. Yeah, you got some good reviewers for your book, right? Yeah, definitely. A big chunk of the PyTest core team has agreed to help out and make sure I don't say something stupid, so I appreciate that. Yeah, that's really excellent. I feel like we've definitely given people a lot of information this week. Definitely.

It's good. And pretty drama-free, so that's always nice. Yeah. Well, thanks a lot for So that's always nice. Yeah. Well, thanks a lot for talking to me today. You bet. It was great to share this with you, Brian. And I'll catch up with you next week. Thank you for listening to Python Bytes. Follow the show on Twitter via

at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.