Python Bytes - #119 Assorted files as Django ORM backends with Alkali

Episode Date: February 26, 2019

Topics covered in this episode: [play:0:52] Incrementally migrating over one million lines of code from Python 2 to Python 3 [play:3:42] Network Automation Development with Python (for fun and for profit) [play:9:59] Carnegie Mellon Launches Undergraduate Degree in Artificial Intelligence [play:11:37] asyncio + PyQt5/PySide2 [play:13:02] 4 things I want to see in Python 4.0 Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/119

Hello, and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 119, recorded live from PyCascades in Seattle. All right, it's great to be here. And this episode is brought to you by Datadog. Tell you more about them later. Right now, I have a bunch of special guests, none of whom are Brian Ocken. More about that in a second. But we have Trey Hunter.

Hello. Dan Bader. Hey, how's it going? Eric Cho. Yo. All right. And all of us are here at the conference and we thought, why not put something live together for you? Now, Brian Ocken decided to punish his teeth by having a painful root canal and couldn't join us in some sort of last minute emergency. And that's really unfortunate because

he was looking forward to be here. So everybody, Brian, we miss you. We miss you, Brian. Right on. Well, let's go ahead and kick it off. I'm going to do the first thing here. And have you guys heard of this thing called Dropbox? Yeah, a little bit. They have something to do with Python.

Anyway, obviously, Guido works at Dropbox. It's a huge Python center of the universe there. And what's really interesting is they're finally migrating to Python 3 and using some of the tools that Guido has personally worked on with like MyPy and static typing and all of that. So that's our first item. And if you were to guess how many lines of code is the Dropbox code that you're working with, you know, that little box in your menu bar, your taskbar, that's also client side Python, which is interesting already, but it's over a million lines of code. So they started way back in 2015, a little hack week side project to prove whether or not maybe they could do it. It

turned out it's going to be hard is what they basically they said. And officially they started the first half of 2017 and the real thing that helped them do this which i think is interesting is my pie have you guys heard of my pie yep oh yeah so my pie is it takes the type annotations or type hints and verifies that you know this function says it takes one of these and you're giving it one of the same things like that that sort of thing did you actually like i i don't think he started my pie i don't think he started it but he definitely works on it one of the original things, like that sort of thing. Did Guido actually, like, I don't think he started MyPy, right? I don't think he started it, but he definitely works on it. One of the original contributors, I think.

Okay. Did he start it? Like, was it started for Dropbox specifically or for the Dropbox code base? Just curious. Yeah, I don't know either, but I know that it was an important thing he's working on. I'm not sure, but I just want to, it seems like Dropbox has been migrating away from like the public clouds for a while, and they've been focusing on just getting things right. So this is probably one of those things where they think for

the long-term growth, it's going to be better than rely on somebody else's infrastructure. Right. Absolutely. It's very interesting. They're stepping away from some of the cloud hosting. Everyone else is running to the cloud. They're like, ah, well, we can make cloud. That's pretty interesting. So let me throw this out for you all, co-guests and audience members and listeners one of the very first things they say in this article is well once we were armed with my pi the first few steps we took was to port our custom fork of python to three five what that's big i'm like wait what, what? They don't run normal Python?

They drop Python? What do they call it? It's pretty cool. It's pretty cool. It cross compiles to Perl. Yeah. It doesn't. Yeah, so I'll just kind of wrap this up here.

But basically, this article that we're covering goes through all the steps of Dropbox moving over. And I feel like if people are going to take the Python 3 as modern Python, and other Python as legacy Python as a legitimate thing, the guy who created Python had better work at a place that uses Python 3, not Python 2. For sure. So I'm super happy to see that's moving along. And also that Guido was a pretty big part of it. All right, so let's see what's up next here. Eric? Basically, I want to talk about what I feel was like underserved community in Python.

I come from a network engineering background and been focusing on network automation using Python. And I think we've gotten to a point where we're big enough to be noticeable. Like it's actually material for the amount of community. I mean, we have new terms such as net DevOps or NRE, you know, not to be subtle differences from the site reliability engineering for network reliability engineering. We have some popular libraries from NetMiko, Napalm, who's been on your show before. And I can't even pronounce that new library, NONER, I think. And NIR will have the link in the show notes. Yeah, you know, there's a lot of free resources out there for people to practice on for either network engineer

who wants to learn more about Python or developers who wants to learn more about network engineering. I think coming of age, I mean, hopefully one day, you know, we're going to have a subculture of Python, just like the data analysis community that for network engineers. So that's, I want to bring to everybody's attention, you could

do it for fun, do it for profit. And, you know, it's a welcoming community. Yeah. And you link to a bunch of resources in the show notes that people who are into that can check out. And yeah, Python's a mosaic. And there's so many people doing different things. And here's just another part of it, right? Yeah, absolutely. I mean, I'm super excited about this. Because I think, as you mentioned, multiple times on your show, it's like, you get started early or started easily. But easily, but you don't hit that ceiling. I mean, I've been doing this for five years and I haven't found that ceiling yet. It's a thought to me. Is that a sign of growth that the Python community has seen where now it makes sense to have a niche

for network automation specifically? I think people are still trying to figure out how this thing is going to go, which is with lots of changes presents more opportunities for people. And Python kind of sort of just emerged in this de facto and speaks to the versatility and the power of the language. I think we're in that phase, we're trying to figure it out. And we just kind of have this trending versus like nobody has the right answer. But that means at the same time, that's where the opportunity lies. You know, you could figure it out and could drive that direction. And I think the developer actually has a huge advantage that everything is virtualized, everything is abstracted away from the physical. So that's my thought at the moment. You know, you could see that I'm not very

clear either. I think it's super interesting that you pointed out how everything's abstracted and sort of cloud programmable. That means like Python has a better chance in the network space if it's not all hardware and boxes and stuff, right? Yeah, for sure. I think one of the challenges for network engineers such as myself going into the cloud is the fact that there's no longer broadcast domain.

Your NIC is actually physically attached to you. So things that we took for granted that were fixed is no longer true. So you get to have like a network NAT gateway that's just arbitrarily attached to your virtual subnet, which, you know, you used to, I think if you work in the traditional enterprise, like the first thing you do when you get the new team is like you subnet it out, you give it an IP address, you subnet. But those are all virtualized nowadays. So you still need to understand the basics, but that basic used to take years to master.

Now it's just a matter of reading a doc. So yeah, hopefully, you know, you guys, you know, come say hi. If you see me at Ansible Fest,

at Cisco DevNet Create, at, you know, some of the Juniper events, you know, come say hi, let's talk. And I think we'll,

we'll could make this potentially make a great community out of it. Yeah. Put Python on the wire. Yeah, yeah, for sure. Buy you a Python beer. Yeah, but it's funny. Python really is a mosaic. I mean, that's, I didn't understand.

Well, I understood a lot of the terms you were using, but what they actually mean, I don't know, because I don't need to know what they mean. In the space of Python that I kind of impart of, this next thing I've got is kind of related to the fact that Python is a mosaic. It's kind of part of the web side of the mosaic of Python, which gets maybe more reputation than it deserves in the sense that there's a lot of folks using Python for the web, but it's not all you can use Python for at all. I mean, data science is huge. But if you have to process data, and it's not in a database, and you are someone who's familiar with Django, there's a thing called Alkali that Kurt made. I can't remember Kurt's last name. Remember,

Kurt's in the room and we actually, it's Kurt Neufeld. So it's funny being at conferences, you sometimes just meet the people who end up, you know, making the things that you're using. So Alkali, I'm not using, but it looks kind of fun because I'm familiar with the Django ORM. And Alkali, it's meant to take structured data, maybe an RSS feed, maybe a CSV file, maybe JSON data, maybe some random homegrown thing that you've got on your team or in your company, and allow you to use a Django ORM-like syntax to query it and also to save it, maybe in some other format even. So it's as if you're working with a database, but you don't actually have a database behind the scenes. You've got some structured file. So it's kind of does that all in memory, which is fun. Right. So maybe you're working with XML and you don't want to

learn XPath or you don't want to write regular expressions against CSV files. Who wants to learn XPath, man? Nobody. The 90s are calling. they want their api back here's my style sheet says nobody ever yes exactly so i think this is the cool project kurt i definitely like that you can point it at even like something an endpoint on atp service and like turn that into effectively a django database and i've heard that there's a branch working on indexes which will like sort of complete the performance side of things. Ooh, that would be really fun. Yeah, no pressure, no pressure. It's going to be released tomorrow, I heard.

I'm just kidding. It's not going to be released tomorrow. It's a long night for Eric. You're shaking his head. Long flight home. I don't know where you're from. All right, before we move on to the next one, let me just tell you about our sponsor

which makes all of this happen. So this episode is brought to you by Datadog and datadog they're really awesome they let you track the performance and errors and requests not just within your python app but across all of your infrastructure like so if you're doing like a kubernetes thing and you've got a flask app and it's talking to nginx and it's talking to postgresql you can like tie all the performance of that entire system together, not just profiling your Python code, which is pretty awesome. So check them out at pythonbytes.fm slash datadog. Get a cool free t-shirt. You get to try it out. It's awesome. Okay. So the next item, that's Dan. Oh, sweet. Yeah. So a quick update here.

The CMU, Carnegie Mellon University launched a undergrad degree in artificial intelligence and apparently that is the first AI degree offered by a US university and you know when Mike told me about it I was really surprised because I thought well you know AI has kind of been like a big buzz word for a while now and why didn't anybody else come up with a degree before that but I guess it always takes a little while to do that and I don't really know what goes into that degree or kind of what, you know, how the curriculum really differs from, let's say like your average computer science degree

or like a data science curriculum. But I just felt it was an interesting development. Yeah, I'm sure they use a lot of Python. We've had computer science forever. Well, first it was like electrical engineering, but I work on computers on the software side. And like eventually I got a real degree like computer science. And then we have like software engineering. But now I think this is a big landmark, like the first artificial

intelligence, like a bachelor of artificial intelligence. Like think of that. That's crazy. And one of the things the Dean said is, you know, of course we'll do CS stuff, but we're also going to focus on things like computer vision, language processing, huge databases, and how to help humans make better decisions automatically. It's pretty cool. So I'm waiting for the day where we have an AI, get a bachelor's degree in AI, just so we can call it a day, and we're done.

Or an AI teaching the bachelor's degree in AI. Yeah, even better. That would be so sweet. My professor's a jerk. It's degree in ai that yeah even better that would be so sweet my professor's a jerk it's written in 4chan yeah so did he use python at all oh god it's gotta be like it's got it must be right it's all java no i don't know it's gotta be python right all right so you all might know that maybe i've been kind of on a rant about async and await and async is programming lately and the next one oh have you also heard that I've talked about GUIs? Like I've mentioned this twice, I think,

like that Python should have better GUIs. Well, this next one is kind of like these things come together, which is awesome. So Florian sent this over to me and it's PySide2 and Qt for Python, the Qt framework. That has an event loop that, you know, a button gets clicked or timer runs or something like that. Well, somebody built some layer that you can plug that into

async and await. So you can have like async def button click handler that integrates with your other async operations happening on your GUI there. It's pretty awesome. There's some examples on how you do it. It's super simple. i linked to one about downloading some stuff and and whatnot so yeah if you're doing anything with cute and you do anything with async then check this out that's uh really really a nice one so that one usually like i know i haven't done qt in a while but gtk uses kind of an object-oriented event loop there right where it's classes so it's taking a class- based syntax and allowing you to use the new async IO syntax, right? I think it's mixing the GUI event loop and the async IO event loop together, because otherwise, I think they would run independently. I think you basically can't have those run on the same thread or something to that effect, right? Like the async event loop would block the GUI

loop or something to that effect. Cool. All right. So the next item we've got on the list here, you know, guys, we're at Python 3.7 now, 3.8 is coming out pretty soon. So we're kind of running out of like minor number space. I guess we could always create more, but whatever. That's a good intro. People have started thinking about, you know, what's going to happen with Python 4.0? Like what would be some cool features that we would really want to see? And so our good buddy, Anthony Shaw, wrote a really interesting blog post about four things he wants to see in Python 4.0. And it's pretty short read, but there's some interesting ideas in here. So we're just going to go over those points here. And so number one is he would love to see just-in-time compilation as a first

class feature. So right now, you know, you've got some alternative Python interpreters like the Piston project or PyPI, I guess, is the most well-known that actually feature just-in-time compilation, and it could bring a huge speed up compared to the plain bytecode interpreter setup that CPython uses. And so I guess the idea would be, is there some way to bring this into core Python?

And apparently there is, and we already have this in some way, or at least we have the infrastructure to be able to plug in something like that. That one would be really big. Because I know there are some companies that the reason they're able to use Python for what they do is PyPy. The fact that it really speeds up with that just-in-time compilation. Yeah, I think it's a big one, right? Like performance. I think the more people use Python, the more relevant the whole performance story

becomes for people because then it's like, yeah, it has a huge impact if you have a small improvement. Yeah, absolutely. There's tons of attempts to solve this problem. Like there's Rust Python and there's Grumpy

and there's all these different attempts on solving. And PyPy, like Trey said, is really awesome, but it has this limitation where like when it kind of, when it gets to the C interop stuff, it can slow down or it doesn't necessarily work with all of them. So it kind of falls back then. And with Pidgin and the work that Brett Cannon and those guys did,

it's really awesome because that's a plug-in to the normal CPython. So it wouldn't be like an alternative thing. So yeah, I would love to see this as well. It'd be great. Yeah, great idea. All right. Item number two is on the wish list is a stable 0.0, like a stable 4.0 release. Is that a lot to ask? I don't know, man, you tell me. I feel like this one, this was because of 3.0 history, right? That, you know,

there were lots of breaking changes that the initial was a kind of a rewrite of the language from my understanding, although I'm not a core developer, I don't know. The central point of that in the blog post here is that, well, you only have one chance to make a first impression really and so maybe python 3 kind of bumbled its way into life or whatever i think now we're super happy that we have it but i don't actually really remember the zero release at a point one release i don't know if anyone does yeah it's like let's not talk about that let's just move on no i'm sure it was great all All right. Static type hinting. I think that's a really good idea too.

You know, we've got MyPy, but it's optional right now. And it would be kind of interesting to see that integrated into CPython or the core language if this is really the path forward. And I'm not actually sure what the roadmap says there. Yeah, I don't know either. It's pretty interesting. I think static typing is super valuable. I think having it mean something in the language,

that would change the zen of Python, wouldn't it? I mean, because it's so much about the duck typing and I don't have to worry about it. It's like, whoa, compilation error. We expected a I runnable of whatever, right? Multiple templated thing. And yeah, I don't know. I don't know about that.

We really changed the face of the language, I think. Yeah. I like what he's recommending here. I'm not so sure about the required static type hinting. Maybe like a mode to run it where you can check it. I mean, we have data classes, which do some validation in a sense. You're wrong, Anthony. No, like we're just, this is some really interesting thoughts about this because, you know, what should go into it? Because obviously it's a big release, right? If you're talking about like Python 4.0, like it better be like a really, really noticeable improvement. Otherwise people are going to go like, oh, which would be nice too.

I mean, if it's just a 4.0 release and everybody's kind of, there's no like upgrade hump like we had with from two to three. That's kind of nice too. Right. there's no like upgrade hump like we had with from two to three that's kind of nice too right well and he does mention the idea of static duct typing putting an iterator in there as opposed to a generator specific type of thing but i don't know how you would really make that a truly generic thing yeah well as long as we don't end up with a python 3 death clock in a pretty good place nice okay so uh the next item we have here is a GPU story for multi-processing. So I guess the idea is that a lot of workloads that people use Python for these days are actually running on GPUs. You know, a lot of, I guess, like the deep learning stuff's all running on GPUs these days. And so wouldn't it be cool if Python 4.0 actually had some facilities to run stuff on the GPU for like parallel computations and had it built into the language. Wouldn't that be sweet? It's an interesting idea for sure.

Maybe like another decorator, like an at GPU. And we're done. Add some tie pins. Yeah, and the last item here on this really interesting list is number five is more community contributions. And I think Anthony is saying that he's already seen a lot more involvement from the larger community. And now that CPython is hosted on GitHub, there's less barriers for people to contribute, I guess, to the code.

And just seeing more growth in that and seeing more people involved in the actual development of CPython would be pretty sweet. I totally agree. What do you think, Eric? A lot of these features, I haven't been coding long enough to have a strong opinion about one way or the other. But I think to me, obviously, you know, optimizing for hardware and who would say no to that. But to me, the 4.0 story would be big in terms of this would be the first major release

without having a BDFL. And I guess we'll figure it out by then how 3.8 came about and all the peps. But this will be a major release where it's determined, I guess, by the committee. So it would be kind of interesting and just see how that transition going and hopefully for the long term and 5.0, 6.0. I feel like even outside of the core developing team, Python naturally has had more community involvement over the years. And it'd be nice to see that with a 4.0 because I mean, even this podcast, like, you know, you mentioned

under pie packages recently. And that's something that that's not a pep that's actually ready. That's something that it may or may not make it into Python. That's a discussion that normally happens not behind closed doors, but in an open space that no one looks in, which is the core developer mailing list, whereas it's on a podcast now. Some random people in Portland dug it up and talked about it on the internet. Getting all the dirt on your Python. Yeah, so that's it for all of our main items. Just a couple of quick extra ones from me.

One, I did an Async webcast, which is available. So if you want a one-hour review of what Async and Await means and why, I think now is the time for Async and Python and you don't have to switch to go. It's already awesome. Just use it. And so you can check that out. I'll link that in the show notes. And then if you happen to be somewhere near Tel Aviv or Israel, at least the first week of June, they're having PyCon Israel, which is pretty awesome. And call for proposals is open just a couple of days ago. So that's those are my extra items and you guys got anything else yeah quick announcement we're working on a new

book for real python and we're going to release through real python it's called the python basics book so it's like a beginner's book for people who want to get into python in the first place and mike actually wrote the forward for it and um it's great but it also kind of duplicates what we had said in the intro so that means we've got to rip out a bunch of stuff and then use this forward as a new intro because it's so much better than what we had. Thank you, Mike. You're welcome.

And shameless plug for the book. Thanks for making me work. So the only thing I have to share is that some things in my world, I have a goal for myself to write more because writing blog posts takes me so much time. And so that's something that I'm just announcing publicly here only so that I will commit to it

over the next quarter or so and there's some kind of big things that folks on my mailing list know with Python more so it's going to be coming up soon. Yeah, sounds great. So I guess we got to close this out with a joke. So we got a whole list of jokes here and I'll just grab two for you guys and let you

all see what you think here so uh why did the angry function exceed its call stack size it got into an argument with itself oh no oh no there's more but wait why did the developer ground their child as in in, you can't go out, you're in trouble, you stay home for the week? They weren't telling the truth. And with that, I think we're going to close it out because that's, what are we going to do with that?

All right, so Trey, Dan, Eric, thank you all for being here. And everybody, thank you so much for coming. Podcast Cades was great. Brian, we miss you, and see you all later. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S.

And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.