Python Bytes - #71 We can migrate to Python 3, careful please

Episode Date: March 28, 2018

Topics covered in this episode: The Conservative Python 3 Porting Guide World-Class Software Companies That Use Python Stop Writing Classes PyPi.org is alive pygame on pypy usable Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/71

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 71, recorded March 22nd, 2018. I'm Michael Kennedy, and normally I would introduce Brian, but Brian is not here this week. So I'm here with Trey Hunter. Trey, hello and welcome to the show. Hi, thanks, Michael. How are you doing? I'm doing great, doing great. Thanks for dropping in. There's a bit of a shuffle with the whole spring break travel chaos going on. And so it's really great that you're here to record with us. And Brian will be back with me next week. So probably a lot of people out there in the audience know you,

but maybe a really quick introduction just so people know who you are. Yeah, I am super happy to be here because I'm a big listener to the show. I do onsite Python training for teams and I help folks learn Python on the internet through weekly things and I tweet a whole bunch. So folks have probably seen me on the internet if anywhere. You work on the internet. That's awesome. Yes. And do things on that internet. All right. Awesome. So before we get to the first item that you picked, I just want to say thank you to DigitalOcean. They're sponsoring this episode

like they do many. Check them out at do.co slash Python. Trey, what do you got for us this week? So my first pick is kind of related to a talk I'm doing at PyCon. I've been researching this stuff a whole bunch. It's the conservative Python 3 porting guide. This was written by a whole bunch of folks at Red Hat. And it's pretty much the best porting guide I've seen for how to take your code from Python 2 and switch it to Python 3. It's not perfect, but none of the guides out there really are. Yeah, this is something that there's really not as much material on as I kind of would expect there to be. I mean, there are tools. We have six, we have two to three. We have Anthony Shaw's Pluralsight course that he just launched, which is really a good course. Well, you know, that's brand new. And Guido just recently made a statement, which I thought was

great, that really solidified the date. Because for a long time, it's like, well, Python 2 is going to go out of support, out of security patches and all that stuff in 2020. And like, that's a whole year, right? Like what, we need a more time frame than a year, right? So when is it actually going out? It's January 1st, 2020. So it is less than two years away now. So people should probably start paying attention to this if they're ever going to. Yeah, this is a really big deal. It's, you know, a lot of my clients use Python 2 and switching to Python 3 is going to be a really big headache.

I do wish this guide used the future library, which I know Anthony uses in his courses at Perl site or his one course there on switching from Python 2 to 3 instead of the modernized library. But I mean, none of these guides is perfect. Some of them are a little bit older than others. This is just one of the best ones I found out there. Yeah, that's really awesome. I really like it. It's called the conservative Python 3 porting guide. Why conservative? Conservative because they don't try to get you using Python, new fancy Python 3 features.

They're just trying to get your code working on Python 3, even if in a kind of begrudging way. You know, if you don't want to switch, you have to out of necessity, this is a conservative guide. I see. So it's not like, oh, and while you're here, wouldn't it be awesome to throw a little async and a wait in there?

Exactly. You know, maybe some stars to force some keyword arguments and off you go. No, this is just like, you need to get it to your older code, your legacy code to run on modern Python. And that's it. No changes. Absolutely.

Yeah, this is you just need to get running. Yeah, I think that's a really good way to do it. Because on one hand, I'm sure it's super tempting. You're in there, you want to make all these changes. But on the other, you don't want to conflate the making it run on Python with adding new features, re-architecting your code and so on, right? Maybe that's a second project. Absolutely. So I have a question for you. Which companies use Python? Well, there are a ton of them, actually. Some of the best companies on the web that you know of

definitely use it. So there's a really cool article on realpython.com by Jason Reynolds. And I think a lot of people come to Python. I don't know. Give me your opinion on this. I feel like they come and they're like, oh, I write a few scripts and I kind of like tinker with Python. But I don't write apps in Python. Like that's a different thing. Do you get this experience?

Absolutely. This is a big thing with a lot of my clients is we use Python. We don't use it for making deployable software our clients use. It's just for scripts. Right, exactly. And so obviously that's possible with Python, but there's so much more that you can do. So I just wanted to pull this article up as some examples for how people might use Python. And if you're thinking of using Python at your company, and they're like,

no, no, we use C sharp that compiles, that's something we can use in large projects. We can't use a toy language, right? Like think about what these companies are doing. So the first one, industrial light and magic, they do all the special effects from any of the sci fi movies, and so on. And they use Python to track and audit pipeline functionality, maintain databases, and basically move items and assets through like a pipeline. That's pretty awesome. We have Google. So in the beginning, of course, right? Of course, Guido worked there.

Many of the luminaries in the Python space have or do. And they said, the founder said, we're going to use Python where we can and C++ where we must. That's pretty cool, right? That is really cool. Yeah, I love that phrase. And of course, Python powers YouTube still to this day, handling millions of requests per second. That's a pretty insane number, right? That's a ton of stuff to be putting through Python there. I know, that's quite a bit. It's really impressive how much is going on there. And actually, there's some really interesting comparisons. Like there used to be Google Video,

which was a competitor to YouTube when YouTube was a startup. And they were doing all their work in C++ at Google Video. And the people, many fewer, like 20 versus hundreds of engineers were writing Python and just like running circles around Google Video. So Google solved it by buying YouTube. It's a pretty interesting history. I never knew that. I mean, I knew they acquired YouTube while having video. I didn't know that Python was a deciding factor there. Yeah, it was one of the deciding factors that let like the small team outperform Google because Google is using C++ for their video service. All right. So speaking of large services, Facebook, they use this mostly on the infrastructure side, but Facebook is really interesting in that they have did a really interesting thing moving from Python two to Python three as sort of like changing the engineering culture. Jason Freed did

this really awesome talk on that. And the whole steps in which they took to make this happen are pretty amazing. Did you happen to see that talk? I did. You know, I think he's given a similar talk at PyCon actually this year. Oh, that'd be awesome. I'm definitely going. Definitely going. Closely related is Instagram because kind of like YouTube, Instagram was its own thing and now it is like engulfed into Facebook. But they run the largest deployment of Django web framework in the world. And they have like 800 million active users. I think maybe the most notable thing was Instagram's keynote at the 2017 PyCon with Lisa Guo and Hui Ding, who gave like the most inspiring talk, how they migrated from old version Django to new version Django and

Python 2 to Python 3 without branching and continuing to ship features. Yeah, that was amazing. I really liked that talk because in the beginning they started with, here's how we did it at a high level. And towards the end, they dove right into the code that they used and the difficulties they encountered. And if you are afraid of encountering difficulties, look at the ones that Instagram encountered because they've got a big code base. Yeah, they have millions and millions of lines or something. It's insane. But the way they did it was truly inspiring. And I felt that was actually like a really clear playbook of, hey, we could do the same thing. We also have Spotify. They're doing a bunch of stuff with Python, primarily as backend services and data analysis. Quora. I think Quora is the best Q&A place on the internet. Always seem to have such like super thoughtful questions and answers. You know, I don't use core much, mostly because whenever I go, it asked me to sign up

or sign in. And so I mostly go to the other ones for that reason. But I have actually found a lot of useful stuff in core before. Yeah. I'm just surprised how much time and energy people are willing to put into answers there. Absolutely. It's interesting because that runs on Python, of course. And they were thinking about maybe C sharpsharp maybe java maybe python maybe c++ and so on and in the end they ended up kind of doing what google did is python where they can c++ where they must netflix obviously netflix is insane they do like in the evenings they account for one third of the bandwidth in the united states wow yeah and they run all their infrastructure management operation stuff over python actually talked to those guys over at TalkPython on episode 16 way back in the day.

Really interesting stuff they're doing. Dropbox. When I think of the center of the universe of Python, I think Dropbox. What do you think? Why is that? Well, Guido works there. That's a big reason. It seems to be one of the companies that is one of major major sort of technology companies that is all in

on python right like even their little widget thing that ships as a client app that is in your tray or your menu bar that's even python yeah they definitely got a ton of deployed python code on desktop computers that they don't have control over which is pretty amazing i think they also have some open source packages up on GitHub, their Python. Yeah, I think they do have a lot of really interesting stuff. And they've had some stuff that they put out and it's kind of like left to go as experiments.

Very, very interesting. So last one I'll leave you with, Reddit. So they have half a billion visitors a month. They had, let's see, it was the fourth most visited site in the United States and seventh in the world. And I guess in 2015, they said they had 82 billion page views, which is kind of insane. And of course, that's all driven on Python, SQL alchemy, all sorts of goodness.

That's pretty incredible. Yeah, awesome. Those are the companies that I kind of wanted to highlight out of that article and just thought like, these are really great examples. And if people are saying, oh, well, we have like 100,000 users, we can't use Python, it's not gonna scale. Like, well, yeah, probably not the real problem. There's probably some kind of architecture

or infrastructure thing that is actually gonna make this work easily. For sure. Yeah, nice. So do you think we should stop writing classes, Trey? Maybe. I don't know if we should stop writing classes entirely, but my favorite, well, one of my favorite PyCon talks,

I tend to tell people it's my favorite because I recommend it so much, is Stop Writing Classes by Jack Diedrich. This is from PyCon 2012. So it's kind of an old talk, but it is definitely the days of Python 2, but all the code in it, all the advice in it is still really applicable today.

Yeah, that's really awesome. So give us the main takeaway from it. One of the biggest bits of advice in it that I tend to tell people is if you have a class that only has two methods, one of those is your dunder init method. And the other one is some other method, especially if it's called call, you probably need a function instead of a class. So this is something I recommend a lot to folks moving to Python from Java and C++, because I teach a lot of folks who are moving programming languages. And you know, Python is just different from the other languages, especially if what you're going to have is effectively a static class, right? Like, yeah, it's just one copy of the variables.

It's very likely that you could get away and you're not doing any inheritance, you're not doing that kind of stuff, you probably could just have a set of functions and module level variables and you get effectively the same output yeah absolutely that's something i see a lot in code on the flip side of things though one of the great things about this talk is it doesn't just show you when not to write classes it does show you a couple examples of when classes are really handy in python yeah that's cool i'm still i'm a big fan of object-oriented programming. I think there's still, you know, people, it fades in and out of popular.

I still think it's a really great way to model stuff. But when you're creating more than one of the things, typically, right? Absolutely. Yeah, it's when you need to make more than one of them, when you need to use it more than once, that's the time where classes need it. They really can be abused. Yeah, absolutely.

So what's up with PyPI, Michael? I'm going to tell you about PyPI. I'm super excited. But before I do, I want to tell you about DigitalOcean. So there's this thing coming up where I might need a whole lot more power for my web server because there's some big promotion and it could kind of end up overwhelming my servers, which so far they've never had an issue. Like you log in, it's like 3%, 4% CPU.

But one of the things that's really cool about DigitalOcean is if you have one of their servers, you can go over there and you can flip a switch. And within just a couple of minutes, it will take it offline for a second, resize, re whatever, however you change the infrastructure. So I want four CPUs and eight gigs of RAM, change it to that. And you could do that for like one day and then go, no, no, back to the one gig, one CPU the next day, right? Spend five bucks and get like some super huge server to handle some spike that you know is coming. So pretty cool. A lot of the stuff that I do on the web is running through DigitalOcean. So definitely recommend them. Check them out at do.co slash Python. They help make the show possible. So another thing that I think is doing really awesome stuff on the web finally is the new PyPI.

Do you remember back when there's pypi.python.org slash pypi? Because it's better if it ends up on both ends of the URL. That's the original gray that's been around forever. What the PyPI has been. And then for a while, there was a PyPI.io, which was like the new one. Then there's a PyPI.org, which they redirected to. I think that's the same site. It's getting confusing here. I know. And so there's been a really, almost a big problem with the old PyPI in that it was based on

just super custom hand-rolled socket processing. And the entire PyPI was running out of a handful, two, three Python files, thousands of lines long, purely custom socket code. No, it's not based on Flask or Pyramid or any of those things. It's just like, yeah, we're just going to start at the lowest level. I think it was more or less a prototype and it just became what PyPI was for many years. And that was actually a big problem because people will come on and, hey, I got this cool idea. I want to add a feature. Like, can I just plug that in as an extension or, oh my God, what is that? I'm not going to contribute to this project. I'm out of here. Right. And so finally, finally Donald stuffed and crew have released pi pi.org as an official thing. You still have that red bar up there like warning. This is like a testing area. It's no longer a red bar like you go there

and it's properly a website going there right now to make sure Oh, wait, it's back again. It was gone earlier. There is it so there is a red bar. I think you can close it. However, this I love that this is officially it's coming up here, which is super exciting. Yeah. So the big news is not the website itself, although that's sort of the portal to it. The big news is when you pip install the thing, it now goes through PyPI.org. Oh, that's really exciting. So this actually is not only the future, it is the present of using pip with Python. Yes, that is where we are. So there's a really cool tweet.

I'm sorry, I forget the guy's name who sent it out that has like graphs of the analytics of sort of the number of requests per second and errors per second and memory usage and latency, all that for the new pypi.org handling the pip install requests. That's exciting. Yeah. So I'm really glad to see that. I guess I did just close that X and it stayed closed. You're right. So I opened it in like an incognito window. It's still there, but it's, it's almost, almost gone. Almost gone. Right. It's good to at least cookie you. That's right. It is good. So one of the biggest challenges you are, you opened this whole section with like getting started and converting from Python two to Python three. And I would say

the biggest thing that people complain about saying this is really hard to get this transition over is the old strings, byte strings, Unicode, bytes, array, madness around that, right? Yeah, absolutely. This is one of the biggest challenges is figuring out what is text and what is data and separating those because Python 2 didn't make a distinction. Python 3 forces you to make that distinction. So there's a nice talk by Ned Batchelder, right, that you pulled up for us? Yeah, absolutely. It's called Pragmatic Unicode. I think it's Pragmatic Unicode or How Do I Stop the Pain is the full talk title. It's counterintuitively, this is from PyCon 2012, and it is mostly Python 2 code. And so if you're

switching to Python 3, this might seem odd to watch this talk, but the reason this is still relevant today is Unicode hasn't changed. The default way we see Unicode in Python 2 versus 3 has changed. And there's a big analogy in this talk that I really like, a metaphor that Ned uses,

the Unicode sandwich, which I'll let you go watch the talk to figure out what that is. But this really helps shape your mental model of how text works versus how bytes work in the Python interpreter. And I think this is really important, especially if you're not familiar with Unicode and you're going to switch to Python 3 or to start using Unicode. You've got to understand a little bit about how it works.

Yeah, this is really, really interesting. And Unicode is definitely one of those tricky things. And I think growing up in the United States definitely at least puts, I felt like put me at a disadvantage of appreciating Unicode, you know, because it's like, well, ANSI, what's wrong with ANSI? I don't know. It seems fine. people is emoji, which is being used even by people who are in English speaking countries. So the rest of the world is finally going, yay, everyone cares about Unicode now all because of these emoji characters, which is sort of sad in a way, but also sort of good because it means we're actually caring about adopting this universally. It's really interesting because it's a bit of a sad commentary on the world that that's what pushes us forward. At the same time, it is good that it's being pushed forward, right? Absolutely. Yeah. Nice. All right. So this last one that I want to cover really has to do with thinking about alternate implementations or runtimes of Python, right? It's easy to say,

well, there's CPython and there's just Python, right? I type Python and it runs, but there are actually many different runt times and i say run times because sometimes they're called interpreters but only some of them interpret right so like um we have iron python python.net jython scython a whole bunch of other ones that are like a look more niche and i'd say probably the most popular alternative one is PyPy, P-Y-P-Y. Have you done anything with this? I have not done anything with,

well, I played with PyPy, but only in one of those kind of like, maybe I'll pip it, or maybe I'll apt to get install it and see what happens. Yeah, exactly. Yeah, so PyPy is a JIT compiling implementation of CPython. So the way CPython works, even though it's written in C, when you first hit your code, it all gets compiled to bytecode. But then those bytecodes

didn't get further compiled to machine instructions, they just get interpreted in like this massive switch statement while true loop. And it just looks at things and it just interprets them as bytecode at runtime. But that's not the same as compiling down to machine instructions, right? So PyPy, surprisingly, does the same thing that CPyPon does until it finds a slow spot and then it switches it to a JIT compiled version, which is interesting. Wow, that's really clever. Yeah, it just finds the slow spots and go, did you call this a thousand times and it's slow?

All right, we're going to make that fast. We'll fix you up. So there's a little bit of a slow startup. So if this was like a script, it runs for half a second and it goes away, there might be no benefit or less benefit, maybe even a drawback. I don't know. But if you use a web app, that's going to be up for six hours and maybe get recycled, or it's a game say, but we're a game that we're running, then that stuff would get jet compiled and be sort of warmed up, if you will.

Yeah. So you're sure here, I think we're talking about games specifically, right? Why would it be a game that this is nice for? Yeah. So there's a, one of the nice ways to make games is with Pygame, right? And this guy, Rene Dunfield, he was at the PyPy Sprints in Switzerland, which were like in the Alps at some beautiful resort. I'm like, man, I need to go to a sprint up here. Anyway, they were working on it. And one of the things they got running is they got the new Pi Pi running Pi game perfectly well.

Apparently, there were issues with it before. And they said on some of the games, they got up to a 30-time speedup by just simply typing something different to start your game. Wow. So they got a 30-time speedup by instead of typing Python, they type just run it in PyPy, and they get 30 times faster. That's amazing. Yeah, it's the same Py game, same code that runs on CPython and so on. So PyPy has a different way to implement C extensions and different APIs. So this is really nice. It has this new way to do extensions extensions and different API. So this is really nice, right?

It has this new way to do extensions for C or assembly language. It's the CPy EXT is getting faster, which apparently was important for the game strong, quicker, and so on. So basically, the people are saying like, it's really nice to have one code base that I don't have to try to like split across these different runtimes. It just runs in both places. Apparently there's a few types of operations that are slower because transitioning into and out of PyPy down into the C level and back is slower than regular CPython apparently, but still pretty cool. That is, that's very cool. Especially if you're using Pygame, this is exciting news.

Yeah, for sure. Like who wouldn't want their game to run? Like you think of like how much people optimize video games to shave, like get a few frames per second here or there. Right. So one example, and this is massive. Like one example they gave is like ray tracing and they had run it on Python 2.7. It took nine and a half minutes and then they ran it on PyPy and it took 18 seconds. So that's a pretty good speed up. That's not just a pretty good speed up. That's an incredible speed up. If my tests took 10 minutes to run and then they went down to seconds, I would be pretty amazed. Yeah, that would be so awesome. So if you're playing with PyGame or even if you're doing stuff on the web or whatever, check out PyPy. It could make a huge difference. Awesome. Yeah, it's good to see that project continuing to grow. That's it for our items. You got any extra news or things you want to share with people? Oh, it's tough to decide

what's new. Going to conferences, going to finding new things on the internet, making new things on the internet. The one thing I think I'd want to share is I launched a thing a couple months ago, kind of quietly, and I've been slowly kind of testing it out with folks called Python Morsels, where I send an exercise out every week. And this is my way of trying to serve the folks who are at my local study group and other people that I meet on the internet who want to learn Python. And they don't have a company that they're working in that does Python that could bring me in for training because I love teaching and I really want to serve individuals. So folks could check that out. It's a pretty fun service. Nice. That's just pythonmorsels.com? Yes. Yeah. pythonmorsels.com. And it redirects right now, but you know,

one day in the future, it'll have its own domain that doesn't redirect. It'll be all grown up on the internet. Exactly. Yeah. That's awesome. Yeah. So I'll put that in the show notes. Yeah. I just have one piece of news to share. I've launched, well, I kind of talked about this last week, but now it's in full swing. Launched my 100 days of Code course with the PyBytes, unrelated to this podcast, but similar name, PyBytes guys, Bob Belderbos and Julian Securia. So really happy to have this course out. It's an 18-hour course that guides people day by day through 100 Days of Python. So that's pretty epic. Yeah, I scrolled through the page for it. It is a

long page. There's a lot in that course. It's super exciting to see that. It gave us a chance to play with stuff that's really pretty far out there and really fun, like building GUIs in Python or different ways to consume APIs and just lots of really interesting stuff that we got to play with. So people can check it out at talkpython.fm slash 100 days, one zero zero days. Put that in the show notes as well. Nice. Yeah. All right, Trey. Well, thank you so much for jumping in and being a co-host for this week. It was fun to share these items with you. Yeah, thanks for having me. This was super exciting.

I hope to be back one day. We'll have you back for sure. Thanks. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at

PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.