Python Bytes - #65 Speed of your import statements affecting performance?

Episode Date: February 14, 2018

Topics covered in this episode: pygal : Simple Python Charting Thoughts on becoming a self-taught programming How to speed up Python application startup time (timing imports in 3.7) AnPyLar - The Python web front-end framework Migrating to Python 3 with pleasure Moving to Python 3 Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/65

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 65, recorded February 7th, 2018. I'm Michael Kennedy. And I'm Brian Ockett. And we got a bunch of cool stuff. I really am excited to share a couple of these things with you. And I'm excited to have Rollbar as a sponsor. So you probably heard about Rollbar from other episodes.

If you want to check them out, we got a special offer at pythonbytes.fm slash rollbar tell you more about that later i kind of want to know about some charts i have got like this just have you ever used seen the test output on uh like jenkins if you have testing no i haven't played with jenkins okay well so in in jenkins builds there's this thing where you can after you run your tests, you can have it display your passed and failed tests in this little graph. It's just a little chart. It's an area chart. I wanted one of those, but I wanted something that would show like five of those charts, a whole bunch of them.

So it doesn't do it automatically. I probably could have hunted into Jenkins or something, but I wanted to build my own thing. So I want to do it in Flask. And I've been trying to figure this out. And there was a great tutorial by, this is going to be a little story anyway, but there was this great tutorial by the Matt McKay from Fullstack that showed how to do this, almost how to do this in Bokeh or Bokeh. Yeah. Very nice. That's a cool project too.

Yeah. But Bokeh, the current version of Bokeh, dropped the charting feature. That's not helpful. But there goes your solution you just found. Yeah, so I'm trying to do it, and I gave up and asked Twitter. And what came back was one of these options was PyGAL, P-Y-G-A-L, which ends up very, it's very simple, but I can, within 20 minutes, i had a flask app running with these charts up so it's not as hugely featured as some of the other graphing applications but it's very cool and it also can export svg so these are small it's pretty small also when you're displaying these that's cool and the the svgs can have basically infinite, very high levels of

sort of zoom ability, right? Because they're scalable, right? That's their first name. That's the S. They also do PNGs, like you said, that's cool. And they have a bunch of great looking charts. Like these are really nice. So you can do like bar charts or pie or the radar ones, all sorts of good stuff. So I still want to get something like Bokeh or something like that to work so that I can make them interactive because these aren't interactive things. You build them and just display them. But for my needs right now, that works,

and it's very nice. Yeah, very nice. And kudos to the people writing the documentation because they have documentation on how to, pretty much they have it in Django, Flask, and pretty much any HTML that you want to throw this in, how to do that. have it in Django, Flask, and pretty much any HTML that you want to throw this in, how to do that. Oh, yeah, that's super cool. And they have nice pictures, which is always really powerful when you're talking about graphs or

UI or things like this. You can just admire the picture. Okay, I want this. Now I'll pay attention. Really nice. So a lot of people send us messages and they ask for advice on like a sort of career path or I'm thinking about this technology or how do I get started in programming or, you know, something like that. And there was this really interesting post over on Reddit under learn programming. And so it's not technically a Python thing, but a bunch of Python people are hanging out there. So that's kind of cool. And the idea was this guy, he gets a guy who wrote me anyway, person posted it, sent a message and said, Hey, look, I'm 31 days into a self-study program on Python. I'm loving every minute of it. A couple of questions. Could you all fill this out just so we can share our experiences?

Oh, interesting. Isn't that cool? So what were you doing before you began self-study and programming? So it's for people who are doing self-study. What made you want to study programming on your own? How did you get started? How long did it take for you to feel confident enough in your skills and knowledge to think that you could be employed? What else did you do besides self-study that helped you?, what's next? And this thread just blew up on Reddit and there's all these really interesting conversations. And so anyone out there who is sort of in the early stages of self-study, maybe they're doing a hundred days of code type thing, or they're in a bootcamp or they're taking, you know, online courses. I know a few places

have good ones of those. I think going through here and reading this, it would be really, really valuable to sort of have that shared experience. Isn't that cool? Oh, that's great. Yes. So there's another story that's linked from there. So this one doesn't technically count as an item, but people might also find it interesting

as a thread to keep following is someone said, there are people who are like 30, 40, like, hey, I'm learning to code in my 40s. And you guys are really inspiring me to realize this is actually possible. It's not too late for me. So there's this article called Stories from 300 Developers Who Got Their First Tech Job in Their 30s, 40s, and 50s linked from there. That's cool. I got to go read that.

Yeah, and I think that might be real helpful to some folks as well. So if you're in this space and you want some shared experiences and to connect with some people, check out this thread we're linking to, because I think it's helpful. On that topic of learning, actually, there's a couple episodes of Test and Code that'll be good. I just did one recently about extended learning through universities, which I hadn't considered before. And then also there's upcoming. I'm talking with Stephanie Hurlburt on talking with mentors and opening yourself up to be a mentor as well. Yeah, yeah. Very cool. I'm looking forward to checking those out.

This is one of those well-planned, awesome transitions. Amazing. So one of the things that you probably don't care about when you're getting started is how fast your imports run. You probably don't even know what an import is at the beginning. Exactly. But import, how long it takes to import your different modules is part of the painful startup process of any Python application. And I didn't really know how to debug that, actually. I never really thought of it before. But in Python 3.7 coming up, there will be a, there is a dash X import time that allows you to, there's a flag that you can run that allows you to investigate and profile all of the time for importing different libraries, which sounds really cool, actually. Yeah, that is really, really quite cool.

And so you can just basically run it and it'll tell you, this library took that long to import. This other library took that long to import and so on. And I didn't realize how long some of these actually took to load. And like any optimization, it's probably never where you really think it's going to be. It's probably someplace else. So having it profiled is great.

And sometimes you just have to import things, but some parts of your system may have too many imports initially and that could be possibly delayed till later. Yeah, you could definitely possibly do it conditionally, right? If you only know in certain circumstances that code's going to run, so you could maybe somehow delay the import until you actually need it. So a lot of a lot of cool things right like let's say you've got a like an editor application or something you all of the stuff that you'd need to i don't know like convert your format so that for saving

you don't really need that all the time you could load it when you when somebody's trying to save right wait wait for him hit the command s control s something like that that's right yeah yeah so very very cool and so this is a feature of Python 3.7. This is like in the new fancy version? Yeah, it's a new fancy version. But I mean, everybody's probably, if you've got an application working in 3.6 right now, you're probably thinking about making sure that it's going to work in 3.7 by now. And you can probably use the 3.7 to optimize current code anyway.

Right, because you want data classes and why not be able to time your imports while you're at it yeah nice yeah so this just came out in beta last week and we already talked about that so here's one more thing you can do if you've got some library that's taken a while to basically import i think that article that you mentioned that you know, John, the conclusion, it says, so I can reduce my time for pip and dash dash version from 800 milliseconds to 500 milliseconds. That still sounds like a long time, but that's actually, you know, that's 60%, 30%, depending on which angle you measure it. And not quite, but it's a big improvement. And it's down into the um the pieces

where it makes it less noticeable from people yeah quite cool all right so before we move on the next one let me tell you about rollbar so rollbar is the thing you integrate into your web applications whether using django flask pyramid super easy to integrate basically to integrate it into pyramid you just put a few things in your config file. And that's it. I don't even think you have to touch your code unless you want to directly interact with the rollbar API, which you often don't have to do. And you can even get it for JavaScript and some other frameworks as well. And it'll basically be there when your app is running. Anytime there's an error, it'll capture all the details, the stack trace, the URL, all the request and response values, even the user who is logged in

potentially. It'll send that off and give you notification over in Slack or email or all sorts of places. So you want to make sure you're not missing errors in your web app. And if you're not monitoring it, I bet you there are errors in your web app somewhere you don't know about. Check them out at pythonbytes.fm slash rollbar. Brian, speaking of web apps, this one we're going to talk about now is it's not the kind of web app you might have originally thought of. I didn't mention it there in that rollbar ad. Anpiler? I'm really sure about the pronunciation. You want to take a shot at it? Anpiler? I don't know. It's kind of fun to try. It's fun to try.

And I think the and is kind of like Angular. The py, obviously Python. The lar, what a great thing to put it in a word. But what it is, is it's a web framework for building front end, rich client spa type applications. So it's a Python framework. You write in it. It runs on the client side in the browser. Isn't that interesting?

Yeah. So it says basically create web apps with the elegance, simplicity, and full power of Python. And you get these sort of reusable components. It's very much like AngularJS in that regard that has like a routing engine. So you don't actually navigate, but you move around the different views of the app. You write all this stuff in Python. It's pretty interesting. So it's got a reactive programming model. It has promises,

standard Python formatting, reusable components, the scope styling, integrated routing engine. That's the navigation stuff I was talking about. So it's pretty cool, actually. I don't know what the underlying technology is for making the Python run, if they've built their own thing, if they're using something like Brython or Sculpt or something else. But yeah, it's pretty amazing. It sounds like it'd be fun to play with. Yeah, it's definitely fun to play with. The one caveat is if you're going to embed Python compiled to JavaScript in a JavaScript file and download it, that JavaScript file may be bigger than standard JavaScript file sizes. So I checked in, I think the minified version is 3.6 megs, which is a little large. It depends on your use case. So imagine this on one hand,

if you're building like a really high traffic website, people use it from mobile or, you know, low speed connections, it's totally unacceptable to put your front end and this. However, if what you're doing is you're building electron JS apps, you're building Ionic and Cordova apps, these sort of offline, like here are my files, run them as if it was a web app. It doesn't matter if the thing is big, right? It just starts up off the local disk anyway. Yeah, that's true. That's a bunch of cool stuff. And in that circumstance, like you just, who cares if, you know, your uncompressed shipping size is another three megs. It's already 100 megs. Like what's 100 versus 103? Nobody cares. Right.

Something like that. So for those situations where it's kind of this offline rich client web experience, it's cool. I like it. Oh, that's cool. That's a good idea. Yeah. So there's some nice ways to play around with it. They got some little demo apps.

So like down the bottom, there's like a little grid of buttons like eight or ten or something you click around there and play with it a bit so yeah it's it's pretty cool if if you want to give this a try and your use case makes a lot of sense right like it's somehow getting that javascript there's not a big problem definitely check it out it cool. The other thing to point out is WebAssembly is coming. WebAssembly is going to be pretty sweet. If somebody can get a really nice version of Python in WebAssembly, that may make a big difference on a lot of fronts. Do you know WebAssembly? I do not. So WebAssembly basically is a new standard for the browsers. I think it's already partially

supported where instead of shipping JavaScript, you ship binary executable equivalents, right? So you've got the problem of running, you download a text version of a big thing, and then you parse it, you do all the sort of runtime stuff to get it executable, and then you execute it, right? You get into like bytecode or whatever JavaScript does. So I think WebAssembly is more or less like we're going to get it executable and then you execute it right you get into like bytecode or whatever javascript does so i think web assembly is more or less like we're going to get it to that last pre-processed step of what javascript would do and we'll give it to you as a binary thing sort of like shipping pyc files in python i think it actually has more effect so it should be a lot smaller because it's small and tokenized and stuff but it should also start up quicker so who knows

maybe web assembly makes this better for somebody sneaking javascript or python through the javascript straw running random binaries off of any website on the internet what could go wrong i think it's only the equivalent of running javascript it's not like fully executing data so yeah okay i think it's no worse than javascript but yeah okay i don't know more than that we'll have to keep track i think we should round out this entire the rest of this episode on just python 3 stuff what do you think i don't think on this podcast we've pushed uh the migration to python 3 enough no we have talked about how bad legacy python is go ahead yeah you go first on this python thing and this is actually just a just a little GitHub repo that is called Migrating to Python 3 with Pleasure, a short guide on features of Python 3 for data scientists.

And it's a pretty quick read, but it with examples of how to use Pathlib so that you can, you know, easily, if people haven't played with it yet, you can, you know, define paths as just like these little just strings with division marks like you would in a browser. It's kind of crazy, right? Pathlib. It's like you put the slashes outside the strings and the division operator becomes like os.path.join type of things yeah definitely and it's but it works great and it makes your code really readable so there's that and it goes on to talk about type hinting and how cool it works with thing editors like pycharm and stuff to help you see what you're doing then one of the things that i did not hadn't heard of before i came about a few things in here is how to enforce types at runtime. I didn't know you could do that, but there's a package called enforce that you can put a decorator on a particular function, and it'll throw an error if somebody tries to pass in the wrong type.

I had never heard about that either. That is really awesome. I don't know if I necessarily want it all the time, but it could be fun to turn on, especially if you're like doing a refactoring or you're like doing some major upgrade. You're like, all right, let's turn this on and just see if it's doing what we think it's doing. Right. Yeah. Especially during like a testing phase or something. It'd be fun. Yeah. And then this kind of hurt my head. There was some function annotations for units. There's an example is how AstroPy does it. And it's things like you can attach to variables like units, like kilograms or something like that, which

that just seems cool. I got to play with that a little bit more. Oh, it looks like, which one is it there's a library i think it's called pint oh right yeah like right that lets you like multiply say like you know this thing time a foot that thing times an inch and then add them together yeah yeah that's it and this looks like a little bit of the same type of story but you might multiply by like a kilogram or a degree and sort of the physicist astrophysicist equivalent of that that's cool like in their example they show uh dividing a meter in speeds in i don't know what it is but um coming up with like uh terahertz we didn't start with

terahertz you started with something else okay now i'm just making up words now i think so pretty cool pretty cool. You also have matrix multiplication in there. Yeah, which I don't use a lot, but it's very cool for people that need matrix multiplication. Matrix multiplication is interesting. It's one of these things that's held up as one of the best examples of the whole PEP process and bringing a new feature into the language.

So from the time the PEP was proposed until it was done, it was like eight days. Wow. And it has to do with how well the people who proposed the idea had already done market research amongst all the scientific computation people. And they had a really clear vision and a strong displayed need. And so it's really interesting that that's in here as well.

And globbing with star star, which is something I hadn't used since one of the Z shell does that. So like you can do recursive subdirectories with two asterisks. And that's kind of neat. And then we threw a whole bunch of other stuff like F strings and floating division. Now is real, real division now

and the rest of it is things that if you've been living python 3 for a while you just kind of take this stuff for granted but um it's a very a fairly complete list of it'd be good if you're in a science community and you're trying to convince somebody to switch to python 3 this article would be a good one yeah definitely, definitely a good one. It sounds really interesting. Like it really brings up, these are the benefits that you get from making this move rather than just the stick of it's going to be expired or you don't go out of maintenance in 2020.

It's like, you know, here's all the good stuff that you're missing that you don't even know you're missing. Yeah, I didn't even list it, but it includes like a dictionary ordering and stuff, which I love that. Right. Yeah, that is really, really nice. I think Enforce might be the big news out of that. I hadn't even heard of that.

That's pretty cool. Yeah, I'm definitely going to go play with that tomorrow. So that's neat. Cool. So let's talk about moving to Python 3. Okay. So I want to give you the- Completely new topic.

Exactly. I want to give you something like this, but from a web development perspective, it's pretty interesting. There's this place called TicketEA.com. TicketEA? I don't know. TicketEA.com is what I'm going to go with. And they said how we migrated TicketEA.com to Python 3 in two weeks. Wow.

So it's apparently a pretty big project that they have running. It runs their whole sort of fulfillment e-commerce type thing and so on and you'll hear the word aphrodite that's kind of their their code name word for their project so they do a bunch of stuff with docker and their first step was to just say okay we're going to change the aphrodite's base docker image to python colon 3.6 slim from i don't't know, what's it, 2.7 fat? I don't know.

Something like this. And they said when they did that, you know, they just tried to run their test and a bunch of stuff went crazy. Like outdated libraries didn't work. Base string had to be moved to string. URL pars had to be moved to urllib.url pars. Other stuff like that, right? So I said, all right, well, the first thing we're

going to do is going to run two to three, right? So this is the utility that can manually fix some of those things. So like every time you see base string, that's STR. Now, we're just going to do that. For example, right? So they ran that against it. And they also look for patterns like, hey, we're using map reduce, or map type functions, and so on, or filtering functions plus lambdas to make it work. And these could really just be list comprehensions, right? So they kind of upgraded the, they took better advantage of like the new language features that were then available to them. So that's pretty cool. And then they said, okay, well, that's the low hanging fruit. Now what? And they had to run their tests, you know, sort of find some more problems. They took the opportunity to upgrade,

at least get ready to upgrade to Django 2 because they are running on Django. So that's cool. They said a couple of lessons we learned, code coverage was 75% or 70%. You know, the more tests, the better. It's good to keep the Python 3 branch up to date with the master branch until you switch.

That's one way of doing it. Like remember the Instagram keynote at PyCon 2017? Yeah. That is a whole nother level of upgrading to Python 3, right? They upgraded all of Instagram to Python 3 without branching, which is, yeah, at all. That's crazy. And they were checking in all the time. So that's another level. You can check out that as well. But this is, I suspect this is more common that you kind of branch. You're like, all right, we're going to try to keep these going. It's a good story to see somebody doing it the other way.

Yeah, for sure. And so they ran Flake 8 against all the stuff to make sure everything was working right. So I don't know, they said they had to modify around 200 files to make Flake 8 happy again with Python 3. But yeah, all good. And then the final thing I thought was really cool is they're running on Google App Engine.

I don't do anything with Google App Engine, but one of the features that these platform as a service have that's really pretty cool is something called traffic splitting. So there's basically like a little slider and you say this version of the app is going to get 60% of the traffic. This version of the app is going to get 60% of the traffic. This version of the app is going to get 40% of the traffic. And you can add in different versions. So you can say, well, let's just send like 1% of our traffic over towards these new versions and see what happens. So you don't completely take the site down. You just may break it for a very small group.

And you could even do that, I think, by like IP address and stuff. So you say only the people that work here get the new version. People on the outside, they get the old version and slowly migrated along. And so they call these canary releases, which I think is actually a pretty cool way to think of it. So there's some neat lessons in here. This is by. I didn't know Google App Engine did that. That's neat. Yeah, I didn't either. But I definitely like it a little bit better now. That's cool. I mean, I don't dislike it,

but it's definitely a feature that would draw me into it. All right, so I think these are really interesting. Yours was on data science. Mine is a sort of web story. And they both are compelling sort of for moving to Python 3. Things that you get and the steps to get there. Definitely.

And we'll keep pounding on it until Python 2.7 is gone. That's right. We will see legacy Python fade away. So you have a webcast coming up, don't you? Yeah, we brought it up last week, but I just wanted to remind people that I'm doing a webcast on PyTest and PyCharm with the PyCharm guys. That's coming up on February 22nd.

So we'll leave a link in the show notes and go register. Hope to see everybody there. Yeah, they should definitely go register for it. There's a couple reasons. One, Brian's going to be there. It'll be awesome. Two, it's free.

So you'll get what you pay for, for sure. I promise you that. Unless you count your time, then I don't know. And then you also potentially, if you can't make it, you'll get the recording notification. If you register and you don't show up, then you'll get a message you can't make it, you'll get the recording notification. If you register and you don't show up, then you'll get a message to say you didn't show up, but hey, you still get it. People should check out.

There's a whole bunch of old webinars there. And it's not just about trying to push PyCharm stuff. It's just people in the community showing some interesting things to do. So it's a good thing that they're doing. I had this project that I wanted to play with, not so much to build in the app, but to play with the foundational bits of the app. So I have, I have a program that sort of demonstrates using cookie cutter from an API mode. So cookie cutter, normally you write, you type cookie cutter space, some template name, and then stuff happens. And

then you have a projects like a starter project, a scaffold type thing for all sorts of various things. So there's also an API for cookie cutter. And so you can put like fancy front ends, or if you've got any kind of application that needs to generate, like a sub project that's going to be messed with by somebody else, if you're doing like developer tools, or you're building something for your company you're like, we're always going to start this way. So here's the CLI and it asks you the questions and then boom, you've got this kind of standard starter app that's integrated into your infrastructure.

So these are all good uses. So I said, all right, well, I want to play with this project called GUI. And we talked about GUI a while ago, G-O-O-E-Y. And so I took my little cookie cutter app that would work with all the pyramid based cookie cutter templates and put a friendly front end on front of on the front of that and then i put a gui on top of that cli app nice yeah so it's really fun and it was it took

it really took like 20 minutes to get it 99 working more on that a sec but so i put a link to the GitHub repository that has that up there. So people can download it. You just clone it, create a virtual environment, pip install dash R, the requirements file, and then you can just run the program and you have a really sweet GUI.

And it's based on WXPython Phoenix and GUI. It's quite cool. The one little hang up is I'm having a problem getting it to run if I package it so I could literally hand out a dot app or dot exe. There's some kind of hang up and I think it's a bug with GUI. I can't tell entirely, but maybe by the time you hear this, I haven't figured it out, but we'll see. Either way, it's a great example. You know, if you can forget the packaging, it's actually an awesome example of a really simple Python GUI app that looks professional. If you can forget packaging, Python is awesome.

The packaging works fine. GUI works fine. A packaged GUI app sometimes works fine. Yeah, okay. It's the intersection of all these things that are cool. There's a small bruise on the shininess of it. But it's still, it's all good. It's all good.

So check that out. There's tons of people working on all these features to make it even more seamless in the future. Yeah, absolutely. Absolutely. Well, good job. Cool. Thanks so much. Brian, thanks for sharing everything with all of our listeners and with me.

Thank you. And we'll talk to you next week. You bet. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm.

If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.