Python Bytes - #118 Better Python executable management with pipx

Episode Date: February 22, 2019

Topics covered in this episode: [play:7:52] Data science is different now Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/118

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 118, recorded February 19th, 2019. I'm Michael Kennedy. And I'm Brian Ocken. And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. More on that later. Right now, Brian, it's, man, it is cold outside.

It's not bad. It's, I would say it's frozen. Okay. Oh, yeah. The whole transition thing. You're so good at that. The first topic is frozen flask. This was sent in by somebody who I probably should have written their name down, but it was recommended by a listener.

And it's actually pretty darn cool. So I've been playing with static site generators a little bit. And here's the idea is if you already have Flask application up, and it's mostly static gate content anyway, or it's got some, the activity is more JavaScripty or something, you can just freeze that into like a statically servable web thing with Frozen Flask, and it's pretty painless.

Frozen Flask looks so cool. Yeah, it really does. And then I also stumbled across a tutorial that's called Dead Easy Yet Powerful Static Website Generator with Flask, and it talks about Frozen Flask, and it also talks about an extension to Flask called Flask Flat Pages, which allows you to have a bunch of pages generated from Markdown. Oh, that's pretty cool.

Yeah, I haven't heard of that. That's nice. And it's pretty fun. I've got to warn you, I tried to go through this tutorial. It's still got a lot of great information. It's well written. It's just, since it was from 2012, it's a little out of date. Some of the imports don't quite work.

So you'll have to do some Googling. But the frozen flask that we also link to has some really cool. It's some pretty easy tutorials on there, too. So it looks fun. Yeah, that looks really cool. There's always this tension between having like a data driven website where maybe I've got a bunch of podcast episodes or something and they all have notes, but I want to have like a whole bunch of stuff around them and have that generated. And of course that would be like a regular flask data-driven app. Right. But if I'm going to do

that with a static thing, like how do I do that? Do I like create a bunch of static markdown? How do I mix it all together? It can be painful. Right. So frozen flask is like, well, you create the dynamic thing, but then you host it. You don't have to deal with the infrastructure hosting. You just put HTML and CSS somewhere and it runs great, right? Yeah, like for instance, I was thinking it'd be a neat way to do this. I was always potentially looking

for ways I can use something also. Like if I've got something internal for our test team, for instance, and we want to look at all of our test data and analytics and all that stuff, having a more dynamic site would be really neat, but then also a periodic generation of like a version report or something as a static site. This would be useful for something like that.

Yeah, it's a place that I just want to throw out as like an extra for this part that I have no experience with but seems super cool and i think you also are familiar with this is netlify yeah yeah so netlify is a place specifically for hosting static sites so it says build deploy and manage modern web projects and it gives you a global cdn continuous integration and automatic free HTTPS. If they give you your own domain name, 100% for free. There are ways to pay for it, but so much of it is for free that this seems like a really good option. Yeah, and they're not sponsoring this episode.

It's just cool. Nope, that's just something that's cool. That's right. Digital Ocean, they are sponsoring this episode. So last week, I think it was, we spoke about the new PEP that is for dunder pie packages yeah and you said that's awesome except yeah it's awesome but and one of the challenges you said was well if you just want to have packages and put them in there and run your app that's totally good it'll pick it up and run but the big the thing that was missing was to run the command line binary apps like pi tests or talks or p serve or you know like micro whiskey all those things

that would then get stuffed into that place that is really nearly impossible to put into your path right yeah yeah well not impossible but not impossible but it's a it's a super pain like every time you create a virtual environment to add it yeah to the path right so so chad smith who had worked on that before uh well we mentioned who was as part of the this python loke for like local python project to let you do like a preview interaction with that sent us a message and said hey actually you should know about my other project called pip x i guess for a pip executable or something like that. And the idea is it will execute binary Python packages in isolated environments. And the closest analogy to this is probably Pipsy, which we have spoken about before.

Oh, yeah. Okay. So the idea is, yeah, you can create these. You can basically, instead of pip install a thing, you can pipx install a thing. And it'll automatically create a virtual install a thing, and it'll automatically create a virtual environment for you, install it and its dependencies. And even it's even nice, it makes sure your pip is up to date, which is almost never the case when you create a new virtual environment. And it just installs it there. And then it puts it in the path.

It puts it in a location on your system that when you install pipx is in the path, but is not in the restricted part of your operating system. So you can pip install anything you need to run. So you can run it from the command line in your path without doing sudo. Okay, so the entry points are going to all go from a bunch of different virtual environments are all going to go to the same place, I guess. Yes, it does a bunch of like linking back some place that's in the path back to the various virtual environments it manages.

Okay, this is very cool. Now I'm excited. Yeah, now I'm even more excited. Yeah, this is super cool. So you can safely install these packages into virtual environments, which is where they belong, but globally, but without messing up other things

or putting them all into one virtual environment that's in your path. You can do things like list the stuff that's installed. So you can see which of the, which it's kind of like brew. You can say, what have I installed here through pipx, which is pretty cool. I'd also run binaries out of the pypackage directory as well. So it's kind of a companion tool to Python loc as well. Okay, nice. Being a Gen Xer, I'm a sucker for anything with X at the end. I know, it's amazing. It's the best. So basically, you just pip install pipx. And then from then on, you can say pip install like ptpython, which is an awesome thing. You could pip install pytest. You could pip install all sorts of things, glances. And there's a couple of ways in which

it's sort of like PIPC, but also better. So one of the things that I really like, you know, I made the analogy to Homebrew before, is you can go and say, PIPX upgrade all. And so everything that you've installed throughout the last six months

or whatever with PIPX, it'll go and find those and see if there's any updates for any of your little Python utilities you've installed. Oh, cool. That's cool, right? Yeah. Yeah, and then there's also updates for any of your little Python utilities you've installed. Oh, cool. That's cool, right? Yeah.

Yeah. And then there's also a comparison, little matrix thing. By matrix, I mean one dimensional, single line. Anyway, there's a comparison to PIPC and it talks about how PIPX is under active development. PIPC isn't in a bunch of other stuff. It's based on Python 3.6 and so on. It uses VENV, not virtual environment. And there's just a bunch of little niceties and you know one that it made me smile is it prints out emojis when it's doing

stuff it prints out little happy face like little indicator emojis on the terminal which you know makes me happy yeah what's the other thing that does that pipenv does that right yeah yeah and yeah kubernetes some of the kubernetes tools do as. It's a trend I'm seeing coming along here. But there's a lot of smiley, happy little things on the terminal, which is, you know, it doesn't hurt. Yeah. Yeah, what's the next one you got for us? This is an article from Vicky Boykus

that's called Data Science is Different Now. It's a fairly lengthy article, but it's some pretty good information. So here's the situation we've got. We've been talking about data science. Everybody's been talking about, say, data science. There's just a lot of buzz around it. So there's a lot of people with just a little bit of training in it going through boot camps or MOOCs or... Do you pronounce MOOCs? I think so, yeah. Okay. M-O-O-C, the massive online courses, yeah. Yeah. Okay. Or theyO-C, the massive online courses, yeah.

Yeah, okay. Or they're self-taught or they get a remote degree or some other training and they want to get a data science job. So there's a whole bunch of, there's a lot of people. The jobs are also becoming more advanced. They have more specific requirements. And also just the job description is sort of vague. So what is it you're actually going to be doing? And when people are actually doing it in the workplace, it seems a lot more like engineering than this shiny new thing that we've been talking

about. That's right. It's just a kind of a good view of reality check for data science, people interested in getting into it. But there's a, she also ends with some advice, not just a complaining about it, but her advice is for people that wanted to get into it. Her first bullet point is don't get into data science, which I just laughed. I actually laughed out loud. I'm like, that's awesome advice. But no, basically, she said, just don't don't follow the crowd because following what everybody else is doing is not going to differentiate you. And she recommends a backdoor approach of starting as a junior developer

or in DevOps or project management or something like that and learn a lot of data science stuff on the job and then build that up into a position that you can work at the company as a data scientist. That sounds like a pretty good bit of advice because I feel like data scientists are a little more on their own than software developers. Yeah.

Like at a company, there'll be five or ten software developers working on one web app or one other automation project. But how often do you have five to ten data scientists all working on the same problem, right? I feel like they're all kind of doing their own thing a lot of times, which as a beginner can make it more difficult, right? So maybe get some of these engineering skills through the approach that you were just talking about

and then branch out into that more solo side of things. Yeah, and then she also has like some skill, a whole list of specific skills for, like let's say you did get a junior developer job or you got some other job, which skills should you beef up on? So she's got a bunch of, listed a bunch of skills to try to beef up on that. Yeah, what are they? Creating Python packages, putting R into production, which I don't like that one. You should use Python. But anyway, what do I know? We'll talk more about that later. Optimizing Spark

jobs so they run more efficiently. I don't even know what that means. Version controlling data. Ooh, that sounds neat. Making models and data reproducible. Version controlling SQL. Building and maintaining clean data in data lakes. I love data lakes and data swamps. It's just funny. Tooling for time series.

Forecasting at scale. Scaling, sharing. Sharing of Jupyter notebooks. Thinking about systems for clean data, and lots of JSON. Yeah, that's cool. All right. That sounds like a good advice. That's a good pick.

Speaking of good stuff and sponsored episodes, this one is brought to you by DigitalOcean. And they, you know, it's February 19th as we're recording this, Brian. And not long ago, five days ago, was Valentine's Day, right? Yeah. Yeah. So DigitalOcean, they actually had a Valentine's Day present for the world. And their present was database as a service.

So until now, you would go over to DigitalOcean, you create a server, and then you would have to put your own database on there or use some other external service like Object rocket or something like that. So now you can get managed Postgres databases on DigitalOcean, which is pretty sweet if you're going to work with Postgres. So you just check the box and they handle all the operational bits for you. It's pretty cool, right? Yeah, that is cool. Neat. Yeah, definitely. So check them out at

pythonbytes.fm slash DigitalOcean. New users get $100 credit to play around with Postgres or whatever else you want to play with. And yeah, they're very good service. Definitely can recommend them. So this next one is pretty interesting. It's not super actionable yet, Brian,

but it starts to approach something that I'm really passionate about. So the thing I want to tell you about is Rust Python. So we have the language Rust, we have the language Python, and somehow they come together. So the Rust doesn't replace the Python, it replaces the C in C Python. The idea is like here's a new interpreter, a new runtime built on Rust instead of on C.

Wow. Yeah. So that feels like a more modern type of language to use than just C, right, to build a foundation. And that might be a little bit nicer if they could get that working really well. Yeah. Yeah, and it's Python 3 only.

So 3, 5 and above. And basically the interpreter is written in Rust. And this comes to us from Frederick Averpill. So thanks, Frederick, for sending that over. But yeah, this looks pretty interesting. And I would say it's definitely under active development. When I was getting ready for this recording, I looked at the site and the GitHub said, last commit an hour ago. So it feels like people

are working on it. You always wonder, is this thing real? People do anything. Now I say that because they're not really ready to do much. You can create a REPL and you can run some basic Python and stuff, but there's not a great standard library written with Rust foundations and things like that. So there's a lot of work to be done. But the goals are to have a full Python 3 environment entirely in Rust with no CPython bindings or no C bindings really. And a clean implementation without compatibility hacks. Okay. Yeah.

So if you want to practice some Rust and play around, there's a lot of things that you could do here that might be fun. I think people who played with this would probably understand regular CPython better as well, right? Obviously, if you re-implement it in another language, that would be good. But they have a bunch of tasks on the issue trackers, and they're labeled very nicely, like good first issue and stuff. So if people want to get started, they can do that. Yeah, also kind of be a fun getting in in the early stages sort of a thing to jump in and help out. Exactly. Like in five years, if people keep working on this, this will be really hard to contribute to.

But right now, it'd be really easy. Yeah. Relatively speaking. So, why do I think this is super cool? Well, Rust comes from Mozilla. And Rust has good support for compiling to WebAssembly, right? Oh, okay. And WebAssembly is the much better, faster, stronger way of running something in the browser. So if this thing got off the ground and it actually worked in a meaningful

way and it could be compiled to WebAssembly, well then, all of a sudden we have a super awesome Python implementation in the browser on the client side. so we get a python really full stack on both ends the server and the client yeah yeah so it would be super cool and and this is um one possible path down that i don't know i don't know that that's the goal but they do talk about web assembly being an option in sort of a toy sense because it's not really quite ready yet but there's some stuff there so anyway that's that was what really got my interest i'm like okay another implementation address is kind of kind of interesting, like, oh, wait, WebAssembly support? That would be a big deal if that came up. Yeah. Maybe in a few years, we'll see something like a book called Python Script,

Just the Good Parts. Exactly. Very cool. All right. Well, what's the next one? The next one is, this is an article a article actually it's kind of a team up between uh mike driscoll and real python so uh mike driscoll wrote an article on real python called the jupiter notebook and introduction and of course we've talked about jupiter notebooks before and a lot of uh stuff around data and around that however this is actually a tutorial that i think was kind of missing and it's just it's not the all the think cool things you can do with it but it's i know i want to use it

no really how do i start and it's uh it really starts from installing it of course up to uh and then just kind of walks through some of the some of the real basics like creating a new once you get it up and loaded you've got a blank notebook or a blank interface and you can create a new notebook in there. And then even something simple like, I didn't know that it was so easy to change the name just by clicking on the name of the untitled notebook that you get right away. And you can click and rename it something else. It talks about how to run cells. I think I played with notebooks for quite a while before I learned the shortcut of a shift enter.

And then, um, goes through some, just some of the menu options and some of the neat features that you might not be obvious if you didn't think of exploring them before. Like, um, file has a save and checkpoint as you're working on a notebook.

You can, um, it's kind of like a version control thing built into it. You can just save it, save a different checkpoints and move on that's cool the edit menu has um i wouldn't have thought to look there for things like uh splitting and merging cells or moving cells around but it's in there and then the uh cell menu has a whole bunch of cool ways to run i knew you could run a cell or run everything but there's also things like run a subset or run everything above this cell or below

this cell or something. At first when I was reading this, there's a couple questions that I have right away when I start with the notebooks that wasn't covered. So I'm going to go ahead and push promote these practices right away. Is when you're

installing Jupyter, if you're installing jupiter if you're installing with pip do it in a virtual environment and then all the installs within your notebooks will go in that environment you can keep those separate from the rest of your work of everything else once you have a notebook up you can pip install within there now with the uh percent pip install that works now thanks to i think at least uh at least Jake Vanderplast publicized it. I'm not sure if he did the work or not. So that's kind of really what I wanted to talk to you about. This is a really good, I think, a good article to send to people.

If you're working with notebooks and you want to share something with somebody else and they've never experienced them before, just send them to this also. Yeah, it's a good resource and very comprehensive. So definitely a good one that Mike Driscoll put up there. Now, I want to round out the show talking about the Python developer survey results from 2018.

This is the end of 2018. So this is pretty fresh from the PSF and JetBrains. We talked about their stuff a year ago when they did it for 2017, the end of 2017. So this is the next year. Have you seen this yet, Brian?

I've only briefly looked through it. Yeah, so there's a couple of interesting things that I pulled out. And one of them is that Python usage as a main language. So one of the questions they say is, so you're here because you use Python. Is that because you use it as a secondary thing? Like maybe you're a C++ developer, but you use it for DevOps or something like that. Or do you use it as your main language? The answer of Python is my main language is up 5% from last year.

That's awesome. Yeah. So now it's 84% of the people and it was 79 last year. So when the PSF conducted the survey. The other big thing to talk about maybe is what do you use python for and the number one thing is data analysis and then web development and then machine learning okay that sounds like a lot of data science and probably a lot of jupiter stuff happening yeah between those top three right and also so that one of the things that's cool

about this survey is now they've been doing it a few years, is they have a comparison, like a trend predictor. Is the derivative positive or negative on this trend here? And so I put those numbers down for 2018 and 2017. So for data analysis, it's 59% of the people now use that for data analysis, use Python for data analysis. That's up 8%. And it's above web development. So you might think it's a zero-sum game, like more people are not doing web and now they're doing data science. But no, like web development's up 2%, machine learning is up 7% to our comment and all that, right?

So pretty interesting. And the other thing they let you do is they let you ask four people who answer this question, let's partition their answers by whether Python is their main language or it's just kind of a secondary thing they do, right? So the main takeaway of this section is that web development is the only category with a large gap separating main users of Python versus secondary ones, like 56 versus 36 percent, 20 percent difference depending on how you use Python. All the other ones are pretty similar. Okay. Across those two, yeah. Yeah, I want to point out a couple other things. Okay. From the survey, 93% say that they're involved with software testing

or writing automated tests. That's pretty cool, right? That's awesome. Or at least maybe that's professional versus hobby. I don't know. Yeah, anyway. Yeah, but it's quite high. And they said they ask you which test frameworks you're using.

For test framework, PyTest is number one. Of course. Of course. That's pretty cool. Another one that's interesting is kind of like last year they said sort of scientific computing, data-oriented, like data analysis, machine learning, whatnot. How does that compare to web development? And it's now 27% web development, 28% scientific development, they call it.

It's like data science plus machine learning. Also Python versus legacy Python, super good news here. Of the surveyed folks, 84% are using Python 3 versus 16% for legacy Python. Yeah, that's awesome. So that's really good.

And it also is growing very rapidly. So like 9 percent more people are using modern python which is great uh web frameworks flask is now the most popular web framework on the survey jango was last year yeah that's interesting yeah but they're they're still really close they're neck and neck for sure then it goes tornado and then pyramid and then databases postgres mysql sqlite those are the top three relational and then mongodb is uh tailing along in the noSQL bit. And then ORMs, SQLAlchemy and Django ORM are tied for that. Anyway, a bunch of like, that's just skimming the surface. There's a ton of cool little interactive bits you can go explore the survey with. Yeah. Neat. Yeah. So good stuff. And yeah, people should go check it out. Of course, the links in the show notes. All right, Brian,

you got anything else you want to share?'s not one of our main topics yeah nina mentioned to us you can pronounce her name but i have trouble with this zacharino yeah zacharenko she uh hit us up and said uh if we could plug this it'd be great mentored sprints for diverse beginners or something that's at pycon i don't know if they've always been a thing but that's something that's uh available this year I don't know if they've always been a thing, but that's something that's available this year. It's a newcomer's introduction to contributing to an open source project.

And we have a link in the show notes. Yeah, cool. You do have to both. There's a call for applications, both for projects and for participants. That's open until March 14th. And a highlight, if you're wondering

if this event is for you, it definitely is. And we'd love to have you wondering if this event is for you, it definitely is. And we'd love to have you take part of it. So that's a neat thing. Yeah, they'll take place on Saturday, May 4th from 2.30 to 6.30 at the main PyCon. So that's cool. Yeah, that's a great thing. I'm glad we get a chance to highlight it. All right. All right. Do you got a joke for us? I don't have a joke.

All right, I grabbed one. This one plays a little better in cartoon form, but I'm going to try it in audio. Okay. Okay. So if you have some pseudocode in like a sample.txt file or something like that, what's the easiest way to convert that to Python? I don't know. How? You change the extension to.py.

For some reason, the pseudocode is very similarly written to Python. And, of course, it speaks to Python's readability, right? So maybe you got to throw a colon in there somewhere. But yeah, pretty much. And I have an extra. I actually have two extras for you. Okay. The first one is this Python song.

And it takes a bunch of like modern songs, and it converts them into like learning chapters. So you learn things like you learn about Python basics, like looping and variables. You learn about classes, you learn about Python basics, like looping and variables. You learn about classes, you learn about web design. So there's like a little different song for each one. Is these folks somewhere like Sweden or Norway or somewhere like that? I can't remember. Maybe Denmark.

And they put together this really funny, like six and a half minute set of little songs I'll put together into chapters. And it's all about Python. I'm totally going to subject my daughter to this. Yeah, yeah, it's pretty funny. And then I'll throw one more in there really quick for you. Speaking of daughters, my daughter literally sent this in like about an hour ago.

And speaking of all the data analysis and data science folks, I think this one might resonate. I'll let you be the judge. So an infinite number of mathematicians walk into a bar. The first one tells the bartender he wants a beer. The second one tells the bartender he wants half a beer. The third one says she wants a fourth of a beer. The bartender interrupts and puts two beers on the table and says, people, you really need to learn your limits. Okay, that's funny.

It's pretty good, right? Yeah, definitely. Pretty good. All right, super's funny. It's pretty good, right? Yeah, definitely. Pretty good. All right, super. Okay. Well, Brian, these joke segments always make me smile. Yeah, definitely. Yeah, thanks for doing the episode, as always.

All right, thank you. Yep, and I guess we will see everyone at Pi Cascades, right? Oh, yeah. That will be the next thing. That's this week. Yeah, we head out in like three days or something like that for the conference. So if you're going to be there, hopefully we'll get a chance to meet you. Otherwise, see you some other place.

Alright. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way.

We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.