Python Bytes - #164 Use type hints to build your next CLI app

Episode Date: January 16, 2020

Topics covered in this episode: * Data driven journalism via* cjworkbench remi: A Platform-independent Python GUI library for your applications. Typer Effectively using Matplotlib Django Simple Task PyPI Stats at pypistats.org Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/164

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 164, recorded January 9, 2020. And I'm Brian Ocken. And I'm Michael Kennedy. And this episode is brought to you by Datadog. We'll talk more about them later. But speaking of data, data-driven journalism. Yeah, let's start there.

So there's this project called CJ Workbench, or just Workbench, something like this. And this recommendation comes to us from Michael Palowski. Sorry if I kind of mispronounced your name there, but thank you, Michael. And it's data-driven journalism as a service, as a platform, along with training on how to use it. Not training on journalism, but training on basically how to do data journalism. So it's a pretty cool idea. That is neat. Yeah, and so what you do is you can go and say,

point it at, I don't know, some website, and say, I want you to scrape this from the website, and then you go and you fill in this UI, you fill in and say, okay, now I want to filter out the empty columns, and now I want to do this, and that transform and other transforms and so on. And so it basically lets you do kind of what you might do with a

Jupyter notebook, but there's no code for the journalist, which is pretty cool. And so you might wonder, well, if there's no code, why are we talking about it? There's all sorts of software that does interesting stuff that we don't talk about. Well, one of the things you can do is if you find that the built-in features are not enough, then you write a few lines of Python and you start extending it for the journalists using Python, which is a pretty cool way. So instead of you worrying about the UI and all the basic stuff like loading CSVs and working with data frames and all that, you just say,

use this, and if we've got to write a little python code we will you can use their system and just add your own little code snippets to it yeah exactly so if you look at the features it has modules to scrape clean analyze and visualize data it's got built-in training like it'll teach you how to use the feature that you've got built in it connects to things like social and google drive and APIs like JSON endpoints, I think. It has recorded workflows, so it's really repeatable. It versioning. But finally, you can write custom modules in Python

and then add them to the little drop-down module library that you can use. Oh, that's neat. Yeah. Yeah. Plus, this is all in GitHub, so that if you really want to hack the backend, you can also.

Yeah, exactly. It's open source. It's on GitHub. to hack the back end, you can also. Yeah, exactly. It's open source. It's on GitHub. You can go grab it, download it, and run with it. But they also have it online as a service. So you can decide, do you want to run your own server? Do you want to use their public server that's in beta or whatnot? So I covered this partly because I think it's really cool.

And I also covered it because, boy, I think the world needs journalism right now. Yes. And yeah, more people saying, I don't know if I believe your analysis. cool and i also covered it because boy i think the world needs journalism right now yes and yeah more people saying i don't know if i believe your analysis i want to do my own analysis on it yeah and here you don't have to commit to a huge amount of work it's cool you pointed all these live sources like you can point it at some website and it can like real time refresh the live data that it's scraping for your article so it can evolve over time and whatnot. It's pretty cool. I might have to try this. Yeah, nice.

Have we covered GUIs yet on this podcast? I think I do remember way back when at some point we talked about GUIs, right? Yeah, at least once. Yeah. What do you got for us this time? Well, so here's another one, another GUI. This one's called Remi.

The Remi comes from Remote Interface Library. I'm not sure why it's called that. I'll go GUIs. Why is it not Rimmel? Rimmel, I don't know. I'm not sure why it's called that. I'll go GUI. Why is it not Rimmel? Rimmel, I don't know. No, no, it's cool. But Remy, a platform-independent Python GUI library for your applications. And yes, we've covered things like this a lot of like,

okay, I want to just scrap the idea of a native user interface and we'll just use web user interfaces. The thing that i liked about this it's so small if there's no dependencies you do i'm not sure why they're pointing you to to do a pip install from their github repo but i'm guessing it's they're not really that great about updating the pi pi version because it's there also but anyway i did this this morning, just did a pip install of Remy from their Git repo. And then I tried their little sample basic app, and I'm like, is it really this easy? And yeah, this is like 30 lines of code or less.

And if you run it, just Python bloop. Oh, first of all, when you pip install it, it installs the package, and then that's it. It doesn't install anything else because it's only dependent on the standard library, which is cool. And then you start it up, and it runs a little server on your thing, and it's really easy to throw buttons together and stuff. It looks pretty good enough, and the code is simple enough that I thought I'd include it because there's a lot of cases where, like in an internal team, where you have, if you want to throw something together where people have access to some Python data or some Python analysis with the user interface and something like this would be cool and it's neat. Python-only Electron.js-ish type of thing without the HTML part. So you define your app and you programmatically define, like, I want a button here and I want

a label there. And when I click the button, I want it to call this other Python function. So you just write pure Python and then it turns the GUI definition into HTML, shows that in a server, and then calls back to your code, something like that, which is pretty cool, right? Yeah, and it feels similar to working with TK back in the day. Within the Python code, you create a label object and a button object, and then you put them in a container. There's no packing so far that I can see, but it has different types of containers that you append them. And then just throw those in a box.

So yeah, like you said, you don't have to write any of the HTML. It's just all Python code. So yeah, for what it's worth, it's kind of simple. Yeah, it's a nice simple little thing, and it looks pretty cool. And yeah, I kind of like it. One of the things I was thinking about doing was, sorry, was we have a switch box that's controlled through, we've got a Python thing that controls the switches.

And sometimes it'd be nice to be able to just see what the current setup is and just have a webpage that just shows that. It'd be really easy to write in Python, kind of hard to do. I don't want to deal with a whole user interface sort of thing but having that i think i could probably throw that together with this tool in like half an hour so yeah and it's a proper local application so it can you know look around on the system and do what it needs to do didn't need to be like really a web server yeah although you can't expose i saw there's a way to to tell it to listen on the network broader interface, not just localhost.

So you can expose this thing to sort of the world for what that's worth. I don't know if it's a good idea or not, but you could. Like I said, we would do it within our internal network. So there's no malicious code flying around internally. Yeah, yeah. Cool. Anyway. You know, I'm starting to get excited again about Qt after Ogi Moore's presentation at the last PDX Python OS meetup.

That looked pretty compelling, actually. Yeah, what a slick program that was. And zippy. Very zippy. Nice. Speaking of slick, Datadog. Yeah, Python Bytes today is sponsored by Datadog, a cloud scale monitoring analytics platform that unifies metrics, logs, and distributed traces from your Python application. The Datadog tracing client auto instruments popular frameworks and libraries such as Django, Flask, Postgres, and AsyncIO even, so you can quickly get deep visibility into your application. Trace requests across services boundaries with flame graphs. I like flame graphs.

Correlate traces with logs and metrics and plot your application architecture with the service map. Sign up for a free trial at pythonbytes.fm.datadog and you'll receive a free t-shirt. Nice. Yeah, they're doing good stuff. Thanks, Datadog. This next one, Brian, you wanted it, but I got it. Yeah, you got it first.

It's cool. I'm excited about it. Yeah, so there's a couple of cool libraries for building CLIs. Flick is probably the most prevalent one. We've got argpars. But people who do FastAPI, the cool new API framework, these folks have now gone, let me just double check that it's actually the same organization.

Yep. Yes, good. Yeah, same org. Perfect. So they've gone and created a CLI definition library like Qlik. And I believe it's actually based on Qlik. Yeah, it's based on Qlik. So you might wonder, well, what does it do?

The name gives you a little bit of a hint. It's based on Python type hints. So Typer looks at the various hints. It can, you can say like, I have a function. It takes a name, which is, you know, name colon stir. It takes a string and then you just use it. When you go run it, it'll say, hey, this thing that we ran, in order to call it, you have to specify a name.

And the name has to be a string and so on. So it just looks at the arguments and the types of the methods you're trying to run, in the simple case at least, and it uses that. That's pretty cool, huh? It's super cool. Yeah. And then you can grow in more complexity. Like you can have subcommands. You can have help on all the commands.

You can actually get autocomplete, you can install autocomplete in your shell, all kinds of cool stuff. So you can create a little app, you say, here's a command, and here's a command. And then, of course, the commands are functions, those functions takes arguments, arguments have types, they can have default values, all that kind of stuff. So you can try to run it, it'll give you help pretty much automatically, which Yeah, this really speaks to me the way this works. I could definitely see myself using it. Yeah, so a really good, clean command line interface

that has, if you're writing a tool with a command line interface, you're going to want help associated with it. You're going to want description of the different methods and what they do, but also with as little boilerplate as possible. You don't want to clutter up your code. And as far as I'm concerned, right now, Typer wins.

There's hardly any extra overhead that you have to add to make this work other than type hits. And if you're not using a main or something, you have to put a little decorator on the different entry points in your program. But really clean. I'm totally going to start using this. It's really nice. And because it can leverage so much of the natural program itself, right?

The argument names, the fact that it has arguments, the argument types. That's really helpful for it. Yeah. So you can also integrate things like Colorama and Qlik, which is based on automatically start using color for its output. If you just have Colorama around, you can also use Qlik completion and Typer. This is the part I was talking about.

It will automatically configure to provide completion for all the shells. That's really cool, too. So the one thing I was concerned about is it's based on Qlik, and one of the things I love about Qlik is that it has a good testing interface. You can write tests against your command line app as if you're writing from the command line, but you don't actually have to spawn a new subpros to test things. And yes, Typer has the same sort of model where you can write your tests against the interface and not have to do a subprocess. Oh, that's really cool. Yeah, nice.

Because, yeah, you don't have to start it, basically shell out and see what happens, right? Or like issue the command line directly to the shell to see what happens. Yeah, really good job. Yeah, the other thing I like about this is you could say, well, we don't want the API of Qlik, so we're going to start from zero

and rebuild our version of what this whole world looks like. But what they did is said, we're going to build a new interface for writing consists, but on top of clicks. So things like Colorama and the completion just plug in seamlessly. I think that's a cool lesson. Yeah, definitely. Also, it's a smaller project for them to have to maintain too. Yeah, exactly.

Exactly. Cool. All right. This next one that you got here is from Chris Moffitt, right? Yes. And this is actually, so I'm hoping this is completely still accurate. I imagine it is. The interface for Matplotlib hasn't changed too awfully much. I know they've had some cool releases, but I want to talk about an article called Effectively Using Matplotlib. And this

is a 2017 article, but in the middle of the introduction he says at first I think I was a little premature in dismissing matplotlib to be honest I did not quite understand it and how to use it effectively in my workflow and I think that matches what my understanding of matplotlib as well I mean when I first needed to put charts or graphs or something of course you try matplotlib and then you're you kind of hit in the face with the complexity of the interface. And you know you can do the simple things simple, but how to do something a little bit more less than simple or more complex is a little daunting.

And then I went out and looked for other alternatives. But now I think the alternatives have their issues too. So I'm ready to go back and take a serious look at Matplotlib. This article is a really good look at kind of understanding it and at least Chris's workflow, how to use it. And one of the things I like is the description that one of the reasons why Matplotlib's interface might be a little confusing is because it's really got two interfaces. It does a mat lab like state-based interface and i'm not a mat lab user but i know that a lot of people have come

from mat lab to matplotlib and so that sort of interface is important to have that transition be easy for a lot of people but then there's also an object-based interface that makes a lot more sense to me and that's chris's recommendation is to use the object-based interface that makes a lot more sense to me and that's chris's recommendation is to use the object-based interface instead and so now after he's gone through it a few times and uh he has some recommendations and his recommendations are to um first learn the basic matplotlib terminology specifically what is figure and axes and around those. And what's difficult for me is that those are just normal words that I think I already understand them, but I need to make sure I understand what matplotlib thinks those things

are. Yeah, sometimes it's just hard to work with matplotlib unless you find an example, right? A lot of times what I, my goal is to hunt the internet for something that looks like i want figure out if i can find the python code that created it and then then i'm good yeah he does have an example in this which is kind of walks through at least one example but if you really want to kind of get your head around it to start with using pandas with matplotlib and so yeah you start your visualization with basic pandas plotting and then add if you want it more complex try seaborne and then if you want to customize it a little bit more he said use

matplotlib native stuff to customize the visualization and i wouldn't have thought to use that workflow so that that's kind of nice it also includes this really kind of cool handy reference that there's a little graphic that he created that has like all the different, what all the axes are. And if you want to customize a certain part of a graph, what that name is within Matplotlib so you know how to look it up. And that's often the secret of programming anymore

is to know what specific word to Google. So I think that's a good thing. Nice. And also I want to remind folks longtime listeners of pile straighter i'm not really sure about the pronunciation like illustrator but pi for the first two words i'll put a link in the show notes and what it is is it lets you take a basic matplotlib plot it launches a gui where you customize the look and feel and then it'll output the code that

would have done that for you oh cool yeah we talked about that episode 137 which is like a year ago or something but anyway still good stuff so i put a link in there for that i'm also including a link in a almost related section maybe there's an article about this but i haven't found it there's a stack overflow answer that has a little, the code you need to generate a Matplotlib graph and put it in a Flask file on the fly without having to save it to a file. Oh, yeah, like a live dashboard type thing. That's cool.

Nice. Speaking about doing stuff on the web, if you're using Django, there's a cool little library that'll let you do stuff in the background. Maybe you want to do some query and update some kind of report, or you want to send out a bunch of emails or recompute some indexes or recompute some search results or something like that.

And it's going to take a little while, and you don't want to just have it block on your main request. There's a bunch of different things you could do. You could set up a separate server with like Celery or RabbitMQ or something like that, but that's a whole nother set of servers and it just gets way more complicated. Sometimes you want to just run it in the background, just in the same process, right? It's easy to share data because you just pass the pointers. So there's this thing called Django Simple Task. And it allows you to run background tasks in Django 3 without requiring any services or workers. Oh, cool. Yeah, basically, the reason it has to be

Django 3 is it's based on async IO. And Django 3 is the first version that uses ASGI. And so you actually need to run it in ASGI server like UV, a corn or something like that so be really aware that you have to run it in its async way for this to work but basically when it starts when the web app starts there's a queue created and some workers to listen to the queue and then you just go to any function you want to run you just call defer on it and that'll kick it off in the background and you could do that now like you could create a thread and just launch it as a background thread. But one, you don't get a thread pooling. But the other is when it's time for the worker process to shut down,

like you're doing a new deploy or something, it'll actually wait until its background work gets done before it allows you to restart it, at least in a friendly way. So that's cool. And you can either give it a async IO coroutine or just a function and it delegates it, either runs it or delegates it to a thread pool if it's not async, things like that.

So it's pretty cool and it's really easy to use, like ridiculously easy. Oh, that's neat. Yeah. So there's a simple little example I put in the show notes where it just imports the deferred operation method. It has two tasks, one of them which uses time.sleep that requires threading, and one uses asyncIO because it's an async method,

so it would run on the async event loop. In the view, it could just say defer one, defer method one, defer method two, and then do your regular work. And that kicks it off but doesn't wait for the response, but does it in a slightly more durable, better way than just launching threads or kicking it off on its own little background thing to go run unmonitored. So in this example that nobody can see, but they will if they look at the show notes,

will the HTTP response happen and the view return before the deferred tasks get run? They might get started, but because both of them, in both of them, they sleep for a second to kind of, that's their work, right? They're sleeping for a second. So certainly the response would be sent back to the user before

those are done. And this is a cool interface because the two tasks, one of them is async and one is not. And the defer call doesn't, you can't tell. Right, you don't have to treat it differently, but if you can do async, great. It'll do that on the async event loop, I think. Otherwise, it just puts it into a thread pool,

which is pretty cool that you don't have to worry. You can just kind of go, here's some background stuff. If it's IO driven, we'll do it this way. Otherwise, just do it that way, which is, I like it. It's nice. Yeah, really nice. Cool. Cool, all right.

What's this last one you got for us? Well, so something happened recently i've got a handful of um i don't open source work on open source projects really as much as i'd like to i'd like to work do some more work on open source projects but i have a few that i maintain myself but project is in air quotes i'm doing because they're really pretty small projects. And a couple of them, cards and Submarque, are things that I just created to help with teaching people how testing works. They're not really intended to be useful projects. And then another one was a plug-in for PyTest.

Actually, I do use it, and it's useful for me, but I started getting like in the last handful of months, I started getting pull requests and issues and stuff. And I didn't know anybody was using it at all. So this was surprising to me, but I wanted to highlight, I don't know if I can't believe we haven't covered this or maybe we already have. There's a website called, um,

pi pi stats.org. Yeah. That's news to me. I hadn't heard of it. It's just a simple interface. Now, the backend might be complicated, but in the interface, it just has like a little box

and you plug in a Python package and it tells you what the download stats, actually quite a bit of history too of different packages. And so I use that for all of my different projects and sure enough, cards was downloaded like, I don't know, less than 400 times in the last month. Submark less than 70 times in the last month. And then PyTest Check, which is a little app or a plugin that lets you have multiple failures per test but keep going during the test.

That was downloaded just a little less than 20 000 times in the last month so clearly that one's more and that's the reason why i'm having more people trying to tell me what's wrong with it so i guess i have to start maintaining yeah this is a super cool interface you get all kinds of graphs and ability to slice it by dates and whatnot and yeah i like it and there's whatnot. And yeah, I like it. And there's also a top 20. So you can look at like, really, what are the top 20 packages on PyPI? And that's why I say that mine are small peanuts, because at the most, it's like 19,000 a month.

But the top 20 all get over 1.3 million downloads per day. Wow. Yeah. And you can look at them, look at the trends. That's pretty interesting. There's probably some cool stuff to go through here. And I guess if you were trying to do predictive stuff or analytics, there's probably a lot of things

to be done here. Yeah, definitely. Also, just occasionally, the top 20 I like because occasionally I'm curious about that. What are the top packages on PyPI? What are people using packages on PyPI what do people use it they weren't what I expected I mean some of the ones I obviously expected were

but I think right now urllib was the top top and I didn't expect that so urllib 3 specifically it must be used by a whole bunch of other stuff like 6 I don't think people are specifically downloading 6 it's yeah no like

i think um request uses both of those and certify and that like hits the top four so python date util is a good one up there boto core pretty interesting i use boto 3 which depends on boto 4 but it kind of makes me frustrated because boto 3 won't use the latest version of BotoCore, but it's all made by the same people. Anyway, there's a bunch of interesting stuff up here. Yeah, I think I'm going to have to look at this some more. Pip is in the top 20. Nice. Got to get started somewhere.

Yeah. All right. Well, that's it for all of our main items. What you got for the extras? Well, I wanted to actually just ask for some help. The last episode we put out was a live recording of the January Python PDX West meetup. Actually, not the entire meetup, but just a... Newsflash, in case people haven't been paying attention, they're near Portland.

There's a second Python meetup, which runs out in Hillsborough. Much better for Westside folks. So, yeah, you put that together. Good job. Yeah, and one of the things that we wanted to do with this one was to, I got this from a lot of people, and in the tech community, well, the Westside has a lot of high-tech stuff. There's a lot of people coming from other languages to Python. And so they might not be beginner programmers, but they're beginners at Python.

We've had this before, people saying some beginner-friendly content per at least one topic per meetup so that people aren't lost with everything. And yeah, we kind of forgot this this last time, but I enjoyed the topics. I'd like to hear from other people as to what sort of topics would be good at a meetup for Python beginner-friendly and just let us know. I think that'd be cool. Yeah. Well, it's easy to get super into like focusing on what's the most latest, coolest, hottest thing, but there's a lot of folks who really just appreciate like setting the ground, like a good foundation. And so, yeah, I think that's a good topic. Yeah. And I, one of the things I'd like to do is possibly get a list of those sorts of things so that we can try to ask people to put together something like one of the meetups

i did like just describing how how pip and virtual environment work and how to use those and there's a lot of basic stuff that people don't even think about maybe that they should talk about and be good to hear more i also want to try to get other people to do it because you and i could probably do those sorts of beginner friendly ones here it'd be fun to get other people to do it because you and I could probably do those sorts of beginner-friendly ones. It'd be fun to have other speakers too. Yeah, absolutely. It would be very fun. If people are willing to get to Portland on a Tuesday evening, then they can come give a presentation.

Yeah. One more thing. I want to do a shout-out to PyCascades2020. I can't remember when. It's in February coming up. February, I believe. I believe it's like 8th and 9th. Let me double-check. It is February 8th and 9th. Let me double check.

It is February 8th and 9th in Portland, Eastside. Wonderful. So people should get tickets so they could come. And if you haven't bought your ticket yet, check the show notes because we've got a discount code for 10% off. Yeah, absolutely. Yeah, it's right there at the bottom. I don't really have anything this week for extras to share,

but one thing that did come out and catch caught my attention yesterday that i think i might as well just throw out there's kind of cool is firefox 72 is out and it has come with anti-fingerprinting which is pretty exciting so i don't know how many people are aware i don't know if you're aware brian that even with ad blockers and blocking like tracking cookies and stuff there's all these weird things that you can do to figure out, actually, it's that same person, right? Like you can ask, what are all the installed fonts? And what version of OS do you have?

And here's a canvas thing. Let's run some stuff on there and like try to detect weird CPU cycles and other kinds. There's all these like weird things you can do in browsers and pretty much uniquely identify everyone running a web browser. Oh, that's lame. Even without cookies.

Yeah. So Firefox is trying to fight back against that. So I think that's pretty awesome and worth checking out. I'll actually link to Ars Technica article about it. Okay, cool. Yep. Well, I guess that leaves us with a joke, doesn't it? Yeah.

The last time we did a joke about describing different software programming jobs. This one, you're going to have to imagine that people who are fans of various programming languages are in like a literature course and they have to write an essay. Okay? Okay. Okay. Have you pulled up the picture? Yes.

I'm already laughing. All right, cool. So basically there's professor looking person with glasses trying to read like a paper, a physical paper and a student looking at it, getting the feedback from the professor. This is important because at the end it gets interesting with the pictures. So for Python, it says, this is plagiarism. You can't just import essay. For Java, it says, I'm two pages in and i have no idea what you're saying

assembly appropriate yeah assembly do you really have to redefine every word in the english language with c it says this is great but you forgot to add a null terminator now i'm just reading garbage i don't really know if i understand the c++ one but it says i asked for one copy not 400 and the professor's buried in books oh i don't get that either but yeah okay unix shell this one's pretty good i don't have permission to read this latech which you often use to describe like scientific papers and like super elaborate mathematical formulas says your paper makes no sense to me but it's the most beautiful thing I've ever laid eyes on. And the final one is the HTML programmer.

And instead of holding a, a paper, the professor's holding a pot and they just, they just say, this is a flower pot. I don't get it, but it's funny. Yeah.

I don't either. All right. So anyway, I thought that was. Yeah, I don't either. All right. So anyway, I thought that was pretty fun. Y'all can check it out. The comic is linked in there, but it's pretty fun. You can't just import essay. This is plagiarism. All right.

Well, thanks a lot, Michael. Yeah, you bet. Thanks, Brian. Good to be here. Thank you for listening to Python Bytes. Follow the show on Twitter at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm.

If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. This is Brian Ocken, and on behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.