Python Bytes - #181 It's time to interrogate your Python code

Episode Date: May 14, 2020

Topics covered in this episode: interrogate: checks your code base for missing docstrings Streamlit: Turn Python Scripts into Beautiful ML Tools Why You Should Document Your Tests HoloViz project A cool new progress bar for python Awesome Panel Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/181

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 181, recorded May 6, 2020. I'm Michael Kennedy. And I am Brian Ocken. And this episode is brought to you by Datadog. Tell you more about them later. Brian, this first item that you picked here, I saw this going around the Twitterverse, and I was like, oh, that looks really cool. Tell us about it. Yes, Interrogate. So it's a new little tool. I think it's kind of new, new to me at least. It checks your code base for missing doc strings or the presence of doc strings. So it's written

and maintained by Linroot. And we were notified by Herbert Beamster, suggested it to us. I like doc strings. I don't like them. I don't have a policy of having them everywhere, but doc strings can help you understand the code quite a bit. I really love how a function docstring is utilized by VS Code and PyCharm and other editors to do things like if you hover over a function, it just pops up a little window with what parameters you can send to it, but it also sends the docstring, shows the docstring if there is one. There's also other tools like Sphinx or PyDoc or Docutils that can use that amongst other things to generate documentation. I don't

really use those. I know a lot of people do though. So it's a problem if you've got some missing, if you really need to have doc string someplace and they're missing. So interrogate is a command line tool that you can use to check your code to make sure everything has a doc string that needs to. I think this is great. Yeah, it's super cool. And well done, Len. Nice work on this one. I think this is really nice because if you have a public library, that should be documented, right?

That should be covered with nice doc strings that give you all sorts of help in your editor, like you mentioned, but also if you generate the documentation, you don't want to have just some random empty, either completely missing or just the function name and just an empty body or whatever gets generated there. So this is really cool, and I could see if you were doing a package that was public, having this part of your CI CD process that if somebody introduces something into say like a PR that here's a new function we wrote like great it failed the CI because it doesn't

have a doc string that'd be awesome yeah and then there's also some common there's some common like caveats like yes all public functions should have doc strings but what about like uh private functions and stuff there's options to be able to turn off things where you might not need a doc string for just an internal helper function or something. Yeah, I totally agree. If it's not really meant to be included in the public API, then it shouldn't necessarily have a doc string. Yeah, it can if it helps to understand it, but you maybe don't need it. So this is nice. Yeah yeah this is super cool well then lynn yeah i wanted to mention also there is

information on the readme about how to use the tool within talks and then within ci workflows so that's helpful yeah this one seems like a super easy thing to adopt if you already have a ci process in place and you're working on something that's public, whether that's public to your people within your company or it's a package that is used by other projects, right? Or it's public as in on PyPI.org. I still think it makes sense to just document it for whoever's consuming it, who's not writing it.

Yep. Very cool. Now, this next one I've been holding off on talking about for a long time because it's really interesting but i didn't feel like i was able to do it justice so the thing i want to talk about is streamlit with streamlit the idea is it allows you to build much simpler data science applications and web apps, I guess you would say. So many people come to Python from, I've got a little bit of computation that I got to do. And so I'm going to write a script that does like 10 things.

Maybe it doesn't even have a function. It just does do X, you know, load data, transform data, pivot data, show graph or something like that right really really simple they're not like pro web developers that are like busting out view or react or something like that to build cool apps but they've got this really cool outcome with python and they're happy but at the same time they would like to have this publicly shareable they maybe want to put it on the internet they want to make a web app out of it And the gap from I've got 10 lines of code that outputs a graph to I have a fully interactive, dynamic, server-backed, JavaScript front-end type of visual website, that's a big gap, right? Yeah.

So Streamlit. Even for me. Yeah, well, yeah, for me as well. Now for Streamlit, the idea is that you write the code in the simple straightforward style that I talked about, but then it turns it into an interactive web application. That's neat. That sounds neat, right? And it does a whole bunch of really interesting tricks

to let you continue to write code in this simple straightforward way and yet interact with like callbacks and stuff. So you might have a function that comes along and says, okay, what I need to do is I want to show, I don't know, a dropdown list in a web app or a slider bar or something like that. And then you write your code that says, get the value from the slider bar. And you might think, okay, well, normally that's wait for the user to click on the slider bar. And when there's a change or some kind of callback and then you have some function that runs on the callback and then

it updates the dom right but this is like straightforward procedural code or imperative code that doesn't have any of those ideas in it so what it does is it uses interesting caching on your function so that if it were to rerun it with the same inputs, it gives you the same answer instantly, like LRU cache type of stuff. And then if any of those controls on the webpage change, it just reruns the whole script. And when you say, get me the value from the slider, it says, well, currently the slider is set at this. And if it happens to change, it just reruns the entire thing from beginning to end. But because everything's cached, it only recomputes the parts that are needed. So you don't have to think about callbacks. You don't have to think

about user interaction. You just write top to bottom code. And here you've got multiple parts of your application that can be changed dynamically that just continue to refresh. And it refreshes it by using JavaScript. So it doesn't even reload the page. It just looks like parts of the page are dynamic and coming to life. This is neat. It's super cool. Basically, if you can write straightforward, top-to-bottom code that works with

graphing or pandas or something like that, you now can create a single-page SPA app. I don't remember what JavaScript framework it uses. It's like React or Vue or something like that. I think it might be React. Anyway, it's not super relevant, but it does all of that work with like hosting it and creating the server callbacks and setting up the JavaScript and

the UI widgets and all that kind of stuff. And all you do is write basically procedural code. One thing that I think is a danger of these types of applications is it's easy to say if you use our magic graphing library in our magic ui widgets in our magic you know data processing library not a real database but our our magic server database that we create then it works right what i like about this is it doesn't do any of those things. It just says you want to use pandas and numpy and plotly or something like that. Use those and then use them in your script. And then we just make it this single page app, which is pretty awesome. Yeah, that was my question. Is this a hosted thing or can I do it like behind a firewall or local thing or do you know? Yes. So with Streamlit, you have a couple of options. I haven't done anything myself,

but you can self-host it or they'll host it for you, I believe. And they have this thing called four teams, which is like they automatically deploy and manage the apps and you just create them and say, here's the code and run it. It's like kind of a platform as a service

for Streamlet apps type of thing. That's a paid part of what they're doing but the stuff i described up until then that is uh free and open source so it's still pretty pretty usable yeah so you have to basically deploy them yourself but they talk about how to do it you know it's not super hard okay nice yeah so i had the one of the guys adrian on who was behind this and has also done some other interesting technical programming stuff with self-driving cars and folding at home, those types of things, on TalkPython just a couple episodes ago.

I don't remember exactly what number it was, but I guess 260-ish, something like that. So just check out that episode if people want to go much deeper. But this is really cool. If you're writing these imperative, simple little scripting apps and you're like i would really like to share this or make it better uh streamlet's cool yeah nice okay i feel like we both have themes for ours your your thing is continuing with documenting and and whatnot and you'll see my theme develop as well this is interesting because um i ran across this um interrog second, but after I ran across this,

an article by Hinnick that says why you should document your tests. He makes a lot of good points. So I believe, and we have this policy where I work, of tests should have doc strings on them to tell you kind of what the test is doing and why it's there. There's a lot of reasons that I have never really been able to describe clearly because you're i mean ideally

your test name should be descriptive and and the code should be clear so why do you need a doc string it's not so that's not something that you're going to call from an api or anything you probably won't type help function you you know, test function, right? Maybe, but probably not. And you also don't call it. I mean, like test methods that you run PyTest with, PyTest calls it. You don't call those tests. But you can get confused in the future. And one of the explanations is,

Hinnick has a good example where he shows some simple code with a simple test name, but the thing he's asserting on doesn't really make sense. And the reason, if you don't know what's going on under the hood, and the reason is because tests often are testing for a side effect of an action. If you're testing the output of a function, that's obvious. It's kind of obvious of the, if I pass in functional tests, if you pass in certain parameters and you get a certain expected output, you can assert against that. But what about actions that have side effects and you need to test for those? That might need some more explanation.

And he writes, this is quite common in testing. Very often you can't ask questions directly. Instead, you verify certain properties that prove that your code is achieving its goals by checking side effects and stuff like that. It's a short little article and I was just happy to have some of that expressed of why you should document. And it might be you that's confused in the future. And then it ties back into the interrogate because, so I was just thinking when I was reading about interrogate, I like oh awesome I really want to put something in place to just make sure

all my tests have doc strings and he lists in the article the exact command line that you need to make sure that just text to make sure that your tests have doc strings and nothing else so this is cool yeah it looks really nice and I

can see how this would lead you down to finding a library that searches for or evaluates these. Yeah, so the test example he has here is testing that the hash, like creating the same object, an instance of a class twice, and taking the hash and making sure that those are not the same because he's trying to test that adders is not implementing false on the class or something to that effect right so you're like what the heck is going on with this like why would you check the hash of a thing is not equal to the hatch of a thing

but it's looking like you said for this sort of like hidden layer that's operating below it right yeah and then also my reasoning and i think he he brings it up, is that often when you're looking at a test, it's because you've really got to get a release out and one of the tests is failing. So you've got to go look at the test and figure out what it's testing for and why it's failing. And if you don't have it clear why the test is important, you run the risk of the person at the time going well i don't even understand what this test does maybe it's not important comment it out comment it out passes you don't want that ci is good we're going home i don't do that but i've heard it done before of course i would never do anything like that but i could imagine that it has been done now i actually have i have some experiences in the past where like there was some project and it had like a beautiful set of tests and then and ci and stuff and i wasn't

working on anymore and whoever took it over it's like oh those tests were really annoying so i turned them off you're like oh my gosh those were there for a reason what just happened so at least like if this gives you some description about what's up, that would be good. Is there anything where PyTest will show the docstring as part of the error report? I think there was somebody that wrote a tool that would do... I don't remember the name of it. I remember running across a plugin that would pop out the docstring for failed tests, which would be pretty cool. Yeah, that would be pretty cool.

Hopefully, the person evaluating the test failures has access to the code. They can go look. Yeah. I'm thinking more of like a CI CD. Like you're not really, you don't have it like loaded up right there. Right. You're just like,

Oh, I'm cruising on GitHub, checking a PR. What the heck is this? Yeah. I actually wrote a, it was an internal thing, but a quick,

uh, plug in for PyTest that would just before every test function, print out the test name and print out the, um, uh, the doc string. If we can look at the logs, it's there. Yeah, that's cool. Very nice. Also cool, Datadog.

So this episode is brought to you by Datadog. And let me ask you a question. Do you have an app in production that's slower than you like? Is performance all over the place? Maybe sometimes fast, sometimes slow. Here's the important question. Do you know why it does that? With Dat sometimes fast, sometimes slow. Here's the important question. Do you know why it does that? With Datadog, you will. You can troubleshoot your app's performance

with their end-to-end tracing. Check out their detailed flame graphs to find bottlenecks and latency in that finicky app of yours. Be the hero that got your app back on track at your company. Get started with a free trial at pythonbytes.fm slash datadog and get a cool little t-shirt as well. Alright, so your theme so far has been pretty clear. Next one of mine, I want to talk about a project called HoloVis. Have you heard of HoloVis? I have not. I'm sure you've heard of

SciPy and those types of things. There's kind of a grouping of other projects, right, under that general banner with like NumPy, Matplotlib and whatnot well holoviz is kind of like that for data processing and visualization and one of its features acts very much like streamlet but in a more general probably a little more work to set up and like right you got to work more in its framework but in a very similar way to what streamlet is doing as well so it's a coordinated holovis is a coordinated effort to make browser-based data

visualization and python easier to use easier to learn and more powerful so it's yeah so what does it do so it has a bunch of tools that make it easier to apply python plotting libraries to your data a bunch of tutorials conda like, like MetaPackage, that when you, you know, Conda install, HoloVis, it actually installs a whole bunch of things. I'll tell you about it in a second. It even has some sample data sets so you can, like, go through the tutorials

and actually get the outcome, right? So it's made up of a bunch of different things. One called Panel, and that one makes creating apps and dashboards for your plots using any of the supported plotting libraries which is pretty awesome it has hv plot to quickly generate interactive plots as hollow views to make your data instantly visualizable geo views for visualizing geographic data data shader for rendering huge data sets param for creating uh user configurable

objects like config stuff and then color set or like color maps so yeah you want to have like nice colors that flow together for your plots and panel is the main thing that i was thinking of that lets you create these single page apps around interactive sliders and drop-downs, around your graphs and other kinds of stuff. And yeah, very cool project as well. Yeah, I was looking at the data shader, and they have some really pretty pictures because of the examples of overlaying data plots over the map of the U.S., things like that.

It's cool. Yeah, or you can plot attractors with 10 million points each. Oh, is that all? And these graphs, I mean, the resulting pictures are, I don't know what, they're like 400 by 400 pixels or smaller. It's like, yeah, that came from 10 million points. Or here's the United States plotting each person in the United States where they're physically located from the 2010

census. So 300 million points of data overlaid onto the United States map. Yeah, it's like a sun. Really is pretty interesting, actually. I think I can see you over there, Brian. You're on the left. You are too. That's right. Awesome. Anyway, yeah, so this is a cool project. There's a whole bunch of different little libraries that are pretty neat. So if you want to try to visualize your data, and we'll come back and talk some more about both these projects towards the end of the show as well. But a bunch of cool libraries all brought together

under this HoloVis project to make them work together. Yeah, nice. Nice. What did you got next? I have another command line tool. So this was by oh gosh rosario and he wrote a little blog post about it but it's a there's a project called live progress and it's a progress bar for python and it's really cool so it is pretty cool command line interface

progress bar so often i mean like even like pytTest has added progress bars for, you know, watching your test finish and things like that. And it's nice to have progress bars. It gives you good feedback and whatnot. But there's limits to what you can do, except for this seems unlimited. It's got a whole bunch of different animations and spinners and things you can combine to make a more entertaining progress bar. So it's fun. Yeah, these are really nice. And these little touches,

they probably don't seem like much, but they can definitely make your app feel more professional and more polished rather than just the answer is seven or whatever, we're done. Yeah, some of the, what was it, pipenv. I don't use it, but I appreciated some of the fun command line interface stuff that they've added to it. And I think adding some fun to a tool is nice.

One of the things I wanted to comment on also was just about the repo itself. So the code is up on GitHub. And the readme has some nice thing features I wanted to call out. So the animated pictures of what it does is a nice touch. We've said this before. We love things like this. I liked that there was a to-do list. So it encourages having a short to-do list,

encourages contributions. And I think even listing things that I've gotten done recently so that people that might not know about those features don't try to go work on them. And then just a short list of things you'd like to have done. And then it has an interesting facts section, which is kind of cool. The code is in a functional style. It uses closures and generators. So, I mean, actually a lot of times I'll look at code not as an example of how to do something. So if somebody's proud of like their user generators, maybe it's worth code checking out if you're trying to learn generators.

And then another feature that I'm definitely going to pick up for things that I work on is change log highlights. So not the entire change log in the readme, but just one or two lines of semicolon separated features per version of change. So this is kind of a nice thing to add. Oh, yeah, that is really nice. Cool. I think it's a great project. If I need progress bar i'm definitely gonna consider this one like i said like i do think they're neat and i do use them sometimes so uh yeah that's cool very nice all right last one's a quick one so we talked about holovis talked about streamlet and panel and how

those are kind of similar so there's this other project called awesome panel which are kind of similar. So there's this other project called Awesome Panel, which is kind of like an awesome list for panel, that way to build interactive data science single-page apps, or not even single-page apps. This is run by Mark Skov-Madsen. I got that close to right, Mark? And yeah, it's a cool project to just show a bunch of examples and a curated list of these. So yeah, if you a cool project to just show a bunch of examples and a curated list of

these. So yeah, if you just open that up, you can go see there's an app that comes up. It takes just a second to come to life. And then on the left, there's like a gallery. There's all sorts of cool pictures. There's also a nice talk. I can't remember who gave the talk. But yeah, there's like a talk from one of the conferences.

But if you go to the gallery gallery you'll see like as you navigate around it's kind of like this single page app so you click gallery you'll see like the main part of the app sort of spin for a second and then come up and then you can go and say oh there's like one of the things that's really cool that i'll point you guys at is this thing called the image classifier in there so i went in there and it's lets you if you just scroll down it says upload an image and you can grab some JPEG and upload it. I grabbed some like random microphone picture that I had laying around and threw it in there.

And then it ran, you can pick different neural networks to run against it. Do you want to run NAS, net mobile, NAS, net large, mobile net V2? Like, I don't know what these are, these glass fires, but you can pick from them. And then it'll tell you in a cool little graph at the bottom, what it thinks that is and it thought the microphone was most likely a microphone but it could have been a shield because it had like a big metal guard on the front of it it's just really cool and so this whole website is built i got it right in panel to show you how you can build a cool panel app but then as you dive into the gallery each one of these pieces is like an interactive sub panel or something like that so it's kind of it's a little bit meta in that regard

but it's pretty cool actually yeah it's really it's really cool yeah and uh yeah it's apparently super easy to work with again i haven't built anything with it but it seems like a much simpler way to get your interactive data science stuff on the internet than learning Vue.js, JavaScript, Blask, SQL Alchemy, etc., etc. Yeah. So anyway, pretty cool. I recommend if you want to check this out, just go browse the gallery. That's the best way to see what this thing can do. I love pretty pictures.

I know. Yeah, this one definitely passes our test for you must have pretty pictures in order to talk about graphical stuff. Yeah, so Brian, really quick on this next one, click on the app for it because it takes just a second to load up. So that's it for our main items. I want to tell you about a couple of things, then we'll get to yours as well. First, one of the features that I really like about visual studio code i haven't really used because i don't use visual studio code in like a meaningful way i use it all the time for

like little bits of editing but not if i'm like doing real projects i probably use pride charm but it has it has live share which is a pretty killer feature so you can say i would like to share my coding environment with someone else who's going to look over my shoulder, do paired programming, just code review, whatever. And they can actually debug and step through and see your code in their editor. And it could even be a different editor. I think there's other stuff support like proper Visual Studio versus VS Code and whatnot. So that's really cool. And I've always thought, well, that would be awesome for PyCharm, but no. So someone on Twitter, sorry, I don't remember who sent this to me, but thank you,

sent up this link to this project called Code Together. So Code Together is a freemium product. So this is not an open source thing. It's like a paid project you can go sign up pay dollars a month for certain features or you can just use the free version but it has that type of experience that i just talked about but for many different editors right so it comes for all the intellij stuff so like webstorm pycharm and so on also works with vs code it works with other things i don't know exactly all the things that it covers but but certainly Eclipse, IntelliJ, and VS Code, which covers quite a bit. So if you were looking for that and you weren't using Live Share, because it didn't exist

for what you were doing, you could check this out. I have not used it. I'm not endorsing it. I'm just saying it looks interesting and it might help people. And then the other one that I want to just quickly mention is related to the first thing. These are totally different things.

But Kevin Van Der Veen sent over a message a week ago and said, Hey, I built a cool data explorer to help you understand the whole COVID pandemic stuff in a local way. And he built it using Streamlit. So I thought, hey, here's an app that someone just sent over that's like a cool running on Heroku Streamlet app. That would be a nice example of what I talked about at the beginning. And it also has the GitHub repo there as well. So if you go and check out that app, like for example, Brian, you scroll down to the second to the third graph below the third graph, there's like a drop down that lets you pick Oregon. And you can go pick that and wow, it looks like we're flattening the curve really well that's pretty awesome although

the last two days have been rough i guess or you can go down a little farther and compare it against different states you could go in there and type like it's colorado you could type oregon or new york or whatever you want and then it'll like auto complete that out of the list and then regenerate and this is exactly like what i was saying. There's some line in the Streamlit code that's just saying, get the tags from the tag selector. And then if you type in there, it just reruns the whole thing,

but most of the stuff's cached and then it'll redraw the graph. It's super cool. So this is both useful, I think, for if you want to try to understand that. It's a cool data science project. The source is on GitHub. But also it's just a cool example of Streamlit

if people want to see that going. And the graphs are fun, too, because you can do box select and zooming and stuff like that. Yeah, these are just Plotly. So they do all the standard Plotly stuff, which is pretty cool. That's kind of what I like about the Streamlit thing, not forcing you to use their random graphing thing, but just other graphs that you might like.

Yeah. All right, well well that's it for my extras what else you got well i wanted to remind people that pycon 2020 online is available and new content is still being posted through the first few weeks of may i think that's when they're wrapping things up yeah that's cool i just saw some new videos go up there yeah yeah one of the new videos that just came up was my talk so multiply your testing effectiveness with parameterized testing oh yeah there it is right at the bottom with a couple of new tags on it yep so i'm excited to have people get feedback from people to see what they thought so it looks

good i like it yeah well done while i was there i was looking around uh there's tons of great talks there's a bunch of tutorials there there's charla's is that new this year they did it last year maybe okay the last two years but it's the spanish language track yeah it's neat i'm glad they're doing it sponsor workshops that's kind of nice to have and then even an online poster hall so that's there so it's cool the other thing i wanted to bring up a quick extra if anybody's following the drama and pi test the drama is over and uh i'm happy with the resolution so hopefully there will be peace in the family i hope so a link to the twitter announcement from pi test yeah they got the whole uh public statement there so the folks who had

dropped out are they dropping back in or what's or is there just permanent out follow-up from that i don't know that any more details i know at least one person um is back in but the all the people i was worried about are signers of this uh message so yeah so probably kind of seems like maybe they're back in yeah all right well that's awesome i'm really glad to hear that got solved and it sounds like exactly the outcome that i would have voted for as well okay so you ready for some jokes i don't know these jokes how about you are you ready for humor yes humor is good humor is definitely good and uh so i found a couple of funny pictures and i'm going to put the pictures in the show notes i don't know if they'll come up in your podcast players some of them do some of them

don't some of you guys say allow pictures in this podcast feed, just like in the player as you're looking at the show notes, but certainly on the website, they'll be there. And these are, oh really book covers. I love these. They're like O'Reilly and just if you've forgotten the O'Reilly books always have the title and then they have an animal that goes with it right so if you wrote a book for O'Reilly the thing is like what's your animal on your book and whatnot yeah so these are like take those ideas put an animal on it but make it silly right there's also usually like some sort of saying at the top of the book so for these you definitely have to read the top also yeah it's right like sort of the subtit the top of the book. So for these, you definitely have to read the top also. Yeah, that's right.

Like sort of the subtitle. Okay, so how about this? I'll read the first one, you do the second one and so on. So this one has like a badger or something on it, on the front of it. And it's kind of like sneaky head down. And the title is Pointless Meetings, The Survival Guide. How to survive all pointless meetings.

I feel like many many many things that used to be meetings these days are now an email and people are like oh my gosh they really can just be emails like why have we been going to all these meetings right yeah all right you want to get the next one next one is overriding your teammates code my code is better than yours anyway it's got a horse on the front I don't know what the horse is I really love this one this one is the essential semicolon

parenthesis sorry quote semicolon parenthesis drop table animals semicolon dash dash and there's no animal because well this is a SQL injection that has deleted it and it says now with user generated content that one's that may be the best yes now with the security hole exactly

related what's next okay this one has a fish on the cover expert hoping nobody hacks you security by optimism and prayer i don't know what the fish has to do with it. I must be missing it. No, but that's good. That's most people's security solution, right? That's true. So the next one is an octopus. Obviously, many legs that can type many things. And the title of the entire book is Exiting Vim, Eventually.

Just memorize the 14 contextual dependent instructions exiting eventually oh i love it i love it yeah so you'll have to visit the website and uh check out these because they're pretty sweet yeah awesome speaking of sweet it's been great to be here with you brian thanks it has been great thanks yep bye bye