Python Bytes - #144 Are you mocking me? It won't work!

Episode Date: August 23, 2019

Topics covered in this episode: Why your mock doesn’t work The nonlocal statement in Python twitter.com/brettsky/status/1163860672762933249 pre-commit now has a quick start guide Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/144

Hello and welcome to Python Bytes, where we deliver news and headlines directly to your earbuds. This is episode 144, recorded August 20th, 2019. I'm Brian Ogden, and Michael's on vacation, but I have two awesome guest hosts. Welcome, Chris May. Hey there. And also welcome, Mahmoud Hashemi. Hello, everybody. It's good to be back. So both of you have been on Testing Code, but have either of you been guests on Python Bytes before?

I think I actually guest hosted once in a similar situation where Michael was on vacation or you were on vacation. I think it was me and Michael that time, so it's good to get the reverse. Cool. And I definitely never have been on Python Bytes, but it's been something I've been excited about ever since I thought I had the opportunity. Dreams do come true. Indeed. I really appreciate both of you coming on. As a reminder to everybody, this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. Get a $50

credit for new users. Lots of good stuff. More about that later. Let's just jump in. I'm going to put Chris on the spot and have him go first since he's new here and we can heckle. Yeah, bring it on. No. Yeah, my first topic I selected was why your mock doesn't work by Ned Batchelor. And it's an article that just came out like perfect timing for my team and I, because there are members of my team who are building an application that's syncing or bringing data from one database server to another across cloud platforms. And essentially, they're trying to do a test-driven development. And they came to me and they're like, how are we supposed to do this? There are all these third-party clients that we're connecting to and trying to get authorization from and all these different things.

And they quickly found out that mocking these third-party tools was hard. I was just so thankful that Ned wrote this article to show us that one problem is that sometimes mocking, the way that they're setting up their mocks, the item that they were mocking never actually mocked the object that they were trying to mock. It's a little hard to explain in video, or in audio, rather. But Ned actually has some great illustrations to show exactly how mocking works and when it works and how it can be messed up. I saw those illustrations. I don't know if he's using GraphViz or what,

but we're definitely getting the point across. Yeah. I was wondering, because I thought he did such a great job with those illustrations like how do you take the time to to design those so it's good to know that there may be a tool behind that well i mean and he's got yeah i mean he used them in his python name and values talk and blog post and uh i always like especially for beginner to intermediate programmers i always send them to that python names and values talk it's a really good use of a half hour. Nice. Hey, look, we're getting bonus content already. Yeah. And that, that actually, um, that names and values talk actually get, makes you a better mock user as

well because it helps you understand how that all works. Plus Ned's just really good at explaining stuff. Absolutely. The other thing I love about this article is he also links to two other resources, another article and a video where people gave examples of when you don't want to use mocking and also good practices when you do. And one of the takeaways I took away from my team is that especially if you're trying to mock third party code, you're going to need to create tests that, of course, test your own application against the mock, but also test the mock against the third party tool. And so this is a bunch of stuff that you're going to invest in. If you want to test with mocks, I mean, if I can, if I can jump in,

what I'll say, whenever you're integrating, right, and there's an SDK or library for integrating, chances are you're not going to use 100% of that library or that service. You're probably going to use a little sliver of it. I know it seems like, oh, great, they wrote all the code for me. Let me jump right in and integrate directly with their library. But it really makes sense a lot of the time to use a gateway or some other pattern to kind of like make explicit which parts of that library you use. Because eventually when you have to integrate with another service or you want to throw it away,

it's just going to be a lot easier to sort of like maintain that side of the integration. So yeah, great post. Yeah. I also like to, I think it was in this post

where you mentioned kind of what you were saying. Usually there's only a thin sliver of the functionality you need. And to use, to create a mock that only does that,

but also to make sure that across your organization or across your team to only have one mock object that handles like that one third-party tool so good well mahmoud what do you got for us so what i've got for the group here is a little library uh slash application that i ran into while doing some research for a talk that I'll get into later. But basically, it's called vermin. And the name's not really explained. But the only reason I mean, the only reason I can see behind it is that Python has snakes and snakes are kind of vermin and vermin kind of sounds like version. And so what vermin is, that's like, you know, v e r m i n. Oh, I just got it. It's like version minimum.-e-r-m-i-n oh wait i just got it it's like version minimum okay so what it

does is it's a rules-based python version compatibility detector with a pretty clever name so what it does maybe you heard the news but basically now all 360 top downloaded pi pi libraries now support python 3 there's like a website for it. And the way that that's determined is, I think Brett Cannon has a library or application which basically looks at the setup py and then looks at the index classifiers. And so if the person who published the package says it supports Python 3,

it takes it as its word and says that this library says it supports Python 3. Now, the top 360 libraries have done so, which is a pretty good milestone for us, especially as we approach 2020. But the thing is that it's based on what people say. So if you, when it comes to your own code, right, like if you're like a team at a company or a maintainer of an application that doesn't necessarily have a setup.py, there's, you know, can I use Python 3? It's not really going to help you because you don't have those classifiers. So basically, vermin, what it does is it scans over every single py file, every single module, and it tells you the minimum Python version necessary

to import that module. And it even gets down to the function level. So basically, you can use this to scan your code and say, like, look, there's a Python 2.7-like feature being used by this function over here in this module. And then you can also certify that, hey, this module over here uses, like, Python 3.5 syntax. Or, hey, that one actually is using a 3.7 thing, but we're only on 3.6. So it's rules-based. It's got over 1,000 rules. It's pip installable.

You pip install vermin, and it has a little command line entry point. And yeah, when I found it, it had like less than 80 stars on GitHub. And I'm hoping, you know, I saw how much work went into this. I'm hoping that maybe it blows up a little bit because it's a very useful tool.

And I'll explain why in my next segment. But basically, yeah, definitely check it out vermin that's really cool i mean i could definitely see yeah it sounds really cool it sounds like a lot of work went into that exactly a lot of work went into it and it's still a little bit quirky i'll tell you like i ran it with uh just plain vermin as described and i got one result and i ran it with dash v which made it kind of like it's supposed to make it more verbose I thought but it actually made it like look harder and it came back with a more accurate result so maybe when you run it run it with dash v or maybe like you know help the guy out I think his name is Morton Christensen maybe help him out and give him a PR make the verbose

act the same way as the non-verbose but uh no great tool regardless ah okay verbose might be more picky interesting well i think vermin looks cool and i'm excited about going from uh not just from two to three but for instance i we now are excited about python 3 8 features or 3 7 features and i want to make sure that you know it's possible to move things around so exactly there's a lot of a lot of use for that and forever, I think. So anyway, I don't want to talk about a new feature. I actually want to talk about something that's been around for a while, but I don't see it a lot in code.

And I just saw this recently, an article called the non-local statement in Python. And I'm going to attempt this name, Abolash Raj. I think that's it. But the idea is if you use a local variable, like if you assign to a local variable, it creates a new thing in the namespace, a local name. And the normal thing that everybody knows is unless you add the global. But global doesn't just make it so that you can assign a variable that's not declared in this function. It also makes it so that it's a global scope variable.

And sometimes that's not really what you want. You just want it to reference, like for instance, if you've got a local function defined within another function, and you want to reference or assign to an outside variable. It's really easy to see when you look at the code. But I forgot this was around. I don't use it very much, but it's pretty cool. I think I run into this occasionally, and I usually just make things global. So, Mahmoud, you probably have more experience than me.

Do you run into this ever? Oh, my goodness. So back when I was, like, really learning the ropes in Python, and I remember it so clearly back in 2011, I was playing with the concept of dynamic scope, which is in contrast to lexical scope. So that's like lexical scope, you sort of

look within your block or your function, right? And dynamic scope is this older concept where it's like you would look up the stack for something. And I think Emacs Lisp still uses this. Kind of glad that Python doesn't. But, you know, in like the mad science, like, you know, mode of learning all of Python's cool features, I wanted to, you know, do something like that. Yeah, somewhere in there, I was like messing with generators and dynamic scope, and I found myself

just hitting right into a wall. And I talked to Raymond Hedger, and he's like, oh's like oh yeah what you want is non-local and that's going to be sometime in the far future yeah anyways I mean uh here we are and it's there and frankly if I use it I'm over that phase so if I use it I feel like I'm doing something wrong and I should probably like factor this in a uh clearer way but that said I'm sure it still comes up. And, you know, I think everyone's probably going to have a little mad science phase of their own. And so let them use non-local and deal with the consequences. Well, right. I mean, it's like one of these features that it's used so little that you're just adding

complexity to the code because most people won't know what it means. So you got to be careful. You know, it's still like it has its own little place. And I'm sure some like someone has a legit use for it beyond like just messing around. So I'm glad it's there still. Yeah, definitely. I'm going to take a little bit of a sponsor break and thank DigitalOcean for sponsoring this show. DigitalOcean has been a longtime supporter of Python Bytes and we really appreciate it.

And not only that, all of the infrastructure for PythonBytes.fm runs on DigitalOcean. We've got our website, our database, and even the audio file hosting is done through DigitalOcean. It's been wonderful, super affordable and solid. There's enough features to get us going and to let us grow, but not so many that it's too complicated. So I think it's a great platform, easy to start, easy to grow, and highly recommended. So if you'd like to give it a shot, visit pythonbytes.fm slash digitalocean. It'll give you $50 free credit for new accounts, so you can give it a try. I think it's a good place to be. Get yourself a droplet.

Yeah, a droplet. So let's start this all over again. And not back from the beginning, but it's Chris's turn again. So Chris, what you got? Well, this week, Microsoft announced that they've improved Python support in their Azure platform. I think in particular, they've announced general availability for a bunch of new Python things, some of which I've so the company I work for, we've been primarily use Microsoft Azure and I've used some of these tools in beta. But so now it's like available to everybody. And I'm excited because reading through the announcement from Microsoft, I wasn't 100 percent sure what was new. But what I was really thankful for is Brett Cannon tweeted a link to it and said, here are the specific points that he was excited about.

The first off is that it's debuting with Python 3.6, but 3.7 support is currently being worked on, and actually I can already see it in preview, which is nice. And especially good news for me is that 3.8 support won't take nearly as long. So I'm hoping that very shortly after they release Python 3.8, it'll be available in Azure. And then the second thing is that there's native async and await support in there. And that's pretty exciting. But really, the other thing that I wanted to pull from that that Brett didn't mention in his tweet is that it's actually pretty impressive, like the amount of stuff that Microsoft does for us. In particular, the Azure functions, it's just amazing to me because they have a way of deploying to the web where you essentially just create a new Git remote and you

just push your code up there. And when it goes, when your code arrives there, it essentially sticks into a Docker image and runs it for you, and it's a pretty fast startup. And all these things that's so complicated, I can't imagine what's all going on, and you don't have to worry about it at all. You just have something that runs on your computer, you push it up, and it's done. And it's really, really impressive. I've never played with Azure Functions. So are those kind of like Lambda functions on AWS?

Yeah, they sure are. And I hear they're probably about as easy. I've never actually met. I've intended to play with Azure Lambdas. In fact, at the local Python meetup, we've had a couple of people talk about how easy it is to do. And I was just thankful that here at work I've been able to try the Azure functions. So it's really nice. Nice.

I'll have to give that a shot sometime. Yeah. Actually, I think Microsoft offers like a free like month or two month thing if you want to do it. And on top of that, the Azure functions are incredibly cheap. I just, they have a calculator up and you can check to see how much it'll cost to do certain things. And I maxed it out. Like I have a function that calls a bunch of API endpoints and saves them to the database. And we realized if we did it 48 times a day against every single website that the company owns, we still would be in the free tier. So it's really impressive what you can do. That's cool.

Well, I'm kind of excited about this next topic. Like super excited. Because we talked about, I'll have to dig up the episode, but we did bring up awesome Python applications before. But we have an update. Well, yeah, I don't remember what it was, if it was test and code Python bytes. But I remember accidentally, like, sort of dropping this idea on the podcast, and it kind of blew up a little bit ahead of when I expected to launch it. But yeah, so I'm here bringing like an awesome Python applications update. And so basically, awesome Python applications, for those who haven't heard of it,

is sort of like your standard awesome list where it's like you're on GitHub and there's a readme and there's a ton of links, you know. And all these links point to applications that are free and open source Python. And so at first it looks like a normal awesome list. One, it's actually generated from some structured YAML. And, you know, when I started out, it was like 25 applications, like Zulip and Sentry sort of came to mind. But then, like, the more I looked, the more I found,

and it grew to something like 180 by the time, like, you know, it sort of went a little viral. And then these days it's like 255. Frankly, it'll hit 260 by the end like you know it sort of went a little viral and then these days it's like 255 frankly it'll hit 260 by the end of the week and so the list keeps growing almost faster than i can keep up with it thank goodness it's somewhat automated and um so this last weekend i was at sort of the local python conference pi bay you know maybe see you pi bay 2020 it was a pretty big hit there i basically you, put in the show notes sort of a link to my slides

and where the video will be and whatnot. But what I did for my talk there was I took, like I said, structured YAML. So I have all of the links to all of the repos. Technically, any awesome list could do this. And I'm surprised that more haven't. But I went and I cloned all of the repos. And this isn't just GitHub. This is

things that are on GitLab, things that are sort of like off any particular software forge. There are things that are on Bitbucket, things that are in Bazaar. You know, it's a whole mix of things. And I cloned them all. And it was something like 30 gigabytes of code. Yeah. And these packages, they date, or these applications, rather, date all the way back to like 1998, like Mailman and gedit. You know, if you're a GNOME user, gedit uses Python. And 95% of them have had a commit

this year, 2019. So these are all very active, and the reason I've compiled this isn't so much that we can go use them, like, of course. I actually used several of them without even realizing that they're Python. But the thing is that as application developers, we need exemplars we can go look to for patterns in testing and documentation and architecture and packaging, things that people do without necessarily writing a blog post about them or, you know,

going on a podcast or giving a conference talk. So basically, I went and I cloned it, and I did all this analysis, and this is how I found Vermin. And so Vermin, the good news here, let me just, I should have led with this, right? But the good news is that, I was surprised by this, fully two-thirds of these Python applications, which are on median, like, you know, 10 years old, two-thirds of them support and use Python 3. Wow.

So I was expecting to find that a lot of these had sort of been, you know, it's tough to maintain open source. And, you know, I thought that they would sort of have maybe, like, sort of fallen behind a little bit, right? Goodness knows that most corporations I run into, well, I don't know about most, but, you know, a lot of them are still using Python 2 for their Python applications. You know, even like, you know, Google, everyone's favorite, right? Even Google hasn't like fully ported over to Python 3 yet. So yeah, but 66% support Python 3. And I've had a bunch of other interesting findings. If you go to my slides, you can check them out. And I'll also be giving the talk at PyGotham 2019.

So if you're in New York early October, I think tickets are available. So there's so many cool things in here. And there's stuff that I knew about once and forgot about, like Eric, one of the Python IDEs. So that's fun. Yeah, I mean, the newest one I found, someone at the conference came up and told me about it, is actually, I think, nia.is or.si or something like that. It's a torrent tracker for anime. And so this particular anime fan informed me that this is one of the largest torrent trackers out.

And, you know, it's written in Python. It's right there on GitHub. And, you know, it's not on PyPI. It's not really discoverable. You need this word of mouth to kind of find these applications and all the lessons contained therein. Like I calculated it out. Over 2,000 years of project maintenance, right?

Like in these repos that was just waiting to be learned from. And, you know, I think the list should grow even further. So if any of the listeners have applications that they happen to know are written in Python, you know, hit me up on GitHub. I'll definitely consider and curate. Nice. Well, this is cool. Thanks. Absolutely. It's amazing to see how much it's grown.

I remember when you were on whichever podcast you were on, and I was just inspired. And now I'm even more so, again, to see how many things have been built with Python. And I still haven't reached the bottom yet. That's amazing. My dream is really to make a static site for this so that you can filter because basically I now have the data, and I'll look at publishing this data set somewhere, but basically

it's like, show me the projects that use Docker. Show me the projects that use PyInstaller. Show me the projects that use Twisted versus whatever. You mentioned AsyncIO right earlier, and I was kind of surprised. There are quite a few older projects that have actually adopted

Async, like AsyncIO rather. You know, like, and it's pretty even split between Concurrent Futures, Twisted, Tornado, G-Event, and AsyncIO. They're basically all around 20% on this list. How about that? Check it out!

Well, I'm going to wrap this up with a, a really easy one. We've talked about pre-commit several times, I think on the show, but if you, and pre-commit is a tool to allow you to run some code before you check in your code to make sure things that are whatever, you've got a style guide or,

or some checks or whatever you want to do. It really is general purpose. But how do you get started? And actually, this is something that I needed myself, the pre-commit documents. Now, they just recently added a quick start guide. And I love it. It just kind of walks you through installing pre-commit, configuring it for the first time, taking a look at what the hooks are, installing the hooks, because you have to install the hooks also, which is something I wasn't aware of. And then the initial attempt to just run your hooks against your project. That's awesome, but I wanted to add in one of the reasons why I wanted to bring it up on the show wasn't just

because, hey, go check this out. but I'd like to add that doing this slowly is a good idea so I tried this with like three or four or five different hooks like flake eight black and some other things and then let it loose on my code base and it changed like everything and I thought I don't really know what's going on here so I'd like to modify it a little bit to say if you're adding this to a project for each new hook add them one at a time to your yaml file install it and run it against your files first and then commit that if you're happy review the changes if you're happy with it your test pass and all that check them in and then move on to the next one i think you it'll be easier for the rest of your team

to swallow the fact that you just changed all the files. Yeah, that's great news. This is one thing when I saw it in the news, I was super excited to see because we want to start doing this here at work. And now that I know that there is a quick start guide to do it, it makes me so much more excited to do it because I tried messing with the GitHub hooks or the Git hooks themselves.

I'm like, I really need to learn Bash. Well, not just that. They're just basically unusable in a team environment without something like this to manage them. So I think Anthony Sotow kicked us off, and he's really just kept running with it right out of the park. He's done a great job. And the community as a whole, actually, too, because anyone can contribute sort of useful pre-commit hooks there,

too. That's awesome. All right. So that's done with our six. Does anybody have any extra information they'd like to shout out? I have a couple of things I'd like to share. First off, the Humble Bundle, easy for me to say, by the No Starch Press, is throwing a special with the Python Software Foundation where if you buy, you can get a collection of the books for a certain price and the PSF gets money, which is wonderful. They can use as much money as they can get, I think. And I'm all about supporting them. And the second thing I wanted to share was I know friends of the show,

PyBytes, have released their newbie bytes, which I know they have put in a ton of hard work into. And I'm excited for them to have got it up and running. I'm excited to like, as somebody who runs a local Python meetup group, I'm excited that there's another resource out there to help people who are very new to Python to learn how to use it. Nice. That's cool. I didn't have anything prepared, but I mean, I guess I'll just shout out again, like, you know, I'll be doing an East Coast tour, like PyGotham and the Maintainers Conference

in Washington, D.C. You know, if you want to get a ticket to one of these, I'd be happy to meet anyone out there, chat about Python and versioning and all this stuff. I should also, like, you know, shout out my wife. She helped me make all the graphs for this awesome Python applications presentation. And, yeah, she was using, like, you know, shout out my wife. She helped me make all the graphs for this awesome Python applications presentation. And, yeah, she was using, like, real Python, I think, to learn a lot of the pandas to do so.

She really, again, just did an amazing job. So thanks, Maya. Sweet. Well, I was going to plug the Py3 readiness, but you already brought it up, so we'll drop a link to the other one. The top 360 packages are now Python 3, which is great. And I was hoping that one of you had a joke. So I only have one. It's not very good.

So either of you got a joke? Joke. Oh, my goodness. You go first. I'll see if it's an act I want to follow. Okay. Well, I was actually looking for some. So I was looking for programming one-liners, looked at a Reddit thread, read a great answer, which was, any joke can be a one-liner with enough semicolons.

I think that joke is obfuscated enough that I'm not going to try to follow it. Oh, I've got a good bad one. So a SQL statement walks into a bar up to a couple tables and says, can I join you? Ah. Yeah, yeah. Okay. All right.

Well, let's leave it at that. So thank you, everyone, for sharing all these awesome items with us, and we'll catch up with you later. Sounds great. Thanks, Brian. Bye-bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T- Bye-bye. Thank you for listening to Python Bytes. Follow the show on Twitter via

at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is

Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.