Python Bytes - #173 Your test deserves a fluent flavor

Episode Date: March 19, 2020

Topics covered in this episode: Advanced usage of Python requests - timeouts, retries, hooks Fluent Assertions Python in GitHub Actions VCR.py 8 Coolest Python Programming Language Features Bento Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/173

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 173, recorded March 12th, 2020. I'm Michael Kennedy. And I am Brian Akin. And we have one of our favorite sponsors, Datadog, sponsoring the show. Check them out at pythonbytes.fm slash datadog. Get a cool t-shirt, get some cool software. Tell you more about that later.

You know, I think people who, when they're getting started, Brian, they write basic code, but they realize that Python is so easy to use because you can just do something like request.get. And hey, you have a website already. Like you've already downloaded like something from an API or you downloaded something, some HTML or a file. And then it's easy to forget that maybe there's more to it. Yeah. I actually totally forgot there was more to it. And we know of requests as being both powerful and really easy to use. And there's just about a million tutorials on how to put something together quickly with requests. So I was really thrilled to come across this article by, I think it's Danny Hadovic,

titled Advanced Usage of Python Requests. He covers timeouts, retries, hooks, and more. And kind of that comment is it's easy to be immediately productive with requests, but there's some really cool things that you can do that I had no idea you could do this stuff. So some of the cool tricks that he runs down, actually all the tricks that he runs down, there's a hooks thing. You can attach a hooks to a session and so you can use hooks to call raise for status on every call so raise for status is a way you can

say well if if when i request something if a certain status comes back then call this other function but like a 404 or there's certain wildcard ones that you might want to like all the error ones you might want to always call something so there's a way to hook that up that's pretty cool oh yeah nice base urls i had no idea you could do this this is so neat instead of doing like uh the full path to a url for every if you're doing a whole bunch of them you can use a set up a base url that essentially gets prepended to everything else later. Yeah. And this comes from request tool belt,

which we covered before. So it like wraps and adds onto requests, which is pretty awesome. Oh, does it? Okay. Yeah. Yeah.

Yeah. Yeah. And this is something I do all the time when I'm writing code that talks to consumes APIs or talks to APIs or whatever is it's like, here's the base url we set and there's usually an if statement like are you in development mode do this are you in production use this other url as the base and then you know just do a slash whatever get but for me it's always been oh that's a great use of base yeah for me it's i've always been like well okay

get base url plus whatever then i'm like yeah, does it have the slash on the end or does it not have the slash on the end? Do I need to put the slash on that? You know what I mean? And it's super clear here how that works. So I really like this. Like I could totally have made use of this and I haven't been, so I should.

And then he covers things like retries on failure, timeouts, timeouts with retries, a little bit of testing and mocking of requests, but there's a lot about that. Yeah, I'm looking at the section here on timeouts. I feel like, do you think people could get frustrated about this or something? What's up with this guy in this card game, man?

I'm not sure. There's like this, like under setting default timeouts, there's like a giphy of a guy just beating a card table to death, right? Cause he lost, he's playing Settlers of katan and i'm guessing he lost anyway it's a great a great giphy to go along with it i'm not sure why that is relevant to here but it's funny uh it's a funny little video but why is it relevant i know this is irrelevant to the the podcast but how is this relevant well it says it's great that stuff will just wait,

but it can really frustrate you when someone's forgot to set a timeout and it halts a program in production. So I'm guessing this is an expression of being frustrated because production's lagged. Yeah, yeah. Anyway, the one that I also thought was interesting, the base URL is cool, is the retry on failure is pretty neat, actually. Yeah, and then you can even customize it to say, well cool is the retry on failure is pretty neat actually yeah and then you

can even customize it to say well not don't retry everything but certain things you want to make sure you retry yeah for sure and how many times and whatnot that's a great feature yeah and then this is kind of clever the last one i thought was clever was the uh if you're you want to mimic being different browsers by adjusting the user agent header request information because you know sometimes you want to make sure that you're whether or not you're testing your stuff you want to make sure that you can deal with different user agents or if you want to try to if some people have have security stuff on where they they only allow certain user agents because they don't want bots coming in well you can write your bot to be a different user agent then. Yeah, absolutely. You could lie to them. Tell

them your Internet Explorer 6. See what they think about that. Yeah. So this next item that I want to cover, I ran across last week and I thought it was really cool. And I'm like, I've been looking for something like this for a while. So have you heard of this library for testing? It's called PyTest. Have you worked with it? Yeah, a little bit. Yeah. Awesome. So Py you heard of this library for testing? It's called PyTest. Have you worked with it? Yeah, a little bit. Yeah. Awesome. So PyTest is super clean and great. And, you know, the way you express the assertions and whatnot is you would, you want to assert something's not none, you would assert the thing is not none. And like, literally, you write that code. And that's really cool for simple stuff like is not none but if you have to test multiple aspects

of a thing like it's not none none it is a number and it's greater than zero or it's between some range that can be a little bit tricky and you've kind of got to write some code it's not not as descriptive or as expressive right that is one of the things that I was kind of looking for is like a library that has more complex tests built in. Like, is this thing a subtype of that or is this string parsable as an integer? I'd like to know that beforehand, you know, like something like that. And I don't want to have to do the little like the complicated, small, but still somewhat complicated and non-obvious code in these assertions. Right. I'm also a big fan of fluent APIs. And what do you mean by fluent?

I would love to see more fluent APIs in Python, the standard library. So what I mean is a function or property that you can call that returns the same object. Oh, okay. Sort of functional. Yeah, a little bit. So if I had, say, a list, I could say list.sort. Some other list operation. If sort would return a list, the list that it was called on, right? And maybe it had an order by, or I guess that sort.

If it had a filter and then you could say dot, right? Transform or something like, right? You could sort of chain these together without doing multiple lines. Yeah. Right? So that's the fluent API. And so this thing i want to talk about is something called fluent assertions and i ran across this from dean egan and he was

talking about it on twitter and basically the idea is it has all these checks like tons and tons of checks built into it i'm not sure how many i would guess 50 different checks. Is that number complex? Is it not complex? Is a value between these two things? Is this set? Is it a set? Is it a non empty? Does it contain spaces? Does this string contain spaces?

Is it shorter than this? Right? So you could say things like given an object, is it a string that contains no spaces? That is all lowercase. You could do that as like one line, just like really clearly in this Fluent API. And it throws assertion errors,

which are the same thing that you would get if you failed an assert directly. So it integrates really well into things like PyTest. That's pretty cool, right? Okay, cool. Yeah. Yeah, so there's an example in the show notes here. And I took this from their docs. It basically says

if you wanted to test something like given two parameters i want to check that the value is not none it's an that it's a float that it's between zero and one and you're given an object and it's not none and it's a type of something right that's kind of complicated or you can go to their api and say check n dot is not none dot is floatbetween. And I thought, this idea is really cool, but I don't like it so much. There's like too many words, right? So I've been working on a PR, and it's basically accepted.

The guys running the project said, sure, this looks great. They've reviewed it, and it's a simpler one, and it uses properties a lot of the time. So it's a little more English, less function-y called. So you can say is n dot not none dot float dot between parentheses zero and one close parentheses. So the things that assert like, I guess, properties about it, like whether it's a float or not none are just properties and the stuff that takes arguments would be still function calls. So I think it's

a really clean way to write these test assertions in nice, simple ways, especially if you're testing a couple aspects. It's not none and it's a float. Yeah, I'm on the fence. I know that a lot of people like this sort of a style. I like your updates to it. I hope that goes in because that's a lot cleaner.

It's a lot easier to read. So good luck with that. I'm perfectly a fan. That sounded harsh. No, that's all right.. I'm perfectly a fan. That sounded harsh. No, that's all right. No, I'm a fan of just the straight asserts myself. I think they're easy to read, but I think having a couple ways to do it, that sounds neat.

What I like about it is it packages up some of the more complicated types of tests, like the character only has, like the string has some characters which are spaces, or things like, it's not as obvious if you write the actual code that does the test some of the times. I don't know. Anyway, that's what I like about these kinds of APIs

is the sort of English telling me what you're looking for more than the code of checking for it. Yeah, and also you're fitting a whole bunch of stuff on one line. It's kind of nice too. Yeah, yeah. This is a way that you can have multiple assertions in one test without people saying, hey, you're doing more than one test. Well, sort of.

Anyway, people can check that out. It's a pretty cool library. If that sounds useful to you, it's already working for the check API and probably will for the is later. Cool, cool. Now, before we move on to the next, I want to tell you about Datadog. Because they're sponsoring this episode, as they have been, great supporters of the show. Let me ask you a question, dear listener. Do you have an app in production that is slower than you like, or its performance is all over the place? Sometimes it's fast, but sometimes it's slow and you don't really know why. That's the most important thing is why

is your app behaving this way? Do you know what's causing it to be slow or to be kind of all over the place? And if you have Datadog, you'll know. You can troubleshoot your app's performance with Datadog's end-to-end tracing. Use their detailed flame graphs to identify bottlenecks and latency in that finicky app of yours. So be the hero that got your app back on track at the company and get started with a free trial at pythonbytes.fm slash datadog. Very nice. Yeah. Thank you, datadog. Good stuff.

Also good stuff, GitHub. Yeah, GitHub and GitHub Actions. I don't know how long they've been out of beta, but GitHub Actions, I think, are available to everyone now. And these are different than their webhooks, right? These are more automation workflow inside GitHub. Yeah. And GitHub's sort of by part of the Azure Pipelines

sort of stuff. I mean, GitHub's associated with Microsoft now. So Azure Pipelines are one way to do actions on a... when you commit something or actions at an event time within GitHub. But GitHub Actions are a way also. They're more of a

lightweight pipeline thing. But for a lot of Python projects, I think they're a very good, clean way to go because a lot of our Python projects are not. If you're building a package, they're kind of perfect. So there's an article called Python and GitHub Actions and it's by Hinnick and it's sort of really cool. He says he's currently recommending that people use GitHub Actions for Python stuff. It's simple, easy integration. So how do you do that? And that's what this article is about. And he goes through running your tests through talks, using coverage,

testing against multiple Python versions, and shows you the YAML that you have to set up to configure GitHub Actions to do that right. And you also have to put a little bit of a, make a change to your Toxini file to make sure this all works. And then I think it's good, he was reminding people that if you've got an open source project, it's kind of nice to clean up your old stuff. So if you are switching from some other CI system to GitHub Actions, make sure to clean up the old stuff so if you've if you are switching from some other ci system to github actions make sure

to clean up the old stuff and then he even goes through and uh tells some other things like um changes you want to make to make sure that you're hooked up to the code to gov system and some other stuff um and then making sure it builds on multiple operating systems the sort of stuff you'd want to do with uh continuous integration most of it is available through Azure, or not Azure, through GitHub Actions. So that's kind of cool. I love that it's all part of GitHub now. That's great. You know, if you're already there, it can be a pain to integrate these other systems.

You can just push a few buttons or a few files and just make it run there. Yeah. Kind of the secret sauce is that they're really the same thing. I think the Azure pipelines and the GitHub actions are all running on the same system, but the GitHub actions is a little simpler interface for people that aren't. I mean, Azure pipelines and the

Azure workflows is powerful, but it is quite overwhelming when you get into it. Yeah, that whole system is pretty much all of Azure is overwhelming to me i go there i'm like why are there 400 things here there's so many things like i don't care about most of these what is this place yeah yeah yeah going back to digitalization okay cool this next one i want to talk about touches on a sort of a similar topic as the request one that you covered

as also the assertion one that i did So one of the challenges of testing your code can be when you're talking to external systems, right? I want to call the Stripe API and I got to provide it all this information. And if you call it more than once with the same token, it'll say, sorry, you can't do that. That token's used. So you got to go get a new token.

And there's just a lot of stuff. And you necessarily want to be calling real live APIs inside your tests. That's going to make it quite slow potentially and so on right yeah you can get charged like uh we use a geolocation service on the training site to figure out which server to serve the video from and that one it's not super expensive but we have to pay per use so i don't want to hammer it in continuous integration and pay more so there's this cool project called vcr.py have you heard of this yeah yeah vcr.py is really cool i heard about this from tim head who was on talk python recently for the binder project and that's going to be out shortly

but the idea is that this simplifies testing things that make hdp requests as well as speeds it up so all you got to do is the first time you run it, you decorate the function. And what it's going to do is going to basically instrument and record all the HTTP interaction, what gets sent out and then what comes back. And it'll save that into a YAML file hanging around, which is called a cassette.

Do you like that? Plug the cassette to the VCR, right? And the next, the second and third and fourth time that you run the test if it finds that cassette file and the same inputs it's like well you asked for this and here's your answer and it just replays it back to you yeah it's pretty clever yeah it's super clever you don't have to worry about if the system is maybe slow or you've got to set things up just right to call it because once it has that little cassette file it's good to go so it lets you test these external service test your integration with these external services in a semi-realistic way because you're really playing back at least snapshot in

time real data that you got from it without any effort on your part lets you work offline your tests are completely deterministic if it passes once it's always going to pass because you always get the same data back. Definitely speed up the execution speed because it's just throwing back JSON that it has in a file rather than hitting an external service and all that. And if you decide, you know what, this request

is stale and out of date, all you got to do is delete that cassette file, run your test once, it'll hit the real system, and then it'll go back to playing the new cassette. Yeah, pretty cool. And then for people who do like PyTest, there's a cool little plugin called PyTest dash VCR, and

then all you got to do is for your test that might use something like URL open, or request or something, you just say PyTest dot mark dot VCR for the test, and then that's it. Magically, it works. Yeah. Okay. Nice.

Yeah. I don't know if it's really magical. I think it's very useful when it can be useful. Yeah. I personally don't have the experience, but I'm going to lean on somebody that does. On episode 102 of Testing Code, I talked with Harry Percival, and part of that that we're talking about

how to set up testing an application that has external dependencies through APIs and stuff. He does talk about both good experience with things like vcr.py and some difficulties like if there needs to be timestamps or different

if you call an API twice and expect to get something different back right well this isn't going to help you but so i recommend if you're running down this route then then also listen to the half hour of episode 102 with harry percival it'll help a lot yeah super we should put a link in the show notes for that yeah to me one of the things that i really like about this is i could go pip install stripe or pip install MailChimp or some other thing that I'm integrating with that who knows what kind of complicated badness it's doing on the inside to make all of its stuff work. You know, and I don't have to think about how am I going to mock out their internals? And if their internals change, how's that going to affect my tests? And I can just say, I'm going to grab this higher

level API that somewhere deep in its guts does network traffic, throw this at it, and then it'll be reproducible. And to me, that's the big appeal. Yeah, and also speed. So even if you're using a test server and not incurring all the overhead

costs of the actual server, even of the actual server. Even with the test server, it's time, it's latency, and you can speed things up by caching the return values. So it's a cool idea. Well, I'm glad you like it. I know it's probably not something you do as much in the hardware world, but yeah, it's a cool one.

What else is cool? What else you got? I got eight cool things. I'm a sucker for listicles, if there is actually good information. And so Jeremy Grifsky wrote the eight coolest Python programming languages features. And I was just smiling the whole time I was reading this. It's a quick article, but it talks about a whole bunch of features. I was reading it thinking, man, this is why I love Python. And I really miss all this stuff when I'm writing C++ code. So there's code examples, of course, in this article, but

we've got list comprehensions and that's something that when you first learn about comprehensions, it's like, oh my gosh, this is so cool. We also have dictionary comprehensions and other comprehensions now. there's all sorts of stuff generator expressions are nice and really helpful slice assignment i sometimes forget that we can do this so you can take like part of a list and assign like if you have three values or something you want to stick in the middle of a list you can assign those with slice assignment it's pretty pretty powerful iterable unpacking so if you've got a tuple and you want to unpack that and pass that to a function

but as separate values you can do that negative indexing i mean you want to grab the last thing off of an array you can say minus one or minus two i love negative indexing so much that's a really clever fee it's so simple and yet it's really nice to just go, I want the thing minus one, bracket minus one. Give me that. That's the one I want. I sorted it. I want minus one.

I've tried to do this in C++. It's a bad idea. So negative indexing is cool. Chaining comparisons. So one is less than X is less than five. To make sure X is between one and five that's not something you can do in most languages and it's just it's how we talk it's how we do math and python has it it's nice uh he finalizes it out with f strings which we love f strings

and then a whole bunch of a big list of honorable mentions and i was thinking as i was reading this like this entire thing plus his honorable mentions at the at the other stuff of like things i could have also talked about that would be a really great just like a an introduction to python course of just like here's a half an hour of why i think python's awesome so it's good i agree i think that would be awesome let's see as i look through this f strings definitely stand out as something that's awesome negative indexing already riffed on that that's cool meaning comparisons list comprehensions and i'll throw all the expressions and other comprehensions in there i love them but i wish they did a little bit more like why can't i sort on a list expression list comprehension or something like that there's just

a few things where it's like oh man if i could just i find myself a lot of times here's the comprehension and then here's the little bit of things that i i wish it could still support i got to do afterwards anyway it would be nice interesting like for example paging right i would like to be able to skip so if i'm on page 5 of groups of 20 i would like to be able to skip so if i'm on page five of groups of 20 i would like to be able to skip four times 20 take 20 with this clause sorted by that and have the sorting happen before the paging like that would be so nice and it's just you know it's on the cusp so half the time i'm like god i love these but why can't i you know whatever the little thing that i wish i could extend a little more database and memory type of behavior, but still great, great stuff.

I would totally miss them if they weren't there. That is for sure. Yeah. Yeah. Cool. Actually. So slice assignments, I would, I didn't never even occurred to me that you could do that with a language. So it's neat that you can do that, but I probably wouldn't miss them if they weren't there because I wouldn't have expected them to be there.

You shouldn't be able to do that. Yes, exactly. Exactly. I wouldn't miss LightSummon at all, although it is neat. You know what I definitely would not miss is bugs in my web app. Yeah, me either. Do you have bugs in your web apps? Well, I always think no, and then I learn yes, but not as often, not that often.

Right. So there's this cool project called Bento that I just learned about. Have you heard of Bento before? Just the lunch style. Yeah, exactly. I do love a good Bento box, but this has nothing to do with that. Maybe the name is inspired, but other than that, no.

So Bento at bento.dev. This is basically an analysis system that will look at your Flask and coming soon, other languages, Django,ql alchemy whatnot look at your flask app or your request calls and look for known bugs especially in the security realm so you don't end up with little bobby tables this is your school calling did you really name your table your son dash dash semicolon, semicolon, drop, whatever? Yeah.

So the idea is you can basically pip install this thing. You call Bento init. It's going to create a Docker container with the analysis tools. And then it's going to run those against your Flask app at the moment. Like I said, Django, SQL Alchemy and stuff is coming coming along and it'll find things like if you have a missing JWT token or you're missing a no opener or the content set or if you're using requests if you're sending username and passwords over an HTTP not a not an HTTPS request it'll automatically detect that and tell you oh very cool yeah it. Yeah. It's pretty nice, right? Yeah. It's very nice. It even does like ginger template checking and stuff. Right. Exactly. So this is open source on GitHub and yeah, you could check it out. It looks pretty

nice. There's a little Giphy. Thank you guys. Well done on your project. There's a Giphy showing how it works right on the page. You can just go to the GitHub repo, scroll down a tad and just watch them find a bug fix a bug and so on yeah very good cool cool anyway so yeah if people have for the moment flask apps in the future it looks like they're coming with other things but yeah you can just run this and it'll check it out there's also a list of all the checks that they have which i put into the show notes so a bunch of stuff like you know some of the obvious ones are like did you ship Flask

into bug mode and stuff, but other things that are not so obvious. That's it for our main topics. Brian, do you have anything you want to share with folks before we get to the laughter,

the hilarity? No, we could use a good laugh, but do you have anything extra? We haven't talked about it yet, but I feel like it's, all the tech conferences

are either canceled or they're on coronavirus watch for being cut, right? Like we've had E3, the big game one, canceled. We have the Game Developer Conference. We had South by Southwest canceled. Some other ones, I think some Facebook's F8, I think, was canceled. Possibly Google I.O. I'm not sure about the one.

I don't remember exactly what they said. But PyCon's still on for the moment. At the time time of this recording there's going to be an announcement tomorrow which may change things but just what I just want to remark like what a crazy time both for the world but also for tech yeah definitely crazy I'm curious what it's going to do for uh now this is totally self-centered I do want everybody to be healthy. But I also wonder, with less people commuting, if less people are listening to podcasts, that would be terrible. Oh, no, they've got a lot of housework.

You all folks out there, you definitely got to keep listening. No, that's actually a legitimate question, whether or not that makes sense. Maybe it does. Here's some of the effects I think are going to happen. I think a lot of companies, especially larger companies, that believe you must have a meeting every Wednesday and it has to be

like two hours with this group and an hour with that group. And you must commute every day into the office, even though you just work and through GitHub and Slack and email anyway, they're going to realize, you know what? We don't actually have to have these big offices and we don't have to have our people always come in. We were at least as well off. And what's going to be like the work-life change that comes from that realization? Yeah, we made sure, I mean, a year ago today, it was rare for anybody to be working from home in our company, even though the work is software for the most part. But now we've made sure that everybody, I think we have out of like 80 people,

something like that, we've got like only a handful of people that are not set up yet to be able to work remotely. I think that's a good change, actually. I do too. It's going to be really interesting to see the knock-on effects. I think there's going to be stuff like that, like, like oh wait we actually could work in this way or we could

hire people from other places or whatever it's going to be interesting outside the just you know the potential chaos of people getting sick and whatnot this has been like talking about the elephant in the room but it's talking about the virus in the room yeah well i hope it's not in either of our rooms so let's just put it like that like right now yeah definitely not there's nobody else in my room. I think I'm clean. So speaking of mysteries, how about I tell you a joke? Oh, please.

Okay. So let me give you a definition straight out of the dictionary. Debugging. Pronounced debugging. It's a verb. Primary definition. Being the detective in a crime movie where you are also the murderer. Yeah.

That's a good one, right? Yes. You probably get that less since you're often coding solo. But there's times where I'm just hot under the collar, mad about some bug in the system and trying to figure out who did this. It's time for some version control blame, some git blame, some subversion blame. Yeah. No.

It was me. It was me. Yeah, I've been there. I've been there like, oh my gosh, now what do I do? Definitely. Yeah. Love that. It's good for your humility level though, right?

Yeah. And then you just remember that next time you're mad at somebody else. Exactly. Or doing something boneheaded. Awesome. Well, thanks for being here. Great to chat with you as always. Thank you. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via

at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.