Python Bytes - #86 Make your NoSQL async and await-able with uMongo

Episode Date: July 13, 2018

Topics covered in this episode: responses 29 common beginner Python errors on one page μMongo Basic Statistics in Python: Descriptive Statistics Strings and Character Data in Python PEP 572: Assignment expressions accepted Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/86

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 86, recorded July 11th, 2018. I'm Michael Kennedy. And I'm Brian Harkin. And Brian, we have a new special guest, Bob Belderbos. Yeah, pretty excited. Yeah, welcome, Bob. Hey, nice to be here. Thanks. Yeah, so people may recognize Bob from his PyBytes with an I, Code Challenge stuff that he does,

as well as the 100 Days of Code course that we did together, right? Yeah, we did a major course on 100 Days of Code in Python. It was very fun. Yeah, it was only, what, 20 hours of video and 80 hours of exercise? Yeah. That was awesome. So welcome to the show.

We're super excited to have you here. Before we get into everything, let's just say thank you to DigitalOcean, our customer, and their sponsor of our show as well. Let's check them out at pythonbytes.fm slash DigitalOcean. You'll get $100 free credit if you're a new user. That's pretty awesome. Brian, if I had to guess, if I were just going to pick out of thin air what kind of topic you might have, it might have something to do with testing. Yeah, I do like testing quite a bit so what do you got for us well we have responses and this is it's there's a funny story around it but this is uh uh responses is a utility library for mocking out requests python library and um

and it's from it's from git sentry so it's uh under the git sententury GitHub account. And it looks pretty cool. It's actually, we'll include a small snippet in our show notes, but it's a decorator approach and other ways to be able to test your, pretty much mock out if you have an application that uses requests to access really any endpoint on the internet, you can use this to mock out those. And it has some dynamicness to it. And it looks actually pretty complete and pretty nice. That's pretty sweet. Yeah. You put an attribute on your function. And during that one,

any call to request like request.get or the response.json or whatever is going to be basically overridden as if you had done that with patch, but it's a little more focused on just requests, right? Yeah, it's focused on requests and it's also designed to go well with the PyTest framework so it cleans up after itself really easily. Nice. How does it integrate with PyTest?

What's special about that? The decorator you put on your test is like responses activate so it'll um and then you're you're going to have it you register a an end point to mark and what response it should give and then all of that gets cleaned up at the end of the test you don't have to undo it or anything like that yeah that's awesome very nice the funny thing is is uh this is exactly what Anthony and I were considering building together. We just didn't know this one existed. And so I wanted to point it out because responses is a clever name to go with requests. I get it. But if I'm trying to find something that goes well with PyTest and goes well with

requests, I might not know to search for responses. So that's why I'm highlighting it out. So I think anybody that's trying to do something like this should check it out. It looks pretty complete. Cool. It looks very useful. So it's not making any call over the network, right? That's the whole point of patching it out, right? Well, it's patching. It'll patch out the endpoints that you specify i don't really know i don't know what the behavior is for other ones it probably lets everything else go through yeah that's one of the things i like about it as you say this if i request this http endpoint then do this rather than i'm just replacing you know request.get with everything right right oh yeah

pretty nice i do see your point about se SEO and discoverability for responses because like response and responses is just such a common word on so many API libraries. Like how are you going to find the testing one? Yeah, I was looking, it says in its description that it's Python 2.7 or newer. Well, so it isn't obvious to me that it was 3x, but it looks like in the investigate the toxin file, it's tested up through 3.5 at least. So it's probably 3.6 and 3.7 compatible too.

Haven't tried it, but... Yeah, probably. Probably. Yeah, very nice. All right, Bob, it's time for yours. What one did you bring? Yeah, I found an interesting decision tree article, 29 Common Beginner Python Errors on One Page.

And I found this interesting because these kind of decision trees or graphics, you get a lot of information in one page you can print out or study in detail. And it's like a flow. It's like an infographic for like getting my code to work. It's amazing. Yeah, it didn't land directly on the article actually. It was Microsoft devs that tweeted it out and that's how I found it on the Python hashtag.

And it goes through this whole flow like what kind of error do you get? And then it goes into these common Python errors like attribute errors, syntax error, type errors. And for each one, it gives like a couple of plausible causes. And yeah, I mean, if you have more experience, some might be obvious, but some are very subtle. And that was like a great reminder,

like comparing three int with a three string that returns false. And for example, when I was going through it, I also could relate to similar things like when you exhaust a file, that also can happen with a generator, right? A generator, you go through it once and then if you do that again,

then hey, it's like an empty list because a generator only, you can only consume once, right? So it was a nice, I guess, exercise to take a lot of info in from one infograph. And I think for a beginner, it's very useful to go through that at least once. And I think the author made the infograph

after having teached Python for a long time in the biology space. So that was also inspiring for me, like teaching others, yeah, how you could summarize that information at some point. I'll try to give a quick summary just so people can get the visual. So it's like an infographing. It says, start here.

Do you get an error when you run your code? Yes. What type of error do you get? And there's a whole branch there. No. Does your code use loops or if statements? If you use an if statement, go check this. It just walks you through some pretty basic stuff. I of like it brian what do you think yeah actually um i this

is one that i've run across before and when our team started using python for testing one of the other engineers actually printed this out and posted it on the side of their cubicle so that people that came and asked them questions could just look at that first. Yeah, that's pretty cool. And I like the fact that it's coming from Python for biologists.com. So we're seeing Python being taught in a lot of different domains, which is awesome. Yeah, that actually is a really good point. It's pretty awesome.

So the next one that I have for us is this thing called MicroMongo, which somehow I have not heard of previously. But Brian, you know, I go on and on about async programming stuff and parallelism and things like that, right? Yes, and I do know that you kind of like MongoDB also. This is like I get to put them together. It's amazing. So one of the main challenges of this async await stuff

actually becoming super useful and super powerful. There's two fundamental problems limiting the serious adoption of this. One is the web frameworks don't deeply support it. So I can't have an async flask method. I have to switch to something like Cort or use Fibora or some other framework, like the main Django Pyramid flask, they don't support the async. So that's thing one, but we're seeing some movement there. But even if you could get that to work, a lot of the ORMs don't have great ways to take advantage of their parallelism or their potential parallelism, why they're waiting on the network talking to the database. Because usually that's what your web app is doing. And

that's why you would want the async is like you could let it do other processing while you're waiting on a database response. So what you need is if you're using an ORM or something equivalent called an ODM for Mongo is the ability to interact with those databases is in an async way. And so this MicroMongo is a very lightweight and small, as you might imagine from the name, object document mapper for MongoDB and Python

that brings this ability to do async operations against MongoDB. Right. So if you wanted to use, say, Quart or Vibora and you want to use MongoDB as the back end, you have to use something like this, which is pretty sweet. Very interesting. I need to go on the async train myself yet. So maybe there's a good opportunity to learn it. So definitely. It's not that I mean, Mongo itself, you don't need that's on the other end. So definitely. It's not that, I mean, Mongo itself, you don't need, that's on the other end. So that's already asynchronous or can be, but it's the document mapper that's the problem. You need the library that talks to the database

to have an async option. So for example, like let's take something that people know more about, probably SQL Alchemy, right? If you go to SQL Alchemy and you create a query and you say like order by or filter or whatever, you need SQL alchemy to be rewritten. So that filter is async is an async method, or it's useless in terms that right. So it's like, the place you want to get to is interact with your database asynchronously. But if the ODM or ORM that you're working with

in the middle, doesn't support that there's zero workaround, you're done, right? You cannot do that. Whereas most of the things you're doing is waiting. But this one will let you choose the foundational bits. So you can say, I don't care about asynchronous programming. So I just want the base to be PyMongo. Or you can have TxMongo. Or you can use MotorAsync, which is something from Jesse Davis, I believe, at MongoDB. Or you're like this, Brian. You can even say the driver,

the foundational database access part of this ODM is MongoMock. So if, Brian, you can even say the driver, the foundational database access part of this ODM is Mongo mock. So if you want, you can just say actually replace the internals with this mocking database layer for testing. Oh, actually, that's pretty cool. Okay, neat. Yeah. So this is this is not I haven't used it. I generally use Mongo engine, but this is really looking quite promising and powerful. So definitely want to give a shout out to that. Speaking of shout outs, let's give a shout out to DigitalOcean. So DigitalOcean is sponsoring this episode and they have a great, very affordable, very reliable service over at digitalocean.com. So our stuff runs on DigitalOcean. We have a couple of servers

doing all the magic, including one doing MongoDB back there somewhere in New York City, I think it is. Anyway, they're super great. Go over there and create a virtual machine. Get an awesome machine for $5 a month. Get it set up in 30 seconds, and off you go. If you want to create something more pre-configured, they have a bunch of one-click apps to, say,

set up like a ghost blog for $5 a month for one click, things like that. So check them out at pythonbytes.fm slash digital ocean. You get a hundred dollars credit for a new user and you'll let them know you're supporting the show and, uh, or thanks for them supporting the show. So Brian, what's up with this statistic stuff here? I've not done statistics for a really long time. I'm one for saying that I've been known to oppose, uh, the stringent math requirements in a lot of computer science and other science fields. But regardless, there's a lot of people that end up having to learn statistics.

And I think statistics is a good thing to learn, at least some of the basics. Yeah, I think we often come into, like even going through computer science, you might not even take statistics. You probably took calculus and differential equations and other stuff that you never, ever use again. Right. Yeah, exactly. Yeah. Let's not go down that tangent too far.

However, we'll all start crying about remembering how much work we put into learning that stuff, right? I know I have never needed to factor a polynomial ever in real life. How about computing the inverse of a non-singular matrix? Sorry, let's keep going on statistics. However, sometimes I do need to know like averages and medians and stuff like that. So this is, I'm highlighting this article. It's called the basic statistics in Python, descriptive statistics. And I like it.

It's just using some simple python to demonstrate how you would build up some simple statistics concepts to teach the concepts like min max mean median mode some of the others like standard deviation it's just a handful of them but those are kind of hard things to get your head around when you're first looking at it so being able to play with the numbers with a computer i think is a neat thing so they give um they give some descriptions on how you would implement some of these functions in python oh yeah that's nice it does a nice job like like for instance if you wanted to figure out the average you can take the sum of all of the all of everything and then take how many scores and it's like the sum divided by how many. Those are good concepts to look at.

The thing that was missing out of it is the little bit at the end, I think, that says, by the way, don't ever do this if you're actually writing Python code because all this stuff's already built in. Exactly. Yeah, you've got the various numerical libraries for it, right? Isn't this stuff already built in?

And, you know, Min and Max are already built in just normally and some of the others. But I actually went and looked and found a – I didn't know about the statistics module. That's built in, I think, as of 3.4 or 3.5 or something. That's right. It's quite new. I don't remember, but I think it might be 3.5, actually. 3.4. 3.4. Thank you, Bob.

Okay. But I threw in some examples. But if you're just using that, it doesn't help you learn what statistics is. So that's why I think the article is still good. that had both standard deviation variance and population standard deviation and population variance. I don't know enough statistics to know what the difference in those are. I know P goes in front of the method name. No, that's awesome that that's in there. Yeah, I think it's interesting that they added that module

into the standard library in 3.4. It's cool. Yeah, I guess maybe they were tired of people implementing them in crazy hand-coded ways. Yeah, and it's got to be faster, written in C, to compute the variance than to do it in a Python loop. That's a good point. Yeah, and it again goes to say that always

keep up with the standard library because there's so much stuff in there that can save you writing code and probably gain you performance. Yeah, absolutely. All right, Bob, what's the next one you got here? This is a big piece. Strings and character data in Python by RealPython.

I don't have the... If it's by RealPython, you know it's kind of like a novella. Yeah, exactly. Because I want to compare it with that Airtools article you featured a couple of weeks ago that was like this long piece that took me an hour to read, but I got so much it with that IterTools article you featured a couple of weeks ago. That was like this long piece that took me an hour to read, but I got so much out of that. And it's the same here. I didn't even honestly didn't finish it yet.

I'm like 60, 70 percent in and always already got so much value out of it because it does this great detailed tour of strings in Python and all the methods you can use on them. And I put some snippets in the show notes because, I mean, for a beginner, it's great because you will be working with strings from day one. But even if you know Python, there's like a lot of these little tricks in there that can just make you faster in Python. If a string is a digit, you can use isDigit on the string, which returns a Boolean. Rather than try to convert it to an int, catch the exception and go, this is not an int. Yeah, that kind of thing. Yeah, so you can use that method. Or for example, you want to look for a white space, a space, a tab, or a new line, there's also an isSpace method that does that for

you. Or commonly with terminal terminal apps you want to have this banner with a text centered well you can just do string center and then the width and then the the the filler character so there's there's a lot of good stuff in there and i think it's worth to go and spend an hour and learn all that stuff because you mean it will just shave off time when you're actually coding and another thing i also found kind of the linking you do when you read such an article for example polymorphism right if you have the index method works on a string but also works on a on a list so i put a snippet in the show notes that i use index on a string in a list,

and you see that they behave in a similar way. Same goes for count. So I found that also an interesting point to highlight. Oh, yeah, this is really interesting. I just threw that into Instapaper, and Instapaper says reading time, 33 minutes. Yeah, this is definitely like a book chapter almost on strings characters. I might be a slow reader, but... Well, I'm sure I would too. And that's not counting that it's like a book chapter almost on strings characters. I might be a slow reader, but... Well, I'm sure I would too.

And that's not counting that it's like code. Code you got to analyze, right? Yeah, you want to try some things. Yeah, yeah, yeah. Exactly, exactly. It's a great find. It's really cool. And people coming from different languages,

I know I was like this. I did not expect that strings would have all of these cool operations on them. Yeah, especially if you're coming from C where strings are really just, you know, character pointer array type things. They're basically just memory, right? Yeah. Yeah.

But imagine you have to do that, leaving zeros to a number. And that would take you a couple lines of code. In Python, it's just number, string, Z fill, and then the number of zeros you want to be penned. And it just makes for shorter code. Yeah, it's wonderful. Zed fill, who's he? Yeah, so you want to have 42, for example, and it needs to be a width of five.

You can just set fill it with zeros. Nice. So I'm going to close this out with something that is surprisingly controversial i don't know i'll see what your all thoughts are so there's a new pep in town pep 572 and whenever i think about these things i always think someone has worked super hard on this and that's really awesome they're making making a contribution. And then, you know, to see people react and I don't know, not totally excited ways. I don't know.

It's sort of, I don't know how to feel about, let's say, but there's a new piece of Python syntax that will allow you to, in a single expression, create a variable and assign to it. So, so often you have to do something to the effect of like, I'm going to create a variable and assign to it. So so often, you have to do something to the effect of like, I'm going to create a variable, set a value. And then if that value is something, I'm going to do a thing or I'm gonna go do something else. So the new syntax lets you put a colon equals to define that and put it all in one line. So instead of saying like, match equals pattern dot search, if match is not none, etc, etc, you can just say if match colon equals pattern dot search is not none all in one, I'm actually even confused about your example, I thought that the point to that

new operator was explicitly to check against none, the point of the thing is to allow you to both create the variable and set its initial value and check it at the same time, whatever you're checking it for. Oh, right. And you don't even have to check it. For example, one of the really common use cases for this is to use it in like a list comprehension or something like that. And anytime you need more than one line, that cannot be put into a list comprehension or lambda expression, right? But now with this one line, a variable creation assignment, potential test behavior, you can put these into inline expressions like list comprehensions or lambdas or so on. I think that's it.

I've only skimmed some of the use cases, but that's what I'm thinking it's for. So this is interesting. The syntax is not terrible. I think it's kind of okay. There's already lots of examples of this in other languages that had pretty decent implementations. So one of my thoughts was like, well, why didn't that get adopted? So both in C-sharp link expressions as well as JavaScript expressions, you have let something equals a value in terms of like a, like in terms of a for loop or something like that, that could have also been an interesting option.

But I don't know, to me it's okay, but I don't know that it's necessarily needed for the language. Like it's not solving a problem that I think very many people actually have. I wouldn't have to use it and see it in other people's code because from the snippet you posted i i find it a

bit confusing it might just be me having to get used to it but it's not the kind of syntax i'm i'm used to from python right which is it's more readable what do you guys think to me it's i it's it's like a one step down the slippery slope you know i mentioned c-sharp and the let stuff and i feel like the c-sharp language used to be pretty nice and it's just a complete train wreck now there's all these little three or four ways of doing the same thing and they're all getting like a few characters shorter but now there's five of them they're all more or less equivalent it's just like whoa why do we keep adding these things to this language they're just unneeded and i feel like this is sort of in that category of stuff. Python's nowhere near as bad.

It might not be the pep itself, but more what we start to introduce and what other syntax changes that might be coming, right, if you open it up for this. Yeah, so what I want to sort of point people at for this is the tweet by Raymond Hedinger and then the Twitter conversation that follows it, which is really interesting. You don't normally see this much conversation in

a thread on Twitter, but there's like 44 messages onto this one tweet and it's pretty interesting. Some people like it, some people don't, but I think it's worth reading through. Nice. So anyway, New Pepin Town, and it's approved, right? This is not proposed. My understanding is this is approved, so it's worth reading through. Nice. So anyway, new pep in town, and it's approved, right? This is not proposed.

My understanding is this is approved. So it's now going to be part of maybe Python 3.8. Yeah, I'll totally use it. But it was one thing that made Python different was that, like in C, if you tried to do check for equality and you accidentally did assignment instead it's wrong and i guess that's why the syntax is is specific so that it's not going to be an accident right and c you could say something to the effect of like if uh match equals pattern dot match do such and such right and you actually meant if those two things were actually equal right so there's a way the the syntax does make that sort of fall throughthrough error case not actually,

you can't omit an equal on accident and actually get assignment. Yeah, you have to explicitly put the colon in there. It's colon equal, so, okay. All right, well, there it is, PEP572, the most controversial accepted PEP I've heard in a little while, pretty interesting. All right, well, that's it for our news items, you guys.

Brian, you got anything else you to share with the world these days what's going on well actually kind of nice uh nice tie-in with this pep 572 i wanted to just talk with somebody about all the new stuff that's proposed actually proposed and accepted already for 3.8 and i think just for 3.8. So that's kind of what Anthony Shaw has been up to lately. So I'm going to have Anthony on testing code and we're going to talk about that. Oh, that's going to be great. I've also realized that doing this podcast with you, it's pretty easy to do it once a week because you're waiting for me to do it once a week. So I'm trying to rope in some other people

and Anthony's one of the people that agreed to try to do test and code more regularly to try to get more of those out. Oh, yeah, that's awesome. Yeah, that's really cool. How about you, Michael? Well, if we want to give shout-outs to Anthony, I just had him on TalkPython as well

about his security article. He's really killing it. Him and the RealPython guys, they're definitely cranking out the content. That's great. But the thing that I guess is the biggest news that has me getting up early and staying up late is i'm working on a new about 10 hour course for data-driven apps in pyramid with sql alchemy and like production migrations and all sorts of stuff that is almost done it should be out next week

i'm just finalizing a lot of the videos this week and it'll be rolling. Nice. So that's what I'm up to. Bob, how about you? You want to tell people about PyBytes real quick, what that is? Yeah, PyBytes started out as a blog with articles and we quickly moved into blog code challenges and that got quite some

traction. So we built out a code challenge platform which you can find on codechallenge.es. So code challenge is spelled as code challenges together, but ending in dot es. And you can log in with GitHub and code in the browser. So there are exercises with automated PyTest. Yes, Brian, it's PyTest. And you can code in a browser and verify those exercises. And yeah, the feedback is great. I mean, people are learning a lot of Python that way

because it's so practical. And so we're growing that. We have a Slack community behind it where a lot of people are joining and we have very good Pythonic discussions. So yeah, I'm excited about this. Yeah, right on. Yeah, it's a cool platform.

Happy to see you doing it. And thank you for being on the show, Brian. Thank you. Thank you, as always. I'm happy to hear you're doing more testing code as well. That's pretty sweet. Yeah. All right, catch you later.

Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.