Python Bytes - #199 Big news for a very small Python runtime

Episode Date: September 17, 2020

Topics covered in this episode: micropython updated respx: A utility for mocking out the Python HTTPX library GetPy - A Vectorized Python Dict/Set isort and black now play nice together easily Scientists rename human genes to stop Microsoft Excel from misreading them as dates Never Run ‘python’ In Your Downloads Folder Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/199

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 199, almost 200, Brian. Almost. Almost. Recorded September 9th, 2020. I'm Michael Kennedy. And I'm Brian Harkin. And this episode is brought to you by us. We'll tell you more about the things that we're doing and what we have to offer later.

But for now, I want to talk about not any huge announcement but a very very small announcement that it's also kind of huge very cool yeah so micro you might say you might say it's microscopic but it's a big deal so micropython has just had a new release and this is significant because the last release from micropython was de December 19th of last year. So it's been, what is that, almost 10 months since the last release. So if you care about MicroPython, this is like a pretty big deal. So version 1.13 is out. And one of the major things is it has improved async IO, async and await support. And it has cool Bluetooth improvements.

And it even comes with the PEP526, that walrus operator. Oh, nice. Yeah. So there's a lot of stuff going on here. They basically, MicroPython, as you can imagine, comes with a micro async IO module. I'm guessing the U is really a mu. U async IO module that's supposed to be more compatible with cpython's asyncio so the main idea is that the task object you can use a task object for each coroutine which allows you to have

like unbounded queues of work and stuff like that whereas previously i think you had to pre-allocate it for some reason i never tried to do asyncio on micropython but apparently it was a little bit wonky and so now it's more consistent. Another thing that I thought was interesting about this release, Brian, was they went through and auto-formatted all the code in the entire repository. And if you look at their repo, there is Python in MicroPython, but a lot like CPython, it's a lot of C as well. I think it's like 80% C or something.

And so they auto-formatted the Python code with black, and that probably won't surprise too many people, but they auto-formatted C with that community's equivalent of black, and get ready for the name, Uncrustify. They Uncrustified the C code. Okay. Do you know about this? No, actually, I'm writing it down because I'd like to check this out. Isn't that great? So you can basically configure how it formats the C code. Okay. Do you know about this? No, actually, I'm writing it down because I'd like to check this out.

Isn't that great? So you can basically configure how it formats your C code. And it's like black. It's like, you know, it just runs automatically and fixes your C code. And boy, if you thought Python code needed some fixing, I'll bet you the C code. And there's more variety in there, right? Yeah, I've seen some ugly C code in my day. Yeah, for sure.

For sure. Because with like the curly braces and semicolonsons you have a lot more flexibility in how you format stuff whereas python forces some of that structure upon you all right they also added blue kitchen apparently blue kitchen is a bluetooth stack and so they added a micro mu bluetooth module as an alternative to the nimble stack as well and there's a unix port and you can build the unix port with the bluetooth support for either of these bindings that's pretty cool that comes with new events so if you're doing like bluetooth discovery and things like that you can like scan for bluetooth stuff more easily it also came with a big memory leak fix for arm chips which is big in the small space ironically so apparently uh there was certain

memory that was not reclaimed by the garbage collector and other types of things so if you've you know and at the same time on these very small devices you care a lot about not wasting memory right it's not like you got 32 gigs like my laptop does so like whatever right it's going to matter if you lose memory on like a little micro python chip and also they set it up so you can run parallel tests against multiple micro python targets so i'm guessing like different chips and different devices you can just say go test them run it synchronize across two or more micropython targets i don't know if that's to speed it up or to like test the whole thing on different platforms in parallel or whatever but it sounds like a good improvement and it does come with a few

breaking changes so people can check that out anyway if you care about micropython and i'm guessing this may make its way over to circuit python as well at some point knowing that those projects are really close. So anyway, this looks really good if you're doing big stuff on small things. Yeah, it's good to see this going forward. And thanks to Matt Trentini for sharing this and bringing it to our attention. I wish I was better at remembering names, but somebody I had on testing code brought up Respex.

Or Respix? I should stop. RESPX. It's a utility I should stop. R-E-S-P-X. It's a utility for mocking out the Python HTTPX library. So somebody brought that up and I had never heard of it before, so I figured we'd bring it up here. So with requests, if you want to mock it, there's a cool library called Responses.

Cleverly named, but hard to search for. And then with HTTPX, if you want to mock it, there is Respex. I'm going to name it ResponseX, but the other part, you just don't write it. It's like the opposite of a silent E. It's not written. Maybe. Sorry, keep going.

It sounds like a great idea. Well, interestingly enough, when you write the two imports out, they're the same number of characters. So maybe that's part of the reason why they did that. Anyway, the quick start shows it's really quick. It's really just a few lines of code to mock out a request response call. You can do things that you would expect, like assert that something was actually called

and what the status code was and things like things that you would expect like an assert that something was actually called and what the status code was and things like that. But you can also have custom content and you can return JSON content of course. There's some examples. What I like is the documentation is pretty nice. There's some examples on how to use it with both

PyTest and UnitTest including how to set up like a mocked API fixture for PyTest so you can have multiple tests using the same mocked endpoint. And there's a bunch of nice things about it. Like, for instance, if you want to mock out a whole bunch of different URLs, you can use some regular expressions to set what URLs you want to catch with that. And then also the content that you return from your mocked object, your mocked request, is you can have a callback that can generate that on the fly

based on what you pass in it. If you have a HTTPX-based application, you definitely need to know about this for testing it. Yeah, absolutely. And one of the things that's challenging about testing the HTTPX is the async stuff, right? And so this is a really cool way to get in there and just make it work, right? Just put a decorator there and it just mocks it out for you like you would expect. Looks very clean. Yeah, pretty easy. And there's

a lot of magic going on under the hood to make that a clean interface but I think they did a good job. Yeah. So, HTTPX comes from sort of an async derivative of requests, which I think is a really fantastic library. And I've done some cool stuff with HTTPX. So, yeah, I'm a fan of this. This is great. Now, before we move on, let's talk about a couple of things that we got going on that people can check out that will definitely

help support the show, and they might find interesting. I know that you've got some updates for your book, is that right? Yeah, the Python Testing with PyTest book, just a minor, the book itself has a minor update, so there's, it wasn't a big enough change to change the hardback

version, but the ebook will be updated. I don't think the hardback is going to be updated. But it's a couple lines of code in a couple of code examples in Chapter 5, so it's a very minor thing. But the big difference is the code download. So we've updated the code download. So even if you're working with the hardback book or using the old version and don't update the e-book, please re-download the code. It'll make your learning experience better

and the main change is that I've pinned the dependencies in the target project or the example project so that everything works better. There was a tinydb is a database that I used in the project and it had some incompatible changes and instead of trying to update everything to use the new the new tiny db i just pinned it to an old version yeah that makes sense i mean it's not like you're trying to teach people tiny db you're like here's a dependency we're trying to get around to something yeah so it's that's not important to the the content of teaching people how to write tests yeah that's the problem of creating content the world

moves on and they're not always compatible with what you did how about uh yeah over at talk python we've got a whole bunch of stuff coming and so what i'd like to encourage people to do is just to go to talkbython.fm or training.talkbython.fm and right there you can, at the front of the training site, you can just enter your email so you get a bunch of announcements. Because I believe we are working on five or six courses right now are under active development.

We've got all sorts of great stuff. And while they're going through it, just sign up to get notified when those come out. Very cool. Indeed, indeed. So a while ago I went on this Twitter journey, let's say, I don't know how to really explain it was like, I posted a quick question that led to a ton of feedback. And then, wow, so many pieces of information and ideas and variations were sent around what I was trying

to say, like, I'd like a dictionary that contains objects that i can access say with different keys like i'd like to access by put a bunch of users and access them by id but maybe also by email or by city and that would come with a bunch so it was it was like this sort of exploration there and one of the recommendations that came over sort of around that is like hey you should check out this thing called get pi okay i don't know where the name comes from but it doesn't tell a lot about what it is but it's a vectorized python dictionary and set implementation and vectorized as in it matches up perfectly with numpy and so pandas and all the things that are built upon numpy to plug straight into them so imagine i want to have a dictionary that has data in NumPy, but lets me treat it like a regular Python dictionary or a set and things like that. So that's what this is. It basically

brings a super high performance Python dictionary and set implementation that automatically integrates into the Python scientific ecosystem, which I is pretty cool yeah yeah and it's built upon this thing called parallel hash map so parallel hash map is apparently the current state-of-the-art unordered map set with minimal memory overhead and super fast runtime so like a c binding and so this is just a python wrapper on top of this c library that's a super fast dictionary and set. So that's pretty awesome. And here's another one for you, Brian, for your C++ adventures. The integration between GitPy and Parallel Hash Map is this thing called PyBind11. So PyBind11 is a compatibility layer between C++ 11 and Python.

So if you want to write like modern C++ and then plug it in easily to Python, here you go. Nice. Yeah. So that's pretty cool. Okay. There's two classes, GP, so getpy, gp.dict and gp.set. And they're designed to be basically similar to the standard dictionaries and sets from Python. But there's a

few differences so check out the docs and then i threw in a quick little example here that has two numpy arrays and you can say here are the keys here are the values put them together and then you can just access different values and it's also typed which i think is kind of interesting right like it has like an unsigned 8-bit integer or something like that much like numpy is but it's more like a python array where you specify the numerical type than it is just an unbounded list right so there's some interesting stuff going on here does it solve your problem that you were looking for no not at all but it's very interesting it's still neat yeah yeah it's still neat speaking of neat we already talked about black once But it's very interesting. It's still neat. Yeah, it's still neat. Speaking of neat, we already talked about Black

once, but it's pretty neat. Let's talk about it again. Yeah, this was sent in by John Hagen and he mentioned that, I mean, I'm pretty sure we've mentioned iSort before. I know we've mentioned Black. But a lot of projects use both. So iSort will sort your imports

so you don't have to. And then just so it's consistent, they're consistent and they're alphabetically sorted and then there's some other things. Right, and there's PEP8 recommendations about that, I believe, about grouping stuff that comes from the library, from standard library versus externals and whatnot. Yeah, and so

you don't have to think about it. You can just use iSort. Black also is becoming more and more popular, I think, or it's just constantly very popular reform it's all of your code but it includes the import statements as well so there was this issue that some projects wanted to use both isort and black and there is and they kind of fight with each other right out of the box if you run isort you'll have one answer, and Black will do something different in some cases. So what do you do?

Well, Black had a configuration page, and we'll link to this in the show notes, that has documents what settings you have to set for iSort so that it's compatible with Black. They also do that for Flake 8 and Pylint, which is nice. But Flake 8 and Pylint, which is nice. But Flake 8 and Pylint are not that complicated. It's the iSort that there's like six settings you have to change. But iSort 5 just came out. And iSort 5 has what they call profiles. And so if you run iSort with profile black,

black is one of the built-in profiles, It will sort the imports such that they're compatible with black. Nice. So now if you use iSort with the black profile and black, also they won't fight with each other anymore. That's very cool. Black also came out with some new changes. So if you're interested in black, check those out. They're not huge changes, but some minor fixes.

The profile feature is pretty cool, and even if you don't care about Black, I think you should check it out. They do, apparently, Django, PyCharm, Google, OpenStack, Plone, Adders, and Hug are other profiles that are included. And also, they're just good examples so that you can look at how different projects are configuring their ISORT. You can configure yours if you want. Yeah, I really like

it. That's super cool. And it is nice to have the top of your files nice and clean and organized in some certain way. It's interesting. There's a lot of people do different linters or code reformatters, but this combination of iSort plus Black is something that's becoming more and more common in a lot of projects. Yeah, yeah. Very cool. So I have something interesting for you. If you were to think of the influence of Microsoft Excel

relative to the influence of all of human genetics and the genome science of the world. And they got into a fight. Who would win? Like a superhero. These seem like apples and oranges. I have no idea. Well, they did get into a fight, and Excel won. So here's the story.

There's an article that was sent over by Chris Moffitt. He's the guy who wrote the Moving from Excel to Python and Pandas course over at DocPython. So we did a bunch of research into like all these funny things and weird things around Excel. So he sent this one over and I thought I'd cover it cause it's fun. So on the verge, there's an article that talks about how scientists have renamed human genes to

stop Excel from misreading them as a date. Okay. Is that crazy? Yeah. Yeah. So you think it was just like formatting and stuff like that so there's like tens of thousands of genes in the human genome and each gene is given a name and a numerical code and then you use to talk about it so like this one controls like what

color of hair you have or whatever right so over the past year or so 27 human gene genes have been renamed all because excel kept rereading them as symbols they're symbols of states okay well the examples are important yeah so like march one no spaces march one is a one of the genes right but it gets converted to 03 slash 01 slash the year, or if you're in Europe, 01 slash 03 slash the year, right? Either way, that's not what you wanted. So March 1 actually stands for membrane-associated ring CH type finger 1, which of course is the 1st of March.

Now it sounds funny, like we're making fun of it, like, ah, whatever. There was a study in 2016 that examined the genetic data shared alongside 3,600 published papers and found that one fifth of them, which, what is that? That's like 700 papers were affected by Excel errors.

Oh man, man. Yeah. Isn't that messed up? So there's a scientific body in charge of naming genes called Hugo Gene Nomenclature Committee. Who knew? There's a committee for naming genes. But apparently. So they published new guidelines for naming genes.

And they said, you have to consider what happens if you type into Excel. If it gets reformatted, that's not okay. Got to pick a new name. So they put like weird suffixes and stuff on these things to make them work. Like March 1 is not like March 1 N1 or something like that. Anyway, so why do I bring this up on the Python show? You know, so much of this work that people are doing there can totally be solved by pandas and Jupyteriter notebook and stuff and so i you know the

guy who was quoted in there was like hey i made these mistakes when i was in grad school because all i knew how to use was excel so here's a bit of a an example along with a plea to help folks who are overusing excel to take a step forward and use, you know, something like pandas and Jupiter. And, you know, you're going to be able to do a lot more cool processing anyway, which I think is great. Yeah. Yeah. Also, one other thing, if you think the geneticists have a exclusive right to these mistakes,

there's a really cool article blog post over on oracle's blog actually called the 10 costliest spreadsheet boo-boos in history and they're really hilarious and like thank god this didn't happen to me well and just like awesome stock photo that they have for this too yeah it's perfect it's so bad yeah it's like perfect it could just come out of excel yeah so some of them are fairly mundane but others like mi5 the british intelligence agency bugged over a thousand wrong phone numbers because the zero zero zero in the last three digits of the number got you know misstated oh no yeah or uh eastman kodak was forced to reinstate financial results for two quarters by from uh yeah 2 million and 13 million respectively due to a spreadsheet mistake

yeah there's just all these crazy examples of stuff just going wrong like oh sorry that was a billion dollar mistake because we were off by you know some random thing anyway there's a bunch of errors like this and it's really interesting to think how you might use the Python tools to not have these errors. I think Excel has too much influence over the world. It does. But if you were going to use Python and you had a Jupyter

notebook, would you run it in your downloads folder? No. No, definitely not. This is another interesting and shocking, wherever you think about it before, but Glyph has written an article called Never Run Python in Your Downloads Folder.

It's not just about that. Okay, so there's your advice. Don't run Python in your downloads folder. But I think it's a good article to spread around and read because it's a nice short tutorial on how syspath works, how it's populated. So Python has this thing called syspath that is, that's where it looks up.

So if you say import something or mostly that, where it's importing things, also the dash M or something, if you say dash M and then some module, where does it find that? And it finds it in lots of places. One is the normal system include directory or the system packages.

But there's other places too. And one of the places where Python uses is the location you're in when you run Python. So a little example he talks about, a lot of people are now using, instead of running pip directly they run python-mpip which i also am including my that in my advice usually because i'm tired of trying to fix people's errors when their python and their pip are pointing to two different places. That's frustrating. So what happens if you run python-mpip install something, some wheel in your downloads directory? Well, normally everything's fine, except if there's a malicious fake pip.py in your downloads directory that happens to have gotten there

because somebody wrote some malicious JavaScript code, stick it it there it's possible i don't know if it would happen but it's possible it's more likely now yeah especially now thanks thanks cliff but then it's going to run that instead of the pip that you expect it to run which is is bothersome so this is hidden there's some extra advice in here i encourage everybody to read the whole article. Understanding how the Python path variable works as well because sometimes other applications will, if the installer of an application can change things and change your Python path, it shouldn't, but sometimes they'll do that out of convenience.

So occasionally look at your Python path and make sure that there's nothing weird in there and maybe contact application developers or if they're doing something odd. If you are mucking with Python path, the recommendation is put absolute paths. Don't put anything relative in there. You want to be able to have complete control over that.

The problem with the downloads folder is not that the downloads folder has weird permissions or higher permissions or anything of that nature it's just that web virus maliciousness might it's most likely to drop the payload there and the python path plus that operation or that aspect is what is likely to lead to trouble yeah right like you wouldn't say don't run it in your documents folder unless you copy virus python files in there yeah and the example was real of like sometimes like let's say you're behind a firewall or something like that and pip install just doesn't work and you haven't figured out how to do proxies yet and you really just need some package you might just download the wheel somewhere okay

you might have that and the wheel might be fine but put it somewhere else don't leave it in your downloads directory put it someplace else and please use virtual environments that'll help things as well yeah absolutely good advice and definitely uh another thing to be aware of that internet it's a scary place yes but we love it's also a good place it goes both ways because we do love it that's it for our main topics but i do want to share two quick things and they both have to do with the internet number one is we just passed five million downloads brian wow that's awesome yeah and uh we're in the top 150 or so of all tech podcasts in the world. So I just want to say thank you, everyone, that's helped to make that happen because that's kind of a milestone.

So that's really cool. Yes, thank you. Indeed. Also, I recently finally broke down and got a Wi-Fi mesh router. Routers? Can you say singular router? It's got to be plural, right? So one of those routers where it has a bunch of different stations

and they all work together. But because of the way it works, you don't actually have to connect to different ones at different places. It's just all one Wi-Fi. And man, I got this Linksys Wi-Fi 6 mesh router, which is quite pricey. But man, it is so, so awesome. So if people are like suffering from being home all the time that i gosh i gotta get better internet my internet was actually pretty good here but we started getting a ton of radio interference from different sources and it would degrade it and stuff so i have my recording area office above my garage it's like little studio apartment thing we built above the garage which is a separate building across from the house and over here if i go speed test i get 400 megabit measured off my wi-fi now that is solid okay so how far away is your are the mesh nodes it's probably 50 feet but it goes through like several outside walls through like a couple inside wall it's like there's layers it's got

to go through but now with the mesh i could put one of the nodes much closer to the wall that i'm close to that spans like that gap between houses okay and how are they talking to each other they have a like their own channel and they do some kind of like back channel wi-fi six thing so anyway super good recommendation yes how about you any extra stuff you want to share with the world no just i'm sort of sick of covid and fires and all that sort of stuff yeah man it's it's looking a little smoky outside and uh yeah the whole west coast and my sympathies go out to the folks in northern california i just had a meeting with someone and oh my gosh is it smoky there it looked like night in the daytime in the background i could see see their window, and it was dark,

as if they were in a different time zone. But they weren't. Not good. So hopefully that all ends soon. But before we end this episode, I got a question, Brian, for you. Are you a real programmer? Like a real one, like a hardcore? I mean, you do see Plus Plus, so that puts you a little closer.

Oh, my God. Imposter syndrome flares every time I get that question. So I don't think so. Let's go to XKCD and answer it. Okay. So XKCD has this cartoon called Real Programmers, and it starts with two people debating about what kind of editor they're using. It says there's somebody working, and they're using Nano, the editor.

Come on, Nano? For real, real programmers use Emacs. And then Brian walks in. Hey, real programmers use Vim. Come on. Oh, yeah, well, real programmers use Ed. No, no, real programmers use Cat and just stick it on the end.

Real programmers use a magnetized needle and a steady hand. Excuse me, but real programmers use butterflies. Everyone turns and looks at the person. They open their hands and they let the delicate wings flap once. The disturbance ripples outward, changing the flow of the eddy currents

in the upper atmosphere, which acts as a lens that deflects incoming cosmic rays, focuses them to strike the derived platter and flip the desired bit. Nice. Of course, there's an Emacs command to do that. Oh, yeah? Yeah.

Command X, Command M, Command Butterfly. Nice. Yeah, yeah. C-M-C-M, C-X-C-M-C-M Butterfly. Indeed. Damn it indeed damn it emax they say does anybody use emax yeah well i don't know but i'm not a real programmer because i don't use butterflies i'm sorry no i don't use butterflies okay well we're in the joke section i gotta share a terrible joke that my wife came up with okay 99 bottles of beer on the wall, 99 bottles of beer, take one down, pass it around. Now everybody's infected. Morons. Hey, that sounds like college here in the US. I

think that's what they're doing from what I can tell from all the news I'm watching. Oh my gosh. Jeez, sorry. Yeah. These jokes are the 2020 jokes. Oh yeah. So this won't make sense in the future. Yeah, yeah, yeah. So in the future, when you speak to a historian and you say, oh, you're a historian, what do you study? Oh, I actually study the year 2020. Like, that won't be enough. You have to say, well, which part do you study? Like the beginning?

Do you study the financial crash? Do you study this crazy election? Do you study like the beginning do you study the financial crash you study this crazy election do you study the pandemic like you got to be like that year is not enough to say what your specialty is you got to pick one pick a quarter yeah it's a crazy time right i study the flyer the fires oh boy anyway well good talking with you as always you as well catch you later thank you for listening to python bites follow the show on twitter via at python bites that's python bites Well, good talking with you as always. You as well. Catch you later.