Python Bytes - #132 Algorithms as objects

Episode Date: May 30, 2019

Topics covered in this episode: History of CircuitPython Algorithms as objects pico-pytest An Introduction to Cython, the Secret Python Extension with Superpowers Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/132

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 132, recorded May 22nd, 2019. I'm Michael Kennedy. And I'm Brian Ocken. And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. More about that later. Brian, how you been?

I'm good, but I'm giggling right now because I've been looking up jokes, but we'll get to those later. Yeah, I always look forward to the jokes, And I think a lot of people out there seem to appreciate them. So yeah, we won't disappoint this time. We got more lined up as always. Yeah. Okay. I mean, like in the past, you maybe used to have to come up with these, but we have the internet. Yeah, I know. It's beautiful. It's easy to be a dad now.

That's right. All right. You want to kick us off with a little embedded Python? Yeah, I was really excited to watch this. So I first one I've got up is the history of CircuitPython. And this was actually on the PSF blog, and I think that a Jesse Jerry Davis put it up. But anyway, it's about the history of CircuitPython and Adafruit and Scott Showcroft, which I met Scott at PyCon, and I knew he was involved with CircuitPython, but I guess I didn't realize he's the CircuitPython guy. So what happened was, and I always kind of wondered about this, like the relationship between MicroPython and CircuitPython.

Right, I knew about MicroPython being for embedded devices, and I'm like, oh, there's now CircuitPython. Like, are they friends? Are they frenemies? Like, one is a specialization of the other. What are they? So what happened was Adafruit thought MicroPython was a pretty cool thing and hired Scott to port it to their, I guess, their SAMD21 chip that's used on many of the boards. So he did it. He's working on it.

And at first it was just a fork fork but now it's a fairly big changed fork there's a lot of differences but scott said it's a friendly fork so they he works closely with the micropython people too and they share code back and forth so there's no no hard feelings there but it's focused on the adafruit boards, but it's also, MicroPython is also focused really just on embedded stuff, whereas CircuitPython is focused on beginners. And one of Scott's quotes is, our goal is to focus on the first five minutes someone has ever coded. That's a strong, lofty goal. But watching some of their demos with the Circuit python it isn't surprising to me you know doing they just hook it up and a few minutes later they've got stuff running it's kind of incredible

yeah that's really awesome and that is a very uh i don't know impressive goal to say we're focused on the first five to ten minutes of somebody programming because yeah that's that's an early point in the life cycle but i I guess you got, you know, it's got to start somewhere. And I do think these embedded devices really do bring like some realism to programming for people who may be programming just in the more theoretical way. Didn't necessarily connect. So it's great. One of the things I didn't know about was with one of the things that CircuitPython does is it outputs like all the print statements go automatically to the serial output so if you could if you hook up your serial output to a connected display you don't even have to have any other device you can just see your prints and stuff so that's kind of neat that's awesome and

Nina Zakarenko actually showed how to use this and like I think it was Visual Studio Code she had that serial output connected so just within Visual Studio Code. She had that serial output connected. So just within Visual Studio Code, like the devices output would just like stream at the bottom. That's so cool. And then we're also including a link to the release notes for CircuitPython 4.0.0. So it's a new major number. So there's some breaking, a couple of breaking things,

but it looks kind of neat. Yeah, that's super. And apparently they're not on the zero-ver philosophy. Yeah, no, that's good. Maybe you read it reverse for zero-ver, like 004. I don't know. Anyway, if you must. But no, that's cool.

It's great that there's a new version of that out. I really think they're doing a lot of cool stuff. The Adafruit folks are doing a lot of cool things, and some of those devices I think were given out at pycon this year as well which is pretty neat yeah so uh we've talked about python being popular right yes yeah absolutely so this next article that i want to cover is by dice.com doing a little bit of analysis of the tob index so we've talked about tob before this is a programming index. It talks about how popular languages are. And one of the areas of major growth in Python has to do with

the data science space and just the scientific computing in general. So this article that I want to highlight, the title is, Are Risks Python Swallowing It whole, according to TOV. So that's pretty interesting. There used to be kind of a debate. Do I use R? Do I use Python? Well, it looks like there's a lot of consolidation, and all this growth in Python is somewhat a zero-sum game for the other languages. So it might be interesting to check out if you're thinking,

should I learn R? Should I learn Python? R has a lot of interesting advantages over Python, but it seems like Python is kind of winning the day. And so R has tumbled out of the top 20 languages and now is somewhere below that. It's very popular with a lot of people, but I didn't even hear about it until like maybe a year ago. For sure. Yeah, it's definitely focused on a more of a niche space so the article speculates sort of not super effectively why they think that is like they ask the question like well it seems that python is winning in data science but why and so i thought maybe i'd throw my own commentary in here and uh there's this idea like i really like to talk about with Python being a full-spectrum programming language.

And what I mean by that is you can get started with the absolute minimum amount of effort. There's no compilers, no linkers, there's no header files. There's a single file, and you can say, print, parenthesis, quote, hello world, right? And that's it. You don't need to think about generators or all the other stuff that like rich programming languages have like think c-sharp java c++ right

you've got to like do all this ceremony to get going but the value of those other languages is like you can go build you know video games or operating systems or you know know, whatever, right? And you can do much of that kind of stuff, not operating systems necessarily, but like you can build real apps, professional apps in Python. And it kind of scales from this ultra beginner part where you can partially understand the language and be effective all the way up to using meta classes and decorators and generators to build, I don't know, Instagram or whatever. And I think that that's actually why Python is winning in this space because the data scientists and scientific computing folks are coming in and they're like,

I don't want to be like a programmer. I want to use programming to solve my problem. And Python lets them start that super easy path but grow into where they eventually find themselves. One of the things also is that people that have to reach towards R or some other programming language, that's not their only software issue that they may have. And if you can solve it with Python, you learn Python, and then you can solve other automation things within your workflow as well. Whereas R is very focused.

I don't know how much you can do with it outside of data science. That's for sure. That's definitely part of the story. Speaking of stories, you've got a little history lesson for us, right? I've had this on the list for a while, but it took me, I just went back and read it yesterday. It's an article by, I think it's Eamon L. Emery called the Missing Introduction to Containerization. And so container systems like Docker and stuff like that, lots of people use them. And I do want to ramp up learning how to use them more.

I'm kind of one of those people that need a mental model of how all this stuff works. And this is it. So it starts with a 1979 release of something called, I don't even know how to pronounce this. Chroot? C-H-R-O-O-T? Chroot. Chroot. pronounce this, Chroot? C-H-R-O-O-T? Chroot. Chroot. Chroot.

Anyway, Chroot Jail, which was a way to isolate a root process and its children from the rest of the OS. But there was a bunch of problems with, it wasn't really meant for security. And I'm not going to read the whole thing, but it builds up. Because of this is open source free bsd improved on it then linux v server and then oracle solaris had some stuff open vz and then google chimed in

with something called c groups and then a group called linux containers did lxc and then you know it kind of built up cloud foundry got got involved. And then in 2013, Docker. And then Google's still doing more stuff with making things easier. But all of this stuff builds up on itself. And I really like that bit of a history lesson of how things built on itself. And then it jumps into talking about the differences, all the different terms you'll find, like a system virtual machine and a process virtual machine and VPS and what all those things mean. Also, the kind of the difference between an operating system container system and an app container. And Docker is a little bit of a mix of both. Also, it's kind of containers and platforms. Just why there's so many things around is kind of described in this. And

then that's only like halfway through the article. The rest of the article jumps into, I think it's creating a container system from scratch based on some of these lib container things. And yeah, I don't want to do that. So, but the first half of it, definitely recommend. It's a good article. Yeah, the history here is really interesting. Like, I guess when I first learned about Docker, I thought, oh, Docker invented containers. But absolutely not.

They were building on LXC and all these other things. And eventually they sort of moved off that. But yeah, they were just making containers more accessible and easier for folks and popularizing it. I think these containers are super powerful and super interesting. To me, I don't know. Do you feel like they're complicated? It seems like the system is made to not be, but I know there's a lot of stuff behind the scenes going on. And I, you know, to be honest, I haven't played with them much or used them. I don't need them for my normal job, but I would like to learn more. How about you? I definitely like them. I'm not doing anything with them right now.

And I've thought about how using containers might make sense. But at the same time, you have lightweight VMs I just fire up, and they're dedicated to a single purpose. So I don't know. It's always a bit of a trade-off of adding more complexity in one place. I feel like it's a little bit of a whack-a-mole problem. I can have more simplicity in some places, but I feel like it's a little bit of a whack-a-mole problem. I can have more simplicity in some places,

but I've like pushed it to another. So for example, right, people talk about containers often being a great way to simplify development environments for junior developers, okay? Yeah. Right, so like I'm going to work in a new company. Our app infrastructure is super complicated. So what they do is they say, well,

here's like three containers, one runs a database, one runs the web front end, one runs the caching tier, whatever the heck it is. And I'm going to just run those. And I'll just develop in that regardless of what my machine setup like, well, that's on one hand, like simpler, I say maybe Docker compose up, and boom, my little environments working. But now I've got to figure out how does my editor do, like, remote debugging of Docker containers? And how do I, like, step across calls between containers in my debugger and all this other stuff? It's like, okay, so I've moved, like, setup complexity to, like, editor complexity or other stuff, right? So I don't know. It's super interesting.

I certainly see them being really valuable for like zero downtime deployments and other kinds of stuff. But yeah, it's interesting. In my wacky corner of the universe, the place where I probably would use them is we often want to spin up a new build server or something like that. And we have it written down

of how to build a build server. Even though it's on a virtual machine, we got to install a bunch of stuff and it's a half a day to get it up and running. And having that just saved off. Yeah, that's a perfect example of Docker, right? Because you can build the container images to just do that setup. And you just say, you know, Docker build and boom, you're ready. Yeah, yeah, I like that. Speaking of containers, and all those good things. Let's talk really quick about DigitalOcean, some of the stuff they have to offer.

So DigitalOcean just this week put their Kubernetes cluster into general availability and added some cool new features. So if you're using Docker, you've got to run it somewhere. And usually just running it directly is not what you want. You want to run it somewhere where you can upgrade versions and do zero downtime deployments and multi-cont multi container stuff. So Kubernetes is a great place for that. So check that out over at DigitalOcean. Just visit Python by set of them slash DigitalOcean and get $50 credit for new users. Yeah, so all

sorts of cool stuff over there, you can provision your servers and optimize it and get it going, right? Really, really nice. And one of the things they just added is free integrated monitoring that'll provide like insight across your clusters and your containers and stuff. So yeah, check them out. Pythonbytes.fm slash DigitalOcean. Super, super cool. So this next thing I want to cover, Brian, touches on something I've been a fan of for a long time, and that's design patterns.

Okay. So design patterns. We talked a little bit about that previously, one of the shows just recently. But this time I want to focus on maybe a topic that is reversed from a lot of the advice that you hear. A lot of times you hear people say, hey, Python developers, stop using classes and stop using objects. Just write modules and functions, right? Yeah, sometimes. Yeah, sometimes.

And that's totally reasonable. A lot of times people come from C Sharp or Java, where it's everything is an object, everything has to be in a class. And so they think, well, I have to do that in Python. And if you've got a static variable in a class, that's, you know, just kind of like a module level variable in a function, like there's no real value to breaking that apart. However, a lot of times there are. So there's a cool article that walks you through in super in-depth, focused on sort of data science side of this object, no object debate called algorithms as objects. So as usually we think of like algorithms as a single function with an input and an output and algorithm textbooks reinforce this notion, right? Like

everything fits nice onto a single page. But in reality, what you end up with are these like giant monolithic functions that are like full of details with lots of cyclometric complexity and passing lots of data around and all sorts of stuff. And it's not nearly as nice. You end up with these functions that like lack readability. And because of that, they lack maintainability. Nobody wants to touch it because it's probably important. It's an algorithm, right? And if it's wrong, it's probably hard to tell if it's wrong. So you don't want to break it. It's just kind of scary, right? So they talk about taking these algorithms and turning them into functions. And one of the ideas I really like that they're

starting with is like, well, should I do this or not? How do I know? So they talk about this idea of code smells. And the idea of code smells comes from Martin Fowler way back in the day, like 1999 or something like that, when he wrote his refactoring book. And these are aspects of code that are not necessarily broken, but they're just like a little bit off. Have you heard this idea, Brian? Yeah, definitely. It kind of makes you wrinkle up your nose. You're like, ew, something's not good here.

But you know, the code works, right? Like it would pass a test. So it's fine. So the code smells, they say, that you should be on the lookout for here are, number one, the function, the algorithm, the one giant function, it's too long or it's too deeply nested.

Like we spoke about guarding clauses last time. Does it have banner comments? Like a huge comment at the top that describes what it does, how it works, the special cases, right? I often say that code comments are deodorant for bad code. So you should just write good code and not do that. So like that's an example. Helper functions that are maybe nested closures inside there,

some weird thing like that, or maybe actual helper functions, that are maybe nested closures inside there, some weird thing like that, or maybe actual helper functions, but they're only used within this larger function, passing a lot of states. All these things are sort of indicators that maybe an object would be much better for wrapping up your algorithm. So it's really, I think, got a lot of concrete advice here.

This actually looks pretty interesting. It's super interesting, and it's full of examples. Like it's a really long article with lots of concrete examples of here's an algorithm. We did it this way. We refactor that to an object and look how much more understandable it is and how much simpler it is. So I definitely like if this idea resonates with you, definitely check it out. And the guy who wrote it said, Hey, when I present this idea to my colleagues or other folks at first, they're like, no, we shouldn't be using classes. Just functions will do. This is, you know, he encounters some pushback, but rarely does he encounter like prolonged pushback after people see the results.

They're like, no, actually, this is pretty killer. I'm just chuckling. And one of the topics he has here is I've got 99 problems. Give me two more. Yeah, it's a good article. Definitely check it out if people are interested. It's very concrete and helpful. So, Brian, I know you're into testing, and I know you're a fan of PyTest.

What are you into, like really small versions of PyTest? Or what is this? I got into this because I've been using Python for testing purposes for since like 2002 or something like that. There were often custom-made Python frameworks or testing frameworks within our company. And then around 2010, I did the same thing and wrote my own. But it took me a while to get it down. I was arrogant and thought, oh, this would be trivial because the basic algorithm isn't really doing much. But Oliver Bestwalter, there was a comment somewhere on Twitter

that the core of PyTest could be written in a couple dozen lines of code. And people were like, what, really? And so, of course, he's a brilliant person. He went out and just did it. So he released a thing called PicoPyTest. It doesn't have assert rewriting and it doesn't have like fixtures or any of the fun stuff, but it's a generally usable test framework that you could run some tests with, written in 25 lines of code.

Dang. And you know, like the first five are just import statements or space and one of them is a method. So I think you could probably get it down to 19 if you were willing to go a little crazy on it. That's wild. So some of the things that it uses are like import lib. Some of these things are clearly put in the language for tools like PyTest and stuff, like import libs, spec from file location,

and module from spec. I don't know what those things do, but it looks like it's a way to sort of gradually load a module and then run parts of it because that's what this code does. One of the things I like about this, one of the reasons why I'm highlighting it is a lot of people think of test frameworks as this thing. They don't really know it's a black box that does stuff. And I think this is a good way to highlight it's not, Python is very flexible and the heart of a test framework isn't that complicated

it's going out and finding files that match like a certain pattern test underscore and finding some functions inside those to call and then just catching exceptions and logging failures it's not that complicated so it's pretty cool that's super cool and I think anybody who

cares about testing or maybe is being introduced to testing should read those 25 lines, not necessarily for understanding, but just to get the sense of like, this is what happens when you run tests against your code. It finds your model, it loads up the functions, it calls it, it does this basic thing and that's it. It's really nice. Yeah. Very, very cool. That is definitely, I don't know what the atomic unit of a test framework is, but that's nearly it. He popped this out in like two days and it took me like two months to write my first version of my test framework. So awesome. Yeah. Super cool. Super cool. All right. Last one I want to cover has to do with Cython, not CPython, but Cython. It's an article called An Introduction to Cython,

The Secret Python Extension with Superpowers. And who wouldn't want an extension with superpowers? Yeah, exactly. They make the statement that they think Cython is one of the best kept secrets of Python. And I agree. Cython will take almost arbitrary Python code and turn it into C. That right there is pretty impressive. If you've a, like, you've got some program and you find out like, oh, this part of, I don't know, some inner loop within an inner loop, you know, within another nested loop or something like that's a little bit too slow. You could probably write like a function in Cython that does that inner loop and make it

way, way faster. Super cool. That's what we hear a lot about Python is you can take some slow parts in and then rewrite it in C if you need to. But you don't have to with Cython, right? Exactly. You don't have, that's the thing. You don't have to rewrite. You could rewrite it in C or in Rust or whatever, or you could just call Cython against it and make it, you know, it basically transpiles the python into c and then

compiles the c over to machine instructions now depending on how you write your code it may still interact with the python interpreter and the python gill or it might not and that could actually significantly determine the performance benefits you get right so you can it'll work with like an untyped item but then it does it as a, I think as a PI object pointer. Whereas if you tell it, this is an actual integer, it'll work with it as like a C int type of thing or something to that effect, right? So you get all sorts of benefits that'll or ways to overcome shortcomings of Python. Say for example, we talked about execution speed, but there's also a keyword in Cython that's called no-gill. So you can just create a context block in Python,

effectively say with no-gill colon, and then that stuff works without the gill in threads. And that's it. You're just guaranteeing that that stuff isn't going outside of it or anything. What the requirement for using that keyword is, and if it doesn't match these requirements, it's like a compiler error for the Cython compiler, is that you don't interact with any Python objects.

Okay. So you basically got to cast them into Cython native objects that can be represented in C, and then there's no reason for the gill because the purpose of the GIL is to interact with reference counting to make sure that that works in Python's object garbage collector. But if you're not working with Python objects, you don't need the GIL. So it's a really great way to free up speed and whatnot. They talk about some of the projects

written in Cython. So spaCy, the natural language processing, UV loop, which is a really fast async IO event loop, significant parts of Scikit-learn, NumPy, Pandas, all that kind of stuff. So, I think the big value, like you said, is like you could go rewrite it in C, or you could just use Cython and make your Python run somewhat like C.

Pretty cool. I actually want to play with this a little bit, because I'm glad you found this article. Yeah, the article is super interesting. One thing that I've noticed, I mean, I didn't read it. I didn't like super inspect every code example, but I didn't see them using Python's type annotations. Right. So they're using like the C def to define types and like of Cython, you can just say like value colon int equals something, you know, like the standard

Python way of saying what a type is for type checking, and that'll actually also be understood and used by Cython for like native types. Okay, that's a question I had was, can you use native Python types? The answer used to be no, but now the answer is yes.

So it's super cool. Alright, well, if you think you need your Python code to go a little faster, check out this article. Check out Cython. It's pretty awesome. Just a good reminder, I guess. We both have a few extras. How about you kick off that section? Sure. I didn't really want to highlight this for too long, but Henik wrote an article called The Price of the Hallway Track. It's just a reminder to everybody. We do hear about going to things like PyCon and other conferences and not actually going to the talks, but doing hallway stuff. He's pointing out that that's kind of lame for all the speakers that work really hard to do that. It's also, even if you intend to watch it later, it's disheartening for people to speak to an empty room or even a mostly empty room.

And he's just asking, you know, don't fill up your day with talks, but pick some, especially ones that from people that are lesser known people. And that sound interesting to you, at least go to a few talks. And I think that's a good advice. I'm going to try to do that next year. Yeah, cool. And what else? The second one is who put Python in my Windows 10 May 19 update by Steve Dower. We did talk about this in the previous episode, but it is officially out and

people are noticing it. Yeah, and Steve gave us a shout out to the Python Bytes episode we did together right there in the first paragraph. So thanks for that, Steve. This is massive. This is Windows 10 is shipping with python 3 and that version of python 3 is auto updating like how cool is that okay within a major version yeah but it ships within a little stub so you don't get python right away you get a little thing that

pops open if you type python it pops open in the app store to download it it doesn't ship with it because it doesn't need to and and still a lot need to. And still a lot of people don't need Python, but that makes it easier. I guess when I say ships with, I mean from a user's perspective. If you tell a user to go through a tutorial and you tell them to type Python 3 and they sit down and they type it, rather than getting error, it says, oh, you have to click this button for this line to work. They click the button and then it works. Yeah, exactly. I wouldn't ask for more. That's really cool yeah how about you got any extras yeah i got a also something super small a pico thing so matt trentini sent over a cool project called the tiny pico which is an esp

32 based board so a tiny little board that has first class support for micro python so brian if you click on that link if you check this thing out it's pretty wild it's um the project you can order it's pretty cheap like 26 dollars it's so small it's incredible it's like two-thirds the size so size so 60 of a double a battery i mean i don't like maybe you know like the middle part of your pinky or something i mean it's a really small board like both the width and the height that is crazy but listen to its specs 32-bit dual core processor full-on wi-fi, 8211 BGNN, Bluetooth, all sorts of stuff, like for $26 at that size. It's so cool.

So I'm going to give a shout out. MicroPython is pre-installed. That's cool. Isn't that super? So if people are looking at embedded stuff and little devices, things, this is really, really cool. Yeah, good for your next SpyCam project. Exactly.

I got another one from Automation Panda, who we met at PyCon. And this is Andy. And he wrote a cool PyCon 2019 reflections, which I thought if you haven't been to PyCon, check out this article. It's like kind of his diary of like what he did and his experiences. It's like his second PyCon he ever went to. So it was kind of a fresh take on PyCon, which is great. That's cool.

Yeah. Yeah, it was fun to meet him there. I just want to give a quick shout out to our Patreon page, which we don't do that enough. So if people want to support the show, obviously visiting the sponsors helps a lot. But you can help support with editing fees and other stuff by doing small contributions. So there's a link at the bottom of this episode to say, here's the Patreon page. So people can check that out. And Brian, thanks to you for putting that together. And one of the things people get is the show notes emailed directly to their inbox if they do that.

That is awesome. And then I just wanted to let people know I'm doing a free one hour webcast in a couple of weeks. And the title of the webcast probably more or less sums it up, but it's 10 tools and techniques Python Python Web Developers Should Explore. That looks interesting. Yeah, so it's all sorts of fun stuff. Like Docker is one of them. Vue.js is another. You know, Vue models and other sort of design patterns as well.

A lot of fun things that people haven't maybe seen there. Yeah. Let's let you kick it off with our joke section. What do you got for us? Oh, you're going to give it to me? I'll steal one of your jokes. Yeah, steal it. Okay. Okay. What do you call eight hobbits? A hobbite. Oh, that's terrible. It's so bad. It's so bad. It wrapped around to good. It like badness overflowed into the good level. What do you call it? All right. So, I got another one

for you. This one may also be bad. This is a little more on the science-y, math-y side, not quite programming, but it definitely has a computational bit to it. So you know Mandelbrot, Benoit B. Mandelbrot is his full name, the guy who came up with the Mandelbrot set and fractals and all that kind of stuff, right? Yeah. And one of the core principles of fractals is no matter how much you zoom into them, there's always more details. And oftentimes, there's super weird reasons that it repeats. You zoom way into a little branch of the Mandelbrot set. There's a baby Mandelbrot set. Yeah. OK?

Yeah. So the question is, for Benoit B. Mandelbrot, what is his middle name? What does the B stand for? Well, it stands for ben while be your bandle brat of course it does yes oh i love it all right well i guess we're gonna leave we gotta do the other one it's so great all right okay so two bites meet the first bite asks are you ill and the second bite replies no just feeling a bit off. Totally left shift on that one. Yeah, absolutely.

Beautiful. All right. Well, thanks as always for being here, Brian. It's a lot of fun to talk about these and share them with everyone. Yeah. Bye. Yeah, bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes

at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.