Python Bytes - #176 How python implements super long integers

Episode Date: April 7, 2020

Topics covered in this episode: * Quick chat about COVID 19* What the heck is pyproject.toml? Awesome Python Bytes Awesome List Publishing package distribution releases using GitHub Actions CI/CD workflows Rich text for terminals psutil: Cross-platform lib for process and system monitoring in Python How python implements super long integers Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/176

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 176, recorded April 1st, 2020. April fools. So, I'm Brian Ocken. I'm Michael Kennedy. And this episode is brought to you by DigitalOcean. We'll talk more about them later. But, yeah, let's just catch up a little bit. What's going on?

Oh, you know, this is one of the few bits of human interaction outside of my house that I get is to talk to you here on Skype video, right? Like, it's a weird time, isn't it? It's totally weird. We normally don't record video, but we do see each other. And we've noticed as we contacted us or connected today that both of us have new haircuts, but they're homegrown haircuts. I never cut my own hair and it was getting bigger and bigger and my hair doesn't get long. It just gets big and poofy and it's more like a Q-tip on top of my head. And eventually I

just break down like, I can't do this. So at first I looked at my daughter who's 19. I'm like, here's a shaver. You got to cut my hair. And she's like, no, no. And so she did an okay job. And that was really good. And then I decided to go back and like, it was still quite a bit actually longer than I wanted. So I gave it a second haircutting as well. And you know, honestly,

it's about as good as the haircuts I pay for. So I'm feeling like this might be a trend that I was forced into. And you're looking good yourself. Yeah, so I normally have like a short goatee trend that I was forced into and you're looking good yourself yeah so I normally have like a like a short goatee thing that I keep and I just have when I bought the trimmer for it I just I only needed one guard because I'm I mean I didn't think ahead and I um just like I don't know where all the rest of the guards were so I only have that like little short guard for the uh

the trimmer and my wife was willing to cut my hair, but it's a little shorter than I was expecting, but I think it's all right. I don't have to worry about it anymore. My kids say it's like a peach, so they come by and rub my head all the time. It looks good. And that's the weird world we find ourselves in.

You know, I'm here drinking coffee out of this Pittsburgh mug, but there will be no Pittsburgh, right? PyCon is now officially canceled. Yeah, that's a bummer. This might be the first recording since it was actually fully, fully canceled. At least the first time we spoke about it, possibly. We might have managed to last time, but there's certainly a tightening up on getting together for conferences and that kind of thing. And, you know, we spoke a little bit about what's going to stick, these changes that we have to

have that are going to stick. Like a lot of meetings could just be emails or they could be remote or whatever, right? And I wonder what that's going to do to the conference space. Yeah, I wonder too. One of the reactions I heard from somebody was even if it was to move towards a virtual, which they were talking about that a little bit, maybe doing a virtual conference,

but that wasn't the thing. It's essentially canceled. Although some people are sending in videos, they're going to have some people be able to send in their presentations as videos. So that's neat. But it's the meeting people. We interact with people online all year long, and it's one chance to get together. So at least with PyCon, I don't think that'll go away.

No, I don't think so. For me, it's not about the talks. I go to talks, and they're interesting. But I don't think so. For me, it's not about the talks. I go to talks and they're interesting, but it's about the stuff that happens in between it. It's about hanging out with you, hanging out with people from other places I don't normally see, making those connections

and just stumbling onto things that are like, oh, I had no idea that this was happening. Let's talk about that. That's the value and that's really tough to replace. Yeah, it really is tough to replace. more video chats will help absolutely all right well i guess everyone be safe out there yeah and follow our lead stay home even if you got to cut your own hair well let's move on to some topics so first off i was hoping to talk about pi project.toml i think we've i'm sure we've covered it before but the brett cannon is is

he still part of the core or did he get voted out i think he's still part of the core ed honchos i think he's on the steering council yeah yeah he wrote an article just uh released recently called what the heck is pi project.toml and one of the reasons why he wrote it and i think there's an interesting side effect is uh so if you're just joining us, the pyproject.toml came out of the efforts with PEP 517 and 518 to define what the file looks like, and it kind of defines what tools are needed to build a project and how to build it. And that used to be mostly just setup tools,

but now there's lots of other tools like Flit and Poetry, and you can make your own if you want to. But these are, this file, one of the side effects is people started adding other, their tool configuration to this file, such as like coverage and talks. Now you can put those configurations into pyproject.toml for those tools, even though it has nothing to do with building. And it just saves you from having an extra file, even though those tools have their own

configuration files. And then Black came around, and Black only has a few configurations available. So instead of creating their own configuration file, Black uses the PyProject.toml for looking up configuration if you want to configure line length or something. And so there's been projects that have added PyProject.toml files for the reason because they're using Black also, and now their builds don't work. And the reason is if there's a PyProject.toml,

even if you're using setup tools, pip will look for the build stuff around that, within the file. So Brett has gone and added, showed the few lines of code you need to add to the toml file to specify how to build with setup tools that's the main contribution for this but it's also i think that people ought to go read the article because it's a good uh good summary of where we're at right now yeah the pyproject.toml is great and it's super cool that you can specify things like here's how you can say build the package right it requires

a wheel and whatnot yeah Yeah, it's cool. I really like it. I honestly need to start working with it. I'm still on the requirements.txt side of the fence, just waiting for things to shake out. But it certainly seems like there's a lot of energy here. Yeah, definitely. Yeah, so definitely a good pick there.

You know, sometimes we talk about awesome things on Python Bytes, right? Yeah. And we've even talked about awesome lists, which are lists organizing various other things, projects, websites, tools, whatever, around something. But last time in passing, we mentioned that there was a cool article by Jack McHugh, where he blogged about the awesome Python Bytes awesome list. Yeah, that was neat. That was cool, that was neat.

Yeah, so now Jack has made this a proper GitHub repository where it has a clear way to contribute to it. You can do PRs and all sorts of stuff. And he said he'll be adding personally to the repo whenever he hears about awesome things like this. Maybe he'll add his own list to it, which would be very meta, right? It would just link back to itself. No, just kidding. But that's really awesome. And there's already five PRs from listeners accepted on this list. Oh, wow. That's great.

Isn't that cool? People seem to be liking it. After we announced it, then we had other people saying, and I saw some commentary about, there was some missing that he forgot about. Yeah. Well, it's just the stuff that really stood out to jack in the beginning right so if you want to go back and kind of look through what some people thought of a year and a half ago that we talked about that was awesome you can go there and if you look at it it comes with graphics like all good things that present things if you can have graphics guess what it's like a thousand words isn't it oh yeah Oh, yeah. Isn't that cool? This is great.

Yeah, it's really, really polished. And you just flip through and like we talked about D type, the visualization for Pandas data frames. And, you know, there's like some nice examples and pictures there. Really, really good stuff. So I think it's a great way to explore some of the stuff that we talk about. A couple of fun projects I saw on there that I just remembered. Oh, yeah, we did talk about that. That was cool.

It was great expectations for validating and documenting and profiling data. PandasVet, which is a Flake 8 linter for opinionated Pandas code. GeoAlchemy, which is like geospatial databases on top of SQL Alchemy. And Vue.py, which provides in-browser Python runtimes interacting with

Vue.js. So just, you know, all sorts of cool stuff there. And I guess related to this, although not technically anything to do with that is we have a super cool search over at Python by set of M slash search. And it's really fast. It even searches our spoken words. So if you want to just find something that maybe we talked about a long time ago or see if it's covered, throw it in there and it'll pop up.

That's really neat. And I love this because I'm like flipping through this going, oh, I want to go back and check that out again. Exactly. Like that was like a year and a half and we only spent like, you know, half an hour playing with it before we talked about it. Then maybe we didn't use it again. But yeah, it was cool.

So I legitimately enjoyed going through the list here. And I thought listeners might as well if they heard it before or even about it, then maybe we didn't use it again. But yeah, it was cool. So I legitimately enjoyed going through the list here. And I thought listeners might as well if they heard it before or even if they didn't. Yeah. Yeah, that's awesome. Yeah. Well, I guess you were going to continue a theme a little bit with your from the pyproject.toml onward, huh? I've been playing with a project for a while off and on associated with testing code called

the Cards Project. and it's a just a little to-do app thing it's just a little sample thing to mostly to play with all of the stuff around python and distribution and testing and and all that and um somebody uh oh gosh it was months ago and i should have their name here name here. But somebody else contributed a pull request to add GitHub Actions. So GitHub Actions are a way to, you know, you can use that for, like, CICD workflows with Python project, with all sorts of stuff, but especially with Python projects. Because, you know, the building pipeline is kind of short with Python. It's not as complicated as other things.

So often you don't really, it might be overkill to use another CI system. And so GitHub Actions are really pretty cool for that. And so I incorporated that and then was looking into the last step was how do I publish to the PyPI? I'd really like to add that. So there's a, I was looking it up and the PyPA, the Python Packaging Authority, actually has a article written called Publishing Package Distribution Releases Using GitHub Actions with CICD Workflows. And so this was like, I'm following through this. I'm actually ran into some hiccups, but I wanted to cover it anyway. My promise is that by the time this goes live,

the cards project will actually be complete because it's almost there. There's a few hiccups I've ran into. PyPI requires you to have an email associated with the package, which is kind of annoying because Flit and Pip don't anymore, but PyPI still does, so i had to do that and there's a few a few other hiccups i don't know if i'll write it if there's just these i'll probably just fix them and have it ready for people to look at but uh if it's too many more i'll have to write a new blog post and i'll associate that with this as well yeah this looks this looks great. And I had no idea about the GitHub secrets. So one of the

challenges is you want to have your personal PyPI account where you are the maintainer admin of that package on PyPI.org. You might not want to contribute that into the source code, especially for an open source project, right? Yeah, that's going to result in some bad things happening there eventually. It talks about how to use the secrets settings in GitHub to store those there and then how to pull those out as replacements in your GitHub actions, which I had never done. That's pretty cool. Yeah, and the workflow that they're suggesting,

which is I think a great idea, is to go ahead and have every pull request or merge to some master branch or to anywhere, any branch, go through the entire thing and try to publish to the test PyPI server. And only ones where you change the tag, which is where you would change a tag if you change your version number, that's when it would push to a PyPI, the real one. But to have that workflow even going through the test PyPI every time you push something, I think that's a great idea. Yeah, it is. But somehow you got to change the version number or it's not going to let you publish even to test, right? Yeah. So I'm not sure. I still haven't figured out how to get

around that. So we'll see. Yeah. Cool. One of the things I love, I love really great tutorials. The internet, I think, is working because of the great tutorials around. And one of the groups that have put out some awesome tutorials is DigitalOcean. So DigitalOcean is sponsoring this episode. And one of the things that they're offering, of course, is they offer a way to get started with, they offer Kubernetes clusters, and you can do that with DigitalOcean. But to get started with hosting and running Linux servers with Kubernetes clusters, that can be a little tricky. And that's why we want to highlight that DigitalOcean has launched their new support center.

And this makes it easier to find answers to your questions, help you get up to speed, get what you need. And you can search across product documentation and community tutorials and forums all in one place. And especially with something tricky like a Kubernetes cluster, or really even if you're new to any of this, their support center is awesome, got great tutorials. So visit pythonbytes.fm slash doc support to see their tutorials. And of course, use pythonbytes.fm slash digital ocean for $100 credit for new users.

Yeah, absolutely. Their tutorials are so good that you can even select different operating systems in different versions so that the steps exactly line up. If you're on, say, Ubuntu 16, you don't have to try to patch that back. The tutorial steps to be different, which is, that's taken

to the next level. Very good stuff. I've heard some from somebody once before where they were actually using a different host and using DigitalOcean tutorials to help them set it up. And then they finally realized, why am I giving money to somebody else? Let's use DigitalOcean because they helped me out. That's where our infrastructure

is. Very good. Now this next one, Brian, this one is best seen through pictures. So make sure you open this up and we talk through it. So like I said, all good presentations, especially stuff that's graphical, has pictures, people. This is amazing. So this thing that I want to talk about is called Rich. Rich text for terminals. So we've talked a lot about how it's great to have GUIs and web apps and stuff like

that. That's very visual for the presentations, which are still true. But a lot of times you just want something in a little terminal app command CLI app. And it would be nice if it wasn't just all one plain color or just, you know, all text left aligned. So there's this cool project called Rich. And Rich lets you have up to 16.7 million colors for your terminal colors, not just like eight or whatever it is, you know, red and light red, like 16.8 million colors. It supports bold, dim, italics, underline, strikethrough, and even, please don't use it, the blink tag. You could put the blink tag out there you can have text that's left aligned centered right aligned justified supports like chinese and korean has emojis like you can put colon apple colon and an apple shows up you can as part of a string put little escapes so you can say this word is bold magenta. And then here's the

rest of the words and just print that thing out. So you have like inline styling tables, you have beautiful tables, like in the terminal. Those are really nice. Those are really nice syntax highlighting for code. So you can print out Python code with line numbers and it'll highlight it. And even has a markdown support. So you could write markdown and it renders it as its own version of rich text not not quite html or something like that but it has you know bulleted lists and like titles and whatnot even as progress bars and logger support and all kinds of stuff isn't that cool yeah the logger handling is actually pretty great yeah so if you're working on a terminal app and you're like we're just going to keep it this way, but I want it to look nicer.

I want it to look a little more professional. This is like kind of your all-in-one. Do a bunch of cool stuff here. You can even have multiple progress bars all updating in parallel on the screen. So if you've got a bunch of downloads, you can indicate them all happening or something like that. A bunch of jobs running. Sweet. So yeah, this is a cool project.

Might have to add this to cards. That would be cool. That would be yeah it definitely would it even has like all of that support for windows if you use the new windows console or terminal i guess is the word that they're they're now using because console i guess is the old school thing and microsoft's like no no we all have terminals now and you can go get that from the you probably don't have that on windows it's in preview but you can get it on the windows store for free and i link to that as well so if you're on windows and you want a better terminal in general that supports this kind of stuff check that out too oh interesting second do you know if that'll roll into the normal just all releases

of windows at some point i would hope so maybe it's still in beta i guess we'll have to see but it would be nice, right? It would be one step closer to parity across all these platforms, which, you know, it's always good. Yeah, I just use the bash terminal for – or Git for Windows comes with a bash terminal. Right, nice. Speaking of Windows, I guess it's any operating system, but I'm actually kind of surprised this has been on our to-do list for a long time.

A library called PSUtil. It's a cross-platform library for process and system monitoring in Python. And I'm actually surprised that we haven't really covered it. We must have covered it in passing a couple times. But I wanted to highlight it because it is an amazing little tool. It's not something I really love when I have to use it, but things like there's time zone control and multiple computers or services running on a different machine with Windows. Like I said, you can use it with, it's a cross-platform thing, but you can use this to grab

CPU utilization, memory utilization, disk usage, network, what ports are being used. And you can even see all of this information based on which process. So you can get like a per process list of CPU usage and everything within that. And then around processes, you can suspend and kill and signal different processes. So if you're, one of the things that we use this for is monitoring our build servers. I know there's other ways to do it, but this is a pretty easy way to go over to another server and grab the CPU usage and memory usage

so we can keep track of all of our, make sure all of our build servers are working and not overloaded. So we use this. It's a pretty cool little tool. Yeah, this looks really great. I think we also talked about it last week briefly when we covered PyTest Monitor, and that was built on top of PSUtil amongst other things. We didn't cover PSUtil, but it's that kind of tools that this enables, right?

Yeah, PyTest Monitor used that also. I'm like, man, I think we covered this recently. But yeah, it was last week we covered that. By itself, it's still a really great thing to use by itself. And the readme is huge, but it shows you a lot of different examples for how to use it. Yeah, it looks super useful. If you're going to do any automation and sort of admin sort of work, it looks great.

One of the great things that I love is the cross-platform part because this sort of stuff you can do on i mean you can do directly with windows and with unix but you know it's different on everything else on everything and so having a cross i'm pretty impressed that it can even exist as a cross-platform thing but it can't yeah no kidding apparently it can and it's awesome yeah definitely definitely a good one so let me ask you this question if i'm gonna store some numbers let's say up to 100 000 how big of a integer do i need to make in python i don't have to make an integer yeah they just are there they're just there it's beautiful right but i know that you've done a lot of c and c plus plus and i have as well back in the day and you used to have to really think about that, right?

Like if you saw a negative 32,000 for a number and you thought you were adding to it, you're like, oh, it was a short and it overflowed. Darn, well, I guess it's, maybe it should be a U short, an unsigned short. It could hold up to 65,000, right? Like this is something you always had to think about. And in Python, you don't.

And how that happens is pretty interesting i didn't really know the internals i kind of guessed maybe something like what's happening was happening but i didn't know and so there's a cool article by arpit bayani and he wrote it something called how python implements super long integers so for example if you tried to take two and raise it to the power of 20,000 in C, you would get infinity. Because it's, well, it's bigger than we can hold it. So it must be infinity as far as we're concerned. But in Python, it's fine. It just gives you a 6,021 digit integer.

And you don't have to declare like I'm working with really big numbers or anything like that. And so Python is pretty cool in how it's transparent with these, right? Yeah. Yeah. So this article digs into the CPython source code. It talks about the algorithms and the data structures that make this happen. So basically, the numbers in Python are represented as what's called a PyVar object. So PyObject, that's the core type of things in Python are represented as what's called a PyVar object. So PyObject, that's the

core type of things in Python. But this is a variable length one, right? And so there's a couple of different types that are like this. We've got lists, we've got tuples, but we also have numbers. And that indicates that they can be of different size, and they can basically grow as they need to. So Python's numbers ultimately are represented by this thing called a long object and that has a PyObject base but then it also has a size and a digit. And these digits are, I think they're 4 or 8 bytes long. I can't remember, it doesn't say here. It's a macro that would expand

to 4 or 8, something like that. But basically it uses a list of digits and it, you know, initially it just uses one of those, but then when it gets full, it adds another and another and another. And what's interesting is their base 230, not 10, not 16, 230. That's weird. Okay. Yeah. Because apparently it can most efficiently use that's exactly that space of its four bytes or whatever as individual elements in a base. So it's pretty interesting. Like there's this ginormous number. And if you were trying to represent it in base 230, it's 100. I'm not going to read it off because it's really, really long, but it's pretty interesting how it uses this. But then when you get into operations, right, if I'm going to add two numbers and they're base 230, like what algorithm do you do?

It's not like based in where you normally do the thing or division or something. So that's also interesting to think about. If you look at arithmetic, it's pretty straightforward. You just add within the digits. And if you overflow 230, you do a carryover like you learned in elementary school. Subtraction is like you do the borrow so it's like reverse but then multiplication in order to keep things efficient uses uh an algorithm called the karastaba taboo stabua algorithm which is a interesting way to multiply two in-digit numbers in different bases and stuff. And yeah, so if you've ever wondered how come you don't have to worry about numbers overflowing in Python,

here's a cool look inside that at the C Python source code and some of the algorithms. That's amazing. Yeah. Actually, that's pretty cool. It is pretty cool. I'm so glad I don't have to worry about it, right? It just happens.

Yeah. One of the definite benefits of Python is this notion of developer time is way more valuable than computer time. So let's figure all this stuff out for everybody else, and then we can stop thinking about it so much. Yeah, for sure. I had a project many years ago where I had to work with an FPGA, and the clock system was such that the timing to set the current time had to divide that out for a multi-radix number. So it wasn't like base 10 or base 230. It was each digit had its own base.

Oh, my goodness. That's crazy. It was kind of a beast. It was a cool algorithm, but it was a fun thing. Yeah, yeah. Love this sort of stuff. Yeah, this is really cool. A cool glimpse behind the curtain, a little bit of the red pill.

You go down inside, see what's happening. One final comment here is there's some funny little tricks you can play on people to ask. You can create the number 10 and then create the number 10 somewhere else and ask if those are the same number, like with the word is. It's true for small numbers is, it's true for small numbers, but it's false for large numbers. And that's because Python pre-allocates the numbers negative five to 256. Oh, wow. Okay. When you have a hundred in your program, it's the same hundred everywhere. But if you have 1,226,411, that was made on the spot.

Because these are pointers. These are not just like four bytes on the stack. These are getting allocated in complicated ways each time. And so they said, look, negative five to 256, we use these all the time. Let's just make them when Python starts. Why minus five? Well, because who uses minus six? Come on.

I have no idea. I can see minus one and zero to 256. But beyond that, I don't really know what they must have some reason yeah i don't know why they probably started with negative one somebody said well let's do minus two also oh let's let's go hog wild and go all the way down to minus five exactly exactly anyway hey that's our six you got anything uh any news for us or any extras i have two really quick things you know know how Microsoft bought GitHub? That was kind of interesting news a while ago.

And actually people were contemplating the effects that would have on open source and all. Well, NPM, the PIP equivalent of JavaScript, is now acquired by GitHub as well. So just an interesting moving and shaking over there behind the scenes. That's really interesting. Unlike PyPA and PIP, NPM was like a commercial venture or something to that effect to try to bring that order to the JavaScript side.

And now, since it was commercial, it's been acquired. So NPM is the JavaScript thing? NPM is how JavaScript spells pip. Yeah. Okay. yeah okay okay quick other one is we're going to try to set up a youtube channel where people can see both of us talking the all are the silly stuff that we do in a fairly uncut unedited way but we're going to try to put a video for each topic so we just talked about the python number thing like just a single video on that.

It'd be easy to share with friends. It'll be something we put up there. So we'll have more details with you soon on the YouTube channel. But I'm looking forward to it. I think people, it'll give people a new look. I think it'll be fun. Yeah, and then people will be able to recognize our faces when we are not walking around

because everybody should stay home. Exactly. Assuming that we continue to just give ourselves haircuts. So we look the same. Yeah. Yeah. All right. Anything you want to share with folks?

Nope. Nope. Just working from home. All right. Well, I had this joke about C and numbers overflowing, and like, oh, why is it negative 32,000? Oh, because it's 37,000 positive, or something like that, right? Well, here's another one. And this one is just to kind of make you feel good about yourself as something like that, right? Well, here's another one.

And this one is just to kind of make you feel good about yourself as a Python developer, right? Yeah. So this one's visual as a lot of them seem to be lately, but I'll go ahead and we'll do our best to describe it. Right, I'm going to let you be the developer. So you do the first three. You got to give a little description though.

Okay, so the dude's 8.30 in the morning. I'm staring at the screen and I comment, stupid bug. And then seven hours later, and this guy apparently grows facial hair really fast because he's already got stubble at seven hours. Oh, it must be Linux. And then the next day, his face is red, and he's even got more hair. And he says, JavaScript's broken? Okay.

Oh, yeah. Bob comes in. The other guy comes in and looks at him and says, oh, hey, Bob. It looks like you forgot a semicolon. Ah, fix. Oh, yeah. Oh, man. Yeah.

So that'll be in the show notes. You can check out the little comic. It's fun. It's by Eric Burke. Nice one. Yeah. Don't miss semicolons. comic. It's fun. It's by Eric Burke. Nice one. Yeah. Don't miss semicolons. No.

There's a lot of things I don't miss. Although I'm doing a lot more C now, or C++. I've got to do it again. Well, it just makes you appreciate when you get to do Python. Yeah, it sure does. And I appreciate my time with you, because this is so special. Absolutely. Thank you.

It's great to get together and chat about this and share with everyone. Thanks. Thank you. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured,

just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. This is Brian Ocken and on behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.