Python Bytes - #189 What does str.strip() do? Are you sure?

Episode Date: July 9, 2020

Topics covered in this episode: Improving Python exception chaining with raise-from Create and publish interactive reports in Python Pickle’s nine flaws PEP 602 -- Annual Release Cycle for Python More git Resources: PEP 616 -- String methods to remove prefixes and suffixes Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/189

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 189, recorded July 2nd, 2020. I'm Michael Kennedy. And I am Brian Hocken. And this episode is brought to you by us, our courses and our books and all those sorts of things. Tell you more about that later. Right now, let's talk about improving Python exceptions. Yeah, this came from Ram Racham.

I'm probably getting his last name wrong, but this is a pretty interesting article. Actually, I had seen this in code. So exceptions are a big part of programming in Python, I think. Some people don't like to use them because they're kind of difficult in some languages, but it's pretty easy in Python. So let's say you're dealing with an exception. You're inside of the accept part of a try-accept block, and an exception happens. And if an exception is raised from an accept clause, which exception should be reported? And I really don't know what the mechanism was in Python 2,

but there were changes made for Python 3 that are associated with PEP 3134 and others that both of them will be reported. So you get both exceptions. But you get something like that. Often you get something that says during handling of the above exception, another exception occurred. Right, which is super frustrating because it's not actually saying the error occurred trying to handle it. It's like I'm trying to add more information to it or give you more detail and more specificity

or something like that, right? Right. So what Ram is pointing out is that if an exception is raised within an except clause, it can be really one of two things. It can be something, I called some code that I didn't expect to have any exceptions and an exception happened there.

And that's what kind of the default message sort of implies. But it might also be, and he has like this long block, I'm just going to read it. An exception was raised and we decided to replace it with a different exception that will make more sense to whoever called this code. Maybe the new exception will make more sense because we're giving a more helpful error message,

or maybe we're using an exception class that is more relevant to the problem domain and whoever's calling our code could wrap the call with an accept clause that's tailored to this failure mode so a long mouthful but basically trying to be nice and raise a better exception or a better message or something and in order to do that all you have to do is just change one thing and it's the from e. So within your exception clause, you accept something as e or as any variable, but e is often used. You raise the other exception with a different message from e and that little from will get you a message within traceback will have both exceptions. It'll say during the handling of the above exception, another exception occurred.

Oh, no, wait. No, it'll say that's what we had before. It'll say. That's a confusing one because it looks like we tried to handle it, but we actually crashed again inside the handling of it, right? Right. So the new thing, if you do the the from me it'll say the above exception was the direct cause of the following exception so that's a little clearer i think so you have

the one on the top caused the second one to happen so you you know that we're in the case of somebody's trying to wrap a an exception with a friendlier exception which is subtle the reason why i kind of i don't know i just like it rom likes it and uh but he didn't see that no that like nobody was using it so he went even further and he went out and wrote a script that can look for probable places where that should be happening and he did uh he did pull requests to setup tools, SciPy, Matplotlib, Pandas, PyTest, IPython, MyPy, Pigments, and Sphinx, which is like awesome that he just went out and did pull requests against all of those projects.

Oh, that's really cool. The other thing he did was he wrote a new rule for Pylint. And so W0707, raise missing from which is kind of weird but but basically pilot will now catch it so if you use uh the most recent it's not released yet but if you run pilot from github you can see that rule but the next time they release pilot it'll be in there yeah this is neat i like it i think it's confusing the way it is now if you don't do the right way and i also often get messages from students saying oh i got this really bizarre error like some you know some other thing and the problem is buried in the other part of

this whole story right and it's it's not clear what part they should be paying attention to so i think this makes it more clear and it's nice. All right. Well, the next thing I want to talk about is publishing data science and other interactive stuff on the internet. Now there's a couple of ways you can do this. You could go to like Google collab and whatnot, but Tim Pogue sent over this new place where you basically custom built for this story and it's called data pain so data pain is an open source framework which makes it easy to turn scripts and notebooks into interactive reports so you can export a jupyter notebook as a report but then it's just static you can't interact with it or if you try to run it then you know there's a bunch of code in there and there's all

kinds of stuff that you probably don't want people to see. So DataPane is a way to create and host interactive HTML that runs against APIs and runs data and so on. It's pretty cool. You go in there and use sliders to adjust what's happening in, say, a graph or filtering or things like that. That's neat. So this is one of those things that it's a paid service, but they have free accounts that you can do, right? Yes, exactly. So I'm not entirely sure the whole spectrum of what's going on.

So it says it's an open source framework. So I think there's an API that you bring in to make this happen. And there might be a self-hosting story. They also have a platform for publishing it too. And on that platform it's free for individuals but i i can't find a price anywhere but it looks like they're going to eventually charge like for a team commercial corporate version i don't really know what's going on with that anymore but it's free for individuals and you can use their open source framework to generate these reports okay and they also have the gallery there.

So I was looking for it, I found it for a while. It's obviously right at the top. It says open analysis gallery for looking at pictures. Exactly. Yeah, so the idea is that you can use the tools that you already know and love. So you can use Jupyter or Colab or Airflow and you can build reports using their framework,

using Markdown as well, and cool libraries that people probably are already using for visualization like Altair. That's what I've been using for my visualizations lately. And then you can either export these as standalone HTML files, which is one option. That's the totally disconnected, just take the library. The other one is you can publish them to data pane for free where you can people can share them and embed them and there's probably this paid corporate thing that i don't have details on nice yeah so you're going to add like forms to help filter and drive the report

uh you can have a talk to apis also they have apis that you can use to deploy your scripts to their server and you can even have integrated setup set up with like GitHub actions or airflow or standard CI CD stuff to like update your reports on data pain.com. So pretty cool. Like you said, there's a gallery I'll link to the gallery at the end of the section. So these kinds of things, you just want to poke around the gallery and get a sense before you mess with

it. Right. I like pictures. I do. We both like pictures. Now, before we go on to the next thing and we look at the ways in which pickle might make you sour, let's really talk about things that won't make you sour.

Your PyTest book. If people are interested in learning PyTest, getting into testing and not just scratching the surface, but really deeply understanding PyTest, they can check out your book, which we'll link to in the show notes. And if they would like to take some courses, we have a whole bunch of courses over at TalkPython.

We're almost up to 200 hours of courses over there. That's a lot. Nice. It is a lot. But like I've said before, one of the things I like about it is you don't have to spend hours at a time. You can spend a few minutes and not lose your place. It's a nice way to set it up.

Yeah, thanks. On the PyTest site, I wanted to reach out to people. I'm pretty available on Twitter or through the contact form on Python. However, I've had people try to email me at the contact form on TalkPython, and Michael just sent it to me nicely. Anyway, I want to have people know. I'd like to know people, if they want to learn how to test more, whether or not they've read the PyTest book. What's stopping you?

I don't know how to get in people's heads of why are they not testing more. So please reach out to me and let me know your questions. That would be awesome. Yeah, absolutely. Now you want to make me sour? I love pickles. Do you like pickles? I'm coming around to them.

I used to really not like pickles, but I'm okay. Okay. I have a really fun pickle story to tell you, but it's like 10 minutes. It's better over here. Next time. But let's talk about Python's pickle. I kind of like it sometimes. It's one of the first things I learned about for saving data and stuff

is you can pickle stuff. So if you've got whatever object, a bunch of objects or whatever in a collection, you can use pickle to serialize it somewhere. I think we've brought it up here on this podcast, but you will occasionally run into people saying, just be careful with pickle or never use pickle or something. And Ned Batchelder put together a post called Pickles 9 Flaws,

where he says, it's not never use it, but, you know, use pickle if you're okay with these flaws. So, and I like this list because I've heard some of these before, but never in one place. First off, it's insecure. So unless you really, I mean, if you're only pickling and unpickling your own data within your own program and there's not a chance of having somebody else feed you bad pickles, then you're okay, but it's not secure. There's ways to have them and have malicious pickles be generated that can cause the unpickler to run random code, which is not what you want. And then he links to a more thorough article on the security if

you're worried about that. Old pickles, second one is old pickles look like old code. And the gist of it is you can't really version this stuff. So if you change the way your objects look, your pickles aren't going to, it's not going to work right. Change your data structure, and you've got to pretty much throw all data away and start over. Yeah, this one is really tricky because it takes the entire object graph. You say, like, here, save this list. This list contains these things.

These things contain pointers. So those other things, it's going to save all of that stuff and if the code that defines the fields of that change like if you rename a field and you try to load it you're done it's not going to be able it's like this blind binary blob that doesn't fit together anymore there's not like a well let me adjust for this it's just like nope yeah nope you can't no longer load that file so it depends but uh not being able to load your file at all or opening a text editor even that might be a bad place basically if you're trying to save code between versions or save stuff between versions it's not a good thing there may be cases where you're just using it to serialize between a live app a running application and you're you're just using it to serialize between a live app,

a running application, and you're not going to save that data anywhere. I think there was an interesting problem. I think it was Instagram that had this problem. Somebody that had done a big talk at PyCon, they were doing caching and something like Redis or something, and they were putting their pickles. They would pickle stuff, put it in the cache, and they pull it out. And then they switch from like two to three and they could no longer read their cache and it like took down the system or something yeah like something really even really weird like that like deploying a new version of the site yeah yeah i'll read through the whole list we can pick out a few they're implicit so it serializes everything even if you don't want it to oh

that's the over serializes the implicit part is it serializes things as class objects, and even if that's not really what you'd want, like his example is datetime. You really wouldn't want to serialize it as a datetime object. You'd want to serialize it as a string or something. The init isn't called, so normally your classes, the init gets called, but during unpickling, it just creates this object without calling init. So if there's any side effects that you need to have happen, that ain't going to happen.

It's Python only. You can't convert it between different languages. It's unreadable. You mentioned it's binary, so if anything goes wrong, good luck debugging it. It appears to pickle your code, but it really doesn't. So if you have a list of functions and classes and objects, those will be in there, but it's just names of stuff. And then when you unpickle it, it creates those objects without calling a net. It's slow, and so the binary part gets me.

But basically, these are all the things that are wrong with it. But there's some ways to get around it. And a lot of people will say, well, there's ways to get around it. And Ned's comment is, if you're going through some of the code to work around the problems with pickle, then just why are you using pickle? There's other things out there. And my favorite alternative is just JSON. So JSON mostly works most of the time. And you you can read it so you can look at what's wrong and go oh i see why it didn't work because i was putting this garbage in there or whatever or i haven't tried these others so there's other suggestions like marshmallow caters protocol

buffers i haven't used those before but the uh what was i going to bring up you brought up something about being binary and and that's one of the parts that gets me is a serializer that goes to binary that makes sense if it's really fast but using a slow serializer to slow and binary come on yeah one of the things that the over serializing that i found is interesting is caching is something we do for like long hard computations or something or just to save some computation you cache the result that stuff will get saved in your pickle also and that's surprising so it's just weird yeah so thanks ned yeah it's a cool article you know the one place that pickle kind of makes sense to me is if i'm doing like a really quick prototype and what I would need to save would be fairly complicated. I'm just like, you know what? I just want to save this.

And if I really invest and I like what we built here, we're going to go and rewrite this with a proper serialization data saving story. Yeah. I could see that. I, all right, just pickle it. Now I don't really often do that, that off very light, you know, it's not very common, but I would certainly consider like, I just need to save this complicated data structure and i don't want really know if i even need this forever but i'm just going to do it for now i might consider pickling but in general it's not super awesome and also the malicious bits makes you you know you got to be careful so the reason why you can't grab jason right away is because there's some objects that are not serializable right yes exactly in some things that are not serializable, right? Yes, exactly. And some things that are simple, like date times, like if you had a really complicated, some structure

that had like, say, date times or other objects in there, you can't just dump the whole thing to JSON, right? So if you were just in a hurry, like, I'm just really trying to spend 30 minutes to figure this out, but I just need to save this and then load it and like, then maybe pickling makes sense. But the fact that you could never be, it's very possible you upgrade Python, you can never load it again or things like that. You can't look at it. Like that makes me never want to invest in this

as like a long-term option. Yeah. The other thing I thought might be a reasonable thing is, I'd like to hear other people's thoughts on what reasonable uses for pickle are. But like in-memory stuff. So you're serializing it just to get it from one part of your system to another or something um yeah yeah yeah like a super deep

copy sort of thing might make sense potentially right um yeah speaking of it might change because you get a new version on python that is now going to be more common. Yeah. So Lucas Lenga, he's been in charge of releasing, release management of Python recently, and he has gotten PEP 602 accepted. And PEP 602 says we're switching from a 17 to 18 month release cycle to a yearly release cycle. Yay. That makes sense.

Yeah. Remember we were talking about something that got into python 3.9 and we were looking at the conversation on the dev channel and it was like i really really tried i submitted this before the deadline but you guys weren't able to like review the pr before the the window closed for 3.8. So because of that, like that feature missed by a couple of weeks, and it was a year and a half before it could actually get accepted, right? It was like really long. So on one hand, this is to shorten that cycle. The other thing is on a given year, when will Python come out? I don't know. Is it an even year or an odd year? Like that's just a

weird way to think of things, right? We have conferences yearly in theory, year like that's just a weird way to think of things right we have conferences yearly in theory things like that right but for python it had been on this 18 month sort of alternating cycle this way they can schedule it so that things like the feature freeze or beta stage happens right after pycon so the sprints can always focus on getting that last bit into c python at pycon and what and it's much more predictable so i think that'll be nice yeah and there's other stuff in life like uh you know holidays and things like that that you can just sort of figure out when a good time is for the whole schedule and then just do that every year absolutely absolutely so let's see There's a little bit of interesting

detail. It says the PEP proposes that Python 3 dot whatever be developed around, I think it's over 17 months across a 17 month period, which doesn't sound annual, does it? No. So the way it works is the first five months will overlap with the previous versions beta period and then there's seven months spent on the alpha release and then three months on beta releases and then finally two months on release candidates so there's the way it's going to work instead of like we're now on 3.8 now we're're all focusing on three, nine, that three is out. There's like an overlap. So basically as soon as the features freeze on the previous, on the what's coming out, right? So like right now, three, nine is coming out. So as soon as it hit feature freeze, three, 10 would start, right? So there's like this overlap as the version that's coming out,

it's getting stabilized and finalized new features and developments already happening on the other one. And the way with that overlap working, it's going to result in a new version every 12 months. There'll be like a five month period of where there's like new features, go to the new one and stabilization to the old one. And this is kind of how commercial software is done anyway. Yeah. Yeah.

You never, it's like no one touches the new one until we've like shipped it. The gold version, like now, like some people are fixing stuff that they need to fix and some people are putting the new features. So in terms of advantages, they call out, makes the releases smaller. So doubling the cadence doesn't double their available development resources and so on. And consecutive releases are going to be smaller in terms of features.

So that's good. It puts features and bug fixes in the hands of users sooner, right? Six months sooner. It creates a more gradual upgrade path, right? Instead of adopting 18 months at a time, you can adopt 12 months at a time. Like you could be, if you're saying like, we're always going to be one version back of what just got released on the Python, right? So if you're saying we're going to run on three, seven in today's world, that's not an 18-month gap. That's a 12-month gap. So that's good. You're a little less far behind if you're on the laggard side of things.

The predictable calendar. So the final release is always going to be in October after the annual core sprint. And the beta phase always starts in May. So after the PyCon US sprint. So there's a nice lineup for that as well. Also, my prediction is that people won't be doing the two version thing. They'll actually do three. I recommend everybody do this anyway.

So let's say like right now, if you've got a project on Python, you should make sure that it works on 3.7 and 3.8, of course, but also 3.9 is around. So test it on 3.9 also. And then when 3.9 is the official official one maybe you keep 3.7 but if you don't want to you you're at least doing making sure you're compatible with the next one the current one and the last one so it'll make more sense absolutely very good now we spent a lot of last episode on git right yeah well uh episode 187 we talked about oh shit git from a zine from julie

evans and i i mentioned that i was uh concerned about i wanted to buy this for my team but i was i didn't know if hr would like the naming of it and uh john place reached out one of our listeners and said you know there's a non-swearing version he actually sent us a ton of great git information so the same little magazine you can get it as dang it get ah shucks ah shucks get the god darn it get all of those would be good but it's dang it get and then also when i was looking around these there's also these were zines that you can buy but these were inspired by by two websites that were put together by Katie Siler Miller, dangitgit.com and ohshitgit.com. There's actually really cool. These are free

websites that have kind of the idea of something went wrong. How do I fix it? And then every single one of the topics is clickable and linkable. And I didn't understand the clicks at first. I'm like, oh, these are links. Do they go to articles? Oh, it just takes me right to where I already was. I don't understand this. Why would I do this? Well, it's so that they're anchors so that you can click on it

and get the URL and send it to somebody. So if somebody asks you a question and the answer is here, you can send them the link and say, here's how to fix it. And then next time they can fix it themselves. So these are neat things in the community. And then also this, okay, do you see the link for get cheat sheet? Have you looked at this yet? with a bunch of commands. But this is an incredible resource. The Git Cheat Sheet is an interactive single-page website that's just beautiful to look at, for one. But it's got five columns. I'm going to pull it up also. It's got five columns, the stash, workplace, index,

local repository, and upstream repository. And when you hover over it or click on the different columns it shows you all of the different git commands that affect that so if you click like click on the local repo for instance it'll show you how to get information from you know diff how do you compare between the workspace and the local repo what reset does does, switch, rebase, cherry picking, and all sorts of stuff. And then different commands that go between the index and the local repo.

And the local repo and the upstream one with pushes and stuff. But it has this nice visual of where your stuff is and how do you get it back and forth. And then if you hover over any of the commands, it shows you the information on the bottom. This is just great. I'm going to share this with my team

for everybody that's kind of, because one of the things that takes a while to get your head around with Git is this idea of what is the workspace and the index and the local repo and stuff. So having a visual of what commands affect which part of Git, it's pretty darn

cool. The last thing I wanted to share was this is a one-pager that you can print out. It's a Git Pretty that is kind of like the Dangit, Git, and OhShit Git offerings, where it's a single-page PNG flowchart that starts with, so you have a

mess on your hands, and then it asks you a bunch of questions that funnel you to what commands might fix it. And I don't know if I'd use this much, but I totally want to print this out and put it on my cubicle wall because it's kind of entertaining. Yeah, this is really cool. I like it.

More Git resources. These are a bunch of great, yeah, a lot of bunch of Git resources. All right, I'm going to close our main items out, Brian, with a simple one okay okay so there's a new pep around strip basically that's the short version before i tell you the long version okay so if i have a string and the string is saturday is the first like so saturday is the number one st and And suppose for whatever reason,

I'm not sure it's a really normal use case, but for whatever reason, I want to get the ST off of the end here. And so I might say.strip ST. What comes out of there? Yeah, that's interesting. So what I thought was when I first encountered the strip function,

I would pass it a string and that string would be taken away from the string that I was stripping it from. So like if I say dot strip st, it's going to take the st off the end. But what it really means is take away all the s's and take away all the t's until you hit something else. So it would be adderday is the one, right? It would take the s off the front even though it's not st it's just s right yeah yeah i mean i guess the idea is you give it a

space a tab and a backslash in and a backslash r and you say take that away and that that destroys the white space right that's probably the foundation right but but you give it a string it doesn't take the string away it turns it into a bunch of characters and it just takes the characters away but what if you wanted to wait take away the st well. It turns it into a bunch of characters and it just takes the characters away. But what if you wanted to take away the ST? Well, you've got to write a bunch of code that goes and finds the ST and sees if it's at the end and then take it away. Or you wait a little while and use a new function that's going to be in 3.9 called remove prefix or remove suffix. So Dennis Sweeney has got a pep accepted to add these two functions that are actually probably what you first thought that strip would do given a string

it's going to take it that string off the ends okay not a big deal but i suspect that there's probably people out there that type more than one character into strip and expect it to be treated as a substring when it's not and so there's going to be in three nine two functions that do that yeah i mean i totally because i get the need to have like a set of things if you want to pass it in like a space and a tab and a new line there's a bunch of things you want to strip off the end well there's easier ways to strip the white space but like let's say there's other like random stuff that might appear at the end yeah uh stripping that but having it be character i was totally surprised by that i didn't believe you i just typed it in

because i'm like well what about the s and is but no it just the strip takes those it's a set of characters that are only at the end yeah it takes the s off saturday and the st off of first but it leaves the other ones alone, yeah. Yeah, so replace doesn't, there's not there was not really a great way in Python to do that without writing your own function that checks that those are only on the ends and then takes them away.

And so now remove prefix and remove suffix. Nice. And just finally, this will be applied to all the string-like things in Python. So unicode str, binary bytes, byte arrays, objects, and collections.userString. So not just against str, but it takes all these other things that are stringish. Oh, nice. Stringish.

Is that a technical term? I think if we get the pep accepted, it'll be technical, yeah. Okay. No. Yeah, all right. So that's it for our main items. What else you got? I've just been working a lot, so I got nothing.

How about you? Got nothing. Well, actually, a couple things going on. So Manning, the book publisher, they are going to have a Python conference in a couple days. I'm pulling it really quick to see exactly what day that is. Actually, I do know what day it is. I think it's the 17th. I think it's the 17th of July. Anyway, we're going to be there. Not the 17th. It's the 14th of July.

So we're going to be doing a live Python Byte at the Manning Conference. I'll put a link to it in the show notes so you guys can sign up. So if you want to see us do a live event, we've done that at PyCon, we've done that at some other events, and it'll be fun to do it virtually, which is basically all we get for our conferences, right? By live, you mean in real time, but online? Right, with audience questions and stuff like that. Oh, okay. So that'll be fun. Also, I did a talk on 10 techniques for web developers with a little bit of a focus on Pyramid, but more generally just for web developers, tips and tricks and tools and whatnot'll put a link to that as well. We also have a humble bundle running right now. It's probably when this episode comes out, got another week or so.

So there we've got a couple of courses for me, but tons of stuff, stuff from Matt Harrison and stuff from real Python and moving learner and on and on great long list. There's like $1,400 worth of Python content and get it for 25 bucks. So that's going like crazy right now so if you're thinking about maybe taking one of the courses for either for me or one of these other guys check it out it's pretty much as good of a deal as you're going to get on those and then lastly lastly abhi sheikh pednecker hopefully i got your name close to right there actually built something really cool for us so at pythonbytes.fm and

talkpython.fm, we have a search API and click on the little search box in the top right. And there's like a link to a JSON API you can use. So he did, of course, what you should do when you find an API and use it. He wrote two telegram bots, and you can just speak to the telegram bot. And you can either install the talk Python one or the Python bytes Telegram bot. If you just text or speak to it in Telegram, it'll search talk Python and give you relevant responses about stuff we've covered. That's cool.

Isn't that cool? Yeah. It's super, super cool. So I'll put the links to get those, those Telegram bots for you if you want. Nice. All right.

You ready for a joke? I am. All right. This one comes from Karen Chi. I don't think she submitted it to us. I think we just found it on Twitter, but here's a quick adaptation. It's not exactly what she said.

I expanded a little. She did it on Twitter, so it had to fit within a couple of characters. So it's a conversation. And Brian, let's just have it together. Okay. Okay. Who do you want me to be?

Oh, I'll be you. No, I'll be me wait okay yeah i'll start it off how's that because it's very confusing the way i phrase that all right hey uh famous engineer inventor is coming over tonight for dinner you want to join us sure who is it oh his name is rube goldberg that name rings a bell which sets off a trap that undoes a buckle and releases a ball that rolls down a pipe. And yeah. And on and on and on and on. All right, well, that's it.

Rube Goldberg, I love those things. I should probably go check the internet for some really funny ones. And with all the coronavirus stuff, there's actually some awesome Rube Goldbergs that people are putting together on YouTube lately because people are at home and they have lots of time on their hands

and their kids are home. Absolutely. Absolutely. All right. Well, I'm definitely going to go check YouTube now. Okay, bye. All right. See you, Brian.

Thanks. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.