Python Bytes - #43 Python string theory, v2

Episode Date: September 14, 2017

Topics covered in this episode: future-fstrings The Fun of Reinvention Sound Pattern Recognition with Python PEP 550: Execution Context Intro to Threads and Processes in Python Alternative filesystems for Python Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/43

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Python Bytes episode 43, recorded on September 13th, 2017. And I'm Michael Kennedy. And I'm Brian Ocken. Hey, it's great to be with you guys. Great to be with you, Brian. We have a bunch of really cool stuff lined up. Like, there's a few things that really surprised me and maybe even inspired me to go create this project that we were just chatting about before we hit record.

So very, very cool stuff. But before we get to it, let's just say thank you to Rollbar. Those guys are sponsoring the show. Check out their offers at pythonbytes.fm slash rollbar, and we'll get to them later. I want to talk about taking the future and sending it into the past, like Marty McFly style. What do you think yeah

yeah this um this popped up uh recently um just this last week and i'm not sure what the impetus was but it is future f strings it's a project you could just do pip install future dash f strings and and then you've got a little bit of a little it isn't a library that you include or it's uh it changes the encoding so you uh well i'll just jump to the chase it's a back port of f strings to even down to python 2.7 and i've tried it in 2.7 i haven't tried it in anything else but it is kind of fun to be able to play with F strings everywhere. So I had very mixed thoughts about this when you told me about this project. On one hand, I look at it like, you know, you're taking one of the really amazing and smooth features of Python 3.6 and making it available in legacy Python, which is kind

of like, do we really want to like encourage that sort of thing? But at the same time, I feel like this also lets you, as you're migrating your code, move to the latest, greatest syntax in Python 2. But then when you actually make the switch, you're not going to have to go like, well, we used format because that was the best option in Python 2. But now we've got F strings.

And so I feel like that's really cool. The other area where I think it's great is Python 3. There's plenty of people running Python 3.5. That's the default on Ubuntu right now and things like that that don't have F strings. I've decided after further contemplation that this is a really cool project and I want to check it out. So tell people what F strings are. Most people maybe know, but what's this thing, this F string thing? Well, it's, so if you're familiar with the format part that you could be,

I don't know how long ago you started using that. I think that's available in 2.7 as well. Yeah, I think so too. In the string that you're printing, you can like leave little brackets to show where you're going to insert something, and then you say like dot format, and then all the things that you're going to insert something. And then you say like dot format, and then all the things that you're going to put into the string. In F strings, you just put an F before the string. And then within your brackets, you just put the variable name for

whatever you're going to put in there. Or any expression can go in there. Right. So you could say like F quote, hello, curly thing dot upper and like uppercase the thing just in that expression or something like that, right? Yeah. Yeah, that's really cool. Yeah, at first I was like, I don't know if I like it, but once you start using F strings, you can't go back.

That's really one of the reasons why I like this project is because I do, I usually use Python 3.6, but there are times where I have to write some stuff in 3.5 or 2.7 or something. And I don't want to go back. Yeah, I know. I hear you. That's great.

I'm going to put a lot of tests around it. So I'm not going to rely on this yet because I don't know if it's really ready for prime time for everywhere. So I would say use caution. Dip into this slowly if you're going to use it. But it's a fun thing. I'm not sure I like how it's doing it, though. It's doing it in an interesting and odd way, yeah.

I don't know how else it would do it, but it's interrupting the encoding. Yeah, it installs a separate encoder. So it extends the UTF-8, I think, the UTF-8 encoding to also do this. So if you're not using UTF-8, I don't know, maybe you could hack it to do something like you're using. Yeah, yeah, it's pretty cool. So you put like a little encoding flag at the top and you install a special encoder and off it goes. Yeah, pretty interesting. All

right, so I want to stick with this Python 3.6 feature angle, even though that's bringing that stuff to other versions of Python. I want to talk about the fun of reinvention. So this was a really amazing presentation done by David Beasley. And it's done in the style of some of his other presentations. And in my mind, like the way technical presentations should be done, you should be up there writing code, showing how things work, not just slides and pictures and stuff, but like showing things, right? And so what he shows though is kind of insane and incredible and your mind will probably

explode if you see this. You'll definitely come away with a greater appreciation for Python 3.6. So what he does is he goes and builds upon some of the new 3.6 features that are below the surface that a lot of people wouldn't even know about like hooking into when a class gets defined so you can basically redefine what that means and things like that so for example he says well let's talk about type ins these type ins are cool but they don't do anything we can add these constraints and assertions inside of our code and that does something but we can't really check that in the tooling. So what if we could rewrite, create this little framework using inheritance and all sorts of other stuff that lets us add these constraints as type hints, and then they're

enforced at runtime automatically. It's pretty amazing. That's incredible. Yeah. Yeah. So whether you think it's a good idea or a bad idea to do that, the fact that you can do that is really, really cool. So you can basically set up a function, and as like a type hint constraint, you can say, and this is a positive integer between 0 and 200. And it will actually check that. It's really something. But mostly it's an amazing look inside. It's an amazing look inside of Python 3.6 and down at the low-level stuff.

So, Brian, knock, knock, who's there? Yeah, I am. Sound recognition is there. Yeah, sound recognition. So there's an interesting article called Sound Pattern Recognition with Python, and I really liked just the... I didn't know you could do this with SciPy.

So there's a, apparently part of SciPy, there's scipy.io.wav file. And I'm not sure if there's other types you can do use, but this article uses SciPy to read a wav file and, and then just has it as data, like an array of numbers, and then does some math on it. And it does, this particular one is assuming that you've got some wave files that have NOCs in them and trying to detect where the NOCs are and how far away they are. And just some basic little logic to try to figure out, yeah, like I said, where the peaks are. I think the author noticed that you need to have some minimum values for how you tell where the peaks are and then some distance between them so you don't have like ringing within one peak.

I like it because it's kind of a, I think it would extend easily to do, I'd like to take it and run with it and try to do some basic oscilloscope measurements with this kind of a thing. That'd be fun. Yeah, it'd be super fun it's it's really accessible like the actual code to figure out the knocks and to process the wave file it's it's really small and approachable and to me what i thought of when i heard this was oh you could build something that that is actually kind of smart so it shows you how to identify knocks right and you should be able to identify like a door jiggling, a key noise, you should be

able to identify like a few of these really distinct patterns. And if you could say like, identify like, what does a door jiggle sound like? What does a knock? What is a doorbell? What does a key sound like? You could put a little raspberry pie just by your front door that says, someone's home, someone unlocked the door, someone's knocking, you know, just cool stuff. Wouldn't that be, I mean, it seems like that's a weekend project with this. It doesn't seem major. Yeah, that would be fun. You know,

you could hook it up with Twilio and have get texts when somebody's knocking at your door. Right. And hook it up with a camera to take a picture of whoever's out there, see what they're doing. Right. Yeah. Yeah.

Send yourself a note. Hey, kids came home, right? They unlocked the door at three o'clock when school's out or whatever. That'd be fun. It sure would. All right. So I think people should take this idea and run with it. And if they do, they should send us a message or even better, go to pythonbytes.fm slash 43 and leave a comment at the bottom about what they created. Yeah, definitely. Definitely. Or even better, they could write a blog post and then email that to us and we'll highlight it on the show. Yeah, maybe we'll even cover it.

That'd be awesome. Yeah. Cool. So before we get to the next item, which is sort of mind-blowing, I want to talk about Rollbar. You guys hear me talk about Rollbar a lot probably, but that's because they're great. Basically, if you run any sort of web app, you should have real-time error monitoring and reporting. So if something goes wrong with the website, for example, pythonbytes.fm, we'll get a notification, it could be in Slack, it could be on our phone, it'll have all the details, you know, this person went to this URL and this

situation, here's the variables they pass to the function that caused a crash, things like that. So if you want to get this for your site or application, just go to pythonbytes.fm slash rollbar and check it out. So we've talked a lot about async stuff, right? Like async, both on the server and on the client. It's pretty mind blowing stuff. But it turns out that, you know, like all changes to software, there's like cascading consequences of adopting a new model or trying something new right and one of the cascading changes is in many of the high performance async processing loops i'm thinking async io but also uv loop which is a really super fast powerful one things like sanic are i think sanic is based

upon it the web framework and so on those things are ultra fast because they rely on async IO, but async IO doesn't use threads. And so that means thread local storage doesn't have any meaning anymore when all of these like concurrent operations are running on the same thread. Okay. Right. So like if I was running a website, I might store into thread local storage,

like the cookies or the authentication. And then later I might ask for it back but that could be changed because you have a blocking i o that released the thread that somebody else then wrote their cookie to right so it's kind of like crazy and there's also in other really interesting places like decimals working with decimals does this numpy error state warnings dot catch warnings profiling tracing all these things use this kind of stuff, right? But it doesn't work in this async await world. So we have a new PEP, PEP 550, that defines a new execution context. And this is from the guys at magic.io who created UV loop. So like their

motivation is really obvious, like they want UV loop to work and this is kind of a something in the way so the pep adds a new generic mechanism for ensuring consistent access to non-local state in the context of out-of-order execution such as generators and co-routines wow it's pretty fascinating right yeah it's very fascinating i hadn't even realized that i mean you think it through obviously it's a problem but i hadn't realized like oh my gosh like some of these like low-level things are just going to be broken. Like how, for example, state is processed in a web request. I'll be very interested to watch this

and see where it goes. Yeah, so maybe it makes it into Python 3.7, maybe not. But it's kind of cool that this is not just like, oh, we made UV work properly, but we're going to make all of Python work properly. So they've got some really nice examples on the article we're linking to. all of Python work properly. So they've got some really nice examples on the article we're linking to. That's really cool that they did it instead of just trying to hack their own way. Yeah, for sure. Speaking of threads and processing and all that.

I often work in single threaded or single process sort of tasks. I really liked this article called Intro to Threads and Processes in Python. It's a very accessible beginner's guide to parallel programming and really kind of what's the difference? Where would you use threads? Where would you use processes? And it's even got pictures with some algorithms to see to show how, like if you're a certain kind of job. I can't remember what the job is that he's doing.

He's doing some data science project for, based on Amazon and trying to analyze a bunch of different stuff from space. And so he's got to run tons of algorithms and tweak them with different parameters and stuff like that. It shows, um, like basically if you use a, uh, one thread, two threads or four threads, and then the same with processes, how it's affected with the running times. And you can watch to see that things are running in parallel and good use of little simple diagrams too. That's cool. Yeah. The pictures were great. The rundown is, um, conclusion at the end is if you're waiting on IO for something, threads are just fine. And if you are CPU heavy, then you want to go with multi-processing. Yep. Until the galectomy happens,

which who knows if that will, but until then, this is definitely a good introduction to it. Yeah. And I'm actually, I'm one of the, I don't know, I think there's probably a lot of people, but having the GIL doesn't really bother me. It does simplify how you, you program. I don't think it's that terrible. So yeah, it's, you know, it's in the extreme cases. I think the main group of people who suffer under the Gill are those doing computational cpu computational work outside of that you're mostly okay okay yeah so in that like async and await and stuff like that i'm gonna expose my uh knowledge of this stuff is um so async and await is that dealt with with uh multiple threads then no it's all one thread but it has a mechanism to say anytime you hit blocking i o go put this part of the code to sleep

and allow it to run another another task another thing until it hits blocking i o and the idea is like most the time you're waiting on databases or networks or something like that. And then you wake up for a second, you do a little processing and then you go back to wait on another network or database or web service. And as long as that's the kind of stuff you're waiting on,

the GIL is not a problem because it gets released. But if you're like trying to compute Pi with a power series or something, something like that, then it would just, it would stop. And the async wouldn't help you with that. Yeah, I just wanted to, so the async in a way there's going to use,

it's going to be in the same sort of family as when threads would be good for you. Yes, yeah, it's in exactly this similar spot, yeah. But just more efficiently. So let's talk about another low level thing. And I found this, I think it maybe even was recommended by a listener, some low level part of it. then i found the whole whole thing one of the problems that we can have

when we're doing unit testing is working with files right that can be a serious pain yeah yeah and if you want to save your files to like say user storage or you want to process different types of files like zip files, regular files. These are all sorts of pains that you have to deal with. But there's this really cool project called Alternative File Systems for Python that makes this all seamless. So the idea is you can work with files and directories in zip archives, in memory, on the cloud, or just as easy if they're on your hard drive.

So they give some examples in this project saying, you could write your code now, just with open, you know, such and such as fin and start working with it. And then you decide what that open means. Does that open mean normal files? Does that open mean something on s3, things like that. You can unit test your code, you can have have in-memory files, but open reads and writes to those in-memory files. You can upload your files to S3 or OneDrive or something like that by just writing to that folder

in that file system and all sorts of stuff like that. Yeah, so the memory file system or the temporary file system that they have would be great for parallelizing tests and things like that. Right. It keeps it all separate and isolated. So some of the back ends they have is they have application data. This is like the special user data locations in various operating systems.

They have Amazon S3 file systems. They have FTP memory. Let's see what else. That's pretty cool. They've got zip for reading and writing zip files. And they also have like SSH file systems for writing to remote servers. All kinds of stuff. And even tar files.

Good old tar files. Good old tar files. Yeah, you can just width open on those babies. Yeah, so this is really pretty interesting. I haven't tried it yet, but it looks quite promising. And it's extensible, so you can add more of these backends if that makes sense for you. That's awesome. I actually think this is pretty fun. I got to play with this. Yeah. Yeah, me too. So definitely check that out. And Brian, that's it for our six items for the week. That's incredible that we've already done.

I know. They're all fun and all interesting. I really enjoyed researching them this week. So what else are you up to? I got an interesting email this morning saying that they've taken off the beta off of the Python testing with PyTest on the Pragmatic website, and you can order the book now, and it's supposed to ship next Monday. That is awesome. Congratulations. Yeah, thanks. So that is what I've been thinking about.

I bet. Nice. I've been thinking about switch. Well, I have been too as of this afternoon. I'm working on a project that would add the switch statement to the Python language without extending it, just like a class that you can basically use. But it's a pretty clever use of defining blocks and stuff. And I think it might be interesting. I'll put a link to a gist or a GitHub repo or something

for you at the end of the show notes. And you guys let us know. I really do want feedback on that because I want to try to nail this. And I think if we could get it into something, we could stick on PyPI or have you do the work actually. PIP install switch. There you go i that might be a thing don't do that yeah don't do that switch laying who knows but yeah i think it would be a really cool feature if we could make it work for the language without actually changing the language because it's already flexible enough yeah cool cool all right well thanks for uh for everything brian and good to chat with everyone thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it

our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.