Python Bytes - #3 Python 3.6 is coming, and it's awesome plus superior text processing with Pynini

Episode Date: November 23, 2016

Topics covered in this episode: - Shipped: Parsing horrible things with Python Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/3

This is Python Bytes, Python headlines and news delivered directly to your earbuds. Episode number three, recorded November 21st, 2016. Hey, Brian. Hello, everyone. Hey. Hey. Hey, man. Great to chat with you. I'm really loving this show we're putting on. This is so much fun. But before I get to the headlines, I actually have some good news about the show.

What's that, Michael? Python Bytes is now brought to you by Rollbar. They take the pain out of errors, and we'll hear from them later. But thank you, Rollbar, for sponsoring the show. That's really awesome. We appreciate it. That's really awesome. Thank you, Rollbar.

Yeah, yeah, definitely. All right, let's get to the headlines. Every week, there's so much cool stuff happening, right? Well, yeah, and this last week, I just read a huge list of articles I read, Python-related. I really enjoyed it. It was a hard time picking out which one to talk about. Yeah, it really was. Yeah.

All right, well, but you had to pick, right? Because we're only doing three each. We're trying to keep this short and on topic. Yeah, and I thought maybe we'd pick the same three, but we didn't. So first up, I've got an article titled, How to Get Superior Text Processing in Python with PyNini. I think I'm pronouncing that right. PyNinny.

Or maybe it's NiNi. I'm not sure. PyNinny? Like the Knights of Ni from Monty Python. Oh, yes. Yeah, yeah. You're probably right about that. This is from the O'Reilly blog,

and it was written by Kyle Gorman and Richard Sprout, I think. Yeah, both who are PhDs and linguistics of some variety or other, right? Yeah, and actually because of that and also because of the title, I was afraid this was going to be sort of a hard thing to get into, but it's a really easy article. And my interest in it is mostly because when I read the Pragmatic Programmer a long time ago, it introduced me to the concept of mini-languages to solve simple problems with this is the data I have and how to change it into the data I need.

Often I used regular expressions for things like that, but sometimes that's not enough. The next step always seemed to be to build up a parser or an abstract syntax tree, and that's way too far that I want to go. I think this article produces a nice halfway ground talking about what, now what they call them is finite state transducers, which sounds like something in a Back to the Future movie.

But it's- Quick, Marty, connect the finite state transducer. Yeah, exactly. But I like working with finite state machines, and apparently this is a finite state machine turned into something else. But the article starts with a small introduction to what finite state machines are, and they're really easy to get the grasp of. If you look at the article, there's some nice pictures.

And then they talk about how that applies to text processing and and then introduces a concept of a finite state acceptor which i think just accepts things and then transducer spits things out i believe i get in this right how this applies to how to translate your thinking from regular expressions to these and anyway it goes through some of this gives an example using cheese names which i love the gives an example using cheese names, which I love the Python tradition of using cheese names and examples. And then it introduces a library called PyNinny from Google to help build these things up. And at the end, I kind of got a little bit lost, but I wasn't like working through the examples myself. It is something that I think I might

spend an afternoon poking through. Yeah, it is quite interesting, right? I mean, one of the things that makes me really happy about looking at this is one, it's not too complicated and two, it gets you away from regular expressions, which anything that gets me away from regular expressions puts a smile on my face. Well, I actually like really like regular expressions, although it is true that, I mean, there's often times with text processing where you just, it doesn't work when you can't do it line by

line. And I'm probably not going to use this right away, but I appreciate this tool introduction so I can put it in my tool chest and next time I have a problem like this, I might look this thing up. Right, you guys will be looking at something, some dating, like, oh, you know what? We need a finite state transducer. We're going to, we're going to solve this. And I totally want to do that in a meeting. I think we should

put a finite state transit seducer in here. So fantastic. Now it does things like pluralized stuff. It'll, you can teach it. Like for example, you taught it what all the cheeses were and then it could locate sentences that were talking about cheese and do lots of interesting processing. Not just pattern matching. It's cool. It doesn't, but it does start out with a simple example of if I tell it all the cheeses to look through some code

and find the cheeses and add a cheese tag around it or something. There are simple solutions, simple problems that it can solve as well. Yeah, it's great to have it in your toolbox. Well, you know where I would like to run it, though? I don't really want to run it on Python 3.5 or definitely not on 2.7 anymore. Oh, yeah, nice transition there. Thank you.

Yeah, I want to run that on Python 3.6. And you guys, there is amazing stuff coming in python 3.6 and they just released beta 4 beta 4 wow so yeah i don't know what the exact release schedule the final release is but i think it's coming up pretty quickly these things are looking pretty solid let me just give you a sense of some of the peps python enhancement proposals that are coming. First up, it seems like no programming language these days is content with their string formatting. And Python is like, look, we only have like eight ways to format strings. We need another. But this new way is actually pretty good.

There's a PEP 498. And what it is about is formatting string literals. So imagine you've got a string and normally you'd like, there's a variety of ways to do it. Maybe the newer way would to be to do something like say, I want to say, hi, my name is, and you have a variable called name, right? So you might say, quote, hi, my name is curly bracket, zero quote dot format name. Yeah. Well, it kind of takes this idea and just says, well, why do you have to do the dot format and use arguments and orders and the number of them? Why don't you just put the name of the variable in the string?

So now you could say, hello, my name is curly name. And now whatever the value of name is, if there's a variable that will be put into the string, as if you did the first thing I said, but that's really cool. Yeah.

It's pretty cool. The only bummer is you have to indicate that it's this new format string. I think it's an F prefix on the string. But, you know, because otherwise you'd have like dictionary stuff mixing in. Just for compatibility, it's got to have a little identifier. But other than that, yeah, it's a really cool way to format stuff. All right. So we also have another easy to understand but really nice one,

a PEP 515, which will let you put underscores in numerical literals. So if you have a number like 1 0 0 0 0 0, and you look at it, is that a million? Is that a hundred thousand? Is it 10 million? Like it's hard to know like what the separators are. So now you can put underscores. So you could have like one underscore zero, zero, zero underscore zero, zero, zero.

And that would, it basically just ignores the underscores, but it allows you to visually see, Oh, that's a million, not a hundred thousand, something like that. Oh,

that's kind of cool. Yeah, that's pretty cool. No, it's not going to be world changing. One thing that is more significant, I guess is in Python three, five,

we have type annotations, right? So I could have a function and I could have a function and I could say it takes a, let's say it says create, function's name is create, and it has a thing called entity. Well, if you were given that function, you could be like, I have no idea what this does. Like entity, maybe that has to do with databases. But in Python 3.5, you could say colon customer where customer is like a class or something. And then the tooling and whatnot would know

to give you the right auto-completion and various type checking and whatnot for that type, right? But that only applied to functions. You couldn't put it on like variables and other types of class fields and other things that are outside of functions. So this PEP 5 to26 brings that to functions. That seems like obvious.

I don't know why they didn't have that at the start. Yeah, it definitely seems obvious. So, you know, maybe you don't care too much, but tooling will care. Linters might care. So like the tools you could possibly get better because of this. All right. So we have 525 asynchronous generators. Asynchronous away is awesome. Yield, yield from are awesome, but they were not friends previously. Now you couldn't

use them together. So you could either have a async co-routine or you could have a generator, but you couldn't have both. Now you can have both. And similarly, we have asynchronous comprehensions. And you know, actually one of the biggest things is the change to dictionaries coming in Python three, six. What? Yeah. So you might think, well, Python has had dictionaries since from the very beginning, when you add fields or attributes to a class, it's actually stored in Dunderdict. I mean, dictionaries are it's keyword values, like all over the place. Dictionaries play a super important role. Well, in this release from 3.5 to 3.6, dictionaries are now 20 to 25% more memory efficient. They should be a little faster.

They are ordered. So the order that you put things in, they remember that, right? When you first started working with dictionaries, you're like, multiple dictionaries can have the fields show up in different times in different orders just it's bizarre right you can keep switching now it's always consistent they order dict part and making them making them fast is good so yeah it's all good stuff yeah it's all good stuff we've got a new secrets module coming so you have cryptographically strong random number generators. Windows, they've now marked the Python executable as long path aware. So I don't know if you guys are aware, but Windows typically has like crummy 260 character path limits for a long time. There's a lot of legacy stuff that applies that to like newer code.

So now long path limitations are gone in Windows, which if you write code that has to run on Windows, you'll care a lot about that. Well, yeah, actually, I try to write code that I don't have to care about Windows, but... Yes, exactly. So if you go look through the link, the release notes, the documentation there, there's a lot more stuff. I am scratching the surface

on all the cool stuff coming in 3.6. So if people need one more reason to upgrade from 2 to 3, well, here's like 20 or 30 more reasons. Yeah, and I like the peps that you talked about. You left the links to those particular peps in the show notes, so that's great. Yeah, yeah, definitely check them out.

Okay. Hey, Brian. Hey. Before we get to the next headline, let me tell you about Rollbar. Well, tell me about Rollbar. All right, man. So Rollbar is sponsoring Python Bytes, and we're really excited about it. I use Rollbar for my

website. Rollbar is a package you install in your Python web app, Pyramid, Flash, Django, whatever. And anytime there's an error, you get an immediate notification and detailed error reports. So, you know, even though the TalkPython websites handle almost 2 million dynamic HTTP requests per month, and they transfer like four to five terabytes of data, I'm not worried about them. Really? No, not really. Because if something goes wrong, my Slack, my email, my phone, everything will be blown up with notifications.

And, you know, I don't get many, but when they come in, like all the info is there. So I often don't even have to debug it. I just go and fix the problem. That's pretty awesome, right? That's very awesome. Yeah, I love Rollbar. So, and the Python Bytes listeners, all you guys out there, you can have the same peace of mind.

Just visit rollbar.com slash Python By bites and sign up for the free tier. You get like 300,000 free errors tracked. Although I really hope you don't have that many. Yeah, definitely. You should do more testing. If you do, you definitely should. Well,

next up for me, I am a sucker for like, I really like efficiency, developer efficiency. It's always kind of been important to me. And part of that is being able to set up your personal development environment to help you succeed. And I know it's different from everybody, and I'm always tweaking mine. I like articles when people write about them.

So there's an article by Dougal, I think, Dougal Matthews, called Create an Excellent Python Dev Environment. And he goes through installing Python, but everybody knows how to do that. But he's got some goals of like, don't muck with the system Python, and set up virtual environments easy. And it's interesting, one of the things I think is interesting about this article is to start off the bat saying why he has particular goals for why he's chosen things. And that's kind of neat.

Yeah, that's cool because you might look at it and go, I would never do that. But maybe your goals are different than his. And you can really understand where he's coming from because of that. That's cool. Yeah. Actually, the first part confused me because he was covering a project that I'd never heard of called PyEnv. I'm more used to V-E-N-V, which just comes in 3.6 or virtual environment. So this is something different, and I just will have to check it out.

But it's something like virtual environments, but it's independent of Python, sets them up separate, I guess. I don't know. Yeah, but I haven't used it much either, so I might be a little off here. But I think it's kind of like for managing and creating and activating different virtual environments. It's like your virtual environment manager. Okay. I think.

Yeah. Next is a PyEnv VirtualEnv, which is apparently similar to VirtualEnv wrapper, which a lot of people use. I don't use either of those, but I still don't get what they're for. Straight virtual environments work fine. I don't understand what I need more for.

You have different goals. Yeah, that's probably it. The next thing he talks about is a thing called Pipsi, which I hadn't heard of and I just used it today. It's a tool to let you install Python-based command line utilities in their own virtual environments, so you can just set your path to them, so you can use them on the command line from anywhere.

That's really cool, just to have these little virtual environments just for a command line utility. That's really cool. I'd never really thought of doing it that way. That's great. When I was talking in one of the interviews I've done, I heard about more people using Atom than before. And I tried to set up Atom and I needed to give it a path to Flake 8. And I'm like, oh, I'll use Pipsi to set up this so that you can see Flake 8.

So I've done that. Oh, nice. But he does talk about, with Pipsi, he usually includes talks for testing multiple environments, which I highly recommend. MakeDocs, which is a, I think it's a static site generator documentation thingy. I've never used it. GitReview, which I would totally use.

It's a Garrett integration for Git, but I don't use Git right now, but at work. And my personal, I don't actually do code reviews on my own code in my free time. And then FlightGate, of course, which is my favorite, recommended, way too static in the analyze stuff. And then as a side note, one of the things he was talking about for this PyANV is because it's compatible with Phish, which is a different shell that I've never heard of. I use Bash. I tried ZShell for a while. I'm always used, I think it's fun to try new shells. So maybe I'll give it a shot. Yeah, it's definitely worth checking out Phish. That sounds cool. I use ZShell as well. I just actually formatted my computer and I'm like, you know,

I might try a different shell this time when I set everything up. So that's cool. I just actually formatted my computer and I'm like, you know, I'm not tired of ZShell this time when I set everything up. So that's cool. I'll check out Phish. I do love to see the way people put these environments together. And I think it really can help people who are new. It can help you discover things that you don't know about like Pipsi. That's really great.

I was just stuck in traffic today, this morning, and was thinking for some reason about this stuff. You know, other people, I don't know what they dream about, but this is what I was daydreaming about. I was thinking, you know, I might try switching to Conda instead of Pip and try using Conda for managing all these environments. I really like the way that Conda like vets the packages and pre-compiles them for you. I was reading about something. It was like, well, as long as you have this compiler set up, it'll probably be fine, but it might not work so well. I'm like, you know what? Maybe I should just use conda.

And like all those problems are solved by the continuum guys. And so I'm also doing a lot of virtual environments with Python three V and V. And the one thing that I like that I've stopped using the main Python REPL and I'm using PT Python. Do you know PT Python? PT Python. No, I don't. PT Python is,

it's basically the regular REPL, but it has a lot better editing and history and auto-completion and all sorts of stuff for what you can do. So like, for example, if you type in a function in the regular REPL, you know, it's like three lines with indentation or whatever. If you want to edit it, you got to remember to like always replay them

in the right order, you know, up arrow four times, enter up arrow three times, make the change, you know, that kind of stuff. This will like, if you hit up arrow for code suites, it'll bring the whole thing back and you can edit it, stuff like that. Oh, okay. Do you work in the REPL very much? Not too much.

Usually I'm in some kind of editor, but when I am there, I've been moving to PT Python. That's pretty cool. I think I'll check it out. Yeah, yeah. Check it out. So what do you got next? Well, a lot of people have been checking out Python lately and there was an interesting article.

Is it an article? It was an article that's posted as a GitHub repository, which is accompanied by a Go program that's generated a bunch of statistics and graphs. And the thing I'm actually linking to is a Reddit article with like 350 upvotes and tons of comments. So you can check out the conversation. But the headline is awesome. GitHub language statistics. Python is the second most popular language on GitHub for active repositories. That's awesome.

Yeah. I mean, Python's growth is just continuing and expanding, and it's such a cool time to be involved in the language. And here's just one more stat to back that up. Yeah. It looks like the graphs you have to click through from the Reddit to the GitHub page. Yeah, yeah. So I'm linking to both.

Okay. So they say we're going to study active repositories because it doesn't make any sense to think about GitHub statistics if something was put there five years ago and hasn't been touched. All right. So they said let's consider the set of active repositories. Those are created or updated during the period, and the period is like a bunch of dates,

like throughout the last three years, you'll see when you get there. And it has to have at least one star. So people have to, somebody has to care about it. Somebody has to care enough about it to have forked it. And it must be more than 10K, right? So it can't just be like,

well, I put up this one file and somebody forked it or, you know, something silly like that. It's got to be like a real project. So if you take all of those together, JavaScript turns out to be the most popular, but JavaScript, I'm sure is highly overcounted. I'm pretty sure that's generally true on GitHub statistics because almost every one of my Python projects has some JavaScript.

You know what I mean? Yeah. I mean, since it's the front end, it's almost every project has it. Right. Almost any web project in any language is going to be counted as JavaScript to some degree in the language category. And so I feel like JavaScript, while it is extremely popular, I think it's massively overcounted in these types of stats.

And it seems to appear that way in the graph but so python displaced java which we can all cheer for that right that's awesome yeah but let me give you some stats that are pretty interesting that will i think this might blow your mind so back in november 2014 there were 1790 active repositories by the four criteria above you know that were python this is just python. November 2015, there was 2,500. And if you look throughout the year of 2015 and early 2016, it would vary, but it's like 2,000 to 2,500, 2,600. By November 2016, there's 10,944 active repositories.

It's just like, it's growing like crazy. Yeah. I love seeing that. And I think one of the things I like, I think has changed in that timeframe is the, I don't know if it can be attributed to this, but the cookie cutter project has made it a lot easier for people to set up something to share on GitHub. Yeah. Cookie cutter is really awesome. If you guys don't know about Cookie Cutter, check it out. It's a cool project. I'm sure it's not hurting.

Yeah, anyway, that's good. Also, it makes me think that I spent the right time learning a new language. So I started from C++. Yeah, absolutely. It's good to be in a place that's growing. Now, I mean, obviously, take these stats. They're at the grain of salt. They're sliced in all these various criteria and so on.

So they might not be perfect, but they give a glimpse into a world that is doing very well and is very fun to be part of. So I'm happy to see it. Okay, we're up to my number three pick, which is handling Unicode strings in Python. It is from, I know I'm going to get this name wrong, Wanli Song, I think, but we'll link it, of course, in the show notes.

I like it because it's a nice article to introduce people about the concepts of text representation in Python, and this comes up, at least, it hurts my head sometimes because I came up with the 2.7 Python world, and I think anybody that learned with 2.7 and had to switch to a more recent 3X version, sometimes it's hard to understand how the switch happens. And there's a nice table for some of the things that are different. And then it goes through specifics of dealing with Unicode

in things like IO boundaries to databases or services. And then also a few other examples are dealing with it, with logging, with JSON encoding, and with Redis even. A nice pointer to a 2012 article from Ned Batchelder called Pragmatic Unicode, which I hadn't read before. So I learned a lot about Unicode in this last week. Oh, yeah, that's great. If you have to juggle that stuff, it can be tricky.

I remember talking to some of the web foundation guys. I think it was Kenneth Wright and requests, but it could have been Armin Roeniger and Flask. I'm not sure. I think it was Kenneth. Anyway, seeing that one of the most challenging things of upgrading their projects from two to three or supporting three let's say is the dramatically different way that bytes go to strings and and all of this kind of

stuff so yeah if you've got to upgrade something that's two to support three, be really careful with the strings. Yeah, and this is another one of those. I'm going to bookmark it, and if the next time I come up with an issue, I'll look through this, see if it can help me figure out how to fix it. Yeah, awesome. All right, what do you got as your last one, Michael?

My last one, I think this one might be a little surprising for you. Let's see. So I've asked many people what their favorite editor is over at TalkPython to me. And I've heard many answers. It would be fun to go back and actually do some data analysis and graphs or whatever, right? But I would say Sublime Text, extremely popular. Emacs, very popular. Vim, I think Vim might be slightly beating Emacs. PyCharm is pretty popular. People hear me go on and on about PyCharm. I love PyCharm.

Really? I didn't know that. Yeah, most people don't know this about me, but I actually do use it now. I do love it a lot. But one thing I came across with was I was fooling around with Visual Studio Code. And this is not like Visual Studio from Windows, the big.NET thing. This is like a lightweight editor.

It's based on, I think, the same underlying stuff as Atom. Okay. It's using the Chrome as its underlying engine and is based on some client-side HTML stuff. Anyway, it's a free editor. That's pretty cool. And it's growing quite a bit in terms of its extensions you can get for it. It's got like cool get integration, and now it's getting better Python integration. So my pick is this thing called the Python extension for Visual

Studio Code, and it runs on Mac and Windows and Linux and so on. And I have a link to the one there. And what's really interesting is, well, the bunch of features, which I'll tell you. But if I had to ask you, Brian, how many times do you think this thing has been downloaded? 23. 23 is good. Well, I would guess thousands, maybe 20, 30,000 times, something like this, right? Like, okay, how many people are installing full Python support? This is like not just standard completion, but it like reads,

if you're writing a package, it like understands your package and will tell you exactly what all the names of all the elements symbolically in your package. It'll do like AST-based refactoring, like really cool stuff, right? Anyway, 850,000 installs. That's actually incredible. Now I want to go try it. Yeah. I mean, when you think of Python editors, almost a million people installed this Python support for their editor. Now, of course, not every one of them is doing Python all the

time and there's not a one-to-one mapping between developers and editors, but this is like way more popular than I realized. And so I thought, well, why don't I play with it? So I played with it this week and I'm pretty impressed for like lightweight stuff. I think this is my new editor. Oh, okay. I was just actually picking it. I installed Adam this morning and maybe I'll try this one out as well. Yeah, they're really, it's really similar to Adam actually. So it's worth giving it a try.

So it has linting including a bunch of linters you can configure including flake 8 it has full-on auto completion as well as support for pep 484 i think that's the type hint it has jupiter support intending code formatting refactoring like i said, code navigation, debugging. You can remotely debug over SSH to like a Flask app. I mean, that's pretty serious for this little plugin, right? That's awesome. Unit tests, all the various tests you're talking about, lots of good stuff. So anyway, I thought this is kind of one of the unsung heroes of the editors. So I thought I'd highlight it.

Yeah, cool. And maybe the next time you ask somebody, they'll say, I use code. Oh, yeah, you use Visual Studio Code all the time. Anyway, yeah, it's pretty cool. So I'm excited to try it out, at least. Okay, well, I think I'll try it out, too. And that brings us to the end.

But we do like to catch up with what both of us are up to. So, Michael, what are you up to lately? Well, I've been continuing to crank out the Talk Python to Me episodes. So the last one I just released was parsing horrible things with Python, which I wanted to make sure I pointed out here because I think that's a really interesting tie

into your first pick of the week because it's another way to parse horrible things that I think is really actually quite powerful and interesting. So anyway, I released that with Eric Rose, and that was cool. That was a really interesting episode. I really enjoyed it. Eric is obviously a brilliant guy.

So that was good. Yeah, thanks a lot. Yeah, he was a great guest and definitely enjoyed talking to him. You last week had asked me if Awesome Python, the GitHub thing, was the same as the Awesome Python newsletter. And I said, I think so. And now I think I might be wrong. No, in fact, the Awesome Python newsletter comes from python.libhunt.com, which is a really cool place for Python news. And I think one or two of the news items throughout the podcast, not this one, but over all of them, I've gotten from that place.

Okay. So, yeah, that's. Okay. So yeah, that's my picks. What's up with me? Python is so awesome. We like to slap it on everything. So it's awesome. Yeah. How about you? What are you up to? Man, I've been doing a lot of writing. I've got a deadline, an editor deadline in about a week, actually exactly a week from today because I'm, I'm trying to put on full steam and it's a, the pie test book and I'm trying to put on full steam, and it's the PyTest book, and I'm trying to get it ready so that it can be printable, and I can take a stack of copies with me to PyCon in 2017 in May. So that's what I've been working on.

The last episode of Testing Code is recorded and partly edited. This was number 25 with Dave Hunt about Selenium, and there's some really good content mixed in with some really stupid questions that I've asked. So I need to go and cut out the stupid questions and then it'll be good to go. Hopefully we'll get that out in the next couple of days. And then I'm really excited about an interview tomorrow scheduled. Hopefully it goes through with a person called David Hussman. He runs a place called Dev Jam Studios.

He also talks about Agile and goes around and speaks to people. He's a really good speaker. Looking forward to talking to him. That would be awesome. Slightly above the code talk about all the testing and whatnot, right? I don't even know if he'll get into testing.

One of the things he likes to talk about lately the testing and whatnot, right? Yeah, and I don't even know if he'll get into testing. One of the things he likes to talk about lately is story mapping, which is sort of high-level planning for an entire project above the writing stories level. And that's pretty neat. Yeah, that's cool. Definitely precedes code. Yeah. Well, anyway, I really enjoyed talking with you this week and look forward to

seeing what cool stuff people put out in the upcoming week. Yeah. Thanks Brian. I enjoyed it as well. And like always, I learned about some cool new things. I'm definitely checking out Pipsy and thank you everyone for listening and we will see you next week. Bye Brian.