Python Bytes - #110 Python Year in Review 2018 Edition

Episode Date: December 26, 2018

See the full show notes for this episode on the website at pythonbytes.fm/110

Hey folks, Michael here. Brian and I put together a special podcast episode for the top Python stories of 2018. We originally recorded this as a Talk Python to Me episode. That's my other podcast. Since it's a perfect match for what we do every week here on this show, Brian and I decided to share it with all of you

for our end of year countdown on Python Bytes. We hope you enjoy the episode and thank you for making this show so successful in 2018. We promise we'll keep it going strong into 2019. Brian, Dan, both of you, welcome back to Talk Python. Thank you. Hey, thanks. Great to be back again.

Yeah, it's great to have you back as always. It's going to be so much fun. And it is time to look back on 2018. Can you believe it's the end of 2018? I just remembered how to write 2018 and not 2017 on all the stuff I have to sign. Yeah, it's gone by fast. It definitely has. It definitely has.

So it's good to have you back. We're going to go through what the three of us have decided are some of the top stories in the Python space for 2018. And I just want to quickly throw out there for folks listening. This mostly does not include data science stories. There's a little bit here and there, but it's mostly pure Python. And the reason is I'm also doing a show on data science year in review. So save those big stories for that one. Don't want to overlap. All right. Well, we're going to start and do a top 10 countdown. And we're going to go from really

interesting to the absolute big news at the end. And you can see what story we have picked to be the number one story of 2018 for the Python space. Let's do a little warm up here. And Dan, I'll let you kick this off. What was our number 10 item? Sounds good. So our number 10 is the Python 3.7 release and more specifically the performance optimizations in there. And I know you had a really good show on the podcast here with Anthony Shaw, where you guys talked about, you know, some of the changes in 3.7. And I just thought that 3.7 should be on this list because it was a really, really good release. You know, a lot of like free performance improvements. It's always nice to see that. And it's just been

awesome to see Python 3 and see Python 3 specifically get so much love and that it's under active development and it's making this forward progress. And I think it's also the release or I think it was either 3.6 or 3.7, where Python 3 is now faster in all of the benchmarks than legacy Python, except for startup time. So I thought, you know, that's kind of a nice milestone for Python 3. I think that's a huge milestone for Python 3. I mean, this is sort of the year that the tide has turned for Python 3 in a big way. I think it was starting to turn before, but really this year, I think that's one of the themes. And this is a major aspect of it, right? That Python 3 is now faster.

There were always these couple of reasons people say, I'm holding out to Python 2. You guys with your fancy Python 3 and your new keywords can just go play because my code is faster and faster beats everything, right? Well, maybe not. Yeah, well, I mean, faster never really beats everything. It's faster development time too. And when you have better data structures, you can sometimes develop faster. However, you're right that a lot of people were holding out because of some of the speed.

And for some applications, that makes sense. But yeah, I think 3.7 is definitely faster. And, and the other thing is a lot of people had already, some people for some of their projects had already switched to Python 3.6 and that was probably from 2.7 to 3.6 was not an easy switch. Having a switch from 3.6 to 3.7 be like, oh, I don't have to do anything. It's just better now. That was a nice thing to have. Yeah, that's super nice. And I think that's a really great point you raised, Brian, that, you know, careful what you're optimizing for, right? Like if it was really all about speed, wouldn't we just be writing assembly code still, you know?

Yeah. You know, I bet somebody good at writing assembly could crush the performance of Python or a lot of C++. But, you know, they would release an app, a small app every 10 years, and it would be buggy, right? Like there's just, it's not the most important thing. That's for sure. One of my favorite examples there is, what's this game called? It's Rollercoaster Tycoon, which was written by one person in Win32 x86 assembly. It's like 100% assembly code.

And it's actually an amazing, amazing game. And it just blows my mind. Like everything was written in assembly language. Wow. And it probably runs like totally smooth on like a 286 or something, right? Yeah, I think you needed a Pentium or something. Pentium. And now somebody's working on like a Python,

going to need, I don't know, multi-core CPUs and whatnot to run at the same performance. Yeah, who knows? But, you know, another interesting thing that's coming out is more and more talk about actually compiling Python, which I think, you know, didn't make the list, but it's definitely something that came up. Like MyPyC, for example, and Cython and whatnot. Yeah, yeah. So if we're going to release a new library or an awesome game like Roller Coaster Tycoon or something like that, and we were going to do it in open source, Probably the main full release after it's like totally stable and spent out for a year would be like 0.01 or 0.03, maybe like a 0.12. I don't know.

Something really, really small like that, right? And that's been a theme that's gone through a lot of open source. So I think I'd like to highlight a little bit of shift in just the versioning that seems to be happening around projects. I want to kick off this part of the conversation with talking about Mahmoud Hashemi's project called ZeroVer. Brian, do you remember covering that on Python Bytes? Yeah, he released it like as an April Fool's thing, but it actually highlighted a lot of things about different projects that were zeroed over after like, you know, years. Yeah, so the idea is kind of like I was describing at the beginning, like so many projects just never even reach version 1.0. And in some sense, it doesn't matter, right?

Is the version number smaller or bigger? That tells you whether you have an older or new one, right? So that's fine. But there's a lot of people, especially coming from the enterprise space, or from the commercial software space that sees things like 0.20 and go, oh, that thing's like a super alpha, we're going to stay away from that until it reaches 1.0. And I think it has a tangible effect on turning off a certain category of folks, especially people who are maybe unfamiliar with open source and things like that. Yeah, I think definitely. I mean, a lot of people do commercial software development, like I do, and the zero versions almost always mean the API is in flux.

You can't trust it. You shouldn't program against it because the developers can change their mind at any time. And it isn't until the 1.0 version where we have a stable API that people can count on. Yeah, and so I think a lot of people perceive it that way, right? Yeah. It's interesting to consider the version number as part of the user interface, you know, for an open source project, which makes absolute sense

because of those effects where people are like, yeah, you know, is this really ready to go here? Is this an early release or whatever? I mean, I'm wondering if it's really going to have a big impact, you know,

if people are moving to something like dating their releases, like just calling them 2018 dot something something, and you sort of get the November release. But it's an interesting experiment for sure. I mean, it definitely looks a little bit nicer

than having, I don't know, like somebody put Pandas here, which had like 83 releases, and it's been running, or it's been active for eight years, and it's sitting at 0.24. Dot zero, dot dev zero. Dot dev zero.

Nothing wrong with that, but I can see how it turns some people off. I don't know how dev zero means production, but to me that just says like, dude, stay away from this. Don't ship this. That's scary. Yeah, I think maybe the grandfather of zero ver

might be putty. That little SSH tool you get for Windows because it doesn't come built in. That little trade app. It's 20 years old, and it's 0.7. One concern that I have with moving to the date-based or year-based version numbers is how, like if you have the semantic versions, you can do things like, okay, give me all the minor patches for this package and all the security updates, but I don't want to jump to the next major version, which maybe has breaking changes in it. And I know that's not a perfect system.

Generally, you want to pin your packages, but it's kind of nice to have the ability to do that. And with, you know, who knows what's going to happen in like July 2018 or something, 19. Yeah, absolutely. So if you look at some of the tools that I know you use and I use, for example, I use pyup.io and use something slightly different, but they let you go to your requirements file or your pip file and say, I want you to keep this on the latest pin version possible with this constraint, where you could say like 0.1.star or something like that, right? Right, yeah.

But how would that work with um you know date based exactly so right let's dig into that a little bit so what are the two options i guess the two options are just like to start shipping like full versions like django is now django 2 and the django 2 the main change from 1 to 2 is we're dropping python to support which is kind of ironic maybe it should be Django 3, but whatever. You know, it indicates a major change like you were saying there, right? So one of the projects that's switching

to calendar-based version is Sanic, right? The async web framework. And what they're doing is they're trying to ship a version, a major release every quarter. So they might be, you know, 2018.07.0, then.1,.2 for like incremental changes. But they're also doing LTS releases around their calendar version. So every December is an LTS version. So 2018.12.0 and so on would be supported for a year and would be the LTS equivalent, the long-term support equivalent, if you don't want to switch to something beyond that.

That's how that group is handling it. Interesting. Yeah. I kind of like that. Then you can glance at a version. And, I mean, if everybody sort of supports something like that, you can glance at all of your versions and go, oh, everything's within a year or so. So it's probably fine. Exactly. Like like brian you know a lot about testing if i told you i'm using web test.2.0.32 is that new is it super old is that a beta version like what is like what is

that right you don't know yeah but if i told you i was using web test 2019.1.0 like oh you must be like testing the the new one that's coming or, you know, something. It just communicates it. So Sanic is doing this. Pip is doing this. And then you threw some others in there, Brian. What are these? I'm not familiar with them.

There's some descriptions of calendar version and semantic versioning, calver.org and semver.org. I love that those are domains. I think Mahmoud did at least one of them, maybe both. He has this knack for creating these little sites and these little projects that just pop out of nowhere. But one of the things I want to point out is I was talking with somebody about calendar versioning and that versus semantic versioning. And one of the things is that the major number, whether it's two or three or it's date-based, often means you're not going to remove features, so you promise not to break customer code within a major version.

The date-based kind of says we're at least going to say it's safe for a year. Yeah. Whereas semantic versioning, you know, a development team can go through major versions as fast as they want to if they feel like breaking things fast. That's true. That's true. It's just when you get the feeling to switch the major version from two to three or three to four, you just do it, right? Yeah, but it has impact. So yeah, I think there's some interest there. So I think another thing that was really interesting about the calendar versioning that I learned talking to Adam Hopkins from the Sanic project was that there's this sense of you need to keep shipping, right? Like if you have a calendar version and it's from last year and that's the latest version, that sends a message in and of

itself that this project is not getting a lot of activity. Whereas without it, you know, you kind of got to be more deeply familiar the project to understand its rate of improvement yeah i think even it'd be cool to have people even if there's not really any changes needed just to release it and say hey we tested it it's this has all been tested on the new versions and you're good to go we updated like the readme file or something right well there's almost always like some minor little thing that can be improved and shipped, right? Another story of 2018 and maybe slightly before, but really was coming in strong in 2018 is Python's kind of popular. Yeah. So Python's very popular. It's taking over the world.

There was a neat economist article that came out and I guess that's just sort of related to highlighting a trend. And the article was, Python has brought computer programming to a vast new audience. But some of the things they point out within there is that right now, according to a Stack Overflow survey, 40% of professional developers use Python, and another 25% wish they did, which is interesting. Just go do it, man. Also, they interviewed Codecademy, which is an online training site, and their biggest increase in demand for 2018 is with Python.

Yeah, that doesn't surprise me. I mean, you think about a very fast-growing technology that also has a lot of career possibilities. Definitely people are going to be wanting to learn that, right? So if you go to somewhere like Code Academy or any other online trading place, like, there's going to be, unless they're doing something wrong, there should be a lot of demand for Python. They noted that since 2014, at least from a survey in 2014, Python was the most popular intro language at American universities. But that was amongst people studying sciences or computer science or something, engineering. But what we're seeing now in 2018 is the growing movement of people that are not traditionally think of themselves as programmers, primarily learning Python.

And we'll see that in data sciences. Embedded development is now moving towards Python in a lot of maker senses. And then also the article also pointed out that more and more people are just sort of worried that all jobs will be taken over by automation. So they want to be part of that. So in order to learn some automation on your own, Python is a they want to be part of that. So in order to learn some automation on your own, Python is a good way to be part of that. You either control the robots

or you get replaced by the robots. Take your pick, right? Yes, exactly. Yeah, also to throw in there, I saw this on Nina Sakarenko's Twitter feed recently. So Python is becoming or has been nominated as the official programming language for education in France.

So I think it's going to be part of the high school curriculum there, which is that's super cool. I think we're seeing the snowball effect, you know, where people it's really becoming the dominating language for beginners and educational language. And I think that's super cool. That is so cool. And when I saw that on Nina's post as well, I'm like, oh, that is amazing. And, you know, I've been thinking a lot about this. Like, why is Python both popular as a teaching language and popular as a professional programming language? Like, C++ is popular as a professional programming language, but they don't suggest that kids start with it, right?

Things like this. I think there's this special balance that when the language was created, it has this way to get started easily and to pull in just the aspects that you want, right? So you can start out with not even functions and then like, okay, I need some structure. I'm going to put some functions in here. Oh, let's start working with classes and things like generators, but none of that, you don't have to understand or use or care about that until you're ready. So like Python has a special ability to start easy and small but expand into professional type of apps. Whereas a lot of them are either you're always a beginner type of language or you're always like, yeah, that's a void star star, kids. Let's talk about that.

Totally. Actually, Mike, I think you called it a full spectrum language uh way back when when we uh did an interview on my my blog with you and um that just really stuck with me because i think that's that's so true you know it's not just it's not like you're learning um you know like the little like turtle like logo or something like that it's like you learn the basics of programming then you're sort of stuck and you have to move on and learn something else but no you can you know all the skills you learn working with python they they kind of scale and grow with you. And that's just a beautiful thing.

It's special. There's not many other languages like that. What a time to be alive. It really is an amazing time to be alive. Before we get on the next one, let me just tell you all about DigitalOcean. They're doing all sorts of cool stuff. Our infrastructure runs on it. Really, really nice and reliable. One of the things I want to highlight this time is their

work with Kubernetes, Docker and coordinating Docker, orchestrating Docker stuff with Kubernetes is a big deal these days. And so they're launching a new Kubernetes cluster over at DigitalOcean. So a really nice way to manage and deploy your container workloads in the cloud. And if you go to pythonbytes.fm slash DigitalOcean and you're a new user, you get $100 credit to Kubernetes all the way if you want. You can run a lot of Kubernetes for $100 in the cloud. So that's pretty awesome. Yeah, very cool. Yeah, so check them out, pythonbytes.fm slash DigitalOcean.

They're big supporters of the show, and they keep us going strong each week don't they yeah i'm very grateful for them yep brian you you touched on that a lot of the growth of python was centered around people who are using python as a tool maybe not as their identity right like python developers like i'm a python developer but other people are just using python maybe for data science or things like that. And so number, number seven in our list is that I'm going to declare 2018 as the year that the number of data scientists have come to equal the number of web developers in Python for professional Python developers or users, let's say. So, you're going to have to back that up a little bit. All right. So there was a really cool survey done by the PSF in conjunction with JetBrains. So JetBrains did the infrastructure for sort of analyzing it and hosting it and so on, but it was promoted just through the PSF and other public sources. So it's not like tied to JetBrains,

but they did this really great survey and it's called the Python Developer Survey 2017, which might sound like I'm wrong in my year, and sort of I am, but this is the very end. This is like, November, December, or something like that of 2017. So it's pretty much there. And they did a bunch of interesting, they asked a bunch of interesting questions. And they found out a lot of. Like, for example, four out of five Python developers use Python as their main language. So when you see people are using Python, mostly they're using that as their main language. It's not like, well, I write in C, but I also have this script that like compiles it in Python or some weird thing like that. Right. So the majority of people using Python use it as their main language.

And that's, that's pretty interesting. But the thing that I want to point out is they said, what do you use Python for? And if you pick where Python is main versus where Python is secondary, you get slightly different answers. But generally, it says for data analysis and data science, 50% of the people use it for that, and 49% of the people use it for that. And 49% of the people use it for web development. And then there's a whole bunch of other areas that like, you know, it's a multiple answer type of thing. So they've also checked those off. But in this survey, basically, they said 50% of the people said data analysis, 49 web development. Does that surprise you guys? No, not really. I think it surprised me. I know

there's a lot of data scientists. I feel like on a web development team, there's more participants than on a data analysis team. I feel like a data analysis is often done by a smaller group of people than like the 30 people that maintain the website or, you know, whatever, right? Like that was my conception. I don't know if that's accurate. But apparently, no. Either there's even more data scientists than I guess, or there's larger teams of data scientists and smaller teams of web developers. So this is all interesting.

Another thing that I thought was pretty interesting around this is they asked people to guess that ratio in this survey. And they gave them like five choices, four choices, something like that. And they said, what do you think the ratio is? Five times the number of web developers to every data scientist? One to one? You know, maybe worse. I can't, like, ten to one. I can't remember.

But when they asked that question to the group, half of whom were data scientists and half of whom were web developers based on the original question the responses they got was only one only 10 percent of people got it right 20 and um most thought that it was five to one five times web developers to one data scientist so i think the general perception is there's more stuff happening on the web around Python than there are data scientists. But I think also what the most surprising thing to me when I saw that result was half of the people that answered that question were data scientists, you know? And so to me, my interpretation is that means that a lot of the data scientists feel

like they're not as big of a representative group of the community as they are. They feel like, oh, we're like 20% of the community. But in fact, they're like half. And I don't know why that is, but I think it's interesting. It is interesting. Well, there's definitely like a, if you look at the hype around different packages, for instance, there's more packages that are web related than there are data science related. Mostly, I think, because a than there are data science-related. Mostly, I think, because a lot of the data science tools have been bundled into these large groups of tools.

Right, like Scikit, yeah. Yeah. So they have a common set of things that they work with. Data scientists like to all use sort of a handful of different tool chains, whereas web developers like to go off and create their own asynchronous library. Yeah, everyone's got their own library

to match their new JavaScript library. Yeah, I wonder if the Anaconda distribution and things like that also play into that feeling, right? You just install it and you just have the tools, right? You don't necessarily go and hunt for every little library as much. This sort of reminds me of, who coined that term? Scott Hanselman, I think,

the concept of the dark matter developer. Yes, I was thinking that too. Yeah, yeah. I think you actually told me about it. So I guess the idea is that there's a large segment of the programming population or people who program for, even for a living sometimes, that are just not visible in the community. know get their work done and it's just not maybe they don't identify as a programmer or it's not just something it's just not something that they do you know publicly or have like side projects and stuff it's a job wrong with that yeah and most people like they don't go to conferences or go on social media and go crazy about their job like like if i was a barista nothing wrong with that but like i probably

wouldn't go to barista conferences and go to barista meetups and like, you know, talk about being a barista on social media very much, right? Like it just started a podcast about, yeah, you actually write some awesome stories. If you did that, make coffee for me. Hey, anyone out there listening to go for it. That's all you. I'm going to get that. I would listen to that. I think there's a lot of, a huge number of companies that have one data scientist or data analyst in their company. And so they feel outnumbered because they're the only one in their company. Or there's like the ratio is very much skewed within their company, but they're kind of all over the place. Yeah. Yeah. I totally agree. And I do think there's a little bit of this dark matter stuff possibly happening here. Maybe you're not allowed

to talk about your analysis that you're doing. So you just don't, maybe you've backed into becoming this data scientist business sort of analyst type person. You started out as like a trader and then you, you kind of were the person that had enough code to like figure out how to do the overall reporting and analysis and whatnot. And you still don't think of yourself as a developer, but you effectively are right. So I don't know. I think there's a lot of interesting angles here. Yeah. Yeah. Also, I was just thinking like, you know,

a lot more secrecy maybe in the, in those areas. Like if you're working on some state of the art, like machine learning thing, I don't know, self-driving cars or whatever. Maybe you don't necessarily want to go out there and share with the world what you just did whereas in the web development work i feel like we're sort of you know we figured most of it out and so people are more willing i think to to share what they're working on at least that's the impression like and often it's yeah i agree and it's on the web already yeah whereas if you're working on the apple self-driving car like you can't even acknowledge existence right you're like working for the cia yeah i work for the state department

okay so anyway if you want to dig into more of this uh i had eva jelawaska and dimitri filipov on they're the two folks who did the this and they printed it in a python so on talk python 176 all right item number six b, what do we got? We have a darkening of the community. This is a very dark topic. It is a little dark, but kind of in a good way. Black is taking over. Black is, there's, and not in a bad way.

So the Project Black is a code formatter that just sort of formats all your code for you. And like PEP8 and some of those Flake8 and stuff tools were to tell you where places where you might want to look at your code and change it. Whereas Black is, yeah, I can tell you if you want me to tell you, but I'll just go ahead and change it. That's awesome. Sort of taking, there's a lot of projects around the web or around Python that are picking it up and using it as their standard. And I think it's been definitely very interesting

in 2018 watching this happen. I love that tool. So it's so good, like, especially if you, if it's a community project, you're working on a project with several developers and it's just so easy to standardize everything and you don't have to have the back and forth conversations of defining your own code style and i mean assuming you like the output that

black produces which i personally really like and it's it's been super helpful for real python we have like a materials repository with some sample projects so like every single folder in there was created by a different person by different author and i went down that that route of okay, we're going to define like a beautiful code style and then tell people how to format everything and make sure it's adhered to. And that's really, really hard to do. And so I just ran black on the whole thing, set it up on CI. So it'll check the formatting, problem solved. And it took 15, 20 minutes to do that. So yeah. But what a lifesaver, such a great tool. So a couple of things about it. It's taken, it's one of the things that takes things like style and format and stuff like

that and takes those out of code reviews. And so code reviews can just be focused on algorithm and design and not on whether you use single quotes or versus double quotes or something. The other thing is Black shows an example. It has part of it is a pre-commit hook so that you can add a Black pre-commit hook to your code. So everybody just as they're checking it in will have Black run on their code. And that highlighted the pre-commit tool for me. And I think pre-commit is being used more and more because of Black using it.

I've heard that more than once. That's pretty awesome. You're like, wait, what's a pre-commit hook? That thing exists in Git? Yeah, definitely. Nice. I like it. The slogan, like you can have it in any color you like as long as you want it in black. Yeah. I didn't know that that was from Henry Ford until somebody told me about Henry Ford saying that about the Model T. Does it have like a theme song or what's going on here? Oh, well every time I use Black, I hear Black Hole Sun from Soundgarden. So I went ahead and just mentioned that in the hard notes. Black Hole Sun

is, Soundgarden was such a good band. They really came from like a special era of music. We had them, Pearl Jam, Alice in Chains, Dinosaur Jr. Like there's a lot of good stuff back there. But this is the only one that seems to be making it to a Python package. So way to go.

All right, Dan, for item number five, I feel like for a really long time, it was like, well, pypi.python.org slash pypi is a little bit out of date and a little bit crummy, but we're working on it. And it just, it seemed to be in this like stasis of like half improved forever, but not anymore, right? Yeah, the new new PyPI, new Python packaging index went live this year. And what an amazing improvement. I mean, like, you know, throughout the whole project, and I know you had a really,

really great show, where you interviewed, you know, a bunch of people working on that transition and launching the new and improved PyPI. And it's just crazy. Throughout the whole thing, it's been rewritten from the ground up the way I understand it. It's new hosting infrastructure. And it's much more friendly now for new contributors. Beforehand, I think it was sort of like a homegrown web framework because, well, it was that PyPI. Old PyPI was created before web frameworks really were a thing in Python. It was like, here's the socket.

Let's start with that. Like, wait, wait, wait, wait, what? And so, yeah, I just love it. You know, like every time I'm searching for a package now, it just looks so much better. And it looks like, it looks good. You know, it's like when you compare it with JavaScript, like the Node.js world and NPM,

I felt like, yeah, that was sort of the gold standard for a while for what an online package repository should look like the the note js world and npm i felt like yeah that was sort of the gold standard for a while for what a online package repository should look like and the features it should have and i feel like pypi is now playing in that league and um it's super nice i think it is super nice as well and it's silly to say that it matters that it looks good because it's just pip install a thing but i think it really communicates a message to people who come into Python or are using Python, but especially people who are new and they're like, oh, what is this stuff? Oh, there's a package injection.

You're like, whoa, that thing is like from the 90s. What is that, right? Like that, I mean, that communicates something about how much the community cares for that kind of stuff, right? Yeah, they also rolled that over with like everybody in the world using it and very little hiccups. Yeah, it definitely is.

Really impressive. Yeah, it's super impressive. So a couple of interesting notes here. When they rolled it out first, it was PyPI.io. And I thought, oh, they're just being part of this hipnew.io. You know, it's like the.ly of yesteryear. You know, it's like the.ly of yesteryear, you know?

It's like Livia all of a sudden became a super awesome place to have, like, your domain name because it had ly in the end. And so io, right, is super popular now. But now it's pypi.org. And the story is the pypi.org was owned by somebody else, and it took a long time for the PSF to get it from them whereas they could go and buy pi.io so that was like an intermediate step in the whole rollout yeah some of the the nasty details of web development exactly and then the other one is this is a pretty interesting example of a popular pyramid web app right they they started out in flask and actually said no we'd rather go with pyramid which i thought was a pretty interesting choice being a fan of pyramid myself and then brian finally marked down right yeah so the the old pi pi was did not support readme mark.md it only was uh readme.

what is that rsl or something or yeah rst yeah rst yeah so i try to avoid restructured text as much as I can. No offense to everybody that loves it. Yeah, I hear you. It's like Markdown, but it's broken. The stuff you put in there doesn't quite work. Just kidding. No comment. Just kidding, just kidding.

I'm just saying the Markdown stuff doesn't always work in there. All right, number four. Item number four, Brian, what do we got? You touched on this a little bit before. Yeah, I think it's been fun to watch. It was unexpected, the rise of Python in the embedded world. We actually did a segment on this on Python Bytes, but there's so many in partly from the maker movement and a lot of people trying to get into controlling their, not just their computers, but their world with Internet of Things and also just playing with, you know, adding lights to your bicycle and things like that with microcontrollers and hardware hacking. Adafruit's been a big part of that, of pushing it.

But there's a lot of projects around Python and embedded. MicroPython is one of those. And MicroPython's being used by both hobbyists and serious developers. Hobbyists are serious developers too, but I think some companies are using MicroPython as well. Adafruit has its own version of this called CircuitPython. Actually, I don't know if it's related to MicroPython or not, but Adafruit has CircuitPython. And then MicroBit is code Python-based for the Microbit processor.

And then we even have Mew, which is an editor that directly interacts with microcontrollers running Python. It's been pretty fun to watch this year. I really think the rights of IoT and Python is just taking root now. And MicroPython is so interesting to me. I mean, the fact that I can take a Lambda expression and tie it to a hardware interrupt just like

never ceases to amaze me, you know? Yeah, that's really cool. And so, and a lot of this is because somebody wants to learn electronics or something, but they also want to make it do things. So you have to learn a programming language. And it used to be, we taught people C and like the barrier to entry for C is huge. Having a Python be able to do that is wonderful. Yeah, it is. So you talk of the IoT stuff. I learned something from Matt McKay recently on an episode, and I want to share this with you and just see what you think. So you guys click on this link here. So here's an IoT device using Python. And the idea is, the title of the article, this is on the Twilio blog, is how I potty train my kids using Twilio and an AWS IoT button.

So it's a little button like those Amazon Dash buttons. And he put it near his son's bed. And apparently there are kids sharing the room. And wanted to make sure he didn't wake up and make a lot of noise and wake up the other kids. So he put this button that basically has like a poop emoji and you click it and it says, I got to go. And the, the,

the parents would come in and like help out the kid while he was learning. And this is written using like the Python API on like AWS Lambda and stuff like that. So yeah, the rise of Python and IOT. So Q4, 10 years and his kids are like, dad, did you that. So yeah, the rise of Python and IoT. So Q4, 10 years, and his kids are like, Dad, did you have to put that on the internet? This is awesome. I love it. Exactly. Yeah.

This is great. Isn't this funny? So yeah, I mean, funny and like an amusing, but also like, actually, that's kind of helpful. I can see how that would be helpful as a parent. Because you definitely don't want your kids waking up and you have to be up for a few hours in the middle of the night. So if an IOT button can help that, then get the IOT button. All right. Item number three, and it's packed full of goodness here, you guys. So legacy Python's days are fading. What do you think? Well, I haven't used it for ages, so. Yeah, I haven't either.

Dan, is real Python running on like what, 2.6? Yeah, not a lot of legacy Python for me anymore, thankfully. That's awesome. Yeah, I mean, it's kind of crazy that it's sort of crept on us, the Python 2 end of life. And now we're not that far away anymore from it. Yeah, we're actually super close and the official item that i'm linking to as this piece of news is guido von rossum's official statement so i don't know four years ago or something it said yeah python 2 it'll be done in 2020 whatever right like that's like a broad range for a period where a thing stops a whole year.

So we needed a more clear date. People are speculating maybe the date of PyCon might be a cool one. Maybe the end, maybe the beginning. So he came out and said, let's not play games with semantics. The way I see it for the situation for Python 2.7 is that end of life is January 1st, 2020. And there will be no updates, not even source-only security patches after that date, period. That sounds like the end. Yeah, it's a good end, though, because the, and some people might complain about the whole security patches, but this is volunteer-driven,

so it's volunteers' time, and I think that to let them work on new things is good. Yeah, absolutely, and it's not like people are surprised, right? It's not like, wait, last month they didn't, nobody told us there's going to be a new Python. It's been going on for 10 years. This will make it 12. It's time. Yeah, and there's a couple of loudmouths to a podcast that have been talking about it for a couple of years. Yeah, I think we're actually making a dent with this referring to Python 2 as the legacy Python. I think it's making a difference.

Not everybody loves it, but I know some people it's been working for. Do you guys know, you know, with the no more official security patches after the end of life date, I mean, obviously this is a big business opportunity for other companies to come in and offer, you know, long-term Python 2 updates and stuff.

Does anybody know who's doing that right now? Like some Linux distributions, I would guess, but I don't really know who's doing that right now? Like some Linux distributions, I would guess, but I don't really know what's going on in that space. Yeah, possibly. The big news that I just heard, so, you know, there's Red Hat Enterprise Linux, right? And they do a lot of that kind of stuff. And the versions that they have live on for a long period of time, but they just announced that they're dropping Python 2 support from Red Hat Enterprise Linux, which to me sounds like, okay, even the laggards are stepping forward. So I think that that's pretty big, but I suspect that there's still a lot of support on the

backend. There's been a few people who've like talked about forking Python 2 and just supporting it themselves as a single individual, but it seems, I'm sure there's going to be some of that, but like as a company, like, let's say, I don't know, your bank of America, but it seems, I'm sure there's going to be some of that, but like as a company, like let's say, I don't know, your bank of America, you're like, dude, we don't need to upgrade. There's this guy. He forked it. We're good. You know what I mean? Like that just doesn't seem realistic. It's also not just the C run, the C Python runtime anymore. It's like so much more than that. So Django no longer supports Python 2, like the scientific stack, like NumPy and stuff are dropping support for Python 2. So even if you still have the older stuff, you know, you're like, well, we just keep running.

It's not just about what's running your code. It's also what you can use in your code. And, you know, that's going to start looking like it's just zooming off into the distance. So not good. Well, I mean, I think Python 2 will actually live on forever in some situations where, for instance, you've got something that doesn't need a security patch because it doesn't touch the internet at all.

And as long as it will run on whatever CPU you're sticking it on, some people have a huge program that they never touch because the requirements never change. And they use it every day, but it doesn't need to be flexible. It doesn't need to be worked on that much. They'll probably upgrade when they have to change the program. Yeah. I'm wondering if that also means that Apple is finally moving to Python 3

and making that the default for macOS, because the latest release, it still ships with 2.7.10, I think. Yeah, that would actually be really, really nice. You know, it's starting to shift in Ubuntu and just different Linux distributions, but also on Mac OS. Yeah, that would be super cool. Brian, you're talking about things that don't require security patches. I just saw a video from NASA talking about their trip to the moon and setting up a base there, which is just awesome. So they had like all this sort of quick, different scenes of people doing stuff. And one of the scenes was them working, they had Sublime open, like editing

a bunch of Python code, controlling like something about the moon mission, which was pretty cool. But I didn't catch whether it was two or three. I couldn't tell. That's pretty cool. Yeah. Someone else on Reddit did catch that the sublime was unregistered, by the way. Okay. I guess a final thought here is we've also talked about this on Python Bites is that there's the pythonclock.org, the Python 2 death clock. And it has been updated for this new end of life statement.

And it reads one year, one month, two days. So if people are out there and they got to start upgrading, like the death clock is ticking. All right. I also got item number two. And that I would categorize as the end of innocence for PyPI and PIP. So we've heard about a couple of issues. one like right before 2018, like late 2017, and then another one again this year about of either a research project a prank or some kind of like bitcoin theft so one of them installed like a clipboard monitor that would watch for bitcoin addresses and soon as it saw one it would like ship that off to like the mothership or

something to that effect but i don't know if anything super bad that's happened, but I guess until 2016, 2017, PyPI as a place was just, it's incredible to me. It's a place on the internet that was entirely unprotected and yet safe. I can imagine that. Yeah. If you had like an open FTP server and you just put it on the internet,

like all sorts of badness would happen to that thing in a day. It would be full of all sorts of things you don't want it to be full of. And here PyPI was going along. And it was just, you know, the community sort of policed itself and took care of itself. Plus, it used to be really hard to become a person that could push things to PyPI. Yeah, maybe it correlates to the shipping of PyPI.org. Yeah, there seemed to be a lot of, like, not a lot, like I'm trying to be more specific, but I think NPM, again, in the JavaScript, like Node.js world, they had a big, they actually was, it was something similar. I think it started with a typo squat except that package, as it was installing,

was looking for your, like if you were registered with the npm package repository, and it would go in and patch the packages that you owned to spread that worm further. I don't know if there was some payload, but it was really exploding quickly. And I guess the same thing could happen with PyPI. I, I don't see why not. Yeah, it definitely could. But it hasn't. And let's not give any people ideas. Sounds cute. We're going to cut that out.

Yeah, exactly. Don't say that. Security by obscurity. Come on. Exactly, right? Security by lack of creativity. Okay. So the two main problems were basically like you pointed

out dan type of squatting which is like we're going to upload a package that takes a copy of django and then slightly modifies it setup.py and publish it under django or django or whatever like how you pronounce it without the J, right? So if people pip install and they misspell, they get your bad thing, right? So that's one. And the other was to take advantage of folks who didn't understand why something wasn't available to them. So like if you tried to use regular expressions and you say re. And there's like, it says, well, re is not defined.

You're like, well, maybe i need to pip install re you know you just need to import it right it's built in so they were targeting the built-in libraries and i believe all the built-ins are not patched and brian didn't we cover something on python bytes that would help guard against typosquatting it would like try to grab packages that also had misspellings around your package name i think there was something like that yeah i think so i don't remember what it was called, but I think that exists and people could find it. I remember listening to that episode

and you guys were talking, I forgot what it's called too, but it generates different variations of your own package names. You can try and register all of those. Yeah, exactly. It's very work intensive. Yeah, so it's interesting on what is the fix.

And I don't know what the fix is. The PSF is trying to get more resources to support pypi and the python packaging infrastructure in general but like largely that's been improve it do things like you talked about previously dan on just shipping pypi.org right but this validation if you have so many releases of so many packages how do you even verify it i don't know it's tricky it's not necessarily something that could be that can be solved with a straightforward technology fix right like essentially you're it's a problem that all of these um let's call them online marketplaces for code repositories or whatever you want to call

them you know if you look at the android app store apple app store i mean the apple app store they have these these like crazy intensive review processes you know considering that there are millions of apps in this app store and they're reviewing every single one. And I think there's even some manual steps involved there. And even there, we've had malicious apps that were trying to steal people's private information. And it's a really locked down platform.

So I think it's really, it's a challenge that we're facing now because we're all connected over the internet and, you know, really relying on these on these package platforms so um it seems like it's something we'll have to try out a bunch of solutions and see what will work there i think we should get some of the all these new machine learning python people to get involved in and take a look at it because conceptually the normal updates for a package are little tweaks on what it's doing. And if suddenly it starts connecting to different servers all over the world, or even just one where it used to not connect to anything, that would be a change. It would be a different behavior than it was ever doing before. That should be something that a machine could catch. Yeah, actually, I think you're right.

I mean, there's almost no scenario where pip install a thing should create outbound connections besides to download the package. Or, you know, even a package that doesn't normally use any connections or use requests suddenly adds, you know, socket connecting or something. That's something to take, you know, at least we can filter it down to things people could take a look at and say, is something weird going on here? Well, it would actually be sweet to have like a CPython interpreter that is fully sandboxed that, you know, can't like, you could just run code that people send you and you can run it safely and it wouldn't be able to write to any folder on your disk or open sockets and whatnot. That would actually be kind of sweet. I mean, it's all

interpreted. So it's seems like that should be possible. I mean, it's going to take a lot of work, I guess, but I think it's possible as well. You know, Dan, you mentioned the online app store as maybe something, I wonder if there's something like that in terms of packaging that we could add. And what I'm thinking is like, if I go build an iOS app, I have to declare the things it's going to ask for. Like it has to ask for access to contacts. It has to ask for access to GPS. So, you know, you declare it needs the GPS feature. It needs these other features, right? So what about a package that declares, like I don't declare any network capabilities, so I don't talk on the network and things like, you know, I can't depend upon packages that do, for example.

On the flip side, none of these have caused a huge amount of damage. And I don't want to go to the point of overreacting so that everybody has to take off their shoes to travel. Yes, I know. Things like that. I know. We don't want to feel good security. That doesn't actually make any difference, but makes life hard for everybody.

Yeah. Yeah, I agree. I don't know what the answer is. I don't know what the fix is or if there is a fix, but I would definitely declare the end of innocence for PyPI. That's my declaration as item two for 2018. Dan, you have the honor of kicking off what is the number one story in the Python space for 2018. What happened? Oh, man.

What a sad honor, though. Well, yeah, I know. It's not about the messenger right well yeah i mean guido van rossum stepped down as the uh bdfl which is both sad and ironic because i guess bdfl stands for a benevolent dictator for life which i think he he still is now technically but he's just going to be less involved with the uh you know the stewardship and leadership of cython. Yeah, he's a little more like the queen of England now, in a sense, right? Like he's still representing Python.

He's still very active in the community, but he just said, I'm going to let other people deal with the decisions around it. And basically went so far as saying like, I'm not going to decide for you how you decide. Like it was even a meta non-decision. So he said, all right, you all have to figure out how to govern yourselves. And it looks like he's coming back in a little bit more in terms of participating. Well, right. I mean, I think that's one of the, one of the reasons is because he, he was the, his involvement with Python was just the stuff

that wasn't fun. And now he gets to be involved with some of the things that are fun, like speaking and mentoring and things like that. You know, I think that's a really interesting point. I'm not sure I would want his job the way it kind of looked a year ago, right? Just, you know, arguing over peps and stuff like that. And you know that if you do stuff on the internet, that it's, there's definitely a very small minority of people that

are, you know, abrasive, unfriendly. It just sucks. But even just a few people like that will suck your energy for sure. Yeah. And we still don't know what's going to happen. So we'll, maybe that'll be the story for 2019 is what happens now. I think you're right. I think 2018 is the year Keto stepped back. 2019 will be the year that whatever, whatever takes heart because it's going to have such a big impact on the future of Python and the CPython project. And so the last update that I saw, basically we have PEP 8000 now, which is a PEP that discusses the various new proposals that have been put forward regarding the Python language governance. And I just recently saw, it was like two or three days ago,

Victor Stinner's post on discuss.python.org where he's comparing the seven Python governance PEPs. So I guess like actually like the different proposals, they're separate PEPs, but they're also listed under PEP 8000. And so there's different models that have been put forward, like, for example, leading the project with a trio

of three co-leaders. There could also be a community governance model where there's no central authority, sort of like a total 1080 from a dictatorship to no central authority. I don't know how that would look like

in concrete terms, but there's different models that are being discussed right now. And I think it's going to be very interesting to see what the CPython team ends up with there. I agree. And it's pretty important, like even simple stuff, like LucasLenga, also Creator Black, what we spoke about earlier, he's doing the release management for CPython for the next couple of releases. And he was thinking like, Hey, it would be great to release this yearly, but we can't even discuss having it released yearly instead of every 18 months because we don't even know how to decide whether we, how would we, who decides, how do we vote? We don't know. So, so many things like that are just like on hold until this gets

figured out. So there's a lot of pressure to make it happen. Definitely. Yeah. It's like, it's a total bottleneck, I guess, for, yeah, like you were saying, you know, just to put out a new release, you would have to, they would have to figure that out first. And so I'm pretty sure that also means it's going to be a decision that's going to be made relatively quickly there. So we can all move on. Yeah, absolutely. One thing you did note, though, is that Gita is starting to come back and be more active in the community again, just not in this BDFL decision maker way, right? Yeah, I thought that was kind of cool.

You know, I don't know. It might be coincidence, but I don't think it was. So as I was putting together, you know, the recent PyCoders Weekly issue, I saw that there were a couple of interviews with Guido out there or articles that you know had like little sound bites from him so he did a really long like video-based interview on the MIT AI podcast so it's actually on YouTube it's more like a video-based show but I think it's also podcast that I thought was really interesting and he also started blogging again so he has this personal blog at I think it's neopythonic.blogspot.com or something like that and he put out a new post there and it's I think it's neopythonic.blogspot.com or something like that. And he put out a new post there. And I think it was kind of nice to see some activity there, again, because

I'm sure it's also a sign that, you know, now he maybe feels like he has more time to do that or feels inspired to do things like that again. And I think that's a really, really positive sign, just for his personal well-being, I think. I think it's a positive sign as well. And just like Brian and I were talking about earlier, a lot of his energy had previously gone into like debating PEP 572 and just dealing with all the decision-making. And now he can focus back on, you know,

what most people would want to focus on, coding, software, cool stuff like that. And it sounds like he's getting back into it. So that's awesome yeah yeah and i mean you know what a what a crazy responsibility to be at the center like at the focus of it all you know and to be to be the one person that needs to make all of these decisions and sure you know like he was delegating a lot of these decisions and everything but the way the c python governance was set up before it was like all focused on

one person and um yeah i mean it's tough to do that for several decades right yeah i honestly i was surprised that he had kept going in that role for that long i mean it's sad to see him step down but at the same time that was 25 years like how many other people have a job that goes 25 years that's high pressure right yeah it's I mean, thank you so much Guido. Yeah, absolutely. Python's a special place and it has a lot to do with him. All right, guys, that's it. Those are our top 10 for 2018. It's been a fun year, hasn't it? Absolutely. Yeah. I think 2018 just has more good stuff to come. So it doesn't seem like anything is slowing down,

and that's pretty awesome. Now, before we round out the show, you have to answer the two questions, both of you. So, Brian, let's start with you. If you're going to write some Python code, maybe test a little something or other, what editor would you use? PyCharm.

Right on. And Stan. Yeah, good one. Dan? I'm still a Sublime guy, although I have been experimenting with Visual Studio Code, and it's pretty sweet, actually. Yeah, I would say the Sublime Visual Studio Code, there's probably a lot of bleed over between those. And with all the activity around Visual Studio Code, it's got to be eating into the Sublime world.

Yeah, definitely, probably. I like what VS Code is. I like watching what VS Code is doing. They're just not quite there with the testing yet. I'll take a look again when they get that up to speed. Yeah, I agree. I think they're doing really sweet stuff. Okay. Brian, notable PyPI package. One preferably that didn't have any badness in it. PyTest. PyTest, right on. Dan? Sweet. So I was going to say black again, but we talked about this at length. So I'm going to say PyTest-ICDiff. So it's a PyTest plugin that gives you better error messages

for your PyTest assertions in some cases. So it'll basically do like a more extensive like diff on, let's say when it's comparing two dictionaries and it'll tell, you know, those keys were added, those keys were removed and it has like color highlighting. And I just found that this is like a really, really sweet little tool, and I started installing it in all of my projects that have tests, and I highly recommend it.

That sounds great. I may have to go check this out. Cool. All right, so a straight sweep for PyTest there. That's great. All right, guys, thanks so much for being on the show and being part of this. It's so many cool things to talk about, and I really appreciate it.

Yeah, thanks for having us. Thanks. Yep. Bye. Bye-bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S.

And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.