Python Bytes - #130 Python.exe now shipping with Windows 10

Episode Date: May 14, 2019

Topics covered in this episode: - pgcli - Papermill - Python Language Summit - Python in Windows 10 Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/130

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 130, recorded on, what is it, May 4th, 2019, live from PyCon. I'm Michael Kennedy. And I'm Brian Ocken. I'm Emily Morehouse. And I'm Steve Della. And it's great to have you all here. Thank you, both of you, for joining us. It's going to be a lot of fun.

I know you both have some pretty cool news to share. So I'm looking forward to hearing about that. And I think, you know, we'll go and get started with Brian's item in just a second. But I also want to say thank you to Datadog. They're around here. They may even hear that. But they're sponsoring this episode, though. So thanks, Datadog.

Check them out at pythonbytes.fm slash Datadog. And they track all your requests across your servers, microservices, and put it all together in one holistic view of your performance and errors and all sorts of stuff like that. So that's super cool. Brian, let's start with your item. Well, here at PyCon, one of the booths here that I kind of had fun with is talking with the people at PGCLI. Hold that thought for a second.

Brian just locked his laptop with his notes on it. That's staying in the show. Okay, so my team actually, we've got a database of test data, error data that we're collecting. We're moving away from just using Jenkins, but we're pushing that all into a Postgres database. And no matter how many times somebody tells me how to interact with the database through the GUI tools, kind of a command line person. So I was happy to talk with one of the people at the booth about PGCLI, it's a command line interface to Postgres. They also have a similar one for MySQL,

but we're using Postgres, so I like PGCLI. And I can't wait to get started using it. It's really fun. It's got some syntax highlighting and like, what do you call it? IntelliSense sort of thing. Autocomplete. Yeah.

And then one of the things, it looks like a Curses interface. And I'm like, well, can I run that on Windows? Because most Curses stuff I can't run on Windows. And they're using the prompt toolkit, the Python prompt toolkit. And that does run on Windows. So cool. We'll be able to run it there. That's my item.

Pretty awesome. I don't do a whole lot of CLI stuff to the database, but I know a lot of people do. That's pretty cool. How about either of you? What's your style? GUI or CLI? I switch between the two for sure. I'm a huge fan of DataGrip, but anytime I'm just doing quick creates and deletes and drops and all that stuff, I usually go CLI. My data work tends to start from CSV and end in Excel and occasionally via Jupyter. So use the most popular database deployed on the earth. Excel. Yeah. Yeah. Excel is my database. Yeah. That one tends towards the gooey side, but that's pretty awesome. All right.

Let's see. I guess I'll go with my next. The one that I thought was cool was a recommendation from one of the folks here at the conference. It's called Paper Mill. And I know, Steve, you've heard of Paper Mill, right? Yeah, I've had a little play with it, but I think you've probably done more justice talking about it than I have. Yeah, I definitely, I've learned a little bit thanks to John Lamb and some other folks. So Paper Mill is a way to take Jupyter notebooks.

And I'm sure you've all are familiar with Jupyter notebooks. And it's just such a different way of programming, right? You go in there, you load up your data, you kind of explore it, you're iterated on it. But what it typically doesn't do is it doesn't typically get called and then return a value, right? It's like this thing you load up, you look at you, okay, well, that's cool. That must be the answer or that's the picture it makes or whatever. Well, with Paper Mill, you can turn Jupyter notebooks into basically functions and command

line arguments that you can run. And you can actually chain them together. Like this notebook then calls on that notebook and then saves it. And it does really cool stuff. So apparently folks at Netflix are doing really interesting things to run a bunch of ETL, egress transform load, that type of stuff. They run 18,000 of those a day through Paper Mill and other stuff. And apparently what's really cool about Paper Mill is if it crashes or doesn't work, you get like a whole graph of Jupyter notebooks that you can go open up and like, instead of just having a log entry or some kind of error in like

an exception trace or something, what you get is actually the notebook with all the stuff that notebooks have explaining what it was doing before it died. If you do anything with notebooks, that sounds like super cool. People should check that out for sure. The workflow is actually a really useful thing. I'm aware of a couple of teams within Microsoft where I work that have, I don't think they're all using Paper Mill. Some of them maybe we don't actually have that much. We don't tightly control everyone's workflow, so they get to use what works for them. But there's certainly some teams that have services set up where people using them are going to send in a query or they're going to do something. And occasionally there's cases that are

not really failures, but they seem weird. It's kind of like if you go to a search engine and you type in something that gets no results, what do you do then? You haven't necessarily failed, but you probably want someone to look into that and figure out what it does. And so I know that we have some teams with a similar kind of thing to what PaperMill can do. It'll generate a notebook to do the start of that analysis. Or there's some cases, some of the security teams that we have, the security research teams, have templates for analyzing malware that's come in. And that's the kind of thing that you can use PaperMill to auto-generate maybe a half-complete notebook with the inputs that you already know and then pass it off to an analyst to work through and put them in a comfortable environment

that they want to work in, which is Jupyter, and give them the tools but have all of the information already there so they're not spending a lot of time loading it up and collecting URLs and copy-pasting. Yeah, parsing log files and all that sort of stuff. Yeah, that sounds awesome. The way it works is basically you define a cell with default input arguments,

and then there's a cell below it that are all the parameters passed. Emily, do you guys do anything with notebooks? Not a whole lot, really. I have a couple of internal tools. We operate at a very different scale, so we are a company of six people right now. But I use notebooks for a lot of internal tools and budgeting and forecasting and whatnot.

All right, awesome. Yeah, well, Paper Mill, check it out if that resonates with you guys. All right. Which one do you want to do your item next? All right. I'll go next. So I wanted to talk a little bit about one of my favorite events that I get to participate in at PyCon. And that is the Python Language Summit. This is my third year attending. And it's really, really interesting to get to see kind of a lot of the inner workings and discussions that go on. So one of my favorite things this year, the Language Summit was run by a few new people to organize it. And Marietta and Lukasz Lange took a little bit of a different direction on it. And we got a lot more community involvement. So we got to hear from projects like Circuit

Python and Beware, and really just got a lot of new voices into the core development community. And I think that that is something that is really going to help us moving forward, finding out not only what we need internally to continue the development of CPython, but also what needs to happen to do things like supporting Beware, which hopefully if you attended his keynote or you can go back and listen to the video, Russell K. McGee's keynote talks a bit about this. And so really just this true cohesion of different groups coming together. And I'm very excited about that. Yeah. And this is the first year that you've had the Language Summit

after Guido stepped down, right? So this is the first sort of independent one. Yeah. I know that he was there, but it probably was a little different. I don't know. I didn't feel any differently. Did you? No, I feel like it was definitely having more community involvement means we were getting a lot more opinions

from outside the core development team. And I think that's going to be really beneficial as we look towards Python 3.8 is approaching its feature freeze, but 3.9 planning is already starting. We've got a few peps that people have been working on towards that. But having had those extra voices come in and call out some of the pain point and putting a face to it as well is such a big deal. So much of the feedback that we get to hear that you always get to hear about software is a GitHub issue and it's just text and it's a made up name and an image

that's generated from an IP address and there's no person there. And how much pain is this? We don't know how much pain this is. And when someone takes the trouble to get to a conference two days before it starts and prepare a talk and come and spend half an hour talking face to face with people, it has a very different impact and you get a lot more information and you get the follow-up. And so there were a lot of chats after some of the talks where some of us heard the things and were like, I thought that was a problem. And now you're telling me it is, let's dig into that. Like, tell me more about what we can do to help this. So I think for 3.9 planning, we're going to get a lot of benefit out of the way this one was run.

Yeah, that's super cool. And of course, the high bandwidth in-person conversations has got to be better than, oh, I put up a GitHub issue or I put up a pep and go read it, whatever, right? Like being there in person has got to help. Yeah, absolutely. And that's one of the best things about PyCon. Those conversations haven't stopped. I walked away from some of those conversations to come to the podcast right now. And probably looking at the people around here, I might make it three or four meters before we start another one. That's right. That's right.

Speaking of three or four meters, I just want to say thank you to Logan for the paper mill recommendation. He was the one who got that one on the show. Thank you. All right, Steve. Sometimes you say you save the best for last. I know that a lot of people are going to be excited about what you're offering here. And you sort of wear two hats in the python world right like you

wear the core dev hat and you wear the microsoft hat and uh you kind of get to talk about them both right now right yeah um and chatting with a few other people we need to get physical hats for some of this because sometimes it's a little unclear which hat i'm wearing in any particular yeah the sherlock hat with python on the front microsoft on the back. Could be fun. Those who saw my brief little chat this morning from the stage will know that was very much the Microsoft hat. But one of the projects I've been doing for a while for CPython itself has kind of been both hats at the same time. So when Windows 8 came out, we added an app store, or Microsoft added an app store as part of Windows, and that's continued on. So there's an app store for getting apps for Windows 10. And that's always kind of been slightly different from downloading

an installer and double clicking it next, next, next, next, next, agree, next. It's been meant to be a lot faster, a lot simpler to get installations, a lot more reliable, a lot more isolated. And generally, you start doing that by making it highly restricted. And so when that first came out, one of my first projects at Microsoft was how do we get Python into this thing? And it turns out the answer was not can't do it impossible. Fast forward a few years, and those things kind of get relaxed over time. And we've reached the point recently, where it totally became feasible to put Python in the Windows App Store. And so we did. And so as of Python 372, that was kind of a preview 373 is considerably more reliable, and Python 3.8 is2, that was kind of a preview. 3.7.3 is considerably more reliable and Python 3.8 is

going to be better again when that comes out. You've been able to go to a Windows 10 machine, open up the store, type in Python and install Python that way. And those apps auto-update? And it auto-updates, yeah. So it'll stick within the major version. So if you get the 3.7 app, sometime when we push out the next version of it, when you're not using it, it's just going to quietly update in the background. And so next time you go back to it, you'll automatically have 3.7 app. Sometime when we push out the next version of it, when you're not using it, it's just going to quietly update in the background. And so next time you go back to it, you'll automatically have 3.7.3 or 3.7.4. I actually get some really nice controls. I can do a gradual rollout on that. So I can set the slider and say, automatically update this many percentage of people, which is fun, but it's not going to take you straight to 3.8 without warning.

That's going to cause you to take another step, go back through the store, pick the new one. But the other half of that project while I was working on that was kind of still like secret within Microsoft. And we knew that we were doing it. We knew that it was going to happen, but until it was actually out there, we haven't been able to talk about it.

Now at this point, it's out there. So some people are already saying that if they're running Windows Insider builds, because they've got the latest update already, that update's coming out in the next month or two. When that hits, every single Windows machine is going to have Python on it. So if you go to a clean... Which means if you go to a clean Windows machine and type Python, previously you get a big fat error message. What you're going to get now is a pop-up straight to the page in the Windows store

with the latest released version of Python, which means you're now one click away from installing it. I actually demoed this at the Language Summit, so Emily can confirm that within like a three-minute lightning talk, I went and installed Python and had it running. The feedback I've been hearing so far is people forget they even clicked install, and they're like, I typed Python and it was just there by magic. And I'm really excited to just take out so many of those steps on where do I go to get Python? How do I get started? I've talked with some of the teachers and the trainers who instruct people on Python. They're really excited because now you can go to any machine that people bring in and type Python and it's either going to work or it's going to put them one click away from it working.

I really love this. I actually, well, part of my job is to monitoring the test environment within our company and our area, but trying to get people to just install Python for even that, for some reason, some people get it messed up. Somehow they install it in the wrong place and it's just a mess. So I'm really happy about this.

This is great. I know Windows has a lot of like enterprise controls and group policies and stuff that i haven't thought about for a real long time but it seems like they could be used for good like can windows stores apps be like set up company-wide so could this just be like automatic on the machines that you know large? I am not personally familiar. I probably need to get familiar because this question is going to come up again and again. Now that we just published it to the world, yeah.

Yeah. With how to roll out this app to a whole lot of people, one thing I can say, you don't have to be an administrator to install it. But when you do, it still installs it as if you're an administrator. So it's going to have a lockdown secured install. No one's going to modify it on you.

So you have like a secured Python install like that, even though you weren't an administrator to do it, which is a really neat thing about the store apps. I almost certainly you can roll it out. But again, it's very few places that I'm aware of are restricting apps from the store because those have gone through a Microsoft review process. We've checked them out and they're basically safe.

Right. They're super isolated. So that helps a lot. There's two ones. They're super isolated. So that helps a lot. That's true. They're super isolated. Super isolated. Yeah. Which does lead to some bugs. And some people have noticed this.

It's like if you write to the temp directory from this, it actually goes to a private one. On the bright side, if you uninstall it, it gets rid of everything that it did. That's cool. And somebody, as somebody who writes courses and, you know, if you write a book, it's always, here's what you type to activate a virtual environment or run the script on Mac OS and Linux. But you don't do that in Windows. You say Python without the three, and then you got to make sure the path is right and you check it. So that's the other good thing, right? Is I can now do Python 3 on Windows. Yeah. So this is the first time

we've actually done this in Python on Windows, but when it comes, it'll automatically go onto path by default. And it comes with a Python shortcut, a Python 3 shortcut, and a Python 3.7 shortcut. It also includes the same for pip and idle. So when you install this, and in fact, right now, until we fix it in probably in 3.8, you're better to just type pip install after using the Windows store package than you are doing the Python-m pip install.

That's awesome. Such good news. I'm really glad. I've wanted this for a long time. I remember on UserVoice, I had posted something like that, requesting Python 3 in Windows, and it got like thousands of upvotes and comments,

and it was just like kind of crickets. Yeah. You know? It takes some time, but we do read those UserVoice. That's awesome. And I'm really happy that you got this. I mean, the impact this is going to have

is going to be huge just because the number of Python developers on Windows outnumbers any other platform, right? Yeah, and it's going to be too late for the people that are listening to this on the recording. But for everyone here live, I'm actually, I've got my talk tomorrow afternoon, which is on this topic.

Python on Windows is okay, actually. For those who are listening, you'll be able to find this online. That's one of my first points is everyone's always surprised by just how much Python is being used on Windows. Because particularly for those of us at PyCon, we look around and we get a very different idea of where people are using Python. Because virtually everyone has a laptop with, you know, some shiny fruit on the back. And that's just not actually the case for everyone who's using Python out there.

So that's conference reality, not reality. That's absolutely conference reality. I joke that I'm the only person in the room with a Windows PC, and far too often, it's not enough of a joke. Yeah, that's for sure. All right. Well, thank you for sharing that and announcing that to everyone here. That's super cool. So I guess let's close out the show with just maybe your main takeaway from the conference. I want to start with you, Emily. Oh, gosh. Your conference experience this year compared to last, something like that. Oh, yeah.

So I think that this is a very unfair question to ask of me because my conference experience this year is very different because I'm co-chairing this year. Congratulations. That is so awesome. This year is very much like running around like checking with my head cut off

it feels like work I'm surprised that I'm sitting for this long but it's really really incredible I'm very very grateful to be able to give back to the community

in a different way and continue to be the voice of the community as PyCon grows throughout the years awesome well congratulations and I'm looking forward

to next year when you're even more involved in the conference. Yeah, Pittsburgh 2020. Right on. And congratulations having your business on Python doing so well as well. It's great to see everything going so great.

You bet, Steve. Yeah. So, well, and again, this is very much a running around like crazy conference. Microsoft is Keystone's sponsor. We have a big booth. We have a lot going on and I've been helping run some of that.

So my experience has been a lot of people actually coming up saying, oh, I've heard about some of your stuff. I've been trying to use it and I've been struggling. Like I've been hitting problems. I'm getting stuck in some places. Are you able to help? And one of the things we came prepared to do, which I was keen to do and help drive through a lot of the planning was actually bringing some of our engineers solely for the purpose of helping people get their stuff working. And so we've had a table set up. We've had people coming up with their own laptops and helping them get whatever it is they're trying to do working. We've had migrating people to Azure pipelines, getting people to deploy stuff on Azure, setting up VS code, a whole lot of

things. And so it's been a lot of experiences of people being frustrated at the start and walking away very satisfied and very happy to have come and actually met a human from Microsoft who can help them with their computer. Well, so a couple of years ago was the first time, the second Portland one. I couldn't get tickets to the first Portland one, even though I lived there. But that was my first PyCon. And since then, the last two years, Michael and I have been spending a lot of our time in a community contributors booth. And this year, I'm very grateful for PyCharm for including us in their larger booth. And then also, we've split up the time. We have more community contributors.

So we're not standing in the booth the entire time. We've really got to enjoy the conference. And I'm really grateful for that. And it's been an incredible experience. So thanks. Yeah, thanks for this theater and this opportunity again. That was awesome. And my takeaway is really, you know, kind of like before I said, just the people, right? Like I get to meet so many people. And you hear these stories, like you changed my life in this way. And like, what I was doing didn't feel like anything like that. I was just kind of like chatting, you know, playing around, but actually the community is great and we do a lot of things for each other and it's just wonderful. So it's great to see you all and, you know, be part of it

and share the stage with you guys. So thanks all for coming in. Are we going to take questions? Let's go ahead and take questions. We'll take two questions. If you guys have any questions. Yeah, Paul's got one. Of course, Paul has one. A question and a half for Emily. Half question. I heard there's some event in Colorado you might want to talk about. Sure. Yes. Pi Colorado 2019. It's going to be the weekend of September 7th and 8th, I believe.

It's that Saturday and Sunday. Inaugural Python conference in Colorado. Come for the Python. Stay for the mountains. Yes, mountains, please. Okay. for the python, stay for the mountains? Yes, mountains, please. Okay, so the actual question. Your reward for becoming a contributor was to take on the most controversial task

in the last 14 decades of Python. Yes. Now that it's merged, looking back on it, was the process easier or harder than expected? To actually implement it? So I don't think that this is a very fair question because I had spent approximately a year and a half plus really just diving into the code that I was going to need to work on.

So I had been figuring out how the parser worked, how the grammar was defined, how all of the symbol table lookups and everything were working. So I think that it was a lot easier to implement than I expected only because I had all of this knowledge. And so it was very rewarding for me to be able to take that year and a half plus of time that I had spent learning and just be able to apply it to something real. And then there are definitely a couple of like tricky bits about how scoping works with assignment expressions. So that part was really fun to figure out and was more challenging, but definitely a

good balance of getting it 90% of the way there and then really having to figure out the tricky bits. What are you planning to work on at the sprints, Emily? Well, I'm actually going to write a bunch of documentation, which is everyone's favorite part. I want to highlight some of the tricky bits about scoping that I think might trip some people up. But then I'm also really interested in kind of diving into some of the discussions around improving Python's parser and swapping that out for something a bit more modern. Follow-up, is the new feature tested? Tested? Yes. I actually use like legitimate test-driven development

for the Waller top writer. Yeah. Yeah, yeah. We've got one more question out there someone wants to ask. If not, you're going to make me tell a Python joke or a programming joke. Now, normally... Oh, yeah. Go ahead, Ricky.

Right. So the question... And I'm going to skip some of the context, unfortunately. But with Python coming in through the Windows Store and auto-updating from version to version, what about all the people that need to run multiple versions? And there's a lot of really good reasons to have multiple versions of Python around. It's actually a scenario that works really well on Windows. And that's something that I'm really

careful to not break, or at least I'm trying to be really careful to not break it. Obviously, one of the main scenarios is testing. If you're going to write library code to share with people then you need to make sure it works on at least the two latest versions probably three and also 2.7 depending on who your users are and so that's something that is my hope is to not let that go away this is still kind of experimental phase so i'm not going to definitively say that nothing will break completely but one thing i will say is that the original installers are not going away. And so just because you can get Python through the store, there's going to be a number of situations where it may make more sense to get it through the regular installer.

And one of those is I don't want it to ever auto-update or I want to know that this is exactly 3.7.1 for some reason, even though that's a few versions behind. Occasionally, there's a legitimate need for that. The other thing is the store version is actually only 64-bit. And so there's some cases where you'll want a 32-bit version still. I've actually had people coming up to me saying, I'm still running 32-bit Windows. And I'm like, I didn't realize you could still get that, but apparently you can. You won't be able to get it from the store, unfortunately.

You'll have to get the regular installers still. Yeah, that's pretty cool. All right. Well, I guess I'll try to remember the joke that I heard earlier. Since I can't type normally what we do, it's pretty amazing. Brian and I, we type pie joke in the terminal, and then we take our favorite one out of there. But it's back there.

So if a functional programmer were to get a pet or a lambda. A pet or a... So if a functional programmer were to get an animal, a pet, what would they get? They'd get a lamb. Duh. Sorry. See? I need this

terminal. Brian, save me, man. Type it. Pyjoke, baby. Friends don't let friends use Python 2.7. Yeah. Alright, well, thank you all for coming out. It's really fun and thanks for listening and thank you guys for being here. It's been great. You're welcome.

Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy.

Thank you for listening and sharing this podcast with your friends and colleagues.