Python Bytes - #93 Looking like there will be a PyBlazor!

Episode Date: August 31, 2018

Topics covered in this episode: Replacing Bash Scripting with Python. pyodide The subset of reStructuredText worth committing to memory bandit Learn Python 3 within Jupyter Notebooks detect-secrets Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/93

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 93, recorded August 30th, 2018. I'm Michael Kennedy. And I'm Brian Ocken. And Brian, you know, last time we had some awesome drop the mic stuff. I think we're going to do it again, aren't we? Yeah, I think it's some really interesting things today. Absolutely.

So before we get on to them, though, I just want to say thanks to Datadog. Datadog sponsoring the showog is sponsoring this show. Check them out at pythonbytes.fm slash datadog. I'll tell you more about that later. Brian, a lot of people get into programming, kind of in air quotes there, by doing scripting around the shell and stuff, right? I think so. I think that's the way some people in the IT world are. Like Bash or even Batfiles.

Or lots of DevOps roles. Yeah, exactly. At the point where I learned Python, I was doing a lot of C++ development and writing little utility scripts in Bash and Perl and stuff like that. And so one of the things that I did, I want to try to learn Python by writing as much Python as I can. And so I started doing more and more of my utility scripts in Python. And so I think that was a good way to just try to jump in and solve little problems.

I could have written it faster in Bash because I knew that or in pearl but uh learning how to do things in python is good so there's a it's a github repo that's called uh replacing bash scripting with python and i think it's a it goes through a lot of stuff that a lot of the issues that that you'd need to do that you normally write in the little utilities and how to do that in python so this is this is fun talks about like dealing with, reading and writing files, dealing with standard in and standard out, standard error, things like that, and writing the command line utilities. Yeah, things like dealing with exit codes.

So if you shell out to some other process, you run a command, knowing whether it failed or it worked, right? Things like that? Yeah, and how to pipe between commands if you're going to pull things together. And then it doesn't jump into a lot of the really cool stuff with Python, but it's a good start. But it does introduce people to the path library and shutil. And those are things. And then a lot of times you'll maybe be using sed and awk and grep and things like that.

And how to go about replacing that sort of stuff with regular expressions with regex. And so this is just fun. I like it to be able to, if you're writing a little utility and it's up to you as to what language you write it in, this might help you write it in Python. It's cool. And I think one of the real key things to actually making programming stick when you're trying to learn it or even learning a new programming language, you know, some other kind of programming,

is to have what you're doing be relevant to you, right? And so if you work in a DevOps world and you're like, well, I'm going to go learn Python. So I'm going to do this data science project. Like that may be totally irrelevant to you. So having this as a resource for people who generally do scripting and automation type stuff, this is really cool. Yeah, and then also when you're learning, the stuff you start out writing, just to be frank, and it's the same with everybody, it's not that great to start with. So starting small is good, so you don't have something huge to rewrite later.

True. I guess a lot of people, I know I did one in the earlier days, went through the experience of going back and reading code I wrote six months ago and going, wow, that's bad. But then that same person who looked back badly on the the old code six months farther down the line you look back and yep that's that's bad too just in a different way right but eventually eventually you get there but yeah having the small stuff definitely helps and i've heard a lot of people that like in a devops world where they just you know you start writing like one or two of your scripts in in python and then you can just you add it. As you learn more stuff, you can do more powerful things. And like you said, having it be relevant to your job is important for what you're doing

for learning. It's a really cool find. I'm glad you brought it up and I'm sure it'll help a ton of people. Now, this next topic, I'm pretty excited about. I'm honestly kind of beside myself about this one. This literally could be a game changer. And so this is called Pi Iodide. And I don't know if you've heard of it before. I had not heard of it until just beginning of this week, I think, actually. So Iodide is a project by Mozilla.

And think of Iodide as like a competitor to Jupyter Notebooks, where the concept of Jupyter Notebooks is you have this web page, and it's like a view into a Jupyter server. And then when you execute code, that code executes on the server, which is a little sketchy because if you're letting people run arbitrary Python code on your server, that's not amazing. So maybe you're using some sort of Docker thing to spin them up and shut them down. It gets complicated, right? Yeah. Well, Mozilla wants to make this environment available to everybody, but in a distributed fashion. And so the goal is to bring this environment to the browser. And when you execute code, it executes on your browser

client side. So you can do whatever you want. Well, that's cool, but the problem is a lot of people want to do data science in Python. And Python, while it's had some JavaScript options for running code in the browser, they've always been like, well, you can kind of do this, and it's mostly like Python and so on. A while ago, as in two weeks ago, we asked the question of, will there be a PyBlazer? Remember, Blazer is the.NET community's attempt to bring the.NET CLR, their equivalent of CPython, into the browser with WebAssembly and make it run there. And they're making that actually work, which was kind of blowing my mind and giving me a lot of hope that maybe somebody could do it for CPython, right? What do you think? Do you think it'll happen?

Well, it's a nice dream. I agree. It's a nice dream. But sometimes dreams become reality, right? So PyIodide is an implementation, a running version of all of CPython, an exact bit of CPython, same thing in the browser, in WebAssembly. That's so cool. Is that incredible or what? I mean, my mind is blown. So I reached out to one of the contributors, Michael D., and asked him, like, this is like

really CPython running in WebAssembly in the the browser and he said yeah yeah so michael works for mozilla this is a mozilla project and they're trying to bring all the c python at least the data science part of it into the browser now there's a few things that it won't do like you can't open a socket obviously tk enter doesn't work because what does that mean in the browser right there's So there's some parts of CPython that are not there. But if it's permitted in the browser, it's identical to CPython. And that, I think, is amazing. That is amazing. I can't wait to play with this.

I can't either. And so they also want to have the data science tooling there. So things like NumPy, Matplotlib, and so on, they have heavy dependencies upon C. So you can't just bring it over and run it in this WebAssembly version of CPython. So you have to take the C code that supports things like NumPy and Matplotlib and get them to compile to WebAssembly themselves. So they've also done that. So you have native NumPy, native Matplotlib on native CPython client side in the browser with web assembly and it's fast i'm still trying to get my head around this how could it be

i know it's i mean it's like binary stuff executing and it's also not that big right the download i i noticed using the web tools when you go and interact with this page, it downloads like CPython.wasm, right? The extension for WebAssembly. And it's 2.5 megs, right? And on my connection, that's like one and a half seconds to get that thing up and running. And then it doesn't have to parse it like it does like two megs of JavaScript. You don't make sense of that, right? Effectively compile it.

It just takes it and runs it because it's already compiled. Okay. That's okay. So that's not nothing, but that's pretty cool. It's not nothing, but it's, it's not, it's not that bad. And you know, you could cash, somebody could set up a CDN for that thing and cash it like say Mozilla. Yeah.

Like say Mozilla. So I think it's awesome. I want to, I want to just say thank you Mozilla and Michael and team for working on this because this, this is amazing. And this could really, really change. It could really expand where Python is broadly applicable. Now, I think it's worth pointing out that this generally is built with the concept of making data science work in the browser, right? Whereas this PyBlazer thing, this Blazer, my imaginary PyBlazer thing that I made up, that was around building single page apps, more of the web side of programming than the

data science programming side of programming. But this thing does have extensions to let you work with HTML and the DOM and events and stuff like that. So you could, it's not a long shot to go from this to having an interactive, rich application for a web app in Python. Yeah. So still not quite where we want, but like really cool. It's really cool.

And I mean, it's totally working for what they're trying to do, but I'm just thinking like, if I want to say replace like a React thing or an AngularJS thing with Python, there might be a few more steps to get like that angle of stuff working. But it shouldn't be that far

because we already have the CPython runtime working, which is really what you need and the integration with HTML. Yeah, so this is cool. I want a lot of people to grab this and run with it. Yeah, I think this really opens up the possibility for some amazing stuff. Once again, thank you, Mozilla. And also, I just want to reflect a little bit. I think it's really ironic or interesting coincidence. At PyCon 2018, as we mentioned when we talked about the Blazor

thing, I think it was Tim Callahan, I'm still forgetting the name, but somebody from Mozilla came and gave an impassioned presentation as the opening keynote of, please let's find a way to use a web assembly to get Python more capable in the web space and specifically in the browser and client side. And I think it's just pretty awesome that Mozilla, another team in Mozilla, actually is the one that implemented this. So that's really, it's really amazing. So these guys working on PyIodide, they may want to document their stuff, right? Yeah, you're so much better at the transition thing than I am. I'm like, oh, next topic. I personally like Markdown. I write, even when I have to use, to be honest, when I have to use Microsoft Word, I write in Markdown and then I convert it to Word. Anyway, I don't know why I was talking about Word. So restructured text is not something that comes naturally to me, but there's a lot of Python documentation that's written in restructured text.

So if you're going to contribute to an existing project, you can't say, well, I can't do any of the documentation because it's all in restructured text. Convert it to Markdown. Just use Markdown. No, we're not rewriting all of this in Markdown. Right. So whether or not you like it, restructured text is part of the Python world. So this is a wonderful article from Simon Willis. Oh, I always had his name wrong.

I always thought it was Wilson, but it's Simon Willison. A subset of restructured text worth committing to memory. And that's a noble goal, but instead of committing it to memory, I'm just going to bookmark this article. But he's got the necessary things for easy documentation. You have paragraphs, headings, bulleted lists, how to do italics and things like that. And then, of course, for documenting code, you need how to do inline code and code snippets and images and internal references and things like that. And there's a lot covered here, but one of the issues that I always had was the tutorials on restructured text.

Restructured text can do a lot, and it's kind of overwhelming, and you don't need all of that to understand enough to help document a Python project. So this is great. Yeah, it's cool. These are the five or six things you need. Headings, links, bulleted list, number, numerical list, images, things like that, right? Yeah, and then how to get your code in there. Yeah, yeah, exactly.

It's great. I'm going to make use of this one that you're sharing here because I'm working on a project where I have to write a lot of restructured text. And, yeah, I basically have doubled my knowledge of restructured text by reading that. Yeah. Because like you, I do Mark my knowledge of restructured text by reading that. Yeah. And I mean, because like you, I do markdown most of the time. So most to be honest, most of the time when I needed to do something in restructured text,

I would just find somebody else that did something like that and copy it and not know why I was doing these things. So having a little little primer and tutorial is good. Yeah, it's great. Speaking of finding things, let's talk about Datadog really quick. So if you have issues, performance problems, bugs, things like that with your application, you should check out Datadog. So they're sponsoring this episode. And it's a monitoring platform that brings together metrics, logs, request traces all in one place.

And they have this new feature called Trace Search and Analytics, and it lets you break down the usage of your Python app around performance for things like, show me all of the operations that this customer or this user did, or all the requests that go to this URL and aggregate the performance of that, which is pretty awesome. So start troubleshooting and optimizing your Python apps today. The free trial and Datadog

will send you a free cute little t-shirt. So check them out at pythonbytes.fm slash Datadog. So I have a theme for the rest of this, the rest of the items I've chosen, Brian. Okay. So have you ever heard that some software has like security problems, people can break into it?

Yeah. You know, even though Python's not compiled, Python code can have it too, right? I assume so, yeah. Well, our friend Anthony Shaw told us about this cool project, little app you can run. It's really a linter, basically. You run against your program called Bandit. Okay. So Bandit is designed to find common security issues in Python code.

So what it does is you point it at your project and it'll load up each file into an abstract syntax tree, run a bunch of analysis over that AST, and then it'll generate a report. And it'll tell you things like you're misusing the Telnet lib or you're using eval or you're doing try accept pass and eating errors opening a sub process with p open with shell equals true i actually don't know what's wrong with that but i should probably figure that out yeah but apparently it's bad shouldn't do it and that's error 602 so it'll tell you you know for example like i wouldn't know that that's necessarily bad i know i've heard it before but i don't remember why it's bad so it's pretty cool and it just has a whole bunch of errors it'll

find and you know security lapses it'll find and tell you about them so this is something that you it's like a static analysis it's not it's not running your code it's looking at no it's it's static yeah it's static analysis oh that's cool yeah isn't that nice yeah yeah it's static. Yeah, it's static analysis. Oh, that's cool. Yeah, isn't that nice? Yeah. Yeah, it's like, I see you did a try-accept and you just ate that error. And that can actually, you know, cause other types of problems. Like maybe you should have validated something, but instead you just keep going. Things like that. So this might be good even for non... I mean, it's definitely important for Python projects that have an interface to the web

or like a web application, but you could use this for a lot of stuff, not just that. Sure. Yeah. Yeah. It definitely lets you know when you're doing things wrong in like a different category, say than probably flake eight. Yeah. Neat.

Yeah. And because it's, you know, you can easily integrate this type of thing into a continuous build integration pipeline. Yeah. I'll definitely have to check this out. Cool. Yeah.

This is, it's, it even probably has a little testing angle to it somehow. Maybe. Yeah. Maybe. Yeah. I'll definitely have to check this out. Cool. Yeah. It even probably has a little testing angle to it somehow. Maybe. Yeah. Maybe. Awesome. All right. What you got next for us?

Well, last week we talked about what Netflix was up to with Jupyter Notebooks. And I mentioned that I've kind of been trying to learn more about them. So we had somebody submit another repo, it's another uh repo but it's um it's a whole bunch of notebooks so it's a what he's put together is learn python 3 i think i forgot the wrong link here learn python 3 with a jupiter notebook and so i was actually playing with this this morning and it was just a blast so we we have uh it's got beginner and intermediate python concepts and you just sort of like can walk through it and one of the things i had to learn of course is to at first is just to how to how to do this i had you clone the i

cloned the repo and just ran and then installed jupiter and then ran the, just say, what is it? You just say Jupyter Notebook or something, and it opens up a web browser. And then you can click different notebooks and run them and see what they do. And the thing that really intrigued me, the bone that he put in there for me to try to grab, is there's two topics on testing. And I've been asked this before of like, can you run PyTest within a notebook? And I didn't know if you could or not.

I heard rumors. But there's two examples in here of how to, two notebooks that have tests embedded in them. So you can run your tests within, so you can define a function or something in a different cell and then write a test for it in another cell and then run it. I think that's really cool. And I had no idea how to run PyTest either in there. And I always sort of saw that as like one of the weaknesses of notebooks.

I have lots of respect for them. But one of the things I don't really like is it's hard to do like how do i run tests against this code this code how do i like get a code coverage report of the code my jupiter notebook is doing things like that right seem much easier when you have a set of files that you can like call directly yeah and i'm still not so this is running pytest within the notebook and i'm not i think there's also a way that i haven't figured out yet to take a notebook and run, if there's tests inside of it, run tests against it. Yeah. There may be in the stuff that Netflix talked about last week, allow you to basically take a notebook and execute it as if it were a function with an

input and an output. And like that in some sense would open up a testing angle to itself, right? Definitely. And especially if you like save the output somewhere that'd be great anyway it was just really fun it was a it's fun to look around the other thing that i like in as an example is um there's the examples of how to do pip install within a notebook it has those at the top because they it doesn't assume that you've installed pytest to begin with oh yeah that's cool and it shows that and then it's using another pip installable package called iPyTest, which is how it's running PyTest within the notebook. So it's cool.

That is really cool. Nice find. All right, last one. I said I was on a bit of a theme. So we talked about Bandit. And Bandit lets you find code that is being either using a bad library or is using a bad pattern, like tracks that pass, things like that. This next one takes a different angle on the same sort of problem

around security and integrity of your application, and it's called DetectSecrets. So their sort of tagline there is an enterprise-friendly way of detecting and preventing secrets in code. And this is actually released from Yelp. Huh. Okay. So on one hand, like using code badly, that's a problem. But also put hard-coding passwords and other things like that into your code can be a problem, right?

Yeah, definitely. And so the enterprise friendly aspect, I think, comes from the fact that it says, all right, look, we assume that your code is just messed up. There's all sorts of badness in here. What do you do if you find yourself in a hole and things keep getting worse? First thing you do is stop digging, right? Before you get out of the hole, you just stop making it worse. So there's a lot of mechanisms in detect secrets to say, let's not make this worse. Let's prevent new secrets from going into the code base. Oh, nice. Now we can start to peel off, like start to like remove some of these things that are hard coded in there.

But let's go and set a baseline and go, all right, if we get any new detections of secrets being entered, that's going to fail the build. But the way it is now, we're going to accept it. And then we can make pass after pass to actually take away some of those and reset the baseline. Okay, interesting. Yeah, it's pretty interesting, right? Also, there's some things that look like they're secrets that should be detected, but you've inspected the code

and said, no, this one's fine. Yeah, exactly. And that baseline thing will let you say, I approve of these, but not anything else. Yeah, cool. And that baseline thing will let you say, I approve of these, but not anything else. Yeah. Cool. Which is pretty cool. And they also show you how to set it up as a get commit hook. So as you know, from the testing space, one of the challenges is like, you can have awesome tests, but if no one runs them or worse, one or two people on the team are very passionate about them and a whole bunch of other people hate them and refuse to run them no matter how much you talk about it the people who care about the test suffer

from the neglect that the people who explicitly ignore it right maybe that person enters some problem that'll fail a test but they don't run their tests so they don't know you get it you try to run your test and then you're hit with their problems as well, right? So this git commit hook here basically means everyone on the team is going to be basically running this as part of a check-in. The whole idea around git commit hooks is interesting. There's a lot of interesting things people are doing, like making sure that, like there's a plaque has a git commit hook as part of it or attached to it so people can make sure their code is still black before they check it in. Yeah, that's the one I was thinking of as well. And it can even automatically reformat your code to the group's format as part of the check-in, right?

I'd be curious to know if there's any other... I'd like to have people let us know if there's other git commit hooks that we haven't talked about that are helping their teams out. Maybe they'd help other people out too. Yeah, it'd be fun to collect a list of use cases of what people are doing.

Yeah, if you're doing something cool with the Git commit hook, check out pythonbytes.fm slash 93 and put a comment at the bottom. That'll be in the show notes, sort of below the show notes for everyone to come see. Alright, well, that's all of our items for this week. Brian, you got anything personal you want to share?

Before we're out of here? No, I guess PyTest came out with a new version the other day, which kind of blew me away because I had submitted a very minor bug, but I had submitted a bug to the project. And I only submitted it like five days ago, and it's already now in a fixed and in a release. So that's amazing. That's a really good turnaround time.

That's awesome. Yeah, that's fun. Anything with you? Got any news? Nothing I'm ready to talk about. I'm working on a couple of new projects. One that I'm doing restructured text for. A couple of new courses are coming down the pipe.

Things like that, but nothing ready to talk about just yet. All right. So how's that for a long-winded no? Well, it was fun talking to you. Yeah, you as well. Thanks for everything and see you later. All right. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian

Aukin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.