Python Bytes - #115 Dataclass CSV reader and Nina drops by

Episode Date: February 2, 2019

Topics covered in this episode: [play:1:03] Great Expectations [play:5:01] Using CircuitPython and MicroPython to write Python for wearable electronics and embedded platforms [play:11:58] How to Rock Python Packaging with Poetry and Briefcase [play:14:59] awesome-python-security 🕶🐍🔐, a collection of tools, techniques, and resources to make your Python more secure [play:18:20]* pydbg* Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/115

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 115, recorded January 29th, 2019. I'm Michael Kennedy. And I'm Brian Atkin. And Brian, we got a special guest, don't we? Yeah. Nina Zankarenko. Say hello, Nina. How you doing? Hey, everyone. I'm very proud of you for pronouncing my last name correctly.

Thank you. Brian and I, we specialize in mangling people's last names on this show, but we try. This time you nailed it. Thank you. For those who don't know me, I'm a senior cloud developer advocate at Microsoft, and my focus is on Python. You can find me on Twitter at NNJA. That's like ninja, but without the I. Yeah, that's a pretty cool Twitter handle. Awesome. And also, this episode is brought to you by Datadog. Check them out at pythonbytes.fm slash Datadog. Tell you more about that. You know, Brian, I expect quite a bit of stuff out of Datadog. I would say I have great expectations when I go use their stuff. What do you think? How about you? Yeah, yeah. I do too. Yes. For a first item, we have Great Expectations.

And it is a package that was shared by a listener. And I don't know the listener because they shared it with you, Michael, and you didn't tell me who it was. I just sent you over the link and I forgot. But thank you for sending it in. Next time we'll do a better job. We're just also trying to make it so that sharing information with everybody else is not an ego boosting exercise because we won't remember your name or we'll mispronounce it unless it's important to you then let us know yeah no great expectations it's kind of cool it's uh this idea that uh we have a lot of tools are out for instance pytest

to to test your code but there's um in a lot of stuff the data that you're running your code through, like in data science or a lot of data pipelines, the data is important too. And being able to check to make sure your data fits what you expect it to fit, what look like is important. So these are some really cool expect calls. They're a bunch of functions that start with expect they aren't assertions so they aren't going to throw an exception but what happens is you like maybe on some of them you pass them a data frame like i'll just give one as an example you expect a column to exist so you give it a data frame and you give it a specific column and you want that column to actually be there and if it's not you get actually regardless of whether the, it comes back in the form of a JSON object.

And it has like a valid, and you can say whether or not, you know, if it failed or passed the test. But it also shows you the parts where it didn't. So this is, there's a little video, actually, demo video that they have where you can see it in action using a Jupyter notebook. And it's kind of cool. It shows you can see it in action using a Jupyter notebook. And it's kind of cool. It shows you exactly where it's failing. So I imagine doing this interactively to look at your data would be helpful, but also you could probably put this in place in some data cleaning steps to make sure things are around, like making sure there aren't any nulls in a column. There's a whole bunch of different things you can assert on or expect on your data.

It's pretty fun. Yeah, it's pretty cool. Some of them are totally straightforward, like expect these values to be in the set. Others, a little more data science, mathematical focused, like expect the chi-squared test p-value to be greater than such and such, right? I honestly haven't done a lot of chi-squared lately, but I don't do that much data science on the web. It's more we use addition and stuff like that. such right like i honestly haven't done a lot of chai scores lately but i don't do that much data science you know on the web it's it's more we use addition and like stuff like that yeah but some of the fun things like uh he was um in the the video example um he expected uh some data to be in a data from a column to be in a particular set and it was either male or female and first off like how

binary of this but anyway the males came a lot of them came back with spaces in them so doing some of this exploratory thing might tell you where you need to add some cleaning steps or dealing with nulls or things like that yeah the follow-up i'd like to hear how people might be able to use this how it's used exploratory wise but how i'd like to see somebody using it in their pipeline stages and how that works. I guess I imagine if you had it in production, you'd have some code that would, if it failed in expectation, you'd write a log entry or something, or I don't know what you do yeah maybe return if you were accepting data over a web method right somebody's doing etl and they're like here we're submitting this new set of data you could return like 400 bad requests to indicate no no something's wrong with the data you sent me things like that possible yeah you know what do you think about this i think it's very

cool and it's probably going to be pretty helpful i like it as well what i like about it is it lets you take what would be a little algorithm you'd have to write to, say, go through and test all, say, the chi-square values and then compare them and then assert on that. Now you just do one line, and it does it on the whole data set, the whole data frame. That's pretty sweet. Yeah, I'm going to be watching the video demo after the show.

Yeah, right on. So the thing that you are going to cover next, the timing is incredible, Nina. I literally got a notification that I had ordered one of these and it shipped today and it's on its way to my house. So I'm very excited. Tell people what is on its way. Would that be the Circuit Playground Express?

It would be. Yeah. Awesome. Yeah. So I wanted to chat a little bit about using Circuit Python and MicroPython to write Python for wearable electronics and embedded platforms. I've been playing with electronics projects as a hobby for probably about the past two years now. In the past few months, I've been focusing my attention on Python for microcontrollers. Right on. What kind of little things are you making with your projects? My last one was Python powered LED earrings,

and I dropped a link to the repo for the code as well as a photo so you can see that. Awesome. Okay, so what I know, I've heard of MicroPython, and I think I've heard of Circuit Python. Are they the same? Are they different? Tell everyone about it. Yeah, there's definitely a little bit of confusion about that. So MicroPython is the original. It's a lean and efficient implementation of Python 3 that can run on these tiny little microcontrollers. And all it needs is 256 kilobytes of code space and 16 kilobytes of RAM. It's incredible. It's truly awesome. Yeah, it's so super low level too, yeah? And CircuitPython is a port of MicroPython, and it's optimized for Adafruit devices.

Cool. So I guess that's the one I'm going to be learning about. Yeah. Yeah. And some of these, the devices that Adafruit sells, they're as small as a quarter. That would be the trinket. But of course, my favorite Python hardware platform for beginners is that Adafruit Circuit Playground Express.

It has everything you need to get started with programming hardware without even needing to learn how to solder. All you need is some alligator clips for the conductive pads. And the board has a ring of NeoPixel LEDs. It has buttons, switches, temperature sensors, motion detectors, sound sensors, a tiny little speaker, and a lot more stuff. Like you can even use it to control servos, which are those tiny little motor arms. Wow. That's really awesome. Yeah. It's a tiny little thing. Yeah. What I really like about this and the reason that I ordered it was I can go to Adafruit and look around and I'm just like, this is too low level for me. I don't really know

what I need. I don't even know if I have a power supply, if I get this little chip or that, like how to put together. I'm like, okay, give me like one thing that has all the stuff to do little projects like you're describing. And I think it's awesome. And doing it in Python is super cool. It only costs $25. Yeah, it's not too expensive. No. And then if you don't want to use Python to program this, there's a tool that you can use called Microsoft MakeCode. And it lets you program these little devices with a drag and drop style kind of scratch-like interface. So that's perfect for kids. And you'll find a lot of examples on their site. Oh, that's awesome that you point that out because I might get my daughter to do some kind of little game, you know, like a Simon Says or something with the LEDs.

Who knows? That'll be fun. Yeah. The Mew editor has some tutorials on using these as well. Right. That's awesome. Super cool. All right. And I like you threw in the link. Yeah. To your code so people can do the earrings as well. Yeah. And like you said earlier, there are tons of guides for Python projects on the

Adafruit website. Stuff from making your own synthesizers to jewelry to silly little robots. So definitely make sure to check that out. Cool, yeah. Well, I can almost recommend it. It looks really, really good. And when I get it, I'm going to play with it and let everyone know what I think. But yeah, it seems like a great little package

that you can get at 25 bucks. You know, it's like, you know, as software developers, you could just try it, right? It's not that big of a risk. It's not like you're getting a new MacBook or something. And Adafruit has a Python for microcontrollers mailing list, and they always drop kind of hot news and interesting new things. So there's a link to sign up for that in the show notes. Awesome. That's a great one. So this next one, I think, is a really interesting use of Python 3.7. So when you think of the main features of Python 3.7,

certainly data classes has to be one of them. Have either of you found any use for data classes yet? Yeah, I just love them. I use them like named tuples. Yeah, they're awesome. Trey Hunter has a great talk on data classes. Definitely. So this is a library that's derived from data classes

and it's a CSV file reader. Now, Python comes with pretty good support for CSV readers. You import CSV, and then you create a dict reader based on a file stream, and it'll read the header and figure out the columns, and then it gives you little dictionaries based on the column names, and that's pretty sweet. But what you get back is a whole bunch of strings corresponding to those values, right? And with this, what you can do is you can actually define a data class that maps the schema of your CSV file, right? So I could define a data class that maybe has an ID and a name and a value or price, right? Maybe it's like products.

So like the ID could be an integer defined in the data class. The name could be a string and the value, the price could be say a float. And it'll actually do all those conversions for you and give you meaningful errors if like it's a non-parseable float or something like that. Isn't that nice? That's incredible. Working with CSV is the bane of my existence. Yeah. So you can just define these things. You get autocomplete in PyCharm or Visual Studio Code for your types, for your rows, because they come back as these data classes. And you get validation. And on top of that, you can actually do cool stuff. Like you can say, with data classes, you can specify either just the type or the type and

the value. And that value becomes the default value. So if only sometimes the price is there, you could put zero or minus one, and it'll just go through and substitute that value becomes the default value. So if like only sometimes the price is there, you could put zero or minus one and it'll just go through and substitute that value. So a lot of cool little nice touches here. Nice. Yeah, I'm definitely gonna check this out the next time that I have to do some sort of CSV parsing

because it's way better to let this thing do the validation and the type conversion and all that and just not worry. I don't think I've been this excited about CSV files in a long time. I know. They are pretty amazing. No, this really makes working with them nice, and I'm excited, too.

All right, speaking of excited, I do want to tell you about Datadog. They're helping make this show possible. So before we get to our next item, let me tell you about them. This show is brought to you by Datadog. They're a cloud-scale monitoring platform that brings, like, metrics and logs, distributed traces all together. You can do auto-instrumenting of frameworks like Django and Flask and PostgreSQL,

which means you can track requests across service boundaries, across machines, things like that, which is awesome. It makes it really easy to troubleshoot your slow Python apps and figure out overall where the time is being spent. So you can get started for free at pythonbytes.fm slash Datadog. And they'll also give you a cool t-shirt with a Datadog

character on it, which is nice and cute. So check them out. It helps keep the show going. Now, Brian, I want to come to a topic that we haven't really covered very much on the show, but maybe I think a while ago we did talk about packaging once, right? Yeah. You want to catch us up on it? I think it's only second to GUIs so far. That's right. There's a fun article called How to Rock Python Packaging with Poetry in Briefcase.

Plus, it has like the phrase how to rock something, and I'm a sucker for that. It's actually kind of a nice tutorial on packaging. So for those of you that just joined us and haven't learned all of our discussions on packaging, this is a nice introduction to packaging and how it fits in Python, and also kind of how the changing of things, changing of packaging, like where Flit, Pipenv, and Poetry sort of fit in with all of this. It's kind of a nice run through of that poetry is one of those things for packaging and it uses the pyproject.toml file and of course we talked about poetry in in episode 100 and uh the detailed nitty-gritty

details of pyproject.toml was in testing code 52 with BrightCannon. But the neat thing, one of the neat things that why I picked this is we also, it also talks about Briefcase. And we haven't talked about Briefcase yet. And Briefcase is one of those tools from Py, the Beware project in is something that you can, it packages your, a Python application as a standalone native application for lots of stuff. It claims Mac, Windows, Linux, even iOS and Android, which is interesting. I haven't tried any of this. The tutorial talks about desktop distribution of code through Briefcase.

And it's kind of cool. And how to get that done with Poetry. I think Poetry is definitely nice. It's sort of an alternative philosophy to P-inf, which is, we talked about that as well. And Briefcase is part of that whole set of small independent tools from Beware, which also is pretty cool. And it's nice to see them working together.

Also, we got Cookie Cutter doing some magic in here as well. Definitely. And then one of the things that I like at the end of this, there's several tutorials on publishing, like how to push your new package to PyPI. But this one I really like because instead of telling you exactly how to push it to PyPI, they tell you how to push it to the test server.

And I think that's an important step for people to do. Before you subject the world to your code, try it out at the test server first. So it's nice. Of course. Yeah, of course. Why don't more of them do that? I don't know. But please, everybody, save the world and push it here first.

Yeah. The packages on PyPI cannot be changed once they're uploaded. You can only add newer ones. So although you've got to admire the rate at which you'll increase your version of your package if you screw it up a few times trying to publish it. That's productivity, right? That's right. So, Nina, this next one that you found for us is one of these awesome lists. And I think these awesome lists are coming along really faster and faster these days, right? We've got awesome Python, awesome Python

applications. What's the next one that's in that category? And this is a new one I came across called awesome Python security. It's a collection of tools, techniques, and resources to make your Python more secure. Oh, that's cool. Yeah. And it's got a lot of stuff for web apps, like the secure.py, which is awesome. We've covered that in the Flask, Flask Talesman, Django sessions, all kinds of stuff, right? But not just the web. There's a whole bunch of other ones. I think, and hopefully all of you agree, that all of your production and client-facing code should be written with security in mind.

And this list features a few resources that I'd come across before, like Anthony Shaw's excellent 10 common security gotchas article that highlights problems like input ejection and depending on assert statements in production. And I also came across a few that were new to me. So the OWASP Python resources, OWASP stands for Open Web Application Security Project. And there's tons of OWASP resources out there. I didn't know that there was a Python specific one. You can find that one at pythonsecurity.org. I came across Bandit, which is a tool to find common security issues in Python. And now Bandit has a lot of really useful plugins that test for some issues like hard-coded password strings in production, leaving Flask debug on in production, using exec in your code, and a lot more.

I linked to the full list in the show notes. And then a few other cool ones like DetectSecrets, which is a tool to detect secrets that were accidentally left in your Python code base. That's cool. Yeah. Let's open source that. Oh, wait. Was our full-access AWS or Azure key in there Python code base. That's cool. Yeah. Yeah. Let's open source that. Oh, wait, was our full access AWS or Azure key in there?

Whoopsie. Oopsie doops. We'll just check it in again without that in there. I'm sure it'll be fine. There's no history. And something I really like about this list in particular is it also includes resources for learning about security concepts like cryptography. Yeah.

You know, out of that cryptography section, they listed one of my all-time favorite packages, which is Passlib. So if you're going to store user secrets and you want to hash them, like passwords are probably the most common. But there could be other things as well that you don't want to store directly, but you want to accept from user and see if you have it. All right. Like you can hash it and that's a good idea. But what you really should do is like take that result, add some salt, then hash it again, take that, do it again, maybe a hundred thousand times, right? Pass lib, that's like one function, like dot encrypt rounds equal 150,000, 200,000, whatever. It's really nice.

That's awesome. To make sure your password doesn't end up on, have I been pwned? Exactly. Exactly. So basically you can say, I want it to take, you know, 0.2 seconds to determine, to brute force or to check each version of the password. And it automatically, because of that, will slow down dictionary attacks against your site. Cause you can only do them point, you know, it'd take only five per second. Right. So, and then there's another one called let's be bad guys. That's interesting. So yeah, a lot of cool stuff here. That's a great project name. I know. It's like a hacker playground.

Yeah, so check out the full list on GitHub. And then if there's something missing that you think should be there, maybe open a pull request. Yeah, absolutely. That's a cool one. I'm glad you found it. All right, the last official item I want to cover

is PyDBG, which is the implementation of a Rust macro called DBG. So just put the pi on the front. Now, I haven't done that much Rust. I've actually been wanting to learn Rust. It looks pretty interesting to me. But the basic idea of this DBG macro is instead of just printing out, like I'm here, I'm here, the value is, you know, printing X as the value, it'll actually give you a higher level statement without doing more work. So if you're

trying to debug something through print statements and that kind of thing, this makes it a lot easier. So you can go and say, like, I have a equals two B equals three. If I could say DBG of a plus B, the output is the file of the line a plus b equals five, things like that. Really, really nice. It sort of shows you in your message where you are in the file, what thing is you're actually printing without having to like come up with elaborate print statements. So pretty cool. Oh, man, I'm going to use this like every day. Do you still use print statements to debug brain? Yes, I do. I love print statements. I use the debugger a lot, but every now and then I'm just like, you know, I just want to print this

out and just see what is happening. Like I, I don't primarily use print statements for debugging, but sometimes I do when I'm kind of exploring that I want it to run, but I kind of want to see what's happening. I'm like, oh, what am I getting back from that API? What is this value? Things like that. Have either of you used watch statements? No, tell us about it.

You can just set up a variable or an expression to watch. And, you know, every time you hit a breakpoint, you're like, oh, I see what's in there. I don't have to type it again. With VS Code or PyCharm? Yeah. I believe you can set up watches with PDB2, but I don't know. I usually do those in a graphical debugger. Yeah, me too. Yeah, I definitely have

used those. I was thinking something different. But so where I'm going to use this DBG, PyDBG thing is a lot of times I've got, we've got test code that generates huge amounts of data, like trace data. And these are stored, and the test runs are really long. And throwing a couple of these extra ones for intermediate values so that failing tests or failing test runs, we can take a look at those post-mortem, things like that. It's an elegant way to have that be done. Yeah, it's pretty cool. What I like about it is it's an elegant way to to have that be done yeah yeah it's pretty cool what i like about is this like so simple right like you place the word print with dbg and you

you kind of got something going on here i also kind of like that it's more explicit you're like this is not really supposed to be a print segment this is just here till i figure out what's going on and we're going to stop this yeah but cool people can check it out thanks for sending it in to our listeners all right so i guess that's it for all of our main topics. But looking at our show notes here, Brian, we all kind of had a second round in the extras. So maybe we'll do like a lightning round one more time. What do you got for us? This is just a cookie that PyTest has temporary directories and temperature factory fixtures for dealing with temporary files. But they've added, as of PyTest 3.9, there's path versions that return pathlib

path objects. And those are just quite fun, and I'll drop a link in the show notes. Okay, that's great. So I want to bring your attention to something, let's say, non-standard in terms of conference presentations. So this is something at Py's say, non-standard in terms of conference presentations. So this is something at PyCon US. So in May, Cleveland, 2019, there's a project called The Art of Python, which is a miniature arts festival focusing on narrative performance and visual art around programming and Python, and basically showcase novel art that helps us share our emotionally charged programming experiences, particularly to do with Python. So it's like five to 20 minute

presentations and a separate little track and the call for papers are open. So if you've always been a theater fan and you program, here you go. Oh, this looks very cool. Yeah. Yeah. That's pretty interesting. So people can check that out if that connects with them. The other one is one of my favorite surveys and sort of put your finger on the pulse of the community items is the Stack Overflow Survey. Well, the 2019 one is open.

So everyone should go out there and represent for Python and fill out the 2019 Stack Overflow Survey. So that's good. And then finally, this gets a little bit back to your pick, Nina. NumPy is awaiting a fix for a critical remote code execution bug. That's bad. That doesn't sound super good. So yeah, I don't know if it has been entirely fixed yet. I don't, you know, this is a couple days ago.

It was not. So the idea is basically there's a problem with the pickle module. Have you ever, could you imagine there'd be a problem with accepting user input straight in pickle form? I can't imagine. So the idea is there's some part of NumPy that you can load pickled data. And, you know, for those of you who don't know, like part of the pickle statement is here's some Python code as a module, and here's arbitrary code to run as part of deserializing that.

So good luck. Oh, boy. Yeah, yeah. That's not so good. So this goes up to at least version 1.10 through 1.16, which at least is January 14th. This hadn't been fixed. So hopefully it's been fixed. So hopefully it's been fixed. But more importantly, if you're using NumPy, and you're accepting user input through the load function, you want to upgrade and you want to be a little careful around that. All right, then, then last one, I just want to throw out there

really quick, I ran across this, and I've known about it for a long time, but it turns out to be more useful than I thought. So I use Google Docs a lot. And I have like sheets in there. And I've got word like documents and stuff. But the problem is, if you use like Google Drive, what ends up on your hard drive, it's like a hyperlink back to the actual sheet, right? So how do you back that stuff up? It turns out if you go to takeout.google.com slash setting slash takeout, that's a lot of repetition. Anyway, you go there, you can say give me all my document format, all my documents, and it'll give them to you in Microsoft Office format. It'll convert the sheets to Excel, it'll convert the docs to Word docs, and then let you download them so you have a permanent version.

Anyway, I thought that was cool and people might find that useful too. Cool. Yeah. Nice. All right, Nina, you got some as well. You're teaching a class that looks really interesting. That's right. Yeah, I'm teaching a two-day introduction to an intermediate Python course on March 19th and 20th.

And that class is going to live stream for free at Front End Masters on each day. And all the course materials I'm going to release for free as well. That's really excellent. And it has an in-person component if you happen to be, where is it, in Minneapolis? In Minneapolis. That's right. You can come to the class. Will Minneapolis be thought out by then?

The class size is about 20 people. That I cannot promise. Hopefully in March, it's a little warmer. Yeah, so up to 20 people could drop in in person. That'd be really cool. The next thing, I recently recorded a series of videos with Carlton Gibson. He's a Django maintainer, maintains a lot of other projects on developing Django web apps with VS Code, deploying them to Azure with just a few clicks, setting up continuous integration and continuous delivery, as well as creating serverless applications. You can watch that video series at aka.ms slash python dash videos.

Yeah, that sounds cool. I definitely check that out too. Yeah, it's great. We got to film in the Microsoft Channel 9 studio, and it's a very well done series. All the bright lights and everything, not just screencasts. That's cool. Yeah, we feel like newscasters. I'm also planning on being a mentor at a brand new hatchery event at PyCon US 2019. That's going to be mentored sprints for diverse beginners organized by Tanya Allard. The goal is to help underrepresented folks at PyCon contribute to open source in a supportive environment. The details aren't out yet, but I dropped a link to where they'll be when they're finalized.

Oh, that's super cool. And there's also things like scholarships or something like that to help folks get actually physically to the event if they need some help as well, right? They can apply for that at the PyCon site. Yeah, PyCon US offers a lot of financial aid. Lastly, if you're interested in Python for hardware, like we talked about earlier, you can catch my talk about electronics projects in Python with LEDs at PyCascades in Seattle on February 24th.

Currently, tickets for that are still on sale. Excellent, yeah, and Brian and I are definitely going to be there. We're all going to PyCascade, so we're going to catch it. Hopefully, everyone else does as well. Great. I'm excited to see you there. And I do have one last thing to sneak in, and that is if you haven't tried the Python extension for VS Code yet,

now is a really good time. The December release included some really killer features like remote Jupyter support and exporting Python files as Jupyter notebooks. And if you're interested in keeping up with future releases, I dropped in a link to the Python at Microsoft blog. Nice. And didn't you all, you as a group, I'm speaking to you as a Microsoft, as a VS Code, didn't you guys just release like an AI powered autocomplete backend for Python as well? That's been around for a few months. I believe it's still in preview mode, but it works really well. The data set was trained on a bunch of open source projects.

Yeah, it looks super cool. So I definitely want to check that. I think I installed it just the other day. So it should be fun. Yeah. Try it. Let me know what you think. Absolutely. Brian, we've come to our joke section, right? Yeah. Nina, you want to kick us off? Yeah.

I found a bunch of really cheesy snake jokes, so here they go. What do you call a snake that only eats dessert? I don't know. It's a pie-thon. Nice. I'll do the next one. How do you measure a python? In in inches they don't have any feet Brian what's the last one what is a python's favorite subject I don't know what is it history

that's bad these are all bad lovely and I will not apologize no those are great thank you for finding those Nina or coming up with them. Either way, they're great. All right, folks. Well, thank you for listening. And Nina and Brian, thank you for being here today, of course. Thank you. Thanks for having me. You bet.

Bye. Bye. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.