Python Bytes - #85 Visually debugging your Jupyter notebook

Episode Date: July 3, 2018

Topics covered in this episode: the state of type hints in Python Flaskerizer PixieDebugger Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/85

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. It's episode 85, recorded on July 3rd, 2018. I'm Michael Kennedy. And I'm Brian Hocken. Hey, Brian. How you doing? I am doing great today. I know. It's another day in paradise here in the Pacific Northwest, isn't it? Yeah, as long as you're not driving, because it's the 3rd of July.

That can be problematic. We do have our traffic issues, don't we? Yeah. Yeah. Well, yeah, before we get to our items, I want to say thank you to DigitalOcean. They've been supporting the show in a major way. And so, you know, like I've said, our stuff runs on DigitalOcean, which is really great. You want yours to run on it, you go to pythonbytes.fm slash DigitalOcean, get $100 free credit for new users. I find myself, Brian, more and more loving the concept of limited use of type hints in just the right places. I was just typing that 15 minutes ago.

Were you? Yeah. Yeah, that's nice. The more I use them, the more I'm warming to them. And there's a couple of resources that came by just recently that I really enjoyed. So there's an article called The State of Type Hints in Python. And this is a fairly thorough article, but it's pretty much like, okay, now it's 2018. What is it like to use type hints now?

And especially, okay, the article talks about if you want to use type hints in 2.7 and prior to 3.6 sort of stuff. And that's doable, but I still think that you get the best experience if you're using Python 3, and especially Python 3.6. Because you can use commented type annotations in 2.7, I believe, because they're just comments. But adding the little arrows for the types and the colons and everything, it's more concise and it looks better. And my personal feeling is unless you really are trying to convert a large code base, which I know some people are, I would switch to 3 first and then add type hints,

if it was me, if you have that option. Yeah, and 3.6 is even better than, five, because you can put types on variables. You can, although I find my, I'm not really doing that when I'm using type hints. I'm just mostly doing it for interface to document interfaces. You know where I end up doing that is where there are core functions that I can't control. So I need to call a function and I know it returns something rich that I know what the type is. And I would like to be able to get better editor experiences, better autocomplete, but it's a, it's a package function, not something I've created. Okay. So then you can say this variable is a, you know, SQL alchemy session, for example. And then it's like, oh, I see. Now you've got add and delete and all the

stuff. And it's amazing, right? Yeah. And one of the things I like about the article and I agree with is the main point of a good reason to use type hints isn't, I mean, it is good to be able to catch more bugs and incompatibilities, but there's a lot of reasons why it makes your code more maintainable and more readable because you just have that option to be able to document it right there to say, hey, this this mean, when you're writing, yes, there are some great polymorphic functions that can take multiple types of type, multiple types. But, you know, most of the code I write really does expect one type. It expects a list or maybe the most generic thing is it expects an iterable. And I'm actually still not sure how to write that in the type shed stuff. I'm sure's a way but yeah this article does talk about things like unions i i ran my pi on my uh on some of my code and i realized that i had uh some uninitial some default values set to none which i wanted to be okay but if they're not none

they're strings or they're integers or something and the type system allows you to add a union type. So you can say, well, it's, it's either none or it's a string. And, uh, it's pretty interesting, but this article talks about a lot of stuff that I really had forgotten about actually is like interface stub files. And, um, I haven't worked with those at all yet. And I know that the type shed is around that allows these, that has these interface stub files for the standard library and a bunch of other popular libraries, but I haven't really used it yet. Or maybe I have, but I just don't know about it. But I don't know. This is good. Yeah, you have an interesting quote in here that type hints should be used whenever unit

tests are worth writing. What's the backstory on that? It's just one of the, in the conclusion of the article, it talked about that. It said, okay, all of this information, but when should you use it? I mean, if you're going to just write a little utility function for, I don't know, for your own use or something, do you need type hints then? And they just came up with this rule of thumb is if you're going to, I quoted it just as is.

It says unit test. I personally would just say automated tests. If the code you're writing is worth having automated tests, it's worth putting, trying to think about where you're going to put type. The other thing I wanted to add was I, I was listening to, I got about halfway through it so far,

but there's a recent seminar that Guido presented at Stanford called optional static typing for Python. And in part of that, it talks about MyPy, but it talks about some of the beginnings of this, of where this came from and how it came to be what it is now. And how are we doing with it now? So it's about an hour and a half, though. So it's a chunk of time to watch. It's a commitment. Yeah, it sounds interesting. Yeah, but I think I recommend even the first 20 minutes for some of the backstory is interesting. Okay. Yeah, that's really cool. I'm definitely a fan of it and I'm glad you're bringing it up.

There's some nice stuff in there. So I have no idea how to transition from Type Hints to MongoDB. So I guess that's a reasonable transition. That's it. So I've actually found a pretty interesting thing on MongoDB. People know that I'm a fan, of course, and one of the big decision points, I think, when you're trying to decide do I go with Django or do I go with one of these smaller web frameworks like Pyramid, Flask,

many of the other micro frameworks is the micro frameworks, you can pick the little pieces. I want SQL Alchemy here, and maybe this part's Alchemy here, and maybe this part's gonna be Mongo. And this part is going to be this other aspect that I would like to bring into this framework for authentication or something. Whereas Django is more like, here are your big building blocks, click them together, right? And a big challenge there can be like the

Django ORM. And the admin stuff, it's all tied into that. And well, if you don't want to use that, then you're kind of out of luck, right? There's a lot of challenges of sort of bringing your own thing. You lose some of the value, at least out of Django, traditionally. So the Django ORM, it depends on a, not a particular SQL database, but it does assume a SQL database. Is that correct? Yeah, I believe so. I believe that's right. And things like the Django admin section, which is really valuable to some people depends upon the Django ORM, right? So like there's these, these layers of stuff. And so if you pull out the bottom brick, there's problems.

But I actually, this guy named Robin sent me a heads up about this thing on Twitter that I had never heard of, but it's pretty awesome called django mongodb connector cool yeah so the idea is you continue to use the django orm just like you normally would except it's now had its back end swapped out to be mongodb and that's pretty awesome and first you're like okay well how much value are i'm gonna get in that right like? Like if I give up, if I continue to use what is effectively a relational ORM, like you can't use the hierarchical documents, right? But what they've done, a little bit like Mongo Engine, is the parameters that go into your query can be more interesting, right? So you would say, like, imagine if you had a blog, you say blog.objects.filter and you would pass like name equals

something in in traditional relational stuff but in manga you might want to traverse like into a blog object and then go find the name and then find that the name is contained in a set and so you just pass different arguments to the filter parameter and boom it becomes mongo hierarchical enabled it's beautiful wow that's actually pretty cool isn't that cool it's so simple yeah yeah and it says you can also use it to connect third-party apps like django rest framework and viewflow and other stuff which heavily depend upon these django models they can now super easily be integrated with mongod well. And it has a hilarious name. Yeah. What is his name?

Django. Yeah, or something like Django. The app, the official Django app you put in the settings file is Django. That's great. Yeah, because for, I get it, Django, Mongo. Django, Mongo, smushed together. Yeah, exactly. Another thing that's interesting is this is a Python 3.6 or higher only and Mongo 3.4 or higher, which is like, I don't know, a year, year and a half old, something like that.

So it's not quite so new. But Python 3.6 is really quite new for the store. Yeah, but I mean, it also seems like something you would start a project with. You probably wouldn't like rip apart an old project. Probably not. I mean, you may have an app you're trying to make better and migrate but yeah i would think you probably wouldn't really yeah yeah i throw that more in there just like it even has like a pretty high requirement for manga db just so if somebody's

got like an older or something or other you can't just jam it in there you gotta gotta make sure you got the new the new goodness okay um so i was actually so this is a not sort of related um i was talking to somebody about Mongo before, and they asked me about migrations from if you're going to change the databases. Does Mongo have that sort of concept? It does have that concept. You do it a lot less often. Much of the time what your migration is is something like I'm adding an additional column here.

I'm adding another table there, right? Like it's not that common that you're say deleting a column or you're renaming a column. It happens, but it's way less often than new column, new table. And in MongoDB, new column or new table is automatic. You don't do anything. It just, it just adapts, right? As you use it. So you don't actually run any scripts or migrations to do the common transformations. But if you're going to say like delete a column or reorganize your data, you would do the same thing and you would run a script against it. I don't think there's something like a limbic where there's like a framework for migrations,

but the concept of like a migration with a script definitely exists, but it's not very common. All right. And I brought this up hopefully because if we're wrong about this, somebody can get ahold of us and tell us. Yeah, if there's some cool framework for doing that, let us know. Awesome.

So what's this deal with Python idioms you got going on here? This was a tiny little article, which sometimes I like that, just a small article of like, hey, I learned this thing and it's helpful. And here's a small blog post about it. And this was Amir Rachnam. Why do I try to pronounce people's names? Anyway, multi-line strings.

So the idiom he's talking about is multi-line strings. So if I've got, and we know there's like a triple quote, you can have strings. Strings can be multiple lines if you use triple quote. And we usually see these with doc strings, but it can be really anywhere you're going to use a string but if you're using it like in the middle of a function it's awkward because it either you have to the all everything but the first line has to be is like over on the left side of your page and it's not indented the right place and if you do indent it over it's including all of those spaces in your string but the way to get around it is to use a standard library function the module or the package is text wrap and the function is ddent so it's text wrap dot ddent and that will um that will strip out. What it does is it looks at all of the lines of the multi-line string

and takes the common spacing at the beginning of it and just rips that off. I see. So it's just automatic. And I use this enough with generating tools and generating other things. And I can't think of a good example right now where I've used it, but I have used it enough times and I've looked this up that I want people to know about it. It's an easy trick. So it's good. Oh, I love it. I didn't know about it. Thank you. This is cool. So I've definitely had multi-line text strings because I've got some big formatted piece of text and I'm like, all right, I, instead of trying to break

this apart and turn it into something I can store in code if i just you know triple quote it and put it in here it's going to be golden but you do get that weird you've got to like indent it to the left outside of your function and it's all bizarro so this lets you can like keep your code looking really nice and it obviously puts the string to like visually into the block that it belongs in it's really nice i love it yeah and i've also like tried to punt and like put define a very global variable or at least a file global variable so that i can get around it but this is cleaner and it's it's good it's a good thing yeah i love it that's awesome other good things include digital ocean so definitely want to tell people about digital ocean because they're sponsoring the show in addition to just us being happy customers there. So you can go from zero to a customized server in 60 seconds.

I've talked about how we've done cool stuff with virtual machines, how they've got spaces, how they've got other various things that are pretty awesome, like the one-click containers for machine learning. One thing I haven't talked about yet is container distributions so if you want to set up your own like docker container you can go click over there and create a core os or fedora atomic server even a rancher os which i've never used but it's got a cool icon and you can just fire one of those up and it's like all ready to go to be your container host for all sorts of cool Docker stuff and Kubernetes. So definitely check them out over at pythonbytes.fm slash digital ocean. Get a $300 credit to play around with those kinds of things.

That's actually pretty cool. I like it. Yeah, I like it too. And I had a digital ocean accident the other day. Oh no, what happened? Well, I wore my favorite digital ocean t-shirt to work and I spilled coffee on it. But luckily it was a gray shirt and it didn't show up as a stain. So it's all good. It's pretty fail safe. Yeah. Unless

you're painting. So how do you feel about your design skills, Brian? Like, are you a fan of like shuffling around CSS and HTML and that kind of stuff? No, CSS is definitely those things where I either rely on a framework or I try to hire somebody. Nice. So something that you might be able to use if you're using Flask, and I know you've played around Flask recently, is this thing called Flaskerizer. Okay, I don't know what the name stands for. I guess to make Flask-ish or something. But the idea is you can go to one of these places that has Bootstrap themes. So Bootstrap is nice. Obviously, it's a CSS front-end design framework.

It's pretty well known. But what's really awesome about it are the themes. People make these pre-made themes for like $10 or even free a lot of times. You can get incredible web designs and you just drop them in, right? You're just putting your logic into the bits where it goes. So this Flaskerizer thing is what you can do is you can download the themes from certain locations that have a known format, and it will convert the bootstrap, the static bootstrap theme into a Flask app. Oh, that's cool.

Yes. Isn't that sweet? So instead of worrying about how it all goes together you just go you know command line boom now i've got like a dynamic running bootstrap theme and flask i like it yeah so there's a there's a couple of sites that they work with i don't know how general this is because i don't know how common bootstrap themes are but i think it's pretty cool and of course if you open up the website and you look around, you can tell that it's a proper design.

This is the GitHub repo because they go from just plain white, like no design, no logo, to clearly a designer with a black turtleneck. Yeah, he's definitely a designer. Yeah, definitely. Dark black glasses too. And a beard. Yeah, exactly.

Oh, he's all good. He's, definitely. Dark black glasses, too. And a beard. Yeah, exactly. Oh, he's all good. He's got all the boxes checked. Very nice. So if you want to do a Flask site and you want a design to go pick a theme and just start working with that design, check out Flask Riser. It could get you up and running really quick. Wow, interesting. I was reading down this. This is one of the few applications I've ever seen or the frameworks that uses Nose 2 as its testing. Oh, nice. What's the story with Nose 2? I only know about Nose.

Well, so Nose was sort of kind of abandoned, but Nose 2 was a, well, I'll offend anybody that's still working on Nose, but Nose 2 was kind of trying to reboot it and redo a lot of the, make a backward compatibility break so it doesn't do some of the things, but it does other neat things and clean up the code base. But still, it started kind of running, getting some steam at the same time PyTest was sort of getting a lot of steam. So I think that there's not a lot of projects that use it, but I mean, it's still reasonable. It's just since there's not that many people working on it, getting fixes and stuff is sometimes an issue.

Yeah, nice. Okay, well, very interesting. Very interesting. So are you learning Python now? Is this your next project? No, but I thought this was fun because I encourage every youth I meet and actually everybody I meet to learn Python. And I always tell them... Good advice.

Good advice. Yeah. The reason I tell them the reason isn't so that you can become a programmer. It's so that you can, like, it's a power boost to what are the other skills you have. So if you're a biology student, also learn Python because now you're a biology student that knows Python. You're now the favorite person in the lab that can actually solve the problem with the data that they got in Excel or something, right? Yeah. And it doesn't matter. It's science

or non-science. If you're an artist that also knows how to code, it's just going to help. There's no downside. And so a lot of times with code, people kind of have, they start writing code and it's precious to them and they don't want to throw it away because they worked really hard on it. I wanted to highlight this article because of its advice. I think it's a real Python article. It's called Learn Python the Methodical Way. I'm just going to read the steps. The steps are make your way through a tutorial or a chapter from what you're learning, a book or something, that teaches you some discrete four to six step skill. You're going to work through it while you're reading it.

Now write down the steps as distinctly and generically as possible. Put the tutorial and chapter and its solutions away and build your project from scratch and peek at the solution and the steps only when you get stuck now erase the whole whole thing and do it again and then go back completely put it away and then erase everything and do it again like a day or two later and this sort of erasing and redoing it i think at least a couple times is good for people because it gets rid of that preciousness. Because the second time you do something, it's always faster. Anyway, I just thought this was neat advice, so I thought I'd share it. I like it.

It's pretty interesting. I definitely feel like when you're new, you feel like I put a lot of effort into that. And so there's no way I'm throwing that away. But the really valuable lesson is actually the second time, it's probably better. And the third time you're like, whoa, this is really a much improved version of what I did. And it takes less and less time, of course, right? Yeah. And also you end up like it's easier to like throw away the cruft. And, you know, if it's just something you spend an hour working on, even if you had to redo the entire thing, it's only a wasted hour.

And you realize that you can really do it again pretty fast. And so the one part I kind of disagree about is I never code without resources anymore. But I wouldn't say try to do it just for memory, but try to use the normal resources that you would otherwise, like Google or Stack Overflow or whatever. Right, exactly. To look up things. I think it's silly to try to memorize the order of parameters to a function. That's what code completion is for and stuff like that. Exactly. That's why we have typefins, isn't it? Yeah. So I totally agree. I don't think these like completely abstract, you know, in isolation sorts of things make sense. But maybe the idea is like, don't just keep following the steps three times, but like try to recreate it from your mind. And to me, recreating is totally fine to go. But how do I actually insert that thing? What is the what is the function i call like that is a totally reasonable google search yeah definitely awesome so we've talked about debuggers before we talked about the breakpoint thing last time one of the areas where doing proper debugging and by proper i mean visual debugging not not command line

debugging but like visual debugging where you see all the the state of the system and everything that can be really tricky in jupyter notebooks right because you don't like attach them they're running in the notebook in your web browser right yeah i've never tried yeah so you can use pdb but that's like a command line thing it just happens to be the command line is a jupyter cell that's that's not the same i don don't think. However, there's this thing called Pixie Debugger. So the idea is you include this Pixie Debugger into your Jupyter Notebook, and then it becomes a visual debugger for each cell that you're working in. Oh, wow. It's pretty cool.

And they have a really nice video, and they've got some nice screenshots of the thing I linked to. And so it has one of those, I don't know what the right word is, like a magic command for Jupyter that escapes out of Python and speaks to Jupyter, like the percent percent. And you can just say percent percent, debug this function or debug this cell, and then boom, it just drops into a thing

where you can like step over, step into, view all the states of all the variables. You can set little breakpoints. There's like a little command prompt REPL underneath when it's paused and you can just ask it questions like type in a variable and it'll show you the value or even change the value of the variable things like that oh nice yeah this is neat yeah so if you're working in jupiter this is really cool and it's interesting because it comes from the the video that the guy did, David Tayeb, it was at the Watson data platform.

So a Jupyter Notebook hosted on IBM Watson, which was pretty interesting. Oh, yeah. So I didn't even know that was a thing. But apparently the IBM Watson data platform is another one of these hosted notebooks. And the reason I bring that up is he's like, well, we needed a really good way to debug these things called what they call pixie apps i believe and a pixie app is like a um a gooey interactive thing that you can put in a notebook that's kind of like a web app but kind of like a desktop app but in a notebook it's kind of funky so i don't know maybe i'll cover the pixie apps as well but that's also

kind of an interesting thing it's kind of like flask running inside of a notebook. It's bizarre, but cool. Yeah, neat. Yeah, so anyway, anyone out there doing notebooks and wishing for a visual debugger, check out the video that I linked to at the end. It's only like two or three minutes, and you'll know whether or not this is for you. It looks cool to me, though. Yeah, and having debugger tools available in any, wherever you're working is good. Yeah, I know. I always feel like I don't need a debugger until I'm like, geez, I really want a debugger for this. This is just not.

Until you really need a debugger, yeah. Exactly, exactly. It's not that often, but when I do need it, I'm like, oh my gosh, I'm so happy this is here. Awesome. All right, well, Brian, thanks you for sharing everything. Anything else you want to give a shout out to while we're talking? No, not today.

Yeah, not me today. Yeah, not me either. So, well, have a happy 4th of July. It is probably exactly that day when this comes out. Oh, you too. Thanks.

Yeah. Thanks. Yeah. Bye everyone. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S.

And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.