Python Bytes - #124 This is not the None you're looking for

Episode Date: April 5, 2019

Topics covered in this episode: [play:0:29] pytest 4.4.0 [play:3:47] requests-async [play:7:10] Reasons why PyPI should not be a service [play:12:35]* Jupyter in the cloud* [play:16:57] Jupyter Notebook tutorials [play:19:28]* Unique sentinel values, identity checks, and when to use object() instead of None* Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/124

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 124, recorded April 3rd, 2019. I'm Michael Kennedy. And I'm Brian Hocken. And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm slash DigitalOcean. Get $100 credit for new users. Tell you more about why that's awesome later.

All right now, Brian, I feel like we're going to go to your happy place. Yes, I'm definitely one of the testing nerds that's around. And I'm excited about the PyTest 4.4.0 released. There's a lot of stuff in the 4.4 release. And that's pretty cool. There's a lot of actually really neat things. One of the things, I'm going to save the big one for last, but one of the cool things is there's a test paths variable that you can put in your configuration, any file.

It tells the PyTest where to look for tests, and it kind of short circuits some of the searching everything under the current directory. Just what, like two subdirectories or something, but not everything? Yeah, like for instance, this is very common. If you've got like a structure where you got tests in one directory, you're sourcing in another directory, and then maybe some documents or something,

and it's a large project. If you just launch PyTest from the top, it'll look at everything, and it really should just look in the test directory. Right, exactly. That makes sense. So that's pretty cool. It's been around for a while, but it's kind of hidden. It just

worked that way. And what they've done now is they've added in the output it tells you which test paths have been set. So that's nice. Yeah, that's cool. And then there's a whole bunch of stuff that's relevant, especially to plug-in writers and a little bit for

test writers. there's actually quite a few things i encourage people if they're nerdy about pytest to check out the changed list the thing that i'm super excited about is there were a couple internal changes that made it so that there's a new plugin called the pytest subtests that works oh Ish. So PyTest has always been able to run unit test code. But one of the things that happened is if unit tests had subtests in them, and subtests are a way to have kind of multiple checks within a test. If you had those in there and one of them failed, it's supposed to keep going and check all of them. But PyTest would stop at the first failure. Now it doesn't. Now it continues on with running all of them. That's cool. So if I've got,

say, a list of five things, and I want to loop over those and test each one of them without some kind of parameterized test or something, I just want to loop over them all and write one test and do that. I would do that with a subtest. Yeah yeah or at least you can yeah and it works pretty good so it really was to the plug-in and the changes were really to fix this little corner case of the unit test support and so i'm really glad that they did that but they also threw in a new fixture called subtests so even if you're not using unit test you can use this new fixture and it has a subtest test context manager that works just like the unit test version, but you can use it in PyTest tests. Yeah, that's cool.

That's neat. Another one that I think is nice is Async. So you have the Async, you know, mark as Async decorator you can use. But previously, if you did that, but for some reason, like, reason like the right plugin to run the async say like paytest-asyncio or paytestrio wasn't there they would just not run but they were still marked as passed and now at least you get a warning and they're not run or something like this that's cool yeah that's that's a oops that's one way to make your test pass like let's just not install that async runner will be will be cool. Yeah.

Well, keeping with the async theme, I want to roll right into requests. So I literally just hung up recording or stopped recording with Kenneth Reitz. We're doing an episode on TalkPython, and we're talking to him about requests a little bit, but mostly other things. Anyway, this next item follows right onto that is request dash async. So currently, without this library or something like it, if I want to use async and await, and I'm doing web requests like microservices or calling like a Stripe API or something, I either do it synchronously with requests, or I can do it asynchronously, but I have to switch out my calling package. I have to use like AIO HTTP dot client session or something totally different. Rewrite my code. There's a package called request dash async, which is pretty cool.

And it's actually a different thing you import. But if you import it as requests, then you can do a wait request dot get a await request dot post and all of that. So it seamlessly fits into your async and await methods. Maybe you're testing with that pie test thing I was talking about. And it's basically the same API, but now it's async friendly, which is cool. That's very neat. Yeah. And like I said, there's other stuff you could use. You could use AIO HTTP, which is nice, but it means you rewrite your stuff, right? And a lot of people depend on like changing DNS resolvers and all sorts of crazy stuff deep down inside a request. So this will let you do that theoretically, at least I haven't tried all the edge cases, but let you do that without

rewriting your code, which I think is pretty cool. Now I did say I was just talking to Kenneth and one of the things he said is they are working on native async and await support for requests. So that's pretty cool, which would make this unnecessary. So why do I bring it up? Well, he said it's probably like not done for a while. It's probably like a year away because they're doing like major rework inside of requests. And, you know, think how many libraries and people depend upon requests. So, and how hard that must make it to change at all, right? Yeah. It needs to not break backwards compatibility. And it's not even just breaking it.

It can't slow things down either. Yeah, exactly. So they're talking about making a new package called request three. So you import request three and it has a slightly different API, just barely. So anyway, here's an interim way to get async and await working with request.

And another thing, there's a cool bit of testing going on here, Brian, I thought you might like. So I can mock out my endpoint. Like, let's suppose I'm trying to call an API or something, and I don't really want it to go to the real server. I want to give it a test one. I can create a Flask app or Cort or Starlet or whatever and create a mock like connection pool or something to that effect and give it my real website implemented in python and that's the mock behavior of the this thing okay well that's neat is there more information about that somewhere it's covered in the github repo that i linked to and if you go to the bottom down there it talks about it i don't totally know

what's going on exactly but for example like you know you want to call i don't totally know what's going on exactly. But for example, like, you know, you want to call, I don't know your API, and you could have like a little local one running on SQL, SQLite or something. And you could like literally give it the app, it won't even go to the network, it'll just like directly interact with the app as if it was the web server. Oh, that's really cool. Yeah, it's pretty wild. So anyway, this is play with that. Yeah, yeah, it's a pretty cool thing to be playing with. And it has some other cool async testing support. So it's got a cool testing angle as well.

Nice, cool. Speaking of services, what do you got next? I did notice that there was an article in the register about a bunch of... I don't know how a bunch, but some layoffs at NPM. Right. NPM is the commercial equivalent of PIP, sort of. And PyPA or Python Pack package authority like mushed into one right yeah i didn't know that it was a for-profit like company but apparently it is yeah that's

interesting all by itself right there isn't it yeah so dustin ingram wrote an article talking about and he dustin is uh one of the people that works with pypi and the packaging authority. And he wrote an article called PyPI as a service. And essentially, there's a couple interesting comparisons between NPM and PyPI. Firstly, NPM has been around for longer, I guess, since 2014. PyPI in 2002. Oh, no, we're older. So PyPI is older. Math is easy. Anyway, they have like $10 million in 2002. Oh no, we're older. So Pipe.ai is older. Math is easy. Anyway, they have like $10 million in funding and whereas in 60, around 60 people working there and the Pipe.ai has like less than

half a million in grants. And then just like, there's only half of a fraction of a single employee and then some volunteers. So there's a lot less people running PyPI and a lot less money. But what if they changed it? Apparently they've had a lot of people ask for an as-a-service version so they could have a private thing. So you could use something like PyPI but with your own stuff that's internal or whatever. And generally just talked about some of the problems with that. One of the problems is it might jeopardize the nonprofit aspect of the Python Software Foundation. That would be bad.

Also, there's a whole bunch of people that companies that donate services and infrastructure to make PyPI run at the tune of about a million dollars a year. And I don't know if, and he mentions that they might not be thrilled about donating that service to a for-profit company. I'm sure they wouldn't. I agree. There's also some of the ecosystem. There's other options out there. However, I've kind of looked into these and I'm not really thrilled with the other options available.

So I actually think there's room for somebody else to create something like PyPI for private use as a service that might be an opportunity for somebody. Yeah, for sure. One of them that comes to mind is PyDisc.com. That's in beta right now, but it's basically that. It's basically the service that people are asking for. Honestly, I totally see the problem here, right? We've got this careful balance of this actually really expensive thing to run

pipe. Yeah. Especially the traffic and the website and whatnot. And then we've got these donations taking care of that. There probably is a business model to be had here, but there's going to be like a, a wicked dip to zero near zero and it has to somehow you have to be able to climb back high enough to cover just the expenses right like that's pretty risky to do

that right also what do you do with the people that are now volunteering do you start paying them what if they only volunteer for a couple hours a week or a couple hours a month do you pay them for that yeah and then, they support some of these alternate services right now, PyPA does, and if they had an incentive to not support the other services and only support their own profit, that's not good. So, anyway,

it's an interesting topic. I honestly think that there's actually a huge potential, and I've talked to people in the community about this, but I don't know that I've talked about it on the podcast before, around around pi.org. Right? If you look at how much traffic is there, there's a huge opportunity for like non invasive ethical ads and other types of promoted stuff to be put on there. And I'm sure they could do more than a million dollars with the amount of traffic they have. But there would be that bad dip, and they might not make it through and so on. But I certainly see this as totally possible that

it could go that way. And it may be, you know, given that they're basically saying, hey, we need to receive a donation of $40,000, $45,000 a month of bandwidth. And if we don't, we're done, right? That's a really scary situation to be in as well they're dependent just on one company so i don't know it's it's super interesting uh the layoffs at npm that doesn't make me sound it make me really want to encourage ipi to go that way right it was interesting at the end the conclusion wasn't that it's never going to happen it's just right now it looks like it's not worth it it would be kind of a pain and it might backfire and so right now they're not looking into it but yeah it's not a never thing that makes

sense all right before we get on to the next one let me just tell you all about some cool features at digital ocean something they just announced is the digital ocean marketplace so it's like one click apps and server configuration for all sorts of tools and whatnot. So maybe you want a discourse server, a GitLab enterprise server, a MongoDB server, or even Django, you can go up to their create account, go into their marketplace and just click the button and boom, you have infrastructure all set up pre configured to run whatever app it is that you want there. So there's a bunch there. And I think you can even create more and add them if you want your project in there. So check them out at pythonbytes.fm slash digitalocean.

Get $100 free credit for new users and definitely get a play with that marketplace. It's quite cool. Brian, if I was looking to run, say, some data science Python in the cloud, and I maybe wanted to do that without paying any money, maybe I was a college professor and I wouldn't have my students do it or even high school, or I just didn't do that without paying any money. Maybe I was a college professor and I

wouldn't have my students do it or even high school, or I just didn't want to pay money or whatever. There's a bunch of these services now that'll run basically Jupiter in the cloud, right? Okay. So there's a couple of big ones. We have like Azure and we have Google, but there's some smaller ones as well. And a friend of ours, Kevin Markham, put together a cool article called six easy ways to run your Jupiter notebook in the Cloud. So basically, he went through and compared six different services, assuming they all had, you know, they got into this list, if they had all the characteristics in that they don't require you to install anything on a machine,

they're either completely free, or they have a completely free plan. They give you access to something like a Jupyter Notebook environment, you can to something like a Jupyter notebook environment. You can import and export real Jupyter notebooks through the IPY in B format. And they support Python language, maybe others as well. Out of all the ones he looked through, there's six that were decent that matched those criteria. So we have binder, maybe you've gone to GitHub, and you've seen like a GitHub repo that has some iPython notebooks in it, and a little binder, like a run in binder button. So if you click that, you can basically use the binder service and run any Jupyter notebook that lives in a public GitHub repo on binder. So you just click a button and say run this repo. That's

kind of cool, right? Yeah, very cool. So that's nice what i like about kevin's article is he goes through and says these are the pros these are the cons so like pros obviously this is free and easy if you already work on github it's just right there if you don't or what you have is a private repo then binder is not such a big help another thing that's big in data science is kaggle right so they have these kaggle competitions which are like here's a bunch of data try to solve a, you know, getting the data to tell you or training a neural network or something like that. So Kaggle is known for that. But they also have this thing called kernels, like a free service called kernel. So Kaggle kernels. And these are kind of like Jupyter notebooks, like super simplified ones. So you can run your stuff

there. That's cool. Maybe one of the bigger ones is Google Collaboratory or Google Colab. And as long as you have a Google account, it's like Google Docs, but for Jupyter Notebooks. You just log in, go. What I really like about this one is it's like Google Docs in that you and I could be working on a problem, and we could just both be typing at the same time and working right alongside each other. So the Collaboratory bit is super big there. Oh, that's nice. So that's pretty good. And then also you can run your code regular on a server through Google

Colab, but you can also run it on a GPU or a TPU, TensorFlow unit, processing unit, which is pretty awesome for something. I don't know if you have to pay for the GPU option, but still pretty cool. Yeah. Because a lot of people don't have, certainly don't have TensorFlow chips laying around probably. Maybe comparable to the Google one is the Azure notebooks. So, you know, this is a big part of what Python and Microsoft are doing. You know, what Python is sort of showing up in in the Microsoft space is over on Azure and Azure notebooks.

So they've got something that's more of like a project, right? It's not just a notebook, but it's like multiple notebooks, markdown files, data sets. So you kind of create these like project folder type things that you can run on Azure Notebooks. And that's cool. Also free. All of them are free. There's something I had not heard of, CoCalc for collaborative calculation. Also super collaboratory, like the Google Collaboratory one.

Yeah, it's pretty cool. It lets you do all sorts of stuff like Jupyter Notebooks, but also Sage worksheets and other things, which is good. And finally, from JetBrains, we have Datalore. And it's not exactly Jupyter Notebooks, but it's like a reimagining of a Jupyter Notebook, which is cool, but you can import and export Jupyter Notebooks from it.

So that's pretty cool. Like all the features and autocomplete and cool stuff that you get in PyCharm, but in a Jupyter-like notebook. Also real-time collaboration there. Those are all pretty cool. If people are out there thinking,

hey, I want to do some Jupyter in the cloud maybe for a course or for a collaboration or something, these are all good options. Very cool. Nice article, Kevin. And hopefully it helps some folks out there. Speaking of Jupyter, what do you got next? Yeah, did you have this down and I just didn't see it beforehand?

Or did we just coincidence? Just a coincidence. Okay, so my next item is Jupyter Notebook tutorials. So let's say you have some people you're trying to collaborate or not. And you have some people that aren't quite familiar with Jupyter Notebooks and you want to get them up to speed really fast. I'm actually in this situation right now, not for me, but with some people I'm trying to ramp up. And I found there's a ton of tutorials about Jupyter Notebooks, of course, out there. But I really liked there's two tutorials from DataQuest. The first is Jupyter Notebook for Beginners, a tutorial. And the second is

Tutorial Advanced Jupyter Notebooks. And it's by the same author, which I should have written down, but I didn't. It starts out incredibly gentle, but it's also useful and concise and quick. You can get through quite a bit right away. And so it even starts with like, if you want to install Jupyter on your computer, how to install it, how to get started up. You don't even, if you aren't even familiar with how these work, how to get going, a little bit of a discussion about the file type, the IPYNB, and then a run through of the interface and how it works to work with a notebook. And then even into data, so like loading data and some important things like plotting data right off the bat, and then even how to share, and then using save and

checkpoint frequently to make sure that you save parts of your notebook. It's like this saving often is something we take for granted with an editor, but it's kind of nice to have this within a Jupyter notebooks also. Yeah, for sure. And I kind of liked that. He says he stops the, the beginning one.

It's long enough to get people started right at there. But then the advanced one, once people play with it a little bit, getting into some advanced things, which like the, the magicsics keywords debugging uh shell commands logging uh using seaborn do using macros and all sorts of stuff in the advanced one which are not it's nice but it's not they're not too long you could probably take an hour or so and go through it and learn what you need and skip what you don't need yet. Yeah, that's cool. I've also been playing with using PyCharm to edit iPython notebooks. It works really good. I would recommend

if people are debugging notebooks, maybe not use write in Jupyter. I would probably use PyCharm for debugging. Yeah, PyCharm is pretty sweet for debugging. Anyway, so that's what I got. Yeah, it's nice. I think those definitely go well together. So people can check out those articles and then try them online. They don't have to install anything. All right, final one, Brian, it comes from friend of the show, Trey Hunter. And it's called unique sentinel values. I did any checks and when to use object instead of none. And I think this is just a really nice reminder, maybe something a lot of folks don't know, or haven't really thought about. Often we have to clear a variable and it may or may not have a value. Think of like computing the minimum of something. So the algorithm is you have the minimum value and you've got to set that to something and then you loop through all the items

and you're going to set it to the lowest. But how do you kind of initialize that? Do you set it to none? Do you set it to maybe negative one? But what if the values could be negative? What is that value, right? So often none becomes this thing that is the it's not yet set value.

But if none is a valid value or maybe in this minimum example, like none could actually be a value that was passed to you in the list that you're trying to find the minimum of, but you need to treat it differently than actually the not having, you know, like it gets really weird, right?

Yeah. So Trey makes a case for this, for something different in this article. I really like this. It's like, instead of sending a value to none, you can set it to object, parenthesis, parenthesis, and that allocates an object, much like none is an object, ironically, singleton and things like that. But object is going to be allocated on the heap and will never, no one will ever pass that thing to you, right? There's no

other use case for allocating an object object, right? You always have something derived from object, right? I don't know a string, a number, a customer, but not object itself, like the base class. So just allocating one of those is like the perfect sentinel value. And there's all kinds of examples where he goes through default values for a function, like default parameters for functions, how you deal with that, how do you deal with like this minimum example, what is the first value before you've looked at the list. And so you end up with really nice checks, like that set minimum equals to object. And then if minimum is not initial, right? Or is not this, you know, you set the initial to like this new object, right?

Then you go work with it. So super simple adaptation that solves a certain class of bugs and makes it really nice. That's interesting. Yeah, it's super simple, but it's, I guess, kind of a good reminder. And there's the example, he re-implements the min function that's a built-in but in python and goes through all the cases to make it work and shows how this pattern is much better than using none i'll have to hit up trey and ask him how to deal with that with type hints yeah because i'm used to oh yeah i know i know you can't just say optional i guess you got to do like union of

object but that's also not a good answer. Because that would be everything. Yeah. Yeah, right. Yeah, it's not so good. Interesting. But it's a cool idea. Yeah, it's a great idea. It is a little bizarre with the I don't know, with the typeins, you're right. But that's okay. I think you can get around it or just put like a, you know, ignore this one line type of thing. Yeah, well

I mean, and it only really will come into place if it's part of your interface to something. I mean, you'd probably do have to be careful with like MyPi and MyPiC and those tools. They'll probably freak out if you don't take into account. So, but yeah, I love the pattern in general, how it fits with type hints. That's like a different, different aside. That'll be Trey's fault.

Yeah, Trey, there's your follow-up article, man. Brian, that's it for our official six topics as usual you got anything extra you want to share with anyone i do i forgot to write it down but you reminded me because you're talking about sentinel values and uh sentinels were part of the the matrix right oh yeah they were scary those are the big metal things that went through like the underground tubes and they would like grab onto the spaceship with uh the good guys in it yeah yeah yeah but there's something that's not scary is the the cool computer screens that would like go downwards right right so one of the things i found which is uh i brought it up because i brought up pytest earlier was a plug-in called pytest dash neo and it runs all your PyTest.

Like normally if you just run PyTest by default, it'll print out like the test file name and then a whole bunch of dots. And the dots mean everything passed or like an F for fail. The PyTest-Neo plugin, if you have that installed, that happens, but it goes downwards and looks like the screen, like a matrix screen,

but actually it looks cool. But it's actually informative. happens, but it goes downwards and looks like the screen, like a matrix screen, but it actually, it looks cool. But it's actually informative. It has all the same information, just in the different direction. Whoa, I need to learn unit testing. Chunk. Yeah, that's awesome. I love it. Brings me back to the 90s. That's cool. Very, very cool. How about you? I've got a couple of things. Let's start with this one. So I recently released a free course. So people can check that out over at training.talkpython.fm or just click the link. I think we've talked about Kenneth Wright's new web framework

called Responder, right? I think so. I think so. Yeah, I'm pretty sure we have. So I decided to create a super short little mini course on it. So I created a almost one-hour long course on Responder and the framework. And it actually goes and and builds that we go and

build out like a cool api using responder and then actually consume it with vjs as well so there's a cool little vjs front end on top of that so if people want to get some quick exposure to responder maybe see some of the cool features that it can do kind of flask like but it does a bunch of other awesome stuff as well it's not it's not just flask check that out so check out the responder course like i said it's free so no no problem there you go check that out and the other one is really cool but i have to really remind myself to use it so imagine this i'm working an editor this is for pycharm or any of the intellij platforms like webstorm or whatever and i've got like a big monitor up a whole bunch of code on the screen and i have my fingers on the keyboard and i want to go up to that section where i'm doing like open something

something as finn or whatever right but i don't want to take my mouse and go over that i don't want to arrow up like 17 lines and then go to the right control whatever so there's this thing called ace jump have you heard of this noJump lets you hit a hotkey. I think it's on Mac. It's Command semicolon. And then if you type, let's say, F-I, it'll go and put a single character on every space that F-I is a substring of on the screen. Maybe the one I'm looking for has a J by it.

So I just have F-I and hit J. And it'll take you straight to it so it basically turns your screen your code into like you do a little bit of a search and then like one character keystroke to jump to that section of code oh that's cool it's super cool it's super hard to remember to do that not just era era era era or mouse or whatever right but it's if i can remind myself to do this it's gonna be great okay well i mean i don't use arrows man sometimes there's vi mode true true true anyway so no that's cool i'm gonna have to try that out yeah so you just basically like type as you search and the

whole thing becomes like sort of quick jump around your your. All right, are you ready for a pie joke? I am. I got to hit it a few times. What's the object-oriented way to become wealthy? I don't know. Inheritance. Okay. That's bad, right?

A good programmer is someone who always looks both ways before crossing a one-way street. Child, dad, why does the sun rise in the east and set in the west dad sun is working don't touch it all right i'll do one more for you because it's about python 27 triumphantly beth removed python 27 from her server in 2020 finally she said with glee only to see the announcement for Python 4.4. Yeah. All right.

That's pretty cool. I guess we'll leave it there, huh? Yep, that's good. All right, Brian. Thanks for being here. Thanks for finding all these items. Thanks. Yep.

Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Ocken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.