Python Bytes - #147 Mocking out AWS APIs

Episode Date: September 11, 2019

Topics covered in this episode: rapidtables Quick and dirty mock service with Starlette Mocking out AWS APIs Single Responsibility Principle in Python Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/147

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to earbuds. This is episode 147, recorded September 4th, 2019. I'm Michael Kennedy. I am Brian Ocken. And this episode is brought to you by DigitalOcean. Tell you more about some cool stuff they got coming up later. Right now, I want to talk about tables. Okay. And really quick, what's 147 in hex?

No, just, I couldn't do that. I don't expect you. Anyway. No, but one of the things I do need to do occasionally, I need to do really fast is generate tables. And I know it's not something that everybody needs to do. But if I've got a whole bunch of data, like in this case, some dictionaries list, and I want to be able to just print that stuff out. They're all the same type of dictionary. They just hold some data. I want to print them out. How do I do that quickly? And for test reports or whatever, I found this called rapid tables and it's really kind of awesome.

Oh, that is so cool. Wow. Yeah. So you just basically, you've got a list of dictionaries and then you want to print them out and you want it to look like a good ASCII table. It shows you how to do it. And it also shows you how to do it with a thing called termcolor.colored. I didn't know this was around. So you can do colored tables if you want. I probably won't do that.

It also converts stuff to markdown and restructured text. That's kind of neat. I might use that. But right away, I just need the just really easy tables. And it says it's super fast in speed, like speed wise, it goes fast. But what I like is that it's not a lot of extra code gunk in your code. You just kind of say, I want this table and you can print it out easily. Yeah, that's super cool because a lot of times like this kind of formatting, it's a lot of Z fill and pad these strings and L pad,

like just like trying to get the spacing and you're putting tabs, but every now and then the tab, like the word is too big. And so the tab will shoot it to the next column. It's a lot of work. And this is not you doing that work. Right. So for all the columns,

you have to go through each of the elements to find the longest one, to figure out how long you, how wide you need to make the column. And it's, and yeah, there's other libraries to do this, but this one I think is pretty cool. Yeah. It's pretty cool. I also like the markdown and restructure text output option. Yeah. Very cool. Yeah. So if you think about what is like one of the most popular packages, what would you say? Take some guesses. Requests. It's gotta be requests. I don't know if it's the most popular packages, what would you say? Take some guesses. Requests. It's got to be requests. I don't know if it's the most used one, but it's certainly the one that has people's

attention and imagination. So what would you think if I told you here is a competitor to request called HTTPX? Ooh, I like the X. It's extra cool. Absolutely. What's really interesting about HTTPX, though, is it's not a from scratch, like throw away the ideas of requests. It's compatible with the request API. So what you can do with requests, you can do with this. So you can kind of drop it in. But it has all these other cool features. And in fact, it comes from this project called ENCODE, E-N-C-O-D-E, ENCODE. And over there, they have a ton of cool things, including Django REST Framework, Starlet, uveacorn, HTTPX,

and also some other async stuff. And a lot of the things happening around here, uveacorn databases, ORM, HTTPX, is all about adding the asynchronous capabilities to these various things, right? And that's one of the shortcomings of Request is it doesn't support an async option. But this HTTP, it does a bunch of cool things around the async space to kind of let you gradually go up the stack of complexity. So you can just be straight

synchronous stuff doing what request does. It has this concept of parallel requests. So you can issue a bunch of requests in parallel, but you don't have to think about actually creating the async event loop and stuff. You just say, I want to run these things all in parallel. And you could even ask it, say, like, I want to process the responses as they come in. So I can say, like, while parallel.hasPendingResponses, the response is parallel.nextResponse. So I can kick off like 20 requests and just process them as they come back in like four lines of code or something.

Oh, that's so cool. I like it. Yeah. Or if you really want, if you're in something like Starlet, where it has actual async support for the method, you do want that to be async. So you can actually flip into like an async mode where those happen the same way I described, but also on the async IO event loop. So it's really cool. It also has some other neat features. So like I said, a request compatible API.

It does HTTP2, which like I could do a request for a page. And as I pull down like, say, five CS style sheets and a JavaScript and an image, you can go get those all in one request. So it has support for that, which is cool. It has a standard synchronous interface, but also async and await if you want it. And Brian, you'll like this one, it allows you to make direct requests to WSGI or ASGI, the asynchronous variant of that apps. So imagine I'm creating a flask app, I don't want to test it, I can actually inside of my flask app, provide HTTP, the app, right creating a Flask app. I don't want to test it. I can actually, inside of my Flask app, provide HTTPS the app, right?

The Flask app object. And then I can issue requests that don't go through the network layer. They just go through this test app instance that I've created. Oh, that's cool. Isn't that cool? So if I want to mock out some behavior or interact with it as if I were externally, I don't have to start a server and talk to it i'd literally just like use the same api here but like directly skipping all the infrastructure oh that sounds fun yeah in like you know a nerdy kind of fun yeah that sounds like

really fun but yeah it is also fully type annotated so it's very like python 3 friendly uh has 100 test coverage and yeah it has support for mocking out things a lot of cool nice features and the fact it's built all this is built on top of requests means if you're already using requests it's like you can adopt it pretty easily because you don't have to change anything but if you want to bring in some of these cool new features it's just you know you start using nice now one caveat i does say that this is in some of these cool new features it's just you know you start using nice now one caveat i does say that this is in some form of alpha it's like pretty early days for this project but yeah it says this this project should be considered an alpha release it's substantially api complete but some areas need to work well you know people start using it people start contributing to it

all of a sudden it becomes not alpha so anyway it, it's out there for people to use. This is sort of the same umbrella area as Tom Christie and the Django rest framework area. So I think it's a pretty good place. Cool. Okay. Neat. Speaking of stuff under the encode GitHub project. So Starlet is part of that as also? Yeah. Okay. So interesting that we did not plan this. The universe planned it to line it up for us. Oh, my gosh. We're wearing the same color shirt also. Oh, my gosh. It's black, so that's not that weird of a coincidence. But anyway, so the article I wanted to highlight was Quick and Dirty Mock Services with Starlet.

This is by a friend of the show, Matt Lehman. Hey, Matt. It's kind of a neat thing. So one of the ideas, I don't know if we've talked about this on the show, but the idea of testing. So if you've got something that you're part of your application or the thing you want to test talks to a third-party service, sometimes you might not actually want to call that third-party service. You can mock or stub out the calls to the service, or one of the ideas is to create a fake service to talk to.

And so that's what this article is about, is how to create a fake service so that you can use it for testing. And Matt writes a really quick one using Starlet. And I think you could use other frameworks too, but I like how simple this is. One of the needs he had was that he didn't want the service to like just come back right away. He wanted some delay because there were some of the service calls that were taking up to a minute or longer, and he wanted to be able to simulate that within his fake service so that you can test the code under test to make sure it is working correct with a big delay. Anyway, like it was a really easy to read write up, easy way to

do that. And I know that that's a, it's a recommended practice for a lot of instances. And this is, but there aren't any very many write ups of how to do it. And so congrats, Matt. I like it. Yeah, that's, that's really cool. You know, when you think of dependencies, you think of, well, my database layer or Stripe API or something, but time is one of those dependencies that's really tricky. Most of the time, one of the reasons why we try to do a mock-a-service is to eliminate that time. You want to make sure that the flow of the data goes quickly, but you don't want to wait for the service during your test. But at the same time, you kind of want to make sure that you're... And in most case test cases, maybe that's the right thing,

but you also want to make sure that your, your application deals with long latencies if that's the normal situation. Yeah, for sure. You might want to shrink it from five minutes to five seconds or something. So the tests don't run super long, but yeah. Or even like, I want to start something and then see in an hour if it's done, you could like, you know,

have to fake out like, you know, daytime dot now. So the next time you ask it, it's like two hours from now, but it's one millisecond, you know? Yeah. Well, and you, you know, make sure your parameterization of that test doesn't make it so that you're running 50 different versions of the hour long wait. Yeah, for sure. All right right, now before we get to the next item, let me tell you about some new services from DigitalOcean. So they've traditionally had recently added Postgres as a service. So now hosted database, which is really cool, but maybe you don't use Postgres.

Maybe you want to use Redis. We just talked about queuing and stuff. So Redis is now a service that you can get over at DigitalOcean, Redis as a service, and also MySQL as a service. So check that out over at pythonbytes.fm slash DigitalOcean, get $50 credit for a new user. They've got all sorts of good stuff coming your way. And of course, all of our infrastructure runs on it as well, which is fun and very good. All right, the next thing I want to talk about is that other larger cloud provider. Have you heard of something? It's some people started using this,

Brian. It's called AWS. Have you heard of this? I think so. Yeah. Yeah. I think like a quarter of the internet at least runs on it. So the Python API, the Python package that used to talk to things like S3 and EC2 and AMI and all the stuff over at AWS is called Bodo. Yeah. Okay. So there's a project brought to our attention by Giuseppe Consolo. Thank you for that, Giuseppe. Called Mocking Out AWS APIs with Modo. So if you're going to mock out Bodo, what would you call it?

Modo. Oh, nice. Now, it lets you do all sorts of stuff. Like if you say, want to mock out an S3 bucket, right? So S3 is like file blob storage. You can mock it out and then like save, quote, save a file to it and then see if it's there or preload it with data and then interact with it. And it's got all sorts of interesting stuff. But if you start to dig through this library, it turns out it's got both some really interesting ways to test stuff. And it's also super comprehensive. So for example, if I want to test working with S3, I can say from Modo import mock S3. And then I just put an app mock S3 on my method and my test method. And now all of my true S3 calls with Bodo actually don't go to Bodo. They go to this Modo mock version. Oh, that's neat.

Yeah. But apparently it doesn't really end there because if I wanted to work with like EC2, I could import mock EC2 and put that on there and say Bodo dot working with, you know, do all the Bodo client EC2, whatever. It's like, I don't know why Bodo is so terribly complicated. It's like super overloaded, but anyway, that's a different time for a different story. You can go and like, ask for your reserved instances and it'll use your mocked out ones and things like that. So just like if you were to go flip through all the stuff that it supports, ACM, API gateway, auto-scaling, cloudM, API Gateway, autoscaling, cloud formation, CloudWatch, CloudWatch events,

Cognito Identity, Data Pipeline, DynamoDB, et cetera, et cetera. And we're only into the Ds, right? It just goes on and on and on of all the different AWS things that it mocks out. So if you got to test AWS, this is pretty killer. Like you can mock out Glacier, IoT, Kinesis, whatever, you name it. One of the notes you added was that it can run as a standalone server mode so that you can even test non-Python code. That's right.

That's right. So apparently I can, yeah, there's a way to run it as a server and then talk to it as if it were some of these services as well, which is pretty wild. It is incredible. And I guess good that Bodo is all of the interface so that it's one point to mock it. Yeah. Yeah, Bodo, the reason I don't like it is you say Bodo and you create a client and then you pass it a string. The string could be like EC2 or S3.

And then how you interact with it has to do with what factory that string drove it to to dry to create a different like you it's really hard to discover what can I do with this thing because it's it's like a generic blob that that can be converted into something more real I don't know it's it's funky but yes in this regard it's definitely very nice apparently your life is more complicated than mine I guess I've worked with S3 and Elastic Transcoder and all that stuff too much. Yeah. Okay, cool. What's next? I was just thinking that maybe we should have done a testing code episode with all this testing stuff. Yeah. But one of the things that we have, I think we've

talked about Mongo Engine before. And I know that you've used Mongo Engine in some of your courses, right? Yeah, for sure. And on and on the sites yeah and it's like a mappers um they don't say orm because object relational mapping because it's not really the r so they say odm for object document mapper that's like the okay the adapted acronym so i didn't know that there were so i knew about mongo engine because of you but i didn't know about others and here's a project called micro mongo or muewMongo? Not sure. Yeah, I think MicroMongo is the way you would say it. I think that's the Greek. Yeah. It's a Python MongoDB ODM. And it comes because the people that made it had two needs. They wanted an asynchronous ODM. And for some reason, they felt like it was difficult to

do serialization and un-serialization of documents with existing ODMs. And so this one does it. So it's an asynchronous sort of thing. It works directly with some of the popular Mongo drivers like PyMongo, TxMongo, that's for Twisted, MotorAsyncIO, and MongoMock. And I'm kind of excited about MongoMock. Yeah, for sure. I got to say, this is probably the most interesting aspect of this because something like Mongo engine, it just works with PyMongo. But this one says, you know what, we actually can work with all these different foundations, which gives it like fundamentally different behaviors and features, right? If you pointed at the twisted one,, it can now integrate with Twisted's Async or its model. MotorAsyncIO now lets it plug into true AsyncIO stuff. Like you could plug it into

Scarlet like you discussed before and those Async methods there. And then the mocking as well, like we're just going to take the foundation and make it a mock version. All those are super cool. Yeah. And then also because of this, I learned about MongoMock, which is a cool way to mock out your Mongo database. So kind of neat. Definitely want to try this sometime. Yeah, it definitely looks interesting. I don't remember exactly when this got created. Do you know what GitHub needs? Is it needs a, this project was created on this day, right? Because you can go back to the commits, but if there's thousands of commits, you've got to like infinite scroll your way back to the origins. But anyway, yeah, it's really cool. I think if I had known about this at first, I might've even chosen this over Mongo Engine,

but yeah, it's really nice. It's nice. It has a 90% plus test coverage. I really like the different foundations, which means you learn like one odm style of programming but then you can use it in these different situations yeah super cool i like it and also i didn't know about manga mock either but now i'm excited about it that looks cool nice all right so let me hit you with um an opinionated piece here and you can give me some feedback let me know what you think about it some parts of it... Okay, I got my tomatoes ready. Yeah, perfect.

Get ready to throw them. I'll get ready to dodge them. So this article was sent over by Tyler Madison. Listener, thanks for sending that over. Now, it's called the Single Responsibility Principle in Python. And the Single Responsibility Principle, or SRP, comes out of this larger group of design principles, coding principles, architecture principles, something like that, called SOLID. So S is the Single Responsibility Principle. O is the Open-Close Principle.

L is Liskov's Substitution Principle. And these even come in really interesting demotivator-style posters. I don't know if you're all familiar with demotivators. You're probably familiar with motivators. Like you will see this probably in the entryway, like kind of near the CEO office. It's like probably got a giant eagle soaring above clouds, like together we can fly higher than you ever dreamed or something stupid like that. Well, the demotivators take pictures like that, but put like negative captions and connotations. So these are really interesting things. For example, the Lisskov substitution one has a duck and a rubber duck in the water. And it says, if it looks like a duck and quacks like a duck, but it needs batteries, you probably have the wrong abstraction, right? So there's all these, there's all these really cool demotivator style posters around the solid principles. And I think the solid principles are pretty neat. Like you can go overboard with them,

but they are pretty powerful. So, this article focuses specifically on the single responsibility principle. And I think talking about it is really interesting. I think the solution is unexpected and it made me think. And at first I'm like, Oh, I don't like it. But after a while, but it is pretty creative and things that make me think I like to share with people. So I guess that's why I chose it. And it starts out by saying this article will guide you through the complex process of writing simple code. And it talks about stuff that you would expect, right? So there's, there's some kind of super long function says this is terrible, you can't test it, you can't reason about it because it's doing five things.

How do you know whether you're doing too much? You can look at measures like the cyclamatic complexity, the Halstead complexity thing, the number of arguments, the number of statements, body length, all that kind of stuff. And that'll help you condense down into, say, reasonable classes or functions. And this is really mostly focusing on functions. So all right, well, let's think of another simple example. I want to calculate the price of a bunch of products given a list of products. And it seems all well and good, except for that it also happens to use some kind of logger object and log out the price. Well, now this thing is doing two things.

It's one calculating and logging it out. What if you want to test it or work with it with its computation bits, but not the logging bit or something like that? So how do you go around dealing with these types of dependencies, right? This kind of goes back to the testing side of things, Brian. How do you deal with things like files and database connections and or database connection objects and you know loggers and whatnot so this guy takes us through this whole sort of thought process which is really good and says you know what we're going to do we're going to come up with callable classes and the callable the callable classes will be given the dependencies and then

you call them like functions allowing you to pass different dependencies or other dependencies and now that seems like a really bad idea but he's using they're using a lot of it and my first thought was okay well that's you know, that's like a curvy shaped weird hammer that I guess you could hit it with. But it's using a lot of interesting Python features. So if you create a class that has, it's a data class, you set frozen equal to true, slots equal to true. It means all of its variables become internal and private only. You said it'd be private or final rather, so it can't be derived from, and you give it a call function.

It turns out to be pretty interesting. And then they talk about some dependency injection frameworks that work with these types of classes. In the end, it comes up with a pretty interesting way to create these things and then call them. I'm not sure that I love it, but it definitely made me think, and I was a little bit surprised by it, so I thought you all might enjoy it.

Yeah, it's an entertaining read to give you that. Look at the very last code fragment in that article, Brian. Okay. The last code fragment that says, like, send postcards equals container dot resolve. Send to these postcards. Yes, exactly. And a little bit above it.

Yeah. Yeah, yeah. So, it is powerful in that you can create these objects. You can pass in mock objects without actually using mock dot patch and stuff. It's interesting. I don't think that I would go write software using this today. But I might use some of the concepts of it, maybe. My opinion is that it is possible to do this, and it might make sure that you don't get fired because nobody else can edit your code.

Yeah. Okay, so I do like the single responsibility part of solid. I am not a fan of the Liskov substitution principle or the open-closed or dependency injection is something that frightens me. These are things that were put in place to work around limitations of other languages like Java and C. And I'm going to get tomatoes thrown at me for coming up with saying this, but some problems are complex. And you can remove the complexity out of one part of your code, but it moves to somewhere else. And maybe it makes your unit tests better, but your system tests are just the same.

Or it makes writing regular code super hard because you're like, where did this database connection come from? I have no idea how I got here. Right? Yeah. It is a thing. It's an interesting article. I'm not a fan.

Yeah, I submit this to everyone as a thought piece, not necessarily as guidance. Yeah, but I'm not willing to say that it's wrong. It's just my opinion. All right. Well, I think we should leave it there for our main topics. Got anything you want to share with folks this week? I've got one quick one. Oh, you'd share it with us then. right the dot es extension i just go challenges put the period between g and es and that's a place where you can go do like practice exercises and it's a cool platform they've created and they

just came out with byte 220 where they walk people through analyzing the python bytes rss feed oh cool yeah so you can go in there and it shows you how to use feed Parser, which is a really cool way to parse RSS feeds. Go in there and actually answer questions. What were they looking at? It was things like, what is the average duration of an episode in seconds? What are the number of episodes with special guests? Like Trey was on the show last week, for example. What are the most common domain names we mentioned?

Like GitHub probably is right up near the top and things like that. So it's pretty cool that there's like this guided challenge to get through our platform, at least our platform's data. Yeah, I'll have to come up with ways in our show notes to break their challenge. No. Yeah, that sounds good. So that's all fun.

People can check that out if they're interested. You ready for some jokes? Definitely. These come from Web Boss's Dad Jokes GitHub repo. So they're pretty short and sweet, but they're in the dad joke style, which is kind of bad jokes that are hopefully funny. So what do you get when you cross a computer and a lifeguard? I don't know. What?

A screensaver. What do you get when you cross a computer with an elephant? Lots of memory. Lots of elephant? Lots of memory. Lots of memory. Lots of memory. Okay. Nice. So I'm going to share this because I made it up and I thought it was funny, but it's

an anti-joke. So a Python developer, a PHP developer, a C Sharp developer, and a Go developer all went to lunch together. Oh my gosh, what happened? They had a nice lunch and all got along fine. Perfect. Of course they should. They're all developers.

Anyway, thanks. Awesome. Yeah, well, thanks as always, Brian. See you later. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news

item you want featured, just visit Pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.