PurePerformance - 068 Swagger-Contract-based Testing for Microservice Delivery at Landbay with Chris Burrell

Episode Date: August 13, 2018

In our previous episode with Chris Burrell, Head of Technology at Landbay, we learned how they got rid of end-to-end testing in order to speed up continuous delivery. Today we discuss how they still make sure that no code changes in their microservice architecture breaks end-to-end use cases by leveraging Contract-based Testing using Swagger and several tools in the Swagger ecosystem, e.g: diff, code generation … - also make sure to check out Chris’ presentation at yCon called “CDC is dead – long live swagger”.* https://twitter.com/ChrisBurrell7* https://swagger.io/* https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources* http://uk.droidcon.com/skillscasts/11147-lightning-talk-cdc-testing-is-dead-long-live-swagger

It's time for Pure Performance. Get your stopwatches ready. It's time for Pure Performance with Andy Grabner and Brian Wilson. Hello, everybody, and welcome to another episode of Pure Performance. My name is Brian Wilson. I was about to say my name is Dinah Trace. Now my name is Brian Wilson. My name is Dinah Trace.

Hey Andy. I'm a little flustered as you know because, and I'm going to take a moment to address this because we are a performance company and And I have to say, even though, hi, Andy, first of all, hi. Hi, how are you? Sorry, what's going on? I'm getting all angry here. You know, I was just praising Microsoft with all the cool things they were doing when we were talking with, oh, come on now, I'm blanking on his name and feeling like an idiot. Donovan Brown. Yes, yes, yes, Donovan.

Doing awesome things there. But Microsoft really, really blew it with the latest Skype update. Just really is not conducive to recording multiple and having multiples running. And I think you're going to have a lot of angry podcasters maybe not even using you anymore unless you fix it. So please fix it, Microsoft. Anyway, now that I got that off my chest, Andy. It feels better, right?

Yes, I feel better. I can relax. I feel better. It feels much better. I can relax. I can breathe. All right. So we get to welcome another member into the Two-Timers Club today, and I don't think he knows what that is, so you might want to just give him a, as you reintroduce him,

give him a little brief idea of what the Two-Timers Club is because it's not like you're two-timing somebody, not in the colloquialism of, youism of 19-whatever-America. You're a dirty, no-good two-timer. Not that kind of way. Sure. So we had a couple of people that have been on the show more than once. That's why I call it a two-timers club, right?

And that's obviously people that want to even extend beyond that, three-timers, four-timers. And Chris Burrell is back with us from Land Bay. Last time we chatted, Chris, I think it was like, I assume, probably like a week ago, we talked about how you are deploying, how you build your continuous delivery pipelines, how you are getting rid of end-to-end testing and with that

focusing on delivering value faster to your end users. And then you brought up a very interesting topic in that conversation we had, and you touched based on consumer-driven testing versus contract-based testing, so focusing more on contract-based testing. Then you also mentioned that using Swagger. And I think there's a lot of things we want to discuss, and doing this in your second time with us. So, Chris, let's jump into it.

Contract-based testing, what is it? How do you do it at LandBay? What should people know about it? What are the good things? What are the bad things? What are the things people need to be aware of when moving down that lane? And then we'll take it from there. Cool. Hi. So yeah, so a lot of questions right there. But I guess at the basic,

you have two paradigms when you're programming. You'll either define a contract first, and then you'll code something behind the contract, or you'll code something first, and then you'll expose an API and so contract really is just a way of handshaking between two services and so one of the things we use land bay is a tool called swagger which is basically just a specification of what would a rest call look like what would an HTTP call look like as I guess a post does it have parameters what other parameters what the types of parameters what kind of return objects what are the error codes and so you can write that you could write that in JSON you can write that in yaml we use yaml slightly

more presentable and then there's all sorts of tools that and the swagger community have made available to just kind of give you a really nice printout or a screen grab of what the interfaces look like you can kind of look at your your models coming back. So in a nutshell, really, the Swagger tool that we use or the Swagger spec that we use is there to define how two services talk to each other. And so if I hear this correctly,

it seems like this is basically test-driven development. But the idea is that you start not from a test, but you start from your contract definition and then you can on the one side immediately generate your tests based on that contract but you can then obviously also start implementing the actual interfaces but the the contract that you define in this case you know using swagger or in json is is the thing that enables automated testing and then obviously gives you gives the developers the input what they actually need to implement yeah absolutely so we use it for testing but we do that testing actual the testing actually happens at compile time a lot of times so

for example we'll we'll build a spec we agree on a contract between the client and the server or multiple clients and the server. And then the server has a kind of richer interface because it's serving multiple clients. So it can generate at compile time what their classes are. And then you've got two options. Either you reuse that particular library that you've generated and you give it to all your clients. And then testing-wise, you've basically done the testing in that everyone's using the same spec there. There's not much else to do, but you can go a step further, which is actually I don't really want to tie dependencies

between two different services. So my client is only going to define the bits he cares about. So he only cares about one of the 10 calls and um in in that particular call that he cares about in the response he actually maybe only cares about two or three of the parameters so that's he really defines a really small area um which means the server can be free to do whatever he likes with the call, so long as there's those three bits. So then what we can do in the pipeline is take the swagger from the clients and just test that they still match the server implementation that's currently live.

So those three fields would be on the master, but the server would also serve 10 other fields, for example, in response, and six other calls for other people to use. Yeah, but did I get this correctly? You generate the client steps or the client code out of your contract definition. That means in the contract definition, you actually have to specify what different types of clients you're going to have. Or do you separate that and say, I have my contract for my backend implementation. And then I have my client contracts for my individual clients?

It might be the mobile app, might be the web app, might be something else. Yeah, so we generate our interfacing code twice off the same swagger. So in the first case that I said, where it's kind of totally build time, we've got one swagger and we'll generate some clients. They might actually be in TypeScript and other clients are in Java. And so the Java ones, they'll use Spring MVC annotations to kind of configure all the endpoints that we expose.

The TypeScript stuff has got all of the same models, but it also uses its own way of making AJAX requests. And then other Java clients maybe have got a Jersey 2 implementation. All of those kind of things are available straight out of the GitHub Swagger repository. But then we've customized some of those code generation things just to tweak them for LandBay,

add a little bit of monitoring in there, track a little bit of the usage and the logging and logging some stuff to elastic search as it goes through um so it's quite flexible in terms of how you do it but ultimately one spec and then generates multiple languages um that's cool and then the so in the pipeline then because you can do a lot of things at compile time validating if all the interfaces are correctly implemented um you last time you also mentioned you can then obviously immediately detect regressions in case the server implementation has changed in a way that will break one of the clients but in order for that to work and i believe i asked the question the last time in order for

that to work you actually have to test with previous versions of clients as well. So does this mean you go back in your GitHub repository, in your history, and then run tests with previous client versions? Or how does this work? So what we do is we know what's live. So we know what we've got live. And generally, we want to make sure that all of our interfaces are backwards compatible. So at our design time, we'll spend a lot of time reworking stuff if our interface isn't going to be backwards compatible. And that might be introducing a new call, getting all the clients to move over, and then removing a call if we can't make it backwards compatible. But if the interface is backwards compatible,

what we can do is at build time, we can check that the new client, the code that's going live for the new client, we can pull the latest Swagger that's live for the server. And then by doing a diff on the two, we can work out if they're compatible. So there's some tools like Swagger diff, or you can code your own. But there's some rules, right? So if you're adding a parameter in the URL, if it's a query parameter, that will be backwards compatible.

So those are quite straightforward rules to implement. If you're changing the URL, obviously that's a break. And so Swagger diff helps us to do that. It kind of takes two bits of Swagger and compares them. So if you're comparing a client, which only defines the bits it's interested in, versus a server, you're basically saying, are these compatible together? And it will output a list of those bits that aren't quite right. Cool. And then the Swagger diffagger diff i assume is a command line tool

so you can call it from your ci or it might even be a jenkins plugin where you can actually call all that yeah that's right so we're actually building that right now and so i think it's a ruby um uh client thing so you you can call it as part of your pipeline and then you just provide two files and it does the diff and then we just parse the diff at the moment. That's cool, that's easy What are the things that you ran into when you started implementing it in this way? Any things that people that want to go down the same route should be aware of? Maybe traps you fell into, problems you ran into that people should avoid?

Yeah, so I think the key question is, do you want to maintain different versions of your API in live, all running at the same time, or do you just want to maintain one version? So we go for the latter approach, which in some respects makes it a lot easier to manage because we've got, say, a service that handles money for our accounts, and we know what the interface is. There's one service that's there, there's one database schema.

Other people will choose to have multiple services around. So that's kind of more the Lambda type thing of, you know, Lambdas just stay out there, you don't need to necessarily clean them up after they're kind of not used anymore. So really, it goes quite straight. So we operate one version only. And I think that helps us because then you know exactly what you're comparing to. So I think that would be the key lesson learned there. As soon as you get multiple versions it's difficult so this works really quite well for internal APIs obviously if you're exposing this to

an external client you're only really focusing on the server side and you either need to ensure you're always backwards compatible or you need to start talking to your customers and say, right, you've got six months or 12 months to kind of upgrade, which is the typical way of doing it.

And I don't think anyone really tries to support much more than a version out of 12 months. Are you by any chance also using your Swagger definitions then once your services are in production to use them maybe as synthetic tests as well? Is this something that you do?

So we're not rolling it out. It's an interesting idea. I think there's a whole host of other tools that allow you to do that. So you can do class path scanning, which gets you all your endpoints and the models assigned to that. On the JavaScript, TypeScript side,

you've got your models in the browser. So we don't use it for that purpose. So I think the key thing is we've actually used Swagger to define the contracts and therefore push the quality testing up front, whether it's at compile time, which removes any kind of traditional type testing, or whether it's at compile time, which removes any kind of traditional type testing, or whether it's during the build pipeline where we're kind of just automating

it. But the idea is to remove it from the needs to put it in production. Oh, that's pretty cool though, yeah. And so if I look at the regular, let's say, API interface, API interface rest API and you have different parameters and obviously different permutations of parameter combinations is this something that that you can also automatically test all the different possible combinations or is this something where you somebody needs to sit down and actually extends the automatically generated tests or how would this work so that I make sure that most of the combinations

of parameters are correctly tested? Yeah, so I think that's the, I think I'd say there's a difference between functional business testing and contract testing. So with the Swagger side of things, you're trying to test that your services

will be able to talk to each other. They'll be able to sort the same language. You're not testing that what they're saying is gibberish, if that makes sense. So in addition to contract testing, we obviously do some kind of functional testing. We mentioned that last week with kind of service testing where we'll check business rules.

I think CDC takes it a step further, so we don't quite do the consumer driven contract testing um and that's that tries to um the idea there is you um write your consumer contract and what it expects on the way back and you can then replay that on the server exactly um but i think again you're not testing semantics there. You're testing the compatibility of whether it works because ultimately what you're running against

is mocked APIs, either a client with a mock server or a mock client with a server. And so the testing is never going to be that realistic. But it goes a step further than what we're doing right now. And with this, I know you mentioned last time that your goal is obviously to push as fast as possible into production.

And then in production, you obviously have your monitoring in place where you do your sanity checks after you do a deployment that everything is correct and still working well but with us both brian and i being you know having a big background in performance engineering and performance testing are you thinking are you can you also use these contract definitions to actually then generate and simulate load tests or generate load tests and then simulate load that actually makes sense yeah absolutely so it depends i guess what tools you'll be using but um i've used um what was it called a long time ago so there's tools like jmeter where um that kind of is based mainly you usually create your scripts by recording real interactions then you customize it but there's a few other tools where you can just hook into code. The name escapes me right now,

but the idea there is you can just create your client and then you just use your REST APIs. And so then you would be able to test your service in isolation, or if they're deployed to a particular environment, then you can just test them straight there. You still have to deploy that code into the environment, but that's easy enough to do if you've got a strong pipeline.

So we haven't done that, but definitely possible. So if your performance test looks like getting some code and then calling the code to run HTTP requests, then that's trivial. Cool. You mentioned in the very beginning there's different strategies on how you can come up with an API or with a contract. You start with the content, then implement it, or you kind of start with the coding and then figure out actually what kind of APIs you have. And either way, over the lifetime

of any service, I assume you are updating your contracts based on usage behavior, based on your requirements, based on feedback that comes back from your consumers. Can you give us some idea on how this works at LandPay? What influences the lifetime and the iterations of your APIs? How does that work? So I think on the whole,

the main influence will be around new features. So most of our APIs are internal. And so as we create a new feature, we may need to evolve a particular API first. I think one big lesson we've learned is originally we created new APIs almost for every use case. So a new story comes along and it's a new call. And that's got its merits because then everything is kind of standalone

and you know exactly what goes wrong if things fail. But I think I'm moving more towards actually reusable stuff, so the whole REST concept of kind of resources, but also just APIs that look further than a particular use case. So then you kind of extend it. Oh, we need this extra field rather than an extra call entirely.

So for example, we show a financial statement on our website when you log on. You could easily think of a developer going in and they create a new REST API, its financial statement slash the account ID. And then that will then go and look at transactions and the transactions will be kind of modified and built into a statement. But you could also build that into a slash transactions slash account ID. And then you just get the transactions out and the client is responsible for his own use

case i.e building statements so we're moving more towards the latter because it favors reusability it's cleaner interface small interfaces and generally from a quality perspective the fewer interfaces you have the better quality you have but do i understand this now correctly in this case your server-side implementation provides fewer APIs, but potentially delivers more data back to the client and then the client is then kind of offering new APIs to the client specific and maybe then they filter data or provide in a different way. Yeah, correct. With various controls in place.

So for performance reasons you may want to flag like the detail of so do you want a full version of this call or do you want the kind of summary version of this call um for on the whole for most use cases that won't really matter um so for example for transactions you would never uh you'd never give back to a user the entire set of transactions from his account from day one. I mean, that just wouldn't scale anyway. So you'd always be building in some kind of date range. But in terms of the breadth of data, yeah,

there's some considerations there in terms of how much data you send back. Cool. Is there anything else from coming back to where we get started today? You know, what is contract-based testing and Swagger? Is there anything else that we want to point people to? I mean, Swagger came up a lot today. So definitely a shout out to Swagger.

And I believe Swagger, they are part of their own company or they got bought, I think, by SmartBear recently, right? Except they're part of SmartBear now. And SmartBear also providing great testing tools on top of Swagger. So if people want to test it out, swagger.io, that's where you find everything. I assume there's a free version where people can get started?

Yeah, absolutely. So Swagger is mainly a specification, which is obviously free. It's an open API. I think it's version 3 nowadays. And then the tools we use are all free. So Swagger CodeGen kind of gives you Java code, C sharp code, TypeScript code from your Swagger interface. And then we don't use the UI, but if you've got external clients,

you can package a Swagger within the UI and just kind of send a package zip file. I've had that on previous projects before Landbay. And then the editor we use just to kind of write. So there's plugins for IntelliJ, but we'll always kind of put it in the editor just because it's got a few extra checks that it makes. So there's plugins for IntelliJ, but we'll always kind of put it in the editor just because it's got a few extra checks that it makes. So that's quite cool.

But yeah, we use none of the paid stuff. I'm a big believer in open source and so on. But somehow these software companies also need to make some money to provide tools. That's when you click the donate button. Exactly. No, and also we are using Swagger at Dynatrace. So for most of our APIs now, REST APIs, you get a Swaggerized UI,

and that makes it obviously easier also for our customers. So we also saw the same benefit as you said, right, having a standard, making it easier, accessible, automated documentation of all the APIs. That's all great. Brian, how about you? You kept quiet today. Yeah, you guys are talking about five to eight levels above my head, so I got nothing on this one today.

Sorry, Brian. That's all right. I'm in a different world than Vandy, so it's absolutely fine. Yeah, I got nothing except for, for i guess hit the donate button that's i can't even come up with funny comments and and i was also so thrown off from that stupid uh it's kind of in the beginning yeah yeah but anyway it was i was i was listening very intently i hope everyone else was i was listening very intently trying to see if I can absorb some of it, which is, I think, what people on my idiot level can do. Now being self-abasing, see?

Andy, do you guys have anything else to bring up on this topic then? Or actually, one thing I want to just bring up slightly. The one thing that did hit upon, you guys mentioned, and this is just so I can maybe sound a little bit intelligent, right? You mentioned keep maintaining one version, right? Because maintaining multiple versions, especially if it's public, it's a lot more important in your case. It's a private API, so you can maintain the one version.

And Andy, I don't know, we have some of those documents we can do the meetups based off of. One of them is the idea of you can either do things like maintain multiple versions or try to make your piece backward compatible. And I guess what you're indicating is that since it's an internal version, you, very large environment where you don't necessarily know who all your clients are, then maintaining more backward compatibility within a, you can more communicate that more directly. But there would be some situations or maybe some of our listeners who might be in a very, very large organization with tremendously gigantic software structures might have to consider that maintaining more backward compatibility

in a private API. Yeah, no, absolutely. So we do maintain backwards compatibility on the whole. I think the sacrifice we've made is we're not going to deploy multiple versions because we're going to always ensure that if we have to break backwards compatibility, then we'll change our clients. In the sense of a large enterprise, maybe it doesn't scale so much. I think the stuff around the pipeline is if you're tracking your clients, so they're still internal clients. So if you're tracking which clients you have, then you can have a data store of those clients somewhere. And then you can test against those and see exactly which ones would break and so you wouldn't be quite in the same situation as say a large enterprise who have got loads of clients but they don't know who the clients are for a particular service and therefore you need to maintain different versions right i think that's the key is is knowing who those clients are and tracking them as you mentioned absolutely Absolutely. That's what makes or breaks it.

Yeah, I think that's also what we see with the people we work with, the backward, we call it the backtrace. If you monitor a service in your production environment, because we capture all the requests that are coming in, but also where the requests are originated from. So knowing who is calling you, what type of functions are they using. So that's a great

way to learn from production monitoring, right? What's really going on out there. And then either then reaching out proactively to these clients that you may not have been aware of before and giving them a heads up about changes, even using it proactively to advise them on better usage of your APIs, right? I think some things that we have observed with other customers is that if you bring up new APIs,

yes, you can communicate these new APIs through blogs, through education, through whatever channels you have. But still, sometimes people don't, you know, read up on everything that changes and maybe stick with older APIs or older calls for far too long. So therefore, I think production monitoring and understanding the usage of the APIs right now allows you to specifically target clients that have not yet been updating their usage to the new APIs and say, hey, come on, you know, we had this new API out for six months now and you're still using it in the old fashioned

way. So, you know, my solution to that, Andy, because this came up because I had posted something about one of our integrations on Slack. And then someone asked about it. And I'm like, I posted on like three channels in Slack. I think it would be very useful if everybody had to wear a shock collar. And then if they did something like that, they get automatically zapped and their screen gets taken over. This was telling you exactly where it was posted and say, why haven't you used it yet? And just shame them and shock them at the same time yeah I could make a lot more of a

fun work environment especially being in an office hearing people scream randomly that's basically inserting malware through your interface great I always have great ideas any rule so Andy want to go ahead and summarize? Sure. Let me summarize this a little bit. First of all, thanks again, Chris, for being on the show the second time. This time talking, obviously, about contract-driven testing. And I think what I learned is it is amazing what we can actually do.

The concept of contract as code, obviously. Everything is code. But if you can write down our contracts as code, we can not only generate the steps for implementing the server, we can also generate the clients. We can, with that, automatically test compatibility and fully automate it. And that's the cool thing.

Now, there's a new set of tools out there that we have to look into. You mentioned Swagger multiple times, which is a great way to define your APIs. There's tools that allow you to diff between versions. So automatically add your pipeline checks into your pipeline obviously, so that you're not breaking any of your API compatibility.

And then I think in the end, we drifted off a little bit to what we can learn from our real users of our interfaces to stay up to date on who is using what and also making sure that we use this data maybe as influence for future API development and how we evolve our API by analyzing the usage patterns right now or educate folks that are still using the APIs in the wrong way. And we definitely, Chris, want to link to your lightning talk. And I'm sure if you have any other talks coming up or other presentations where you talk about this topic, or maybe if you can recommend anything that people should look at or where you started your research on it, then please let us know. We're very happy to put this on the podcast page as well. That's it from my side. Well, thank you, Andy. Yeah, I was going to mention the Skills Matter

page for the lightning talk myself. I actually had a chance to watch it before this all started. You would have thought that would have been enough to get me up to speed, but my brain just isn't as advanced. Anyway, yeah, check that out. And I'll reiterate Andy's sentiment about any pointers for people getting started, good places to start. We'd love to know them and share them. And I'll also put out to any of our audience members as well, if you're doing this stuff and you have any fun tales to tell, make sure you get in touch with us and let us know.

Maybe we can have you on the show talking about it. We'd love to share everyone's experiences and learnings with this. Chris, any final thoughts from you before we take off here? Probably just because we're talking about resources. The GitHub page for Swagger is pretty good, and especially if you drill into the Swagger code gen side, if you go all the way down to the modules, you see there's a whole list of all the supported languages. I saw earlier during the call that it's kind of a Jmeter

is supported, for example, so we're talking about performance testing. But everything is like five, six different variants of TypeScript. So really, the GitHub repository is quite a good source of information just from looking at folder names. Okay, excellent. Yeah, if anybody has any feedback, reach us at pure underscore DT, or you can send an email the old-fashioned way. But we don't have an address for handwritten mail. Although, if you want to send us a letter, contact us through an electronic means,

and I would love to give you an address to send it to. Anyway, you can email us at pureperformance at dynatrace.com. Anything else? Or is that it? Thank you. I want to say thank you again for, or welcome Chris to the two-timer club

There was a competition. I think we have a few people have been on three times I don't think we've had anybody on four times yet. I forget I forget where we're at Andy It's gonna be very disappointing because I don't remember Where we're at and even who's at so so it really doesn't Whoever gets to the next level. I don't even know what it is. We need some monitoring for that and keeping track of it. We do. We do, absolutely. Yeah.

All right. Thank you so much. Thank you. Thanks, guys. Bye-bye. Bye-bye. Bye-bye.