Screaming in the Cloud - Episode 44: Disagree In Commits Console Recorder for AWS

Episode Date: January 9, 2019

Do you have some spare time? Can you figure out an easier way to do something? Then, why not build some software?! Today, we’re talking to Ian Mckay of Kablamo, an Amazon Web Services (AWS) consultancy. He is the author of Console Recorder, which is a browser extension that records your actions in the Management Console to convert them into SDK code and infrastructure as code templates. Some of the highlights of the show include: Timeline to build Console Recorder Infrastructure as Code: How to code repeatedly without starting over and take ownership of what you built by hand AWS vs. Individual Achievements: People asked AWS for years to create something to record console click-throughs that Ian did in his spare time Console Recorder support for any browser that exports Web extensions Sharp edges of what’s expected of Console Recorder to speed up development Management Console’s unreadable responses require reverse engineering Console Recorder: Recommended use cases and areas How to alleviate security concerns with Console Recorder Changes to Management Console that may break things Ian’s past, present, and future projects and products Words of Wisdom: If you don’t like something, just fix it yourself Links: Ian Mckay on Twitter AWS Console Recorder Kablamo AWS CloudFormation Terraform MediaLive Jeff Barr re:Invent CDK Google Cloud Platform AWS Management Console AWS RDS AWS Lambda DigitalOcean .

Hello, and welcome to Screaming in the Cloud, with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This week's episode of Screaming in the Cloud is generously sponsored by DigitalOcean. I would argue that every cloud platform out there biases for different things.

Some bias for having every feature you could possibly want offered as a managed service at varying degrees of maturity. Others bias for, hey, we heard there's some money to be made in the cloud space. Can you give us some of it? DigitalOcean biases for neither. To me, they optimize for simplicity. I polled some friends of mine who are avid DigitalOcean supporters about why they're

using it for various things, and they all said more or less the same thing. Other offerings have a bunch of shenanigans with root access and IP addresses. DigitalOcean makes it all simple. In 60 seconds, you have root access to a Linux box with an IP. That's a direct quote, albeit with profanity about other providers taken out. DigitalOcean also offers fixed price offerings. You always know what you're going to wind up paying this month, so you don't wind up having a minor heart issue when the bill comes in. Their services are also understandable without spending three months going to cloud school. You don't have to worry about going very deep to understand what you're doing.

It's click button or make an API call, and you receive a cloud resource. They also include very understandable monitoring and alerting. And lastly, they're not exactly what I would call small time. Over 150,000 businesses are using them today. So go ahead and give them a try. Visit do.co slash screaming, and they'll give you a free $100 credit to try it out. That's do.co slash screaming. Thanks again to DigitalOcean for their support of Screaming in the Cloud. Hello and welcome to Screaming in the Cloud. I'm Corey Quinn.

I'm joined this week by Ian McKay, who's currently working at an AWS consultancy called Kablamo. But the reason I invited him here is a piece of software that he wrote and I tripped over over the past week on Twitter, which is the source of all things that we find on the internet, called Console Recorder. So let's start at the beginning. First, welcome to the show, Ian. Thanks, Corey.

Happy to be here. Always great to talk to new and interesting people doing exciting things. So in your own words, what is Console Recorder? So the Console Recorder is a browser extension that lets you record your actions in the management console and convert that into SDK code and infrastructure as code templates, such as CloudFormation and Terraform. So it winds up going through effectively what people click through while they're building out their infrastructure and converting that into CloudFormation. That's right. Yeah. So it supports more than CloudFormation. It'll support the Python Bodo library, CLI, JavaScript, Go, Terraform, and just recently the CDK library.

I have to start by asking, was this something you did yourself? Were there other people involved in building this? No, it's just something I did in my spare time. It started off with me using the MediaLive Bodo libraries libraries and the create channel function for that is just tremendously long. It's like six or seven pages just for the request. And I spent two hours just trying to develop this and spec out all the properties that I needed. And I thought there's got to be an easier way for this. So I did some searching around and couldn't find something. So I just decided, hell, why not just do one myself? When you say you did this in your spare time, how much time are we talking about? So I probably started this in around September. And it's taken about this long just

to reach about 100% coverage with the CloudFormation templates. I'm sitting around 40% coverage with Terraform and about 20% with the APIs. Does this only work on specific limited services at this point, or is it more or less everything in the console? Yeah, so my target is to support everything in the console, and it will support everything for a CloudFormation template at this time, which is about 330 resources. Terraform I'm trying to spec out now.

It should be done in about a month, I would estimate. That's got about 450 resources and I'm supporting about 180 there. And as far as the APIs go, there's about 5,300 different calls in the AWS landscape. And I'm supporting about 17% of that at the moment. So I started off by focusing on the CloudFormation side of things, and then I'm working on the other side stuff now. I think one of the reasons that this has sort of resonated is it echoes around the internet. At the time of this recording, I think Jeff Barr referenced it somewhat recently as well. It's sort of taken on a bit of a viral sort of aspect to it. And I think a lot of that seems to come from the fact that many of us have been asking for years for something like this from AWS.

For those who aren't familiar, what generally tends to happen when you build something in the console is you click around, you experiment until you get something that works the way you expect it to. Cool. Now, here in 2018, or really this past decade, there's this concept of infrastructure as code. How do you wind up doing this repeatedly without having someone else clicking in the console in the same place as you did? And the official answer from AWS has largely been, okay, take everything you've built in the console, throw it away and start over and do it in CloudFormation. And other things as well do support this as you've referenced Botto and Terraform and the new CDK. But there's never been a direct take ownership of the thing I just built by hand. And that's been a,

I guess, sort of a sore spot for an awful lot of people for a very long time. Yeah, I agree. A lot of feedback that I've got just recently when this is starting to blow up a bit on Twitter is that constantly I'm hearing, why wasn't this a native feature? Why wasn't this just here to begin with? And I spoke to one of the solution architects when I was in reInvent recently, and they said, we looked into this two years ago, and it was just deemed to be too hard to implement,

just because there's so many different variation in the way that the console works, because these are in siloed teams. And I hear what they're saying, and I believe them, that they're being sincere when they say that. But as you said, you did this in your spare time in the last couple of months, and you've now gotten it to a workable state as a single person. I get that AWS is built around the concept of small teams, but it seems to me that it wasn't as much, I guess, heavy lifting, for lack of a better term, that has to be insurmountable for a number of years. I mean, what did you see that I guess

they didn't? Or was this just a matter of you deciding that you'd dive in and see what it took? And it turns out, surprise, you're incredibly productive when you start working on these things. And by spare time, you mean the 16 hours a day that you're not at work. Yeah, that's right. There's a couple of different ways you could take this in that you could record the actions that the console is doing from a point-and-click point of view. But the way I implemented it is to record the network responses that come in and out of the console

and then use that to translate into what the SDKs and the templates and those sort of things expect. You've also built this into an extension for both Chrome and Firefox? That's correct, yeah. So it should support any browser that exports web extensions. So technically, it would still work in Opera or Brave or any of those sort of browsers as well. Yeah, realistically, once you get out of the major browsers

that you tend to see in almost every environment, then it starts to be a little bit of a point of diminishing returns. I'm sort of astounded it works on something other than just one. The fact that you've to see in almost every environment, then it starts to be a little bit of a point of diminishing returns. I'm sort of astounded it works on something other than just one. The fact that you've supported multiple browsers and multiple output languages is kind of astonishing. Yeah, it's interesting. I started off with just the Python and the CloudFormation support, but as I saw the different tooling that is in the community,

I've decided to expand it out a bit. The most recent one being the CDK, which is not even in a production release yet, but is being actively developed. So I'm working closely with that team to make sure that that is up to date. One thing that I have always praised GCP or Google Cloud for has been that when you build something in their console, they spit out what amounts to a congratulations good for you here's how to do what you just did programmatically so as you go ahead and click through the console which i admit is my preferred way of experimenting with and trying new

services and things i haven't really wrapped my head around yet and turning that into something that you can use programmatically it just seems like you've effectively duplicated that particular feature, which has been high on people's wish lists for a long time. So forgive me if I'm coming at this from a place that it seems too good to be true. What are the, I guess, what are the sharp edges on this? Where is it not going to necessarily do what someone might expect it to? Yeah. so it's important to remember that this is doing as best as it can to record the actions from the console, which may not necessarily map to an SDK call or an API call. So it can be particularly verbose, especially if you want the very bare bones configuration in your templates or your calls.

So if it says, I'm going to be explicit with this property that nobody really uses, it will map that property over. So sometimes it can be just a bit too verbose, but it's meant to be a good baseline for people to speed up their development. So it's not just like an out of the box tool. Like most people should have a bit of tweaking to deal with it after it's been generated. Yeah, I guess that's sort of my next question, given that I've been too busy marveling at the fact that this exists. Credit where due, I've been a little slammed the last couple of days and haven't had time to play with it myself directly. How usable is what it spits out, as far as being able to take that, drop it into an existing environment and iterate forward? Or

does it require a fair bit of manual massaging to turn into something that is repeatable and usable? Yeah, I think out of the box, it's very well supported. Most calls and templates should be able to be inserted straight out of the box, assuming those resources that you just created in the console don't exist anymore. There will be some slight deviations and there's obviously going to be some bugs because every single call is manually mapped. So if anyone sees any bugs, then happy to reach out and then approve that and fix that in the code. As you went through this and iterated forward on it, presumably you were experimenting on this yourself as well as you did this. Did you encounter any bugs in the console itself?

And how did that unfold? Probably perhaps not some bugs, but the way the console works and the way the network calls are is that it's so diverse for each different service. Down to the point where some network responses from services are borderline unreadable. CloudWatch, VPC, data pipeline. I'm looking at you guys. The console responses are insane and requires a significant amount of reverse engineering in some cases just to figure out what's going on there. And there's definitely some cleanup that we can see in the console responses. So you see uppercase and lowercase deviations a lot. There's that classic problem with ARNs and AMIs in the community where it deviates between the different

cloud formation and Bodo. And then this is just one layer on top of that. Thank you, incidentally, for pronouncing AMI correctly. As listeners to this show know, that's something of a sticking point for me. And I think you're right. I think that if we take a look culturally at what drives Amazon and how they build things internally, they're famous for their small teams working on individual services. And that works really well at getting a particular service from conception to a 1.0 that the world can use in a relatively short period of time. But whenever you wind up seeing the shared services that everyone has to use, it tends to wind up in a bit of a mess. The example that I live with in my consulting is always the bill, where all the different services

interact with different billing dimensions, and it turns into a bit of a mess. The console is another good example. People who know what a front end is supposed to do, of which I am not one of them, have been complaining about the user experience in the Amazon console for a long time. And some of us who aren't that deep into that space just tend to accept that it's going to take a little bit of work and be difficult to massage into place the first few times you do things. We sort of view it as an incentive to start doing things programmatically with CloudFormation or one of its similar services. So I guess what I'm wondering is, are there certain use cases that you see where this would be ideal for? And or are there certain areas where you would absolutely recommend no one ever try this and expect it to work? It's good for most use cases. I don't see any particular services where this doesn't work out very well. There's definitely

not 100% support in the CloudFormation console for all services. So you won't get any recognition calls or any of that sort of thing. So you still need to use that. But I haven't seen any services where it's definitely just don't use this. Perfect. One question that I would be remiss if I didn't ask, are there security concerns that people should bear in mind when installing this extension because they read about it on Twitter or heard about it on this podcast or saw Jeff Barr talking about it. And I guess, how should they reassure themselves that this is something that is not going to destroy their company out from under them? Yeah, that's a very important question. I think the first thing to know is that this is in a record-only aspect, with the exception that you can potentially block

readable calls as an option, which is not named by default, but it's a record only extension. And look, there's always going to be this concern where potentially the extension can be taken over by a third party. For those people, it's always available to look at code and it's always available to download directly from the GitHub repo, analyze the code and make sure it's not sending data externally or that sort of thing. And you can install that directly locally. So if you do have those concerns, you can basically vendor the extension as it is now and download it and use that. And then you can see exactly what's going on. A pattern that I've used myself and would recommend if people have a sufficient level

of security concern is I disable console access for myself in my production account for the things I use that power this podcast, my newsletter, my consultancy, etc. But I have a developer account where I can click around and do all kinds of interesting things, but contain no actual data of my clients or people who subscribe to the newsletter, etc. And the rule is something has to be built programmatically to get into the production account. So there is no, I guess, production secret in the developer account. To that end, if someone winds up putting secret credentials or whatnot into the console, does that wind up getting captured by Console Recorder? Yes, it will. So that's part of the post that we put on security is that you should always check your templates for any credentials or any secrets

that may not be necessarily good to keep in code or committed and make sure that they're extracted out at that time. So it's a very brute force tool. It's not going to look into any specific code. It's going to say, this is what you put into the console. So this is what I'm going to put into your code. If you go ahead and do a recording of console recorder yourself, and then hand the output to me without any tweaking or any editing, ignoring the credential piece for a second, is that something I can take and run and have success with almost immediately? Or do I have to go through and factor out things such as hard coded account numbers, specific availability zones, for example. How, I guess, portable is it between accounts? Yeah, it tries to do the best job it can in recording resources that connect to each other.

So, for example, if you were to enable the response interceptor and then create a security group and an instance at the same time, it will link them in the CloudFormation template so you don't have a hard-coded SG reference or an IDash instance identifier. This is, I guess, sort of a question that isn't necessarily going to help you with anything, but if you would go back in time and we're starting this project again today,

what would you do differently, if anything? When I started this, there was a lot of change in the way that console responses came out. So I made the conscious decision that any recordings that come in would be purely mapped as code. So it's just basically a bunch of statements. If the regex matches this, then this is the expected output. I'd like to do this a lot more programmatically in the future, but I haven't determined a way yet that I can do that

that cross-references it in a nice way because there's always a bunch of data meddling that I have to do sometimes to map it to the correct response. So when the console gets back an ARN and the CloudFormation wants a name, then I have to do that conversion. One thing that wasn't particularly surprising to me, though I suppose it may be to folks who are not as, I guess, overexposed to Amazonian culture. When I saw this and talked about it with a couple

of my friends who work at AWS, you might expect their response to be angry or bitter or start pointing out problems. But the response across the board that I heard was, isn't it amazing? People are thrilled to see something like this come out. And it stings a little bit just from a usability perspective. It seems, and this is unfair to the excellent work that you've done, but it seems like it's not that hard to have developed over the past, what is it now, 12, 13, 14 years that AWS has been a going concern. Yeah. And the advocacy from AWS has been incredible. There's nothing but support from the PM, and the advocacy from AWS has been incredible. There's nothing but support from the PMs and the solution architects that reached out and said, this is amazing. Yeah, it really is an amazing accomplishment.

I don't want to even slightly come across as downplaying what you've built. This is one of those tools that I will be shouting from the rooftops anytime it's even slightly relevant, which is a disturbing amount of the time. Yeah, and we are seeing a bit of a consolidation in the console just recently. A lot of the console experience is being a lot more shared, a lot more common UI. So I think RDS and Lambda were one of the first to implement this sort of new UI response

and any new services get this sort of new UI look, I guess you could call it. And it does actually map to the network calls. Network calls are a lot easier. They're usually just JSON formatable. So any new services, they're pretty easy to map out of the box. I can usually get things like transit gateway, day one support, pretty easy to put in. But it's a lot more of these older services that are in this legacy mode where it's a lot more chaotic. To that end, do you foresee a future in which there are changes to the console that wind up breaking things? For example, they release a new service or they refactor the UI for an existing service, and that winds up causing

problems for the extension. Yeah, I've seen it already. Recently, the ECR console got a new overhaul. And what came with that was a new network mapping throughout the whole experience. So I had to redo all that mapping by itself. So, you know, things will break in this when AWS iterates on their changes, but the idea is that I'll keep up with them. Are your plans just to keep this as an open source project that you work on as a hobby? Are you looking at potentially turning this into a business to some extent? Are you using this as a portfolio project? If you don't mind the question, what are your plans for this other

than look at this neat thing that I built? Yeah, it's mostly just a portfolio project. I plan to maintain it as much as I can. I've got a lot of other development tools, which I'm trying to give just as much love to. But yeah, I don't plan to commercialize it. I just want it to be a good tool for the community. And it certainly is at that. For people who are listening and having the same reaction that I am right now, oh my word, what else have you built? Is there anything relevant to AWS or cloud that you've built in the context of a developer tool? Sure. So before I started this project,

I worked on a Visual Studio Code extension, which synchronizes with Cloud9 environment. So live synchronization of both files, but also cursor positions. So you can do like a Google Docs experience from both the Cloud9 experience and your Visual Studio Code IDE. So that was a fun project.

And I got a lot of feedback from there as well. I must have missed that one entirely. But that sounds almost as, I guess, earth-shaking as this one. I'm wondering if AWS is going to acquire you personally at this point where it's just, yeah, we've decided to acquire you. And your response is along the lines of, I'm a person. You don't get to acquire people like that. And their response is, watch us. So this is one of those things where it seems like a single person has been going after some of the more annoying aspects of working with a cloud provider and just picking them off one by one. Yeah, I'm always of the opinion that if you don't

like it, just fix it yourself, whatever that may cost. What, I guess, words of wisdom, for lack of a better term, would you have for someone who sees an annoying problem like this, looks at it and says, oh, well, if AWS hasn't done it, or someone else hasn't done it, it must be too hard and not worth tackling? Oh, just for sure, make a POC. Do a proof of concept, figure out whether it's viable. There's a lot of times where it just won't be, and then you can move on. But if it's a viable concept, you can open source things, you can commercialize it, just do it. And it doesn't have to be this huge, styled, clean, tidy project that you do. It can just be this little demo. And a lot of the times, the community will tell you whether it's a good idea or a bad idea.

Did you communicate with folks in the community as you were building this out conceptually by hand? I mean, looking at the history of this on GitHub, it very much looks like it was developed completely in the open, but it doesn't appear that there have been other people submitting code to it. Yeah, I haven't got any other contributors to this yet. There's been a couple of issues raised, but no other people have really contributed to it and pull requests are welcome. But I haven't really exposed this. I've just been sort of working towards my coverage numbers actually to make sure that it's a viable enough project before I really tell people about it. But yeah, a couple of the engineers in Edwards seem to have picked it up. I know it's making the rounds at the moment. So yeah, I guess it's out in the open now.

It absolutely is something of beauty to look at. And just it's neat. If for no other reason, if you never touch the AWS console or even AWS, I still think there's value to this in that it shows that a single developer who has an idea can release something that one of the largest companies in the world has not been able to get out the door. There's a lesson somewhere in there. I'm

not quite sure what it is yet, but it's absolutely inspirational from where I sit. Yeah, it's great. It's one of those things where everyone's just like, oh, why isn't this embedded? This is great. And it's just really good to get that sort of community feedback. Well, it's something that I think has been a very painful spot for far too long. If people are interested in other things you have going on or the words you have to say, where can they find you on the internet? Yeah, so I'm primarily on Twitter and GitHub. And your usernames are?

You better post the link. Fair enough. They will be in the show notes. Thank you so much for taking the time to speak with me today. I appreciate it. Cool. Thanks for having me on. Ian McKay, currently the sole developer behind Console Recorder. I'm Corey Quinn, and this is Screaming in the Cloud. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold.