Screaming in the Cloud - The Hari Seldon of Third Party Tooling with Aidan Steele

Episode Date: March 16, 2022

About AidanAidan is an AWS enthusiast, due in no small part to sharing initials with the cloud. He's been writing software for over 20 years and getting paid to do it for the last 10. He's still not sure what he wants to be when he grows up.Links:Stedi: https://www.stedi.com/GitHub: https://github.com/aidansteeleBlog posts: https://awsteele.com/Ipv6-ghost-ship: https://github.com/aidansteele/ipv6-ghost-shipTwitter: https://twitter.com/__steele

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. Couchbase Cappella. Database as a service is flexible, full-featured, and fully managed,

with built-in access via key-value, SQL, and full-text search. Flexible JSON documents align to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com slash screaming in the cloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella, make your data sing.

Today's episode is brought to you in part by our friends at Minio, the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. Getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere.

And that's exactly what Minio offers. With superb read speeds in excess of 360 gigs and a 100 megabyte binary that doesn't eat all the data you've got on the system, it's exactly what you've been looking for. Check it out today at min.io slash download and see for yourself. That's min.io slash download and be sure to tell them that I sent you. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by someone who is honestly feels like they're after my own heart. Aiden Steele by day is a serverless engineer at Steady but by night he is an absolute treasure and a delight because not only does he write awesome third-party tooling and blog posts and whatnot around the AWS ecosystem, but he turns

them into the most glorious, intricate, and technical shitposts that I think I've ever seen. Aidan, thank you for joining me. Hi, Corey. Thanks for having me. It's an honor to be here. Hopefully, you get to talk some AWS and maybe also talk some nonsense as well. I would argue that in many ways, those things are one in the same. And one of the things I've always appreciated about how you approach things is you definitely seem to share that particular ethos with me. And there's been a lot of interesting content coming out from you in recent days. The thing that really wound up showing up on my radar in a big way was back at the start of January 2022, for those listening to this in the glorious future,

about using IPv6 to use multi-factor auth, which it is so... I don't even have the adjectives to throw at this, because it is is first, it is ridiculous. Two, it is effective. And three, it is just, who thinks like that? What is this? And what monstrosity have you built? So what did I end up calling it? I think it was IPv6 Ghost Ship.

And I think I called it that because I've recently watched. What's that series? I was recently on Apple TV. The Isaac Asimov. If it's not Paw Patrol, I have no idea what it is because I have a four year old who is very insistent about these things. It is not so much a TV show as it is a way of life. My life is terrible. Please put me out of my misery.

At least it's not bluey. That's the one I usually hear about. That's Australia's greatest export. But it was one of the plot devices was a ship that would teleport around the place, and you could never predict where it was next. And so no one could access it. And I thought, oh, what about if I use the IPv6? Oh, Foundation. That's the one.

Foundation. That's how the name came about. The idea, honestly, it was because I saw, when was it? Sometime last year, AWS added support for those IP address prefixes. IPv4 prefixes were small, very useful and important. But IPv6 with, what was it, 2 trillion IP addresses, per instance, I thought there's got to be fun to be had there. 281 trillion, I believe.

281 trillion. It is sarcastically large space. And that also has effectively, I would say in an InfoSec sense, killed port scanning. The idea, I'm going to scan the IP range and see what's there just because that takes such a tremendous amount of time. Now, here in reality, you also wind up with people using compromised resources. And yeah, it turns out I can absolutely scan trillions upon trillions of IP addresses as long as I'm using your AWS account and associated credit card in which to do it.

But here in the real world, it is not an easily discoverable problem space. Yeah, I made it as a novelty, really. I was looking for a reason to learn more about IPv6 and subnetting because it's a term I'd heard, a thing I didn't really understand. And the way I learned things is by trying to build them, realizing I have no idea what I'm doing, Googling the error messages, reluctantly looking at the documentation and then repeating until I've built something. And then I built it, published it, and it seemed to be pretty popular. It struck a chord. People retweeted it. It tickled your fancy. I think it spoke to something in all of us who try not to take our jobs too seriously.

You know, know we can have a little fun with this ludicrous tech that we get to play with. The idea being you take the multi-factor auth code that your thing generates, and that is the last series of octets for the IP address you wind up going towards. And it is such a large problem space that you're not going to find it in time. So whatever it is,

automatically connect to that particular IP address because that's the only one that's going to be listening for a 30 to 60 second span for the connection to be established. It is a great idea because SSH doesn't support this stuff natively. There's no good two-factor-off approach for this. And I love it.

I'd be scared to death to run this in production for something that actually matters. And we also start caring a lot more about how accurate are the clocks on those instances all of a sudden. But I just love the concept so much because it hits on the ethos of, I think, what so much of the cloud does, where these really are fundamental building blocks that we can use to build incredible, awe-inspiring things that are globe-spanning and also ridiculousness.

And there's so much value in being able to do the same thing sometimes at the same time. Yeah. It's interesting. You mentioned like never using MProd. And I guess when I was building it, I thought, you know, that would be apparent. Like, yes, this is very neat, but surely no one's going to use it. And I did see someone raised an issue on the GitHub project, which was talking about clock skew. And I mentioned- Here at the bank where I'm running this in production, we're having some trouble with the clock. Yeah, it's- You know, I mentioned that the underlying 2FA library did account for clock skew, 30 seconds either way. But it made me realize

I might need to put a disclaimer on the project. While the code is probably reasonably sound, I personally wouldn't run it in production. And it was more meant to be a piece of performance art or something to tickle one's fancy and to move on, not to roll it out. But I don't know. Different strokes for different folks. I have gotten a lot better about calling out my ridiculous shitpost things when I do them. And the thing that really

drove that home for me was talking about using DNS text records to store information about what server a virtual machine lives on or container or whatnot, using Route 53 as a database. And that was a great gag.

And then someone did a Reddit post of, this seems like a really good idea. So I'm going to start doing it. And I'm having these questions. And at that point, it's like, okay, I've got to break character at that point. It's, yeah, hi, that's my joke. Don't do it. Because X, Y, and Z are your failure modes.

There are better tools for it. So yeah, there are ways you can do this with DNS, but it's not generally a great idea. And there are some risk factors to it. Okay, A, B, and C are the things you don't want to do. So let's instead do it in a halfway intelligent way, because it's only funny if everyone's laughing. Otherwise, we fall into this trap of people take you seriously, and then they feel bad as a result when it doesn't work in production. So calling it out as this is a joke tends to put a lot of that aside. And it also keeps people from feeling left out. Yeah. I realized that because the next novelty project I did a few days later, I'm not sure if you caught it was a, it was a Rick roll over ICMP V6 packets, where if you would run ping 6 to a certain IP

range, it would return the lyrics to music's greatest treasure. So I think that was hopefully a bit more self-evident that this should never be taken seriously. But who knows? I'm sure someone will find a use for it in prod. And I was looking through this, like, this is great. I love some of the stuff that you're doing because it's just fantastic. And I started digging a bit more into things you'd done. And at that point, it was, whoa, whoa, whoa, wait a minute. Back in 2020, you found an example of an issue with AWS's security model where CloudTrail would just start, if asked nicely, spewing other people's

credential sets and CloudTrail events and whatnot into your account. And A, that's kind of a problem. B, it was something that didn't make that big of a splash when it came out. I don't even think I linked to it at the time. And C, it was examples of after the recent revelations around cloud formation and glue that the fine folks at Orca Security found out. That wasn't a one-off because you'd done this a year beforehand. We have now an established track record of cross-account data sharing and potentially exploits. And I'm looking at this and I got to level with you.

I felt incredibly naive because I had assumed since we hadn't heard of this stuff in any real big sense that it simply didn't happen. So when we heard about Azure, obviously it's because Azure is complete clown shoes and the excellent people at AWS

would never make these sorts of mistakes, except we now have evidence that they absolutely did and didn't talk about it publicly. And I've got to level with you. I feel more than a little bit foolish, betrayed, naive for all of this. What's your take on it? Yeah.

So just to clarify, it wasn't actually in your account. It was the new AWS custom resource execution model was you would upload a Lambda function that would run in an Amazon managed account. And so that immediately set off my spidey sense because executing code in someone else's account seems fraught with peril. And so... Yeah, you could do all kinds of horrifying things there, like use it to run containers. Yeah. Thankfully, I didn't do anything that egregious. I stayed inside the Lambda function, but I looked, I poked around at what credentials it had and it would use CloudWatch to re-invoke itself and CloudWatch gets recorded in CloudTrail. And I won't go into all the details, but it ended up being that you could see

credentials being recorded in CloudTrail in that account, and I could sort of funnel them out of there. When I found this, I was a little scared, and I don't think I'd reported an issue to AWS before. So I didn't want to go too far and do anything that could be considered malicious. So I didn't actively seek out other people's credentials. Yeah, as a general rule, it's best once you discover things like that to do the right thing and report it and not proceed to, you know, inadvertently commit felonies. Yeah, especially because it was, you know, my first time, I felt better safe than sorry. So I didn't see other credentials, but I had no reason to believe that I wouldn't see it if I kept looking.

I reported it to Amazon. Their security team was incredibly professional, made me feel very comfortable reporting it and let me know when, you know, they'd remediated it, which was a matter of days later. But afterwards, it left me feeling a little surprised because I was able to publish about it and a few people responded, you know, the sorts of people who pay close attention to the industry, but Amazon didn't publish anything as far as I was aware. And it changed the way I felt about AWS security, because like you, I sort of felt that AWS more or less had a pretty perfect track record. They would have advisories about possible Zen exploits and so on,

but they'd never published anything about potential for compromise. And it makes me wonder how many other things might have been reported in the past where either the third party researcher either didn't end up publishing or they published and it just disappeared into the blogosphere and I hadn't seen it. They have a big earn trust principle over there. And I think that they always focus on the trust portion of it. But I think what got overlooked is the earn. When people are giving you trust that you haven't earned on some level, the right thing to do is to call it out and be transparent around these things. Yes, I know Wall Street's going to be annoyed and headlines, et cetera, et cetera. But I had always had the impression that had there been a cross-account vulnerability or a breach of some sort,

they would communicate this and they would have their executives go on a speaking tour about it to explain how defense in depth mitigated some of it and or lessons learned and or what else we can learn. But it turns out that wasn't what was happening at all. And I feel like they have been given trust that was honored, and now I am not happy with it. I suddenly have a lot more of a, I guess, skeptical position toward them as a result. And I have very little tolerance left for what has previously been a staple of the AWS security discussions, which is an executive getting on stage for a while and droning on about the shared responsibility model with the very strong implication that, oh, yeah, we're fine.

It's all on your side of the fence that things are going to break. Yeah, it turns out that's not so true. Just you know about the things on your side of the fence in a way that you don't about the things that are on theirs. Yeah, it's an interesting one. I think about it and I think, well, they never made an explicit promise that they would publish these things. So on one hand, I say to myself, oh, maybe that's on me for making that assumption. But I don't know, I feel like the way we felt was justified, maybe naive in hindsight. But then, you know, I guess I'm still not sure how to feel because I think about recent issues and how a couple of AWS distinguished engineers jumped on Twitter and to their credit were extremely proactive in engaging with the community.

But is that enough? It might be enough for, say, to set my mind at ease or your mind at ease, because we are, to put it mildly, highly engaged. Perhaps a little too engaged in the AWS space. But Twitter's very ephemeral. Very few of AWS's customers. Yeah, I can't link to tweets by distinguished engineers to present to an executive leadership team as an official statement from Amazon.

I just can't. Yeah, yeah. So the lesson we can take from this is, okay, so, well, we never actually said this. So let me get this straight. You're content to basically let people assume whatever they want until they ask you an explicit question around these things. Really? Is that the lesson you want me to take from this? of very explicit questions that I will be asking you going forward if that is in fact your position and you are not going to like the fact that I'm asking these questions. Even if the answer is a hard no, people who do not have this context are going to wonder why are people asking those questions? It's a massive foot gun here for them if that is the position that they intend to have. I want to be clear as well, this is also a messaging problem. It is not in any way a condemnation of their excellent folks

working on the security implementation themselves. This stuff is hard and those people are all stars. I want to be very clear on this. It is purely around the messaging and positioning of the security posture. Yeah, yeah, that's a good clarification because, like you, my understanding that the service teams are doing a really stellar, above-average job industry-wide, and the AWS security response teams, I have absolute faith in them. It is a matter of messaging. And I guess what particularly brings it to front of mind is,

I think, was it earlier this month or maybe it was last month, I received an email from a company called Sourcegraph. They do code search. I'm not even a customer of theirs yet. I'm on their free trial. And I got an email that, I'm paraphrasing here, was something to the effect of, we discovered that it was possible for your code to appear in other customers' code search results. It was discovered by one of our own engineers. We found that the circumstances hadn't cropped up, but we wanted to tell you that it was possible, it didn't happen happen and we're working on making sure it won't happen again and i think about how radically different that is where they

didn't have a third party researcher forcing their hand they could have very easily swept it under the rug but they were so proactive that honestly that's probably what's going to tip me over to the edge into becoming a customer. I mean, other than them having a great product. But yeah, it's a big contrast. It's how I'd like to see other companies work, especially Amazon. This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's

S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense. The two companies that I can think of that have had security problems have been Circle CI and Travis CI. Circle had an incredibly transparent early on blog post. They engaged with customers on the forums and they did super well. Travis basically denied Stonewall for ages. And now the only people who use Travis are there because they haven't found a good way to get off of it yet. It is effectively DOA. And I don't think those two things are unrelated. Yeah, no, that's a great point because, you know, I've been in this industry long enough, you have too, to know that humans write code and humans make mistakes.

I know I've made more than my fair share and I'm not going to write off a company for making a mistake. It's entirely in their response. And yeah, you're right. That's why Circle is still a trustworthy business that should earn people's business and why, Travis, I recommend everyone move away from. Yeah, I like Orca security as a company and as a product, but at the moment, I am not their customer. I am AWS's customer. So why the hell am I hearing it from Orca and not AWS when this happens?

Yeah, yeah, it's not great. On one hand, I'm glad I'm not in charge of finding a solution to this because I don't have the skills or the expertise to manage that communication. Because like I think you said in the past, there's a lot of different audiences that they have to communicate with. They have to communicate with the stock market. They have to communicate to communicate with. They have to communicate with the stock market. They have to communicate with execs. They have to communicate with developers.

And each of those audiences demands a different level of detail, a different focus. And it's tricky. And how do you manage that? But I don't know. I feel like you have an obligation to when people place that level of trust in you. It's just a matter of doing right by your customers on some level. Yeah. How long have you been working on an AWS side environments?

Clearly, this is not like, well, it's year two, because if so, I'm going to feel remarkably behind. So I've been writing code in some capacity or another for 20 years. It took about five years to get anyone to pay me to do so. But yeah, I guess the start of my professional career, and by professional, I want to use it in the strictest term, means getting paid for money, not that I am necessarily a professional, coincided with the launch of AWS. So I don't have any experience with the before times of data centers.

Never had to think about Direct Connect. But it means I have been using AWS since sometime in 2008. I was just looking at my bill earlier. I saw that my first bill was for $70. I was using a C1X large, which was 80 cents an hour, and it had eight whole CPUs. And to put that in context...

Eight vCPUs, technically, I believe. Were they using the eCPU model back then? Yeah, no, that was vCPUs. But to me, that was extraordinary. You know, I was somewhere just after high school it was the netflix prize was around if you're not sure what that was it was netflix had this open competition where they said anyone who could improve upon their movie recommendation algorithm could win a million dollars and also being a teenager i had a massive massive ego and no self-doubt.

So I thought I could win this, but I just don't have enough CPUs or RAM on my laptop. And so when EC2 launched and I could pay 80 cents an hour rather than signing up for a 12-month contract with a co-location company. It was just a dream come true. I was able to run my terrible algorithms, but I could run them eight times faster. Unfortunately, and obviously, I didn't win because it turns out I'm not a world class statistician. But... Common mistake. I make that mistake about myself all the time. Yeah. I mean, you know, I think I was probably 19 at the time. So I had my ego did, uh, make me think I had, I was one, but it turned out not to be so. But, um, I think that was what really blew my mind was that me and nobody could create an account with Amazon and get access to these incredibly powerful machines

for less than a dollar. And so I was hooked since then. I've worked at companies that are AWS customers since then. I've worked at places that have zero EC2 servers, worked at places that have had thousands and places in between. And it's got to a point actually where I guess my career is so entwined with AWS that one, my initials are actually AWS, but also this might sound ridiculous and it's probably just a sign of my privilege that I wouldn't consider working

somewhere that used another cloud. Oh, I think that's absolutely the right approach. I had a Twitter thread on this somewhat recently, and I'm going to turn it into a blog post because I got some pushback. If I were looking at doing something and I was coming to the industry right now, my first choice would be Google Cloud because its developer experience is excellent. But I'm not coming to this without any experience. I have spent a decade or so learning how, not just how AWS works, but also how it breaks. Understanding the failure mode and what that's going to look like and what it's good at and what it's not, that's the valuable stuff for running things in a serious way. Yeah, it's an interesting one.

And I mean, for better or worse, AWS is big. I'm sure you'll know much better than I do the exact numbers. But, you know, if a junior developer came to me and said, which cloud should I learn or should I learn all of them? I mean, you're right. Google Cloud does have a better developer experience, especially for new developers. But when I think about the sheer number of jobs

that are available for developers, I feel like I would be doing them a disservice by not suggesting AWS, at least in Australia. It seems they've got such a huge footprint that you'll always be able to find a job working as an AWS familiar engineer. It seems like that would be less the case with Google Cloud or Azure. Again, I am not sitting here

suggesting that anyone should, oh, cloud, you're insecure. We're going to run our own stuff in our own data centers. That is ridiculous in this era. They're still going to do a better job of security than any of us will individually. Let's be clear here.

And it empowers and unlocks an awful lot of stuff but with their privileged position as these hyperscale providers that are the default choice for building things i think comes with a significant level of responsibility that i am displeased to discover that they've been abdicating and i don't love that yeah it's an interesting one right because yeah like you say, they have access and the expertise that people doing it themselves will never match. So, you know, I'm never going to hesitate to recommend people use AWS

on a campus security because a company's security posture will almost always be better for using AWS and following their guidelines and so on. But yeah, like you say, with great power comes significant responsibility to earn trust and retain that trust by admitting and publicizing when mistakes are made. One last topic I want to get into with you

is one that you and I have talked about very briefly elsewhere, that I feel like you and I are both relatively up to date on AWS intricacies. I think that we are both better than the average bear at working with the platform. But I know that I feel this way, and I suspect you do too, that VPCs have gotten confusing as hell. Is that just me? Am I a secret moron that no one bothered to ever tell me this and I should update my own self-awareness? Yeah, I mean, that's been the story of my career with AWS. When I started, VPCs didn't exist. It was EC2 classic. Well, I guess at the time it was just EC2. And it was simple. You launched instance and you had an IP address. And then along came VPCs. And I think at the time I thought something to the effect of, this seems like needless complexity. I'm not going to bother learning this. It will never be relevant. In the end, that wasn't true. I worked in much larger deployments where VPCs made fantastic

sense and made a lot of things possible, but I still didn't go into the weeds. Since then, AWS has announced that EC2 Classic will be retired. End of an era. I'm not personally still running anything in EC2 Classic, and I think they've done an incredible job of maintaining support for it this long. But VPC complexity has certainly been growing year on year since then. I recently was using the AWS console, like we all do,

and no one ever admits to, to edit a VPC subnet route table and i clicked the drop down box for a target and i was overwhelmed by the number of options there were nat gateways internet gateways carrier gateways i think there's a thing called a wavelength gateway eni and i think i was surprised because I had to scroll through the list. And I thought, wow, that is a lot of different options. Why is that? Especially because it's not so relevant to me.

But I realized a big theme of what AWS has been doing lately is trying to make themselves available to people who haven't used the cloud yet. And they have these complicated networking needs. And it seems like they're trying to reasonably successfully make anything possible. But with that comes additional complexity. I appreciate that the capacity is there, but there has to be an abstraction model for getting rid of some comes additional complexity. I appreciate that the capacity is there, but there has to be an abstraction model for getting rid of some of this complexity.

Because otherwise, the failure mode is you wind up with this amazingly capable thing that can build marvels, but you also need to basically have a PhD in some of these things to wind up tying it all together. And if you bring someone else in to do it, then you have no idea how to run the thing.

You're effectively a golden retriever trying to fly a space shuttle. Yeah, that's interesting. Clearly, they must be acutely aware of this because they have default VPCs. And for many use cases, that's all people should need. But as soon as you want, say, a private subnet, then you need to either modify that default VPC or create a new one. And it's sort of going from zero to 100 complexity extremely quickly because, you know,

you need to create route tables to everyone's favorite NAT gateways. And, yeah, it feels like the on-ramp needs to be not so steep. Not sure what the solution is. Yeah. I hope they find one. As do I. I really want to thank you for taking the time to speak with me about so many of these things. If people want to learn more about what you're up to, where's the best place to find you? Twitter's the best place. On Twitter, my username is underscore underscore steel, which is S-T-E-E-L-E.

From there, that's where I'll either widely speculate on the latest releases or link to some of the silly things I put on GitHub. Sometimes they're not so silly things, but yeah, that's where I can be found. And I'd love to chat to anyone about AWS. It's something I can geek out all day, every day. And we will certainly include links to that in the show notes. Thank you so much for taking the time to speak with me today. I really appreciate

it. Thank you so much for having me. It's been an absolute delight. Aiden Steele, serverless engineer at Steady and shitposter extraordinaire. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an immediate request to correct the record about what I'm not fully understanding about AWS's piss-weak security communications. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less

horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.