Two's Complement - Yak Shaving, Part 2, Also Live!

Episode Date: June 20, 2023

Ben and Matt finish shaving the yak from the prior episode. While waiting for DNS certificate validation to complete, our hosts discuss the "branch based environment" approach to infrastructure, and consider how serverless services make that model a bit cheaper.

I'm Matt Godbolt. And I'm Ben Rady. And this is Tooth Compliment, a programming podcast. Hey, Ben. Hey, Matt. So, we were looking at the problem with our SSL certificate for twoscompliment.org. In other words, we wanted to be able to host from just twoscompliment.org, not www.twoscompliment.org, which sounds straightforward.

And through the miracle of podcasting, we recorded that many weeks ago, our sort of attempts to fix it, but never quite got there and so i figure we should probably finish the job try and get it so that our website's actually working and um everyone can laugh at how much we've forgotten between what may be back-to-back episodes as far as our listener is concerned but what for us a month has passed the fact that you think we're going to get this fixed today is very ambitious, and I like that attitude. Well, we've got half an hour or so. Let's give it a go. Let's see how far we can get, at least.

So we had a whole bunch of Terraform-y stuff was how we left it. Right now, if I remember this correctly, our plan was to create an AWS Route 53 domain. Yep. And then change the domain to use the wildcard certificates? I think so. Yeah, we could use a wildcard cert, or at least a cert that has multiple hosts listed, one of which could be a wildcard. But it could just have www and the no domain, which I believe is what Compiler Explorer does, but I can't remember. I think actually Compiler Explorer has like empty and star dot godbot org or whatever so i have right now okay so looking at

this so we had kind of terraformed some of this up before um and right now there is a certificate that we have for www.toothcomplement.org and there's a little to-do here that should be just toothcomplement.org and then I have another little to-do here that says subject alternative

names equals and then square brackets quote star.toothcompliment.org right and I have a little bit of for each magic in the Route 53 record that I think attempts to

create a record for each of the things that it sees in the certificate and I have this i think because i have done this once before i say this to work and i copy pasted some of that in here but now i have zero memory of how it will all fit together got here let alone how it works so this is what happens when you put things down for a month and then don't pick them up again right let me go check I have a project on my

github that I think this maybe came from so let me go see if that is even remotely true and if so we can crimp from that yeah and I guess I should try tasty penny.com that does work and it is secure.

So I have done this on that site. Yeah, my recipe tracking website called Tasty Penny, I have done this. Tasty Penny? It's Tasty Penny. Yeah. I don't even know. Where is that thing? Yeah, you know, it's like all recipe websites are terrible. Because they're not really recipe websites.

They're advertising websites that are trying to put as many adverts between the obvious thing you want, which is the damn recipe. Right. And try to, you know, get enough SEO from the text that they're putting on there about like, oh, I traveled to Paris three years ago and I had this wonderful, you know, whatever. But yeah, I can't even find, am I like not logged in? Oh, I'm logged in as, yes, okay, that's what's going on there. I'm not. Oh, you're on AWS right now. I can't see my private repositories.

Oh, no, I see you're logged to GitHub. I'm trying to find where this stuff came from, so I'm going github but i'm logged in as a different user so i can't see my whoa you have more than one user i do i have an aquatic user for my official aquatic things and then i see me which is less official you're less official you're not the official ben reidy i am not the official ben reidy i am the i'm just the casual Ben ready. I see. Um, okay. So yes, tasty penny. Uh, here's some terraform. It says site.

And yes, I think that is exactly where that came from because that looks very similar, except some commented out stuff. Uh, before we get too excited, if I go to tasty penny or if i could type tasty penny dot org com com com no dub dub dub it is working no ads no junk just tasty apparently is the uh there's the little uh byline for this and right connection is secure it says and connection is secure, it says, and certificate is valid. And the common name is TastyPenny.com, and there you go. And looking at the – yeah, you've just got a certificate, and it only mentions TastyPenny.com in this certificate.

Now I'm going to go to www.TastyPenny.com. And connection is secure, cookies, all the things. Maybe it redirected me then it might have i see but obviously in the interim it it was it served up something which did not upset my browser yeah in terms of security if you wanted to know for sure i certainly could but yeah that's awesome all right so this is going to be a good thing to crib from because it works that's what i'm checking is like before we get all excited and changing it let's just see that we're heading the right way uh-huh and i will curl it yeah so yeah when i do a curl dash v i see server certificate subject cn tasty penny start date expire date which is uh in may okay

well keep that in mind uh and then uh picking up rocks subject alt name host www.tastypenny.com search star tastypenny.com perfect okay so that's a good sure amazon analog then of that and we know what to look for when we when we do it for two's compliment yeah so you know the question with this is how do you fix the airplane while it's in the middle of the air and uh i would hate to uh you know apply some terraform change here that all of a sudden makes our podcast disappear for you know 24 hours or however long the dns is poisoned or whatever it might be you know so how about this can we make a change straight away to change the ttl of the dns down to like two days and just apply exactly as is but with a really low ttl which means that already or

like two hours which means that we're starting to promote the idea that we're going to screw this up and we want to be able to undo it, right? Exactly, yes. A good friend of mine once told me that if you can't test it properly, then at least make it cheap to roll back. Yeah, okay. Sounds like that guy was making shit up as he went along. Okay. Let's see here. So right now the Route 53 record is set to 60, which is in seconds, right?

Oh, I think so. Right? So we might be, I mean, DNS has its own mysteries I don't think that currently AWS is the name server for ToastComplement Got it, so that means that we can make all the changes

we like here and just use NSLookup with the server being or host or dig or whatever the cool kids use these days and test that it's doing the right thing How would we confirm that that is true i'm going to do ns lookup and i'm going to set type equals any and i'm going to do twos and sorry for my offensively loud keyboard twos complement.org and it tells me non-authoritative

answer name server is ns2.hover.com. Yeah, that's... Blah, blah, blah, blah. And then address is 216.40.34.41, whatever. Yeah. Okay. Beautiful. Now, if I were to set the server to be... Do you happen to have an AWS DNS IP there?

Uh, let me go see if I can do that. One moment, please. And meanwhile, I'm looking at the Tasty Penny website going, this looks great. I want to... I have some updates I want to make to it. I want to make it a little bit more tablet friendly, because it's not

right now, but it, you know, it has some good recipes on it. Yeah, I want to sign into the console i don't know if you got this thing recently where i finally had to separate my amazon.com you know oh shopping shopping password and my aws password yeah a reminder of like it used to be that amazon was a bookstore yeah that's right yeah it is pretty bonkers at one stage actually i had a problem where um i enabled two-factor authentication on one or other of the two and it affected the other even though they were supposedly independent i think there's still some link between the two different account names now and that was the one and only time i ever

spoke to an amazon person on the phone while they were trying to reset it oh this is an interesting problem yes yes yes so i have the twos complement name servers yep there's four of them anyone will do let's go ns-68.awsdns-08.com dot a w s d n s dash zero eight dot com wow that's a beautiful thing oh the so it has an ipv6 address there you are that's crazy okay so now i've just said server that and i'm typing to his compliment mint.org again and now it tells me uh the name servers are now i can see the other data dns uh servers at amazon which is like n50 and 11600, all this kind of crazy things. And apparently it has an address of 202.251.192.68 is what it's resolving to, which is that the alias to the load balancer? Yeah, it's a CloudFront distribution.

Okay, so what I'm going to do is I'm going to look up. Yeah, it's a CloudFront distribution. Yeah, okay. So what I'm going to do is I'm going to look up. No, yeah, it doesn't. Oh, it's refusing me. I'm going to do this on another one. Those 205, 251, 122, 68. Oh, that says NS68AWSD. I mean, it could be the same IP addresses for all I know.

So, yeah, we need to look at what the CloudFront distribution is set to just to see it. I mean, obviously, this is just us testing the water here. This should all work out. But while you do that, what this means is CloudFront is essentially a caching proxy in front of all of the AWS infrastructure. And when one creates one, one tells it where to get the information from that it's going to be proxying and edge caching. And it gives you sort of a unique distribution name. And then that maps to an IP address or a DNS that you then use to point your web services at and then whatever you land on knows how to serve up from cloud front your web server your web traffic

your web whatever so i have a r and i have a distribution name but i'm having a hard time finding what is the distribution address oh wait no maybe this is i think it is and i i think it might actually be a full fqd the distribution name is a okay yeah i think i do yeah okay uh this is this is gonna be a little painful okay this all right well maybe yeah go on is it as something or other it's no it's abiglongstringofcharacters.cloudfront.net. Okay. So if you've got that in your console, why don't you copy that and just do host space that

and see if you get an IP address that looks like this one, and then that would sort of confirm. Why don't you do that? I'm going to copy www.tooscompliment.org. I see 54.230.18.99, 54.230.18.49, 54.230.18.82, 54.230.18.69. Got it. None of those match because I've just realized that NSLookup was giving me a bad answer. It tried to connect and it got timeout. And then what it's doing is it's just telling me all about the domain there is no there's no a record

associated with twos compliment.org so that's what we need to fix okay there's also no a there's no a record associated with www.twoscompliment.org right because isn't there not going to be isn't it going to be this like different kind oh what are the what is the name of that type of dns record that's like well it's not specific to amazon but it's like well it's called sort of alias around so like the the underlying problem here is that there is no such thing really as much as people would love there to be there is no such thing in dns as an uh a c name which is what we really want for the um uh the the naked domain like so what you might want is dub dub dub dot whatever to be the address of

a machine and then if someone puts in the thing without the dub dub dub you say hey this is the same as dub dub dub dot which is a c name a kind of a symbolic link, if you like, at the DNS level. But unfortunately, you can't have a CNAME record for a naked domain itself. You have to have an A record. And the problem with that is that the CNAME actually needs to point at the CloudFront distribution because Amazon want to move it around. They want to change it. Yeah, yeah. And so what typically happens is that DNS providers will have a product

where they track the DNS entry for the CloudFront end that you've got. And they'll just keep periodically changing your A record. But Amazon natively supports this. So we should just be able to configure it. So I think we're just missing the configuration in the Terraform and an application should just make this work here without affecting the real Tooth Complimental because it's still being served up by Hover.com. Right, right, right. So I'm actually looking at this now, and this makes sense to me, which is I've actually got some commented out stuff in this Terraform that does, I think, exactly that. And that is what my Tasty Penny terraform does.

And looking at my Tasty Penny configuration in Route 53, I can see an A record there that is a very strange-looking A record because the value of it is that big long list of characters.cloudfront.net, right? Well, not the same one. Oh, that's interesting. Yeah. And that is for the www one and for the sort of bare domain yeah that sounds so that to me seems like amazon you know doing an a record

you know trick behind the scenes let me i'm gonna have a very quick look at how i did this for some other website that I'm involved in. Route53.tf. Where the hell are all this stuff? Oh, yeah, I actually have modules for this because it's so awful that I have so many stupid things. Main.tf. Okay.

I set a CNAME, and the records are the, yeah, something like root53's address, a.fqdn. So it's kind of looking up somewhere else. This obviously makes for great radio. Zona, the alias, name. Yeah, okay. It looks like it's an alias that i'm setting so i do for both the the a record and the aaa record i have um an alias stanza inside of the terraform itself so it's not an address record even though it could be and it has a name a zone id and some other bits

and pieces in it and i don't know if that corresponds to the thing that you're looking at now yeah i think that is i think we're looking at the same thing here so i've got yeah alias name equals and then i've got a variable which holds the cloud front distribution dot domain name and then another thing that's cloud front distribution dot hosted zone id and that essentially configures the a and the aaa for the top level name which is in my case you know godbolt.org or godbo.lt or compiler explorer.com for all the times it's instantiated which is like the foreach that you've got but i think we only need one of these so you could probably just write it out longhand right now yeah well i i think this would actually just work if

the certificate was the if i switched so kind of parsing through this now and having some vague memory of what we did here um i think the this will all work if we can just just change this certificate to be a wildcard certificate so if i were to change that in the Terraform and then try to run it, would it just replace the existing certificate with a wildcard certificate? I think so. I think so. I have some magic to do that too, if needs be. So why don't we try that?

Let's give that a try. What could go wrong? Right. Well, in theory, people could start getting certificate errors going to choose compliment. Because I do think that this is the real certificate. This one will be the real certificate. Yes. The DNS can do whatever it likes, but we're about to tell CloudFront to use a different

certificate when it's pretending to be us. Yes. Which is probably why I stopped here. I mean, YOLO. Let's do it. Did you make a new certificate actually you already made a certificate well i was gonna i mean can okay wait a second stop if i change this terraform it's not going to make a new certificate i have to go and do it manually i don't remember if you i mean you can absolutely have certificates created in terraform too i don't know if we did did we do that last time okay well let's do this let's start by making

the change in the terraform and doing a terraform plan and seeing what what the heck it thinks yes always always a good start where are we now is what am i is what I have on my computer an accurate representation of what the cloud provider thinks I've got? Right, right. Well, I mean, so I did this once and it said it was up to date, but I'm going to change it. And now we're going to do a plan again. And then we're going to see what Terraform says about what it feels like it wants to change. And I'm going to make this look very much like the existing one that I have for my recipe project. Right.

Okay. And I found the certificate stanza that I have for my site so we can steal from it if need be. Okay. So I'm going to do terraform plan. Yep. What does it say? It says three to add, one to change, two to destroy. And so it is going to, yeah, say AWS ACM certificate,

two's compliment.org must be replaced. Okay. Because you've changed the subject alternative names in there, right? I changed the subject alternative names in there, right? I changed the subject alternative names, and I also changed the domain name from www.toostcomplement.org to toostcomplement.org. Perfect. Perfect. And then it says AWS CloudFront distribution S3 distribution will be updated in place. Yep. And then it says, AWS Route 53,

twoscomplement.org, bracket, star.twoscomplement.org will be created. Yep. And then another AWS Route 53 record for twoscomplement.org

will be created. That sounds good to me. Let's do it. What could go wrong? Well, let's get a list. Many things. All right. All right. All right.

Do I have an applied? Oh, I do have a terraform applied. All right. Firing the rockets. Firing the rockets. If we wanted to troll our audience, we should cut off the audio in the middle of the supply. It's like, wait a second. How does that?

That doesn't... Yeah. Doesn't work like that. Well, ironically, your internet just ditched then, for me. So I'm like, oh, he's joking about the connection going down, and then I'm like, you froze on my screen.

Oh, man. Which was epic trolling in its own right. Alright, it says, destroying, still destroying, about the connection going down and then i'm like you froze on my screen oh man which was epic trolling in its own right all right says destroying still destroying still destroying still destroying it's destroying everything oh and we got an error all right is there some create before destroy thing that i've got no this is a access denied not authorized to perform ACM request certificate. Oh, no. This is where we go into the IAM console and we give this service user that we're running and adds a whole bunch of permissions that it shouldn't have. You are far too clever. I just have to run it myself. I don't actually manage the...

No, I don't. I don't manage the IAM in here. Okay. So where is this user? Tasty penny. And yeah, we're gonna attach a permission. This is gonna be... What is the name of this service certificate something AWS

certificate I think it's this one I don't even know who knows AWS certificate manager private yeah you're this is outside of my purview of understanding we'll's not even the right word we'll try this one yeah and see what happens and if this doesn't work then we'll remove that we'll

take that out you know if it ain't fixed don't break it if it yeah if it don't don't leave it broken more broken yeah that's like yeah the programming by coincidence thing, I think. Have we talked about that before? Maybe not. Yeah. Maybe not. Maybe we have not. So that didn't do it, so I'm removing the policy because that did not fix the problem, so I don't want to create a whole other problem by putting something in there

that wasn't in there before. But I... AWS certificate... So this guy should have this already. ACM. The Association of Computer Machinists. Yeah. No, not that.

What? What? Oh, I guess I can go and look at this, actually, and see. Yeah. This user, I thought... Are you the right user, though? Oh, it's this user i think i thought are you the right user though oh it's a different user i'm an idiot i'm looking at the tasty penny user which clearly does which already works like this i did this already i guess i should have thought of that before it's like you have a user that does this go look at what they do i thought i'm a doofus. I think I was maybe thrown off by the fact that

the user that I have for this has the original podcast name. Can we talk about the original podcast name? Oh my golly, this is all these things. I should be taking notes. Alright. Certificate. Give me all your certificates. uh all right all right certificate give me

all your certificates are belong to us when i've just gotten a certificate manager i know if this is this is certainly completely off topic but i've just been given the okay to push an update to compiler explorer which i will do in the background of this so the continued tapping noises will be me pushing a kind of cool thing to compiler explorer okay we're creating we're creating a certificate all right oh uh so that's a good sign i'm pushing compiler explorer six 725 to production from the staging environment unrelated to this podcast but you know we're all tapping we're both tapping away at keyboards if we've got to fill the air with talking or some description so yeah so it's interesting to talk about how we would do this if this were not just

our hobby podcast yes right so because right now we are literally testing this in production right um which i've heard is a bad idea. We've all seen the meme, the most interesting man in the world meme with him with his little bear going, I don't often do testing, but when I do, I do it in production. I do it in production. And that's not our MO in our day job. So if anyone's thinking that this is the kind of cowboy activity that we would do if it was anything other than you and me chatting, no. So how would we, Ben? How would we do this?

How would we do this if it was – well, so obviously you want to have a separate environment for testing this out. But the trick with creating that separate environment is how do you know that your separate environment is a copy of the state of the environment that you want to change for real? Right? is a copy of the state of the environment that you want to change for real. Right. Right? Which has the additional problem of it is it's going to take you some time to make these changes. And in a large enough organization or in a large enough project, that means that the environment, the production environment may change while you are working on making the changes. Right? So you might be able to make a copy of your production environment as it stands right now. may change while you are working on making the changes, right?

So you might be able to make a copy of your production environment as it stands right now and then make some changes to it, test those changes out. And while you're doing that work, someone else might be doing the same thing and making other infrastructure changes to the main environment. Yeah. So when you finish that, you need a mechanism for basically reapplying the changes that you made on top. It's almost like a fast forward in Git, right? Yeah, yeah, yeah.

You need to reapply the changes that you made on top of the environment as it exists now, not as it existed when you started working on the new thing that you wanted to add, right? Yeah, yeah, yeah. when you started working on the new thing that you wanted to add, right? So I feel like the only way to even have a hope of being able to do this is to just automate everything, infrastructure as code style with Terraform. Like I feel like- And have the only thing that pushes any of this stuff

to be the main branch of your GitHub repo so that you've kind of post hoc already merged everything in at the point of where things are applied. You kind of get a merge commit queue at that point, right? The only thing that's really making changes to your production deployment is the head of the line where all of the intermediate branches have to definitionally have been merged in. Otherwise it goes, oh, I'm rejecting you because you're not at the latest. I have to get it again or whatever. That kind of feel or are you, yeah. No, yeah yeah i think it i think

it is that and and then being able to sort of rebuild your test environments based on changes that are have been actually deployed so being able to either tear them down and build them again and then reapply the new things that you did or merge a change in in a way that's realistic like like you know it's it's probably like the order of operations uh potentially can result in this in the same environment where it's like i had some environment and then i applied someone else's change and then i applied my own change uh that is probably that is representative of what is going to happen in the main environment when you merge your change, flipping them might not. Yeah, yeah. If you apply yours first and then theirs, you might get the same thing.

Hopefully, if Terraform works the way that it says on the tin, but you might not. So you have to think about how that's all going to get applied. So speaking of Terraform that doesn't work uh it said error updating cloud front distribution right the specified ssl certificate doesn't exist isn't in the u.s east one region isn't valid or doesn't include a valid certificate chain okay well so i don't know if we destroyed our other certificate and made a new one or what just i think you do up arrow return and see what it does the second time. Because some of these things have disgusting, like, oh, it takes a while at the back end of...

Which is not ideal. Especially a certificate, right? Yeah. All right. I'm going to go look at the CloudFront distribution. Yeah, that's a good idea. And see what state it's in right now. It says it's enabled.

Can you curl the site real fast and just see if it returns anything i can certainly cut it if it gives you some sort of weird certificate error um uh oh hang on a second dub dub dub doc two's coming on dog importantly because that's exactly what we're trying to fix right you see this is why it's a problem for me this is why we have to fix it because i i'm too lazy to type dub dub dub and or even say it this is why it's a problem for me. This is why we have to fix it, because I'm too lazy to type or even say it properly. Yeah, no, it's working fine still. Okay.

Whatever it is. I probably created the new certificate and was trying to flip the cloud front to it, and it went over, and it was like, no. I mean, you've got the console open, too, so you can actually have a look in the ACM certificate thingamajig and see if it's there or not. Oh, yeah, good call. I know. So we've deliberately not shared screens so that I have to

ask Ben what he's seeing so that you dear listener can actually sort of hopefully follow along. I don't know how much anyone will be able to follow on what we're doing here. Yeah, I see. Okay. So, yes. So I see four certificates

in here. Two of them are two's complement ones. One is the www one that is issued and in use and eligible for renewal. And another one is without the www, its status is pending validation. Ah, so. So we may have to wait. It's status is pending validation. Ah. So. So we may have to wait. There's usually a DNS validation.

That's how these things. Did you have, what type of validation did you have? I mean, it could be that you might have an email right now because it's like, hey, are you really sure this is your certificate? Oh, interesting. Mine set up for DNS, which I think, because 50 whatever monkey uh is in cahoots with itself it can basically set its own dns records and then requit oh but there's the problem now we've got two now we have two problems we can't we won't be able to use dns validation because you honestly haven't flipped the flag yet for the real DNS provider to be Amazon. Yeah, I could copy those things over into the other one.

You certainly could. If it tells you what the challenge is that it's put in the DNS, then you can put them in. I mean, I could go, if it added it automatically, I could go look. That's true. I could go look in round 53 and be like, what did you add to this thing? Yeah, that's true. And just copy those over uh but yes i agree with your assessment of the situation here yeah which may have been

this rings a bell from the last time we did this and like hey yeah this thing might take a while which won't be good won't be good radio of like yeah we just have to wait two days yeah yeah yeah um so i don't see any new records maybe it's not set up to do it that way so i mean if you look at the acm it's the certificate does it say why or how to author to do that thing i'm a jake um it says pending validation status group additional names and you don't have an email or something I can't remember how this works if it's not set that way I see just checking my email because some of those addresses you put a little forward on I don't know that it's yeah all right spam folder real fast.

Yeah, I can't help it. I'm going to go and find my certificates. No. Certificate manage. Oh, of course, I need to log back in again. How do I do this for? For your magic penny. Yeah.

I'm just looking at mine, and I can see, yeah, in use, yes. Renewal, eligibility. Right. Okay. So I can see, oh, no, that's, yeah, I can see that if I've gone to one of my certificates and I can see that it has in the sort of more information inside the console itself under the ACM, it's got a list of domains and it tells me status and renewal it's this type and then it's got c name name and c name value those are the two things that need to be put into the root 50 thing oh and there's even a button that says create records in root 53 but

you can click oh well obviously you don't want to do that because we don't necessarily yeah yeah right okay let's i'm gonna go into my other registrar doodad. Right. And I'm going to go to twoscompliment.org. Why is there... Oh, I've got two domain names for twoscompliment.org. One of them is the misspelling. Be very careful.

Very careful. Yes, but I've got to make sure... I mean i mean it's not gonna hurt anything if i do the wrong one but like what hurt fewer things yeah so dns and then we're gonna add a record and it's gonna be a c name which you can copy paste from thankfully from the other thing yeah yeah and then that's how honestly how much of software engineering or administration goes through the clipboard? I mean, it's just... Oh, my God. So much.

So very much. I'm going to set the TTL to five minutes. Wonderful. And I add this record. And then I'm going to do the same thing again for the wildcard. Yep. Yeah, you've got the two I can see for each of my domains I've got

two thingies. And then of course we have to hope that it notices this within. Yeah. Alright, so yeah, I've got two of them set here. And it's probably a good sign that

I actually had another one for the www certificate that is in here i can see right okay so now there's actually three right right but these are all like interim things so like just to sort of recap in case that we're we are trying to prove to amazon that we own that domain name and one of the many ways that we can prove that is to make a change to the dns records with some magical things that they've. And one of the many ways that we can prove that is to make a change to the DNS records with some magical things that they've given us. These are the CNAME records that we've just been talking about.

This allows Amazon to say, we believe you own that domain and therefore we will issue you a certificate for that domain that says you own the domain and we signed a traffic and all that kind of stuff. Now, this is an interim step because eventually Amazon themselves will be the people that are serving up the domain name. And therefore they just know we own it because we transferred it to them in some capacity.

But we're not there right now. So as an intermediate step where we want to just be able to test it by getting ourselves a new certificate, we are going to issue the certificate, use your existing DNS to prove that we own it, and then apply that certificate. And then finally, we can move things over if we're happy the certificate looks good. Okay. Exactly.

Is there a way to poke the AWS certificate manager and say, hey, can you- Can you take another look now? One ping only, please. Yeah. One ping only. That's such a good uh that's amazing i don't know if there is maybe maybe i can do this here i can delete it i don't think i want to do that uh request yeah is that maybe gonna make a new one i can say that's probably well this is where we nhx free events yeah this might be

we just wait you know 10 minutes for us well compiler explorer is 67 through doing an update very excitingly in another window all right behind you so that's uh man so then we were talking right two things we talked about one One, obviously, we sort of briefly mentioned was the idea that in our day job, the way that we do this is that the CI build in main applies the production configuration. And so it's been through all the testing, and there's not like the two people fighting over two independent things changes along the way, because you always are seeing the union of whatever has been merged into the trunk. Correct. But then how do you test it how do you test a separate like thing how would we i mean so in compiler explorer i have some very hard-coded staging and beta or beta just to de-confuse people honestly i've had this conversation so many times with americans they're like what beat and they're like thinking like egg betas or like pub boxers. No, not that kind of beta.

No. Beta. And so those are very special case for me. And they share kind of a lot of infrastructure as well, because at one stage I was trying to save money. Nowadays, I think actually probably this is a false economy. But there are better ways of doing it, or at least there are different ways of doing it,

rather than having just some very special hard-coded things that we sort of push things through. And I know you've been involved in a lot of those recently, so do you want to talk a little bit about some ideas that you've had about how it should be done? Yeah. I don't remember if we've talked about this on the podcast or not, but we lately have been doing a thing with a data warehouse project that I'm working on where the branch in GitHub represents an environment.

So we don't have a production environment. We have a main environment because we have a main branch. Because the main branch is that. Right. It's not special case in any way. It's not special case. There's like a couple of additional protections for deleting things. Got it.

And that's it's it other than that it's identical to every other branch and identical to every other environment and so when you create a new branch it you know says oh this environment doesn't exist I guess I need to apply this terraform I apply the terraform every time so the terraform just has

more work to do this time and it you know spins up all of the infrastructure I apply the Terraform every time, so the Terraform just has more work to do this time. And it, you know, spins up all of the infrastructure that this project requires. And it's doing that, obviously, from a fork of the Terraform file that was just in the main branch. Got it. And is therefore a copy of the infrastructure that is running in the main environment. So you wind up making an exact copy of whatever the environment was at that time, right? Right. And so that will all get created. It will then automatically deploy to that environment.

And now you have a completely separate running copy of that system. There's a different URL that you can go to that's got your branch name in it. And you can play around with it. You can test things out. And then as you push changes to that branch, it goes through the exact same process. It applies any Terraform changes if you have them. It deploys the new version of the software that you built. And then you can sort of iterate and continue on working in that.

And then when you have something that you're confident is correct, you know, all the tests are passing and maybe you've done some exploratory testing. I think this is especially important with the sort of cloud-based services that you use on some of these projects, because it's very difficult to test them, obviously, like from your, you know, your workstation your laptop so the only real way that you have to test them uh in any sort of exploratory sense is um by using them for real exactly as we've been doing right now i said that because we don't have this setup we are experimenting directly in prod right exactly right exactly right and so once you're confident that your changes work and that all your software works with

any other infrastructure changes that you have made, you can atomically merge those things back into the main branch. So your infrastructure changes and your software changes that may be interdependent on each other all get merged into the main environment at the same time. The same sort of Terraform application process that you used in your branch then gets applied to the main environment at the same time uh the same sort of terraform application process that you used in your branch then gets applied to the main branch your new software version gets deployed and if everything goes according to plan uh now you've updated your environment while while doing so in a way that gave you high confidence that the changes that you were making were actually

going to work before you tried to do them right right and presumably like we've discussed before if it if it doesn't at that moment in time the hope is you could just revert that commit to main and it goes back to everything before as long as terraform does its job and as you know if anyone from hatchet corp is listening never i don't distrust it in any way uh it's pretty reliable so you can almost bet the farm on on it doing the right thing most of the time yeah yeah there are going to be some situations in which it can't figure out some path to go from wherever you were wherever you are but really i would say 99 times out of 100 uh it does exactly what you it would expect it to do so if you revert that change in the in the main environment it's going to then have a different terraform configuration, and then Terraform is going to try to change that configuration.

You obviously have to be careful of things, and this is why we have a few individual protections in place. If you were to, say, add an S3 bucket or add a data store or add some other thing, roll that into production, write some data to that data store, and then realize that you had another unrelated problem. If you were to roll that back, it's going to delete your data store, right, by default. And so you want some additional protections in there to say, like, hey, if you ever try to do this, just don't.

Right. Right. Sounds awesome. From my own personal experience experience the trickiest part of this is when you start doing refactoring in terraform and you want to like say well i do have 10 running ec2 instances but they've got terrible names in the terraform and i want to rename them in terraform which means i have to do this unfortunate two-stage thing where i changed the name and i don't want to delete them and recreate them i want them to be this and there's ways and means inside terraform of like using state to actually say okay i'm renaming this thing

in the actual uh state and if you go full automated you don't have the little breathing room to do that where i'm like i have to kind of literally call around people and say okay i'm doing some like surgery on terraform i'm going to rename this thing which means i have to rename it in the backing store which is a terraform command and then i'm going to change the text file and then i'm going to terraform plan and it should say no changes needed i'm like good because i didn't really change anything so i don't know if you've had any uh experiences with that stuff yet or do you just say i haven't had to go through that process yet um right part of it is because and and i think this also is sort of related to another uh potential trade-off with this approach that i'm talking about is that your

branches can get very expensive yeah like if you have lots of infrastructure that has like a per hour cost to it load balancers for example exactly uh then you know running a branch can be can be very expensive right and so one sort of side effect that that i have kind of seen or felt working on this project is that it leans it it it leans me toward using more like serverless things and things that can basically scale from zero. Scale to zero, right. So, yeah, if you have like auto-scaling groups, you say, well, they start out at zero and the first request that comes in, unfortunately, it's going to be delayed. But that's fine for this.

Right. Yeah. Exactly. Or as you say, lambda type things. Yeah, lambda type things. I mean, there's lots of them out there, right? But it sort of has me using those things more because I know that, you know, we're going to be creating a lot of these branches and we want to be able to iterate. And it's like, yeah, if you use them, you want to scale up to be able to test them.

But, you know, if you're not using some particular functionality in a branch because you're testing something else, you don't want to pay for it, right? So, you know, for better or worse, it's sort of like the architectural direction of this project has headed in that way just for cost reasons. That's really interesting. As I say, like, well, on the extreme end of like what Compiler Explorer does, I'm like deliberately sharing a whole bunch of things so that I don't pay the cost for the load balancers and the storage or whatever. And the other thing that we deliberately don't bifurcate is the storage of a whole bunch of stuff because we have you know three terabytes of crap and you know there's no way i'm gonna keep deploying that to a new environment every time one gets spun up and similarly i want to be able to create a short link in one domain and test that it still works on the old version or the new version and stuff like that not sharing tables

behind the scenes so there's some sort of edge cases with that but i would also like to be able to say no i just want a whole new copy of the whole thing somewhere else so i can make a wholesale test and i have the advantage on this project of it's a data warehouse so one of the things that it is supposed to be really good at is copying data around so and so when i want to copy data from one environment into another turns out we have a lot of great tools for that already that's part of your mo yeah yeah yeah so we sort of lucked into that but otherwise it would be kind of painful like you'd either have to have a thing where you have you know like maybe read permissions into the main environment from any of the branch environments so that you can sort of right and every time you do that you're sort of slightly eroding the nice guarantees that you had before about like the isolation of things and

whatever but sometimes you just this is what i mean this is what makes it engineering and not science or art right it's like there are trade-offs all the way through this right right right and we have had one situation thus far on this project it's been going for about six-ish months now something like that we've had one situation where a change in a branch environment leaked over into the main environment. And this was because of this thing. We had some data in the main environment that was being reused for testing in the branch environment and additionally, we had a permission that was set incorrectly, right? And what had happened was basically the system running in the test environment saw this main environment data and said, oh, I need to go disable this object, this thing, this resource. But it was the main resource, right? And it went and it disabled it in't have had the permissions to do that but you know permissions

in in aws and in terraform can be a little tricky to get correct as discussed today you know it's not necessarily the easiest thing to get right yeah yeah it's not like you can write tests for those kinds of things so you just have to sort of like i don't know if i mean i know that because there's a testing framework for aws permission AWS has a built-in permissions thing where you can run what-if scenarios, but it's very much as a service. It would be cool if there was a standalone thing that allowed you to sort of write these things where assert, like, given this environment and this user, assert that they would not succeed in deleting this file that would be pretty right pretty cool maybe something i guess you could you could maybe do a thing where you like decorate parts of the aws sdk and you say run as if i had this policy right and then you could like try to do operations against a basically like a non-existent environment

and say like you know you don't have to give me the result but just tell me what i would have been would i have been permissioned to perform this the real trick though is that it's so incredibly complicated it's not like there is a policy that's you know the user the iam role has a policy the user has a policy the machine you're running on has a policy the then on the receiving end like oh the the bucket has a policy that grants anyone with like a name who's you know ends in a queue they're fine they can write to me you can do literally anything right as well as the other way around so i mean who knows yeah yeah how is our certificate doing uh let's let's give it a go because we're're running out of time here, and this might be a rambly third part coming, where we actually get it to work for reals.

Yep, yep, yep, yep. Okay. Drum roll. Certificates. It says it's issued. Let's try running the terraform again. Terraform! What a wonderful way to end, if we actually... Well, I say end. We've still got more work to do, right? Because we say we still got more work to do right because we always work more work to do uh-huh yeah issued in use no so hopefully that will turn to yes here in a second and we modify our cloud front distribution which says it's modifying okay so that's cool and then the has it made the dns change because that's something i've still got open in the terminal is i'm still dns looking up to.org to sort of see if they're...

I guess once the CloudFront... Yeah, it has to be after the CloudFront stuff. Yeah, so I would expect the CloudFront distribution would use the new certificate, but I don't know. I'm trying to remember what we did here. I don't know if to remember but you haven't put the alias into the DNS yeah and even if you yeah let's see here

actually no I think it might let me go take a look here so I'm still not getting it on that and I'm talking directly to it to the DNS that should be reserving up these requests there's no caching going on. So I don't see it in the console.

I'm hopeful that when this Terraform applies that it will actually add that. Got it. So at the moment it's modifying the CloudFront thing. And presumably because in the new DNS records you use a var that comes from the CloudFront domain, that is its unique name, it probably depends upon it. So it's waiting for that to be applied before it does it, even though we know that it would be kind of okay. So, all right.

Well, CloudFront takes a while. Oh, crap. No, that's still coming out. Okay. Well, that's quick to apply, though. So we can probably. Yeah, I'm going to have to add that. Yeah, that's all good.

And Compile Explorer is rolled out, which is other good news. Okay. And then, yeah, so that's applying. Yeah, the CloudFront takes a while as it has to kind of get the okay from all of its geographically diverse regions before it says, oh, that's good. My guess is that I'm also going to have to add in A couple of these guys We're so close

I'm actually going to say to the people who I'm supposed to be now meeting That I'm not going to be there Most of whom have said they can't make it anyway So this is fine Apparently we have a day job as well There is that I should probably check

On the other computer That I'm not being hassled or harangued. Long silence will be cut from the podcast during the edit we can as we wait we can do the magic okay all right so that applied so i'm going to do one more plan for these other route 53 changes and then i think i think we're at it and then getting close now I have an undeclared resource probably because I spelled whose compliment wrong I guess I do that all the time

yeah well I put a.com instead of a.org that'll do it that's not really a misspelling, is it? I mean, strictly speaking. No, it's just wrong. All right, let's do this plan. Okay, what does the plan say?

The story so far, the CloudFront domain is using the new certificate, and now we're about to apply the DNS changes that will be still not used by the Internet at large, but will be used by my console but it's set up to use amazon directly so yeah we're creating two we're removing a uh uh route 53 entry and adding two more um oh because one of them god oh what happened i think i did a.com somewhere. Oh, really? Maybe it doesn't. Hopefully after this runs

I just named something.com. It wasn't actually like a domain name. Oh, okay. Right. It was just like a variable name effectively in. Yeah. Alright. Yes. Okay. So still running. We'll make this look really cool in the edit because it'll just work first time, every time.

Yep. 60% of the time. Works 100% of the time. All right. So we're wasting an application. Tried to create record set a record, it already exists did you manually make one before or have you duplicated it accidentally in the terraform and terraform hasn't noticed this mistake which is i think that is exactly what i did yeah that's that's my mo because like a

terraform will go this looks valid to me and they'll do the plan and it said this is what i'm going to do and then amazon turns around and says, no, no, those are the same thing, you fool. You already got one of those. I told them we already got one. Two's Confident that it was already declared main TF. Oh, yeah. Is that what this is? Oh, yes.

Okay. So I do have in this thing the, yes, the verification records. Oh, I just went to just the naked toothcompliment.org, and that has applied. I can see that it has lots of A records for all of the various different servers. So that is insightful. We just need dub, dub, dub to be the same,

which you're working on, presumably. Yes. And see, this is're working on, presumably. Yes. And see, this is one of those examples, incidentally. If you've already created one of those things in the console and you've got it in Terraform as well, which that's one of those things where you adopt an existing, you know, Terraform import. And that's harder to do in an automated environment, unfortunately. But then, you know. I really feel like you got to be all terraform or no terraform you know what i mean like living in the middle ground is just that's true but like

you know you have legacy projects for example where you need to do a lot of adopting of what's there and my my usual trick is to um write a skeleton of my best guess as to what i think a resource that i already have looks like and then import it and then go plan and then basically copy the inverse of everything it says it's gonna do back into the terraform oh I'm gonna delete oh yeah I had that all right what does it say what does our survey say uh it's saying try to create resource set to scumpland.org a record but it already that's because it does already exist now I can see www.to scumcomprom.org is also those addresses. So that's good.

Good from a it's working, but not necessarily good from it's going to work each time we apply. Because it thinks there's something there that is... This is almost certainly something that I'm just copying wrong. Great googly moogly. Sometimes when we say stupid things like this, it makes me worry, not worry, it makes me feel sad for you when you have to do the transcription of these because the automatic stuff has got no hope

with a lot of these words. Yeah, true facts. So those two records look right. And yet it thinks. And yet. Did you switch from having one that was managed by, oh, it should be deleted then. No, I was going to say, did you move from a four each

or two or four each from not a four each or stuff like that? Is that potentially the problem no i've had that before where it's tried to like create something before it destroyed the old version and they happen to have the same name and it didn't realize that they were going to stomp over each other but that doesn't sound like this video records first name value and type uh you know the other thing i'm going to do is i'm going to open this up in uh some jet brains tools so that i can get the terraform plugin to tell me if i've done anything but the thing is terraform would tell you yourself right you know terraform validate and terraform itself plan will do at least its side usually the problems come when it tries when the rubber hits the road and

it doesn't know it doesn't properly model what the provider is going to do when it actually applies these things. So it has no idea that those things already exist. Alright, here's what I'm going to do. I'm going to comment out the verification no, the two ones that we need are the top

level domain and the www. So I'm going to comment out the record for the verification because we did that manually once already. And we can always just blast those all the way in both the console and here or whatever. Yeah, let's apply this. Let's try and get the closure of knowing that it applies cleanly and then I think we're pretty much done

here. Yeah. Yeah, yeah, yeah. Did that apply cleanly now? Yeah. Oh, it's gone. I mean, we'll see. We'll see. But it's trying to do it. 20 seconds. Stay on target. Oh, the console's looking good, though.

I got two records. They both point to the cloud. That's what we wanted to see. I mean, I'm seeing that on my side as well here. So I think we're there. Yep. yep and yes terraform okay complete then i think we can declare almost complete victory at this point we don't fully understand why those those other records were either they're not there or whatever maybe amazon's putting them in automatically as well as you trying to put them in manually or something like that that would be my guess now because it's

managed by them already um so you just just leave them out and then terraform never needs to know they exist right and it'll just work so final work for this then is to uh double check the certificate looks good which i think it probably must do and then point the top level domain registrar at aws change domain, the DNS records, DNS servers, sorry, to be Amazon's ones and or move the whole thing. It's up to you how you end up. Yep.

No. And then that should. And then finally we should bump the TTL back up to something kind to everybody. That's the other last thing that no, everyone forgets myself included is that like, well, if you don't need it to be 60 seconds,

then it might as, I mean, who knows who say anyone pays attention to these ttls properly anyway right right cool well there we go we got success we did it we got success hopefully by the time this airs people will actually be able to go to https to scumpliment.org and it will just work it'll just work fabulous awesome okay my friend until next time Until next time. Until next time.

You've been listening to Two's Compliment, a programming podcast by Ben rady and matt godbolt find the show transcripts and notes at www.twoscompliment.org contact us on mastodon we are at twos compliment at hackyderm.io more at inverse phase.com