LINUX Unplugged - 342: Shrimps have SSHells

Episode Date: February 26, 2020

A radical new way to do SSH authentication, special guest Jeremy Stott joins us to discuss Zero Trust SSH. Plus community news, a concerning issue for makers, an Arch server follow up, and more. Special Guests: Alex Kretzschmar, Brent Gervais, Jeremy Stott, Martin Wimpress, and Neal Gompa.

So something interesting happened today. A little piece of me dies every time I hear someone pronounce GIF as Jif. And so Jif peanut butter has actually teamed up with Giphy to help settle the debate once and for all. And has created a Jif GIF can or container for their peanut butter. And you can actually order that on Amazon now. This is an attack on me. This is absolutely an attack on me. Oh, hey, friends.

You did it. You got the right show. Welcome into the Unplugged program. My name is Chris. My name is Wes. Wes 342. It it. You got the right show. Welcome into the Unplugged program. My name is Chris. My name is Wes. Wes, 342, it's going to blow the doors off. Might be our best yet. You know what I love? SSH.

Me too. Who doesn't, right? And our guest, Jeremy, will join us in a little bit to show us some things you never knew SSH could do. At a scale which blows my mind, there's things in there that are just going to maybe change the way you think about SSH, but potentially change the way you log into your systems in the future. It's a real cool, really, really cool chat we had.

So he'll join us in a little bit. But we also have our standard affair of community news, some pics, some follow-up, and a lot more, like our virtual lug. Time-appropriate greetings, Mumble Room. Hello. Hello. Happy New Year's Tuesday.

That is a lot of people. That is a great turnout. Wimpy, it's great to see you again. Welcome back to the Unplugged show. Hello to everybody. Normally when we have a great turnout like this, I'd read the names out, but there's so many in there. It's just not possible.

So great to see so many of you. Today is one of those days where it's great to have friends, and there is an aspect of Luggs that we don't talk about a lot, and sometimes it can be a bit of a support group where it's where everybody knows your name, and it's just what you need. And today is one of those days for me. We'll get into that in a little bit. I have something I need to update everybody on. But in the meantime, you guys are making all the difference for me today.

I really appreciate that you're here. So if you'd like to participate in our virtual, please do because it's just getting better and better all the time. We do have the new time, which will throw some people off, but I think it's worth it. What do you say, Wes? Should we start with some great community news?

Yeah, we've got a lot to get into. Super excited to get the announcement out for the old Manjaro project. Version 19 of Manjaro is out. And, of course, in there is tasty updates to the different desktop environments. As well as, I haven't tried this yet. I was thinking about trying to get the old upgrade in before we started. Of course you were. Still loving it on the ThinkPad. But I was, I just thought, you know what?

I got a lot going on today. Today is not that day. But I believe in there now, and maybe actually Wimpy, you may know, I believe in there now they have really smooth integration for snaps and flat packs, like just the old double click and it launches right up and gets installed. Yeah, this is, I can't speak to the flat pack packs i don't know what the extent of the work is there but knowing uh the attention to detail phil and team have shown snaps i expect that both enjoy first class support there now i've had a chat with phil he's been showing me some of the development

as it's been going along and yeah uh they've've got all of the packages available side by side, so you can take your pick where you need to take a pick. You know what I noticed, Wes? They're using the LTS kernel. Hmm. That's a good idea. Huh. We didn't think of that.

There's a good reason for that. I wonder, you know, we actually, Wes and I, don't have the LTS kernel installed on our Archbox right now. And for the life of us, we can't quite remember why, other than we just felt like being dumb. I think we had the conversation, and I think it went something like, ah, screw it, let's see how far we can push this thing. And it's been great for the show.

Why do they use the LTS kernel? It must be for drivers or something, right? Yeah, well, I know one of the reasons is the fact that the NVIDIA 390 kernel is not compatible with anything newer than the 5.4 kernel. So I think it's there so that people that have got older generations NVIDIA cards have still got a support group there on an LTS kernel. I think an LTS kernel is a pretty solid idea on a rolling distro. I'm a big fan after last week. All right. Well, there was a story that I think we've all been, I'm wondering, I'm wondering if you can guess what it is, Wes. There's a story out there that we have been led to believe we'd have an update that caused inconvenience for a bunch of

enthusiastic purchasers. And we now have a resolution and an update. I'm trying to make it cryptic because I don't want to give it away, but Wes Payne, one guess. The Raspberry Pi 4 finally got its USB-C act together. That's right. Woo! That's right. That means I can finally buy one.

I've been waiting for them to fix this. Yeah, yep, you got it. So I guess, what was it? There was anything that was marked with an E cable for USB-Cs, and that was like one of the smarter USB-C type cables. Right. It wouldn't work with the Raspberry Pi 4. Yeah, they had some non-standard wiring,

so it thought it was actually an audio device and wouldn't give it all the juice. You know, I mean, USB-Cs got lots of fancy things, but you got to get those negotiations right if you want all the power that you have. Yeah. So it meant that some adapters worked and some didn't. Obviously, the one they supplied did, but one of the nice advantages in theory of USB-C

is we've all got all of these things laying around. With those first generation, they might not work. Yeah, I want to power my Raspberry Pi with my laptop adapter. Come on. So the register reached out to Eben Upton, right? Eben? Eben? Sorry, Eben. It's not you. It's me to Eben Upton, right? Eben? Eben?

Sorry, Eben. It's not you. It's me. Mr. Upton. That's the way to go. Mr. Upton. And they asked him about this and like, what the hell? Where's this at?

And he said, yes. Indeed, they had rolled a fix into their PCB design for manufacturing. And he would expect that it has begun to reach end users by about now. So if you're buying a Raspberry Pi now or in the near term future, it will likely have the fix. So if you would do me a big solid, if you get one of these and it does seem to be fixed, please let me know. I'd love to update folks and say, yes, they are actually out in the marketplace now. There was another interesting little tidbit to this story, though. When they were talking to Upton, he confirmed that the 4 gig version was really the one flying off the

shelves. But I guess before the 4 launched, they thought it would be the 2 gig version. I think that's probably because of price. They thought the price was too high on the 4 gig version. Meanwhile, I was over here being, could you make an 8 gig version? All right, I'll take all their input. I can get that. That's what I want. Yeah. I mean, they were kind of getting the data back in September showing the 4 gig was performing better. So I guess what this really is confirming, actually, is that that is still the case.

The 4 gig is still the top performing seller, which could mean maybe that 8 giggig version will show up. That would be great. Fingers crossed. I'm running three. I wanted to say four, but the fourth one's just for playing with. I'm running three in production at my home.

One does all of the things, probably too many things. And then the other is just doing Pi hole and one network monitoring tool, SmokePing. And then another is doing all of my security cameras using what's that project called that I can't remember? Shinobi? Shinobi, thank you. And I mean, I've been using them for months now. It really has felt like the first sort of

like, all right, I'll just deploy this and it'll work and it's powerful enough for all of my needs. It's not, you don't have to compromise too much. I'm pretty happy. And I'm actually kind of feeling like I reaffirmed that decision to go. The Raspberry Pi for me has always been a toy. And as an old sysadmin, the idea of putting a toy in production for my server seems like a bad idea. And I still think I'll regret not getting more frequent DDs of my SD cards because I know one of those suckers is going to die on me soon. Hey, that's on you, buddy. I know. And there are ways you could get around it.

You know, you don't have to use an SD card, but I actually kind of like the floppy disk like nature where I can just copy them. And anyways, where it has actually been pretty great is how freaking low power they are. Even the Raspberry Pi 4 is way less power than an x86, even a low-end x86. The whole load for my entire Lady Jupes RV with all of my smart lights on and all of my Raspberry Pis running, my Wi-Fi, my network switch, my router, all of that, and all my LED lights, all of it together is less than 200 watts.

Everything. That's fantastic. Even on a cloudy day, I get that in solar power. So I run my Raspberry Pis off of solar all day long. That's pretty cool, and I don't think I could do it with an x86 PC in the Pacific Northwest. Right, you can really just optimize. It's just enough compute, just enough resources,

and you don't have to compromise. And silent. You know, that's the other thing. Maybe they'll die a year or two sooner than if I was not using them in production. But at that price point, realistically, I could replace all of them every year. And the form factor is so easy.

You know, you kind of grab it, you just slot in your new SD card, plug in the, you're done. Well, and realistically, right, you're going to replace them when the 8-gig version comes out. Yeah, especially the one that I have that's running Plex and it's running a bunch of other containers like a document wiki and a bunch of network tools and all these other services that really push the Pi. I was really kind of seeing how far I could push it, and then I was like, okay, well, now it's invaluable.

I pushed it really far and made it invaluable. You do have backups, right? Every night. every night. But, you know, it's still not the same thing. I have all the data backed up and all the applications are in containers. But I'd still have to burn a day getting the OS all reconfigured and all of that. So I just don't want that. I've insulated as much as I can. Sounds like you need to teach Dylan how to make copies of SD cards. I like that you and Cheese keep coming up with jobs for

my son. This is good. I'll get a list from you guys and then I'll start getting them. These are great chores. Sorry, Dylan. Son, today I'm going to learn you how to DD an SD card. I think I might DD rescue it. I don't know. Does anybody in the mumble room have tips for how to duplicate an SD card in the Raspberry Pi? Don't use

DD or DD Rescue. What should I reuse? This is good. Well, you know, GNOME disks can restore disk images like SD card images onto SD cards. That works really nicely. It's not just using DD in the back end? No, of course not. It's for animals.

We've evolved since then, Chris. That was like, you know, 30 years ago. Okay. All right. I'm totally down for giving that a go. And it's so low impact to test it, right? Here's what I was thinking my process would be. Shut down the Raspberry Pi, pop it out of the Pi,

pop the current production SD card in my ThinkPad, image it right off the built-in SD card reader, and then pop a new one in and restore the image to that, put that in the Raspberry Pi, see if it boots. Yeah, sounds good. And Gnome Disks has create a disk image and restore a disk image, so you can do it all with a UI and not run the risk of DDing over something critical.

Yeah, but what if he's on plasma? I'll install Gnome Disk, it sounds like. Hey, I will say Gnome Disk is actually really great. I'm running it on everything, even on Plasma. I've definitely taken a look at Gnome Disk. Is there like a certain point where it's like you got to have at least this version or has it just been good enough for a long time now? It's been there for as long as I can remember.

I mean, I was certainly doing lightning talks about this four or five years ago. I thought someone was going to say Clonezilla to me. That's what I thought I would get suggested was Clonezilla. But I like this. This is much simpler. Chris, would not a good alternative to avoid the manual intervention here be to simply run some rsync over the network to your backups? I mean, it's not as complete a solution, but it won't take you a day to restore. Yeah, and just rsync the file systems around. Yeah, I wonder, taking the whole theory of the

Raspberry Pis are not so expensive that maybe it wouldn't be impossible to buy another $45 Raspberry Pi and have it running as sort of a cold backup that's syncing every day. And then if one dies, I just pull the plug out, plug in the other one. Why do you need to restore data and move data around? Aren't you running everything in containers? Have a Docker Compose file stored somewhere off site, GitHub maybe, and then spin up your containers, plug in your persistent storage, and you're good to go, aren't you? Yes. Mostly the data is solved for.

It's really trying to avoid the hassle of spending the afternoon installing Raspbian, getting it updated, getting the Docker repo added, getting Docker installed, getting that data reinstalled. Avoiding all of that stuff that takes a guy like me an afternoon because I'm doing three other things and it's a Saturday. I want something that's nearly like plug a new SD card in, power it back on, and it's pretty much good to go

because in theory, the data would be safe because the data is stored on a separate disk. Have you heard of my friend Bash Script? He can do a lot of that for you. Yeah, well, that's what Brett was saying is maybe just automate so that it's always synced up. Yeah, I kind of like it. Or Ansible, or

there's a lot of ways to solve this particular turkey. That is actually part of my problem. I got so many options. I can think of a half a dozen ways to do it, and then the internet could probably think of a hundred ways I could do it. You should probably pick the one that will be the best content for the future. That's what I'm going to say.

Yeah. So, so I should say I should wipe all of them and put Arch on there. Well, I mean, I think the latest Manjaro release will be fine. We'll see about that. I'm actually,

I actually think if I were to reload them, I'd probably, I'd probably still for today, I'm not sure, stick with Raspbian. With the Raspberry Pi 4, it's been semi-necessary, although I think that's changing.

So I may just relook at this in a few months and then redo it then. You know, probably one of these will just fail on you and then you'll have to figure out how to restore it and you'll figure it out then.

Undoubtedly, undoubtedly. That's totally how that's going to go. No problem. Well then, let's do a little housekeeping. There's a few things I'd like to tell you about. Number one, you have to go check out Brent's recent brunch. He sat down with Heather Ellsworth from

Canonical. She's a software engineer on Canonical's Ubuntu desktop team. She's also a GNOME Foundation member. And she has quite a story, including a background in physics at CERN. And just a great conversation overall. It gives you kind of an insight, too, at daily life at Canonical as well. It's fun, right? It's one of my favorite brunches so far.

Yeah, Heather's awesome. Her chat, we could have chatted for probably two hours, but maybe that'll come sometime in the future. But anyway, she's fascinating. She's super awesome. I encourage everybody to check it out. Absolutely. Also, well, okay. So I have a little bad news.

We have been trying to get this sorted out for a little while. I was hoping I wouldn't have to come on air and say this, but we will not be attending scale this year. And it's a little last minute. And the reason why that bothers me is because I know there's already some folks who have made travel plans to come out and see us. You guys are awesome. But the reality is just didn't work out this year.

is just didn't work out this year. The merger happened just right around the time of Fest season and budgets are getting finalized and it just didn't iron out. So we decided to put our efforts and resources into Linux Fest Northwest, which is going forward. It's a shame. You know, we love coming out and seeing you guys. You've heard me talk about how important the in-real-life connection is a million times on this show. And these are always our chance to actually practice what we preach. So it is unfortunate. Elle will be there. She'll be doing her talk, Confessions of a Cis Admin, in Ballroom H on Saturday. Definitely don't want to miss that.

So she will be there. You can go say hi to her. But your buddies, Chris and Wes, won't be making it this year. And it kills me, actually, to be honest with you. But we will try to make a great Linux Fest happen, and hopefully next year we'll make it work. I apologize that we weren't able to get the news out sooner. We were just trying to make it all work, but it just wasn't really in the cards. Wrong timing this time. Yeah, yeah. But

it's going to be a great scale. Oh yeah, we'll be looking on and we'll be jealous and watching conference talks when they're available. Hope everyone has a great time. Jill, tell people where they can come find you because I know you're going to be at scale. You've always got so much going on. Yeah, so I will be at the Linux Chicks of Los Angeles booth and running

that booth and I'll also be doing interviews on the floor for Linux Gamecast. And we're also going to go live from Linux Gamecast. So and at the Lutris booth. I'm going to be everywhere. Jill, you are working it this year. That's awesome. Yes. Scale is such a great event. Linux Fest Northwest is our hometown event. But Scale, I think, is one of the best events in the West. Yeah, and I'm going to miss my Jupyter Broadcasting family. Yeah, well, we'll miss you too.

Yeah, it's always a good chance to catch up with everybody. That's one of the things we love about it. But perhaps next year. All right. Well, then, the housekeeping goes on. One other thing I'll mention, go check out Headlines, linuxheadlines.show. We're doing something kind of neat here. We're trying out a new format, a three-minute podcast. That sounds crazy. How could that work?

I like it. I just put it on as I'm getting down the road, and then my podcast player just plays the next thing. It's just sort of the way I start off. linuxheadlines.show, three minutes or less, everything going on. And here's what we try to do here. We try to do something that's not hypey, that's not just taking PR releases, re-massaging them, and then putting the headline on it that makes you want to click it. We've got a team of people that are dedicated to trying to get that accurate coverage in there, trying to get it just right. And then sometimes to our dissatisfaction, we all argue over the details for a while before we publish it to make sure everything's really massaged just right. And it gets reviewed by a team of us before it's published every day.

You'd be amazed the amount of work we're putting into a three-minute show. But I think it's kind of important, especially if you're in the industry. You want to keep aware of what's going on. You want to be current, sort of like due diligence in a sense, but you don't need to commit to two hours or an hour long. Just get the headlines every weekday at linuxheadlines.show.

And then the stuff that really needs a deeper analysis, further discussion, we'll either do it here or we'll put it on Linux Action News on Sunday. And of course, we've got links to everything so you can find out more. Bob is indeed your uncle on that one, I say. So check that out, linuxheadlines.show. And last but not least, a plug for the old Telegram group, just rocking it out.

People are awesome over there. Some super good conversations. I thought for sure it would fall apart as it got huge, and it's gotten better. It boggles the mind. jupiterbroadcasting.com slash Telegram. Go get in a 1500 and stronger group over there that are talking about these shows, Linux, and everything in between. Jupyterbroadcasting.com slash telegram. Yeah, we actually just crossed 1,600. That's amazing. And it's just a nice place to be. Great conversations basically all the time.

That is so cool. All the time. Now we have a little bit of a public service announcement that I want to get into on the time. That is so cool. All the time. Now we have a little bit of a public service announcement that I want to get into on the program. This is something that's going to impact those that are in the makerspace or love to see technology, cutting-edge technology, and see where it can go and

tinker with it. We cut now live to our reporter on the scene. Alex, I'm hearing that people are getting very upset about new drone changes that are coming to the States. Well, not strictly Linux. Actually, a lot of these drones do run Linux. And I was kind of hoping maybe you could update me

because I have not properly educated myself. So the thing that I'd like to bring to the audience's attention is something called Remote ID. And Remote ID is effectively an electronic license plate for all model aircraft. it's not just drones so we're talking about helicopters planes anything that flies essentially is that's an unmanned vehicle uav so on december the 31st of 2019 the faa published a proposal for what remote id is going to look like and you have 60 days to comment and that

period ends on March the 2nd. Now what happens after those comments are reviewed is the FAA will publish a final report and the new rules will take effect 60 days after that. So why is it such a big deal? Well for me there's several things that are an overreach here. So let me just break a couple of the key things down that kind of rub me up the wrong way. Firstly, I build all my own drones, like racing drones and that kind of stuff. And as it stands, you will not be able to do that anymore because you will need to buy 100% FAA approved components. Else you will be restricted to FAA recognized flying sites otherwise known as freers now most of those are AMA American Model Association fields so I don't know if you've if you're into FPV or drones and you've ever tried to go to

a plane field you get the stink eye and you're not really very welcome there so you'll only be able to fly self-built models three years after these new rules go into effect at these freer sites now a further kicker is that they're not approving any new freer sites ever because as they assume in their rules, all craft will be remote ID capable. So there'll be no need for new freers to be approved. Well, that's just not true. So in terms of where you can and cannot fly, you will not be able to fly in your own backyard on your own property. You will not be able to fly in parks, anywhere that is basically currently legal. mean obviously national parks and stuff like that are still out of range but i think it's just a huge overreach to try and regulate the

the model aircraft's base that sort of sub 400 feet altitude space so that people like amazon and fedex and ups can start flying commercial delivery drones. So if your aircraft is not remote ID capable, you will only be able to fly it at a pre-approved freer site. If your aircraft can share location via the internet, so this addresses most DJI type products, you will need to subscribe to a subscription service, which hasn't yet been decided. So let's assume a minimum of five dollars

a month and at that point you can only fly in a 400 foot sphere around you so that includes up but also sideways so 400 feet that's it you have this drone that can go five kilometers and you can do 400 feet so i think it's pretty egregious i I think it's a horrible overreach. Now, what can you do about it? You can leave a comment on the FAA website before March 2nd. Please try not to make it an emotional type, reactional type comment that, you know, FAA sucks, that kind of thing is not very helpful. But instead focus on the specific areas of remote ID that you think are unworkable or an overreach and propose alternate solutions such as app-based systems. Lance is something that already exists for part 107 commercial pilots to

essentially say, hey, I'm going to fly here through an app, get pre-approval to fly in that airspace. And just remember that as this proposal stands, it's going to pretty much kill the remote control hobby as we know it. Yeah, it's a shame because it seems like that community has been really blossoming. It's a makerspace thing now, like building these little drones as a maker thing. It's a huge part of the reason I'm into electronics and self-hosting and all that kind of stuff. I mean, I got into building my own quadcopters and learned electronics through that. And that's led to a

whole bunch of other stuff that anybody who listens to self-hosted will be familiar with. So many prominent YouTubers have released really excellent content on this topic, including Flight Test, Joshua Bardwell, Roto Riot, Mr. Steel, et cetera, et cetera. So please leave a comment for the FAA, even if you don't live in America, just leave a comment anyway. LetAA, even if you don't live in America, uh, just leave a comment anyway. Let's try and get the numbers up. Um, if you want to help make a difference, you can also join me and thousands of other people, uh, remote control pilots who are going to be in Washington, DC this Saturday, the 29th of February for a protest outside the FAA headquarters, uh,

just off the national mall. So, um, thanks for letting me raise the FAA headquarters, just off the National Mall. So thanks for letting me raise the awareness of this issue, Chris. Yeah, we want to talk about it now since the timing is critical. There you have it. There's Alex with our drone update. He didn't mention it, but he's actually being carried by a drone right now. Thank you, Alex, for that very important public announcement. Yeah, go speak on it if you can.

Make some comments, make some noise, and hopefully there'll be some reconsidering done. So now I want to talk about Zero Trust SSH. This is something that is so cool. Jeremy's going to join us and talk about things I never knew were possible for SSH authentication. It truly is the magical, magical fruit in open source. And he's got some problems to solve that

I'd never really contemplated before. All right. Well, we all use SSH. And I mean, I dare say love it, right? From port forwarding to proxying to just getting to a server securely. It's super handy, but we don't all use it in the best way. I mean, I'm looking at you, Chris, you're one who uses password-based authentication. A lot of times, unless it's like just a few of the regular machines, I just stick to passwords. But you at least know you have other options, things like public-private

key pairs that you can use to get onto servers. Well, there's another way to do it, and it has some advantages, and Jeremy had a great presentation all about how you might actually use it. SSH has certificates. And Jeremy, thanks for joining us to tell us more. Oh, thank you very much.

Yes, password-based auth, it is sort of maybe the first introduction you get to SSH. You know, you sort of start it up and then try and SSH straight away, and you're prompted for your password, and you think, oh, well, you know, this is pretty familiar and it works no problem. But then as you pointed out, there's that public-private key authentication. And maybe if you're like me, you sort of tried to try it out because it was a bit of a more exciting way to connect, you know, or you've just gotten tired of typing in your password all the time.

Or maybe even if you're at a company, you might be required to use private-public-key authentication. So there's some distinct advantages with private-public-key versus password-based. You're kind of authenticating your laptop to the server instead of your password. So there's some advantages

that you can't accidentally mistype your password into the wrong prompt, for example. There's some other neat advantages of public-private key. But like you said, the third way that I discovered is already in OpenSSH since 7.7 point something, I can't remember, is certificate-based authentication. It's not an X.509 certificate. It's a kind of a lightweight certificate. Really, all it is, is your public key that's been signed by another private key. A little bit of metadata like expiry and username, things like that. But that's all it is really. So instead of having your public key and private key, you have your public key plus that signature and your private key. And you can't have a chain

of trust like in a normal X509 certificate, like a TLS certificate. So you can't have like a root authority and then an intermediate and then things like that. It's just one. You just have one level of delegated trust. That's right. That's right. But when you go and SSH to a server, so with public private key, what you need to do is put your public key on the server. Right. I think we've all been through that, having to share your private key with someone or copy the public key over so that it's there.

And maybe you've got a little script to help you with it. Exactly. And maybe you're giving that to your, you know, to your IT team. It's like, Oh, here's my public key. Except with, so with the SSH certificate, you, you never have seen the host has never seen you before. Right? So you, you try an SSH to it. And as long as your public key has that signature of the trusted certificate authority,

then the host will go, oh, great, this is you. This is Jeremy. I'm going to let you connect. So it kind of switches things up here. If I'm getting this right, you can tell that host to trust anything signed by your certificate authority. And then it doesn't have to have any specific keys added. Anything that's been signed, it'll automatically grant access to. have to have any specific keys added, anything that's been signed, it'll automatically grant access to. Absolutely. So for an individual, I mean, unless you have hundreds of servers, it might not be so much of an advantage over public-private key itself. Because you, I mean, it's not necessarily any more secure cryptographically. Right. It's using the same keys and the same infrastructure. Exactly. Yeah, but when you start having

more than one person or a team to manage keys for, I don't know if you've been through the pain of managing a team of public keys. I've definitely managed a Chef repository to push those. It was not a lot of fun. Well, I mean, even just

adding another one. I mean, my previous company was Puppet. And the flow for adding a new user was quite literally, you know, get pull, get commit, add the public key, make a pull request, get someone to review the pull request, you know, then merge the pull request and then watch the CI server deploy Puppet and wait for an hour. It was a bit involved for just adding a user to the system. It's some time that I just shouldn't have been spending doing anything like that.

I should do something else. So I really wanted to automate team management of SSH on hosts. And it doesn't matter where your hosts are. They could be in Amazon, Google Cloud, or Azure kind of thing, or it could just be your own servers in your closet somewhere. But yeah, SSH certificates will really help a team manage their access. Because if you imagine your hosts suddenly don't need to know in advance that your user is going to connect to it, then you can bring up new hosts all the time.

But as long as the users can go to that certificate authority, get their SSH certificate signed, and then connect to your host, it'll be smooth sailing. That sounds really nice, honestly. But how do you go about making sure all those certificates get signed? Is that another manual process or can you automate? Yes, that's a good point. And so, you know, have I just swapped my problem

for now public key infrastructure and managing public keys, certificates? And you're kind of right, because it's maybe a more difficult problem even than just public keys. create the certificate authority in as light as possible way. And we happened to use Amazon, so AWS for most of our things. So I built a certificate authority in the Lambda function. So it's not a new concept, actually. Netflix, I found it first with Netflix. They have a project called Bless, Bastion's Lambda Ephemeral SSH Service. Bit of a mouthful, but they were still using this jump host idea and it was connecting users SSH to that first jump host and infrastructure, and then it would go off to the Lambda function,

which is really just a bit of code running somewhere in Amazon, who knows where, could be on their vacuum cleaner, I don't know. So you run your Lambda function, and then it will sign your certificate for you, sign your public key. Sure. So you send a public key, it'll sign it and send you back the sign key. That's correct. So what Lyft did, which was quite interesting, is they decided, well, we don't really want this bastion host because it's just, you know, complicating things and our infrastructure is maybe not set up for it. So they decided to make a Python client that worked on developers' laptops. and sign its public key. And it would invoke that Lambda function itself and had a really clever KMS auth thing to sort of prove your identity to the Lambda

to say, yes, I need my certificate signed. And remember, all the Lambda's doing is just takes your public key, it's got a private key pair, private key, public key pair, and it just signs your public key. It needs to know what username to put into that certificate. So it, it needs to know what username to put into

that certificate. So it kind of needs to know your identity, a good sense of your identity. Because if I could get my certificate signed with someone else's username, then I'd be able to connect as them. Right. You've still got this trust layer you need to take care of. Exactly. And actually, it's funny in like a zero trust model, the idea being that you don't trust anything inside your network or outside your network. So, yeah, kind of each piece should be on its own. You don't have this firewall and everything beyond this firewall is this wonderful garden. The funny thing is zero trust is almost like more trust, right? Because you're adding all these extra layers.

Right, now you're authenticating everywhere. Exactly. So one thing I tried to do with the Lambda that I wrote was to use OpenID Connect to authenticate to the Lambda. So you kind of like sign in with Google

to get yourself that certificate. It just creates that identity token, which is this bit of signed JSON JWT by Google, and you can pass that over to your Lambda, and it can verify, yes, your email address is correct, and it can put your email as your username, for example. So then in this mode, onboarding, I'm thinking, I mean, here at Jupyter Broadcasting,

we've got some G Suite going on, right? So I could configure a new user there, which I probably have to do anyway. Right. And then if it was all set up correctly, they could just have access to servers. That's the dream, that a new engineer starts and they have their laptop and they already have the access that they need without you having to do anything other than put them in the correct groups in your directory, right? Right.

I mean, this is kind of a solved problem if you think, you know, Active Directory, LDAP, you know, like there's all these, that's all existing. But what I'm seeing is a lot of companies moving away from having, you know, those traditional directories and going to sort going just cloud native services. So you might just have G Suite, for example. And if you don't have a directory, like Active Directory, how do you now leverage some of those tools

that are all set up for it? If you're not running LDAP, what's the alternative kind of thing? And I think this is a good alternative. When you say that there's companies out there that don't have directory services too, I think to our audience that is using Active Directory or LDAP, that sounds ridiculous.

But Wes and I have and may currently have worked for companies that have no LDAP, no Active Directory. It is truly, it's a G Suite authentication system. Maybe you're running Slack, you're like, none of these things necessarily need that backend infrastructure. Right. Absolutely.

So for the company I was at, it seemed a bit silly to provision LDAP just for managing this. Right, and you're going to also need people to manage LDAP, right? That's its own arcane art. So help me understand how shrimp plays into this, because I loved that you got a name for a component of shrimp during the talk too.

That was a great moment of your talk. Oh, yes. Well, I was going through why I named it shrimp. And so shrimp has shells, right? Shrimp have shells and got a great groan from the audience for that one. Lightweight, right? So I guess you could have a really massive shrimp. I don't know. All those shrimps are pretty lightweight. And then someone yelled out, shrimps on the barbie, like a classic Australian expression. Comes from an advertisement ages ago, I think. Yeah. Yeah. So I had to integrate that somehow. I was looking for a name for the backend components,

so maybe that was the barbecue. You know, it's like shrimps on the barbie kind of thing. Yeah, I don't know. So shrimp is the name that ties all of this together, that it's the different components that enable you to use Lambda for this aspect. Right, yeah. So shrimp is the, there's two parts. There's the Lambda, and then there's the client on your laptop. So you

need something to go and automate the process of like giving your public key to the Lambda and then receiving back the signed certificate and using it in your connection. I mean, you can use just SSH Keygen. There's nothing special about what these tools are using. It's just sort of automating it in a way that really makes it easy for developers to get going. I think the idea is, well, my idea was if you make it easier but also a little bit more secure, people will jump on it really,

really quickly and really want to use it. But if it's just slightly more inconvenient, even though it's more secure, people will try and find a way around it. So true, right? If you can give them an easy and secure option that's right at their fingertips, there's no reason to choose the wrong path. Exactly.

So I want a method that they don't even need to, you know, put your public key anywhere. You don't need to like copy and paste anything. You're just up and running already. So what part of that was the client actually runs an SSH agent? So normally your SSH agent manages your private keys for you. And so it's a program running on your laptop Maybe it's even on us on a partially on a smart card or a YubiKey. Sure your SSH agent kind of manages that Where is your private key?

So I wrote an SSH agent. Well, actually, the Golang crypto SSH module is really great. They have an agent implementation there. So it's almost like just... You plug into it. Yeah, exactly. So I used the SSH agent to provide this Lambda-based certificate signing feature. So when your SSH client says, hey, I need the list of public keys,

I'm trying to connect to your server, it will ask the agent to say, what have you got for me? And at that point, the agent can say, just hang on one second. I've just got to go and just check what's in my, you know, Lambda closet over here. Just give me a few minutes. And it goes off and signs your, it actually generates a new private public key and then goes and signs the public key. And then it returns to the SSH client and says, oh, yes, everything's normal. Don't worry about it.

Here's the public key and private key. So to the SSH client itself, none of that matters. He just gets back the key that it needs and it ships it off to the server. Exactly. The advantage of that is other tools that use SSH work just great, like rsync or scopy or maybe even Ansible or something like that. Those kind of tools, they just will be none the wiser. Some other ways of doing this kind of rely on wrapping the SSH command.

So if you wrap it in another command, but that almost breaks that flow. So you can't really use those other downstream tools that are using SSH underneath. Well, Jeremy, thank you for joining us. What time is it where you're at right now? I've just hit 7.30 in the morning. Thank you for getting up early to chat with us. We really appreciate it.

And we will link to the talk. Anything else you think we ought to link to? Obviously, we'll link to Shrimp. Any other suggestions? So that Shrimp repository, I've got the good repo for that. One thing I didn't mention, which I found in the last week, and it's not actually in that talk, is um, is Keybase. Have you guys, have you guys heard of Keybase? Oh, oh yeah. Yeah, absolutely. Keybase.io slash Chris Lass.

Okay. Well, you're going to have a new follower very soon. Um, well, Keybase, there's a blog post, um, where they've implemented their own SSH certificate authority on top of Keybase. Oh, neat. And it's the most bizarre chat ops use case I've seen, where essentially, because there's no secrets being exchanged when you sign these certificates, right? You're giving your public key, and then the server, your certificate authority is signing it. You know, that bit's done in private. But then it gives you back your certificate, which is also a public thing.

Right. There's no secret exchange between the machines. Exactly. And it can do it over the Keybase chat. And it uses the Keybase API to do it for you. So it runs on your computer. It uses your Keybase if you're signed into Keybase. And then you have a chatbot on the other end, also listening on that same channel.

And when it says, hey, I want to sign my public key, it goes, okay, what's your username? And because Keybase has such a strong sense of identity, it knows, oh, yes, it's the right Chris. I mean, I haven't reviewed their source code, but if it's true that they claim end-to-end encrypted and the way they've implemented it, it will be very, very difficult to impersonate your chat message. So it really is coming from your laptop that it should be your request for a public key. And then the chatbot just signs it and then here you go, here's your public key, which is kind of cool because the SSH certificate authority doesn't need to be publicly listing on the internet. It doesn't have to be a Lambda function that can be invoked somehow or a server running on EC2 that is listing on some port.

You just need to be able to connect to Keybase. Exactly. It's kind of like an outbound connection. It could be a Raspberry Pi in your office, for example, or wherever you're, you know. I've tried it out. It's actually really neat. That does sound really neat. It's worth including. I gave this talk at Auckland OWASP Day, our security conference here in Auckland,

and I included that part as well. It's just awesome. You know, I love all of the ways you can use SSH. It's just been proven such a flexible tool to build on, I love all of the ways you can use SSH. It's just been proven such a flexible tool to build on, I guess. No kidding. Couldn't have said it better myself. Jeremy, thank you so much for joining us and going through some of this.

I really enjoyed it. My pleasure. Thanks for having me. It was great that Jeremy was able to join us and talk about that. But it's really still worth watching his whole talk because we've only scratched the surface. One thing we didn't get into but is really neat about these certificate-based solutions is you can set the expiry time really low. So for Netflix as an example, they just sign their certs for two minutes, just long enough to let you SSH to whatever box you're actually trying to get to, but then you can never use it again. Whoa, a two-minute SSH cert, that's it. So when they issue the certificate,

they just can set a tiny expiration window? Yeah, and you can do other things like limit where that's coming from to a particular host or IP ranges, say. All of these just go so much in the other, more secure direction from adding a key to a server and then never touching it again. But you'll want to check out other stuff.

There's lots of neat ways to actually make this work at scale. We'll have links to all the tools, including Jeremy's excellent shrimp tool. That's such a great name. It was nice of Jeremy to join us because he was in New Zealand, right? Yeah. You know, one other thing I loved about his talk is it was actually contained inside of an SSH

certificate. He wrote a Python script to display the talk. And then you can actually, with the certificate, you can use the force command option to basically, like, when you SSH to the server, it runs that command. And then the command ran his Python script, which then started up and displayed his presentation

in the terminal. It's fantastic. I am both, like, intrigued and a little perplexed by the idea of using Keybase chat, but I could see some advantages, especially in certain networking conditions. Huh.

Keybase is something we should probably talk more about. Remember how SSH can do all these crazy things? I mean, I just kind of use it the most basic way possible. I feel fancy when I do X11 forwarding or copy a file. That's, oh, look at me. I'm fancy. That's pretty great. We will have links to all of that as always. But before we go into the picks,

I thought maybe we could do a little follow-up on the old Arch upgrade we did last week, which did not go so well during the show. No, it did not. Oh, boom, boom, boom. We got a new kernel, which, you know, wasn't yet supported by the ZFS on Linux project, so that didn't work. Pause there. Here's a great example of when you are kind of rolling the dice with Arch on a server.

Because there is no central control making sure that you don't release a kernel that is incompatible with the version of ZFS you're shipping. In a traditional distribution targeted specifically at the enterprise, those are the very kinds of things that they work out and prevent hitting you in an update. Now, we know that going in, we were just kind of playing fast and loose. And there is a side repo like the ArchCFS repository, which has kernels and those things a little more in sync. We're not using that. This is just the stock kernel at the moment. At the moment.

You did try the LTS kernel for a bit, but it had issues as well, right? Yeah, there was some bug going on. It looked like that was going to be resolved more quickly than the GPL symbol issues that seemed to be lurking for kernel 5.5. So the old snapshot and rollback system worked mostly smoothly, except for you had to go remove a particular folder or file somewhere, right? Yes. So when we didn't have support for ZFS, that didn't get mounted.

We are integrating Docker on that host with ZFS, so it wants to go talk to ZFS, doesn't find it, but does end up making a folder under varlib docker that when you restore ZFS support means ZFS won't mount cleanly. Right, because now all of a sudden something exists there. Just had to go clear that out, reboot, everything came back.

So to be clear, we were using essentially butterfs snapshots to restore the system back. Yep. And then we have ZFS snapshots for the data. Exactly.

But our particular issue was is that the ZFS array would not mount after the kernel upgrade. And then even after we rolled back, it still failed to mount because it had created that directory. But nice fixing. I mean, you had it all fixed within a few minutes. Yeah, it wasn't a big deal.

And, you know, there's a lot of options that we could do. I'm sure that the latest kernel will get support before too long, so you can also do upgrades where you just leave the kernel out of upgrading, right? I mean, that's a fine kernel. Or install it with the LTS kernel. Or get an LTS kernel. That is probably the better idea. Yeah, this is legitimately

why we're doing this, is so you don't have to. If you ever wondered what it's like, our philosophy here is Arch with belts and suspenders and everything backed up, containerized or VM'd. Nothing actually running,

for the most part, on the Arch host box. That's the theory. And I think we kind of proved it out with these rollbacks. This issue here with that directory getting created only happened because we tried to bring the system all the way back up without the right ZFS module. If we had not done that, that issue would not have happened.

The rollbacks worked for the most part with that one exception. Yeah. And if I just remembered to delete that file before rebooting, it would have been.

So potentially, if this ever were to happen again, that rollback process you went through would be 100% effective. So how long in total would you say, now that you've done it,

if you ever had to do it again, would it take you? Five minutes. That's not bad. And a reboot in there as well, right? And a reboot, yeah. So that varies depending on your slow server like ours. Yeah. We could have also

just downgraded the kernel from the package cache as well, but it was more fun to use Snapper. Cubicle Nate wants me to just switch it over to SUSE. Yeah, one day we might try that. You never know. Yeah, that might be next in the old server reboot queue. See, what I'm concerned about is we'll be all like,

hey, let's try out 2004 Ubuntu LTS. We'll put it on there, and then we'll have no reason to remove it, so we'll never try anything else again. Well, I was just thinking that with Arch, and okay, yeah, so we did have to futz around a little bit. That did take a little time to go investigate, and here I am on the ZFS on Linux GitHub looking at issues.

But it also means I don't have to reinstall. There's no next Fedora that's coming out or even a next LTS release. That's just work I'm never going to do. Right. I mean, probably not because, you know. So maybe you diffuse that work throughout the lifetime of the box instead of sort of bundling it all up at once. That's the theory.

Although you still have updates on all systems. They're not all just as crazy and wild. That's the theory. Although, you still have updates on all systems. They're not all just as crazy and wild. That's right. We've got not one, but two app picks this week, and one is a collaborative ASCII canvas. What? Yeah, I mean, haven't you ever just wanted to hang out with your buddies and do some collaborative ASCII? Because that's what it's all about.

Yes, most days. Yeah, I mean, that's what, like every Thursday? Yeah, I mean, it's not collaborative antsy, but, you know, it's ASCII work for a time being. Or say you're SSH-ing around with your two-minute Netflix keys and you want to get with your buds at Netflix for two minutes and jump onto the server and collaborate on some ASCII. SSH-ing around is one do.

Yeah, SSH-ing around, that's what we do. I think we could do show notes in here. We'll just have to change our format a little bit. Go to two columns. Makes sense. So Kolasky, I guess that's how you pronounce it, is a pretty neat little app that'll allow you to collaboratively draw on an ASCII canvas together. Because why not? It's great.

It's great fun. That's pretty good. But I'm trying to live the Encurses lifestyle cheese. So what do you got for me if I just want to go Encurses all the time? So, yeah, I mean, if you want to Encurse, you know, your way into a display manager, you can check out LIE, which is a terminal display manager, which essentially will allow you to log into all of your favorite desktop environments, Budgie, Cinnamon, Deepin, Enlightenment, GNOME.

Wow. Okay. I never knew I needed this, but I'm kind of in love. Finally. I mean, how long have we all been waiting? Well, doesn't it feel silly to have this whole display manager thing anyway? It just gets in the way. No kidding.

And good news, works on Dev1 too, so no systemd required. And there's also a feature so that you can enable the PlayStation Doom fire effect. Oh, good. If you don't know what that is, I'll drop a link there into the channel, but it's just a beautiful

ASCII fire effect. See how these things kind of tie together? You see, and people say Linux is just for geeks. Look at this. Right? Come on. Super nerd fun. Think point made.

Yeah. Those are great. Thank you, Cheesy. Those are really great. Even if you're not going to install these, I encourage you to check out the links because they have screenshots and videos. They're just fun.

Super fun. Speaking of fun, I hear they're about to have some fun over there at the Ubuntu podcast. Things are in the works again, Wimpy, for a brand new season. They most certainly are. Yes, season 13 confirmed. And we're in preseason preparation stages at the moment. Should start recording in March. Was there any discussion to go right to 14 you know

just for the you know avoiding bad luck no we've decided to fully embrace the 13. oh i like it good for you open to podcast.org i'm going to give a mention for my website chrislast.com not only do i have my chris last cast there but that's where i'll be keeping links to what i'm up to and some of my social profiles like my key base, I think, is on there, as well as future things. So check out chrislass.com for that. I think you have some awesome pictures of your RV up there too, don't you? The new solar setup? That's on worklifeandrv.com, Jesus. Oh, which you can get to from Chris Lass. That's true. It is linked on chrislass.com heyo

ba-dum-bum shhh you know we should mention we know we haven't plugged for a little while speaking of pictures is the gallery oh yeah

jupiter.gallery go check out some of Cheesy's work there that's going to be updated here at LinuxFest Northwest and so if you are there and you do take pictures and you want to share them

just email me and we'll get those up on the gallery. LinuxFest. Now that's what I'm putting everything into now. Putting everything into LinuxFest now. That's all I got. That's all I got.

Go get more Wes Payne over at techsnap.systems. And he's on the Twitter. Where are you over there? At Wes Payne. How? Guess what? I'm at Chris LES. Aw.

Did you know? You did know. You can admit it. Yeah, I knew. At this point, if you didn't know, there'd be something seriously wrong, dude. You know, I think even like the network has one. I think it might. At Jupiter Signal.

What? But does the podcast have one? At Linux Unplugged. Oh my gosh. We're all over Twitter. They're like tribbles. They're like tribbles. Of course, we're live.

We'd love to have you here. It makes all the difference. Please join. JBLive.tv on a Tuesday starts at noon Pacific. We usually actually get rolling a little earlier, 1130. Yeah, come in early for the excellent pre-show. Yeah, and of course there's post-show. You hear a little tiny bit of it in the release version, but there's a lot more every Tuesday at JBLive.tv.

We'll see you then. 🎵 Tuesday, I'm blood program. Oh, I think J-Bot had a little much too much to drink at lunch, I think. You know, we shouldn't have given him access to the tab. I don't understand why J-Bot gets lunch on Tuesdays and we don't. How did that work out? How did he get that arrangement? We don't get lunch and JBot does? Come on.

I don't like any of this. This is not a fair arrangement. So we will have to rerun the title votes here in a moment after Wes. Sometimes it saves it. Sometimes. We'll see. Wes will go and try to rescue our other Arch server in production, which has been in production for a bajillion years, actually.

Yeah, I'd like to point out this one was not my idea. It has been running forever, though. So when we get that going. Forever ever. Forever ever. In a very special way. So not one that we would recommend. So, Brian, I thought you weren't going to be here.

Did you just get all your work done super quick? Were you a super efficient photographer? Yeah, you know, the great thing about being a professional is you can get things done faster. So everybody's happy. Client's happy. Photographer's happy. I got lunch right in before heading here. It's all great.

Brian, I don't believe for a moment you did something quickly. Nuh-uh. Nope. I know. Actually, though, when you see him bust out that camera, man, he'll just boom. The next thing you know, he's taking pictures. Like, it just goes from zero to taking pictures. The camera, yes, but editing or anything.

Oh, well, yeah. No. He's not done with the editing. He's not done with the camera. Hey, give him a break. He just took the pictures today. How's the life on the farm been, Brent? Aren't you farm-sitting?

Oh, man, yeah. I got to farm sit the last few days for good, good friends of mine who have a farm about maybe like 45 minutes from here. And I've worked on their farm a few times. What? What? Well, hold on. We got a Chris Lastcast about this. This sounds castable, my friend.

Did you take care of the animals? Yeah, so it's still, you know, the deep of winter here. So they only have like laying hens and their two dogs. How big's the farm? Oh, so they do mostly vegetables. So I think they're farming like two acres of like basically a huge collection of vegetables. Oh, I'd love to. Okay.

All right. Don't tell me too much. Don't tell me too much. I think this is, I think this is castable. You know, I have, I have family, I have a family farm think this is castable you know I have I have family I have a family farm on my mom's side up in Canada

oh um what yeah how do I not know this yeah well we got something to cast about my friend

I mean secret secret farms I get like a mental image that Brent's just like out in the field picking carrots and stuff like just having a snack that's what I did

last time I was there there's a version of me that just nopes out of the internet and I'm just like, I'm out and I just go to the farm and I spend the rest of my life on the farm. And I think it sounds wonderful. You know, Chris, I was actually, while I was there, I was trying to disconnect a little bit to use it as a means of just having a bit of a quieter, you know, 24 hours. And I was thinking back to the Linux Unplugged episodes

where we were talking about some strategies to kind of deal with burnout and stuff like that. And I was thinking, hmm, something I try to do often is have these little like 24, 48 hour little retreats. And you don't have to go very far. You just have to go somewhere a little bit different. All right, hold on. I got a response. We can't talk about this. We can't. We got to save it.

Write that down. You're getting too deep, boys. You have the notebook. You write it down. We can't talk about this. We can't. We got to save it. Write that down. You're getting too deep, boys. You have the notebook. You write it down. I didn't bring it in here with me today. That's something we should touch on. I know. Well, I have my backup notebook.

I do have a backup one. I have a whole podcast about keeping notes in notebooks. I got a backup notebook. It's just not my notebook, so I'm not as inclined to want to write in it. You know what I mean? Like, I've got an attachment to my notebook. I'll also point out the six computers that are around. I'm no longer writing on computers

anymore, Wes. So actually, I have specific note insights I'd like to share too. Are you also hanging your clothes up on the clothesline, just sort of becoming Brent? I wish. I wish. I tell you what, I'd consider it. If I could make this mate jacket last another few years, I think I'd do it. I'm still wimpy.

Oh my God, what are the chances the day you show up I'm wearing the Monte jacket? I think you might have summoned him. I think I did. Well, there you go. Yeah, I think that's how it works. That is so funny. Yeah, I just started wearing it and, oh wow, that's a weird twist. You're going to have to start wearing it every Tuesday. I got mine the same

time I sent yours over and it's still in regular use. It's just the perfect temperature this time of year right now for it. It still in regular use yep it's just the perfect temperature this time of year right now for it's not too much it's not too little and it accommodates my ever expanding sizes you know because they cut my I seem to just fluctuate in size all the time

now and it sticks with me I appreciate that too