LINUX Unplugged - 429: Starlink's Linux Secrets

Episode Date: October 27, 2021

We attempt a live production over Starlink, and dig into the secrets of this giant Linux network in space.

This episode's probably going to sound like hell because there's really no way to remove rain from a recording like this because it's a variable rain. We are in just the middle of what they call a bomb cyclone. We're actually at the tail end of it now. And it has been dumping just for days. We've had a literal atmospheric river over the Pacific Northwest for days. There's no way for me to record without some kind of weather sound, especially because I'm out in the woods. And it's kind of a special out in the woods trip too. Hey, Chris, did you do a rain dance at all? My podcast dance and my rain dance are kind

of the same dance. I'm not very good at it. And I think that's what that must've been what happened. I think it might be time for you to try the other one. Hello, friends, and welcome back to your weekly Linux talk show. My name is Chris. My name is Wes. And my name is Brent. Hello, gentlemen. This episode is brought to you by A Cloud Guru,

the leader in learning for the cloud, Linux, and other modern tech skills. Hundreds of courses, thousands of hands-on labs. Get certified. Get hired. Get learning at acloudguru.com. Well, this show is going to be an interesting one. It could just be a total disaster, and we are all going to find out together live because there's a lot of technology that's brand new layered on top of this episode. Starting in front of me, we've got a Fedora 35 workstation

wired up with pipe wire that is routing audio between myself, Wes, Brent, the mumble room, OBS, all of it happening in a rather fragile setup. But what's even more exciting, that is connected to a Starlink dish, which is broadcasting this to the world via space. And the entire pipeline from my broadcast machine to the dish, to the satellites, to the ground stations that receive the signal, to even the systems that bridge

the ground systems to the internet all run Linux. I had a great opportunity to learn a little bit about how Starlink is set up. Just scratch the surface kind of details, but fascinating nonetheless. So we're going to share those Linux details about Starlink today, and I'm going to give you a review of my impressions. I have been out in the woods during this crazy storm. This today will be the fourth day. I got out here and of course had to find the right spot. Took me a little while. I had to

navigate around because I needed a good shot of the sky. I got Starlink going and I've been monitoring everything from its performance, its latency, even the power draw because I'm running all of this off grid. So we're going to talk about that in the show today. Of course, we've got some community news, some Ubuntu concerns we want to share with you, some pics, some feedback, and a lot more. So before we go any further, let's say time-appropriate greetings to our virtual lug.

Hello, Mumble Room. Hello. Hello. I don't have the little counter for Mumble, so I actually don't have the count today, but thank you for making it. Welcome in over Starlink. Please excuse any kind of weird hiccups, because not only is there a storm going on right now, and I'm broadcasting from space, I'm pushing this thing. We have two different remote audio feeds for Wes and Brent. We have the audio feed for Mumble,

plus we have every device in my RV that's connecting and communicating to the internet, and I'm live streaming to multiple endpoints via OBS. So you can just imagine that is a lot of send and receive happening over this dish right now. It's a very precarious setup, not to mention the software is technically in an unreleased state still. So please excuse any oddities, but this is a real try it. We're doing this by the seat of our pants episode to really put the technology in production and see how it holds up. So it's going to be a real make it or break it kind of thing. And that in itself will be part of the review. But we recently had a review on the show and Brent West and I were talking about this after the show. And we felt like maybe something didn't go very well. We didn't handle it quite right. Maybe actually, maybe that's not

the right way to put it. I think the right way to put it is we didn't have our eye on something. Yeah. Chris, you mentioned that you wanted to make good on how we touched on it last time. Yeah, this is awkward because we looked at the upcoming, back then, this was weeks ago, we looked at the upcoming Ubuntu release and we thought 21.10 looks pretty standard on the main version. And we just thought, okay, it's not a big release. It's got a few notable things, but it's not, this standard version isn't blowing us away. And as you might recall, we therefore decided that maybe we'd give the flavors a shot for this release. Chris was over on Mate, and I chose Kubuntu. Yeah, and that was nice, you know? I think we liked both of them quite a bit. Good versions

of the desktop environments for both of them as well. I think you and I both liked that quite a bit. But frankly, we didn't really pay much attention to the main release. I had spun it up briefly, tried it, didn't really pay much attention other than just to check out the GNOME experience. And then after the show, I had an opportunity to use it a little more, just not really even in a review mode, just using the main version of Ubuntu. And I realized we missed an opportunity, but like kind of in a bad way. And this isn't an Ubuntu bashing session because I love Ubuntu and I'm very grateful for the work Canonical does there. But it is out of all of the versions of Ubuntu that I've ever reviewed, which is literally all of them. It's not even like a, it's not even like

a humble break. It's just a fact. I've reviewed literally every release of Ubuntu ever in some podcast at some point. And so I've spent time with each one of them, comparing and contrasting them to the previous versions. And 2110 disappoints. And I'm worried about a general fade here that we're seeing. And this isn't a bash session. This is like, as a community, let's align our expectations. And I want to chat this with you guys, just sort of hash this out here in a moment. But let me tell you what I'm seeing, okay? I'd like to know if people in the mobile room are seeing some of this stuff too, so tag me in a bit. But to start with what I've seen, the main version of Ubuntu is just

on the surface, it feels a little underwhelming. The GNOME 40 integration, which we waited for, still feels a bit unfinished, to be honest with you. Beyond the dock, which they've added back in there, and the Yaru theme, there doesn't really seem to be much integration that's happened that really kind of warranted punting the release like this of GNOME. And the Yaru theme is actually created by a community outside of Canonical. So that isn't, I mean, I'm sure there's some work inside Canonical to coordinate that. And that's awesome. Thank you for doing that. But it seems like even like the biggest

piece for GNOME 40, which was the theme, wasn't really done by Canonical staff. And it just seems like a real minimum effort kind of thing, Wes. Yeah, you're just kind of wondering how much work was really spent here. And what does that say about where Canonical's focus is? Yeah, it's like maybe not enough staff kind of a thing. It was rough, too, because I didn't really appreciate the difference between the DeB and the Snap of Firefox in the flavors, because the flavors chose to, like in the case of Kubuntu, just ship the DeB. But using it for a bit, I was actually pretty disappointed in the implementation, because the flavors chose to, like in the case of Kubuntu, just ship the deb. But using it for a bit, I was actually pretty disappointed in the implementation.

And this isn't a snap bash either, because I think universal software is great. And I think you were pretty hopeful, right? I mean, we kind of touched on how both Canonical and Mozilla had put some work in here to try to make Firefox as a snap a good experience, but that's why it's important to touch on it. We did cover that aspect of it, but the actual usage of it was disappointing.

The launch time is slower. Even on relatively high-performance systems, it takes longer to launch than you'd think. And the stability was less than ideal. So we've also had emails in that have told us that certain hardware products they use for their company login systems don't work with the snap. That's kind of unfortunate. But for me, before Starlink, I was on LTE and I'm downloading an ISO image and I'm sitting there for 45 minutes and I'm watching

this slowly trickle over my cellular connection as the providers will allow me to have it. And just desperate for it to finish because I've got to go. I don't have a lot more time. Five minutes left on this download. I'm doing nothing in Firefox. It's sitting on the new tab page and I have the download menu open so I can watch the progress of the download. And I haven't touched the machine. I'm even not even, I'm chatting on a different computer. I'm just watching the Firefox download like you should never do, you know, watching a pot try to boil kind of thing. Five minutes left, no one's touching the machine, and Firefox just Insta-crashes.

Just boom, gone. And of course, I could resume the download, but it went back like 40%, so it added like another 25 minutes to the download, and I couldn't make any sense of it. And that's just the worst experience. And the thing that really got me upset was I realized, holy shit, this would have been a, just a, they could have just accomplished the same exact goal if they just use the flat pack and that wouldn't have been a worse user experience. And that's really frustrating

because I do appreciate what they're trying to accomplish. And I don't know if the flat pack has the same collaboration that the snap does. It may very well may not. I don't know, but I do know it would have been a better user experience for me because I've had experience with that and it launches right away and it doesn't crash on me. And just that realization that they chose to do this snap effort and they've created a worse experience for me now when the flat pack could have worked even though it's not their tech i appreciate that but it could have worked and it would have been a better experience and it's it's those kinds of little compromises that i think

have become pervasive throughout the ubuntu experience gnome 40 instead of 41 and what we got was an updated theme and a doc and it it's not really feeling all of that complete. But beyond just my experience, when I look out into the internet to see what others is, other than the trashing it's getting on YouTube, the real ones seem to be there was a major data loss bug in the upgrade process. For users of ZFS, surprise, surprise. Hey, that's me some people when they upgraded to 2110 had their zfs file system irreparably corrupted it was noted in the release notes but only after

2 808 words and then the internal response perhaps even more damning, was rather messy when you look at the history of the bug. It seems initially, from what I can tell, it got assigned to a staffer who didn't even work at Canonical. And then it got moved around and it did eventually get fixed. But this is a freaking data loss bug. And it just seemed like nobody was behind the wheel over there. I'm hurting over this because Ubuntu was a shining beacon on the hill of desktop usability and

what 2110 represents is boring at best and disappointing at worst. And I'm not sure what that signifies, but I have a sense it means we need to realign our expectations

of what we are getting as a desktop product out of Canonical. So here's my question to the class, and, Bran, I'd like to start with you, and then I'd like to open up to the mumble room, and then Wes and I will close it up. But does Canonical need to be doing more? Is the sort of minimum viable desktop that is usable by OEMs, that can be rebased with flavors, that provides you a fairly consistent experience, is that enough?

Because maybe what's really Ubuntu's advantage now is vendor application support and community support, and Canonical doesn't necessarily need to be pushing the edge with Pipewire and GNOME 41 and dedicating more staff to this. Brent, what do you think? Is it me that's the problem here? Well, I don't think Canonical is necessarily interested in being like the cutting edge of technology anymore for desktop users on a whole. They are doing some cool things to, you know, try to push the envelope. But I wonder, you know, Popey has come on here so many times in the past saying that the number of users of sort of the vanilla Ubuntu desktop is really impressive.

I don't think he ever shared numbers, but I just remember sort of cementing that into my mind that, oh yeah, it's way more than we probably think. So to see that desktop experience start to disappoint in many more ways than one, and to do some experiments like the Firefox Snap, but clearly not to have tested it enough, it feels, like you said, really unfortunate. And I have tons of respect for people on the Ubuntu desktop team. We know a lot of them. And I think they're going through a hard time, and it shows. And I think that's really unfortunate, really. Minimac, this is the time right now with this release

to work out these kind of kinks, I suppose. Yeah, I'm a long-time Ubuntu user, and the last version before the next LTS was always the version where they introduced the big system changes to be able to test them. So I was quite amazed that Pipewire didn't really make it into the default distribution. It is available. You can switch to it, but it's not enabled as default.

And when it comes to GNOME, I don't know. There are a lot of changes coming to GNOME, and I have the feeling that they haven't really decided what version of GNOME they will use in 2204. So a lot of thinking and work going on there, I think. So, Neil, I think part of this is there's two different users of Ubuntu. There's the LTS users, there's the interim users.

But Minimac touches on a good point there. There isn't a lot that's being tried in this release. We're not going to get, we're probably not going to get GNOME 41 or 42 in the LTS version. We might not get Pipewire by default. Wayland probably won't land by default. More and more things are kind of falling behind, and I worry that ultimately will make the LTS desktop less appealing. What do you think of that? Wayland by default actually happened in 2104. So props to them. They actually did finally start doing it. But the larger point, I think we've kind of forgotten,

is that Ubuntu wasn't about LTS or fast moving or any of those things. What Ubuntu started out as, and I think this release, as it is, kind of exemplifies it better than any other release in the past, is it is taking a snapshot of Debian SID and rebranding it and cutting it for a release. Now, I'm not trivializing that. Saying that as a simple statement, that doesn't make it a simple task. It is very difficult to do that kind of stuff. But that is essentially what the original goal of Ubuntu was, was to take Debian and throw some spit shine on it, replace some of the creaky Perl stuff with more useful user intuitive Python tools, and do a regular cadence of releases so people could see how the

Debian distribution was evolving because Debian releases very slowly. This is the goal. And I think when you look at what's going on in Ubuntu 21.10, it shows, right? Like it's taking what's in Debian SID, it is applying the Ubuntu Delta of branding and other customizations, and it is shipping that. The Firefox Snap versus Flatpak, you know, that's a different, different discussion for another time. But like, I think people are going to be increasingly disappointed in Ubuntu as it goes back and goes to its roots of being essentially that. And in the past, they were a lot more they deviated a lot more from that during the unity days and the early days around, you know, investing in infrastructure stuff and things like that. that at this point, it is just enough desktop built on top with, you know, some spit and shine and customizations to reflect the Ubuntu brand to ship and deliver. That's what it is.

Wes, I like to look at a lot of these things through the lens of the enterprise market and the server market. And I wonder if maybe this isn't just a clever strategy that leaves enthusiasts like me kind of disappointed, but is actually a keen move on Canonical's part. Like they're building a product that still brings value to their server product because it's a platform developers can use to make sure applications work great on Ubuntu without 200 people working on staff to build this thing.

Is that maybe the appropriate way to look at this? How does it serve the server market? Or do you think broader than that? Well, I mean, you have to at least consider that, right? I mean, resources get allocated kind of at the end of the day where it makes sense for the business. And it's just kind of strange to see a shift where I always felt that with the Ubuntu desktop,

okay, it might not be the shiniest, it might not have as new things as Fedora or what I could get in Arch and Assemble myself, but I could get really close to something approaching the state of the art in the most user-friendly or close to that that you could get in the Linux world. And having that slip, yeah, it does seem like it might echo where some of the leadership and the resources are being directed at Canonical. And I don't think they're slipping away

on the server side of things. They've got to just be putting those resources over there. Well, and can you really blame them if a lot of the technology stack now is at least sponsored in part by Red Hat. They're probably their largest competitor, right? The GNOME stack, the SystemD stack, I mean, they would argue, oh, we're not Red Hat projects, but they're influenced and they're sponsored.

I mean, that seems maybe semi-fair to say. And so at least I could see that perspective from Canonical, and I could see kind of the idea of just sort of, all right, we'll just go with the flow. And I mean, they've been burned, right? I mean, there's that whole mobile saga. There's Mirror. There's Unity. There's a lot of interesting things that they've tried and have advanced.

And maybe there's a feeling of not quite getting enough out of all those efforts to really keep putting in that kind of above and beyond labor. Yeah. Maybe I've got it all wrong. We'll see. You know, at the end of the day, it still makes a pretty damn good desktop, especially when you buy it with a piece of hardware. And what really struck me was just, I guess I expected more from that Firefox Snap experience. I was pretty shocked when not only was it having problems for me, but if you look at just about every review out there, there's complaints about

it. But then we started getting emails from people that said, you know, I can't get logged in now to my company's web app when I work from home. I thought, well, that's why you don't upgrade your work machine. That's why you stick with the LTS on the work machine. But, you. But at the end of the day, it's pretty easy. You can still get the deb of it, of course, or you can just go get the tar file from Mozilla. Or as you say, the Flatpak. Yeah, that works pretty good. So something else I use as a Flatpak is Element. And you might recall Element.

They're the makers of probably the most popular Matrix chat app. And this next story almost seems like too good to be true for a guy like me. the makers of probably the most popular Matrix chat app. And this next story almost seems like too good to be true for a guy like me. The team behind Element today announced Element One. They're pitching it as a complete messaging package that bridges WhatsApp, Signal, and Telegram via their hosted service for just $5 a month. Whoa. I mean, okay. You could make all this stuff work before, right? Obviously, you can host your own bridges,

or if you're an enterprise or someone paying for managing a whole server and team accounts, that kind of thing, this was possible. But this is kind of a new offering just to users like you and me. Now, they say that there's unlimited usage with some, quote, fair usage limits, but for their part, hopefully, these are, quote, generous. Yeah, it seems like maybe that is just a ban, like automated spamming and stuff. So if you trigger like some serious thresholds, then it's no longer unlimited, but they didn't really come

out with any specifics there. You know, Chris, when you brought this to my attention, I thought it was kind of amazing and something I've been waiting for for a long time. I don't know if you feel this way, but having eight different chat applications feels a little daunting to me. And how it brings together stuff that we already use is great, especially from a place that we seem to have been loving for the last little bit. So it feels like there's a lot of promise there. And for someone like me who doesn't want to tinker to try to get all this stuff working, it's a nice, reasonable, at least for me,

fee to put all the hard parts together for you. But I did see one caveat about the end-to-end encryption. It seems like for now, the chat services you bridge to has a broken end to end connection. So I know they're working on that. As we know, end to end encryption is not the easiest thing, especially with various services. So I'm interested to see where that'll come. They do write, the bridges operate in elements trusted EMS environment with no content scanning or data mining, but currently bridged conversations are not stored and encrypted in Matrix. So that's something to consider and to watch out for in the future.

I appreciate them making that disclosure up front. And that is a pretty big limiting factor for a lot of people, I would imagine. I'm not a Signal user, but if I used Signal, I would imagine I'd be doing so specifically for the security aspects of it, and that would concern me. I am a heavy Telegram user, and I have never used Telegram for its encryption or privacy. Now, it's just a great chat client. That's the thing that's appealing, right? Yeah, totally. I mean, I expect the encryption to be compromised or will be compromised one day. I'd love it not to be. I expect friends to keep our conversations private, but I do not rely on the encryption.

There's also another aspect here that's just, it's interesting to see how, you know, EMS and Element and the folks behind Element and Matrix are trying to, you know, find paths towards sustainability. And maybe this kind of level is enough, right? That $5 a month might be enough to actually entice people to do it. Is that too much? It's close. It's pretty close.

I think it's like the right price for a user. I think that's why they priced it there. It is the right price for something I'd be willing to try. However, you and I have hands-on experience running a matrix server. And I don't think if we charge the listeners, if we charge each user of the matrix server $5 a month, I don't think it would cover the run cost of the matrix server. And we have a smaller matrix server than they do. You know, that may be true. I've got to hope they're better admins than we are, though. Linode.com slash unplugged.

Go there to get $100 in 60-day credit on a new account, and you go there to support this here show. Linode's where you can host everything. There's 30% to 50% less than the major hypervisors out there that are just going to try to lock you into their platforms, and they've been around for 18 years before AWS even started, so they're really, really dialed in on how to do this. That's why I run all kinds

of stuff up on Linode. Every single show we do is powered by a show doc. Everything we're going to talk about, the links we collect, if somebody contributes a new link suggestion during the show, we collect it all in our show doc. It's like the master doc for our show. And every single one of those is composed on a Linode. We use HedgeDoc hosted up on Linode to get collaborative real-time Markdown editing. It's like Google Docs, but for Markdown and modern and clean looking, and you can host it yourself. And we love running this kind of stuff on Linode. These tools that make producing these shows possible. And you could do for your projects, for your business, or maybe you just want a place to put a blog online.

It all is so straightforward with Linode. And with that $100, that's them saying, we really want you to try this. It's not just like a fly-by-night thing. Like $100 is really something you can use to try out the service. That's confidence on Linode's part. And their mission remains unchanged

since the day they began. Make cloud computing simple, affordable, and accessible to all. That's part. And their mission remains unchanged since the day they began. Make cloud computing simple, affordable, and accessible to all. That's it. It's easy to understand. It's straightforward. And I think it jives a lot with the mission of Linux. They also are really good about staying in tune with what the community likes because they're part of the community, right? They got into this stuff because they're passionate. That's why they started Linode is because they saw where Linux was going. They played with the technology. I can really click with that. I totally understand following your passion and starting a business around that. And I think it's been their secret

sauce. You see it in the documentation, the things that they write up, like how to install custom Linux distros or host Plex using object storage for essentially unlimited Plex storage, all these kinds of things that they don't have to do, but they focus on that stuff because they want to include everybody. Not just the people like me that are trying to run their business infrastructure, but anyone. And the pricing reflects that too. And then the additional services on top of that,

like cloud firewalls, DDoS protection, VLAN support, and S3 compatible object storage, and a lot more. But really, just go try it for yourself. Go use that $100. Go see what you can do. Go kick the tires, as they used to say. Just go to linode.com slash unplug. Go there to support the show.

It's like saying, hey, thank you, Linode, for supporting my favorite Linux podcast, and you get $100 if you sign up with a new account for 60 days. Try out their super-fast networking. Maybe even play around with some of the more advanced features like their Kubernetes or Terraform support. Or just set yourself up a box to do like an SSH jump host like I've done. This is something too I recommend. Give yourself like a support system in the cloud. It's got your tools. Maybe you have some jump hosts set up to there. Something that if you get to that box, you can support your friends and family or maybe your work machine or whatever it is. Oh man, it makes life great. Go try it out. Linode.com slash unplugged.

This is actually the perfect time other than the storm, but the storm has been going on for days to record this because if we were, if we would have been going on for days, to record this. Because if we would have recorded this on the first day I set up Starlink, I'd be just gushing, right? I'd be elated. I would be just going on about how amazing it is. And to their credit, Starlink couldn't be easier to set up. Everything you need comes in the box already wired together. So you don't have to figure out even where the ethernet cables plug in. It's already all plugged in and they have color coded things. So the dish cable is gray that goes into the gray end of the POE injector. The land side is white. So the plugs are white. The cables are white. Everything's

color matched in that regard to make it super straightforward so i set up their router and i set up their dish and for about 15 minutes it sort of sorted itself out and i was getting like 40 megabits i'm like oh you know 40 megabits then once it sort of had settled i did another speed test and i got 240 megabits on that test. Whoa. Remarkable. Yeah. And you know, you have to get a good clear shot of the sky, but their app makes this really easy to do. You fire up the app and you say, I want to check this area. And they have like an augmented reality checker that gives you a really clear indicator of where you still need to scan. Then it maps out the whole area. And then it's got some data models on the app. And if it's your first time ever running it, it actually takes a little bit to generate it.

But once you've run it once, it generates much faster. And it analyzes the image you just captured and tells you if the Starlink dish will have any trouble connecting to the sky. Oh, that's great. Can you imagine? Like you're trying to set up in a new area, and you've got no idea if it's going to work or not. It's so awesome.

That app, they nailed that. And I noticed that the router that they ship, which is a nice slick-looking custom unit, and I later found out, we'll get into this more, but it uses a custom version of OpenWRT. So it's a little Linux box. But this is a good point, right? This is ISP, but of course,

you know, us Linux nerds here, we want to know the pieces. Is there bridge mode? How does it actually work to get you an IP and talk into Starlink? Oh, gosh, it's interesting. And it's so cool. So they have an aux port. Obviously, it comes with Wi-Fi, and it's a little AP. But they got an aux port on this thing that you can plug your router into and get an Ethernet connection.

But it turns out, and I really hope they never change this, it goes a step further. Impressively, the little dishy McDish face that connects up to space actually has a very powerful custom SpaceX silicon Linux box in there. And everything you need to get connected to space is in the dish. If you have a solid LAN setup, like probably a lot of our listeners do, with a router you already like and Wi-Fi access points that you already want to use, you can take the router out of the picture altogether. Oh, that's great. And so I have gone directly from the back of the dish into my peplink router. And now I manage it like I do my three LTE connections. It's just one of the connections now in my setup. I am so grateful that they, because that router is nice. And if you didn't have something, you know, you were setting up from scratch, that'd probably be one of the best ISP routers

you're going to ever get. It's powerful. It's got open WRT on there, and it's got a good little powerful access point chip in it. But for me, I've invested years in a LAN setup that is very custom tailored to my mobile needs. And so the ability to go directly from the back of the dish into the PoE injector and then into my router means that the work I've invested into this setup to allow me to be mobile and work from the road is not wasted. It's just applied to the Starlink now. And I really am very happy about that. And I'm extremely impressed by the capabilities of this little dish, because it's passing through the DHCP information and everything like that. And it is using, I believe, that Linux system in there is using their patched version of U-Boot, which they've put all of this

stuff upstream. We'll have a link to their Space Expl github that has all of their patches for open wrt it has their patches for the precision time protocol that they're using on the satellites which is an open source time protocol they're using that on the linux boxes in space to keep time they've done some patches that's upstream on their github it's really great um the full stack from the router they ship you, the dish itself, the satellite it connects to, the ground stations, the ISP peer stations are all Linux from top to bottom. And of course, the development systems

at Starlink's office in SpaceX, also Linux. And get this, Wes, I got a few details about this and you can suss out even more stuff in the AMAs I'll have linked in the show notes, but, um, they deploy a full software stack refresh, a full, like the Linux OS and everything to all of the satellites in orbit once a week. Oh, wow. That's, that's impressive. Could you imagine? And they're using their own custom chips up on the satellites. They have a patched version of the Linux kernel, so it does preempt RT because everything needs to be real time up in space. Indeed.

And so they're submitting their patches up for that, too. The connection itself is pretty good and solid. I'd say on average I'm seeing, with good clearance, about 140, 150 megabits. I get ping times between 20 milliseconds and 45, 50 milliseconds. I'm getting 54 right now. But for me, this enabled a new kind of off-grid working

that I've been doing now for the last few days. Every show we've ever done on the road, remote, if we've been at System76 with an Ethernet connection, or if I've been in some hellhole in Tucson, every show we did remotely was mixed through our studio. I'd come in with an Opus Audio feed. Wes comes in with an Opus feed. The mumble room comes in. We mix it all together in the studio. I'd come in with an Opus audio feed. Wes comes in with an Opus feed. The mumble room comes in. We mix it all together in the studio. It sends it out to the OBS machine where we can record the members only version of the show and we can live stream it out to the internet. And the real

compromise we were making there was all I really had was a low bit rate audio stream back to the studio because I had to compensate for LTE. And if I needed to control something or check in on the mumble room, I'd have to use something like Rust Desk to remote view the desktop in the studio, which of course would then take bandwidth away from my VoIP session, lowering the quality and making me drop packets. And it's just, it was a mess. And Rust Desk is great. Don't get us wrong, but boy, it's hard to do that live in a show remotely on LTE and a big old desktop screen. Right.

And there's just so much, only so much you can expect with LTE, and I have a pretty optimal setup with three different LTE connections that I have bonded over a VPN that goes to three different servers at Linode running a custom VPN OS that combines it back together.

Like, I am doing best-case scenario LTE compensation, and it's still limited in what it can achieve. But with the bandwidth availability of Starlink, I've done Coder Radio, and now Linux Unplugged completely live. Brent and Wes are coming into my RV that's parked out here off-grid in the woods. The Mumble Room's coming in here. I'm mixing everything live. Brent and Wes are coming into my RV that's parked out here off grid in the woods. The mumble room's coming in here. I'm mixing everything live. The live stream is streaming out over OBS. The

member recording is happening here in the RV for the first time ever. And you guys know I've done a ton of remote shows. For the first time ever, I'm actually producing the full show remotely. The studio could be powered off right now. This is a remarkable difference in what is capable by going from a high-end, I'm talking very high-end, LTE setup to Starlink. And Starlink right now is $99 a month after you get the $400 and something gear. It's pretty impressive, the change that this makes. It's sort of a, for remote workers, it's a paradigm the change that this makes it's it's sort of a for remote workers it's it's it's a paradigm shift in performance and you combine that with an impressive power draw when you look at all of the reviews online right now they're all a little old and they talk about

how the dish is pulling around 100 watts of power. For me, that would be a huge deal. I would need to find a supplemental source of power, or I would need to increase the overall power capacity, battery capacity of Lady Jupes. 100 watts consistently, 24-7, is a huge deal. I mean, right now, as I'm recording the show, I actually have this up. My overall power draw for the entire live broadcast setup with everything pretty much turned off in the RV except for my Raspberry Pis is around 215 watts. I can probably make it like this for quite a while. But if I added another 100 watts to that, it'd really be a problem. It's no joke, especially during this storm where there's really no solar to be had.

But I am impressed because through a series of what I think are probably software improvements, maybe some hardware revisions, they've gotten the power drawdown from 100, 110 watts to around an average of 35 watts, especially when you take out the router, you actually lose, you drop like another eight watts. So it's even, it's using less power than that, really. I've seen it peak as high as 75, but this is only for a short period of time. And on average, it's around 35 watts. What I did is I grabbed a smart plug that does power monitoring that feeds

statistics into my home assistant instance. So I've been monitoring the power draw of the dish from the moment I powered it up. And so I've got historical data now over the last four days that shows me its overall power draw and 75 watts is as high as it got. It really seems like the team just made some impressive improvements to the power use. And an interesting note is cold weather will bring the power draw up of this dish so it can heat itself up to melt ice. And a lot of the YouTube reviewers you'll see out there, they're going to tell you it's got heating elements in it and that it turns these heating elements on to melt the ice. in it and that it turns these heating elements on to melt the ice. But in actuality, what it really is, is that the software just detunes the radios to run really inefficient and their inefficiency generates heat.

I mean, yeah, right. The power's got to go somewhere. Right. That's great, though. And I mean, simpler, too. You don't have to ship these extra heating elements and extra parts. Yeah.

It's a really clever little piece of hardware, that little dish. And it self-actuates, and it moves to track the satellites and find them. It's really something to watch. And it seems that there are a series of projects going on right now to pull the statistics that the dish tracks.

Because the dish is tracking your speed your latency your packet loss and that's all available in the app if you bypass the router and you hook directly to the dish the app doesn't work now there's two solutions i love that starlink makes this available because it lets you geek out so if you want to bypass the router but you still want to pull up the app and get the stats, you have to add a static route. And I'll have the details in the show notes. And it's basically a route that you put on your router that says, when you go to this 192.168.100.1 machine, route it to this WAN interface. And then that will forward that to the WAN interface on the dish, which is set up to listen at 192.168.100.1, and it will retrieve the information for the app, and you can then use it with your own LAN.

But the other route to go is there is a series of Python tools called Starlink GRPC tools, because they're using GRPC on the DISH, and it can pull the stats remotely and store them locally. And our buddy Jeff Geerling has a Docker setup that will pull in the gRPC tools and Grafana and then input that information directly into Grafana so you can have historical information of how the dish is performing right there on your Linux box. Oh, yes. It's nice to have these kinds of tools available. And it's so rad that Starlink didn't lock this stuff down because it makes it so much more usable in my existing environment that I've spent a lot of time curating to work for me on the road. So I hope they never prevent this kind of flexibility. It enables a new kind of remote

living. It's like good for my psyche to get out here in the woods. Like when I'm out here, I feel so much better. I'm not stressing about financial stuff or the state of world affairs. I'm like living in a little cabin out in the woods. And now I have internet that actually allows me to work out here. So in the past, I'd drive into the studio when I was out in the woods. I'd just jump in the car and take like a two hour drive early in the morning, work out of the studio all day, and then drive back home for two hours to my RV out in the woods. And that's fine, but it's, it's a silly use of gas, especially with the cost of fuel right now. And now I can just stay out here and it, it actually improves my, my, my state of mind, my mood. It's such a special place that I was just

telling Brent before the show, like, I'm, I'm not sure if I want to bring you out here. Like, you know what I mean? Like, I want to show it to you, but I don't know if I actually want to take you here because it's so special to me. But you got to show off Starlink in person. I want to see this thing. This might apply to my setup too. And something tells me Elon knows where it is now. that up too. And something tells me Elon knows where it is now. Yeah, he's probably watching. You know, I had kind of an emotional moment. It was silly. Like, cause you know, the speed tests were like one thing. Oh, okay, great. You know, 240 megabits. Wow. 150 megabits. Wow. That's great. I'd never see those speeds. But what actually hit me a lot harder than any speed test that i did was just seeing vs code download and coming down at like 11 megabytes a second i've never really experienced an internet

connection like that at home before and so you know when everything i do is online and every every app every device like everything's like using the internet and the quality of your internet affects so much just even how your devices perform because of their damn insistence on connectivity. It really felt like a shift for me. And I give it the full stamp. Even having the caveats of it's not fully out of beta yet. You know, some people experience sometimes like up to eight minutes of downtime a day. I think some of that might really be based on your

sky clearance because having 100% clearance here, I've really had fantastic connectivity. I think over the last four days, I've maybe had a total of 35 seconds of downtime last time I checked the app. And my average ping, because of course I've been running a smoke ping to try it out here on my machine is around 35 milliseconds back to Google and stuff like that. Today, it's actually kind of been the worst. So we're kind of trying this on the worst case scenario and it's still holding up. Chris, I'm curious about how it's working. Like you're testing all this in what you said was like four days of rainy weather. Do you think it's going to get better or is it just not affected by sort of the clouds and the such? That'll be interesting to see.

It seems rather unaffected by it. I think snow would be a bigger issue, especially a heavy snowstorm. And I was concerned it would get blown over. We were initially on the coast where I was first going to set it up. And the wind speeds were getting up to like 60 miles per hour out on the coast. So we came inland to our spot in the woods, which is just at the foot of the Cascade. So if it wasn't so gray, I'd be looking at the Cascade Mountains right now. And the wind wasn't as bad here, but we had this big channel of wind that came down right into where where we're kind of in this groove of around

mountains. So it sort of channeled the wind in here and it hit us real hard for like a half hour. And every tree around us had like all of its leaves blown off. In fact, I think my solar panels are partially covered in leaves right now. And I sat there and watched the dishy to see if it would get blown over because it's probably 25 miles per hour wind with a lot of like, you know, things debris flying through the wind or through the air. But it just stayed. It didn't it didn't even flinch. It just seems really sturdy. It looks kind of like it'd be a top heavy thing, but they give it a pretty sturdy base.

So it seems to have held its ground. I mean, yeah, I storm tested it should be ready for anything, right? Yeah, I think maybe the snow, maybe the ice. We'll see. But I'm kind of just, I'm struck by when I started looking into what Starlink and what SpaceX is doing here, I realized this was way more complex and a way bigger deal than I think as a society we've really caught up to. SpaceX is about to switch over to using their Starship rocket, the big boy that they want to take to Mars. And that's going to increase their deployment capacity from something like 40 to 60

satellites at a time to something like 400 satellites at a time. Oh, wow. And just in order of magnitude. Yeah. And the capacity, more capacity means faster speeds. It means more users. It's like they can add capacity by putting more of these things up in the sky in a way that like Comcast can't really, they can't just constantly be laying fiber all the time. And there is that element to this. Before I got it, I was giving a lot of thought to how do I feel about a lot more satellites up in space that you're going to be able to see from the ground is it going to be kind of feeling like we've junked up the sky but the reality is is that starlink isn't the only initiative too

like bezos has an initiative to do this um indeed and spacex sees this as part of an interplanetary communications network like they see this as building infrastructure to enable communications to the moon and to Mars. But they also just recently acquired a company that was working on a system like Starlink, but for IoT devices. that could be kind of smaller and didn't necessarily have big dishes, but could still receive these lower powered signals from smaller satellites up in space. And SpaceX just purchased that company. And so they may be also looking at

launching into the IoT space down the road. They also just were granted a few months ago, I think it was back in July, they were requesting, I don't know if it's been granted yet, but they were requesting permission to test a mobile Starlink setup for buses, vans, boats, and airplanes. And just doesn't that make so much sense? Yeah. I mean, you could see how it would be a great service for airplanes. It's probably even

a faster connection than I'm getting. You've been talking really great things about Starlink and your experience, but there's got to be some stuff that didn't go so well or some stuff you're worried about. So what's your least favorite thing about your Starlink experience to date? You know, I think in part it's that it's the better than nothing beta. So things could go down. You don't really know. And so here I am, you know, now I'm kind of becoming dependent upon it. So there's that aspect of it. Also, I'm worried about moving this thing. Yeah. you and I have talked about theft as well. True. I could see at a campground somebody stealing it.

That's why it would be nice to have it on the roof. But they expect you to use this in a service area. And you really shouldn't go beyond that service area. It's supposed to be fixed at your home. And moving it is tricky. And it requires updating your service address and hoping that there isn't overcapacity in that area. And perhaps they will allow you to move it, but that's not what this dish is licensed for by the FCC. And it's not how they've built the system. They've built the system such that you're at a service address and they use those service addresses to figure out where their

biggest demand is at and stuff like that. So it goes into some of their planning, I would assume. But additionally, you could move it like I did to set it up here in the woods. And then say, I want to go back somewhere that's more back in civilization. And I could try to move it back to there. And it could tell me, nope, sorry, overcapacity in this area. So there is that. Additionally, you know, the power draw, it's a lot better than I expected. Absolutely. You know, between 35, 40 watts, 45 Watts on average continuously. But that also has made a pretty big difference on our day to day here. Well, that and having zero solar, but it's, so it's really kind of worst case scenario, but my batteries, you know, they, instead of being in the low seventies or

high sixties in the morning, now they're in the f the 50s the you know the mid to high 50 percent range now when i wake up not the end of the world but it's a lot of power draw and that's a lot of other things that are running that's not just the dish that brings that down that's my fridge and my servers and my networking gear and a lot of little things are running my cpap and noisemakers and so but it makes a difference and it's about five to 10% of our battery draw. And that makes a difference when you're, when you're living for multiple days off, off grid, I'm, I'm now watching power a little more closely. It's like sort of a precious commodity right now. And so it's a downside, but because of the trade-offs, it's kind of worth it. And I'm not going to lie, Brent, as a space enthusiast, who's also

obviously a Linux enthusiast, this really brings two things I like a lot together. So I'll overlook some of that stuff. I'm curious if you've considered adding your automation enthusiast to the situation. Is it possible, first of all, without too much trouble to automate the dish? So for instance, you could turn it off, you know, past midnight or something to save power for eight hours or something. I may actually do that. So that was another reason I put it on a smart plug.

So the dish itself draws power through PoE, power over ethernet. They supply you with a very high-end custom PoE injector because this thing can draw a lot of power if it needs to heat itself. I mean, I think this thing allows for like up to 150, 160 watts or more than that. It's a lot of power for PoE.

So you probably want to use their custom PoE injector. So that is what I plugged into a smart plug. So not only can I monitor the draw of the dish, but I can also turn it off from Home Assistant. And I may very well do that is set some sort of automation. What would be incredible, what would be like next level and very doable because this, these,

those GRPC tools I mentioned, they're all Python tools. Home Assistant's a Python project. It really is just so ripe for the picking to take the data that's on Dishy and make an integration that feeds it into Home Assistant. So then ideally you could do something like, when my average transfer rate drops below this threshold for X amount of time after 11 p.m., switch off the smart plug. That sounds possible, but just all the bits aren't there yet to connect it. Somebody needs to make a Home Assistant integration for Starlink.

That would be amazing. I'm curious too about the downtime from the moment, you know, you turn the power back on until you have a usable connection. It's quicker than I expected. I think it was 30 seconds or so. Gosh, that's actually really super reasonable. Yeah, I was impressed. I really was. I think I, you know, I stepped away for a second, grabbed something, came back to my desk and it was pretty much back up and going. I liked that a lot. recording, everything we messed up, plus a big pre and post show, which is like a whole other show. It's a way to say thank you to our members.

And then if that's not your thing, you like it short and tight, nice and quick, we have that too. We have a second feed that is limited ads, but the same full production. And you can get access to that by going to unplugcore.com and support the show. If you have any feedback, topic ideas, or thoughts on Fedora 35,

which we'll be covering next week, let us know at linuxunplugged.com slash contact. And join our Matrix community at linuxunplugged.com slash matrix. And the LUP plug happens every Sunday. I popped in there for a bit this Sunday since we had delayed the LAN recording. I took the opportunity to say hi to the LUP plug. It was a good time. That's noon Pacific in our Mumble room. Mumble details at linuxunplugged.com slash mumble.

How easy is that? Well, how about some feedback? James has a wormhole-like app that he'd like to share. Hey guys, I want to throw out a cool file sharing app I've started using pretty heavily. It's in the vein of Magic Wormhole, but can be installed with a single binary. It's written in Go. Sorry, Chris. It can actually perform direct peer-to-peer connections as well, and has the ability to resume interrupted

transfers. There's even an Android app for it. It's called Croc. And actually, this is something we've talked about before, but it's been ages. And this seems like a good chance to revisit it because I, for one, didn't know that there was Croc GUI, an Android app on F-Droid. I just might have to try that one out. Yeah, this looks pretty neat. And it looks like the GitHub page was bumped just about 20 days ago, 18 days ago. So it's still alive out there. 20 days ago, 18 days ago, so it's still alive out there. You know, the idea of these kinds of things is you take a command,

you point at a file, and it makes that file immediately available for transmission over your LAN or the Internet. It's like the easy button for sending files around your LAN. Don't bother with secure copy. Don't bother with Samba. Use, in this case, Croc or what we've used at Wormhole before. I like that it does local P2P. I don't know if Wormhole does that. We'll have to look. There may be some good comparisons out there of all of these tools. Hopefully there's one of those awesome GitHub lists that kind of makes that clear. The one thing

I will say is I think Magic Wormhole does still have that social network effect. I was using Croc. I really love how easy they make it to host your own relay, which, you know, if you're exchanging private files, not that it has to travel over it, but just any of the setup of the exchange

you might care about and you can just type Croc relay and be up and going. But when you're actually transferring files with other people, seems to me more people have wormhole installed, at least right now. Fair enough.

Hey, we have a pick before we get out of here. And I know Optimus Gray in our chat room is trying this one out right now. It's the official repository for KDE Connect on iOS. It is finally here. Yes, there is a companion app that works with iOS 15 that lets your iPhone get in on the KDE Connect fun. It's up on test flight right now, which is like the test area for apps sort of outside the app store. And we'll have a link in the show notes. We call that poor man sideloading over here in Android world, Chris. It pretty much is exactly what it is, Wes. It's like sideloading with rails on rails, sideloading with bumpers. Oh, iPhone users,

we put up with so much. But it's really nice to see KDE Connect for iOS now. It's been a long time coming. Yeah, KDE Connect is amazing. And kind of taking it for granted on Android, I didn't realize there wasn't anything on iOS. So here's hoping this makes it to the App Store sometime soon. Yeah, I wonder if this is the kind of thing Apple would allow in the App Store.

That's a good question. Yeah, wouldn't if this is the kind of thing Apple would allow in the App Store. That's a good question. Yeah, wouldn't that be nice? It'd really be nice if they acknowledged that kind of stuff. Oh, boy, I have lots of complaints and thoughts about that, but I'll save that for the Coder Radio show at coder.show, if you haven't checked that out. Also, holla at our friends, A Cloud Guru, on social media. They're just slash A Cloud Guru on social media platforms

like YouTube or Twitter or the, I don't know, the Instagram. I don't know what you do, but if it's a social media, they're just slash a cloud guru on social media platforms like YouTube or Twitter or the, I don't know, the Instagram. I don't know what you do, but if it's a social media, you can probably find them slash a cloud guru. Pow. If you do that Twitter thing, we're at Linux Unplugged and the network is at Jupyter Signal. If you're a big Mastodon user, I guess go to linuxunplugged.com slash contact and tell me why. Tell me, I've been thinking about it, but I don't know, man. I'd rather just go all in on Matrix. And you can find a whole network of podcasts over at jupiterbroadcasting.com, like the self-hosted show, like Linux Action News, like Coda Radio.

You'll probably find more Wes Payne if you go look for it. I mean, there's probably lots of it over there, right? Just a bit. Why not go catch up on some old brunch classics over at extra.show? They're still up over there. Go listen to a little brunch. Spend your morning with Brent. That's what I do when I'm missing him. I just play a brunch episode. Oh, that's so nice of you. If I play your episode with Wes, it's like I'm hanging out with the both of you. Oh, that's what you do when you hide in your secret spot you won't invite us to. You know it's a Wes shrine. You just don't

want to say it on air. You know it's true. Links to everything we talked about today at linuxunplugged.com slash 427? 429! 429! I don't think we're ready for 430. No. No, although it will be our Fedora 35 review episode. We're going to do a Fedora 34 exit interview, and we'll do a

Fedora 35 review. So what I'd like to encourage is go try it out and let us know what you think during the show. So go get Fedora 35. It'll be out hopefully like to encourage is go try it out and let us know what you think during the show. So go get Fedora 35. It'll be out hopefully next week. Maybe not. Depends on the bugs.

Uh, but you know, they release it when it's ready and we encourage that. So no razzing from us. Feel free to join us live. We do it. Rainstorms or not. Sorry if this episode sounded like hell.

I'm sure Joe was trying his best, probably swearing me. Just I can picture it now, but no he tried. Just no he tried. There's nothing you can do. It's a bomb cyclone. But we do it live regardless. JBLive.tv on a Tuesday at noon Pacific

3pm Eastern. Thanks so much for joining us on this week's episode of Your Unplugged Program. Even if you're not here live, we still appreciate the hell out of you.m. Eastern. Thanks so much for joining us on this week's episode of Your Unplugged Program. Even if you're not here live, we still appreciate the hell out of you. Thanks for subscribing. Thanks for downloading. Thanks for reviewing the show, all that kind of stuff. And we'll see you right back here next

Tuesday! Thank you. Let's go find a title, jbtitles.com. I don't know, maybe it'll sound all right. How did it sound in the LAN so far? Did it sound okay, Wes? Yeah, you know, it comes and goes, so there's definitely a few moments where it'll be clear, including the land intro where you're like, oh yeah, yeah, nope. It's definitely raining behind Chris. Yeah. When you and I were doing land, it was raining hard. No kidding. We'd stop like between a story or something to like check something and the rain

would stop. And then right as we got started again, just pouring down rain. That's why I don't do this very often. But, you know, I wanted to really, truly put Starlink through its paces and really see if this was legit or if it was hype. And, I mean, here we are now. This is the second live show where I'm really pushing this thing. We're talking remote audio streams to Wes and Brent to the mumble room. I've got a couple of computers. We're talking remote audio streams to Wes and Brent to the mumble room.

I've got a couple of computers. I've got like 60 devices on my LAN that are probably chirping to the internet all the time. I've got my web browser, my chat programs going. And of course, we're live streaming over OBS. And it is raining like the dickens. And it's really holding up right now. My ping time to Google is 35.2 milliseconds right now while I'm doing all of this, while the connection is theoretically fairly slammed. So I thought what I would do is pull up the real-time power usage for the show and see how much power the dish used while

we were doing the show. Because I have discovered through monitoring this thing that it uses more power to transmit than it does to receive. And if you think about it, to receive is just sitting there listening for signals from space, right? But to send, it actually has to send that up into space. So it uses more power when you're transmitting more. Yeah, exactly. Here we are. Right now we're at 39 watts in real time. And let's see, around 9 a.m. when I first started really cranking away for the day,

56 watts. During the show, around 47, 46, 46, 35 watts right now. It's got down to 35 watts while we're doing the show. Wow. That's amazing. Think about that. For a Linux box that's communicating with space. With freaking space.