LINUX Unplugged - 369: Double Data Rate Trouble

Episode Date: September 2, 2020

The Raspberry Pi might be getting a small software fix that makes a big performance improvement. Plus, we attempt to combine two internet connections with Linux live from the woods! Chapters: 0:00 Pre-Show 1:07 Intro 1:55 SPONSOR: A Cloud Guru 2:35 Lenovo Linux Laptops 11:21 Raspberry Pi Storage Speedup 13:31 SPONSOR: Linode 17:45 Linux Unplugged Core Contributors 18:58 Fedora 33 Bug-a-Thon 20:55 Using Two Internet Connections in Linux 25:11 Policy Routing 28:32 Net-ISP-Balance 31:46 Diving into Policy Routing 33:42 Speedify 39:35 Feedback 40:32 Pick: tunshell 43:16 Outro 45:46 Post-Show Special Guests: Alan Pope, Brent Gervais, Drew DeVore, and Neal Gompa.

Okay, I've been waiting since we hit the record button. I wanted to ask both of you what your naming conventions are for your home machines on your LAN. And Drew, I want to start with you. I name them after astronauts. So the laptop that I'm using right now is Aldrin. And then I've got, you know, Grissom is my phone. And Level is my tower. That's clever.

That's pretty good. And not too far off from mine. If you want to switch it up, like say you've got some servers off somewhere, you can just choose a different mission. That's pretty good. Okay, Wes, what about you? Star Trek ships, of course. And I always make sure to name my admin account Odo.

Mine feel kind of boring now in retrospect because I don't have a ton of machines on my LAN. I name my machines after the moons of Jupiter, of which there are many. So the computer I'm talking to you on is just really simple IO, just IO. And so it's like one of my favorite host names to connect to because it's very short. Hello, friends, and welcome into yet another fresh episode of your Unplugged program. My name is Chris. My name is Wes.

Hello, Wes, and hello, Drew. Hello. Hello. Gentlemen, we have a fine show today. We solved a problem they said would cost thousands of dollars. This is one of those episodes where we'll often talk about the pros and cons of commercial platforms. But this is one of those things that Linux does that the other OSs can't touch. Wes and I have come up with a couple of clever ways to use a Raspberry Pi to bond multiple LTE connections into either a load balanced failover, or we'll even talk about a way you can completely bond them.

And to sort of prove a point, I am broadcasting from the woods today. So we'll talk about that, plus we have community news and all of that to get into. I want to say thank you to A Cloud Guru. This episode is brought to you by the all-new A Cloud Guru, the leader in learning for cloud, Linux, and other modern tech skills, hundreds of courses, and thousands of hands-on labs hands on labs get certified get hired and get learning at a cloud guru.com and also made possible by our core contributors thank you to our members i'll give you an update on that a little bit in the show as well but also time appropriate greetings to the mumble room i uh i was late on the draw there so i wanted I wanted to get you guys in before we went too far. I appreciate you being good sports about it.

Hello, everybody. Good to talk to you. Let's start off with some community news that I'm pretty elated to talk about, and that is Lenovo has begun rolling out their Fedora Linux laptops, and Ubuntu ones are coming soon. The first models are appearing with Fedora as an option, the first one being the ThinkPad X1 Carbon Gen 8. Not only has it rolled out with Fedora, but it's right there front and center, and it's also the best price. And, best of all, it doesn't come with a whole bunch of bloat.

As to, quote, the Fedora team, Lenovo respects our open source principles. The only software they will have is from the included repositories. No extra fluff. Now, so far, it looks like you'll have to be in the US to get one of these, but fingers crossed that more markets come soon. I love that it's vanilla Fedora. I think that's really great. Gaming on Linux has a great write upup, and Liam writes, we also know from what's already been said to expect Fedora Linux to be rolled out to the ThinkPad P1 Gen 2 and the ThinkPad P53. It seems like Ubuntu is not too far off either, and will probably land on the ThinkPad X1 Carbon Gen 8 soon, and then other systems as well. Love this. This is pretty exciting. This is like the moment, you know, where we start to see real Linux adoption happen at the OEM level. Neil, it's available in other places outside the US,

you're saying? So when I talked to Mark Pearson, who's the Lenovo guy who's managing this relationship with Fedora, it's a matter of getting the web teams for each of the individual regions to update the websites to get them showing up. It is available in the web portals for the United States and Canada today. It is available through telesales, or basically if you call them. It's available globally. It will show up in the web portals in various regions, I think over the course of the next couple of weeks or so. But don't hold me to that.

I don't know for sure. I don't work for Lenovo. Yeah, sure. No, very good. Good to know. It does remind me, Drew, that there's been some background projects getting in place to kind of make this a first class experience. Yeah, I think it's worth mentioning that Lenovo were some fairly early adopters of the LVFS project, too. So this has been a long time coming and they've been fairly good community members, in my opinion, just in the fact that they did launch into that LVFS system so early in the process.

It's nice to see, you know, like some formal support too, because what, so many of us have been using these as great Linux laptops for years and it seems like Lenovo has finally caught on that. Hey, yeah, people love this stuff. Yeah, there is that aspect. It's like, yeah, catch up guys. Come on. But what do we think this means for the vendors that have shown up for Linux and been here for maybe like a decade, like System76? Is this going to hurt them, do you think?

I don't know if hurt would be the right word. I think it's good for the overall community that we have more choice, but it does maybe say a little something about the vendors who have been doing this for a long time. now that, you know, Dell and Lenovo are both offering, you know, first class Linux support. I don't know. I'm hoping not because I'd like to see things like, you know, Tuxedo and System76 continue to thrive. And I kind of doubt that that's really going to impact them too largely just because, well, System76 is still a big player in our community. You know, and I looked at these laptops, there's like the advocate in me that's like, I need to put an order in. My Linux laptop was supplied by a cloud guru. And so I need, you know, I'm like, I would really like to have a Linux laptop. I would very much like to have one. But when I look at the lineup, even though I'm compelled to like support this initiative, I've always believed in voting

with my wallet, you know, right now, like these first three months of JB being indie are like the just most critical. Like if I can survive these three months and then make it the next six months and then make it to the year mark, like that's going to be a real accomplishment so i it's like i really i'm really in a hard spot i i look at it i'm like it just doesn't quite do enough for me now this isn't an ad this is totally just my opinion but i look at something like the oryx pro and i think that's a lot more meat on that bone and if i'm only going to be able to spend my very limited funds once, I get a lot more for that dollar from something like the Oryx Pro. And maybe because for me, like a GPU is kind of important if I only have, you know, one machine like that. You're probably also going to get, you know,

things like a better support experience. Yes, Lenovo is now supporting Linux directly on these systems. But, you know, that's a big organization with supporting multiple different platforms and operating systems versus, you know, a smaller shop like System76 where Linux is what they do and they're all passionate about it. Well, and Lenovo is pretty new to supporting Linux too. So, you know, what's the quality going to be? I'm not saying it's bad. I'm just saying it's an unknown. Right. They're going to be figuring some of that out still. Yeah. And I tend to agree with you here. I've been a longtime ThinkPad user,

but especially lately, it feels like you've been getting less bang for your buck with the Lenovo models than you would with, say, a smaller distributor. I still think it's significant because it is a brand legitimization. It is an option that a lot of developers are going to be comfortable with and enterprises.

But there should be some knock-on upstream effects that might help the other OEMs as well, right, Neil? The advantage of having a big OEM like Lenovo directly and properly supporting Linux in a very direct way across the globe is that they're directing all of their suppliers to also support Linux the right way. That is driving them to move work upstream, contribute to the correct projects, get their drivers in there, make sure the hardware enablement's there. And it shifts the prioritization for their suppliers, too. Like, you know, I still hold out a tiny flame of hope. You know, it's a very small flame of hope

that NVIDIA could be convinced to make their open source drivers for NVIDIA cards work a lot better. And my hope is that because Lenovo is such a big user of NVIDIA cards on all of their professional lines, and two of the professional lines that are going to come with Fedora later on

are going to have NVIDIA cards in them, that it might help, you might help drive this case towards we should better support Linux out of the gate rather than this crap situation that we're all stuck in right now. And that's good for Lenovo, that's good for the Linux consumer, but it's also good for all the smaller OEMs that just don't have the purchasing power and the might to get a parts supplier to respect them. And to be frank, I love a lot of what System76 does, but they just don't have that might that is required to force a supplier to do things the right way. And Lenovo does. And I hoped 10 years ago when Dell was starting to do this with Project

Sputnik that they would do the same thing. It hasn't panned out to the degree that I'd hoped for. It will be interesting to see if we see any messaging from Dell in response to this. Popey, I feel like I want to go to you to get some historical thoughts on these ThinkPads and these Lenovo's officially launching these Linux lines. And if we zoom out in a few months, we'll have Ubuntu versions as well. What do you think about all this? Well, obviously, I love the idea that my preferred laptop vendor of choice. You? Yeah, I know, right? Are making it easy for me to make that purchasing decision. And later this year, I get my laptop refresh money from my employer, and I get to choose which laptop I'm going to spend my money

on. And obviously I'm going to spend my money on a laptop from a vendor that supports Linux. And so there's now choice. I get a laptop refresh amount of money every three years. And every three years, there are more places that I could spend those dollars. I'm delighted by this, obviously. That is an interesting measuring stick. You know, when you come up for air every three years to start shopping again, and you can kind of take a poll on what's come since last time. And you're right, it's more choice than ever.

We're also seeing distributions work closer with hardware providers too. So I think we're going to see more stuff there. Yeah. When it does come time for me to get a dedicated Linux machine, thankfully I have my Pinebook Pro right now, but honestly, I'm a performance maniac. So when it does come time, I'm going to have an even harder choice, which is a good thing. I think that's code for you break systems. I just am a demanding user.

Right, right. Thankfully, speaking of upstream knock-on effects uh performance on the raspberry pi 4 could be looking a lot better for those of you using sd storage a manjaro developer has brought to light that only the single data rate mode is currently being used for micro sd cards and emmc storage with the raspberry pi 4 model b. But get this. Ready? With a two-line, yes, two-line kernel patch, the double data rate mode can be enabled. Yeah, Tobias Schramm sent out a kernel mailing list post identifying that it looked like the controller and the board circuitry

appear to support double data rate mode just fine. He also checked on the signal integrity on the data lines for the micro SD card slot, and so far hasn't found any issues. Now, so far, no one from the project has really responded to the patches yet, but it was just posted this weekend, I think. So we'll see. But it is exciting. You know, not everyone has the time to set up some sort of external hard drive, or you just want to, you know, small factor, and SD card makes the most sense. Yeah, this machine we're using today for this little testaroo that we're going to tell you guys about, we're running it off the SD. I mean, it's not great, but when you just want to get a Pi going,

it's pretty nice. And if you're running Manjaro on your Pi, I think they're going to patch this on their version of it, so those folks will take advantage of it. Yeah, right away yeah you know mangero is making a very compelling argument uh for the arm platform however for these devices that we're using i've stuck with 2004 because i kind of use them for a little more like reliable like server type work uh i want the lts 2004 but i also want something that's built to

be server grade and as much as I love Arch, and I've put it on our server in the studio, I don't actually want to run Arch on my Raspberry Pis that are acting as network routers. Turns out one Arch server is more than enough. Yeah, yeah, really. But this is a great little thing to see. And you know, one of those little improvements by code, which I love updates that make your system faster. Right? I mean, like if hardware is just sitting there and suddenly with a little bit of tweaking, you can make it better for everyone using it. That's really exciting. All right. Well, I have something pretty exciting. I am very excited to announce that

we have one of our first sponsors since going independent has come on board the Unplugged program. And it's a company that I've wanted to work with for a very long time. And they were one of the first on my list when we started talking about going independent again. And I am thrilled to say this URL for the first time. Check out for a special offer for Linux Unplugged podcast listeners

and new Linode customers. Visit linode.com slash unplugged and receive $100 towards your new account. You know, these new ads we're doing, they're going to be tight, efficient ads. This is not representative of what the ad will typically be, but I wanted to take a moment with this first one and tell you personally that for me, this is a milestone in my career because I have followed Linode from afar for a very long time. And when JB stopped doing all sponsorships, uh, I wanted a pretty clear separation of church and state. And so, uh, about two years ago for all my personal projects, I started spinning

everything up on Linode because I thought, you know, nice and clear and separate all of the work stuff's on one, all of my personal stuff's on the other. And I watched them become an extremely competitive company where they really have evolved their product over the last couple of years to the point where it is extremely sharp, it is extremely well put together. And the best part is they have been deep in the community for a long time. So they're one of those companies that is part of the Linux community. I have gone to many events and I've seen them at those events talking to people. So check out linode.com slash unplugged. They have plans starting as low as $5 a month. They have dedicated CPU plans. They have GPU plans. They have block storage. They have the one-click

app marketplace that I love, but something else that's really cool, something I played around with a lot with setting up different WordPress, and I was able to play with different styles of WordPress setups, is they have these stack scripts that will deploy a stack for you. And you can just audit the script. You can see this is where it's installing Debian. This is where it's deploying this package. Really easy to write your own as well. Native SSD storage, 40 gigabit network connections, industry leading processors. They have data centers all over and they have job postings as well, which I want to mention for those of you that are looking for work right now, Linode.com slash careers. It's not related to the sponsorship. I just think that's something you should know about.

But if you want to get that $100 credit and you're a new customer at Linode.com slash unplugged, go spin up a box on linode's infrastructure it's really cool and i am so happy to have linode on as a sponsor but also to have them here just as we're going independent again so they're one of the first companies helping make that possible they started in 2003 so they've been around a long time just like Jupiter Broadcasting. They support the Kubuntu folks and other members of the community. I've wanted to work together with them for years. And so when this deal worked out, I had a real moment. I had a moment where like, okay, this decision to go independent is going to work.

Like, it's really thrilling. And I would encourage you to support our sponsor because they're making what we do possible right now. And I think this is a great fit. Linode.com slash unplugged. Get $100 off. And a big thank you to Linode for sponsoring the Unplugged program. I just recently set up a WordPress instance. I did it in seconds.

I was seeing what I could do to just do a completely self-contained, self-hosted podcast platform. It's awesome. We'll be telling you more about it. Linode.com slash unplugged. I just signed up. I just used your code and just signed up for Linode because I actually needed a quick and easy VPS to sign up for and having $100 thrown in was an absolute bonus for me

because money is tight and I really appreciate that. So thank you. I've just signed up. Yeah, thank you to Lenovo too. I'm looking at doing the same. They're a really cool company. And they're one of those that as a businessman, I've wanted to work with for years,

but then there was like another relationship that kind of made that impossible for a while. And then we weren't doing ads. And so this has just been an opportunity that it's really exciting. It feels pretty great. It feels like a great fit for the show. Also want to say a huge thank you to our Unplugged Core contributors.

We are already halfway to the goal of essentially the revenue of a single sponsor. We're about 50% now. It's like 48% the way there, which is incredible. That's an amazing first response. The founder discounts went away super quick, but the membership is still available at unpluggedcore.com. We have two feeds for you. One feed that is the entire bootleg live stream, lots more show, lots more show than what makes it in typically. The other version is an ad-free feed. There is some small ads in there that are contractually obligated, but it's like

just the main fully produced show ad-free. And you get that as a, as a podcast feed in your podcast catcher, you choose either one. And it's a great way to support the show and not only keep the ad load down, but let us be picky and choosy too. So that's an unplugged core.com. And thank you everybody who has become a core contributor. It has been a very emotional week for me. And that's one of the reasons I'm out in the woods, seriously, is just so I can kind of process everything that's happened.

And that support, you know, it helped me feel like I was, it gave me some confidence that I was doing the right thing here. The audience gets your back. They do. They do. They do. Let's also mention something else we're working on. And that is coming up next episode. So we're doing a Fedora 33 bug-a-thon to test Fedora 33

on September 8th. We're going to do the show and then we're going to just move over into testing. I want some people to show up and throw some hardware at the ButterFS install. Let's try to find some issues. And if we do, let's work with each other to do informative, helpful bug reports. Our idea here is we're trying to line this up with a testing week with Fedora. So we're doing this while the project is in a mode to receive these, and we're going to try to do well done bug reports that actually help them.

And hopefully we'll have some people advising us and guiding us on how to do that. And we'll be on the live stream working with people and can help bounce things around in the mumble room. I'd like to do some brainstorming in the LUP lug this Sunday. And if you haven't tried it yet, maybe it's a good time to go get Fedora 33 and start banging on it.

The, uh, test week is going on right now as we record and it wraps up on the eighth, which is the last day. So, and that's the, that's when LUP is.

So that's next week as we record this episode, September 8th, 2020. If you can make it to jblive.tv, join us for the live show and then stick around to do some bug smashing or show up when you can. It'll be a live bug-a-thon. I don't expect this to be a tremendous success or anything like that, but I'm trying to learn how we can do this best so we can make this something we do a little more on the regular for other projects and

distributions. Try to do something that targets the project schedules that can kind of land when it's most beneficial. So we're not overwhelming them with low information, low value bug reports, but try to get it right. And that's what this is about. And we're kind of doing our first tests with this next week, September 8th. And you know how we love Fedora. Yeah, we love to give those guys a, you know, a good community test because the ButterFS switch is a huge switch and it's a very public thing. And if we can maybe help avoid some issue that comes up that causes people heartburn, it's a win-win for everybody. Okay, so that's everything. Let's talk about the little trick

that Wes and I pulled off using two separate internet connections with Linux. We are using MiFis, but in theory, you could combine this with, say, a home broadband connection and a backup cellular connection that's tethered to your phone even, doesn't have to be a dedicated data device. There's a lot of ways you can actually mix and match this to bring maybe fairly reliable connectivity

to an area that maybe is having an issue. I'm utilizing it because, well, let's see how to put this. You're a hill person. Yeah, I am out in the woods and I don't have like a broadband internet connection. I don't have like Comcast doesn't come to my RV. That's not a thing. I thought you were just like dragging a fiber cable out the back. Yeah, right. Or I have a big satellite pointed at Elon's Star a thing. I thought you were just like dragging a fiber cable out the back. Yeah, right?

Or I have a big satellite pointed at Elon's Starlink array. Maybe someday. Yeah, maybe actually. But right now, the most practical approach, this is crazy, but the most practical approach is I track down old, grandfathered, truly unlimited data plans, and I contract with them. And I have an AT&T, I have Verizon, I have others too. I actually have Ting back in the mix now, but I have several others. But for this, I wanted to see if I could take a benefit, if there was any benefit from

using both my AT&T MiFi and Verizon MiFi at the same time and use a Raspberry Pi to sort everything out in between. And there's issues here that have to be accommodated for, like, you know, MiFis are pieces of crap that often require stupid interaction, you know, and those kinds of things. But overall, the idea was, these are two very expensive connections. Can I leverage them at the same time? Instead of traditionally connecting to one, using that, and then speed testing it, connect to the other one use that speed test it and then okay well i'll use this myfi today and then tomorrow i'll use that myfi instead let linux sort it out and just use them both just use them both and there's there's a lot of ways to crack this between like vpn bonding which would require uh like a node on a on a Linode somewhere or some kind of remote gateway

that would combine your connections. But that's not the direction we went for this. So you could, and some have even talked about doing with WireGuard. Didn't we see that as well, Wes? Yeah, you know, there are a few ways to make this happen because, well, it's Linux and that's how these things work, especially with networking. And I think this would work really well for a lot of places, especially if you weren't like Chris and constantly roving around this large country, because you do need a VPS at the end. And you basically, you know, you've got your two internet connections, and then you set up a tunnel to the VPS over both internet connections. And then you just use the regular kernel, you know, interface bonding support that exists just right in the

kernel itself to bond those together on your local side. And since they both end up at the regular kernel, you know, interface bonding support that exists just right in the kernel itself to bond those together on your local side. And since they both end up at the same place, it just works. And then the VPS acts as the final gateway to the internet. Now, a big plus with this too is you only show up as one external IP address, right? You just show up as the IP address of that VPS. But it's a little more complicated. You also have to have a VPS in play, which not everyone does. And it might be tricky for you because where you are in the country might not correspond to where that VPS is. And, you know, if you have latency there dominates everything, maybe that's not so great. So before we talk about what we did instead, to accomplish what Wes and I were able to pull off just using free Linux and a Raspberry Pi. There is a commercial product made by a company called Peplink.

And these are the products you usually find in buses, trains, boats. And this product is called the Max Transit Duo. And it combines two LTE connections and it has external antenna connections. And it's a little router. It may even be running Linux itself. It has a bit of a UI to manage all of this. It makes a lot of what Wes and I did simple. The catch is the product is like $1,100.

Woo! Yeah. And then the bonding service to bond the connection is about $1,000 a year. Wait, what? Yes. That's a little more than a VPS. Yeah. So when I looked at those prices, I said,

I bet we can figure out a way to do this for free with Linux. And sure enough. So the approach that we went with is it's more, I guess, akin to load balancing. Wouldn't you say? It's like a policy routing solution. Yeah, you know, if you're still using ifconfig, well, that's just fine. But you really should check out ip route 2 and the IP command because it exposes a whole bunch of really neat kernel facilities that you might not otherwise be aware of.

And one of those is more advanced policy routing. Now, normally when you're routing packets, it's kind of all about the destination. But in our case, if we have two outbound gateways, two ISPs that are connecting us to the internet, things can get rather confused if the connection comes in from one and then goes out on the other. Yeah, TCP doesn't like that. No, no, right? I mean, I wouldn't like that. But thanks to policy routing, you can set things up. So that never happens. If it comes in on an interface, it goes out the same interface. And the kernel actually has built in native support

to have different weights and to load balance between those connections. So you can just set up multiple routing tables, one for each of your different ISPs. Then you set up some IP rules, the command is IP rule. And that sort of, you know, sets things up so that you don't run into those confusing problems and things going out the wrong interface, make sure if it comes in on one goes out that same one. And then tell the kernel how you'd like to preference them. Either give them the same weight and it'll just split the traffic equally, or if you want to weight one more than the other because it's faster, well, you can do that too. Now, this isn't perfect in a lot of ways, right? Because for any one connection, it's not going to use the bandwidth of both links.

But if you've got multiple users on your LAN, say, or you're just downloading for things from multiple sites at the same time, well, there, it will work. And so this is something that we played around with because here where I'm parked in the woods, amazingly, there's a tower not too far away and I get 95-ish megabits on one connection and about 60 megabits on the other. What. Yeah, it's great. So we thought, well, let's wait the 90 megabit connection a little heavier, and then that'll be the primary connection. And then the 60 megabit connection is the one that gets used when the other connection is busy.

And the next level kind of trick to this was to set up that little Raspberry Pi as a router for the LAN and then just set the default route on my boxes on the LAN to point to the Raspberry Pi. Yeah, what's also nice about doing this yourself with Linux is it makes it pretty easy if you wanted to, say, like, let's say you've got the kids at the RV, Chris, and you want them to use the slow connection because it doesn't really matter for, you know,

whatever video they're streaming. Well, you can use the faster connection to actually get some work done. Well, you could set that up too. Now, Drew tells me that there's something similar to this that the Ubiquiti products do. Yeah, so you can do pretty much all of this in their EdgeMax line. And it's running Viata, which is essentially just like Debian, but repurposed for routing. Oh, cool.

So you can buy like a little EdgeMax router. Some of them are only like a hundred bucks. That's what I use at home and literally log into a shell, install Debian packages, adjust the routing manually, do all of that within a little box that's designed for routing, already has multiple ports. You know, whichever model you buy is going to determine which port, how many ports you have. It's really cool tech. That is neat. Okay, I would definitely check that out. Well, like all great things in Linux, so all this stuff is like built in, but there's also some projects that sit in front of some of these tools. Like the one we were using, Wes, was, if I recall, NetISP Balance, right? Yeah. And so where things, you know, you might want a little more support. And I should also note that some other Linux firewall-based projects like OpenWRT or Shorewall, they've got tools in

place to kind of manage this for you too. We didn't want to go that route. We were kind of rolling it ourselves to play with the underlying technology. But if you want to get fancier, you want to make sure, let's say say that one of these MiFi craps out and just totally drops off for a while, has to be power cycled and come back online. You want to make sure

that those cases are handled in a robust way. So there's a couple tools out there. We'll have them all linked in the show notes, of course. One of them was just an easy link status monitor that did the hard work

of sending pings out your various interfaces and then running external scripts when something changed. And that seemed super useful, really nice to customize. But I think for your case, we wanted something that was a little more plug and play and just forget about. That's where NetISP Balance came in, which is basically just a fancy Perl script that sets all of this IP-based policy routing up for you and has a few IP tables tricks up its sleeve too

to make this automatic. It's also got a nice failover mode. So if you don't want to load bounce, you just want to say have a backup connection, maybe that's a MiFi and you have a regular cable connection or something, that works too. Yeah, that's the part where I could see it

being really useful for remote offices that maybe have connectivity that have a broadband connection, but it's a little flaky. I've definitely had clients that have been in that situation before. Or people at home that always want to be able to have some connectivity. Maybe it's for their alarm system monitoring or their camera system. You could have it fail over to something like a Ting MyFi that you're only paying for when you use the data. Or what I'm doing here is Google Fi is like the third backup. data. Or what I'm doing here is Google Fi is like the third backup. If for some reason I have no Verizon and AT&T, it's unlikely that I'll have Google Fi, but it'll give it a shot. And that's

pretty neat. And you just have this really easy to set up config file and it makes it simple. I assume you probably have to have mailing set up on your machine, but it makes it very simple to generate alerts when an event occurs as well, which is really nice. So you don't have to sit there and watch it. It'll just send something to like an email inbox. Right. You can get an email saying like, hey, just so you know, your main connection went down. I'm now using your more expensive backup connection just to be aware. Yeah. So that's NetISP Balance. And it's just one of those great open source tools because a lot of the heavy lifting is actually being done by the kernel and the OS tools.

But like Wes said, this Perl script really kind of wraps it all up and makes it nice and approachable for you. Yes, and it'll handle some of that monitoring for you too, right? So it's going to be doing, you know, you can configure it with endpoints that you want to test. It's going to send out periodic pings. You can also configure it to say like, oh, well, if packet loss exceeds a certain threshold, well, then consider that link down. All kinds of options like that. And another sign that I really liked, I mean, as you said, Chris, it's just basically using a whole bunch of built-in stuff to the operating system. It's got a debug mode.

So if you want to go investigate what commands it's running or just use it as a script to orchestrate this and then customize it, run those commands yourself, that's possible too. Mm-hmm. yeah, good point. So, okay, let's talk a little bit about policy routing a bit more because that was kind of key to make this work. Traditional IP routing systems just route packets by comparing the destination address against a predefined list of routes that it can go to.

And at certain points as well, I've got nowhere else to send this, I'll send it to the default gateway. And that's my understanding of routing. So how does that differ from policy routing? Well, policy routing is just more flexible. It lets you use other information that's available. Maybe the source of the packet,

maybe the protocol that's going on, basically a whole bunch of other options instead of just the destination. And usually the way this works is you've got basically this set of rules that you can have, and then you can add additional routing tables. This might be familiar for folks who are used to, you know, maybe more network grade routers that have VRFs, virtual routing and forwarding tables that exist. And it just adds a lot of flexibility because traditionally you just

have that one main routing table, but actually under Etsy IP route to slash RT underscore tables, just by adding a line in that file, you can add basically as many as you want. And then once you've got these tables set up, they act as little routing namespaces. So you can configure different default routes, you can set up access, and then you can set up some rules

that make sure things based on either the source or the protocol or whatever fields that you have access to through policy routing, go to the right table. And then those tables just have routes that work as normal and then direct the traffic on. So you can make sure that, say,

if you have different lanes that you need to get to, you can set it up so those go at the shortest hop possible. And in our case, that's the magic that says, oh, look, you came from this interface over here. Let's make sure that you choose the routing table that sends you back out that same interface. And in the past, you came from this interface over here. Let's make sure that you choose the routing table that sends you back out that same interface.

And in the past, you could do this too. IP tables has a marking facility. So you could use IP tables that would set a mark on a packet if it came in over that interface. And then on the outside, it would identify that mark and then know how to send it. But this is just a much cleaner approach. Now, from like the totally other end,

this is something we didn't play around with, but we did come across Speedify, which is a commercial graphical user application that promises to combine multiple connections on your Linux desktop or on, I think, other OSes as well, including maybe mobile. And it also include VPNs in that potentially. And it combines all of the connections

into one VPN back to their server. It's a commercial product. It's like $3 a month. But they kind of promised to do this only. You don't have to have a separate device. It's acting as a default gateway. You just throw it on your laptop and connect a couple devices up. And then this app gives you a GUI to bring them together.

Didn't try it, but it certainly from the screenshots looks pretty decent. So that's something if you're interested, if you're interested in this, but don't want to go through all that rigmarole, you could give that a go. But for me, it was worth all of the rigmarole to get a Linux box set up and get all this done and figure out how to do all this because that's sort of like the groundwork for a good edge device. And then I could say, make a wire guard endpoint, or perhaps I could put a squid proxy on. So that way I'm not sending as many web requests out over the MiFis. And instead I'm answering them locally from a cache and setting it up as a local DNS

cache seems like a no brainer as well. And maybe it's an edge cache for steam possibly as well. If I throw a USB hard disk on there, all of a sudden, this little Raspberry Pi that's managing these multiple connections can start doing other things that are clever to take the load off of my MiFis and make the overall experience faster as well. And so that's why I thought it was worth the effort. Right. And, you know, I think it's a good use case here, too, because you might ask, well, is the Pi really up to something like this?

But since you don't have a fancy, you know, gigabit fiber connection, the speeds on your outbound really aren't too bad. So it seems to be working nicely. Yeah, I think that's a good point to make, is if you're maybe connecting to something considerably faster, you might want a faster machine as the router. I also want to clarify here, too, that, you know, net ISP balance was really neat to find also because

while policy routing is not too complicated in the simple case that we were talking about, and we'll have linked in the show notes just the sort of the basic guide that we followed when we were setting it all up by hand. But if you're new to networking or networking with Linux, and this all sounds a little bit over your head, you don't really have to worry about that with NetISP balance. Because basically the bare minimums that you need are you got to have your multiple connections. They got to have IP addresses. In our case, that was just DHCP to the MiFis. And then it literally took care of the rest. I mean, you did have to set up in the config file. You got to tell it what type of service you got, you know, like which are your LAN interfaces

and which ones are ISPs and which devices that correspond to. And you can also, that's where you can customize the weights and like what IP to ping to check if the connection is up. But that was all we configured and it set up everything else. Yeah, and it's really simple because it refers to stuff as LAN, ISP1, ISP2. And so it's not confusing at all. You know exactly what connections you're dealing with, which I thought was really nice and it made it super simple to read.

I could see us doing other things down the road to accommodate the fact that these are crappy MiFis. Like using tools like FullSM to accommodate the fact that these are crappy my files like um using tools like uh full sm to monitor the links of them and then when it detects that some sort of link has failed triggering another script that does a usb port reset to reconnect the my file i could i could see little workarounds needing to come up because really for me ideally like the one thing that the peplink device has is that the sim card slots are fully integrated into the actual device. They're not these USB MiFi devices that are running their own crappy OSes. That's probably the biggest limitation here is, yeah, it's not like a real interface or anything or a fancy high-end SIM interface.

It's just a crummy MiFi. Yeah. Another thing that came to my mind that would be a neat stretch goal here is, you know, set something up to do speed tests across the various links and then shift the weights as those speed test results change. Yeah, that would have been useful today because before we started recording, I did all the tests and the AT&T connection was far above the connection to use for our remote VoIP setup here. 15 minutes before we went on air, the AT&T connection was unusable. It was dropping packets to our voice server like crazy for some reason, out of nowhere, just every second or third packet gone, sometimes multiple packets in a row gone. And the Verizon connection out of the batch all

of a sudden was the best connection. And it got me thinking like, boy, it'd be really cool to have some sort of health check script that just does a sanity check back to the studio and then picks the connection that performed the best in that health check. Yeah, wouldn't that? That would just be so slick. So pro. It'd be so pro. I tell you what. Colonel points out that there is the GIL or GINet devices.

Like I have their little mini slate. They do have one with dual SIMs. I saw that. I was looking at that one, actually. So I'll probably eventually switch this up. In the meantime, I kind of want to see how far we can take it. So we're going to play with it a bit. But it was it was really fascinating to get, I would say, at least the core functionality

of that thousand dollar device for totally free with Linux. Right. That was one of those things like Windows can't really do this. Mac OS can't really do this. This is pretty awesome like the fact that the kernel is just totally helping us out by keeping track of the routes and it's easy to bond these connections and then it's easy for an open source developer to create a pearl script that sits on top of all of this that i mean it's it's pretty awesome stuff and like the policy routing is like you're saying it's like enterprise grade routing

stuff so it's and i have it right here on a tiny little $40 pie. Yeah, that's what that's what's really exciting to me is it's just like the kernel has a really rich networking stack. And yeah, OK, if you've got, you know, crazy high line rate speeds, maybe it's not going to cut it for you. But for anything we're doing, totally more than enough. Yeah. So we'll have links to the projects and some of the other projects that we saw that could be useful in the show notes. If you're curious about maybe setting up a backup connection or combining two connections, those resources should be a good starting spot. Well, Mr. Payne, what do you say we move to the feedback? And we have something submitted by Jay that we may adopt.

We may fork this ourselves. submitted by Jay that we may adopt. We may fork this ourselves. This is a getting started with the Jupiter Broadcasting Matrix Guide, a quick start, including a video walkthrough to make it super easy, but also just a step-by-step guide for getting on the Jupiter Broadcasting Matrix. And we have a link to Jay's blog where he wrote all this up. And he said, guys, feel free to take as much or as little of this as you want. And I just wanted to get it out there right now. We may combine that with a guide that we've already started, but we're kind of doing a hundred things at once right now. So it was just great to have somebody write that up. So thank you, Jay. And if you'd like to join us on the new matrix server, check that out. We have a link to it, the Linux Unplugged.com.

I think this is just a great, you know, just enough of a guide to where I found Element decently easy to use and get started with. But for folks who've never tried it before, go check out the video. It'll have you up and up and chatting with us on Matrix in no time. Let's talk about this pick that we've been sitting on for a couple of weeks because we've been super busy. But this pick is really special for for one reason and one reason alone. It's written in Rust. So I'm going to give a guess, Wes, it's pronounced TunShell.

Because, you know, it's like tunnel and shell. Yeah, I think you got it. Okay. It's pretty neat. It's a simple, secure method to remote shell into ephemeral environments. And I like that they've got this question right at the top of the readme. Why would I use this over my well-established SSH client? Good question. You wouldn't. The use case for ton shell is predominantly quick ad hoc remote access to hosts which you may not have SSH access to. So let's say you've embraced the craze, you're using

AWS lambdas for just about everything now, something goes wrong, that's not really an environment you control. And usually, that's part of the value proposition, right? It's a quote unquote, serverless, or at least, hey, it's Amazon servers, they're just running your app for you. But sometimes you might need a little more information. But you can't set up an SSH server. Well, now you can, because Tunshell is a statically linked, pre-compiled single binary that all you need to do is just download it, run it, and boom, now you've got SSH access.

I love that. Don't have SSH access? Well, we can fix that. That's pretty neat. I'm glad we finally got to this one. We have others that are in the winds. Is that the right, in the rafters? In the wings.

What does that mean? What does in the wings mean? What does that even mean, Wes? I call BS on that saying, Wes. Ouch. I've decided right here on the show. But that doesn't mean we couldn't use some more suggestions. So head over to linuxunplugged.com slash contact

if you've got a pick for us. Diddle, you say you know what in the wings means? Yeah, it's a theater term. In the wings means backstage. Okay, that makes sense. Waiting to make their debut. Right, they're backstage about to come out on stage. Okay, now, all right.

I retract my statement. I like it, actually. I think that's pretty great. All right. Well, yeah, we'll link to that in the show notes. TonShell. And it's the remote shell into an environment that, it's like a subspace environment. You know, this is how in Star Trek, they would get into a computer that was inside a subspace envelope is they would use TonShell because it's an ephemeral subspace environment. You know, this one for me, actually, I discovered it too late. Just a few weeks ago, I was trying to debug some problems in an ephemeral environment and I didn't have a tool like this and it was driving me crazy. So I think this is going to be one of those picks that actually makes it into my toolbox. All right. Well, don't forget about that bugathon on September 8th after Unplugged. Show up for the show if you can and then stick around to help us make Fedora 33 even better. We need your help.

I think it's a great way to kind of contribute. You know, we use Linux and desktop Linux. We take advantage of all of these great projects and work of open source developers. And even if you're not a Fedora user, it's not a bad way to just sort of stop by and give back a little bit and participate in the community as well. So that'll be next Tuesday as we record this, and we encourage you to show up. Also, again, thank you to our core contributors, you guys. Wow. Amazing. And also just a little interesting stat, Wes.

It seems to me when I looked like it's split 50-50, like 50% of people are using the bootleg feed and 50% of people are going with the ad-free feed. It's not like one is particularly more popular over the other. Well, when you've got two great options, it can be mighty hard to choose. Oh my God, Wes. Unpluggedcore.com. We appreciate your support over there. Also, a big thank you to A Cloud Guru. Check out the new A Cloud Guru at cloudguru.com. And also, a big, huge thank you to Linode for sponsoring the show and making it possible for us to come back in the I am thrilled Linode dot com slash unplugged. Go check out Poppy on the Ubuntu podcast and the new show, which we'll have links over there. Yeah. Hey, hey, hey.

Go check that out. It's it's a good time. Mr. Payne, what else should we mention? Maybe the show has a Twitter account. We hardly ever mention that. At Linux Unplugged, good way to get show updates. Anything else? Come hang out with us on our new month Matrix server.

Sure. Got to get good about the plugs on that one. I'm at Chris Less. He's at Wes Payne. The network is at Jupiter Signal. And Drew is at Drew of Doom. Thanks so much for tuning in to this week's episode of the Unplugged program. Oh, Linux Action News is back. Check your feeds. See you next Tuesday. And Drew is at Drew of Doom. Thanks so much for tuning in to this week's episode of the Unplugged program.

Oh, Linux Action News is back. Check your feeds. See you next Tuesday. So Westpain, I can't believe I forgot to mention Linux Action News is coming back. So as we record Monday, probably, the next Monday you're hearing this, there'll be a fresh, brand new Linux Action News in your feeds. I feel so bad. I should have totally put that in the show. There's so much going on right now.

It's a good problem to have, but I don't want to overwhelm people with the pluggy plugs either but that's a big one i'm very excited about land it's a literally huge news it is very big and um you know for me it's like uh it's like it's a lifestyle change it is it's like i went a decade plus doing shows on sundays and then just stopped and now it's coming back the only the biggest downside for me is it lands during the luplug so that'll be tricky but i'll probably i'll try to maybe pop in early i'm not sure i'm not sure what to do because we record at noon on sunday which was funny because the luplug started

sort of right as the next sunday i would have been doing land became luplug time i think this just means you need to drag joe into the luplug for a bit. You know, just dabble, hang out for a few, and then go get your work started. We all know you like to chat before you work anyway. Yeah, I could run two instances of mumble. He wouldn't even have to do a thing. Don't give him the choice. He'll love that. Surprise!