Embedded - 521: Are You The Tiny Domino?

Episode Date: February 20, 2026

Kenneth Finnegan entertained us with stories about accidentally contributing to the internet's ability to network. Wondering how the internet works? All those terms about IPv4, IPv6, BGP, OSPF, CDN and other alphabet  soup? Check out the YouTube videos by NetworkChuck. Kenneth writes about his adventures on his blog, The Life of Kenneth. Some of the posts related to this show are: Creating an Internet Exchange for Even More Fun and Less Profit Building an Anycast Secondary DNS Service  Building the Micro Mirror Free Software CDN  We also mention FCIX aka fcix.net or the Fremont Cabal Internet Exchange You can also find Kenneth at @[e-mail] where you will find more about half-dollars, nickels, and trains. If you also secretly long to run a locomotive, take a look at the Run-A-Locomotive program at WPRM. The title is related to the XKCD comic 2347: Dependency. Transcript

Welcome to Embedded. I am Elysio White, here with Christopher White. Our guest this week is the internet. Wait, I guess this week is Kenneth Finnegan. And I think we're going to talk about the internet. Yes, I'm from the internet. Hello, Kenneth. Welcome to the show, finally. Hi, Chris. How's it going? I'm doing all right. Could you tell us about yourself as if we met at a super con luncheon lunch? table. Yeah, so I started life as a mechanical engineer. That lasted until about the second

week of my master's program when I had my first meeting with my thesis advisor, and that managed to annoy me enough that I switched to electrical engineering. I got a job as a solar cell at a solar cell startup testing solar cells for satellites, got then moved over into actual electrical engineering, building semiconductor wafer plasma etch equipment at Lamb Research, and then accidentally created a whole bunch of load-bearing parts of the internet. And so I now work, I worked at Arista in their technical assistance center, helping ISPs fix their own internet as their support helpline. And then now I work at Nvidia, building the AI networks that Chris really loves to think about and talk about constantly.

And then on the side, I just have all sorts of wacky hobbies that get wildly out of control. So there's the XKCD comic where there's a bunch of pieces, blocks all put upon each other, and then there's one little tiny domino at the bottom. Tiny domino that everything really depends on. Are you the tiny domino? Are you the tiny domino? I am one of the many tiny dominoes. Do you have a security detail?

I do not, right? And that's the scary part is like everyone, the most misleading part about that comic is that everyone's like, oh, yeah, there's that one small project that everyone relies on. But that's a single small project that everyone relies on in every single different dimension. And so, like, I would describe myself as one of the 250, 500 dominoes that keep the, the internet running. So I'm in like no way like the sole load bearing part of any one part of the internet, but it's terrifying how small the pieces get at the bottom. It really is.

Okay, but we're going to do lightning round and you're familiar with the show, so I won't tell you the rules. You'll just have to remember them for yourself. Are you ready? Yes, yes. Would you rather fight ten, half dollar coins or one hundred nickels? Oh, I carried nine half dollar coins with me on a daily basis, so hands down, I would would fight 10 half dollar coins because I think I can just take on one of them myself.

Okay. What is the worst internet protocol? The Border Gateway Protocol or VATP is the worst internet protocol. It's also the only Internet protocol we have that is useful. I did say routing protocol in the question. Well, I mean, there's other routing protocols other than the Internet. Okay. What's the worst interior routing protocol?

RIP. And I can tell you for a fact that RIP is still used in many production very large national networks. I just didn't want you to be mad at OSPF because I implemented OSPF once. Oh, no, OSPF is great. Like, OSPF is probably my favorite routing protocol for underlays, so no complaints there. Well, for all the people who don't currently understand what the questions are about right now, let me ask this one. 042, 440, 460, or other.

It's definitely other. My favorite configuration is BB. Seriously, I have no idea what's going on either. It's not just you. So what you're asking me about is the wheel configurations of steam locomotives. And I'm not at all a fan of steam locomotives. I'm much more a fan of diesel locomotives. And so when you look at a diesel locomotive, the number of axles on each group of wheels

will dictate the configuration of drive for it. And so BB indicates a two-axle at the front and a two-axle at the back, where many modern locomotives that you'll see will actually have three axles on the front and three axles on the back, so it'll be a C-C configuration. Do you own a conductor's hat?

I do not own a conductor's hat. My head is a little awkwardly large, shockingly, and I spend most of my time in rail yards wearing a hard hat. And so I don't really have much need for... It's probably what they make you wear these days, isn't it? Well, yes. How do you most effectively convince someone they need to own two miles of telegraph wire?

Oh, so this is a very fun game that I love to play. So in the Bay Area, we have all sorts of electronic swap meets and salvage. surplus stores that at this point are mostly gone. And so the game that I loved playing with all of my friends as we were hanging out at like Hal Ted or weird stuff or at the Silicon Valley Electronics Flea Market is we would see something and I would try and convince my friends to spend their money to get it. And my crowning achievement in this whole game was that one time I managed to convince one of my amateur radio friends as we were standing there in a parking lot looking at a vendor

trying to sell two miles of rusty telegraph wire still in the original case, that he would definitely use that to build like dipole antennas and long wire antennas and stuff. And so I got my friend to lay 20 bucks on the table and carry two miles of rusty telegraph wire back to his car, drive it home, and then sit in his garage for five years before he finally came to his senses and threw it away. Did they know that they were playing a game? Yes, my friends have regularly accused me of, oh, dang, you're doing that thing again, where you're trying to get me to spend my own money, aren't you?

Have you ever networked a locomotive? I actually have. Locomotives have a very standard, oh, is it, it's 68 or 72-pin network connector on the front and the back of them, and it's this big cable that looks like, it's about the same diameter as like a gas pump hose. and so you can network two locomotives together so you can drive both of them from just one of them. Do you want to ask the follow-up? Have you ever locomotive to network?

I don't think I would say I've locomotive to network. No, I've managed to. Have you ever steam power to network? Oh, heck no. No, steam power is the worst. Okay. Complete one project or ask a dozen? Ask a dozen? Complete one project or start a dozen.

doesn't. Or Aska doesn't. So I, my brain works in this wonderful way where I kind of get the best of both worlds in that I love to spend years architecting projects in my head and on paper. And so I have a dozen projects that are like in the block diagram and pipeline. And then when I actually physically start materially building one of them, I can focus on it pretty well and get it across the finish line. or give up on it pretty quickly. All right. Now we have to ask, favorite fictional robot.

Favorite fictional robot is Wally from the fact that like Wally was a like a self-healing robot that like was so self-sufficient, which a lot of robots, and they really portrayed it in a way that a lot of robots is like, this robot would never sustain itself in any reasonable, useful way. And do you have a tip everyone should know? Yeah. experts love to talk about whatever they're experts at.

And so, like, people need to be less afraid to reach out and say, like, hey, I saw that you wrote this paper or this blog post or you've been posting about this. And I have these kind of, like, specific and constrained questions. So, like, can you answer these or can we, like, jump on a call and just talk about it for 30 minutes? Like, people are much happier to talk to you about things than you would think they are. I often hold myself back from talking about things because I, I don't know at what point people get bored. And so, yes, if you ask me to talk about something and you truly want me to talk about it, you may have to ask twice, but after that you might as well just get a cup of coffee and sit back because I will be forever.

Just ask me about origami. It's very embarrassing. And it's important to ask it in a way that it doesn't make it insurmountable. So I get a lot of email questions where people are like, hey, can you tell me how to get in a networking? And it's like, no. So science. Tell me about science. Exactly, right?

So you need to come to them with an attainable ask of like, hey, can I just pick your brain for 30 minutes and not in a skeezy way where we're trying to pay you $400 to disclose like industry secrets? But like I'm just earnestly interested in this thing and want to talk to you. So networking is hard and it keeps getting harder. What do embedded developers need to know about networking and how can they acquire that knowledge? That sounds like, tell me about networking. telling me about American, doesn't it?

No, networking is very hard. And the thing that at the subsystem embedded level that I see a lot of people struggle with the most, is that people really need to start, people need to be much more familiar and comfortable with the vocabulary and the risks around the field of study of distributed systems. Because everything is a network, right? Like, anywhere, you know, like, obviously you have things that look like networks that are, you know, you've got cellular modems or you've got Ethernet links or Wi-Fi. And then you've got smaller things that don't look like networks like I-square-C or SPI, which have a lot of the same failure characteristics and same dynamics as larger networks. And so people need to really get more comfortable with problems like, you know, the two general problem and consensus process.

protocols like raft and really appreciate like when the internet, you know, when the network fails, whatever scale of network that is, like what is my fallback positions and like how do I, how should my system respond appropriately to it and handle it gracefully? So basically we should be checking the return codes on our spy calls? I mean, yes and no because, I mean, yes. Yes. Like, you should be returning the, you should be checking the return codes on your spy calls, but then you should also be asking yourself questions like, well, what if it comes back with a bad answer, right?

And what is the failure? So, like, the one that we see constantly, or that, like, I have to interact with a lot is that, like, a lot of people don't appreciate that temperature sensors, like those, like a little I-square-C, temperature sensors, have a really bad habit that if you manage to ask them at just the right moment, they will, come back and tell you that whatever system they're currently measuring is 64,000 degrees. And so, and it's like, you know, it's like, that's a little hot for earthing. Yeah. And so it's one of those things that, you know, it's like, you know, not only do you need to be checking the return codes, but you need to be asking yourself, like, well, what if I get bad information off the network? Or what if I make an SPI call and it never comes back, right? Like, what it? You know, like, there's all sorts of these sorts of problems of, like, you know, like, what is every single different failure mode that when I'm, and inside of the microcontroller talking, trying to talk to anything else that I need to handle in some graceful way?

Because, you know, saying that your system drops out a cellular reception and just absolutely nothing works anymore, like, doesn't make for a very robust project. There is how the application works with the internet or works with a degraded network or a non-existent network. And then there is the, I don't know how to get started with networking. It is big. And sending up a raspberry pie for the first time in your home network is not trivial. Like, what is the IP address? What is an IP address? All of these features, but that's not what we talk about when we talk about networking.

Do you have a good resource for how people can just wrap their head around? Internet addressing to start with? Internet addressing to start with. Mac addresses and all of that stuff. Big problems of networking, not the specifics of like why OSPF is really good for when you want to multicast. and multicast means you want to send lots of people the same... That's not good for multi-calf. Whatever.

I don't know. Good for it. I mean... But there are certain protocols that are better for certain things. I don't want that information, clearly, because I don't have it. Right. There's tactical information for everyday people who encounter network, and then there's operator information for people who are running networks,

which are different domains, sort of, with some overlap. Yeah. And I would say that we're kind of really in this... We're on the tail end of the sweet spot. as far as on YouTube, there's tremendously valuable resources, you know, like networking Chuck and these channels that, like, we'll, like, sit there and talk to you about the basics of, like, this is what a router is, this is what a switch is. This is what a Wi-Fi access point is. I'm really, I've never had good answers for like, hey, how did you get into X, Y, Z?

Because, like, usually my answer is, well, I downloaded the I-Triple E spec for it and just read or I sat down in a coffee shop with all the RFCs printed out and read through them and highlighted them and put flags on them. So I was like, the way that I approach information, I think, is not a very kind or accessible manner for most people. But I would say, like, there's so many good resources out there right now that have not quite been overrun with the AI slop that's trying to re-monetize the same venues. Yeah, I don't think get a summer job at Cisco in 1995 and have somebody hand you a pile of RFCs. is really a good answer for most people. No, it really isn't. I mean, the man page is still one of the best places to get information.

And I don't think people even, I think there are a whole bunch of people who have no idea that man pages exist. Yeah. Well, you first have to have to learn the terms. And, I mean, but like not only just man pages, but like just it is so easy now to be able to build something that you can then like just look at it. Right. Like the burden of like you get me a Raspberry Pi, get me a $60 travel router and build an entirely completely zero stakes network on my desk. Right. Like, you know, I can then, like if I need to learn about, you know, what does this packet look like?

I can set it up and do a packet capture and I can look at that packet. And then I can change something and see how that changes the packet where, you know, it's like now for, you know, so for a couple hundred dollars, you can build an entire. lab environment that lets you play around with any of this sort of networking stuff at just smaller and slower speeds than exactly what everyone else is doing, right? Like, there's no reason that you can't have BGP running between two virtual machines on your laptop. But you have to know what BGP is, which I don't, so could you explain? Yes. So BGP is the Border Gateway Protocol. this is the routing protocol that's used between every separate network of the Internet. And so when you think about the Internet, you're looking at it from the perspective of you are a customer connected to one Internet service provider at your home,

where they hand you one cable and they say the entire Internet is this way. Right. It's a vast ocean. Yes, it's just that way. But once you move from your leaf right on the very end of the tree and you start moving back towards the branches to the larger branches to the trunks, the answer of which way is the Internet becomes much more complicated. And so your service provider is going to be connected to anywhere from two to hundreds of other service providers or service providers who are then also. connected to other ones. And so there's ultimately about 60,000 unique entities that make up the Internet that all are connected to each other in certain places.

And then BGP is how those two service providers, wherever they meet, can exchange information between their two routers on the edges about, hey, if you want to reach this block of IP addresses, I can get it to you via this path, you know, the sort of sorts of cost metrics. And if you want to go to these blocks of IP addresses, I can get you there or I can't get you there. So this is kind of like airlines. I mean, you used the tree, which I think is great, but it's hard for me to visualize how you do to cost metrics with a tree. Yeah, airlines would be a fantastic analogy as far as like, yeah, if you're in some little town like San Luis Obispo, that technically has an airport that has one flight into SFO a day. Once you get to SFO, you then have many different directions that you can go on many different flights to all sorts of different places that may not necessarily get you exactly where you're going in one hop, but we'll get you closer in that direction.

And multi-hop might be cheaper, but it might take longer. And there are all of these different cost metrics that you have to balance. And honestly, you as a packet or you as a person don't really get those. It's your ISP manages that. The packet, I mean, all that stuff gets programmed into hardware. So there's just kind of those are locked in unless something changes. Yeah. And unfortunately, the things change constantly.

Right. Something that people don't really appreciate is like, so like, well, to finish the thought, it's, yeah, like the cost metrics, which is like the A. path and lots of other things. And a lot of the packet routing decisions between the service providers isn't really what is the optimal way to route this packet for the best experience. It's often a, well, I like this airline more than that airline, or I want to earn points over here versus over there.

I have a service agreement with this one. Yeah. And so the BGP is a often called a routing policy. expression protocol that also happens to forward packets. But it's much, much closer tied to the actual business objectives of how does our service provider want to interact with that service provider in not optimal lowest latency, highest bandwidth criteria always. If I look at the, I want to say, Lego website, that is located in Denmark?

What, who, how does anybody get paid for that? Yeah, so the, for the traffic. Right. So to finish like the, the pop the stack is one, sorry, we're going to be all over the place here. Yeah. So the internet is constantly broken. And so when you're looking at a website in Denmark, you're very likely not actually looking at a website. in Denmark. You're looking at a website that originally came from Denmark, but you're probably

connected to a server that's much, much closer to you, because the burden of actually getting, like, you clicking on a link and getting a packet from you to your service provider to their transit provider, from that transit provider to another different transit provider than going through a cable underneath the Atlantic Ocean gets very difficult. And so, most large websites will have content distribution networks acting as they're like front doorstep into their website in a hundred or 200 locations throughout the world. So even if the website's nominally from Denmark, you're probably connecting to a server, you know, a couple dozen miles away from you who it will have some parts saved already of the website and other parts, it will reach back and, find the best way to get back there. And so, you know, from a payment and money perspective, you are paying money to your

service provider to give you internet access in your home. They are then paying some other, like a tier two transit provider to give them access to the whole internet. And then the tier two and tier one transit providers, which are like, progressively larger and larger networks from like a geographical perspective, they interchange packets with each other on a, what's called a settlement-free peering session where they both say, hey, I can reach about a quarter of the internet and you can reach about a quarter to the internet. So if we just kind of equally trade about the same amount of traffic, it kind of works out

in the end that, you know, both of us can just trade traffic with each other and our customers can all reach half of the internet without us having to pay anyone else. And then someone lays a new cable across a giant ocean. Is it the tier three people who do that? It's anyone who needs it, right? And I mean, a lot of cables will be cooperatives between like these five very large tier one service providers all need extra capacity between these two places. and so they'll form a new holding company to, like, go lay one specific cable for a project, and then they will share bandwidth capacity across it, right? But it's not the historic, idealistic world where it's like you have these nice, neat tier one networks at the top,

and then tier two networks, and then tier three networks. If that ever really did exist, it definitely doesn't exist anymore, and you have this much messier, well, Facebook and Google are bigger networks than the big networks, but they're not really transit providers so much as end content providers, but they're laying their own cables because they have larger bandwidth demands internal of their network than anyone has external to it. So the not useful answer is anyone, anyone will do it depending on what they need,

and it depends. I mean, that makes sense. It seems like the sort of thing that would go through phases of vertical integration and then specialization and then vertical integration and specialization and what you want when you're vertically integrated and want to handle all your own traffic and charge other people to do traffic is different than I just want to buy traffic. I just want to be in charge of what I'm doing. I don't want to worry about the rest of the world.

Yeah. And then inevitably geopolitics, right, comes in and plays lots of parts as far as like, oh, you want fiber connectivity from this country to this other country. And, you know, oh, you want to lay fiber down through the middle of what is otherwise a relatively unstable area. And so, you know, like you go look at places like South America where it's often easier to lay subsea cables from your country out and then back into your own country versus having to go. forge roads through the middle of your own country. I mean, that is such an odd concept. Of course, laying cables in an ocean is going to be super expensive. Yeah.

I mean, just the, there's the physical, not all two-mile cables cost 20 bucks. Yeah. And then there's the personnel and the maintenance and the just drop the boat. Drop it's going to be expensive. And the cables get cut or bitten by sharks, which is, of course, my favorite. Bitten by sharks. Yeah, a bunch of them. Until they stopped making some of the cable stuff, the coating out of shark deliciousness.

They were relatively thick. Okay. Well, so my understanding is, so the issue is that, like, so when you have a, you know, a thousand kilometer strand of glass that's, you know, as big as your human hair, that's going from one country to another, you can't possibly pump enough light into it on one end that it comes out the other side usable. And so in the middle at intervals of this cable that you're just dropping on the bottom of the ocean, you're putting amplifiers in there to regenerate the signal. And unfortunately, those amplifiers need power from someplace. So part of this cable you're laying

down is a power rail to energize these amplifiers. And power, unfortunately, gives off magnetics and I guess sharks are somehow sensitive to magnetic signals. All kinds of electrical signals. They hunt using the electrical signals of the fish because the... Oh, here we go. Never mind. Yes. So long story short, once you even lay the cables, they tend to break, be it either by sharks

or, you know, quote, accidentally broken, unquote, by nation state fisher fishing vessels. I do think, I want to go back to one thing you mentioned about the, the caching stuff because I don't think people realize just how much of the internet is cloud flair in AWS. Yes. Like most of the time you go to a website, you're hitting one of those two things. Yeah. And that really comes down to the fact that the speed of light is painfully slow.

And so like the concept of like everyone and everywhere in the world being able to access your website coming from one server in one place. Like I mean, A, that's a very high stakes single box. you have sitting there. But secondly, the speed of light means that once you, as you move physically farther and farther away from that one place, the user experience on even the human time scale starts getting noticeably and noticeably worse.

And so a lot of the modern good high-end internet experience where you're like, yeah, you expect to be able to click on something and the website immediately opens, a lot of that relies on this fact that you are as a, as a website host, like if you set up your own little website, you're then paying some other organization or you have to grow your organization to build out this infrastructure so that when a user clicks on a link, they're accessing a copy of your website that is geographically very close to them, which is hopefully more or less representative of what your website actually looks like back on the source of truth, ultimately.

It's interesting to me how both different and the same the Internet is from when it was first kind of getting going in the 90s. You know, BGP existed then, all these other routing protocols, the basic fundamentals of it were all the same. DNS existed. We built networks out of all of this stuff. But the big routers now are very different than the big routers. But they're still running BGP. But we're doing everything differently with contact distribution networks and cash. and all this stuff. It used to be that, yes, you did go from your PC to, you know, a server in Denmark.

That was the way it worked. But we've taken these tools that existed that are pretty old and crifty now in some ways and rebuilt them in a completely different way, which is kind of fascinating to me. IPV4. Yeah, well. It's a good segue. Yeah, IPV4 isn't going anywhere fast. No, no, I was assured in 19...

97 that IPV6 was going to be deployed pretty widely by 1998. Fully accepted by 2000. Yes, absolutely. Not for all these damn ISPs using ATM. Never mind. Yes. Thankfully, ATM is mostly gone. Oh, God, it's not completely gone?

No, no, Lord, no. Nothing ever dies on the intro. I know. Okay, so we've talked about when these Tier 1 networks meet like slime molds fighting on the edges. With suits. With suits. Yes, with suits. Slym molds and soups?

Does it be like that for a title? All right. So they meet and they exchange traffic through BGP. And is this an internet exchange? It can be, right? And so when you're two extremely large organizations, like two extremely large networks and you specifically want to trade traffic with each other, it's pretty easy to say, like, hey, I have 300 gigabits per second of traffic I want to send to you.

You have 250 gigabits per second of traffic. want to send to me, it's probably worth it for both of us to build our networks into one building and have a fiber physically running from my router to your router. And that cross-connect between your two networks often happens in what's called a carrier hotel. And so these are just sometimes deliberate, sometimes accidental buildings that end up being these kind of sweet spots where lots of different internet service providers. all build into the same building, and they can then physically directly connect to each other. And so, and these buildings are often kind of nondescript.

And so, like, if you're driving, so, like, you know, local to the Bay Area, not that this will help a lot of your remote listeners, but, like, if you're driving up 101 and you take the 85 off ramp, right on the right side, you'll see this complex of four big buildings. that's the Equinix SV-1511 campus, which is where an extremely large portion of the internet traffic flows for all of the West of the United States. And so when you're extremely large, you can justify the cost, expensive, like, I need a 400 gig port from my router to this other network's router.

It's expensive in that you have to have a physical port on your router, but it's also expensive in that whatever carrier hotel you're in will often charge you a couple hundred dollars a month for the rental space for the two millimeter cable in their tray running between your two networks. That's ridiculous. Yes. All right. So it's the most expensive four square millimeters of real estate that you're going to pay for is that cable up in the cable tray.

And so when you go down in size from like the largest 15 networks down to the other 60,000 and networks, this idea of cross-connecting with each other individual network starts getting problematic because you've got an n-squared problem there that densely connecting with 120 other networks requires 120 routers ports on my router and 120 cross-connect bills from the data center. And so internet exchanges are this service provider for the service providers, where you essentially just have an Ethernet switch in one rack with a bunch of ports on it. And then anyone who wants to trade traffic with everyone else plugged into that switch can come to the Internet exchange and say, hey, can I get one port on your switch to get effectively direct connectivity to, you know,

100 other networks that I'm, for any one of them, I may be only trading a few megabits of traffic, but aggregated together, I've got, you know, a couple gigs. worth of traffic that I can send to the switch of the internet exchange, which then, you know, end-to-end scatters that traffic to where it actually needs to go to get to the rest of the internet. What is the Fremont-Kaball Internet exchange? So the Fremont-Kabal Internet Exchange, or F-C-I-X, or as it is properly pronounced, fuck-ix. I'm not even sure I heard that.

Sorry. This is an internet exchange that a couple of friends and I originally started as a joke and accidentally it became successful and a load-bearing part of the internet. Okay. So, I mean, you told me that it is very expensive to house hotel, have guests at vacation at Internet exchange hotels. How did you accidentally do this? So there's a very long backstory to this that we're not going to have time to get into about how one of my friends challenged me to set up my own autonomous system or like to be one of those 60,000 autonomous entities on the internet because it was going to be an impressive operation. and like I needed a rack in a data center for lots of other reasons. And so we eventually ended up in this situation where in this one aisle of the data center,

so in one aisle of one suite of a data center, we ended up with about five of us immediately adjacent to each other, each with our own autonomous systems in one rack. And, you know, it's like, well, I've got an autonomous system. You've got an autonomous system. clearly we should peer with each other to be able to trade any traffic that needs to go from my web server to your web server.

And so being right next to each other, we just pulled an Ethernet cable up and over the wall between our two racks, which once or twice you can kind of get away with, but when you've got six racks next to each other, that's 25 cables between your six racks. And the data center eventually notices and says, hey, guys, we didn't pull any of these cables,

and you didn't pay us for any of these. So you need to knock this off. And so we went, okay, okay, all right. You caught us. Kenneth's rack happens to be in the middle of this kind of block of racks. And so on the router in his rack, we'll set up one segment for this Peering Exchange, and everyone will just run one cable into that rack,

and we'll rip out the. other 20 cables and our end and squared problems becomes a crossbar in the Cisco 6506 router I was running in my rack. And so we were having dinner with the owner of the data center later and someone casually mentioned, oh, hey, so like, you know, you guys were complaining about all those extra cables we had. We fixed it. We started an internet exchange.

and Hurricane Electric, who was the data center that we were in, the owner of it, Mike Lieber, is one of the largest advocates for free and open peering. And so when he hears that a couple of smart guys and his data center has started their own Internet exchange, he got very excited about this and went to all of the salespeople for the carrier and the data center and said, position and sell this Internet exchange in our building as one of the person. for other networks to build into our building. So, okay, I want to back up just a little bit. You and some friends decided to have your computers talk to each other. Yes. And due to an overgrowth of cables, you decided everyone would talk to your computer and you would just pass messages like you were in fifth grade.

Yeah, exactly. But you're really only talking to each other at this point. Yes. And then you have dinner and some guys like, oh, you know what? That's really cool. And it's free and it's open source. We'll make this sexy and let everybody else talk to Kenneth's router. Yes.

And so the data center started pitching as part of the value add services in their building of not only to provide you power and cooling and security and, you know, access to these XYZ service writers that give you the whole. whole internet. You can also connect to this random guy's router in this one rack and from there talk to anyone else in the same building for free. Let me let me let me just stop here. I was there 3,000 years ago when the 6500 started coming out. They're not that fast. How much traffic are you exchanging? So the 65 guys talking to each other. It was just the five guys talking to each other, right? And so the 6,500 did not last very long. Okay. Because once you've got real actual networks coming in and saying, hey, like, we would like a 10-gig port on your exchange, please.

Exactly. For anyone who doesn't know, the 6500 is about the size of a very large suitcase, and a high-density line card on it has eight 10-gig ports on a single line card, which is about the size of a pizza box. And so very rapidly, we were faced with this problem of, oh, my God, we need better switch hardware than the $6,500 because buying this thing on eBay for $400, like, did not get us where we need to be here. And at that point, I was, you know, posting on Twitter, rest in peace about this and posting on Facebook about some of these shenanigans. And my godfather's son, the son of the son of. my godfather. Your family friend, you know, I've known him the whole time I was growing up. He reaches out and says, hey, I see that you're starting an internet exchange.

I work at Arista, which is like one of Cisco's big competitors building, you know, huge 50-teribit Ethernet switches. Would you like us to sponsor you and give you the switches you need to make the internet exchange successful? I love this. I love this. And there's no way you'd say no to that because it would be so much fun to, to, to, to get the good gear. I mean, somebody offers me a nice o-scope. I'm like, yes, please.

Okay. Yes, exactly. And so I met up for lunch with him. He handed me a pile of, you know, end-of-life obsolete switches to them that were still 64 ports of 10 gig. So, you know, it was able to move the traffic we needed, absolutely no problem. I then on Twitter, I guess, flex optics reached out and said, hey, we would. love to be the optical transceiver sponsor for an internet exchange. And so they sent me an entire case of the optical transceivers to put into the switches. And Hurricane Electric said, well,

you're an internet exchange in our building. So the first cross-connect into the internet exchange for every single member is also free. I feel like this is one of those stories where you start with a paper clip and end up with a house. Exactly. Exactly. So it started as a joke between the five or six of us and the one aisle. The right people hear about it and the right people, you know, know of the types of shenanigans I get into, and then eventually I end up with, you know, $100,000 worth of networking gear given to me for free and Internet Exchange with an annual OPEX cost of $260. And I'm moving 50 to 80 gigabits per second of the Internet. Where is that now?

I mean, do you still, where is it now? Yes. So it's still running. So we're up to about 120 networks in the building. And so we have about 120 networks that are tied into this fabric, which the peering fabric has now grown to the point where it's about seven switches, all just in a stack. A good way to visualize this is imagine a bowl of spaghetti and then just look down straight into a bowl of spaghetti, just yellow fiber running everywhere. It's absolute chaos. But are you still running off of donations and enough, like, you said 260, which is, you know, I would cover that.

That's amusement money. But if a line card goes down, that's inexpensive. Yeah. So, yes, it's totally a volunteer run, right? Like a lot of, you know, every year, like, we have new members that come and say, hey, like, you know, this is great. Like, you're adding a lot of value to our network. I'm getting a lot of value out of this. Can I just give you a couple thousand dollars?

And so, you know, when we have some sort of hardware failure that, like, I actually have to open my wallet and, like, go buy replacements for it, I just kind of have this floating pool of like a lot of enough people have given any, you know, somewhere between small and extremely generous donations. So that, you know, we're able to cover the costs of this thing. and it's really just, you know, seven Ethernet switches and a couple little virtual machines running on my own network. And so the work and effort on this thing is connecting a new member and troubleshooting that initial link up. Once they're on it, like, it just sits there and moves traffic for the Internet. And, you know, we don't have 24-7 on call because, like, it's just three of us. And so if the Internet exchange goes down in the middle of the night, it's just going to stay broken until one of us wakes up. But you're passing like a quarter of the downland capacity of North America.

No. No, no, no. We're only moving 50 to 70 gigabytes per second. The local carrier hotel facilities near us in South San Jose, it's moving like a quarter of the year. So they do have a 24-7 on call. All right. Good, good. Yeah.

Armed guards, that sort of thing. Yeah. Okay. So you accidentally did this, but you also made a DNS hosting service, which we don't quite have enough time to talk about. But it's another of these internet things that you can start small and suddenly it becomes very important. And you did that there too. Do you think there's a trend? Yeah.

Are you going to start your own cache experience? So I have, right? He did. Well, yes. Yes, right. I guess it is. So the FCIX Internet Exchange kind of then grew into the Micromere project where we're providing caching services for free software.

And as global was this DNS service that I built that then became successful. And all of it is really just a demonstration of like when people talk about networking, like, yeah, when you're in college, you need to work on networking. And it's not the like awkward, like, yeah, go. to the career fair and glad hand people and trade business cards. It's like really a lot of like finding and getting to know the right mentors and the right people that and then like demonstrating to them that like you are reliable and you're serious about learning what it is that you want to achieve. And so like having that sort of credibility makes it makes it so that like I'm able to

go to random service provider or not random, but friends of friends. service providers and say, hey, would you be willing to give me a virtual machine in Europe to set up this DNS service? And so I was able to build the NS Global DNS Authoritative Service entirely from free donations of virtual machines in just various places from friends or friends of friends. And so, you know, that's where it's like having something like Twitter, rest in peace, or like my mastodon of being able to this post about, hey, I've got this predicament, Like, who has the solutions to this just kind of sitting around and is willing to toss it in my direction for the amusement

of watching to see what happens. And now, after all of that, if you download VLC, there's a good chance you're getting from your mirror, right? Yes. So if you download VLC in North America, there's about a 70% chance that it's going to come from one of my servers. And many times I've needed to get an image for Ubuntu, and I've just gone straight to your servers

because I can't be bothered with the official servers. They're too slow. Right. Well, so, yeah, so when you go to abunto.com, you click the download link, they are routing you to one of these caching mirrors for them. And so we're about, depending on how you measure it about 20% of the download capacity for Ubuntu. And so you might land on one of our servers that's pretty fast. You might land on someone else's server that's not quite as fast. So, yeah, a lot of people have realized that Kenneth runs pretty good download mirrors. And so they'll just. go directly to one of those micromere site. So if you ever see at MM.fcic.net URL while you're downloading free software, that's coming from the same project.

That's coming from Kenneth's surfer. Yes, surfers. There's about 40 of them. So these turned out to be a little bit bigger, more successful, more work than perhaps originally anticipated. Certainly bigger.

I would not say that, I mean, a very important part of these sorts of projects is focusing on how do I not make it a lot of work? And so things like robust, you know, CICD pipelines and hands-off provisioning and, you know, playbooks and checklists so that like I don't have to, like, so it's like it isn't a lot of work, right? There's, it's a lot of, okay, I need to do the same thing that I've already done 15 times, right? I just plug the programmer in and click go on my build pipeline. line. And so it was a, it's been, these projects have been a harsh lesson in becoming acutely aware of

exactly where are the manual steps in any sort of project and how do you eliminate them. That's actually, I mean, there are a lot of times where that piece of information, do you identify the manual steps because you're like, oh, I've already done this or can you pre-identify them? it took a lot of experience and a lot of pain to get to a point where you're able to look on paper and systematically identify that. And so a lot of it was just the hands-on experience of, wow, I've already done this six times. And I accidentally did it five slightly different ways. And I would love this to not take 20 minutes anymore, but I'd love it to take two minutes. And so I go spend 150 hours developing a solution to save me 15 minutes every single time I go to do something.

And so it's not an easy thing to just look on paper and say, oh, that's where the bottleneck is going to be or that's where all the human errors are going to be. Yeah, and things change. Like, you know, it's a scaling thing, right? So, you know, I do this three or four times a year. I'm just going to forget how to do it because I can relearn it once a quarter. Right. And then suddenly something happens and you do it 30 times a year. Well, now that becomes much more of a, okay, I need to either become an expert in this or automate it somewhere.

Exactly. Why didn't you sell out? I mean, it is, there are a lot of people making a lot of money doing what you're doing as a hobby. Yes. And I get a lot of offers, as I will regularly get offers to buy various little projects off of me. And I've been nominally, a lot of the motivation for not selling out is that, you know, like, you know, update yodell.net is a project that I've, I'm standing behind with my credibility. So if I sell it to some sketchy organization that's going to continue serving updates for end-of-life Dell servers, I don't know if they're going to start serving malware off of it at some point. And I just don't want to ever run the risk of being associated with that.

I mean, I have the luxury and the blessing of having a job in tech so that, you know, like, paying my rent this month isn't something that is top of mind for me. So I don't necessarily need it. And in some ways, I arguably have sold out. I got a job at Nvidia selling shovels in the gold rush that is AI. So I feel like in some ways I kind of have sold out. sold out a little bit. But you haven't. You've made money off of the skills you've learned.

Yeah, I'm not going to call that. I'm not going to call that self. Okay. Well, it's, I really enjoy it. Like, I just love the infrastructure and, you know, how does things actually get moved from here to there? It's just such a satisfying and gratifying, like, looking behind the curtains and understanding enough to build it, that it's like, you know, people love to take apart, you know,

when you're, when your kids, you love to take apart, like your printer or something to figure out how it works, but what if you go take apart the whole internet and figure how it works and put it back together and get the futz around to your own little pieces of it? Are there projects you're working on now that are in danger of becoming one of these?

I'm just trying to figure out how to raise that yet. No, it's the running joke. It's like, oh, someone offered a whole kind of spear. He's going to go do it again. At this point, I'm really trying to restrain myself is I kind of my my dance tickets pretty full because each

one of these little projects like yeah I get it mostly automated and it's mostly hands free but you know there's a little bit of background chatter and it kind of they start adding up by a death of 100 paper cuts there and so I've been trying to

not think about any other world problems on the internet that like you know with a little bit of technical skill I can really you know start putting to rest And so, like, the network time protocol is kind of the one that I keep not be able to stop thinking about. Oh, please fix that. I'm really trying to.

Never work on time. Never work on time. Yeah. No, no. No one's got time for that. And so I've been trying to avoid that sort of thing. And so, you know, like, so this year a relatively new hobby for me as I got into coin collecting. And that refreshingly has absolutely nothing to do with it.

the internet at all. Given your history, there's a 1% chance you're going to start your own mint. Okay, well, actually... Well, actually. Not entirely incorrect, because I actually have minted my own coins before. And that was...

I went back to when Make Magazine came out with that awful make-crypto-coin. Right. Right, I remember now. Yeah. And so, you know, Dale comes out with this, like, make-crypto-coin, and, like, everyone in the make-crypto-coin, and, like, everyone in the maker community, like, unanimously went like, what the hell are you doing? Like, crypto is like the antithesis of everything make.

And so I sat down that weekend and went on Alibaba and found a custom arcade token vendor. And I drew up the artwork for a, hey, make something awesome. It has like the word make on the front of it. It's got like a mig welder and a pair of knitting needles on the back of it as like the of first and the reverse designs for it. And it turns out that if you want to press your own custom arcade tokens, the tooling die cost is like 60 bucks. And the minimum order quantity is 5,000 tokens.

And so for $600 all out the door, I managed to get 5,000 of these custom brass tokens made and shipped to me. And I was handing those things out like candy for a couple of years. And I was like selling them off. And so like I managed to demonstrate like, look, make magazine who's nominally focused on making things, put out this scammy crypto token thing that ultimately ended up going nowhere. And one guy with a couple hundred bucks managed to actually make something that physically is substantiated. And numerous people will hand, trade these things back and forth at, you know, maker beatups and events for like, hey, you made something awesome. And like I wanted to kind of like John Wick style show appreciated.

for what you've been achieving. That's pretty cool. That is very cool. I am supposed to ask you about trains. Yes. What should I ask you? So trains are what originally prompted me to become a mechanical engineer, which didn't last very long. Well, the mechanical engineering didn't last very long.

The trains are forever. Trains are awesome and amazing. The trains started as while I was growing up, my dad has always been a very large rail fan. And so we would, you know, during the 90s, Congress was always talking about killing off Amtrak. And so we made a point of doing a lot of family trips that were taking Amtrak long haul train rides, you know, up to Washington or out to Denver. And then, you know, we would go to the railroad museums like the one in Sacramento. Prento. And back in the 90s, all the museums and stuff would have these racks of trifold pamphlets of other tourist destinations. And so my dad found one of these pamphlets that was for the

Western Pacific Railroad Museum in Portola, which is very, very far north, like up in the trucky Sierra Nevada area, that has a locomotive. rental program. What? So if you want to learn how to drive a locomotive, they have this program where you can go and rent one of their locomotives for an hour and drive it around the museum property, and then you're done. And then you get a certificate and you can drive it wherever you want after that, right?

Does the certificate say, I'm a good conductor? Because I need it for some of my jokes. Oh, so unfortunately not. Because a conductor and an engineer are two very separate roles. And so to be a good conductor, you have to join the society and attend a couple, about 150 hours of training and hands-on experience to get your Breakman certification and then your conductor certification. And then you actually will be a certified conductor. But you still won't be able to pass that much electricity through your body. Exactly.

But so long story short, so my dad gets involved with this railroad museum and drags me along with him. And since the museum is a live operating full-scale historic preservation of the Western Pacific and the equipment that operated on the Western Pacific Railroad, the museum falls under the FRA safety guidelines and G-Corps. And so as a 12-year-old kid, I wasn't really allowed to do much on the hands-on operating 12 inches to the foot model railroads that much. But thankfully, there's a lot of adjacent activities to running locomotives and trains in that you have to maintain locomotives and trains. And so my summers growing up were spent at this railroad museum with these crafty old mechanic guys mentoring me. teaching me how to weld and how to take apart locomotives and rebuild them and, you know, service an engine that has 150 gallons of crankcase oil as opposed to the five liters in your car. And so I got to spend all of this time at what is a really fantastic hands-on museum that also, like, we fell in love with the place because of this rental program.

It's called the Run a Locomotive Program. and then I've just kind of always been involved in that place driving trains. That is extremely cool. How many times are you allowed to, you know, blow the train horn when you're renting? So we do have a public crossing on the typical route, and so you need to blow the horn every single time that you're crossing the road crossing, And so you will get to blow it many times. And that's museum.wp.lives.org?

Yeah. So wplives.org. And then I think, yeah, that's why I think the museum.wp.lives.org is the main landing page for you want to come here and see what is that. Book now. Okay. I think that's all. I have other things to do now. I have so many more questions for you, but we are almost out of time.

Chris, do you have anything outstanding before I close up? No, no, I think we'll have to leave that for another day. Absolutely. I don't know how other things we could talk about. Kenneth, do you have any thoughts you'd like to leave us with? Yeah, I mean, I think that something that everyone should really take away and gna on is there's a lot of things in the world that you kind of take for granted and don't engage with, right? And so coins are something that I've just become infatuated with recently.

And it's one of the things that you handle on a day-to-day basis. But like stop and think about like what does every single symbol on these things mean? Or like how do traffic signals work? Or what is the logistics of like, where? where my coffee comes from at work. Like, I think there's a, the entire world is so fascinating and has so many neat things going on with it that other people are making happen just because it's a mundane part of their worlds.

And I feel like particularly now in this day of people getting into these smaller and smaller silos of understanding and reality, make a point of really noticing what you take for granted and like, what's, what's just kind of. happens that you don't think about because there's, it's such a neat world out there that has so many cool little things that become stupid little fun facts that you can whip out at maker parties and events. Our guest has been Kenneth Finnegan, senior solutions architect at NVIDIA, and technical director at Fremont-Kiball Internet Exchange. Thanks, Kenneth. It's good to talk to you. Thanks, guys. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon supporters,

coffee supporters. Thank you to our Slack group for their questions. And of course, thank you for listening. You can always contact us at show atembedded.fm or hit the contact link on the Embedded FM website. And now a quote to leave you with. This is Susan Sontag saying something very close to what Kenneth just said. Do stuff. Be clenched. Curious. Not waiting for inspirations shove or society's kiss on your forehead. Pay attention. It's all about paying attention. Attention is vitality. It connects you with others.

It makes you eager. Stay eager.