The Changelog: Software Development, Open Source - It's not always DNS (Interview)

Episode Date: March 8, 2024

This week we're talking about DNS with Paul Vixie — Paul is well known for his contributions to DNS and agrees with Adam on having a "love/hate relationship with DNS." We discuss the limitations of current DNS technologies and the need for revisions to support future internet scale, the challenges in doing that. Paul shares insights on the future of the internet and how he'd reinvent DNS if given the opportunity. We even discuss the cultural idiom "It's always DNS," and the shift to using DNS resolvers like OpenDNS, Google's 8.8.8.8 and Cloudflare's 1.1.1.1. Buckle up, this is a good one.

What's up friends, this week on the changelog we're talking about DNS with Paul Vixie. Paul is well known for his contributions to DNS and agrees with me on having a love-hate relationship with it. We discuss the limitations of current DNS technologies and the need for revisions to support future internet scale, the challenges in doing that. Paul shares insights on the future of the internet and how he'd reinvent DNS if given the opportunity. We even discussed the cultural idiom, it's always DNS, and the shift to use DNS resolvers like OpenDNS, Google's 8.8.8.8, and Cloudflare's 1.1.1.1. Buckle up, this is a good one. A massive thank you to our friends and our partners at fly.io It's simple. Launch apps

near users. Fly deploys straight from your source code into micro VMs that run on their hardware in 30 plus regions on six continents. You can launch an app for free today at fly.io Well, friends, I have some good news for you. It is launch week once again for Sentry. And I'm here with Rahul Chhabria from the product team at Sentry.

So, Rahul, what can you tell me about the launch week this year for Sentry? In March, we're making a huge investment into our product platform. We're trying to make it faster, better. In November, we shared a sneak peek about our new metrics offering. So now developers will be able to define custom metrics they care about and monitor how quickly their app is responding to the business measures they have to be accountable for, plus also like the customer experiences that they've committed to. that's gonna be available in an alpha people can sign up to get access uh we'll turn it off for them like in a couple days once they write in and they can get going right away now on top of that it's like we are looking more at how

do we make the product smarter now i know the world is talking about ai and ml and they're all solving like we think like entertaining problems, but Sentry is taking a more thoughtful approach to it. We are trying to look at what is the developer trying to do? Like our goal is not to have you sit in Sentry all day long. Our goal is not have us like be a tab that you need to keep open. Our goal is to have this be a tab you open when something is wrong, give you the information you need right away, tell you how impactful it is to your user base, and if you should care, and if it is something you should care about, here's how to fix it. So we're taking a broader look at how developers use the product. Where's the noise they're seeing? Are they seeing repeat

events? Are they seeing things that are not critical rise to the top and have to automatically resolve them or ignore them? So we're going to make Sentry a little bit smarter with artificial intelligence to give you a more prioritized view of the issues that really matter. So you can solve them quickly and move on and not be distracted by random rage clicks that are, you know, just ghost issues. Those are the two major things coming out. It's like thinking about more like defining the metrics you care about, also figuring out ways to like organize your issues so developers can actually solve those problems faster. And then we're also working on a few features for our mobile developers. Like Sentry is a platform that works with any technology you want.

But for mobile developers, there's always been this like, wait a second, there was an error. But hold on, let me go dig up this device and see if I can recreate it. I can't recreate it. Okay, let me go look at the stack trace. Like it's definitely something there.

I'll just fix it, push release. And hopefully those are like the crash rates go up and the crash for user rates go up. But there's, it's always like this idea, like I the crash rates go up and the crash for user rates go up. But there's, it's always like this idea, like I still need to figure out like where does that bank of used old devices for someone running, you know, iOS 13 on an iPhone 11 somewhere.

So we're giving them the ability, we're previewing the ability for mobile developers to actually see what happens on an end user session. So that way there's no question about the problem or the latency issue and building out more performance capabilities so they can see exactly how fast their app is performing.

Those are the three big things we're planning on talking about aside from core platform announcements, some integrations and cool partnerships we're working with. Yes, the big investment in machine learning and artificial intelligence. Okay, Sentry's launch week happens March 18th through the 22nd.

Check the show notes for a link to the launch week page. We'll be showing off new features, products. You can tune into their YouTube channel or Discord daily at 9 a.m. Pacific Standard Time to hear the latest scoop. Or if you want to get swag along the way, enter your email address at the page we'll link up to get swag all along the way. Or join the Discord, whatever works for you. Head to Sentry.io. That's S-E-N-T-R-Y.io. I'm sure they'll link it up somewhere.

Or check the show notes for a link. While you're at it, use the code CHANGELOG to get $100 off the team plan. Again, use the code CHANGELOG. Go to Sentry.io. We had Alan Jude on the show to talk for you, BSD. And he said to us, hey, if you're into DNS, you should follow Paul Vixie. And I followed that to Paul Vixie on LinkedIn. I said, hey, Paul, we would love to talk to you. I'm into DNS.

Adam, are you into DNS at all? Oh, yeah. Yeah, I love DNS. Love, hate DNS. And thankfully, Paul was gracious enough to give us some of his time. So he's here now with us. Welcome, Paul. Welcome to the show. I'm here and happy about it. Do you like DNS, Paul? Yeah, what's your feelings? You know, I'll bluff it like you were just saying, but what I want to point out is none of the technologies of the Internet were designed for the scale it's now seeing. So I'm wondering why we ran out of address space

or why routing table ingestion is such a problem and why fragmentation doesn't work and all the rest of that, because it was essentially a laboratory toy. It was built by a bunch of government contractors. They communicated with each other and did that perfectly. But then it got into a fight with commercial protocol suite called OSI. And OSI was very much a telephone company creation. They were going to bill us by the pillow character,

kind of in the style of X.25. And a lot of people said, we don't want that future. We want a network that the world itself built for itself. And it turned out that the internet protocols were just far enough along we'll keep and take it. DNS being an example, but TCP also, IP also. Mind, we didn't have the hardware to support encryption, so we just didn't have it.

We didn't even have a placeholder for it. So there's been this periodic, let's change everything. Here's my rototiller, let me go in there and turn it this periodic, let's change everything. Here's my rototiller. Let me go in there and turn it back to raw dirt and change everything. We probably should have done that with DNS. I am maybe the person who did it the most, but it needs it again. And it's now too big. There's no way to have a flag day. So we are stuck with a bunch of things in DNS that should have been revised with scale and work. And then we inevitably have some opportunistic revisions that were back patched in by somebody who had a business case that required them and they got it done in a way that

now we're all living with that. And so some of the instability that we're seeing is because it's too old and some because it's too new. But either way, it's chaotic. So if you read Eric Raymond's book, if you go on the bazaar, this is a bazaar. What happens if we don't make these remedies? What happens to the internet that we know and love or the connectivity we have with LAN, WAN, etc.? Yeah, so I'm not sure that any end user will experience the painfulness, but I think every deployer, every internet operator, every innovator, every implementer of protocols has always

sort of felt the pain of, you know, gee, I need to operate infrastructure. It needs to be able to support the following needs. I need some software to do this part and some software to do that part. You might even write some of your own software. Then you start looking at the specification and you find that it's incomplete. You go looking at the other implementations and you find that they're incoherent and inconsistent with each other. What we actually have is a best effort system at each other. And that what we actually have is a best effort system at global scale.

And so there's always something that is sort of a threat to stability or a threat to our profitability or our ability to go home for the weekend and visit our families. But yeah, those people like to live that way. So we're not going to treat this as an emergency. But let me give you a very specific example. It has to do with IT packet size. OMS was originally a UDP protocol. It has grown beyond that. And some of that's controversial.

And we've talked about that a lot. But as a UDP protocol, it meant there was no endpoint state. In other words, the kernel of the initiator where the questions were coming from was not trying to remember anything like a TCP session or port numbers or any of that. It was just saying, okay, you sent a packet. It's gone, and I have no remaining burden. And then the response comes and gets delivered to you. And again, the kernel had no state burden. And that was very necessary when the fastest computer on the ARPANET was a BAC 750, which was about 450,000 instructions per second. It did not want to be putting extra burden anywhere. It had to be as simple as it could be. It gave it a certain austere beauty.

But the trouble with an EDP is if you want to send something, you want to send a response, let's say, that is bigger than whatever your network can contain. Most of us are on Ethernet. Ethernet is still 1,500 bytes as the maximum transmission unit. And so if we want to send more than that, we can't. We have to make a choice. And every choice we could make will be a bad one.

One choice is to truncate it and say, well, this is what will fit. And then a little indicator telling you, there's no ask this question, that the answer is incomplete. And then you, when you receive the answer, you could try and be intelligent and say, well, I see it's incomplete. Let's see what's there. It might be enough that I get work done anyway. That's not what happens. What they do is they say, ah, it's incomplete, so I'm going to start over again with TCP. TCP doesn't have message length limitations, but it requires kernel state. It requires three round trips and a minimum of seven packets to exchange one question for one answer. And that's crazy talk.

That is an awful lot of network state and overhead just to ask a question and get an answer. So what you really want to do is be able to answer all questions in the size of a 1985-era Ethernet packet at $1,500. And that's a terrible trade-off. So what we'd prefer to do would be to send a larger answer. And the IP protocol permits this. It permits you to say, here is a datagram. It's an IP datagram. And it won't fit in one Ethernet packet. So it's going to get chopped up into several different segments, basically, or fragments. And you'll be able to reassemble them on the far end the internet since 1970 until now. And pretty much anything that wasn't necessary, we need fragmentation, so let's figure out how to make it work. We want to find a quicker path to get into Sloop tonight to read about path into you discovery.

Okay. But it doesn't work. And later with IP version 6, which has been trying to take over for a little over 20 years now, it took a different approach. It said, yeah, we're not gonna fragment packets inside the network. Fragmentation is necessary, it has to be done at the sender. And the sender, of course, since it can't discover

the size of a packet that would get through, doesn't know what size to use. And that in turn means that no receiver is ever going to have the ability to reassemble fragmented IP version 6 data branch because they were never set, they were never tested, they were never deployed. And so we're just kind of stuck. We're way out on a limb. We can't send messages that are large enough to contain even the current set of answers with DNS security signatures, let alone what will come in the post-quantum world. And we all know this. We're in the river. We can hear the waterfall. We know we're headed

to the waterfall. But we all have important things that we have to work on right now, or we don't even make it to the waterfall, so we don't. And as we get closer, you're going to see newspaper headlines about yet another Y2K style debacle for the whole industry to worry about together. When do you think that will occur? Fortunately, quantum computing is always 10 years out. If that changes and we end up needing to have post quantum crypto so that it remains impossible to factor large numbers and it remains impossible to store and decrypt information, then we're going to have a problem because we'll be trying to move toward a type of crypto that simply won't fit EDPDP. And we won't have

fragmentation as a backstop. We're just going to have to do TCP with everything. So that means there's a tipping point. If we get close enough to real quantum computing to where we need post-quantum crypto, we're just going to be using TCP for all DNS, except the part that doesn't get the memo and therefore does not switch, and therefore doesn't work or doesn't adopt post-quantum crypto at all. And I don't want to sound like this will be an emergency for the world. Well, I think it was not an emergency for the world. It was just kind of a big deal on the newspapers but there's always going to be something like this it's only in the case of the internet that we do it to ourselves do we need a new internet because i have to do it jared silicon valley have you watched that tv show paul by any

chance no thank you paul thanks no never would you are you not a fan would you never do you just not watch tv is there a reason why you haven't watched Silicon Valley as a TV show? No, I watch a lot of TV. Okay, cool. Let me just give you a picture. It very much simulates and satires a lot of the last 10 years of Silicon Valley. That's why it's calledahn Valley. And in this TV show, Richard Hendricks creates an algorithm that compresses something so well, it does like a 4.8 on the Wiseman score, for example, like compression we've never seen before, for example, maybe five something. I don't even know. It was, it was massive. It was a breakthrough. And so he had this idea to create a new internet. And so the whole show is essentially about stumbling into this compression this algorithm creating this platform and the platform really was eventually a new internet do you think

we need a new internet is that what tcp will offer but tcp requires a handshake right so it requires more than udp which is just sending packets blindly in a way right how do we create a new internet if you think we need a new one? Well, the cheeky answer to your question. Give me the cheeky version. Yeah, I like this. Because I don't know what technology we will be using 50 years from now to do global commerce and email and messaging and everything else.

But I do know that it will be called the internet. We're not going to rename it. So some shorter term examples of how that might work, and you can see in the web, and the web community is, they're large, they're new, they're young, passionate. In many cases, they don't understand that the internet is still here. They still have an internet, but we have a web that's kind of built on top of it. And so they look at some of the limitations here and they just sort of wave their hands at them and say, yeah, we don't need that anymore.

It's clearly outmoded. And this audience is certainly familiar with HTTP and therefore HTTPS. Many in the audience will know that something called HTTP-2 was created 10 years ago to deal with some of the limitations of HTTP-1. In particular, they wanted to have multiple objects in flight kind of interleaved with each other, rather than a whole bunch of connections in parallel, which creates other resource exhaustion problems. And this was cool. This was very cool stuff, but it still lived on top of PCB. And the trouble that you'll have living on top of PCB is it's reliable. It is a reliable stream protocol.

Every octet that is transmitted will be received in the order that it is transmitted. So if there's any congestion-related loss or any other kind of loss, then that part, the part that was lost, will have to be retransmitted. And until it is retransmitted, you won't, as a receiver, be able to receive anything that was sent after the part that has to be retransmitted. It will all just sort of sit there in various queues throughout the pipeline until they'd be delivered to you in sequence without loss. And that'll mean if you've got HTTP2 and you are interleaving multiple objects, lose one TCP segment of one of those objects,

all of the transmission of the interleave objects will also be delayed while the stream reassembles itself and gets back into synchronization. So now we have an alternative to that. That was not understood as a problem when HTTP2 was being developed. Now it's broadly understood. And so now they're developing HTTP3, which also is called QUIC, U-U-I-C. And what is amazing about this is that it lives on UDP.

UDP. And that means it still interweaves, but there's no head of line blocking. If you lose some packet of some object, yeah, that'll have to be retransmitted. Other things can still go on while that's happening. Now, if you wanted to squint just right so that you could have your wish about how to interpret this, you could say that each of those is a reinvention of the internet because it is a fundamental change to the thing that we do mostly which is to look at pictures of cats. And, you know, inevitably, in the last month or so,

there's now a proposal that says, yeah, QUIC is not the be-all and end-all, as it turns out, because a lot of networks block UDP. Hey, welcome to my world. So they've proposed making QUIC live inside of TCP. Kind of missing the original point of why HTTP3. I was going to say that was the big idea was not to do that, right? You know, if you have, and I, you know, I was a 20-something once upon a time and, you know, I came in and I wanted to reinvent everything and I didn't necessarily know the history of

why things were the way they were and what would be the hard part of reinventing it. And so, I expect that every new generation of 20-somethings for all of our humanity's future will always do that. And so, that's what they're doing. They're learning bit by bit, piece by piece, why the problems that they waved their hands at were more difficult than they realized. And that we weren't just idiots. We didn't sort of choose what we chose without knowing what the alternatives were.

That's okay. You know, I raised a generation of teenagers. Yeah. Unfortunately, it's kind of like when you have a young child and the stovetop is hot and you tell them that it's hot and it's going to burn them. And some kids will just be like, OK, therefore, I'm not going to touch it. Very few, though. Most of us have to actually touch it for ourselves and learn the hard way that what that person said was correct because they have to experience it. And so we do reinvent things.

And I don't know any other way, I guess. I guess maybe reading the history books. Maybe if we had better history, I think these conversations are hopefully helpful to those who are paying attention. But if you are going to reinvent DNS with all your experience and hard-earned wisdom, maybe just give like a 30- second for those who haven't been exposed, just a 30 second of how DNS works, broadly speaking. And then you can dig into some of the details of what you would do differently if you could actually just start fresh today and get global adoption, like everyone's going to do it. So you don't have to worry about that part, which is actually like you described, like the impossible part, right? Yeah. it turns out rebuilding the airplane in flight is hard.

There's only certain changes you can make. But if I didn't have that constraint, what would I do? Well, first, as to DNS, it is a request-response protocol. It is eventually consistent. So the authority data, which is edited and published by whoever owns a certain zone. Like I own redbarn.org, for example. So everything that ends in.redbarn.org comes from me as an editor and publisher. And some data changes. I renumber something, I can publish the address in my zone file in the already data, but not everybody is going to notice that

right away. If there are copies of the old address out there, they will have to time out because without a cache, if every question from every end user always had to go all the way to the authority to get answered, we could not have scaled nine orders of magnitude during the lifetime of the DNS. That hash is absolutely crucial. But it will eventually, there is one source of truth, and if you have some stale data, it will eventually work its way out. And that's kind of generally the way the system works. Now, the specific things that are causing us problems are mostly in the representation. In other words, the binary format of these messages turns out not to be as extensible

as we'd like it to be. And so whenever we're talking about some way in which DNS needs to serve a new purpose, there's always this question, well, where are we going to put that? How are we going to express that? Will it fit? Will it be ambiguous? Can we find a way that old clients will not be confused by new answers? And so forth. So that's where we spend probably 80% of our time as a DNS technical community. And to understand one trivial example, look at internationalized drone names. We started out with just a bunch of United States of America contractors connected to this network and they all spoke English and they all used ASCII. Well, not only some of the IBM people were using. They had converters. That was a well-understood problem. And so, it all made sense. We weren't going

to have company names that had whom lots or other special characters, and they certainly weren't going to be in kanji. It's all going to be just too excessive. But to be a global internet for all of humanity means you have to outgrow that. It's just not reasonable to ask a bunch of people who have no other reason to learn English to learn English so that they can type in domain names and represent their own family names and school names in English. That was seen as maybe not an oversight, but certainly something that had to be corrected. Well, it turns out that that ASCII assumption was built way, way down deep into DNS.

And the way it mostly appears is case insensitivity. An uppercase A and a lowercase A mean the same thing. They're carried differently, so you can see what it really is in the authorities' data. For example, 3Com.com, when they created their domain name, that C really had to be an uppercase C because that was their trademark. So they wanted 3Com to be the number three, uppercase C, little o, little m. So that all works. But it is the same name as 3Com without any capital letters or 3Com with all capital letters or whatever.

And that business of case folding where 26 of your symbols are semantically equal to 26 others of your available symbols meant that there was just no way to add something like any of the international character sets. You just couldn't do it. So we had to do this whole took a decade. We had to do this whole thing to do name prep. We would take that data and essentially convert it

into Base64. And then the far end was seeing Base64 names when they were expecting ASCII, which it was. But they would then often just display it. You wouldn't see all this base 64 gibberish in the browser bar because every client in the world had to understand that if it's an internationalized name, it'll look like this. Here's how you unpack it.

And so something like that has come to DNS every five years during my decades with it. So to answer your question, I would start with an extensible encoding. I don't know if I'd use JSON, but it would look like JSON. It would be like JSON or maybe the binary version of JSON. It would be something that was not designed to be fast for a VAC 752 encode and decode and was a little bit more flexible in terms of future representations. I would cut down the number of assumptions

and constraints that are designed into the encoding itself. And if you did just that, that would probably take half the pressure off of the future. Who pays for this? You said it took a decade, this work you did. I don't expose your employers necessarily, but somebody's paying for the progress of the internet.

Who pays for this progress? A lot of companies see it in their best interests. They need a better internet to provide more value to their customers. And so they send people to the normal engineering task force meetings where these protocols are debated and ultimately ratified. They will often fund an employee to work on what is now called open source software. Although when digital equipment was funding me to work on it,

that name hadn't been created yet. But if you're in the tech business, you have to be an innovator here. You have to have a seat at the table, and you do that by contributing. There's no way you could get an internet without openness. You can't have openness without a big tent. Aside from that, one of the non-profit companies that I started and ran for a while is Public Internet Systems Consortium. And they still exist. I left them in 2013, but I am on very good terms with them. And they're still very much in the thick of all of this. And we used to simply accept contracts

either from companies or from government agencies. And one of the DARPA, Advanced Projects Research Administration, the DOT, paid us to add DNS security to the BIM software. And it's a normal-looking software development contract. We want these features. We want them by these dates. And this is the milestone of when we will pay you how much.

And after we signed that, then everything went back. And that's the way a lot of this stuff got done back in the 90s and the 2000s. Because there were things that had to be done where there wasn't critical mass of commercial interests who all saw, yes, that's vital. Or if they did, they said, it's vital, but I don't want to work on it in conjunction with my competitors. But they would all be willing to go to, let's say, the W3C or the Apache Foundation or us, and say, look, you're in Switzerland in this situation. As long as you do what the IETF has decided will be done, then we're totally ready to pay for it.

So, I mean, this is all the norm now. It was pretty controversial in 1990, but it's the norm now. What's up, friends? I'm here with one of my new friends, Zane Hamilton from CIQ. So, Zane, we're coming up on a hard deadline with the CentOS end of life later this year in July. And there are still folks out there considering what their next move should be. Then last year, we had a bunch of change around Red Hat Enterprise Linux that makes it, quote, less open source in the eyes of the community, with many saying,

Relo's open source, but where is the source, and why can't I download and install it? Now, Rocky Linux is fully open source, and CIQ is a founding support partner that offers paid support for migration, installation, configuration, training, configuration, training, etc. But what exactly does an enterprise or a Linux sysadmin get when they choose the free and open source Rocky Linux and then ultimately the support from CIQ if they need it?

There's a lot going on in the enterprise Linux space today. There's a lot of end of life of CentOS. People are making decisions on where to go next. The standard of what enterprise Linux looks like tomorrow is kind of up in the air. What CIQ is doing is we're trying to help those people that are going through these different decisions that they're having to make and how they go about making those decisions. And that's where our expertise really comes into play. A lot of people who have been through very complex Linux migrations, be it from the old

days of migrating from AIX or Solaris onto Linux, and even going from version to version, because to be honest, enterprise Linux version to version has not always been an easy conversion. It hasn't been. And you will hear that from us. Typically, the best idea is to do an in-place upgrade. Not always a real easy thing to do, but what we've done is we have started looking at and securing a path of how can we actually go through that? How can we help a customer who's moving from CentOS 7 because of the end of life in July of this year? What does that migration path look like and how can we help? And that's where we're looking in ways to help automate from an admin perspective. If you're working with us, we've been through this. We can

actually go through and build out that new machine and do a lot of the backend manual work for you so that all you really have to do at the end of the day is validate your applications up and running in the new space. And then we automate this to switch over. So we've worked through a lot of that. There's also the decisions you're making around, I'm paying a very large bill for something I'm not necessarily getting the most value out of. I don't want to continue down that path. We can help you make that shift over to an open source operating system, Rocky Linux, and help drive what's next, help you be involved in a community and help make sure that that environment you have is stable. It's going to be validated by the actual

vendors that you're using today. And that's really where we want to be as a partner from not just an end user perspective, but as an industry perspective, we are working with a lot of those top tier vendors out there of certifying Rocky, making sure that it gets pushed back to the RESF, making sure that we can validate that everything is there and secure that needs to be there and helping you on that journey of moving. And that's where we CIQ really show our value on top of an open source operating system is we have the expertise. We've done this before. We're in the trenches with you and we're defining that path of how to move forward. Okay, ops and sysadmin folks out there, what are you choosing?

CentOS is end of life soon. You may be using it, but if you want a support partner in the trenches with you, in the open source trenches with you, check out CIQ. They're the founding support partner of Rocky Linux. They've stood up the RESF,

which is the home for open source enterprise software. The Rocky Enterprise Software Foundation, that is. They've helped to orchestrate the Open ELA, Thank you. dot org and of course if you need support check out our friends at ciq at ciq.com Is it infeasible to design and spec and provide a reference implementation of a DNS2 and then just let people opt into it somewhere the way you'd upgrade to H2 over H1 and do the typical campaign of like, I'm thinking about Let's Encrypt, you know,

like they had a big effort to encrypt all internet traffic and took them a few years. They're definitely not at 100%, but they've hit critical mass, I think, at this point, and they've had lots of success. It seems like with the right core entities involved and a good spec and implementation, that's the kind of thing that you could get done,

don't you think? I would like to say so. You know, I tilted windmills. I've never taken on something that was going to be clearly possible. I don't think I'm ambitious about the sure thing. And at the time that I left nonprofit service and went back into the commercial world in 2013, I was trying to do exactly what you say. I left nonprofit service and went back into the commercial world in 2013. I was trying to do exactly what you say.

I was trying to figure out, well, if there were a replacement to the DNS protocol that you could opportunistically adopt. So you as an information publisher, you as, let's say, a smartphone maker, creating a lot of DNS requests, trying to fetch a lot of information, and just try to enumerate all the different entity types in the entire DNS economy and say, well, how could they speak both for some transition period, which by this time is 30 years, but anybody who was an adopter would immediately get some improvement that features better performance, less resources, whatever it is, so it'd be an incentive to adopt. In a way, that's how HTTP2 and now QUIC or HTTP3 are doing it. I was really tinkering hard with this at the time that I just kind of said that's it. I've done 18 years of nonprofit service.

I need to go think about my retirement and went back into the commercial world. But I'll tell you the biggest impediment to this is that everybody wants to be the creator of that. You've heard of crypto and Bitcoin and digital currencies. All of them followed each other along this model of, we don't like fiat currencies, we want to be able to have money via a matter of private contracts without governments being able to either inject money into the economy or control interest rates or whatever. And the problem is nobody came in later and said, it's a really good idea. I'm going to adopt whatever one is most popular. What they did is they came in and they said, whoever owns this is going to be a trillionaire. And they created their own. It's like Linux

distros. Why would you use the existing ones and invent your own? So some of those very same people have come along and said, yeah, we could use a blockchain to encode domain names so that takedown was impossible. So that no matter whose trademark you were infringing or whose intellectual property you were infringing, there'd be nowhere to target a lawsuit as you, the person having standing in the court, all about the lawsuits, it'd simply be nowhere to go to request takedown. And wouldn't that be better than having government? I'm not sure it would be, but I'm pretty sure that that type of anarchy might do more harm than good. Well, I didn't discuss that. It would be a very happy opportunity.

For sure. But nobody said, hey, let's all get together. Let's work together. Let's create something like that so that we'll have critical mass. Then we'll all be able to live in a world that has those features, those capabilities. No. Everybody said, whoever cracks this nut is going to be a trillionaire. And so they all launched a parallel, and they all came to a stupid debt.

Yeah. I mean, I'm following that logic. However, I think HTTP 2 and 3 is a better corollary to DNS, isn't it? Where there's, I mean, with cryptos, there is the cash incentive of being really rich. But it seems like with dns the incentive is if your company makes money off the internet the internet works better people get their names resolved faster more securely etc etc with less load and everybody wins like why do you want to

be the inventor of that necessarily couldn't we all just play nice why can't we all just get along that's a very uh compelling vision that you're painting there so poetic jared but the fact is depending on when you were born you see certain parts of the current system as necessary or terrible and getting everybody to just agree on sort of what use cases should it support and what should it look like, right? If you get into a group of web people, they're going to say, yeah, the web can just do this. We're just going to do this over HTTP, or we're going to add extra headers,

so-called meta headers to the HTTP above the body to say, and this is where all the DNS security information is. There are several existing standards. That stuff doesn't work well if you're not otherwise in need of the web. It's just you trying to access the file server at your office. You might not have been planning on using web protocols, but the web people think that you should. And the people who don't agree are going to say, well, no. And so I don't imagine that we're going to have another unified vision. Okay. And let's just say, hypothetically, DNS2 is a possibility. It doesn't matter who's writing it,

who's speccing it. It doesn't have to be you continuing your retirement or whatever you're trying to do as to not be involved in the long-term of that if you want to. But at what point does this not become just software but hardware? Like Nix, how much of the hardware layer has to change to support the new software layer? Is it just simply all of Cisco routers,

all of Unify, Ubiquiti routers? How does that trickle down into hardware manufacturing and just the hard stuff, really? Hardware is the hard thing to change. It's what's called hardware, not software. How does that work whenever you want to do something like this, that you have hardware that also has to follow and support? I think that that problem would be avoided. Earlier I talked about how in the early days of the web, we didn't have crypto that was fast enough to support the crypto hardware fast enough to support HTTPS. And so we just sort of didn't do it until you needed it. And then you bought some hardware to assist you with that.

And nowadays, I'm thinking about a number of different NIC vendors who make PCI-X16 cards that you can stick into your server that will do a lot of the offloading for you, checksum computation, segmentation, reassembly, and so forth. If you need to operate at 100 gigabits a second, or coming now soon, 400 gigabits a second, if your CPU has got other things to do than sort of shoulder every octet through the bus, then you can get a very smart NIC and driver support for it.

And, you know, as we all know, Moore's law will continue giving us its annual gift. And the time will come when you don't need that hardware anymore and you're just doing that with your CPU because you have so many cores, so much cache, and so forth. So I think any protocol in order to succeed at all would have to be the kind of thing, be very difficult in the short term. But in the long run, it'll just be the way everything works. And so I don't see that hardware support is going to be called for in any of this. What will be called for is some hard decisions. I mentioned earlier fragmentation and it doesn't work and it got worse on an IPv6. It worked a little bit and IPv4 doesn't work at all now.

And the packet size we use on our Wi-Fi is Ethernet cells, IEEE 802.3, and that is 1,500 off chats. And I knew one of the people whose name was on the Ethernet patent. He was my mentor at a mini-computer company back in the late 80s, his name was David Boggs. And I had an opportunity to listen to him talk about the old days, talk about being in Xerox and inventing Ethernet on the team that invented it. And so I'm in a position to know secondhand that the intent was the packet size would continue to grow so that as we got faster networking, we would also get large packets. And he was a genius in a lot of ways. I miss

him every day. But his idea about this was every time the clock rate gets you 10x, in other words, you go from 10 megabit to 100,000 gigabit, or I guess gigabit to 10 gig and so forth, every time you get 10x of clock, you should probably give about a third of that packet size so that the number of packets in a given unit time doesn't also go 10x. But your packet count and packet size each go about a square root of 10, around three. And had we been doing that all this time, a lot of things would be simple that are currently very hard. We certainly wouldn't care if the fragmentation didn't work, if the package we could send had gotten larger over time. But they didn't. And the reason they

didn't is that the internet market relies on sort of backward compatibility. When somebody adds 10 gigabit networking to their office network, they don't make everyone switch it once. They just say, new ports will be 10 gig, but the old ones will still be 1 gig, and we're going to run a network bridge, a layer 2 bridge, to connect the old one to the new one. And that won't work if the packet sizes on the new one are so big they can't be bridged backward

to the one that made the market exist in the first place. So Ethernet is effectively trapped at 1500-odd bits for all time to come. Yes, a lot of us have turned on what we call jumbograms, so 9100 bytes, which turns out to be a very convenient size to get 8K plus enough room for all the inevitable headers and crap that you put around your 8K. So if you're running jumbograms, your NFS is going to be faster. Your SMB is going to be faster. Everything you do is going to be faster. You just can't use that when talking with people outside your own house or your own

campus because there's no way to discover whether your ISP can carry use that when talking with people outside your own house or your own campus because there's no way to discover whether your ISP can carry packets that big or whether the far end. So because we don't have that, because we didn't do logs inevitably caught with the intelligent obvious thing that everybody would do, you have one third of your clock wounds to packet size. Anything we do with DNS in the future, it's going to have to take that into account. That in turn means we'll be making the assumption, I guess we could probably send about 1,400 bytes plus

room for all the headers and stuff that is added. Now we've got to find a way to connect several adjacent packets together so that we can do essentially application level fragmentation or else we've got to deal with the handshake overhead so i predict knowing the idea culture as i do if we start now then within no more than four years we will come to an agreement that central issue no less than four years huh yeah well i mean that is a tough one i mean unintended consequences right it's very convenient to be able to incrementally adopt

or incrementally upgrade a network i mean understand why it got stuck there because some networks are so large it's just financially infeasible to ever upgrade if you do it all at once like it has to be done incrementally and so what are you gonna do forklift upgrades are very hard to argue for yeah it sucks that we're stuck though we're just stuck right there for and it's all because of the wiring basically right like how hard is it to really change ethernet in a building well not just the wiring but all the devices in between. They have to all support the larger frames. Well, even the wiring alone, just like going from like Cat5 or something in its predecessor to 6 to carry more load even.

You can transmit Tinkabit over Cat5, but it's not going to be reliable at length. You'll have interference, you'll have packet loss, stuff like that. Well, so I think you're confusing two issues. If there is a Cat5 link somewhere in your network connecting one switch to another, and you are using Cat7 at every endpoint, and you have endpoints that are trying to communicate with each other,

and they all see a 10-gig network, but there is this Cat5 in the middle somewhere. Cat5 is probably running at 1 gig, and so it simply won't be a bottleneck. And that's where the wiring will hurt you. But it turns out that's a solvable problem. Map out your network, then move all bottlenecks. Get the core of your network upgraded first before you start adding endpoints that want to operate at the same speed. But crucially, for this topic, the packets will

look the same. And it's just a matter of the bridge. Yeah, I understand we've got slightly different encoding on the Cat5 cables versus the Cat7. The Cat7s are using all four pairs and things like that, but that doesn't matter. That's an active bit of electronics that can just be built to do the work. That problem that we're having is there is no signaling by which an endpoint can say, I'd like to send a request to a file server,

and I would like to let that file server know that it could send me a 64K response, which by the way, it would not be all that large when you look at how much faster 400 gig is than 10 meg was. 64K packet size is not absurd. There's just no way to tell it. By the way, that's what I'd like you to do, because you don't know. You have no idea if the network between you and a file server or between you and somebody else on the internet could tolerate a larger than 1,500 or 1,400 octet packet. And so that has to be envisioned. We will need new ICMP

message types on the internet. We'll probably need new various Ethernet level packets similar to what you do with bridge discovery. We're going to need interoperability testing. We're going to need to make sure it falls back reliably. You can never invalidate the install case. And the appetite for that doesn't exist. There isn't a consortium of companies who collectively believes that they will be able to deliver more value if they embark on something that will cost as much as the Apollo mission

and take just as long. Well, let's talk about the good side of DNS because this is very much its limitations, which is where you operate and where there's lots of future thinking things but people do some pretty cool stuff over dns and they use it they abuse it we had haroon on the show from thinks who makes uh what do they call them honeypots they make like this is security service that they call them canaries and they install them into your network and they're

honeypots and they call and they phone home and they let you know if people are trying to i'm doing a terrible sales pitch for haroon sorry haroon if your system has been compromised basically but the point is is that their entire all their entire fleet of canaries all communication that it does is over dns and they do that because it's convenient and easy not to have to deal with NAT reversal and other such things. You can just DNS your way out and DNS your way back in. And that's surely not what it was designed for,

but just a cool use of the protocol. And I'm wondering if there's other things people do. I'm sure you've been exposed to all kinds of stuff that people are doing. Using and abusing DNS, Paul. Your thoughts on that topic? I have. And because of my sort of affinity for the software and the protocol back in the day,

I was befriended by the actual inventor, Paul Moth. He and I have been in close touch over the decades since then. I dare say that some beer has been drank over the topic of what would you intend. I could tell you that the scope that they had at ISI UCI, wherever they were, that we need something to replace the old hosts.txt file that the internet's going to fix someday. And the scope was really just that. There's no need to be able to do dynamically

what we're currently doing by having this file that everybody pulls down by FTP once a week. And he made sure that his system would do that. Otherwise, it would not have fulfilled whatever development contract they had. However, he significantly overshot. He had a vision for a much more generalized system. There are many more data types than just, here's the IP address of the server. And so he made it very extensible. It is because he overshot the mark that we are using the DNS in so many cool ways today.

So a couple of examples. One is my own work. I created the first distributed reputation system and the first anti-spam company. So Anytime you hear about something called the RBL, that was us. There will never be email sent or received in the future. A million years from now, we're still going to use the RBL. No, I didn't patent it.

I didn't think of that. I didn't think of that. What we did, my co-architect on this was Eric Zygast, and we wasn't thinking of how he was going to change the world. He just wanted to get this out of his routers and into his servers so that it was likely to be better. And that turned out to be a really attractive model for a lot of people.

It had to be that as an email receiver, a server, an SMTP server, was heard a connection from somewhere, and that connection, should it be accepted, would then allow the sender to initiate various email transactions. Here's where it's from, here's where it's going, here's the body, et cetera. So it just had to be that the SNTP receiver would make a DNS lookup where the name being looked up was the IP address of the sending server backward in the usual kind of way.

And then it was under my domain, rbl.maps.biz.com. Maps was the name of this company, and spam spelled backward. It was also the mail abuse prevention system. We were very clever. And we got rapid adoption. We got way beyond how many queries per second we could tolerate on current infrastructure in a matter of months because there was nothing else like this, and commercialization and privatization meant

that all of a sudden the Internet was going to include everybody, not just people with government contracts. So right place, right time, right technology, but this is not what the DNS was meant to do. What it did, as you said, it traversed NATS. It was completely transparent. It didn't have to do anything at the far end in order to be able to do these lookups. So by using DNS to convey reputation data, we could

just say, hey, the address that you asked me about, that has sent a lot of spam lately. We have it in hand. We have proof of this. And that meant you could just reject it. And spammers took a while trying to figure out how they were going to look down this. And that meant she could just reject it. And spammers took a while trying to figure out how they were going to look down this. They did, or else they didn't stand with it. But that was what would be the first wide-scale use of DNRs for something that had nothing to do with the old host file. The second one that I saw a couple of years later was licensed key lookups. And I think this was Symantec. I don't know who it was.

But what they wanted to do is be able to give away antivirus software with every PC that was sold and then have it be that you get 60 days for free and then after that you have to pay money. And so they would have it be that every one of these PCs would create a random looking license key. And when you paid, you were essentially paying them to allow that license key to operate. And what they would do is use that license key as part of their DNS lookups or their

antivirus signatures. And it worked perfectly. And it let them build a global antivirus empire without having to sort of have every PC reach out to the mothership in the way that we all see today. The third way, and this has made it the best, was Dan Kaminsky. Dan has since passed and I miss him a lot, but he used to have a tradition where he would go to DEFCON in Las Vegas and they would just put him on the schedule. He didn't even have to file a proposal. They just put him on there and it would be something involving DNS. Then we'd all go there and see what it was without knowing any more than that. One year it was DNS tunneling where it was using the query direction as a way to transmit data and the response direction as a way to receive it. And so if you had a DNS tunnel endpoint on your laptop, which is how we demoed this, and it was talking to some DNS tunnel gateway somewhere that returned your DNS tunnel data back into normal

packets, then you could, for example, use a hotel room Wi-Fi without paying for it, because they had to allow DNS to work, otherwise they couldn't make their paywall work. Same thing for coffee shops and everywhere else. And so, they just used the fact that DNS was open by default and his demo was Skype. And he held a full motion video conversation on the big screen with somebody somewhere using DNS tunneling. And all of us were just mind blown. We didn't think that it would ever be fast enough to do that, but it just, it was just huge.

So yes, DNS turns out to be a lot, have a lot of room to grow. Resilient. I would describe that as being resilient, right? Cause like in any, in a lot of scenarios that can do compelling things.

Or flexible. Flexible. Sure. Okay. You know, most humans are pretty lazy. You know, we will invent what we have to invent and we'll reuse what we can't. Especially programmers. Programmers are the laziest humans. And the fact that DNS does so many things so well means that it's almost the top of

mind for a, a system developer at this point. There's even a t-shirt somewhere that says, oh hell, forget all that, let's just put it in DNS. Because that's what we do. Because if you have this global, coherent, sort of eventually consistent, reliable, semi-reliable database, it'll turn out to be just the right amount for almost anything you want. What's up, friends? This episode is brought to you by ImageProxy. ImageProxy is open source and optimizes images for the web on the fly. It uses the world's fastest image processing library under the hood, LibVeeps.

It makes websites and apps blazing fast while saving storage and SaaS costs. And I'm joined by two guests today, Sergei Alexandrovich, author, founder, and CTO of ImageProxy, and Patrick Byrne, VP of Engineering at Dribbble, where they use ImageProxy to power the massive amounts of images from all of Dribbble.com. Sergei, tell me about ImageProxy. Why should teams use this?

Everyone needs to process their images. You can't just take an image and just send it to users' browsers because usually it's a megabyte of data. And if you have lots of images like Dribbble does, you need to compress and you need to optimize your images and you need them in the best quality you can provide. That's where ImageProxy shines. Very cool. So Patrick, tell me how Dribbble is using ImageProxy. Being a design portfolio site, we deal really heavy in all sorts of images from a lot of different sizes, levels of complexity. And when we serve those images to the users, those really have to match exactly what the designer intended. And the visitors need to

receive those images in an appropriate file size and dimensions, depending on whatever their internet connection speed is or their device size is. And that was just a constant struggle really to really thread that needle throughout the course of the platform using a handful of different tools in maintaining that right balance of a high degree of fidelity, high degree of quality without sacrificing the visitor's experience. And when we were exploring using image proxy, we were able to spin up using the open source version of the product, a small ECS cluster to just throw a few of those really tough cases that went through our support backlog, looking at some of the cases people were reporting. And almost to a T, we aced every one of those

cases. Wow. So it seems like image proxy was really impressive to you. Yeah, Adam, I just have nothing but positive things to say about using ImageProxy. The documentation is crystal clear. Out of the box, it does everything you need it to. Tweaking it to meet your specific needs is incredibly easy. It's wicked fast. It deploys real simple. And our compute costs have gone down over the open source tools that we've used in the past. Even including the ImageProxy Pro license, it still costs us less to use and gives our visitors a better experience. So as an engineer, I like using it. And as an engineering manager, I like what it does to our bottom line. So ImageProxy is open source and you can use it today. but there's also a pro version with advanced features.

Check out the pro version or the open source version at imageproxy.net. The one thing I love so much about this is that no matter which you choose, the open source route or the advanced features in the pro version, you use the same way to deploy it. A Docker image, one that is from Docker Hub that everybody can use, it's open source, or one that's licensed differently for those advanced features. That's the same delivery mechanism via Docker Hub. It's so cool.

Once again, imageproxy.net. Check out the demo while you're there and check out the advanced features for the pro version. Once again, imageproxy.net. There's also a refrain that you probably heard at some point, Paul, was it's always DNS. You heard that one? Something along those lines. I have. This is the culprit of many lost hours of debugging,

only to find out it was DNS the entire time. And I wonder what your thoughts on that cultural epithet, not epithet, but this idiom that we have, and why it's the case that that's gotten that reputation. Well, I think the reputation has been earned. The statement is not inaccurate. It's merely misleading. How so?

So any company who comes into the internet and says, yeah, we want to deliver value, it's likely to look around for opportunities. What's not working well today? Sometimes their solution will just be, let's relax the constraint and Lunk will be the company to go to. And a lot of people have come in with online services, for example, that used to be enterprise services. Well, actually, for example, you think about Dropbox or any of the file service companies,

we all used to just buy a lot of hard drives and plug them into a lot of servers and so forth. But it turns out for a lot of what you need storage for, you don't care where it is and you don't mind that you have to go across the wide area network to get to it and you're happy that they're backing it up instead of you and so forth. So there's a lot of value to be created that takes the form of disorientation or just simple disruption. And that's not a bad thing. In fact, had we gone the other way, had TCDIP not won the war, had we been on the OSI protocol suites as developed by the phone companies, none of that would be possible. We'd only be able to do the things that they wanted us to do, whereas the internet is designed

to kind of let you try almost anything. It's a so-called permissionless innovation. So one of the things that got done with DNS was done by OpenDNS. And that was to say, you know, people hate their ISP DNS service, or they hate something about DNS. So we're going to create a global Anycast DNS service, OpenDNS, so that anybody in the world

can simply stop using their own enterprise DNS or ISP DNS or any other DNS, just use us. And we will be more of a liable. We won't data mine their queries to figure out where they're going and send them ads. So they actually did that for a while. We won't block things that, you know, we're not an anti-state. We're not going to say, you know, you can't reach this because it might be harmful in some way. Although there's always somebody out there being harmed and lawsuit you if you were a central service at that time. So there are some costs there. But they just wanted to centralize something that used to be distributed. And it worked really well.

But, you know, they were a growing for-profit company and they needed to figure out. And it worked really well. But they were growing a for-profit company and they needed to figure out, okay, we're here, we have customers, we have a lot of users, how do we monetize this thing? And so they did end up, they did this strange thing, they intercepted queries for www.google.com. And instead of giving back the real address, which would be the Google web server, they gave back their address of their web server. And it did not falsely indicate that it was Google. It said, this is the OpenDNS search engine. And then you would type something into the search bar, just the way we would anyway. And they didn't have a search engine. They couldn't answer it.

All they would do is then forward that question on Google and then tunnel the response back toward you. But it gave them an opportunity to associate your interests, the keywords that denoted your interest, with your IP address. And then they sold that data to advertisers. So when you then later reached some web server, that web server could ask the question,

hey, this IP address, tell me what they're interested in. And you might be able to imagine that Google wasn't super happy about this and believe in one so far as to say, please stop. The story is that people at OpenDNS said, you know, there's no law that protects you in this way. We're not breaking any law by giving back the wrong answer. And we're certainly not costing the law any money because you're receiving every bit of query data that you would otherwise have received. Google is still going to be able to make its whole business plan work. Somebody at Google would probably say,

yeah, but we didn't want Google to get free access to the thing that we monetize, so we don't want you to be in an intermediary here. But OpenDNS was resolute. They were not going to stop. And that, in my opinion, is why we have 8.8.8.8 today, is that the only way Google could prevent OpenDNS from continuing to intermediate itself between Google and its search customers was that Google had to build a bigger, more popular system. You know, once they did it, it was inevitable that we got 9.9 and 1.1. Think about it, the IP version 4 protocol is 256 octets in that first octet. So there are maybe 250 more companies who are going to get out

there and try to get 11.11 and 12.12 and all the rest. because if you can put yourself in the middle of DNS queries, then you can learn a lot. And then you can take that learning, and even if you're totally privacy respecting, which according to their stated privacy policy, Google is, and I have no reason to doubt it, you can still learn a lot that is not privacy violated. And so why wouldn't everybody and his brother try to create a system that would cause millions of people to send them their most vital information, which is what they're working on and what they're interested in. Okay. So let's fast forward, right? You're asking why is it always DNS? Yeah.

Okay, so it used to be that if you were a CDM, like Akamai, a content delivery network, that you could simply operate the server or whatever, Microsoft.com, for example. You'd get a query. You'd answer it. Somebody would ask, where is Microsoft.com? And you'd look at the source of the query and say, gee, we've got 35 copies of that content around the world. The one closest to you is that one.

And then we give you an IP address as part of the answer to your DNS question that was the mirror that was closest to where you were coming from. This went away once OpenDNS and Google and everybody else started doing this, because the place where the DNS question would be coming to you from was not the end user. It would be, and it wasn't their ISP, it wasn't their house, it wasn't anything that would help you predict where the web fetch was going to be coming from. And so, a collection of companies who had monetized things to the point where they no longer worked, proposed additional complexity with vast privacy risks, and then deployed it. And it is the standard for the internet today. It's called EDNS, which was my thing.

So I'm blamed for this sometimes. EDNS client subnet, ECS. And it's just a way to amend your query from your 8.8 server or your OpenDNS server or 1.1 or 9.9 server. You would amend your query by saying, and furthermore, the question I'm sending you is due to an end user who is on the following network. So, if you're planning on doing the CDN thing, you craft an answer for them based on that address, that's the address to use. Don't use my address because I'm not where the web fetch is going to come from. And boy, there were a lot of bugs and there are still a lot of bugs. And so, by sort of getting in there and saying, this is my leverage point, this is how I'm going to innovate, this is

how I'm going to shim myself into the internet ecosystem so that I can add value and get paid for it, DNS works less and less well. And so, when somebody shows me that, says, oh, Paul, you're such an idiot. You created this terrible thing. It's always DNS. Well, it's not my DNS. The world has taken DNS for a ride. And there's no guardrail where you're driving it. And it doesn't want to work the way that you want it to work.

And I'm not surprised that you're having the trouble you're having. Sorry, long answer to short quote. I like that. it to work and i'm not surprised that you're having the trouble you're having sorry long answer to short quote i like that i mean you point out a great um a great point obviously with the fact that where you resolve your dns at whether you choose it's open dns which was very popular back in the day right that was i had no idea about that backstory between open dns and google and but that's true where you point your d DNS to is you give a lot of power to them. Speaking of, Paul, where do you point your DNS? What resolver do you use? Well, there are two me's. I'll answer differently.

So I have a day job and they provided a laptop and it does whatever it does to go through the corporate DNS environment so it's logged and filtered and everything is done however it is that the IT security system wants. On my own laptop, I have a VM on that laptop that does nothing but run a DNS server. So I carry my DNS server with me wherever I go.

In my house, and back wherever I go in my house. And back when I used to be a startup guy, we simply ran our own DNS servers on CRAM so that we could log, so that we could filter, so that we could get all of the benefits of that information leverage locally. And I have a friend, Tom Burns, who has created something called the Personal DNS Firewall. It's a company called ThreatStop.com. And this part of what he does is free to all personal customers as well. And I really would like more people to say, you know, I heard Vixen say that I shouldn't let nobody see my DNS traffic and that there's a free thing i can install on my laptop just do it all locally

yes you did hear that yes you should do that good answer running it local i suppose on that note though like how do you not you know call out i suppose to say a different resolver that's popular like 1.1.1.1 from Cloudflare, et cetera, because how do you run your own DNS resolver, I suppose? How do I not know how to do that? So I assume that you've got some laptop running cell operating system and that it's got an IP address, and one of its IP addresses is 127.0.0.1. Right.

And all you do is grab some open source DNS server that has been compiled and packaged in a laptop that you have, install it there, tell it to listen on the loopback address. When you configure, you know, if you have a Linux machine, it would be etc. If you have a Windows machine, it would be etc.config. If you have a Windows machine, it would be somewhere in the registry. But one way or another, just tell your system that 127.0.0.1 is the name server. And then run a name server there. It's just that hard. I mean, it's just that easy. Well, I'm still learning about DNS.

I do not claim to be a DNS, you know, I'm a novice, really. I do run, and you may know this very well, PyHole is popular out there for a lot of home labbers who... PyHole is huge. Yeah. Any of your listeners who are interested in running their own DNS server and they don't like what I said, they should use a PyHole. It's great. So I use a PyHole. Actually, I have two, and I have it low-balanced. But inside of PyHole settings, it has upstream DNS servers, which I have set to Cloudflare. Is that not the same where you have upstream DNS servers? That's the default config.

But one of the appendices shows how to configure it, talk directly to the root name servers, and discover content without going through an intermediary resolver. Okay. That's something I haven't learned to do yet. So all this time, I've been so proud to be using PyHole. I've said it at least a thousand times in this podcast, right, Jared? At least. And didn't know that there was an alternative way to configure it

so that I can just resolve direct. One of the things you can also do with a bi-hole is to have your own filter list over and above whatever you subscribe to. You can just say, yeah, here's the various advertising servers that I want to insert don't exist so that my web browser won't go fetch them. That's what a lot of people use it for. But it is absolutely possible to make the five-hole ignore intermediate name servers. But I also want to speak in defense of your ISP's name server. One of

the reasons that ISP name servers got a bad rap and thus created the opportunity for OpenDNS, Google, Quad9, and CloudFlare and so on, is that they kept doing the wrong thing. They abused their position in your data path to data mine you and target you with ads and all the rest of that stuff. They don't so much do that anymore. I know as an example that Comcast has adopted a completely hands-off attitude toward their customer DNS traffic, and I know their team. They are really good. If you were a Comcast customer,

you don't need to use Cloudflare in order to keep your information safe. And it's worth looking online, finding out what is known about whatever ISP you have. If it's not Comcast, they may still be pretty good because now the world is watching them in a way that they weren't 15 years ago. Beyond, I suppose, the praise you've given for Pilehole here, what do you think about, I suppose, the open source, I guess, program itself? Like not just saying, hey, more people should use it. I feel like Pilehole cracked a nut where it was like just never thought of.

Like rather than solving, like you had said before,ns resolving at the laptop level which is a device you do it at the network level which means that the entire network benefits from the fact that pilehole is on the network and you control it what are your thoughts on that like considering what you know about dns given privacy etc so i love your question because i know that you feel it in your heart that something you really want to know. So let me just fill in a little bit more of the back story. There didn't used to be ISPs. We just had networks.

The internet was a network of networks. And we went to peering points and you spoke BGP and whatever it was you were going to do, you did. And you provided whatever services you went need. So at the time I took over maintaining the Bind software, the late 90 needs. It had a 100% market share because DNS wasn't sexy in the way that it's now. There's no money to be made disintermediating people and it ran everywhere. It was on every single network. That's just how the world started. This thing where now people, this may include you,

came in after. They don't know that that was the origin story. They think that Cloudflare has been here forever and Google and OpenDNS and so forth. Now, they came in in the early to mid-2000s. There was a rich history, decades, of everybody running their own DNS server. And yes, that gave you a network effect. You could say, you know, hi, I'm part of this campus, some concrete tilt-up down along Highway 101 somewhere, and I have a connection to CSNet.

And yeah, we have our own name servers. And everybody in the company who wants to use TCP IP protocols is using those name servers. And so we get to share one answer that we fetched from the outside world among every internal user who wants it. That's how this all started. So for you to ask, could that possibly work? It's a little odd for me. Yes, that can possibly work. Okay. I did come after that era. Okay. The only reason we're not doing that is that it didn't make enough money for enough people. Otherwise, you would have been born into a world where that's the law.

So this thing you mentioned from your friend, personal DS firewall, does this, and then PyHole itself, does it, I think PyHole requires a bit more for someone to adopt it. Do you personally advocate for, would you suggest, is your prescription for folks out there that care about their privacy to run their own DNS server, whether it's personal DNS firewall or pile hole or something like it? It is, but I want to admit to some of the endpoints that we encountered.

So if you're an average American apartment or single family dwelling, and you've got whatever connection you've got from DSL, whatever you've got, and you've got whatever connection you've got from, you know, outcome, DSL, fiber to the home, whatever you've got, and you've got this old modem or your own gateway box of some kind that connects to the outside, maybe it's got a Wi-Fi access point, maybe it's got new wired ports and a book ringer, whatever. That's your situation. You want to run a Pi hole. Well, you're going to have to,

number one, get yourself a Raspberry Pi, install the image, fiddle around with it a little bit, make sure it works. And then number two, you've got to get into that gateway box. And it's probably answering on the web port on 192.168.1.1, and you probably have a password which you've written on the side of the unit because you've been told not to forget it. You've got to get in there or configure the DHCP service inside that box so that when anybody signs on to your home network and they get an address from you

and they get told what their gateway is, they're also told to use your PI hole as the DNS service. Or they're going to use you, the gateway is, they're also told to use your Pi hole as the DNS service. Or they're going to use you, the gateway box, in which case you need to reconfigure that gateway box so that instead of going to the ISP name server, it goes to the Pi hole. That is all hard. Internet as scale, everything I just said is somebody swimming upstream. And I don't want to make it sound like it's going to be super simple. But once you understand what those issues are

and you're willing to outpour them and vote, then the pie hole is one answer. Another answer is, you know, you don't need the pie hole image per se. Another answer is, you know, you don't need, well, the Pi hole image per se. Take a Raspberry Pi and whatever version of Linux it came from and install Unbound or any other open source name server out there that has a package for whatever version of Linux came on your Raspberry Pi. You can just turn it on.

The defaults are pretty reasonable. It won't do the kind of ad blocking that Pihole is known for, but it will absolutely give you a local listener that everybody inside your single family, well, or your company or your apartment or whatever will all just start using that. So this is so non-magic. And the benefit to them is obviously they're not now freely giving their lookups to the isp to the resolver that they've chosen which is hey i'm google come use 8.8.8.8 because it's easy it's fast it's whatever whatever they bless it as and even clouther we're fans of them

but i i think they have a family edition which i think is pretty interesting like you can use a different ip address or i guess dns lookup ip address to have a family-friendly lookup zone where if your family's looking up things that are inappropriate or just sort of fringe to families i suppose it's protecting young ones on those networks like that seems like a good benefit i understand why they're it, but they are getting the ability to sniff it, right? They're getting all of your lookups and that's not good, I suppose. They are. But I want to say that, again, I've read the privacy policies online for OpenDNS and Google. I have no reason to think that they aren't implementing exactly what they say. What they say is they don't sniff. And so I think there is a valid enterprise value proposition for these companies who just want to say, look, I'm in the business of providing internet-related services.

I will be more successful with that if DNS doesn't hurt so bad. I want to offer this service to make sure people have access to at least one reliable, high quality DNS service. I don't think that's a lie. I just don't think that you should have to go that far and trust that far. If you're using a DNS server whose operator is in a different legal regime than you. Maybe the privacy law there is not the same as the privacy law where you are. And maybe that it won't be them, it'll be somebody between you and them who wants to data mine your queries and optimize your ads and all the rest of that. And so now the IETF has said, well, because people are talking to distant name servers by default, we have to encrypt all of it. Well, if you're going to encrypt all of it, then you have to go figure out how do I get the encryption keys so that I can know, you know, how to encrypt the

data if I'm sending it to that service. Well, and you go talk to other things on the internet, it sort of turns your otherwise your tiny little island, the network of which the internet is a network of, you're turning that into this viral cell in the body of something very large and you're depending on everybody else to be able to do the right thing. So I don't have a pitch that says, gee, if you use my stuff, you'll be safe. My pitch is if you use open source and control it yourself and don't go off net unless you need to, that will maximize your economy and your experience. Paul, as we close out here, I'm curious, you know, you're, you've been in the industry a long time, a lot of experience. Before we started recording, you said you, your family has a ranch.

You have some, you know, things that you're doing outside of the technical world. What keeps you going? What keeps you in the industry? Why haven't you hung up the shoes? What's the saying? Why aren't you retired yet? In the nicest way possible, I'm not trying to push you out. I'm just curious. What gets you going in the morning to come back to work every day? Well, it all started in 1980 when my high school guidance counselor explained to me that I would be in the 11th grade again next year because I hadn't turned in any homework and terrible students on. And I remember thinking to myself, I think I know a better way because I know how to program computers. And I'll bet there is somebody somewhere who will pay me more than their own wage to do it. So, that kind of put me in the right place, right time, San Francisco area, 1980s, internet was just about commercialized and privatized, units was just about to become a household word. And so 30 years later,

when I was receiving my award from the Internet Hall of Fame, I said, I spent the first half of that 30-odd years making communication possible. And then, because it succeeded so well, I spent the second half trying to make it safer. And when I exited my fifth, and what I hope will be my final startup in November of 2021, I didn't really want to do it. My heart was in continuing on, finding another investor. We were in the black.

We just weren't growing fast enough. But investors that had said, no, we'd like to be paid, so we sold. I thought, okay, that's it. I have been such a bomb thrower for the last 30 years that I am going to be unemployable unless I start another company, which I am just not pertinent to do. But I was wrong. I was wrong in two ways. First, I was unemployed for the first time in 41 years. So I learned something horrible about myself. You should try this, see what you learn. Which is, I learned that if I don't have a reason to get out of bed in the morning, I don't. And that was intolerable. But then I had the problem of I didn't have a team over and above my family. I wasn't on a team

and I didn't have customers to protect. And so the cloud company called me up, said, we don't care that you're a little bit of a bomb thrower. We can go right in. And I was so glad that I had a team to go join and customers to check. That is my particular psyche. I need those things. Well said. So no end in sight then. Because why? Because you need it.

I don't want to tie it to my desk. But someday i will definitely get too old to do this like yes something out of my co-workers to tell me that that's happening okay that's a good strategy somebody's gonna if you got some honest co-workers around you you know no shortage well it's been fun digging into the villain and the hero called DNS. Yes. I've never had a chance to sit down with someone like you to go as deep as we have with you. And I really appreciate you taking the time with us to entertain our questions.

And yeah, it's just been awesome. Thank you so much. This has been a lot of fun. Thank you guys very much for picking up and man. Thank you as well loved it so it's not always dns or maybe it is i don't know i mean for me it is i don't know about you all but if something's going wrong it's like what dns again come on But I also learned something too. That little bit about the pie hole and not using the common DNS resolvers like OpenDNS, Cloudflare, or Google. I mean,

I'm doing that as we speak. It's a weekend home lab project, and I'll report back to you in Slack. Speaking of, the Slack community is 100% free. You can go to changelog.com slash community. There's about 7,000 folks in Maine on Slack. So you'll have lots of people to talk to. And there's always something fun being talked about. High signal, low noise come hang with us later this week on change logging friends i'm solo because dare's getting some much needed r&r and i'm talking to a good friend of mine robert ross founder and ceo of fire hydrant it's a good one make sure you listen of course thank listen. Of course, thank you once again to our awesome sponsors for this episode, century.io, S-E-N-T-R-Y.io. Get a hundred bucks off the team plan using the code changelog. So cool. And of course, our friends over at CIQ and their awesome work on Rocky Linux, big fan of Rocky Linux. Check them out, ciq.com. And of course, rocky linux big fan of rocky linux check them out ciq.com and of course rocky linux.org

and also our friends over at image proxy image proxy.net real-time processing for your images on the web open source and also available as a pro version with lots of advanced features check them out image proxy.net and who could forget our awesome sponsors and partners, Fly.io and Typesense.org. They love us. You should love them. And, of course, to Breakmaster Cylinder, the banging beats from the beats master himself, Breakmaster Cylinder, bringing it every single week. So awesome. Well, that's it. This show's done. Thank you. Outro Music