The Changelog: Software Development, Open Source - Let's talk FreeBSD (finally) (Interview)

Episode Date: January 17, 2024

This week we're joined by FreeBSD & OpenZFS developer, Allan Jude, to learn all about FreeBSD. Allan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there & answers every curious question we have about this powerful (yet underrepresented) Unix-based operating system.

This week on The Changelog, Adam and I sit down with FreeBSD and OpenZFS developer Alan Jude to learn all about FreeBSD. Alan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs Alan gives us a brief history of BSD, tells us why it's his operating system of choice, compares it to Linux, explains the various BSDs out there, and answers every curious question we have about this powerful yet underrepresented Unix-based operating system. I learned a lot, I hope you do too. But first, a big thank you to our partners at Fly.io, the home of changelog.com.

Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io. All right, Alan Jude on the changelog. Let's do it. What's up, friends? This episode of the changelog is brought to you by our friends at Vercel. And I'm here with Lee Robinson, a.k.a. Lee Rob, VP of Product over at Vercel. So, Lee, let's talk about Vercel's front-end cloud.

I know Vercel supports over 35 frameworks, but the question I often hear is, is the happy path on Vercel Next.js? Are you optimizing the best experience, the sugar on top, so to speak, for those using Next.js? Yeah, it's a great question because of course we're biased because we make Next.js,

that we really like Next.js. But the idea behind the front-end cloud and the idea behind a concept we call framework-defined infrastructure is that we want to make it as easy as possible for you to write code in the frameworks that you prefer and still get to take advantage of all of the cloud infrastructure primitives and the developer experience platform that Vercel provides in that same model. So tangibly, this framework-defined infrastructure concept means that I write code in my framework native way, and Vercel is going to do the heavy lifting of transforming that into managed infrastructure. So in Next.js or in Nuxt or in SvelteKit or Remix or all of these frameworks that we support through framework-defined infrastructure, I'll write my code in their framework API. So I want to do server-side rendering. Great. I'll do it through their conventions. When I deploy to Vercel, our platform is going to transform that code to work with our managed infrastructure that's very scalable

and also flexible for what developers need. Okay. So help me break down what you mean when you say the front-end cloud. I get Vercel, front-end, front-end frameworks, all that good stuff, but break that down for me. What exactly is the front-end cloud? We believe that there should be tools designed and built by and for front-end developers. We want to have tools that help enable you not only from the framework and the code that you're writing with React or Vue or Spelt, all the way up to the platform that allows you to deploy your code, not only in US East, but around the world as well too. And I think historically, a lot of tools for developers have been built backend first, have been built for the backend developer. And if you think about AWS, right, it's a great backend cloud.

It has a suite of tools that you can pull from and really craft and build anything that you want. But when you're building the front end of your application, there's a lot of things that are undifferentiated. It's like, I don't really want to have to stitch together all of these different pieces and set up my pipelines and set up all of this front end infrastructure. I just want to build a really good product. And really what the front end cloud means to me is focusing more on the product experience and focusing more on the user experience and just

letting a platform like Vercel's front end cloud take care of all of that infrastructure and tooling for you. Very cool. I love it. With zero configuration for over 35 frameworks, Vercel's front-end cloud makes it easy for any team to deploy their apps. Today, you can get started with a 14-day free trial of Vercel Pro or get a customized enterprise demo from their team. Visit vercel.com slash changelogpod to get started.

That's V-E-R-C-E-L dot com slash changelogpod. All right. We are here with Alan Jude. We are here to talk about BSD, specifically free BSD. Alan, thanks so much for coming on the show. No problem. Thanks for having me. We're happy to have you here. We're happy to fulfill a listener request.

Shout out to Vladimir who writes, I enjoy listening for discussions about the state of modern Linux distros, episodes about Red Hat, Debian, Nix, etc. But I don't recall any about FreeBSD. I'm sure lots of people heard high-level facts like it's another Unix-like or its networking stack is better than Linux or something about ports or macOS was based on it, etc.

I'd love to learn why and who the OS is for and how it's different. So there's your task, Alan. We brought you here to tell us these things. Why and who is the OS for and how is it different? Yeah, I guess to start, you go back into the prehistory of computers basically. Let's follow back. With the original version of Unix that came out in the 70s.

And because of the consent decree that meant that AT&T wasn't allowed to, in order to maintain its monopoly on the phone system, wasn't allowed to compete in some other spaces, including computer software. And it had to license its patents to universities very cheaply. And so it ended up providing copies of the Unix software with the source code to a bunch of universities so they could build on top of it and do research based on it. And that eventually led to what was called the Berkeley System Distribution, or BSD, at the University of California at Berkeley, where they made their own version of it and kept tweaking it and adding the

things they felt were missing. And it was one of the kind of pioneers of the open source movement, although back then it was more mailing physical tapes of code back and forth between these different universities and one person trying to compile that all together because there was no GitHub yet. So somebody literally had to kind of be like a copy editor and put all the different pieces contributed by different people together and make it into something. And that went on for a long time and worked quite well. But as time went on, AT&T decided that was worth more and more money and the other problem was you know people who had worked with this stuff decided that oh we want to actually make a product

out of this and sell it or even sell support for it and so on and so a company called BSDi started up to sell this BSD operating system to regular people. And they maybe made the trademark mistake of having the phone number 1-800-ITS-UNIX. And since AT&T owned the trademark on Unix, they were very displeased with this. And it resulted in a bunch of lawsuits about, you know, some of that code is copyright AT&T. You can't just go selling it.

And it quagmired the BSD code in lawsuits for a couple of years. And about then is when Linux actually started, like when Linus decided to build Linux. And there are quotes from Linus saying that if BSD hadn't been tied up in the lawsuit at the time, I probably would have never started Linux. I would have just used BSD.

Right, that's interesting. But those all got settled and it turned out there were like four files that just got rewritten and there was never actually something that was being done wrong necessarily. But it scared people off enough and delayed the development long enough that Linux gained a lot more popularity in the meantime. But in general, then after the last versions of that, the university was kind of done with it after dealing with the lawsuits and everything and so they came up with the bsd license which you know compared to the gpl

is much shorter you could fit it on a business card the modern version has just two clauses in it don't remove the copyright and reproduce the copyright in the manual it's basically the only requirements the original bsd license had a couple more including like you can't use the name in the manual. It's basically the only requirements. The original BSD license had a couple more, including you can't use the name of the university to promote your product, just because this code came from the university. And you have to include the line that this includes code that came from the university

in the manual. But of course, as you can expect with something open source, when that proliferated and you had 200 different companies having contributed code to something, and you had you know 200 different companies having contributed code to something and you have to list all 200 of them in the manual for every product it got a little out of hand so the advertising clause got removed from the license to make life easier for everyone but that big the difference there in the license is is what makes most of the difference

in what bsd is and why people choose it over Linux oftentimes. Because the license doesn't, it's what we call a copy-free license, whereas the GPL is a copy-left license, right? The idea with the GPL is that it has a very strict requirement that if you make changes, you have to give those back and make those public. And that you have to provide, if you build a product out of it, you have to provide the source code for that to any user that asks. Whereas the BSD license is copy-free. It's basically you're free to do whatever you want with it. The only rules you have to follow is you can't claim you wrote this when other people did.

So you can't just delete the copyright statement. And you do have to acknowledge that you used other people's code in the manual somewhere in the code. And that's the only requirement. So if you're a company that's building an appliance, like a washing machine, and you want to build in a network stack so it can do a push notification to your phone to tell you when your laundry's done, instead of having to write your own, you could take this open source one and just build it in. And then like at the back of the manual, you just have a page of copyright notices. Whereas if it was GPL, you could probably do the same but then anybody who asked you have to mail them a cd with the source code for it and if you made any changes the license is viral so it infects

those changes and you have to give those away freely as well and that can cause maybe it's not that big of a burden but it can cause a lot of extra headaches for compliance right making sure Making sure that you're following that license all the time and you don't accidentally not can be a big impediment to companies wanting to embed or build a product based on's the people that have been using real Unix since a long time ago, or people that got started in computer science in university at the time wherever university was using BSD. And so that's just what they started with. And then there's people that were building a product

and chose BSD because of some of the technical features that it had, like FreeBSD pioneered the practical implementation of containers in the year 2000, whereas we'd only saw containers and namespaces on Linux come much, much later. Now it's gotten maybe a lot more built on top of it since then, but the concept came originally from BSD.

And then a lot of companies, as we talked about, can take the BSD code and build a product on top of it. And a lot of the reason why BSD continues to thrive is that those companies see the value in investing back in free BSD. So while the license doesn't say you have to give your code back, if there's any code that's not specific to their product and is not their intellectual property, they have an advantage of contributing that back. So it means when they go to build the next product on a newer version of BSD,

it's that much less diff that they have to try to rebase on top of. So I'm curious, Alan, why you use or choose BSD personally. What's your story coming to it? So back in the late 90s when I was a teenager, I finally got on the internet and I discovered IRC and thought this was really cool. And I wanted to host an IRC server, to have my own IRC server instead of using the big popular one with all the people on it. Because that way I could be in charge because I was 13. And so I asked around a little bit and learned about it. And I apparently needed this thing called a shell account, which is it was pre-ssh but you'd tell that

into somebody else's server and be able to leave a process running and it turned out a good chunk of the servers you could get a shell on back then were bsd and so that's what i started and what i learned and also especially back then in the early days of linux there were a lot more root kits and vulnerabilities and problems if you know you were running Linux for, especially when you're doing things like hosting something public that you're encouraging people to try to connect to. And so that's where I got started with renting, sending them. I was 13, so I didn't have a credit card. go to the post office and get a money order and mail it to some guy the next state over to let me run something on his computer on his T1 line because I only had dial-up and use that for a while. And it happened to be a FreeBSD 2.2 box. And so that was the first kind of non-Microsoft

operating system I had ever used. And then after a couple of years, I decided to go beyond that and start a hosting provider to do what that guy had been doing, but on a bigger scale. And so then I installed FreeBSD 4, and that's when I really learned to be a sysadmin. Because you have to be really good at being a sysadmin when you're purposely letting other people connect to your machine and run processes on it. Yeah, for sure. And so very quickly learned to use some of the features and to be able to lock stuff down. So FreeBSD's networking stack has a feature called dummy net, which allows you to, originally created for doing research at universities, allows you to simulate different network conditions. So you can like purposely make it really high latency, like it's a satellite link, or make it have a certain ratio of packet loss, like it's a Wi-Fi link. But it also allows you to just say, hey, any connection that matches

this rule is limited to this many megabits per second or whatever. And so being able to use that to deal with the fact that, you know, I only had so much bandwidth from the provider and I needed to share it fairly across hundreds of people and how to do that. Interesting. Do you think you stay with BSD because that's what you grew up with or were you ever you know called to the dark side the linux world what made you stick and stay that's why you used it but why did you stay uh mostly just because everything made a lot more sense to me things were organized in a much different way and partly i think because when i the few times i tried to use linux it was never the same linux a bunch of times in a row.

Somebody would be using Red Hat and somebody would be using Debian and another one would be Slackware or something. And they're all so different. It was very confusing, whereas all the FreeBSD machines were kind of this one holistic thing. And also the fact that in FreeBSD, there's the kernel, like in Linux,

but also the user space stuff, like core utils and all that other stuff are shipped as one package so the tools are always integrated and and don't have to deal with trying to work on a different version and so you don't have some of these weird artifacts you get with Linux where you know each distro decides to ship a different version of like the DU utility but with the same kernel or whatever and so you know i remember being very excited after years when uh gnu core utils du finally got the dash capital a flag for showing the apparent size of a directory so if there's compression it can show it

but you know that version only ships on newer ubuntu and if you're still using centos you didn't have that feature and And it was just like interesting. But mostly it's just once you know a tool really well every time you use a different tool you just hate the fact that it's not the tool you know. And I think that's the main thing that's really kept me. I think a lot of things perpetuate for that reason.

Yeah but also just there's just been a one of the big things is there's a very clear separation between what is the operating system and what is the packages I installed. So in FreeBSD, if you do a fresh install and you go into the package manager and list what's installed, the list is empty. Well, some tools are built into the operating system like TCP dump and netcat and some of the basic stuff like that and like the compiler. Those are part of the operating system. And then anything I decide to install goes in a prefix directory.

It goes under user local because this is a local modification I made. It's not part of the operating system. And it means that I can have completely different packages that are separate from the operating system. And so those packages are always fresher than what you, like if you're on a Ubuntu and the last LTS is 2204. So most of the software you have is the software from 2022. Whereas on FreeBSD, if you run the latest package set, you have the software from this week.

And that can make a big difference as far as trying to use a new feature in an application or just you you know, you're running a web server and they've released a new version that has a feature and you want to use it. The fact that I don't have to wait till the next LTS release or try to shoehorn it in there. Do you like Linux? Are you a Linux hater? Are you a Linux lover? Do you divide the line? How do you do it? There are some cases where I have to use Linux, like doing video transcoding stuff. And the only, well, NVIDIA makes the graphics drivers to use the GPU as a GPU on FreeBSD. They don't make the drivers available to do the video transcoding stuff on FreeBSD. So in my sysadmin experience, I have,

there are some Linux machines and I use them, but I often find myself trying to make them look more and more like BSD just for my own comfort and realizing that that's probably just making it worse. BMC beats made me want to group hug everyone. BMC beats made me want to group hug everyone. What's up, friends? This episode is brought to you by our friends over at CIQ. CIQ is Rocky Linux's founding support partner. They support the free, stable, and secure Linux operating system called Rocky Linux.

And I caught up with a recent episode of their podcast called Flops and Threads. Episode four was an Ask Me Anything show on OpenELA. OpenELA is a collaboration created and upheld by CIQ, Oracle, and SUSE. And Gregor Kircher had this to say about OpenELA. OpenELA really came to be just through the collaborative nature of open source and organizations that are really trying to solve similar problems. And CIQ, Oracle, and SUSE all came together to see how do we leverage each other's work in an open and collaborative way and then share that and collaborate further with the wider and generalized open source community. And that was really the idea. Just how do we rely on the resilience and all the benefits of open source for this critical

platform that we, the user base, our customers, and everyone in the community seems to rely on? And that was really kind of just the basic idea to start off with. Wim Kockertz, head of Oracle Linux development was also there and had this to say. So my name is Wim Kockertz. I work for Oracle and have worked there for 27, 28 years, something like that, 28 years. And one of the teams in my org is the Oracle Linux team. So we've been doing all the Linux work at Oracle since the early late 90s, early 2000s, but really mostly starting in the early 2000s. The Oracle Linux

team is a big part of my day-to-day job. And of course, we're very excited about what's going on with the open ELA. And it's been a very interesting last few months. Always something new going on in the open source and certainly Linux world. So never at all moment. Certain events have gotten people to come together more. And I think the result really, this is an opportunity to make it easier to be more out in the open than it was before and do some shared work to be more efficient. A lot of work that we will now do jointly, we've all been doing individually as companies and not just companies, but also communities. There's lots of community project versions out there that do something similar where folks working for some, you know, where

their day job is something different. So we have lots of processes in place today for QA, for automation, for validating patches, for doing check-ins with sign-off buys, just like what happens with the Linux kernel. Those are sort of the types of structure that we've built into our environment for Oracle Linux. But now we can all work together to bring that more broadly. So it's very exciting. I think we're on the right track here. And to round off the discussion,

Gregory Kircher, explain what it took to start the Open ELA and an invitation to the community to get involved. So to announce this, we had to have a few things already kind of defined in emotion. Alan, myself, Wim, Thomas, and others, we've all already been talking about this for, oh goodness, for a little while now. And we finally got it to the point where we had a charter. We're in the process of forming it as a nonprofit legal entity. We've been really just focused heads down on getting that. Now that that's been out there, we've spun up the slack. We have a lot of people in the community that have joined and want to be

part of this. Now we're actually working on the technical environment to start identifying what are the kind of core tenants development-wise, and what are we going to be building moving forward, and what are the requirements of that? Once we have that, we kind of have the foundation done at that point, now we're really not only inviting people to join this vision, but now we're actually going to be inviting people to actually join and be part of the project itself. Okay, check the show notes for a link

to Flops and Threads, episode four. Also check out openela.org, where CIQ, Oracle, and SUSE have come together for a trade association of open source enterprise Linux distribution developers. Once again, openela.org. You can also check out ciq.com where you can learn all about what they're doing to support the enterprise linux platform and again check the channels

for a link to flops and threads episode four so i was first exposed to bsd in college i've never run it so now you know my experience level so this is my level of personal experience was same never run it. So now you know my experience level. So this is my level of personal experience was same. Never run it was at college. This is like turn of the century computer science department, university of Nebraska. And there was like this pecking order of nerds and it was like correlated with population size. So like most of the people are running windows and they're like at the bottom of the pecking order. But there's most of them. And then some of us, see how I'm now personalizing it, some of us had branched out and learned to run Linux

on our laptops or our desktops. And there was less of us, but we were cooler than the Windows people. And then there was one guy, one guy who ran FreeBSD. And he was at the top. He was the hardest of the hardcore. And we were all kind of looked up to him and was kind of afraid. And he also was a bit snobby and looked down on us.

And I just was afraid of it from then on out. Because I was like, only one guy knows how to do it. I'm cool with Linux. It's already hard enough. I'm already not a Windows person. These are very juvenile thoughts, of course. But it seems like that's the view I saw of this divide of the Linux and BSD people, is there's more Linux people, less BSD, but BSD is my even harder core. Does that resonate with

you guys? Or is that just what my perspective looked like at the turn of the century? I definitely have seen that perspective. Oftentimes, especially back in the early 2000s, I would say that BSD definitely wasn't harder. It was probably much more well-documented. And if you had wanted to, you could have done it and probably even had less difficulty. Some of that has maybe inverted a bit now. Linux has gotten a lot more mature and there's a lot more documentation out there than there was back in the early 2000s. Yeah, I'm trying not to have too much bias here. But there is that general perception and I don't know how well deserved it is.

Yeah, or how it started or whatever. BSD is this higher core, but for a long time it was definitely, if you were doing business with your computer, if you were doing business with your computer, if you were making money hosting something, you wanted to be doing that with a BSD and then Linux was this kind of, it's what the kids are doing, but it's just a toy. And that has changed a lot.

It's been 20 years and more than that. And IBM spent a couple billion dollars making Linux better and then separately since then bought Red Hat. But I think part of it is also just the assumptions that BSD makes is kind of, I guess the best way I've heard it phrased is Unix doesn't stop you doing something stupid

because that would also stop you doing something clever. But it does mean that, you know, there are no guardrails to keep you from shooting your foot off. Something that would make me think about BSD differently, which I really haven't considered so far yet because I'm an Ubuntu user primarily, 2204 or 2310. I do have one system stuck at 2310 that I will upgrade whenever, I guess, 2404 is available.

I don't know. So sometime this year, I'm going to upgrade that 2310 machine. And I'm waiting for the day, basically, for the LTS. I think April is roughly when it will be. But I'm a ZFS lover. As a matter of fact, I own a book that you may know about. I don't know if you know this book. It's like, it's got your name on it.

It's called Free BSD Mastery ZFS Edition, essentially. It's the, I got it because I want to learn more about ZFS. And the reason why I bring it up is because as part of preparatory for this conversation, like any good podcaster, I went to chat GPT and said, can you tell me some things about BSD that I may not know? And I know the overlap is obvious with ZFS. And I was surprised when it said that the integration with ZFS and BSD could be better because of a more compatible licensing mechanism. And so I kind of want to hear more about that because I love ZFS. All my file system storages I have are all ZFS-based.

But here I am saying I use Ubuntu. And meanwhile, I probably could get a better experience if I would just try FreeBSD and then not just read your book to learn about ZFS, but also how to use ZFS on FreeBSD. So with that in mind, the integration with ZFS seems like it could be better or more stronger because of just various things it has incompatibility with the license. Now, that may not be true for OpenZFS, and I know ZFS has had some licensing history that may not be pertinent, but is that accurate with what you know? Is ChatGPT smart in saying that ZFS can be more tightly

integrated to BSD because of its license compatibility? Yes. Okay, sweet. ZFS and OpenZFS are licensed under the CDDL, which is a slightly modified version of the Mozilla public license. And so the CDDL says, I'm not a lawyer, don't take my legal advice. But in general, the license says that you can make a binary out of cddl license code and you can license that binary however you want so that you can make a product out of it or whatever and so you can make a zfs binary like kernel module or whatever and license that even under the gpl so it'd be compatible to integrate into the linux kernel but the gpl itself requires that any code you let you link into it has to be licensed under the GPL.

And the CDDL doesn't let you just change the license on that code to be under the GPL. And so that's where the kind of incompatibility comes in. But it's not a very strong one, and Ubuntu is pretty sure that their legal way around it is working. But because the FreeBSD license is literally two sentences, right? Don't take off the copyright and reproduce the copyright. It means that we could pull in all of ZFS, the code base directly into FreeBSD. So if you go to the FreeBSD GitHub and the repo under sys contrib, there's a directory called open ZFS that has a whole copy of the ZFS source code. So this means ZFS is completely just built into FreeBSD. So the FreeBSD bootloader has full

support for reading from a ZFS pool. And so you can boot directly off ZFS. So even if you're using all the advanced features of ZFS. And so this allows a feature called boot environments, where, you know know your root file system if it's zfs it's actually a virtual file system and you can have multiple different ones or you can clone it and take snapshots so you know when i'm preparing to go to a conference or something i have my laptop that runs freebsd that i give my presentations off of and you know i use a desktop most of the time when i'm at home. So I haven't used my laptop much in the months between conferences. So I'm going to update the OS because it's been a while. But, you know, that's when everything has the greatest chance

to break, right? When you're installing all these package updates and so on. So if with ZFS, I can take a snapshot of my root file system and save that as before I broke everything and then update stuff. And it all seems to work. And, you know, I check at home and I can play stuff on the TV. So my HDMI is working. Everything's fine. But say I fly to the conference, I get there and something isn't working right. I can just reboot in the bootloader, say, hey, I want to use the image of before I broke everything instead of, you know, the latest version of FreeBSD. And it will roll back my root file system. But importantly, it doesn't roll back my home directory. So I still have the latest version of my slides, but I have an older version of my OS that still works. Because whatever latent problem I found that, you know, X Windows didn't

want to support a certain resolution or whatever broke. And so that can be really powerful. Or if you've ever tried to update the firmware or the OS on a little router or something, they often have two images, right? They always have the one you update and then there's a spare one. Even switches do this. So that if you somehow botch the firmware update,

they have a second firmware they can boot from in the meantime. So FreeBSD had a feature like that before it had ZFS called NanoBSD that let you do that, right? You partition up your disk into two big partitions and one little one. And the little one would hold the configuration and the two big ones would be like version A and version B of whatever you're trying to install, whether it was a firewall or router or whatever. And you'd be running off A, so you'd update B, and the next time you boot, you'd switch to B, and then you ping-pong between these two images. But ZFS allows you to do that with an unlimited number of images

because each is just this virtual file system in the pool. You can have as many of these as you want, and you can access them directly from the bootloader and decide which one you want to use. And when you're switching to one, it doesn't make the other ones go away. All the data is still there because ZFS is copy on write, so only the bits you actually changed

are taking space. So having seven copies of your operating system doesn't take that much space because you only have to store the differences, not all of the data seven times. And so that allows much deeper integration. But also if we go back, when we talked about the fact that

the base system utilities like top, for example, are not a separate package that came from a different maintainer somewhere that is part of the FreeBSD operating system. So when you start top on FreeBSD under the line where it tells you about how much free memory you have, it has two lines about ZFS telling you this is how much of your memory is being used by the arc. This much is for frequently used files. This one's for recently used files. And this much is, you know,

data that's pending being written. And, you know, here's what your compression ratios are in memory and all this other information that would only be there if you had ZFS. And because FreeBSD is that integrated with ZFS, the top can just be compiled against the version of ZFS that you put through your system and be able to get that information all integrated. Pretty rad. Yeah, very rad.

I think I'm going to be playing with free BSD soon, Jared. I'm sure you will. Yeah. But Adam, why not open BSD? So that leads us to our next line of questioning here, because you were talking about Linux distros. Right. Certainly less BSDs. I could count them on one hand. I only know three. I'm sure there's more. Free, Open, and Dragonfly, but I'm no BSD person. And NetBSD.

Okay, so four. Is that all of them? There's GhostBSD, which is more like a distro. It's free BSD, but includes a GUI by default and is more like a Linux distro in that they took the OS and then added a bunch of packaging and defaults on top of it. And if you want to run FreeBSD on a laptop

as a daily driver type machine, GhostBSD probably is a better place to start because it will be more familiar to somebody used to Linux. Whereas FreeBSD out of the box includes no packages and no GUI. You can install GNOME or KDE or MATE or whatever you like whereas FreeBSD out of the box includes no packages and no GUI. You can install GNOME or KDE or MATE or whatever you like but you have to know to do that whereas with GhostBSD it comes separate. So with the BSDs, the NetBSD, OpenBSD and DragonflyBSD

are not really analogous to a distro. They are all originally based on the same code that 4.4 BSD Lite from the University of Berkeley, but since then while because they're open source, patches might have gone back and forth, they are since then completely divergent projects.

So they're forks. Yeah, they're forks of the same original code from the very, very early 90s. And so being 35 years later, a lot has changed in between there. So they are part of the same family tree, but the kernels are quite vastly different. And kind of the goals of each of the projects is very different. So NetBSD is really designed around portability, especially back in the early days of Unix. That was one of the big things, right?

Some people had like a PDP and then they had VAXs and all these different computers that were completely different architectures. Before the invention of C, every time you had a different kind of computer, you had to rewrite the operating system to like the assembly language of that computer, right? It wasn't like we have now where the x86 is an ARM

or the two main kinds of computer and we don't have to deal with very many others. There's PowerPC and MIPS and maybe and a couple things like that, but we're not dealing with hundreds of different architectures. There's like three or four popular ones. So NetBSD wanted to maintain that and works on all of those weird architectures that we just mentioned and a bunch of ones you've probably never heard of and a bunch that only existed in Japan and never shipped anywhere outside of Japan. And so NetBSD's focus is really that portability

and being able to run on anything. A lot of their ethos from the beginning was there are these bigger mainframe-type machines and they don't make an OS for them anymore. We can be the open-source OS that people can still use to to run these and that has a lot of value but it doesn't have quite the same mainstream appeal as you know running commodity servers with you know regular applications but netbsd did pioneer a very interesting thing called package source pkg src and this is a package manager that's not specific to one

operating system so it's supported on like various different versions of the bsds linuxes solaris and its derivatives lumos and so on and it's allows you to get some kind of basic applications on stuff like people are like oh i need modern python on hpux and package source is a way to do that and that came out of the NetBSD project. And then OpenBSD has really had a focus on security and is kind of like a research operating system where they're trying new concepts and ideas to make more secure computers.

So they invented a bunch of common things we do. They're the maintainers of OpenSSH that every operating system uses as a way to securely connect to a remote machine. That's maintained out of the OpenBSD project, along with lots of other applications. They have their own BGP server, and they maintain a BSD-licensed version of rsync, and lots and lots of other applications that you might actually use. But also have pioneered a lot of concepts. They have a system called Pledge, where

an application can say at the start after it's done its initialization where maybe it needed root access for something, it can pledge that it will only use these system calls from now on. And if it ever tries to do more than that because maybe somebody used a buffer overflow to try to make the application do something it's not

supposed to, it will actually the operating system will say, you promised you wouldn't do that, and will stop the program. And this allows you to sandbox the program. FreeBSD has something that's not exactly analogous. It's called Capsicum, and it's a lot more fine-grained, but it means it's a lot more difficult to retrofit existing applications to that more locked-down sandbox.

It works very well if you're writing a new application and design it from the beginning to support this compartmentalization. But Pledge is a very easy way to retrofit existing applications to make them safer. They also invented a couple other concepts, like they relink the kernel, all the different parts of the kernel,

together in a different order every time you boot so that everybody's machine is different and then XFlight won't be able to just work on every OpenBSD 7.4 machine. That's smart. And what about Dragonfly? What makes that one different? So back at the end of FreeBSD 4.11,

so FreeBSD now is at version 15, but back at version 4, they were looking at how to support basically multi-core systems. This is just when we were getting past the five, it's like you had one CPU that was one core, and maybe you had a second socket for a second CPU. And that's all there was. But as we were getting into this idea of having multiple cores and even threads and so on the model that the kernel had to use for managing all the processes needed to change and freebsd picked one specific way to go and one of the developers really wanted to go in a different direction and so he forked freebsd and created dragonfly bsd that's tries to apply the concept of message

passing and so on a lot more and seems to have gotten some traction in like the high frequency trading space uh because of the way the message passing applies so that's that project is out there uh freebsd ended up changing what it was doing to even a third option but since freebsd6 the the process model has worked very well to support that and you know with freebsd 14 or 15 uh we can support up to 1024 cores on a machine and i think linux is going to get there soon but i think most versions of linux are still stuck at not able to support more than 256 cores which you know seemed like lots uh even only a couple of years ago but now that i've you know personally used machines with 192 cores it's like oh we're definitely going to exceed 256 like any minute now.

If you're listening, you may remember the early days of the internet where open networks like HTTP and SMTP led to an explosion of websites in online communities. Building a fan site and connecting over shared passions led so many of us to careers in software and I'm one of them. Back then it seemed like anything was possible because the internet of the 90s was built to democratize information not consolidate it with a handful of big tech companies. Read, Write, Own, Building the Next Era of the Internet is a new book from startup investor Chris Dixon that explores how network architecture plays out in our online lives and the decisions that took us from open networks governed by communities of developers to massive social

networks run by internet giants. Read, Write, Own is a playbook for reclaiming control and for reimagining applications so users can own, co-create, and even profit from the platforms they use every day. From AI that compensates creators to protocols that reward open source contributions, this is our chance to build the internet we want, not the one we inherited. Order your copy of Read Write Own today or go to readwriteown.com to learn more. So if you were just beginning with BSD, which BSD would you choose? Like is free BSD the choice that people should select first?

Should they choose the portability one? How do you choose? I suppose based on what you just said, it makes sense that FreeBSD is probably the one you'd say. I'd say if you're trying to install something on a laptop, GhostBSD, which is FreeBSD with a default GUI and so on, is probably the best place to start. When I started, I was never trying to install Unix on my machine at home.

It was always, this is going to be on a server somewhere that I'm going to SSH into. And so that is definitely one where FreeBSD excelled. I had a FreeBSD machine I rented in Michigan that I installed as FreeBSD 6.1 and upgraded in place over only SSH. It had no KV or like any remote access upgraded all the way to i think freebsd 11 before i finally retired the hardware because it was too old

and so just that the reliability and because you separate the applications from the operating system upgrading the operating system is a lot less perilous than so often now. It's like, rather than upgrading the OS in place, I'm just going to reinstall because that'll be safer, right? Yeah, that's exactly why

my machine is at 2310 and I haven't done any sort of changes to it because I'm like, I don't want to mess up this machine. And I feel kind of stuck, basically, until 2404 comes out and is blessed as an LTS or whatever then I'm like then I'll move to that probably and I'll probably still even then tiptoe

around it because it's a precious little system what about the home labbers out there who are running Proxmox they just want to play they don't have bare metal to install it on can you run FreeBSD as a VM in proxmox just to tinker and play do you do you know of a good tutorial that you can suggest for that or is it just pretty easy it's pretty easy um like on the official freebsd website you can download already built like qcav2 images that you could just fire up uh or you can get the installer iso and and just do an install the installs yeah okay that's That's easy. Pretty easy. I personally put a bunch of work

in making it less complicated to get ZFS by default. But every time I watch someone do it, I learn more about how not easy maybe it is. Okay. Well, maybe you can watch me do it and see if it's not easy. Yeah, but part of the problem of working with something too much is

I can do the freebsd installer with my eyes closed i know exactly the order of the menus and stuff so like literally put me in front of a keyboard and turn off the screen and i can finish and install a freebsd and that makes me maybe not the best person to try to make the installer easier for anybody else to use the most committed that's for sure yeah because i feel like after this conversation i'm gonna test on on proxmox first and then i'll probably take my zima board or i don't even know can you do a free bsd install on a raspberry pi like it yep there are pre-built images for raspberry pis and like the rock pro and the pine board and And there's a whole list of supportive platforms.

Gotcha. And it's really been nice now that like ARMv8 is pretty standardized where you don't need a special one-off image for every different board now where there's just like this ARM64 image will just work on any modern little ARM board. Right. I was going to say like, it doesn't seem like there's any hardware limitations to playing with FreeBSD.

Is there any that you're aware of that's... Not especially. Certain Broadcom Wi-Fi chips aren't well supported, but that's really Broadcom's fault because they make it impossible. That's their problem. Well, it's FreeBSD's problem, but it's, yeah. And they're pretty popular too. But the Intel ones,

luckily Intel, when they commit the drivers to Linux, they dual license them under the GPL and the BSD license so that FreeBSD can use that source code to get the drivers working under FreeBSD as well. Both for Intel for Wi-Fi and their built-in GPUs. And that goes a long way. What about a FreeBSD system that runs obviously ZFS because it's built in it's open ZFS built in

right? But it's really intended to be a Docker machine running a particular Docker container

that runs Plex and needs access to a GPU and stuff like that. What kind of can you do that with FreeBSD?

Is that going to be a problematic system to build? There's a couple different ways to approach that. Docker specifically doesn't really support Linux, although the more generic concept, the container runtime stuff, there is growing support for being able to do that directly on FreeBSD. For the specific use case you're talking about,

it might make more sense to use beehive which is freebsd's hypervisor okay kind of like kvm on on linux and you could pass through the gpu so you could have a vm that would run you know your docker container and be able to pass through the the gpu doesn't have to be a dark container i mean is there a more native freebsd way to do things that isn't docker related yeah uh freeBSD's container concept is called Jails. They basically are a chroot that contains a different installed OS that can have other packages and so on.

And they have a restriction on like which IP addresses it can use. And you can lock it down. It's since grown a bunch of features like the network namespaces kind of like on Linux, where you can have this basically be almost like a lightweight VM. It has its own network stack, its own files, everything,

and you can do it. And that's how I run Plex on my FreeBSD ZFS machine, is I just have a jail, and then I just, inside that jail, I installed the Plex package from FreeBSD's package system. Like that. It's too easy, okay. Yeah, and it just passed through a mount

from my storage into there and so this way plex can only see media data set from zfs that i wanted not you know i didn't want to indexing my photos that's why like my docker compose setup because i pass through via docker compose which mounts i want you know the plex docker to know about. And it can only see those and that's it, essentially. And that's why I like that process. But I figured there's probably a more native way to do it. Yeah, and those concepts were originally done in 1999 on FreeBSD by a company who wanted it.

They were just trying to support PHP 3 and PHP 4 at the same time as a web host. And so they paid for this feature and they said, we need this as a competitive advantage. But a year from when we finish the project, you can open source it. And so a developer built it on top of FreeBSD for this web host and they used it and then it became open source. That's awesome. And that concept has continued to grow and Sun saw that concept and built their concept of zones based on top of that and that's why zfs has specific support for delegating data sets to a container because solaris had this zone concept that they borrowed the idea from freebsd and then full circle

eventually zfs comes back to freebsd and linux and support for those has landed. And actually, my company did the support in Linux for containers with ZFS. So in ZFS 2.2, which will come out with Ubuntu 24.04, you'll be able to actually ZFS zone a dataset to like an LXD container. So then root inside that LXD container can see that dataset, make its own child datasets, make its own snapshots, and so on.

So one of our customers wanted to be able to run Docker inside of a container. They do hosted CI, and so they needed each customer's stuff to be separate from each other, but each customer wanted to run Docker. And if you just did LXD without this support from ZFS, Docker couldn't use the ZFS driver and get all the advantages of ZFS when using Docker. So many layers there. It's like Inception. Am I sleeping now or later? Am I going to get the kick and wake up yet? What's happening here? Well, especially because you can do an unprivileged LXD container, which has a root user inside that as far as that user is concerned

is root. But as far as the consistent is concerned, it's just an average user that doesn't have any special permissions. For sure. You can have the same user ID in all these different containers, but actually maps out on disk to a different user ID so the system can tell the files apart. And yes, it can get very Inception-y.

Yeah, I bet. What about networking? What can you do fun with networking around FreeBSD? I know that you've got some cool stuff in there, but like if I wanted to tinker with build my own router, just do fun things with ports on the machine I have access to, whatever. Give me some fun stuff. Yeah, that's one of the main way people get started with FreeBSD is using one of the kind of open source appliances based on FreeBSD, whether that's like PFSense or OpenSense as a router, which is a router firewall with a web GUI.

And that's how a lot of people get started. Or there's TrueNAS. If you have a machine, you just want to be your ZFS machine rather than trying to learn it all yourself. You install TrueNAS on it, you get a web GUI, and now you have a ZFS machine. That's core, right?

Not scale. Scale is based on Debian. Isn't that right? Yeah. So TrueNAS Scale is based on Debian. Isn't that right? Yeah. So TrueNAS core is based on FreeBSD and TrueNAS scale is based on Linux because people wanted Docker support. That's right. I was in that crowd going, please.

Okay. One of the ways that people tended to get first exposed to BSD is using it in one of these appliance type situations and then wanting to do just a little bit more with it. And so getting a different machine and starting with a plain install of FreeBSD and learning it more themselves and building something more advanced than the kind of out-of-the-box appliance is meant to do.

What about in your network? Like what are you doing with FreeBSD? You mentioned Plex, what else? Yep, so FreeBSD has two main firewalls that you can pick from. One called PF, which is actually originally from OpenBSD, but the FreeBSD version is quite a bit different now. And then IPFW, which is the original firewall and has some very interesting capabilities around forwarding and changing things.

So using FreeBSD as a router or a firewall is very common. You can use just a NAT router for home or even like a full like BGP router for core internet. Lots of ISPs use the BSDs for their routers because they have very good uptime, all the tools built in, you don't have to worry about the packages changing up from under you.

And long-term support, right? Like each main version of freebsd is supported for five years uh so you know if you use the 14 branch or the 15 branch each of those gives you a whole five years and makes you know generally matches the life cycle of machines so that you don't have to worry about upgrading very often or in particular there's a guarantee in freebsd of stability so if you build something for FreeBSD 14.0, it's guaranteed that'll still work on FreeBSD 14.4 when that comes out.

And only between major versions can anything break that might make your life difficult when you go to upgrade. But stuff you can do in the network, like I said, lots of different options with firewall. And we talked about at the beginning, dummy net for traffic shaping. And it's really interesting capabilities there. but it also has a concept called vnet

where you can have a completely isolated separate copy of the freebsd network stack so in those containers they can have their own complete network stack that means they can have their own firewall so each container can have its own complete separate firewall with its own firewall rules they can even have conflicting ip. So you have two containers that have the same IP addresses. And as long as you don't link them together, they won't see each other and won't have a conflict. And this can be very useful for simulating networks, right? If you need to build something where you have 100 machines all emulate some network for work or whatever, you can do that all within one machine

and because it's using containers instead of VMs, you're not having to dedicate a bunch of memory to every one of those and run out of RAM trying to start 100 VMs. And just a lot of composabilities, it has a bunch of these different layers where you can compose stuff together and be able to create

whatever you want, including using WNET between those VNets to control how much traffic can go where and filtering and all that stuff. Well, what about this scenario? So I have two PyHoles and I have a DNS mask load balancer, essentially machine in between them that I point my router at that single ip address i have two pile holes and i low balance to those two piles based on dns mask if i want to do that in the free bsd world what would you suggest if i want to replace that machine with a low balancer

built on free bsd yeah so you can just install dns mask on free bsd that's it is one option there are other options. Too easy, Alan. Give us a harder way. What might be more interesting is actually on replacing the pie holes with something running BSD

and using CARP, which is actually another technology from OpenBSD, the Common Address Redundancy Protocol. It allows multiple machines to basically share an IP address. And so if one of the pie holes

goes down, the other one will notice, hey, that PI hole hasn't said I'm alive in the last 200 milliseconds. I'm next. And it'll take over. And that way the traffic, you never notice that one of your PI holes went down.

And so it's very common, like web server, DNS server, whatever, to do this so that as soon as a machine goes down, another machine takes over for it. And you never have to worry about it. And then FreeBSD is very popular as a load balancer. You can use RelayD, depending on the

application, like what protocol you're trying to load balance or all the other common stuff like Nginx or HAProxy and any of those applications. I think in this case it's DNS itself. I'm an idiot when it comes to protocol. For DNS, you can also just configure your client

to know that there are two different DNS servers and try them both without necessarily needing a load balancer. Because I can't imagine your DNS load on your pie holes is so heavy that you need two of them. No, it's for fun. It's just a learn and tinker, really. Exactly, and that's doing the carp thing so that you can just unplug the cable from one of them and watch the traffic just instantly switch to the other one

and know the applications don't notice is, you know, a really good skill for a system to practice in their home lab. You'd be surprised by network traffic. It's pretty intense around here. Just kidding. It's just... In that case, Alan, how would you actually know,

because you want to be able to have diagnostics on that down machine, how do you know if you have two addresses, two machines sharing the same address and one hops in and the other one's down? Usually what you would do is they would each have their own actual IP, and then they would share a floating IP. Kind of like you would use in the same concept like Amazon and DigitalOcean and other provide in the cloud. It's not quite Anycast because that requires BG Yeah, like an Anycast kind of a thing.

It's not quite Anycast because that requires BGP, but it's the same idea, yes. Okay. Of those machines communicate and they have an election decide who's the leader and that person does it. And then if you look at ifconfig or some of the other metrics you get out of BSD, you can see, oh, I think I'm the leader or I'm a follower or I know that things are broken and I'm down. And you can also do that across interfaces. So for example, if you're doing this with, say it's your NAT router that you're trying to make redundant, if the internet facing interface goes down, you want it to also purposely take down the LAN facing interface on that router

so that the other router that still has internet will take over. And so you can have this kind of, you can define these relationships so that, you know, if this connection goes down, sure, we might want to change the IP address on the internet side. But also, if we don't have internet, we don't want to advertise us as a router on the internal side. And so one of the other interesting protocols that's available on FreeBSD and OpenBSD is called PF Sync. So if you're using PF as the firewall to do the NAT,

it can sync the state between your two routers so that when you fail over to the other router, it already knows about all your open connections and you don't have, you know, your currently open connections disconnected when you had to fail over to the other router. Whereas normally that state only exists in memory on your router, especially with consumer routers,

that state table is usually the thing that gets full and means you want to reboot your router. Are you sure that works though? Yes. It sounds like it wouldn't work, like it would fail. I'm sure it works. I've seen lots of really big important internet nodes using that to stay online.

I love the idea of Adam having a fleet of Raspberry Pis that are all load balanced in order just to handle his internal network's DNS requests. Just two. Follow Gerhard, Jared. You know that? Two of everything.

If you're into DNS, you should follow Paul Vixie. Okay. And he's written a thing to actually detect the DNS over HTTPS and break it on purpose. Because he's like, my network is a totalitarian regime. I decide who connects to what. And I control the DNS. Firefox doesn't get to decide to go use Cloudflare for DNS. No, no, this is my network.

Oh, I like that. It can do that? Yeah. Basically, it uses the firewall and FreeBSD to intercept the outgoing connections. And then when it sees the ones for HVS, it's like, oh, I'll send a DNS request to it. If that one replies, then I know that's a Doh server.

And nope, that one's not working anymore. He gave a great talk about it at a BSD conference. The video's on YouTube. So that'd have to be out of ban though, right? If you have an HTTPS request. Well basically this would be on your router, you can't use any DNS server except for my DNS server.

Right, right, right. But that detection mechanism would be outside the regular connection, right? You're not like man-in-the-middle-ing it. Right, yeah. The firewalls trigger something external to go and check, is this host actually a Doe server? And if it is, then no, we're not going to let that packet go through.

Smart. Man, to get Paul Vixie on the show, learn how to become totalitarianists inside our own networks, Adam. I want to do that. I'm attempting that. And I feel like I'm being swindled by Firefox. Thank God I don't use Firefox, but whatever. It can do that.

Actually, can Firefox do that? Google's worse. Chrome is worse than Firefox. Truth. So bringing up Firefox makes me think of just consumer-grade software. And what I think of with obscure operating systems is lack of packages, lack of updates. I'm a Debian guy from way back, so I know what it's like to be pleasantly behind the curve and then go grab some different sources because I want that package that's newer than 18 months ago or whatever it is.

How is it with BSDs? Because I assume they probably have less access to software than any other operating system out there. Is that wrong? It depends. So on FreeBSD, we have the system called Ports, which is basically a directory structure full of make files that allows you

to compile a lot of common applications. In total, that generates something about 40,000 packages. So not a small amount of software is properly supported on FreeBSD. And so, you know, Firefox and Thunderbird and Chromium and whatever else you might want to use. And like we talked about, like DNS mask and whatever other applications you might want. Plex, you said was Plex as a native install too, right? Yeah, Plex including the version for if you have a Plex pass. That's cool. And lots of other stuff.

And so in ports, there are two flavors of ports. The default is called quarterly and that's updated basically the versions of the applications are updated once every quarter and then outside of that it's only like bug fixes and security fixes to try to, like you said, kind of pleasantly behind the curve a little bit to avoid the churn. But there's also the other set

called Latest, which is literally, you know, rebuilt every couple of days. And it's all those packages, the brand spanking newest version. So there's a great website called, I think, Repoology that compares the package repos

on like Debian and BSD and all the other ones. And in general, FreeBSD is usually the second freshest just behind Nix, who are somehow crazy and just keep everything even more up to date. So generally, not every single application is available, but most popular stuff is, except for things that contain too many linuxisms

that are really really specific to linux yeah biggest pain in the last couple of years have been more and more applications assuming systemd is available uh which isn't necessarily the case on a bsd that doesn't have systemd but in a lot of those cases we've been able to write shims that can pretend to be enough systemd to get the application to work anyway. Well somewhat more importantly how up to date are the LLMs with helping somebody learn OpenBSD or

like this conversation is going to go away you won't be here to just ask questions of later on when I'm tinkering can I ask a chat GPT and get good answers what's your experience there? You probably don't use it because you don't need to. I've used it a little bit. He writes the books that chat GPT reads get good answers? What's your experience there? You probably don't use it because you don't need to. I've used it a little bit. He writes the books that chat GPT reads.

That's true. That's part of it. And that, you know, there's a lot written about the BSDs, especially, you know, FreeBSD had the best documentation through all the early 2000s

and so on. And all that's still out there. The number of people helping keep it up to date is not grown fast enough to keep up with the increasing breadth of the number of things that need to be written about. And so if you're looking for somewhere to contribute,

the documentation is a great way to get started. And as I was kind of talking about with the installer, someone who's very new to it is the best person to write the documentation on it because they don't know anything about it already. The problem you have is once you know a lot of it, there's things you forget that you know and that the person reading the documentation won't know.

And you tend to skip a couple of steps because they're so automatic, you don't think about them. But it means the documentation suffers from that. And so don't think that you're too much of a noob to help the documentation we actually need your noobness before you get better at it is the best time to write the documentation okay how does one interact with the community so let's say i'm ready willing and able i'm a freebsd noob and i want to help with the docs how do I even get started with that? So we've done a lot of work in the docs in the last couple of years

so they are now written in ASCII doc which is marked down plus a bit and so you can just go to the GitHub FreeBSD and go to the docs repo and make a change and open a pull request but if you want to interact with the the community there's all the common ways uh there's a matrix channel there's a discord there's a forum there's a mailing list kind of depending on how old you are which of those you prefer

and you'll find uh your people there okay it is really interesting to see the stratification of you know people of a certain age want a mailing list, next age was forums, and then it was Discord, and then it was Matrix or whatever. Looks like there's a free BSD documentation project primer for new contributors as well. Yes. Which is a split HTML or a single HTML page, you can just jump in there. That's linked to from the repo. Yep. And, you know, that's also a great place to get started as you're trying to follow those

instructions if you see some extra detail that would be helpful to someone who's trying to do this for the first time then that's a great place to contribute that's cool but yeah compared to the old days when the documentation was written in doc book xml and had uh all these terrible things you had to learn and all these tools you had to be able to do before you could start working on it now that it's basically marked down if you can write a paragraph you can contribute to the docs All these terrible things you had to learn and all these tools you had to be able to do before you could start working on it. Now that it's basically marked down, if you can write a paragraph, you can contribute to the docs. How does the FreeBSD project make progress? What is the governance and how does it all work? The FreeBSD project itself has what's called the core team, which is a group of nine people elected from people who have

right access to the repos every two years. And so instead of having, you know, one person in charge, like Linus Torvalds, instead in FreeBSD, every two years, we just elect nine people to make up the team of people that help decide, you know, who gets right access to the repo and deal with if people are having an argument over how to decide what to do, that happens there. But really the core team's more about trying to direct people in the same direction and cheerleading efforts led by individual developers rather than strictly deciding exactly what the project's going to do next. It's much more sustainable governance

in that we're not depending on one person who can get burnt out or just be bored of it or just disagree with everyone. By having an elected body that changes on a regular basis, it means there's lots of chance to get new people into the leadership roles. I did two terms of that myself a couple of years ago. Very cool.

If I was coming to FreeBSD from my world, Ubuntu, Linux, you know, only really, what would I have to unlearn? Like the core things like I know as a Linux user primarily, and I'm going to unlearn them to do the FreeBSD way. I think the biggest one is you have to stop assuming your shell is Bash. But you can just install Bash and have it on BSD if you want. It's just not

the default. What is the default? Just bin sh, which is the original Unix sh. It's mostly equivalent to what is ash, which is what dash is on Ubuntu. So it's just support, doesn't support arrays

and a couple specific dashes, but not a big deal. I think the biggest thing that you'll find surprising is that all the software you install doesn't go into just like user bin or whatever. It goes under user local bin

and the config files for anything that doesn't come with FreeBSD will be user local ETC. Basically, we prefix all the paths of software that doesn't come with FreeBSD will be user local ETC. Basically, we prefix all the paths of software that's not part of FreeBSD with user local. And that means that if you screw up all the packages, you could, in an extreme case, just delete that whole directory and reinstall everything without touching the operating system and it wouldn't break anything. Okay, that's it. Those two are pretty easy. Yeah. And I guess the third one is that there's not slash proc by default on BSD. Most of the stuff you would find there will be under sysctl on FreeBSD instead.

There is a slash proc you can get, but it doesn't have very much in it. Or if you're using FreeBSD's Linux emulation, where you can actually run unmodified Linux binaries directly on FreeBSD and it will translate the system calls from the Linux version to the FreeBSD version. There's a linprocfs which provides a slash proc that looks like Linux but it's not there by default. What about like you said it's like systemd and stuff, what about?

So FreeBSD still uses cron for periodic jobs and it has its own other system called periodic which is more about like I want to run this every Sunday and it's going to do like a security audit of my system. There is a system called periodic which is more about like i want to run this every sunday and it's gonna do like a security audit of my system there is a system called devd which uh can automatically respond to things from devices so like oh you just plugged an ethernet cable in do you want it to automatically try to run dhcp on that interface or you know you plugged in your headset do something uh some of that stuff but not something quite as holistic and all-consuming

as systemd yet having something like that maybe like launchd from mac os or or some other thing that does some of that i could be interesting there a lot of those building blocks so there are like supervisor things that can do some of the individual components of what systemd does that you can have but there's not something that tries to bring all of those different pieces together yet. But it does mean that when a service doesn't start, it's much easier to figure out why if you're a system that understands shell scripts. But I understand that at the same time, there are interesting functionality that there's no direct analog for.

So being a BSD guy and mentioning LaunchD, curious your take on macOS as being rooted way back there. Isn't Darwin and BSD like the same core starting place? So Darwin's mostly its own kernel, but the basic utilities like LS and DU and all that stuff were borrowed from FreeBSD 5 originally and then slowly updated over time. There's a lot of BSD code in Darwin,

which is then underneath macOS, but there's also a lot that is very specific to them. Sure, sure, sure. So have you used macOS? Can you compare it to your life in the BSD world? I've used some macOS. It is somewhat comforting to have the command line tools you're used to, even when you're on a pretty operating system.

But I don't have that much experience with it. Fair. But, you know, Microsoft borrowed a lot of code from BSD as well. Like, if you've ever run netstat on Windows or ping and traceroute, those are all originally from the BSD code base. Really? Yeah. I would expect them to be fundamentally different.

I guess the way you ping things isn't super low level or I don't know. Well, it's more that the whole concept of having a TCP stack, when there was one available open source for free and it was the reference implementation, they've changed a lot about it over all the years since then. But the reason why some of those commands match up is because the original code was...

But some of that even predates FreeBSD, right? This is going back to the early days. The primordial ooze. Yes. When DARPA was funding the development of the internet. Yes. When the government, when DARPA was funding the development of the internet. Yeah. I don't know.

I'm pretty sold, Adam. What are you thinking? Seems cool. I'm thinking FreeBSD is an install away for me because I'm about to There you go. about to spin it up.

I'll probably play on Proxmonx for a bit and then find a reason to swap something out and actually go bare metal once I learn enough. But it seems like it's got the right kind of building blocks really for me i mean yeah it's maybe the best criticism of freebsd is it's a lego set it has all the building blocks but that last 10 to make it into something perfect is not included

yeah sometimes so like with the zfs stuff it's quite polished but you know for example freebsd make it into something perfect is not included. Yeah. Sometimes. So like with the ZFS stuff, it's quite polished. But, you know, for example, FreeBSD's had jail since the year 2000, but didn't quite get to having something like Docker where you're just saying, I want this application and it gets set up as a container for you. Yeah. There are lots of tools to make containers and then you go inside and install the packages or whatever, but not something where there's just a repo of already built containers that you can just slurp down and start running partly because you

know there are definitely security problems with just slurping down an image that never gets updated versus using the package manager to constantly keep the packages in a container updated but uh that's what i would say is that you're expected to add that last little bit to turn it into something which is also why you see so many appliance type things built on BSDs because they're designed to be these

this Lego set that you can build on and then add your custom stuff onto and not have the same license compliance issues that you might on something that's GPL licensed.

Yeah. I'm really interested to try out a machine with ZFS. I know you keep saying ZFS, I think because you're Canadian, right? Are you Canadian? Because I'm Canadian. I thought you were Canadian.

I picked that up along this way. You were saying process. I used to live in Canada, so I know. And it's not Toronto, it's Toronto. Anyways, just showing my Canadian cards there. So ZFS is important to me, obviously as a file system and storage system and then running plex you know i'd love to see how that works for me with i think i have a an intel 13900k

cpu that is the main system for that and it was mainly because i wanted to transcode 4k very easily and have no issues whatsoever. And so I built a system around that CPU and ECC RAM, ZFS, obviously, but it was an Ubuntu system, not FreeBSD. My Plex machine is my build server. So it's a dual E5-2690. So it's 40 threads at 3 gigahertz. So I didn't bother with a GPU for transcoding.

Yeah. Well, the problem with GPU versus CPU I found is not that the CPU can't handle it. There's certain things the GPU does video codec wise. It's not that you got so many CPUs and they're strong. It's that you really need the logic.

In particular, I think it's SmartSync, if I recall correctly, in the Intel GPU, the iGPU. QuickSync. QuickSync, yeah, thank you. I was thinking Smart, but Quick makes more sense, and that's true. But QuickSync generally is much faster than doing it on the CPU, but you can't get the same quality as you can get on the CPU.

But on the CPU, it's so much slower that usually at any reasonable bit rate you don't care about the difference in quality because without a magnifying glass you're not going to be able to see it. Yeah, and I don't know too much about the way Plex interfaces with the hardware but I know that there's the hardware encoding option

inside of Plex and I think it leverages that iGPU and QuickSync specifically because Plex has worked well with the Intel CPU and so I didn't want to go the hardware. I want the easy, I wanted the easy button really. So I went with the Intel QuickSync iGPU system with lots of cores so that when I want to do other things on that system, that it's a multi-use system. It's not just a Plex server, but it's, that was one of the primary

uses of it just to see what I could do with it. But I really did not want to have an issue with my 4K stuff. I wanted to be able to rip 4K and watch 4K and also transcode 4K with zero problems. And I had problems on other CPUs and other setups and that's what I ultimately landed on. Is it the best system ever? Maybe not, but that's what I chose. Yeah, I don't tend to use the transcoding very much

because I mostly watch on my TV that just takes the plain 4K and not have to, you know, I'm not trying to stream it to my phone and dumb it down to my phone screen. I got kids. So, you know, that's why. Yeah, my niece and nephew are the fourth busiest user on my Plex. There you go. There you go. Very cool. Well, that's that would be my first use case would be that playing more with a native version of OpenZFS. Obviously, I'm using OpenZFS on Ubuntu,

but it's not maybe the same way you can set it up with FreeBSD. The ZFS code itself is basically the same. So since FreeBSD 13.0, it's been the OpenZFS version as opposed to the previous Illumos version of ZFS. So in FreeBSD 13.0, it's been the open ZFS version as opposed to the previous Illumos version of ZFS. So in FreeBSD 13.2, I think it's ZFS 2.1.6,

which is almost identical to what's in Ubuntu, which is 2.1.5. But if you install FreeBSD 14, you'll get ZFS 2.2, which you wouldn't get on Ubuntu until April. Yeah, I want it now. And so you'll get access to all those new features that are there. Yeah.

And then, as we talked about, all the integration. So there's a tool in FreeBSD called BECTL for controlling those boot environments that'll let you take a snapshot of your system before you try the next thing in your lab. That way, if it breaks, you can go back and unbreak it. Or even if you do break it, you can keep the broken one around

to go back to it and work on it later. Yeah, that's what I like about the built-in nature of ZFS in the operating system is that you can do things like that. I may not do it every day, but it'd be nice to do it whenever. I'm like, okay, this upgrade is risky. Let me snapshot prior to and do the upgrade.

And as you say, if it gets botched, roll back. I don't have that option now. It kind of sucks. You went to try to get most of the way there, but got distracted. Yeah, for sure. What else, Jared? Anything else to cover on this BSD journey with Alan? No, I think we've

pretty much covered it. For me, at least. What have we not asked you, Alan, that you're like, man man why did you not ask me this question about my love for bsd or zfs or being assisted man whatever your podcast maybe even the two and a half admins i'm a fan i listen to that not every day mostly when it's about zfs which is almost every episode but uh a couple things i can drop if you are interested in bsd i'm not the host anymore but there's the

bsdnow.tv podcast comes out every week there's news about the BSDs if you want to keep just keep pace of what's going on there or see interesting articles or people have wrote up

how to's on doing different things on BSD maybe Adam can write up his journey with setting up Plex if he ends up doing that that's a great one

and then like Adam mentioned I host two and a half admins podcast. If you do Sysmin stuff and want to keep pace with what's going on or just talk to some people that have been doing it for 20 years and have lots to

share, then definitely check that out. And then for my day job, I co-founded a company called ClaraSystems.com and we do support for ZFS and FreeBSD as well as development. So we build custom features for FreeBSD and ZFS. So for example, we talked about the support

for LXD containers on Ubuntu built into ZFS. That feature came to our company and said, we want this feature and we built it and upstreamed it and made it part of OpenZFS for them. And we do the same thing for ZFS on Linux and BSD and for the BSD kernel itself. Very cool. Well, thank you for your book.

Thank you for all the knowledge you put out there. Podcasting, of course. I mean, it's one thing to know. It's another thing to tell it. And you do a great job of doing that. I really enjoyed that book of yours as a primer for me with ZFS. And so I appreciate that. I'm pretty sure I bought it.

I think I bought it. Yeah, well, there's the advanced version, the advanced ZFS that follows on to that if you make it all the way through the first one. You know, I think I may have skipped that one. I bought that, the initial one, and I think I just got on my way. I don't know if I ever went back to the advanced. I mean, I'll look at the table of contents and see what's in there. Yeah.

It's mostly the more tuning in and getting deeper into replication and stuff. Yeah, that's the fun stuff. I should get that then because I like to send, receive, replicate, clone, all those fun things. Mainly, it's one thing to stand up a system. It's another thing to keep it up

and also to back it up and to ensure that if something happened to that machine that your data isn't gone. Yeah, well, especially if you're using Proxmox and every VM is based on ZFS, being able to replicate that to another machine so if the Proxmox machine dies, you can just fire it up on a different machine

is really valuable. And being able to use the incremental replication to keep a copy in sync without having to re-replicate all the data all the time. That can make a world of difference there. And that is in this other book, the master version of it? Yep. All right.

Yeah, so FreeBSD Mastery Advanced ZFS. Advanced, that's right. Yeah, so both of those are at zfsbook.com. Yeah, don't let the fact that the title says FreeBSD frighten you. All the ZFS commands work exactly the same onbook.com. Yeah, don't let the fact that the title says FreeBSD frighten you. All the ZFS commands work exactly the same on Linux. Yes.

Some of the tunables are, you know, it'll be sysmodules, zfs, blah, blah, blah, instead of what it is on FreeBSD. But outside of that, everything is the same now that the ZFS on Linux

and the FreeBSD ZFS code are the same code base. I concur. And hopefully in the next couple months, the mainline support for the community macOS version of ZFS will land in OpenZFS. Yeah, that'd be fun. That'd be cool.

And that way, the same code will be available for all three operating systems. Okay. We'll link up those things in the show notes for the folks. Check that out. Alan, thank you so much. Appreciate your time. Yes. Thank you. There it is. An entire episode dedicated to BSD. Finally, I know some of y'all have been waiting for this one. We're happy to fulfill that request. And if you have requests of your own, send them to us at changelog.com slash request.

We read and consider each and every one. So even if your episode idea doesn't pan out, it still influences our thinking and we love hearing from you. Also, did you know our transcripts and show notes are open source on GitHub? Seriously, they are. That means you can help make them better by just editing a little markdown and opening up a pull request. We love when that happens and it's a small but powerful way you can be part of our changelog community.

Another way to do that is by telling your friends and your colleagues about what we're up to. Word of mouth is still the number one way people find us. And when they do, they'll thank you for helping make that happen. And we'll thank you right now. Thanks, for real.

We appreciate it. Thanks also to our partners at Fly.io, to our Beat Freakin' residents, Breakmaster Cylinder, and to you for listening. We love that you choose to spend time with us each week. Coming up next, Adam is joined once again by YouTuber TechnoTim to talk

the state of the home lab. That's all for now, but we'll talk to you again on Friday. Thank you. Game on.