The Changelog: Software Development, Open Source - It's a TrueNAS world (Interview)

Episode Date: March 22, 2024

This week Adam talks with Kris Moore, Senior Vice President of Engineering at iXsystems, about all things TrueNAS. They discuss the history of TrueNAS starting from its origins as a FreeBSD project, TrueNAS Core being in maintenance mode, the momentum and innovation happening in TrueNAS Scale, the evolution of the TrueNAS user interface, managing ZFS compatibility in TrueNAS, the business model of iXsystems and their commitment to the open-source community, and of course what's to come in the upcoming Dragonfish release of TrueNAS Scale.

What's up, welcome back. This is the ChangeLog. I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog.com, and I'm going solo today talking to Chris Moore, Senior Vice President of Engineering at IX Systems. IX Systems is responsible for all things TrueNAS and you know I love TrueNAS. We discuss the history of TrueNAS starting from its origins as a free BSD project, TrueNAS core being in maintenance mode, the momentum and the innovation happening in TrueNAS scale, the evolution of the TrueNAS interface, managing ZFS compatibility in TrueNAS, the business model of IX Systems and their commitment to the open source community, and of course, what's to come in the upcoming Dragonfish release of TrueNAS scale.

All my home labbers out there, put your hands in the air. This is a good one. Of course, a massive thank you has to go out to our friends and our partners over at fly.io. That's the home of changelog.com. That's where we live. As Billy Madison said,

go to your home. Launch apps near users too easy. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents start for free at fly.io well friends i have some good news for you it It is launch week once again for Sentry. And I'm here with Rahul Chhabria from the product team at Sentry. So Rahul, can you tell me about the launch week this year for Sentry?

We're making a huge investment into our product platform. We're trying to make it faster, better. In November, we shared a sneak peek about our new metrics offering. So now developers are able to define custom metrics they care about and monitor how quickly their app is responding to the business measures they have to be accountable for, plus also like the customer experiences that they've committed to. And that's going to be available in an alpha. People can sign up to get access. We'll turn it off for them like in a couple of days once they write in and they can get going right away. Now, on top of that,

it's like we are looking more at how do we make the product smarter? Now, I know the world is talking about AI and ML and they're all solving like, we think like entertaining problems, but Sentry is taking a more thoughtful approach to it. We are trying to look at what is the developer trying to do? Like our goal is not to have you sit in Sentry all day long. Our goal is not have us like be a tab

that you need to keep open. Our goal is to have this be a tab you open when to keep open. Our goal is to have this be a tab you open when something is wrong, give you the information you need right away, tell you how impactful it is to your user base and if you should care. And if it is something you should care about, here's how to fix it.

So we're taking a broader look at how developers use the product. Where's the noise they're seeing? Like, are they seeing repeat events? Are they seeing like things that are not critical rise to the top and have to automatically resolve them or ignore them? So we're going to make Sentry a little bit smarter with artificial intelligence to give you

a more prioritized view of the issues that really matter so you can solve them quickly and move on and not be distracted by random rage clicks that are, you know, just ghost issues. Okay. Sentry's launch week is happening as we speak. Check the show notes for a link to the launch week page. I'll be showing off new features, products. You can tune into their YouTube channel or discord daily at 9am Pacific standard time to hear the latest scoop. Or if you want to get swag along the way, enter your email address at the page. We'll link up to get swag all along the way or join the Discord, whatever works for you. The next step is to go to sentry.io slash events slash launch dash week.

Again, that link is in the show notes. I am here with Chris Moore. Chris, I've been looking forward to this conversation because I am a TrueNAS fan, and I hear that you work on TrueNAS. You're the Senior Vice President of Engineering for IX Systems, so glad hear that you work on TrueNAS. You're the senior vice president of engineering for IX Systems, so glad to have you here on the ChangeLog. How are you doing? Oh, really good. It's a pleasure, Adam. Hey, I'm a geek at heart, and I love to come and geek out and talk TrueNAS all day, so this should be fun and entertaining. Looking forward to it. How long have you been doing what you've been doing for IX Systems? I mean,

give me a spectrum of your experience. Sure. So I'll give you the brief history bio. So I've been with IX going on just about 18 years now. So that's a pretty good career, especially in the tech industry. That's a long time. It is.

Staying with one company. Almost the age of ZFS. Just about. Just about. Which, I mean, that is an oddity in our industry. Just talking to someone who's been with a single entity that long. So, I've been with iX a lot of time now. I started kind of messing around with BSD Unix back in the 90s. I was like most software engineers. I did a stint in support, so a lot of time on the phones dealing with customers in the early, early days. And then I came over to iX, I think, what 05, late 05, early 06 kind of timeframe.

Originally, I was the guy who did a PCBSD, which was a desktop version of FreeBSD. So we took things like KDE and different window managers and wrote a custom graphical installer and all that good stuff that made FreeBSD desktop, if you will. Did that for IX for many years. And then about six, seven years ago, I got pulled into the TrueNAS side. We were having a lot of success with

the FreeNAS and TrueNAS product at the time at IX Systems and started doing a lot of QA, helping out with those efforts within IX. And then over time, one thing led to another into the senior vice president role I'm at now, where I have a team of engineers, both hardware and software engineers, everyone from doing UI all the way down to file system development on ZFS. And then quality engineers, automation engineers, people doing testing, docs, IT, all that good stuff wrapped up around the engineering side of the fence here at Ike Systems. I think it's kind of interesting because my journey here has been like, I came for the ZFS and I kind of stayed for everything else, you know? Like, it's kind of funny. My first adventure with ZFS was like bare metal Ubuntu and like managing it from the CLI. No UI, right?

Just understanding the actual Zpool or ZFS commands from creation to configuration updates and stuff like that. Creating pulls literally in the CLI, like a crazy person, you know what I'm saying? No, hey, that's how a lot of us started. So you must have started then ZFS on Linux came a little later. So I had a little bit of a headstart, I bet, because we got that in FreeBSD a bit earlier on. Right. And I remember even in the early days, I wanted ZFS integration and PCBSD. We were still running on the older UFS file system on FreeBSD, which, you know, I mean, it worked.

But ZFS was the shiny new awesome thing. And, you know, it still is a great file system even for desktops in that way. But, yeah, same thing. I was one of the first guys working on support for it to allow the bootloader to do things like boot environments where you could make a snapshot and then roll back if, say, an update went bad. Like all that good stuff. I cut my teeth on a lot of those things in Proxmox installation is a dual-disk ZFS-backed installation. So the operating system level SSDs are dual-disk and ZFS. However, my TrueNAS box, I don't believe is, unless it is by default. Is it by default?

You mean like boot device mirrored kind of thing? Right, like redundant drives. Yeah, yeah. So TrueNAS, it depends entirely how you installed it. If you only have one boot device, it's not mirrored, of course. No, it is actually. Now that I think about it, I have two boot devices, so I guess it would be mirrored. I just can't recall what I did. Yeah, if you went and set that up, then you should have mirrored redundancy.

We see less and less people doing that these days, but in the early days of FreeNAS in particular, that was a lifesaver because a lot of people were using cheap boot media, USB sticks in a lot of cases, and those are not great, as you can imagine. That $20 or $10 USB stick you got at the local Walmart probably won't last that long. That's not what I'm using, Chris.

Hopefully you're doing better than that. But in the early days, that was the thing, was you'd go buy cheap boot media, USB-based, and then mirror it, and that was kind of your first level of protection from losing your boot device. These days with modern M2s, a lot less of the home labbers are doing that. But it's still cool to have the option. We have found on the enterprise side over time, especially with SSDs,

they tend to wear out right around the same amount of time especially if you're running mirrored so it's not really worth it yeah so you're not buying yourself much apart from if you have a catastrophic failure one of them really does bite the dust horrifically yeah in my case i'm literally on ssds not m2 ssd or mvme So it's literally two HD store, 250 gigabyte drives. That's my setup in this case in particular. But hey, those

might die at the same time too. You just never know. I mean, you never know. For now, I feel pretty safe though, Chris. So don't harsh on me. You're good. You're good. I'm not going to jinx you here. Hey, just take your config back up every once in a blue moon. Anytime you make major changes and then worst case scenario, okay, if they both bite the dust, you reload, you restore your config,

and you're up and running again. I mean, I'm running with a single M2 on most of my devices. I usually use like a 120 gig boot device. And I just back up my config. I had one of them actually fail on me about six months in. And I just swapped it out and I was up and running a few hours later. It just wasn't a big deal. I suppose, could you back up that config then?

We're getting a little ahead of the story to some degree, but quickly here. Could you back up that config on an automated basis inside of TrueNAS to, say, a store that you have on the drives or on the system, I suppose? So there's a couple ways you can do that here. Yes, if you're willing to use the API, our TrueCommand product, which is our single pane of glass kind of TrueNAS companion, does that for you automatically. So it will go and take a snapshot of that config

every time changes are detected, actually. And then you can go pull it back out of the TrueCommand UI and restore it if you need to. So a couple of different ways to do it. There's people out in the wild who do scripts and all kinds of stuff to email it to themselves every so often.

It's not a very big file. Well, now that I think about it, like what sense would it make to have it on a ZFS store in a system I can't boot? What, to have the – Well, like if my primary operating system drive took a dive and doesn't work anymore, and I'm backing up to stores that are on the system I can't boot. Yeah, that doesn't help you. You need to back it up off-site. That's why it doesn't really help. Unless I can move those drives to a different machine. You've got to have a way to retrieve it. So that doesn't make a lot of sense. That's why most of the people either set up something to automatically email them the script or fetch it periodically via the API or

use the TrueCommand product and just let it do that for you. Let's get into the meat and the meat and the potatoes as they say. So I mentioned true NAS, true NAS has a couple of different flavors. You got core, which is the old school version of it, which is the, actually the new old school version of it.

And you have scale, but the history goes beyond that. You mentioned, I think free NAS in your introduction of like your history at IX systems. Like, how do we get to true NAS scale or core? What was before that? What has been ixSystems' adventure into hardware sales backed by really good open source? How's that worked?

Sure. So FreeNAS, ixSystems has been involved with that since 07, I believe. It's been an interesting journey. It started as a FreeBSD project just to do basic NAS functionality by a fellow out in France. And he ran that for a while and kind of ran out of steam and was ready to move on and do something else. So IX Systems adopted the project, if you will, at the time. And kind of built it up over the years. I mean, in those early days, it was even pre-ZFS. We're doing UFS NAS. But then ZFS came along and like just kind of changed the game for everyone.

Like this was the file system you wanted to run. Zettabyte means something, right? Like Zettabyte file system. Oh yeah, copy on write being have instantaneous snapshots, all the data integrity stuff built in. That was a game changer for FreeNAS back in the day. And so IX just built that up over the years. Eventually we launched an enterprise product line based on that, some of the FreeNAS back in the day. And so, IX just built that up over the years. Eventually, we launched an enterprise product line based on that, some of the FreeNAS Pro line, if you're

old school enough and remember that. And then over time, that got rebranded into TrueNAS. So, TrueNAS was our commercial offering and FreeNAS was the free open source offering. Well, a few more years went on. I came along the scene and we went under a major effort to unify the software because for a long time that we ran two different builds. We would do the free NAS open source bill, and then we would take that and go package it up and build the enterprise thing. But the enterprise thing was still all open source. So it wasn't like we had a real reason we had to do that. It was more just a legacy. This is how we've always built it. So we came along and said, you know what, let's see if we can unify these images.

They're all open source either way. Let's have one image that does everything. And that led to the branding unification too, because now it's a little awkward. You have a free NAS image and a true NAS image. We decided to go ahead and rename them all to just true NAS. We ended up with true NAS core, basically the free thing you download off the website and then true NAS enterprise, which is the thing that's paired with our hardware, which we dubbed TrueNAS Scale. And again, porting the same open source middleware. So a lot of it's all sharing the same base. It's just we swapped out the BSD underneath to Linux.

Actually, Debian-based is what we are today. And we did a lot of work to port ZFS and make, I guess not port it, that's the wrong word, to make it very portable between the two platforms. We wanted to have the FreeBSD ZFS and Linux ZFS be 100% in parity if possible, because you need that flexibility for people who are migrating back and forth and being able to share pools and all that good stuff. So we spent quite a bit of time to make all that work happen in the ZFS side. We had to add some kind of messing functionality you wouldn't have had on your Ubuntu back in the day, which was extended attributes and ACLs support the full NFS before style ACLs on Linux. That just wasn't a thing yet. So we undertook those efforts to get them into the scale product to now where we're at today,

which is pretty much feature parity with everything we had on core, plus all the other fun little benefits we got being on the Linux side of the fence with access to more fun things to play with. And I suppose now you might be at another branding conundrum potentially with like, there's some concerns out there like, is core going to die? Is scale the future? There's, I see Tom Lawrence, I'm a fan of Tom sharing this information. There's just some confusion, I suppose, again, in the TrueNAS water. Can you explain that and what might be happening? So I'll do my best

to try and put people at ease and explain the situation. So core, again, FreeBSD-based. It's a lot more stable. It's mostly in a maintenance mode, and we're trying to keep it very rock solid, and we're going to keep supporting it. Again, we've got enterprise customers running this, so it's still here for a while to go. And then on the Linux side, that's kind of the up and comer. That's the teenager in the house, right? A lot of energy, a lot of new exciting things are landing there and we're seeing all the momentum shift to that variant of the product. So that's not a real secret. Again, anybody who's paying attention can see that with their own eyes. But that said, you know, we're not dropping people on their head. We're going to support core as long as it makes sense to do so. And especially while we have enterprise customers who prefer to run that platform, just because some people don't want to do any kind of upgrade or migration. They're pretty happy just to set up the NAS and not touch it again for five years. So makes sense. So we're going to do our best to support both, make sure security is good in both. But again, it's not a secret. Scales were the innovations

happening these days. So if you want that shiny new feature, it's going to be on the scale side. And if you're a home labber, this is particularly important for anybody who's a do-it-yourselfer. The hardware support's just flat out better on the scale side of the fence. The Linux kernel, again, no offense to the free BSD guys. It's been a great product. And I've been a BSD guy for almost 20 years. I was on the core team a couple of times. So, you know, that's where my heart was at.

I get it. But at the end of the day, just as far as sheer breadth of hardware support and kind of features and functionality and all the bells and whistles, that's going to be on the Linux side. So I didn't hear anybody say that Core was on maintenance mode, and you said it's in maintenance mode. Well, maintenance mode in the sense that we're not going and actively breaking things, adding new features and functionality. We're very conservative there. So whether you call that maintenance mode or not, it just means we're very careful on what we're putting in. So you're not going to see big, shiny new features land on the core side.

It's more about making sure it's rock solid. The support and features it has doesn't regress, doesn't change, stays fast, stays stable. That's what we're all about right now on the core side, because honestly that's what a lot of people just want their NAS for, is they want to send in the corner, not touch it, not look at it, and hopefully everything just runs and ZFS does its thing. And so we're going to aim to keep providing that experience. Now, not to say it's not that on the scale side.

Scale, especially over the last couple of years, has reached a point where it's just about 100% as stable as core is, especially for core NAS functionality. If you're running SMB, iSCSI, NFS, it's more or less an equivalent experience, maybe better if you have hardware that's not as well supported on the BSD side of the fence. Gotcha. Yeah, I found myself, so we had Alan Jude on a while back

and he reinvigorated and clarified, I would say, the fact that BSD, free BSD, is not Linux. So hammer that home for sure. Whereas someone who doesn't really, I guess, swim deeply in the Linux waters, understand the difference between a BSD or a free BSD or the different variants there. Or I think there was a different word for it there.

Whereas in Linux, you've got variants. And I think in BSD, they're just like, I don't know, targets or something. I forget what he called it. I can't recall in this moment. Kind of like distros versus it's an operating system. Right. Distros are, because you think about Linux, it's a kernel plus a bunch of other things to turn it into a distro, right? FreeBSD at its roots is historically a complete Unix-like operating system. That means the kernel, the user land,

all that stuff's designed in concert to work together. And there's pros and cons to that model versus say the more modular Linux model. So FreeBSD, you'll end up with a vanilla system and they're always all the same. They're all just FreeBSD. And then you go add your application layer on top of it which is your packages and your ports ways to extend it but there is like a core layer if you will which is more than just a kernel it's also the user space

the shell the bin utils all that kind of good stuff the compiler is even baked into the base system all that whereas linux is a little bit more you get to pick and choose and pull pieces out and put certain pieces in and that's how you end up with red hat or debian or seuss or whoever yeah for sure so i guess the question is if somebody is new to trunas they should choose scale right at this point that would be my recommendation especially for a home labber i mean if you're just starting your TrueNAS adventure, I would recommend you start with scale first. Again, if you're procuring your own hardware and all that and you want the best shot of it just kind of working out of the box, no questions asked, scale is going to be your best bet. Plus, you'll have access to some of the shinier new features.

The UI is a lot prettier. Actually, I'll back that up, too. We've done a lot of work on the scale side to reinvent parts of the UI to make it more user-friendly. So if you're just getting started in particular, you'll probably find it a little easier to get that initial SMB share created or bring up your pool for the first time if you're not like a ZFS god already, right? Right. So those are the kind of things we've done. So it has those niceties. CLI junkie.

Yeah. If you're a CLI junkie, you could probably jump into core and just be like, I get it. I know what it's doing. I can see what it's doing from the UI. And I know on the back end what that means. But if you're a little bit more of a novice in those waters, I'd say start with scale. You'll probably just end up with a better experience overall. Yeah. I'm using scale. I considered using BSD because of the conversation with Alan and just reconsidering my adventure, I suppose, into Linux and which one I should choose. And in

my case, it is a Homelab environment. At the same time, it does have changelogs archives on it. So it's also sort of like Homelab slash enterprise production. Like those, that ZFS store cannot die right that would be catastrophic i do have a backup plan rage is not a backup plan i get it good i'm glad you said that

because you were gonna make me if you didn't and i think that trunas itself scale or core is very helpful with having a better backup plan because of just the u UI and the built-in replication. Sure, I can do it as a CLI, you know, junkie, so to speak, from my beginning days with ZFS, which I'm comfortable with, but you got to trust your own scripts and the code that I wrote in concert, maybe with a LLM or a chat GPT-4 or whatever, you know, like, so I've got to, I've got to trust that stuff and I've done it and I have trusted and I've tested it. But at the same time, I was like, is this really viable? Am I trying to be, you know, a bash script kitty for my ZFS backups? Like, do I want to do that? Or should I trust TrueNAS? Should I finally take this hardware I've got and move it to a TrueNAS option? I did that recently in the last month.

So familiar with TrueNAS, but I don't have years of history, whereas I have years of history with ZFS directly. I guess that being said is, I'm not really sure where I'm trying to go with it, except for that you got a great UI for all this stuff. It does seem very stable. However, I have hit a couple things that really just drive me crazy, and I can't believe I hit these bumps.

We can talk about that now, or we can go deeper into some other worlds. What do you think? Yeah, no, we'll talk about that. But first, I just wanted to make the point, I think a lot of people hit that place like you did, where it's like, am I going to be a weekend warrior script kitty and beta testing my own custom bespoke solutions? All I need to do is backups, right? It seems like a simple task we all have. So a lot of people hit that fork in the road. And at the end of the day, yes, TrueNAS is all open source and built on open source. And sure, you could go roll your own Arch or Debian and do it all by hand. Right. But some of us have lives we want to live also

outside of just sitting there troubleshooting our own, you know, gear. And so TrueNAS comes in, it's like, look, we've done all that work for you. It's a pretty good size team we have here just to make sure things like replication just always works. Backups always work. They're reliable. Because once you go down that rabbit hole, it's like, oh, well, I got the replication working, but how do I monitor? Well, I got to set up alerting. How do I do that? How do I make sure that? And it just on and on it goes, right? Well, how about data integrity? Am I monitoring my disks enough?

Am I getting alerts properly? Am I backing up? Now I got to back up to the cloud too because I need offsite. So now I'm scripting our clone or something else, you know, and it's just like, at some point it's like I got better things to do. I felt that pain.

I was like, I'm going too far with this. Yeah. And I was like, well, you've got a great UI. You've got smart monitoring built into Trina. There's so much into it. The data replication I did like, I'm going too far with this. Yeah. And I was like, well, you've got a great UI. You've got smart monitoring built into TrueNAS. There's so much into it. The data replication I did recently, moving data around, I was like, hmm. I just pushed a button, made sure the data set was there, and chose to replicate from one data set to the other. Because I was moving from disks to disks within the same machine, but different pools.

And so I was moving from old pools to new pools, essentially. Import of those pools, et cetera, all that good stuff. But I mean, in my CLI ZFS scenarios, it was a lot more just challenging just because you had to remember all the commands and don't get it wrong and doing zfs send receive by hand i mean you got to have a slight master's degree or a good tutorial with clarity so that you don't mess it up and you don't want to mess up a multi-hour send receive right like if it gets there and at the end you have an error like what what a waste of time right yeah or you screw up the remote side and your snapshots aren't in sync now now i gotta start over like it's it's one of those things it's a lot of pain and time to do it right and do it well or you can just use something that we literally if you're home lab or provide for free already that does exactly that and not only is tested by us but by

the other hundreds of thousands of users doing it every single day. So it's like at some point you just hit that wall where it's like, this is just a lot easier to deploy on our system. But what's cool with your background is there's no real mysteries or secrets. You kind of know what we're doing under the hood. There's ZFS replicating under the hood and they're doing snapshots and they're doing all the stuff that I'm familiar with. So that should hopefully provide you a bit of comfort and other folks comfort that there's no black magic here.

It's, it's all very straight, open, open source things. And that's good. That's something we pride ourselves on. We're not, we're not doing anything secretive in that sense.

It does give me comfort in that. And I would actually, if anybody came to me and said, Adam, what, what should I do? If I'm getting into ZFS, what should I do first?

Well, I would say eventually move the true NASAS but I would say maybe at first stand up your own pick your flavor of Linux in my case I'd probably say Ubuntu pretty solid out there install ZFS I mean in a moment with some disks available even in a virtualization state

like a Proxmox with virtual disks you can toy around with a proverbial ZFS box that is for storage and learn. And I would say go with the CLI for a bit, just so you know what it's actually doing and the different things you can do with it. Just for the same reason you just said, like now you know, Adam, that what we're doing behind the scenes is just simply those same commands, but behind a GUI. Because that does give me comfort. And if I needed to, I can go into the shell inside of TrueNAS and do my own thing.

I'm sure if I wanted to. You may not like it or prefer it. Maybe you can't. I don't know. I don't have that much experience with TrueNAS, how much you allow or suggest folks to actually go into the shell. We do give you access to the shell. You have SSH, and it's there for a reason.

We tend to steer newbies away from it just so they don't foot shoot because, again, once you go into the shell, we're basically taking the guardrails down, right? That's right. So you could do something good. You could also do something very bad. So we recommend, look, if you're old enough to drive and you kind of know what you're doing, absolutely.

If you want to go double check it, this is the beautiful thing. It's open. There's no secret. So you can go drop to the shell and say, I'm going to run ZFS list. I'm going to check the snapshots. And it's really helpful if you think

you see a problem in Trinaz. You can go check it and investigate it. We can do it for debugging too. It's fantastic. But again, I would caution people, if you're not a ZFS person and you have data you care about, like maybe try it in that virtual environment first and kind of get your feet wet before you start running commands that, you know, chat GPT maybe told you, you don't quite know what they're doing under the hood. But by the same token, if you already grew up in that environment and know your way around, absolutely feel free, go take a look, see what it's doing behind the scenes.

Yeah. And, uh, you know, if you see something that we could do better, like we're all ears. What you're about to hear are real reactions from PagerDuty users in response to seeing signals from FireHydrant for the first time. PagerDuty, I don't want to say they're evil, but they're an evil that we've had to maintain. I know all of our engineering teams, as well as myself, are interested in getting this moving the correct direction, as right now, just managing and maintaining our user seats has become problematic.

That's really good, actually. This is a consistent problem for us and teams, is that covering these sorts of ad hoc timeframes is very difficult. You know, putting in like overrides and specific days and different new ships is is quite onerous oh and you did the most important piece which is didn't tie them together because that's half the problem with pager duty right is i get all these alerts and then i get an incident per alert and generally

speaking when you go sideways you get lots of alerts because lots of things are broken. But you only have one incident. Yeah, I'm super impressed with that because being able to assign to different teams is an issue for us. Because like the one the one alert fires for one team and then it seems like to have to bounce around and it never does. Which then means that we have tons of communication issues because like people aren't updated. No, I mean, to be open and honest, when can we switch? So you're probably tired of alerting tools that feel more like a headache than a solution, right? Well, signals from fire hydrant is the alerting and on call tool designed for humans, not systems. Signals puts teams at the center, giving you the ultimate control over rules, policies, and schedules.

No need to configure your services or do wonky workarounds. Ingest data seamlessly from any source using webhooks. And watch as Signals filters out the noise, alerting you only on what matters. Manage tasks like coverage requests and on-call notifications effortlessly within Slack. You can even acknowledge alerts right there. But here's the game changer. Signals natively integrates with Fire Hydrant's full incident management suite. So as soon as you're alerted, you can seamlessly kick off and manage your entire incident inside a single platform. Learn more or switch the day at firehydrant.com slash signals. Again, firehydrant.com slash signals. so the uh the one issue i've had and i'm surprised by it i do not expect you to solve it on this

conversation but maybe tell me if it's common is that i was following once again tom lawrence i was following his tutorial because i'm a tail scale fan. I love Tailscale and everything, every machine I have is on Tailscale. Now, my early issue with TrueNAS Scale was that I didn't know how to use Tailscale with it. I like to access my SMB shares from different locations via my Tailnet. And so that was my turnoff and why I kept running my Ubuntu CLI self-ran version for a while because I can run Tailscale in Ubuntu too easy. I didn't know how to do it with TrueNet Scale. And I think you have applications that you can install.

And you choose a pool. So when you set up apps, for anybody who's listening to this, I'm in the sidebar essentially. I choose the apps section and you can check available apps and you can search for any of them. And the tutorial basically is to install the Tailscale application, enter in your, you know, your auth key from Tailscale that you can set up, all that good stuff. And then boom, you've got an application running. I think as virtualization inside of Kubernetes, you can clarify all this, Chris, as I'm done here. But then the thing just never fully booted. It was like deploying.

It was like stuck at deploying. And so I could never fully deploy this application for Tailscale simply just to enable my box to have, so I can access my SMBs via my Tailnet. That's my long story short. That's where I'm at. Okay. So I'll give you a little background on it.

So yeah, it's using Kubernetes on scale. It's really running Linux containers, Docker images, if you will, under the hood. So no real mystery there. And I'm not a tail scale user myself, so I can't necessarily troubleshoot it. But I can tell you when it says it's stuck in the deploying phase,

what you need to do in the UI is it'll show you which container instance is running. You click the three dot menu and say, show me the logs for it. It means for some reason, the container is refusing to start all the way and get to a healthy state. Now it could be it's saying, oh, your API token's bad, or it was a space, who knows what the error message would be. It could be something that simple, or it could be like, oh, I'm starting up and I don't recognize this network config. Again, I'm not a tail'm starting up and i don't recognize this network config again i'm not a tail tail scale user so i don't know what their specific error messages are but that's the first place you'd go hunt it down and show me the logs to that container

let's see why it's not starting okay or starting fully and nine times out of ten it's going to be right there at the bottom of the log like i stopped because blah right it's probably obvious this is definitely now that i know this definitely something i got a trouble shot shoted better in the moment and you gotta understand the way we do the apps the reason you see that deploying is it tries to heal because sometimes apps i mean let's be honest it's software software sometimes has bugs they can crash they go yeah so kubernetes true now is they're going to try and be helpful and automatically try to restart, and that's why hence the deploying. It's like, okay, well, I'm going to try this again.

Maybe it's a transient. The network was down. Maybe, I mean, who knows what. And nine times out of ten, that works great. But if you're deploying something fresh for the first time and you can never get it to that online, fully running state, go check the log for the container. It'll give you the info that hopefully helps you troubleshoot

and figure out, like, oops, I fat-fingered putting the log for the container. It'll give you the info that hopefully helps you troubleshoot and figure out like, oops, I fat fingered putting the key in or something. And that might be it, honestly. I mean, it's probably the key now that I think about it. I should not have blamed TrueNAS. I should have blamed myself and done better with that. I didn't know you could take the logs on these apps. That's cool.

Yeah, if TrueNAS screwed up, we'll own that. If we screwed it up, file a ticket and we'll go take a look and see if the apps worked or something. But again, until we have that info, we just, you know,

can't go anywhere yet. Which I guess the next step there would be like the forums. You got amazing forums. I mean, I've already learned so much about ZFS just from your forums, not even TrueNAS related.

There's one person in particular. I think I emailed that person. I'm going to have to, he's the Grinch. Is his name the Grinch or something like that? I emailed that person. He's the Grinch? Is his name the Grinch or something like that? I recognize that name from the forums, yep. Does that person work for iXsystems, or are they just super passionate? No, we just got a lot of – I mean, some of the folks on the forums do work for iX,

and usually you'll see like an iX badge or something on their name, like myself and others. But we have a lot of really passionate users who do this as a hobby. Some of them do it for work too, for their day job. self and others. But we have a lot of really passionate users who do this as a hobby. Some of them do it for work too, for their day job. They also have TrueNAS at the office, just like they do at home. Honestly, that's how we get into a lot of businesses is people in Friday at home, sysadmins, and are like, this is amazing. This is great. I really should be running this at work. Why are we doing that other thing that we all hate to administrate when I can use the same thing I do at home and it'll do all the same functionality. But yeah, back to the forums, there's a lot of really good, helpful people there.

It had a reputation in the early days for being the Wild West, forums being what they are. But these days, if you haven't come back and checked it out, the forums have gotten quite nice. And there's a lot of good stuff coming there. Definitely lots of info there. And I've appreciated the depth. And I think there's times whenever you can have curmudgeons in forums with like snarky responses and like, you know, don't double, don't, you know, whatever. All the reasons why people are like. RTFM.

Yeah. Forums are not that great. I think that there's a somewhat tame nature to my, in my experience with the forums around TrueNAS, which I was quite happy with because I think I've seen some presence of iXsystems folks in there as well as the Grinch and others, if that's truly his name. I can't recall if that's his actual name or not, but let's assume that that moniker works in this moment. And if not, then I'll apologize later. Those people are very helpful, and they're also very passionate.

They'll help you learn how to set up pools, and if you're config like a mirrored scenario versus a you know raid z2 or z3 like you're considering different options they'll talk you through what your scenario might be and give you that feedback sometimes a little snark but that's okay that comes with it to some degree but you gotta figure if you're getting help for free you can put up a little bit of snark it's just we try and make sure it doesn't get out of control. Absolutely. Absolutely. And then I think my next one I'll mention to you, and you might be aware of this one, or it might be a me issue.

Hit me. Hit me. Okay, so I've got my box right now connected to the network via an Ethernet port. It's got dual Ethernet ports on it, gigabit ethernet. But I also have a dual SFP plus NIC on my box and I've got the connections. I've got the aggregation already set up. I'm running Unify. I've done that before.

I did it as the Ubuntu box as it was beforehand. But whenever I try to set up that, like I add a new interface that is a bonded interface, all those, you know, all the things, I guess there's some sort of UI that after you do that, it sort of puts it in a temporary state of like X seconds.

Yeah. It never lets me make that connection finally. I've tested with ping that it goes to the static IP all that good stuff like the ethernet port doesn't go away and my new dual aggregate bonded SFP plus

static IP address via link aggregation does not come online. But I have tested that it will come on temporarily with ping but it won't let me move to it. Is it coming online too late for you to make it in time? Because I think it's like a 30 second timeout or something. Well, that's the funny thing is like,

I don't get that UI whatsoever to confirm it. It's like keep testing changes. Oh. Like it's got two buttons, test changes and like stop and like, and I forget what they actually are without doing it, but there's never the one that says, yes, keep configuration. Like that button is not there. It's supposed to pop up when you say test changes they're supposed to be a modal window that pops up that says confirm and click okay and that's to present

prevent you from locking yourself out of your box where you make a change and now you can't reach your nas and you're stuck going to the console um you're on the latest kobia 2310 too yes i had actually had just updated before I did this. I was like, you know what? Maybe there's some new features that just make it easier. I don't know. Maybe that was wrong. No, no, no.

I don't have an open ticket I know of around this, but do me a favor. Let's get me a bug ticket with a debug file, and we'll go take a look and see if we can identify why it's doing that in your case. Okay. I'm not aware of others still hitting this issue. I think back in the early days, we did have a handful issues there, but those should all be resolved now. So if there's some other case we haven't encountered yet, like send it over, we're going to go get it addressed. Right. It's good timing because we're going into the RCs and final release for didn't again i didn't spend a ton of time trying to figure this out i just kind of hit a couple walls tried a couple more times and i was like okay i'll figure it out later it's not that important to like suddenly be on a i mean i would love to have a bonded aggregate you know 10 gigabit connection to my network but hey it's not that big of a deal for now eventually it's better you know it's not

the end of the world in the moment they had said something like you couldn't have an ethernet port be active while you also add another aggregate so that could be something where like maybe i didn't deactivate the you know existing connection you know so maybe that was it were they all intended to be on the same subnet is that why exactly yes that might Yeah, we don't want you to have multiple interfaces on the same subnet at the same time. That actually could be the reason you're prevented from moving forward. That's intentional now. Okay, I can already hear people screaming, complaining, Linux allows it. Linux allows you to do a lot of things, and sometimes things that are very bad from a network administrator standpoint. So having multiple interfaces on the same subnet, yes, Linux will allow it,

but we try and do everything we can to prevent you from foot shooting and having to troubleshoot your network later and find out why things aren't behaving properly. Which route should it be taking? I ping out of one interface, it's coming in another, and everything's getting confused, and now my services don't behave the way I expect. So that is still a limitation we have in

the software, which we do try and keep you more to a little bit saner network config. So yeah, in your case, I would say remove the original subnet from the other ethernet device and then see if you can apply it again without actually seeing it and sitting down and troubleshooting it with the team. That is something I will throw out there. So again, people have different opinions on that. And if you ask on the forums, you'll probably will get flamed, like, ah, do proper networking kind of thing. But it's- Well, I think this comes to the UI,

which I do want to get to, because that's kind of where I want to, less to troubleshoot further, but more to clarify. So I have, you know, an ENO01 or EN01, I believe, is the Ethernet port there for it. It is unclear what I should do to add a new link aggregate

interface because while I know I have the existing one, it's scary to reset it because even when I push reset to maybe remove it, it says this change can interrupt connectivity and must be tested before making permanent. I'm like, of course, I don't want to lose the one I have. So I'm thinking like if I'm adding one that is on the same subnet, part of your UI should be like, hey, you've got one active already. Let's establish a new one you're trying to add and kill the old one. Like the flow should do that, in my opinion. That's where I think we could probably improve, which is why I'd really love to get a bug ticket and just confirm that's what we're hitting here

because that is something where we could probably make it a little more user-friendly to say, hey, we detected you're trying to set this up with the same subnet here. We should clean up the old one for you at the same time. Do you want to proceed? Right. And that's not happening.

And that could be a me issue. It could be a you issue. I don't know. We'll figure it out, though. Again, get us the debug. That has all the gory details, and my engineers can rip that apart and say, okay, now we see what he's got set up. I'd be happy to do so, I'm sure.

Right? Like, yes, more trouble. Let's fix it. So let's move on now then to the interface. What I love most about TrueNAS is really this interface. Has it always say this good like it is barring the couple issues I've mentioned it's really easy coming from a you know CLI ZFS you know junkie so to speak or a cowboy so to speak now it's like well when I'm at my dashboard I have network I've got memory I've got my CPU I've got all of my

polls available you know when the last time the the scrub ran, how many disks. Everything is at my purview. The path it's on, available disk space. Such a beautiful UI. Has it always been this nice? Well, it's not always been that nice. We've always had a good UI. That's always been a key piece of the TrueNavs experience is the fact that you want a UI to do things

so you're not shell scripting and doing it all from the command line all day. But, gosh, it's been a good seven years now where we switched off a django and went to angular for the ui and set up proper api ui rest all that good stuff and so ever since we did the angular ui it's gotten better if you've seen the core ui that one's a little bit more um organic growth i guess if you look at it you can kind of see the evolution of FreeNAS and TrueNAS where we added things and new pages got added. And so when we did scale, when we were first kicking off the project, we kind of went back to the drawing board and said,

okay, we've been doing this now for 10 plus years, just kind of throwing things in as we add new features. New Cloud Sync, new this, new that. Can we design a better UI that helps unify a lot of these things into less pages? Because there was a bit of page fatigue prior on the core side, which you can still see today if you go look at it. There's just a lot of pages to choose from. So instead, we consolidated things on scale. So instead, you'll see things like a data protection page. And when you go there, you got widgets that show you, here's my cloud

sync tasks. Here's my ZFS replication tasks. Here's my snapshots. Here's all these things that are related to data protection, right? Or for example, the networking page has all the networking things on it versus 10 different pages of different items. I'm trying to think, same as storage data sets. We've done a lot of work to make ZFS management kind of painless through the UI because ZFS is really stinking powerful and there's a lot you can do with it. And so the challenge and the art is how you take that and then make it something somebody can click through and it's pretty and gives you all the feedback you need. So it's come a long way. I'd say the last year and a half, two years,

it's really come into its own and it's shining the way we kind of envisioned back in the days when we even started the Angular UI six, seven years ago. So pretty happy with it. Not to say we're complete. We always have a lot more things we want to do. I'll throw a little sneak tidbit out for your listeners. So in the fall release, we're looking on adding some more dashboard widgets of different sizes so you can customize it a little more fully and add GPU feedback to the dashboard too. So if you have an NVIDIA or an AMD discrete GPU in the system, you might be able to get, you know, your temperature and utilization and all that good stuff. So we're going to keep improving it.

And I think there's still, you know, each page I look at, I'm like, ah, I'd love to fix that. But, you know, all the big things that we wanted to do, I think are starting to come together. And people see that when they install scale for the first time. Yeah. You'd mentioned you're a senior vice president of engineering. So I would assume that the interface for the thing you manage, and I guess senior vice president, is under your purview. Is the UI something that you pay personal close attention to,

or what's your personal take on it? I'm not only a senior vice president, I'm a heavy TrueNAS user myself. So absolutely, I pay attention to it. I run my own rigs at my house. I have my own rack in my office at home. My personal TrueNASes are two I've built because I also like to assemble hardware and I've been that home labber just like you are. So that's where my roots are. So yeah, absolutely, I pay attention to it. We have a UI team and they handle the designs and the implementation side of the UI, but it doesn't mean I don't go in and give my feedback. I think maybe they cringe when they see me coming because they know he's got

something for me. Oh my gosh. He doesn't come around too often to yell at us, but now he does. Well, not yet. Give us stuff to do. I absolutely love to do that though. Cause a lot of times I'll go home and on the weekends, sure. I'm fiddling with my own Trunaz system. I'm installing some new apps. I just set up a home assistant on it and got that all dialed in. And along the way, I was like, hey, there's some things we could do better

in the apps UI. We could do this. And so I go make my Jira tickets and say, here's the things we should address on the next version. And eventually it ends up on a roadmap. I guess that's one thing I get to do with the title, right? Is I can go and drop things in the roadmap like that. So I do take advantage of it from time to time. Well, let's talk about then, since we're talking about that, let's talk about leading the teams of engineers that, you know, at the most deepest levels at the Linux levels with ZFS and you mentioned before keeping I believe you mentioned core and scales

ZFS, I don't know what you call them, you know, in sync or, you know, making... Feature compatibility, all that. Compatible, yeah. Because that's a big deal too. Like, I don't know, I have moved ZFS pools from systems to a virtualized TrueNAS system inside of Proxmox a while back and had some issues because at the ZFS level, you've got different versions and there's import compatibility,

there's flags that are features that may or may not be in your distro of Linuxes or BSDs available to you. So there's things that can happen. Like ZFS can move around, but you might lose features and you might even have pool issues. So what do you all have to do to sort of manage the core to be stable and work with all Linuxes and stuff like that?

I guess Debian is not all Linux. Let me give folks a little primer on the ZFS side. So ZFS, of course, it's a file system, but it does have versioning in it. And so a while back, they added feature flags as ZFS. So when a new ZFS 2.2 rolls out, it may have a handful of new features. You know, block cloning is a good example. That's where when you make a copy of a file, it's actually

not copying all the blocks. It's just doing a pointer. So you save space, you get like inline do you do basically. So that's the type of thing that's a feature that's turned on when you upgrade to a newer ZFS version, right? And it sets flags on the pool to say this pool is using these features. Well, of course, that's hard to roll back. If you go back to an older version of a Linux or something that doesn't have that modern ZFS, it checks those feature flags and says, oh, I'm halting. I don't recognize this flag. And this means I can't read the data properly. I don't understand it. That's intentional. That's how it's always going to be. It has to be that way. It's the only way you can get new features into a file system like ZFS.

So you have to take a little bit of care when you decide to roll forward. So one of the nice things we've done in TrueNAS is you do have a chance to try before you buy. So if you really. Yeah. So I don't know how many folks know this. I didn't get that chance. I don't think it wasn't clear to me. I just was like, upgrade. It depends on where you're coming from and if you're upgrading. So say you're upgrading from a version of scale that's a couple years old, and now there's a new, you know, ZFS will say 2.1, and now you're coming to the new Cobia and it's got 2.2.

We don't automatically upgrade your pool for you. That's very intentional. And you might say, well, you should do that for me. No, we give you the chance to make that choice to hit the upgrade button of the pool later because maybe you decide you have to roll back. Oh oh the driver i was using doesn't work as well here there's a bug or some reason i need to roll back to the old or i need to maintain compatibility so i can export a pool and import it in another system whatever the reason is we give you the option to go click the button to say i'm now ready to commit and upgrade to the newer version

of zfs and turn on the fancy new features, right? So that's something that shows up in the disk page. When you go in, you'll probably see an alert too that pops up after a while that says your ZFS version is running an older version. Click here and you're ready to upgrade kind of thing. So that's kind of the story there. So again, if you know you're going to be using

pool interoperability between different systems, just pay attention to that. Just be aware that if I have older systems, just pay attention to that. Just be aware that if I have older systems, maybe I don't want to upgrade this one to the newer version. Same thing if you're replicating, you could end up in a situation where potentially a replication stream needs a newer version of the pool on the receiving side. So it's just, again, something to be aware of as you go to your updates and say, okay, the way I do it personally, so I have two ZFS or two TrueNas at my house and i replicate

between them of course one's the primary one's the backup and so i often will upgrade the backup system first make sure everything's stable there because the newer backup system should always be able to receive the older zfs stream no problem once i'm content everything's good on the backup system then i go upgrade the primary and then once i'm good everything's running okay then i click the upgrade pool button and life moves on that That's a good pattern. I like that. That's my personal strategy. People usually will use some variation of that. And then on top of it, I'll back up to offsite as well. Usually some sort of S3 bucket somewhere. Gotcha. So I did say I didn't get the option to try before I buy. I guess it's tried before you buy by default.

Yes. By default. Right. Cause this is like, you get no, no option to try before i buy i guess it's tried before you buy by default yes by default right because this is like you get no no option to not try by you know before you buy now again the caveat i should have mentioned this is if you're installing fresh it is going to use the current version of zfs so it's not like when you go create a pool for the first time it's going to create it with the latest version whatever that is so if you for example if you're making a new fresh trunas and you know you need to support an older pool format you want to install an older latest version, whatever that is. So if you, for example, if you're making a new and fresh TrueNAS and you know you need to support an older pool format, you want to install an older TrueNAS for the initial pool creation to create it with that older format before you upgrade forward

into the new version that has the new pool features, if that makes sense. In my case, I was migrating from, again, that Ubuntu self-built box or system, I suppose is a better word to use and so I was moving pools from a non-Trunas system got to try before I buy because you know it was like hey yeah let's recognize this as a ZFS pool here you go it's there by the way you can upgrade this and like if you do there's no going back kind of thing like that was the what the paraphrase of what the dialog box said and i'm like i will upgrade in a minute or two let me give me a give me a chance to think about that but we're just fine no we want you to think about it it's it's designed to make you stop and think and go am i ready to do this because part of it too is

you can't use the rolled back boot environments because again if you're going back to older software you need to be know that those are not going to be accessible once you upgrade the pool to a newer version. So you did ask about it in context of core and scale though too. So what we do try and do is keep the two versions running as similar as the EFS version. The version we have in Cobia and the Dragonfish that's coming out next month is based on 2.2. And then we're getting ready to go into the release candidate and release phase of 13.3 of core, which will have the same 2.2 version, OpenCFS. So again, we'll have parity between those two to allow people to move back and forth as needed,

replicate as needed, et cetera. I guess it's important too when you consider the in-quotes maintenance mode that you mentioned earlier for core, maintaining it, you have to maintain, I would imagine that you have mixed environments. Some folks that choose core and they're trying to modernize, in quotes, modernize onto scale, that you want to keep those pools interoperable. Absolutely. You're not locking somebody out of like, okay, well, just because you've chosen to

have a more legacy system like core, that's still amazing, still stable, still maintained, but, well, you're not locked from moving data around with replication. Absolutely. It's still pretty common where we see somebody, again, in my scenario, two Trinases. A lot of times people, one of them is the older core version, and that's the backup system that's receiving the backups.

Fine. And then I'm using scale so I can get the shiny new apps feature or some functionality that I want to use on the primary. Completely valid. Some people do that, but we do our best to make those kinds of scenarios possible. So people have ultimate flexibility. How do you think about at your level? Because you're pretty high up with iXsystems and managing TrueNAS. How do you feel about, I guess, competition? How does how does IxSystems make money?

What is your business model? Why in the world do you give away this amazing software for free as part of the open source movement that we all love and are here for? But how does it make sense and how do you compare to competition? What do you consider competition

to TrueNAS or iXSystem itself? Sure. So I'll try and avoid mentioning competitors' names. Mention their names. Don't hide. You know, but. Put them out there. But no, more seriously, like a lot of competition wishes they have what we have in the sense of an open source community with that many users.

It's phenomenal. We like to make the claim we're the largest storage company in the world because in one sense we truly are which is by the sheer number of deployments of people out there running trunaz in the wild which is amazing that does a lot for us as a company and that really in many ways is the secret sauce right it's like you know we put our money where our mouth is you don't got to spend hundreds of thousands of dollars to get access to a product you can literally go download it for free try it see if it works it works. Does it do what I want? Fall in love with it. And then you come to IX and say, now I'm ready for the enterprise grownup

version of it because I got to go to work and I have responsibilities and we need SLAs and I need all the support and all that good stuff. So I guess I'll talk about the business side of a little bit. So IX systems, of course, we're the makers of Trunaz. We do all the primary development, all the testing, all that good stuff on the software side. And then on the enterprise side of the business, we offer Trunaz as an appliance. And that competes with more of your traditional vendors out there, right? You can think of who they are out in the wild. But we're the kind of young spry guys who do the really cool stuff and offer a lot of neat functionality that's just all inclusive. It's there. We're not nickel and diming you for license fees and all that good stuff.

But we take that software, we marry it to different hardware appliance platforms. The key thing is we offer both single controller variants. So if you're not in a high availability need situation, you can do that. Or we have the HA platforms, which everything's fully redundant. So one chassis, two discrete controllers in there. They all have redundant access to the storage and we can do failovers between them. So if you do have, you know, catastrophic system failure or something, it can fail over to the other controller and you're back in business within seconds, right?

Because it's all accessing the same ZFS pool and it makes upgrades super simple because you can fail over between them. It's quite nice. So we do that on the enterprise side and that's very popular. Our customers really love that. And then we do things a little nicer like the proactive support as well,

which means we're getting alerts and notices if a drive's starting to behave flaky or if we detect some other air condition on the box, we can reach out to you and let you know, hey, we've detected something on your rig here. We're going to send you a new drive or we need to schedule a call to go dive into this deeper and see what's going on. So really popular on that side. One thing I will speak about a little bit is I don't think a lot of people realize how much work goes into doing an enterprise product like that. So we come out of the home lab space,

right? We're used to going on Amazon or Newegg and buying our parts and putting together our system and kind of do it yourself. And that works to a degree of works, right? But to go to the real next level of enterprise level functionality and stability is huge. Now we're talking, we spend a lot of time working with vendors, firmwares, trying to make everything as reliable as possible to get all the nines on uptime

we can on every single platform we sell and support. You know, that's not something you get if you home lab it. A lot of times for home lab, it's fine, it's good enough. But for an enterprise that says, man, I'm running critical

infrastructure off this, I cannot accept any kind of downtime. like there's a lot of extra work that goes into trunas to polish it to that level to make sure it's just rock solid stable for the most critical of environments and the beautiful thing is the open source community benefits a lot from that too because a lot of those fixes end up in the open source side of course and then a lot of it is on firmware and vendors and all that good stuff to make sure that everything's compatible and it's just hunky-dory and hot plug always works and yes enclosure management always works and you get the nice visuals and you can tell the remote hands in the data center which drive to pull by just looking at a graph and saying okay it's a second down third to the right go pull that one that's flashing you know that kind of thing wow yeah a the right. Go pull that one that's flashing. You know, that kind of thing. Wow.

Yeah. A lot of work goes into that. Don't pull the wrong drive. That's really the difference. But another thing is, you know, because we're open source, we're able to work with our community to make sure the software is battle tested and we can be led by our customers a little better. I don't have to sit here as the senior vice president and guess where the market's going and make bets on that and try and hope I get it right. I got a community of people constantly giving me feedback. I mean, that's what this interview is. You're giving me feedback on how you use TrueNAS. I can hear from people directly in the field, in the trenches, working on it at home, at work. This is how we

use the software. This is how we use the hardware. These are the kind of applications we run or the kind of services that we run. Here's what we wish work different or work better or the new thing that we're interested in. Like we get that feedback in a heartbeat. It's quite amazing. And it's daunting. Don't get me wrong. I've hired some folks who've come in from open closed source competitors. And it is a little daunting to come into kind of an open source company that lives by that open source eth that lives by that open source ethos and they come in go oh man all my work is visible everyone can see what i'm doing and i'm like no no embrace it that's a good thing we have no secrets we have nothing to hide i don't

have to go and tell customers trust me i can say well you can trust me and you can go look at the code and see exactly what we're doing that's i I love being in that position. I'd rather be in that position than not. What's up, friends? This episode is brought to you by ImageProxy. ImageProxy is open source and optimizes images for the web on the fly. It uses the world's fastest image processing library under the hood, LibVeeps. It makes websites and apps blazing fast while saving storage and SaaS costs. And I'm joined by two guests today, Sergei Alexandrovich, author, founder, and CTO of ImageProxy,

and Patrick Byrne, VP of Engineering at Dribbble, where they use ImageProxy to power the massive amounts of images from all of Dribbble.com. Sergei, tell me about ImageProxy. Why should teams use this? Everyone needs to process their images. You can't just take an image and just send it to users' browsers because usually it's megabytes of data. And if you have lots of images like Dribbble does,

you need to compress them. You need to optimize your images and you need them in the best quality you can provide. That's where ImageProxy shines. Very cool. So Patrick, tell me how Dribbble is using ImageProxy. Being a design portfolio site, we deal really heavy in all sorts of images

from a lot of different sizes, levels of complexity. And when we serve those images to the users, those really have to match exactly what the designer intended. And the visitors need to receive those images in an appropriate file size and dimensions, depending on whatever their internet connection speed is or their device size is. And that was just a constant struggle really to really thread that needle throughout the course of the platform, using a handful of different tools in maintaining that right balance of a high degree of fidelity, high degree of quality without sacrificing the visitor's experience. And when we were exploring using image proxy, we were able to spin up using the open source version of the product, a small ECS cluster to just throw a few of those really tough cases

that went through our support backlog, looking at some of the cases people were reporting. And almost to a T, we aced every one of those cases. Wow. So it seems like imageProxy was really impressive to you. Yeah, Adam, I just have nothing but positive things to say about using ImageProxy. The documentation is crystal clear. Out of the box, it does everything you need it to. Tweaking it to meet your specific needs is incredibly easy. It's wicked fast. It deploys real simple. And our compute costs have gone down over the open source tools that we've used in the past. Even including the ImageProxy Pro license,

it still costs us less to use and gives our visitors a better experience. So as an engineer, I like using it. And as an engineering manager, I like what it does to our bottom line. So ImageProxy is open source and you can use it today, but there's also a pro version with advanced features. Check out the pro version or the open source version at imageproxy.net. The one thing I love so much about this is that no matter which you choose, the open source route or the advanced features in the pro version, you use the same way to deploy it. A Docker image, one that is from Docker Hub that everybody can use, it's open source, or one that's licensed differently for those advanced features.

That's the same delivery mechanism via Docker Hub. It's so cool. Once again, imageproxy.net. Check out the demo while you're there and check out the advanced features for the pro version. Once again, imageProxy.net. How does that dovetail into making money? So how do you, you're talking about the business of it but what the model is i imagine that from the outside it seems like support slash sla type things on the software side where you have a was it a trunas enterprise is that is there another flavor of trunas out there trunas enterprise product so we have things like our M series, which, again, dual controllers. And those guys can scale up to 20 betabytes plus on a single system.

And we're talking beefy systems. When you say controller, what is a controller when you say that? A controller is just a computer inside a chassis. So you've got two controllers, two discrete computers. Again, 100% redundancy. So you can lose an entire computer with its power supply, with its network, and it keeps on trucking. It fails over the other. The storage automatically goes to the other system within seconds.

Any shelves that are connected with extra drives go over to the second controller, and you keep on moving. So instead of an outage now, you get a blip. It's like, oh, storage went away. Oh, it's back. It was like even though I just lost a stick of RAM or something, right? What kind of installs require that kind of like, at what level are people operating like that? Can you give me some examples?

Sure. So a lot of high-end enterprises. We could talk about media and entertainment where they're doing a lot of video editing. We could talk about hospitals. We could talk about universities with students on 24-7. We could talk about finance and banking. I mean, we play in a lot of verticals as far as the storage product goes. Another one, virtualization. Of course, I should mention that

a lot of people are running it as the backing for their VMware and you need that high availability because of the VMs you're running on top of, which might be part of your critical infrastructure. Right. Like my Tailscale app. So important. Absolutely. Very important for Tailscale. We got to get that right. Okay. That makes sense then. But yeah, yeah. So it's, it's a combination of the appliances and that kind of rock solid reliability. It's the support, it's the white glove support you get in that sense that, you know, we were the makers of the software and the hardware and it's one unified platform and environment. Some of the troubles, some of the competition has when your software defined only is you don't control the hardware. So now you're dealing with all the,

oh, I just threw it on this Dell and it doesn't work right. And the driver's funky and the HBA doesn't respond right. And the hot plug didn't work because the firmware doesn't match. It's a lot of things when you go to that level of enterprise that you have to work out in advance. You can't make your customers bleed on it. So y'all are making your own hardware, like from the ground up, you, you, you know, do you work with that team at all? Yeah. Yeah. So, I mean, we're not making our own motherboards and stuff. We take a lot of the best in commodity and we assemble it, we put it together. We do a lot of the chassis design.

We do a lot of the stuff in between that makes all that work. Yeah. Yeah. It's I have those teams under me. That's the, our platform team, which those guys are amazing. So shout out to the platform guys, if you're watching this, but we have a, I'm actually here. I don't know if I mentioned, so I'm sitting over in the Knoxville, Tennessee area. It's Maryville. It's a little suburb of Knoxville. This is where our IX innovation center is. That's the building I'm in here. We have a lot of our software and hardware engineers. And, uh, down in the basement, we have a nice R and D lab, you know, clean room, static floor tiles, all that good stuff, right? It's very nice. And that's where we do our product design and prototypes. And so we

got all kinds of good stuff on the bench and we're testing thermals, testing voltages, checking sound levels, you know, designing new products. That'll be next year's offerings. You know, a lot of neat stuff happens down in that lab. And it's almost like a kid in a candy shop walking down there sometimes. It's like, oh, I want one of those. I want one of those. That looks neat. I don't know what you're working on there.

Tell me about that. Sure. And then, yeah, it's fun. We give tours of it occasionally to customers and vendors that come in. And it's a lot of fun. I would love a tour. Neat place. That'd be awesome.

If you're ever in the area, let me know. Okay. So, assembling, kind of making your own hardware, which is a big deal, I think.

I mean, is that a big business for you? Yeah, that's what pays the bills. Can you, not so much sharing revenue numbers, but can you share ratio? Can you say,

well, our hardware business accounts for 60% of our revenue or like, how does it slice? Well, what I can say is the TrueNez Enterprise business accounts for our revenue of our revenue? Or like, how does it slice? Well, what I can say is the TrueNAS Enterprise business accounts for our revenue.

That is what pays the bills and keeps us funding TrueNAS, the software, the things that you guys run and download for free. So absolutely, that's our business model is we put it out there. We have people fall in love with it. They enjoy it. And eventually they say,

wow, that system I built at home, guess what boss at work, you should buy one of these because it's half the price of the competition you were just looking at. And not only that, I know it and I love it and it's amazing. And they have amazing forums. And we have amazing forums. Shout out to the forums. Do you get a different access? I suppose you get support. Do you even need forums at that point when you're an enterprise customer? No, we wouldn't make our enterprise customers go to the forums. They call up, they talk to our support team, we handle everything. That's part of the experience. The support actually sits in this building too, a huge

chunk of them. So when you call up, odds are you might be talking to somebody here in Maryville. And yeah, support team's amazing. They have access to developers just a couple floors up and we talk and make sure things are working for our customers. But you really do get that superior support experience working with IX because again, we own it all. Hardware, software, all the features, we can service it all very well. So platforms under you, engineering's under you. Would you consider UI engineering? Is that a different sliver? Maybe it helps if I can just lay it out. So yeah, it's not just all me. So I'm not taking all the credit here. I got a team of a VP.

I got a director. I got managers and supervisors of various groups. But we have software engineering groups that's under a vice president. And that's a couple teams, you know, backend, middleware, which is a lot of the Python and brains of TrueNAS, which is where a lot of the magic happens. We have a UI team, which, of course, is the pretty UI you've been talking which is where a lot of the magic happens. We have a UI team, which of course is the pretty UI you've been talking about. Those guys work with the middleware. And then we have a backend OS team, which does ZFS things and Linux things. And we have to

control all aspects of the stack on the software side. This is something else people don't realize when they talk about, oh, you have an engineering team. A lot of people think an engineering product is just a UI sometimes, which in a lot of cases, that's all it is. But no, no, we're talking down to the kernel, the file system, all the way up to the middle layers, up to the top, to people running it on their laptop and accessing a web interface. So I have those folks. We have a platform team, again, the hardware guys who then build the cool products, support the cool products, and then marry that to the software. We have SQA and automation teams. So that's testing, manual testing, automation testing. So every time a beta or RC or new release drops, we go through sprints of testing cycles here just

to make sure we did a good job as engineers and make sure we're putting out good quality code, even for our free community users. We want it to be great for everyone. And then we have a performance team. So we're always interested in what kind of performance we're getting, what's possible, what can we do better. Those guys spend a lot of time talking to the OS and file system guys

because that's where a lot of times we're tweaking and tuning things to maximize performance, squeeze ZFS every bit of it we can out. And then we have, let's see, so I did the performance team. We have documentation. So you mentioned forums a lot, but I don't know I did the performance team. We have documentation. So you mentioned forums a lot, but I don't know if you've seen our handbooks and guides.

Those are all open source. Those are awesome as well. Those are great. And those are all open source too. We even open source the guides. So people can, community people contribute those occasionally as well. So that's all open. And we have a docs team which does all the cool artwork and photography for the enterprise

products, but also make sure the guides on how the software works and all that's kept up to date. And then I guess with another hat on, I have the IT kind of DevOps-y kind of groups as well, which is infrastructure and hosting the websites, making sure the update servers are up and all the good things you need to do to run a product like this. Wow. So it's a busy group. And like I said, I can't take credit for it all. I got a great team underneath me. So by no means is this a crystal on item rise. So what do you find most joyful in your, in your work? Like, where do you get, where are you at your happy place? Well, for me, it's, I get to work on something that I use and love and enjoy. And there's a lot of people who don't

have that luxury. So I consider myself very fortunate in that regard. Not only is it, I get to come to work, I get to see other people and I get to work with amazingly talented people, engineers, and all the folks on our teams who make an amazing product that I get to go home and geek out on, on the weekend and play with outside of work even, and do personal things on and just love it. That is special. And again, I wish everyone had that opportunity. I know a lot of folks don't, it's just a job. And we run into those folks where it's like, yeah, I do storage at other big name company. It's just a job. I can't afford the gear. I don't have access to it. They don't have a free version. I can't afford to spend a hundred thousand dollars to buy that system to put in my home rack.

But what I love is the enthusiasm you get from running a team of engineers and Ixians, what we call ourselves internally, Ixians who also are enthusiasts of the product. If I could draw an analogy, if I was a car company, it's like a car company for people who are just car aficionados who love it and they go driving on the weekend. And like that's their hobby in addition to what they do for work. So I'll let you in on a little secret. When I'm hiring people or interviewing and talking, that's what I'm looking for. Is people who have that passion like I do or that drive who want to build something cool that they would want to use and will use and are using. Those are the kind of people we attract here at IX. And again, it's infectious. It's a great atmosphere and environment where people just, the energy builds.

You know how when you get enough people in the room who are all excited about the same thing, good stuff just kind of happens. That's the environment I get to come and work at every day. So it is really a pleasure. Yeah. It's only got a fun job. I mean, I'm a fan. I could probably enjoy doing what you do

i mean that seems pretty fly next time you see it opening there you go i got a job i got a job i'm doing my job now chris you know oh i see how it is you know i got a job this is my job hey there you go there you go but at the same time you know we make great partners adjacent you know the brand change all media we make great partners adjacent you know the brand change all media we make great partners in other ways too so definitely opportunity for us to find ways to work together and i'm already telling people about trunas you know so you are now an honorary ixian either way sweet yeah yeah i dig it i was you know going back i guess one layer whenever i was moving to the trunas system i i was happy that I could try before I could buy because I didn't have to upgrade those pools. And I was thinking, well, I think what I'm

doing here will work out. And so then I had enough confidence to actually import the pool fully and upgrade it. But I was like, you know what? This TrueNAS thing is really sweet. I was confident in a couple of days just because I had had that history of CLI management, which was just fine. I'm really happy for those days. And then part of me was like, man, now I can't type in zpool status dash v and be happy or run a scrub manually. I'm giving up part of my hacker mentality, my hackerism, so to speak, in a way. Do you feel that at all? Like go into a GUI?

No, no, no, not at all. Don't get me wrong. I love the GUI and I use the GUI and that's what I enjoy, especially because I get lazy on the weekends. I'd rather go play with whatever the thing I want to do is as opposed to having to script everything. But the nice thing about TrueNAS being what it is in open sources, we don't restrict that access.

You go turn on SSH and you're right in there and you can run your Zpool list and see everything. That's true. I personally actually have a handful of scripts that I keep on my storage pool that what I do is I'm a prolific shell scripter, right? So I'll go shell script something up that does a thing on the TrueNAS and I'll go into the UI and set it as a job to run either boot up or periodically or whatever the script is. Right. Again, this is for advanced users and this is the beauty of open source. You can kind of do what you want with it. So I wouldn't recommend it for everyone if you don't know what you're doing, but you know, if you're like us and you have that experience and you need to scratch that itch and go do something a little off the beaten path, go for it. I will caveat if you break it, I'm not going to be able to help you so

much you know you kind of went off the rails on your own but you know the nice thing is you have that flexibility because i mean even uh true now is the dragonfish version we have a command you can run to install the dev tools so if you want to go hack on the system and again open source we want contributions if people have cool things like send send them in, we've enabled that. So it's almost like an SDK. You can run one command and it turns on apt and all the Debian commands. So you can go and do development work, right? Which is fantastic. We, that's how we basically develop it here. Right. And we've enabled that to our community to do. I mean, not a lot of commercial closed source products. Do you have that kind of flexibility? You just don't, but that's, again, some more value add we can do with TrueNAS, which is, oh, you want to add a feature?

You need to do something a little custom because you got a really strange workflow? Fine. Do it. Like I said, just don't come complaining to me if you break it. That's on you at that point. For sure. Well, you've mentioned Homelab a couple a couple times and i have as well home lab is a is a big deal you've mentioned enterprises where you make your money but i gotta imagine

there's some money in the home lab space like have what what is the state of hardware for home labbers are you guys looking at that market more closely what's happening there so we do address some of that we have our mini line of true nazASes. We have our minis, which are like little desktop units that are quiet and efficient. You can do four drives, five drives, eight, whatever. And then we also just this last year unveiled the Mini R, which is a rack mountable mini. So great little system. You can get those if you're a home labber. Again, some home labbers want to go build their own, which I get.

That's part of the fun, right? It's the hobby thing. But if you're like, yeah, I'm a home labber, but I don't want to go do my own hardware and I want something that IX supports and has all the testing and vetting, or I want like, say, the enclosure management bit, because that's a piece that's really hard to do when you build your own hardware. We can't just map things for you automatically. That's a perfectly valid option.

And so, yeah, we offer those. They're up on Amazon. They're up on the company website and store. And we see people grab those. And that's a good gateway device, especially even larger enterprises. Maybe you'll try that to just see how it performs in their space and, you know, kick the tires on an official platform before they go and scale up. I just wonder if there's like a middle ground there, like for the folks who are home labbers, but kind of like enterprise home labbers because they want to build their own thing. And you might say DIY and in some cases DIY seems like a pejorative, but it's not really

because you're a DIY yourself as well. I would potentially spend several grand on a new machine if I can like buy maybe a chassis from you all and blessed components from you all and be able to put my own CPU in there and obviously my own RAM. Like, is there a world where iXsystems will eventually support that kind of natured home library? Because I can imagine there's a big market for you out there

considering how ubiquitous you are in the open source. Absolutely. So I will say we don't have that world here today, but there's no reason I couldn't run that through our product team and see if there's a world tomorrow where we could do something like that. No promises, but. I think you'd be surprised.

I mean, there's lots of, I mean, I've done so much research. There's obviously no end to different pieces of hardware you can get out there. And then where you can get them, you know, generally is like Amazon, new egg, CDW. I mean, you can be a home user as a CDW customer as well. Like I've bought drives and ran from them and whatnot. And it's sort of this home lab world is sort of like this,

this middle ground between not quite enterprise, but they have enterprise tinkering desires. Yes. Yes. You know what I mean? And so I don't know how to frame that as a business standpoint but they're not uh just buying an appliance and move along they're like can i buy good chassis can i buy good components from a worthy source not just random places we sit

kind of halfway we see some folks do that today it was like the mini r they may buy that put in more ram you know put a gpu in there example, because Plex is an extremely popular app people run on TrueNAS. They want their transcoding. So, again, we're doing a little bit of that on the periphery, but not quite to the extent you're describing. But, hey, something we'll consider. I'd love it if you would run that up the ladder or, I guess, down the ladder potentially. Sideways. I'll go to the product team.

Yeah. Yeah, exactly. Sideways. I'll go to the product team. Yeah. Yeah, exactly. Sideways then. And just see because one, I would probably be a customer. And two, I could imagine that there would be more opportunities out there. Now, I guess it depends if you can actually make money from it. Does it make sense for you to focus on it? Yeah.

Yeah. Is it financially viable and all that? For sure. Part of the issue is if you sell something like that, can you support it? And then you lose money on the support and all that good stuff. So, you know, you got to do that math and make sure it makes sense for a business. Yeah, absolutely. But in the meantime, we'll keep providing the same high quality, awesome TrueNet software that we always do and, you know, do our best to try and support it, you know, on other hardware configurations. You know, we're not, we're not unsympathetic to it,

but at the end of the day, we don't control every variety of every motherboard and NIC and all those things out there. So it is a challenge. Well, when it comes to storage in particular, it's about trust. I'm down to DIY. I'm down to Tinker and HomeBuild or however you want to frame it. But at the same time, it's storage, and I want to trust the components I'm getting. There's some things you can get from Amazon and be like, totally cool, whatever, right? Like a heat sink for an NVMe. Yeah, I'll buy that from Amazon. But if it's an HBA, I'm not going to buy it from eBay unless I know it's a decent source, right?

Like if your HBA poos the bed, I mean, that's like serious for your storage system. Oh yeah, yeah. And you buy one that's been heavily used and it's already got heat damage or some other issues. I've seen that. It happens. You got to be careful what you get off eBay, yeah. Yeah. You buy one that's been heavily used and it's already got heat damage or some other issues. I've seen that. It happens. Got to be careful what you get off eBay, guys. So pay attention. Absolutely. Well, what should we cover in closing? What else have we not said, Chris? I know that we've kind of covered a lot of it. We kind of went deep in some areas, maybe not so bit of the future and what's to come. So I mentioned a little bit about Dragonfish. You're on the Cobia release today. First, let me explain our release cycle, because this is a little bit different for people who are new to the scale world who come from core. We used to be on kind of a 12-month release cycle with the Linux Trunez scale product.

We're now on a six-month cycle. So we do day and date versions. So 24-04, which is April of this year, and then there'll be a 24.10 and then so forth and so on. So every six months we'll be dropping some sort of new major release that has features and all that. So the next one, which is in the queue, is 24.04, which I know you mentioned Tom Lawrence. I think he just did a video on some of the cool features that are coming with that. 24.04 Dragonfish is important for a number of reasons. Number one is the biggest thing

that a lot of core users have been waiting for is the ZFS Arc functionality has been changed on scale. So you can use the same amount of RAM on scale as you would have on core for Arc now for cache. That's a big deal for people. That's something people have been waiting for for a while. We got that in. That's really kind of that last feature parity thing

we needed for the majority of core users who are kind of waiting to move to scale. So that's in. That's great. The SMB side. So if you're an SMB user, that's been beefed up. Significant performance enhancements for that. Multi-channel supported. And some of my favorite new functionality is you have things like auditing built in. So now TrueNAS can keep audit logs of your SMB user accesses, audit logs of user interface accesses. You can also see SMB session information now. So you can log in and see in real time who's connected to what and what sessions are running and all that good stuff. That's very, very nice. Let's see on the other side here.

A lot of improvements on the back end, kind of some ZFS performance thing, block cloning is enabled. Again, that's that inline dedupe. So if you're copying a file, you're an old school ZFS guy. So you know, if you copied files between datasets before it actually had to make a copy of the file to move it over and that took time, right? Well, with block cloning now, again, it's just making pointers. So you don't actually have to sit and wait for that, you know, 500 gig file to copy between a data set. It's just instant like that. So very nice. And you get the dedupe benefits because you're not storing a second copy of all those blocks. So that's a cool feature that's going

to be in Dragonfish. A lot of little new UI improvements as well. There's a fantastic reporting feedback button up at the top where you can, with a click, bring it up and submit us a screenshot of what you're seeing on your screen along with a debug. So if you do run into a problem, we can hopefully triage it and fix it just that much faster now. That's really an amazing little feature.

And then just the usual assortment of scattered performance things. Linux kernel 6.6, for people who are curious, we try and follow the latest LTS kernels from upstream Linux. So if you're building newer hardware systems, that'll be a much needed improvement for a lot of folks. But yeah, that's some of the major things. And we're excited to get that into people's hands.

The beta is out now. When we're recording this, I'm not sure when it'll air, but on Tuesday. Next week, one week from now. One week from now. When we're recording this, I'm not sure when it'll air, but on Tuesday. Next week. It'll go up one week from now. One week from now. Okay, perfect. So in that case, the RC is dropping March 19th. So RC1 should just be out now. If people want to try that out and test it, that would be fantastic. One thing I will give props to our software QA team here is we have invested heavily in our

test procedures over the last couple years in the team and how we do the testing, the automation, the manual, all that good stuff. The RC releases are closer to what, say, a.0 was a few years back. I mean, I regularly run betas and just don't have issues. I mean, the number of issues we have to fix is shrunk so much between beta and RC and.0. It's pretty pretty amazing that team serves a lot of credit so if people are ready to try some of those new features by all means give rc1 a whirl let us know if you see anything we'll fix it before the dot o yeah but uh yeah it's pretty solid platform and we're really thrilled to get that out the door so i i did some checking while we were

tilling out here i suppose i thought this is how unaware I was with what release I was on. So I'm on Bluefin. Oh, gosh. So when I go into my update section, that's the train I'm on. Okay. So are you saying I should train, I should move my train to Cobia?

Yes, I feel so much better now that you said that because that networking issue you had, I was positive we had fixed something around that in Cobia already. So you'll need to take some time here after the call or whatever, switch your train to Cobia, upgrade to 2310.2, and hopefully your network issue actually is resolved as well. And in terms of ZFS, that's always stable from train to train, I'm assuming.

Like you're just adding features or performance improvements. We update ZFS versions, but the number one thing for us on the true net side is data integrity and stability. So yeah, that's kind of scary to move from like beta releases, whatever. That's why I'm like, hey, I don't want to mess with this

because it's data. No, no, no. At the end of the day, ZFS is really the shining star under the hood and we have to have that stable. So that's the number one thing. Matter of fact,

if anyone even hints at an issue with ZFS, that's like the top of everybody's stack internally here. We're going to go investigate that until we know what's going on. So, yeah, even on betas, we don't ship with like really crummy ZFS code. That's too risky. So let's say there's somebody out there that's a version of me.

They're like, man, I'm going to switch my train from Bluefin to Cobia. What will I expect? What can they expect with this transition? Yeah, I have to go back and remember the Cobia release notes now, but a handful of new features went in there. Matter of fact, I don't know if you have the ability to show the screen, but if you go to the trunaz.com site, trunaz.com slash docs, what you will see is on that landing page you have a list of kind of the different versions, core, scale, there's our true command product and then our hardware enterprise systems.

And you'll see on that scale, for example, 23.10 release notes. That's Cobia. So if you go bring that up, you can take a look. And I'll just kind of highlight a few of them here. But the apps page got a huge overhaul. So that's a lot cleaner, a lot better UI design. The storage pool creation page got a redesign as well. It makes it easier to do large scale deployments.

We're talking hundreds of disks. That's a big deal. We also optimize to let you run up to 1200 plus drives on a single system. So think about it. 25 petabytes on one TrueNAS. That's pretty incredible. The multi, oh, SMB and NFS v4 multi-protocol. That was a biggie.

So we do have a lot of customers who have mixed environments, say Windows and Linux, or for whatever reason, they need to be able to run SMB and NFS at the same time. And starting in Cobia, we now officially support that with NFS v4 and Samba, and it preserves all the locks. So if a file is open on the Windows side, your NFS v4 client's not going to trample it, right? You don't end up with corruption. That's a big one. That opened a lot of doors for people. Let's see. Some stuff on the enterprise side for Alua support for enterprise. DRAID was another one. That's a newer ZFS pool layout you can use.

If you haven't read on that, I'd suggest go reading up on it. It's pretty interesting. I'll check that out. Yeah, yeah. We all supported over NetData. Now it's the backend system that's doing all the stats collection. And speaking of that, there will be some exposure of the NetData UI in a future release of TrueNAS coming soon. So people might keep an eye out for that. I think that's some of the main highlights, but at this point it's at a.2 release.

COBIA is pretty darn stable. It's what our enterprise customers are running in the field. So if you're not already on that and still hanging out on Bluefin, I'd suggest upgrading. Now's the time. Like I said, I'm a newer TrueNet as users. I'm not sure how to, you know, I haven't gone through much upgrading or, you know, train changes before.

So I was like, I'm just going to stay where I'm at. I don't know where I'm at. I think I'm good. You know, here's what to expect. It's super simple. You go to the UI and you see a train that says, Ooh, there's a new train that has the new version, 2310 Cobia. You change train.

It'll say, okay, updates are ready. Do you want to apply? You'll click apply. It'll probably nag you first. Do you want to back up your config file? We always recommend that. Just back up your config, store it on your desktop or laptop or whatever so you have a copy and then you just hit the button and kind of wait and it reboots and then it comes back that's all there

is well i know the one feature i'll definitely appreciate would be the arc changing because like why have 128 gigs of ram and dedicate only half you know to zfs i want to give it more because like this is a storage system yeah give it as much as i mean like i usually go to like 80 you know for most cases i don't know what you recommend for that but like 80 is a good number for a dedicated storage system it is and there's some calculation on the zfs side that happens it looks at the amount of ram you have and how much can we safely go and all that so i forget the exact formula off top of my head but it's somewhere in that 80 90 percent of memory depending on how much ram you have in the box will end up being used out of box that's in the dragonfish beta that'll be in the rc which hopefully just released and everyone can go grab

now again that's that's a worthwhile feature right there a lot of folks are happy yeah i'm doing it right after this call i'm doing it there you go so when this when this show goes out i'll probably be in the outro telling people how i did it and how fun i was well then go go do the beta or do the rc1 if you want if you want to get that oh really i should go to beta absolutely the dragonfish beta come on let's yolo here we can do that confident we can do i'm running all my personal stuff and i have i mean it's not a huge system i got 35 terabytes that I store on my home system. So it's data that matters to me. My wife would kill me if I lost all of our family photos and stuff.

Right. I mean, we're talking, I'm in a ditch outside if something happens there. So. Oh, yeah. I trust it. Dug by her. Yeah. Happily.

So. Okay. So your suggestion is to skip Cobia and go straight to Dragonfish from Bluefin. If you're adventurous, well, no, I would go to Cobia first. Make sure that upgrades good. You want to stair-step it. You want to go to the next release. Make sure everything looks good and is happy.

And then if you're adventurous and you want to go try out Dragonfish, give it a shot. And then just don't upgrade your pool if you need to roll back because beta is too scary. Fine. You can do that okay and so when dragonfish comes out of beta and it's a full release will be what next april or yeah this this upcoming april so we're about i was thinking like last year i'm bringing for some reason when i said next no it's hard to believe it's like 30 some odd days from right now is when the full release comes out. So it's right around the corner, like I said, especially with RRC coming out. We have quite a few people, I think, 1,500 plus running the beta right now.

Usually RRC, you're talking 10,000 start running it. It's pretty good numbers. And at that point, if it seems stable and people aren't complaining about anything you care about, you can take the plunge if you want. There you go. All right. Well, Chris, I'll take your advice. After hang up on a do what we just talked about.

Thank you so much for taking the time to just geek out with me on all the things you do for fun and for hobby, it seems. Sure. And I appreciate the hard work you put in for all of us to just enjoy hobbyists and enterprise alike. It's a lot of fun to – I find it to be a lot of fun to mess with storage. I'm like a weirdo, I suppose, in the world. And I'm just in the same room with a bunch of other weirdos in the TrueNez world because not everybody is like super fanatic

about ZFS as I am. And that's just how it works. So there you go. That's the part of what makes work fun around here, man. I get to come to work with people who, again, if I go anywhere else, I'm the weirdo in the room. But here it's like, dude, I'm in the club. You're accepted. Yes. You belong here. In fact, you run the show. That's right. All right, Chris, thank you so much.

No, it's been a pleasure, Adam. I really appreciate it. Thank you. Yeah, thank you. Well, there you go. TrueNAS scale is the future. TrueNAS Core is in maintenance mode officially. That means that it is being maintained, but it's not being innovated on. No new features necessarily, fixes, etc. Support, of course, in supporting enterprises. But the real future of TrueNAS is TrueNAS scale. Which makes me super happy because I chose TrueNAS Scale. And again, even though Alan Jude is very, very influential, let's just say, with FreeBSD, and I did explore

various FreeBSD scenarios for my home lab, ultimately, I came back to Linux winning. Doesn't mean that FreeBSD or any of the BSDs out there are not cool, doing cool things. Or for you, it just means it wasn't for me. And my home lab is now powered by TrueNAS Scale as the primary storage engine. And my Plex couldn't be happier. There you go. So hopefully you've got something awesome out of the show with Chris Moore. So much fun

digging into this stuff. Seriously. I mean, can you believe I get to do this for a job? Talking to people like Chris about ZFS and HomeLab and enterprise and hardware and Linux and storage. I mean, it's, it's a dream. And thank you for listening because without you, this dream would not be possible. And if you want to go one more layer in, there's two ways you can do it. First, the easy button. Join the community, changelog.com slash community. Free to join. Everybody.

Everybody's welcome. There are no imposters here. Join our Slack and we'll see you there. The second option is, of course, ChangeLog++. That's our membership. That's where we give you ad-free versions of the show. Drop the ads. Get closer to that cool ChangeLog medal.

Bonus content like today. Hey, there's a bonus for Plus Plus subscribers and so much more. A free sticker pack directly mailed to you. It's so cool. It's so cool. It's so cool. 10 bucks a month, 100 bucks a year, changelog.com slash plus plus. It's better. Man, it is better. It is better. Okay. So thank you so, so much to our awesome sponsors today. Century.ioio Use the code ChangeLog Get $100

One Zero Zero Buckaroonies Off of the team plan for you By going to Sentry.io And using the code ChangeLog

So cool And our friends at FireHadron Rolling in the deep With OnCall Yes, they are leading the market It is a full-fledged You can go use it with OnCall. Yes, they are leading the market. It is a full-fledged, you can go use it today, OnCall solution. Deeply integrated with all of FireHydrant. So, so stoked with that team and what they're doing.

And of course, to our friends over at ImageProxy, the best on-demand processing for your images for your front end. Love ImageProxy. ImageProxy.net. It's open source and they also have a fully upgraded paid version that you can use as well if you have significant scale needs. Again, ImageProxy.net. And of course, to the home of changelog.com, fly.io io that's where we live billy madison said it best go to your home go to your home and so we listened

fly dot io and those beats from breakmaster the beats freak in residence themselves breakmaster cylinder good beats good beats okay i'm gonna go tinker with my trunas installation and

I hope you do too. Bye, friends. Outro Music