The Changelog: Software Development, Open Source - From Sun to Oxide (Interview)
Episode Date: May 22, 2024Bryan Cantrill, Co-founder and CTO of Oxide Computer Company, joins Adam to share his journey from Sun to Oxide -- from Sun and Fishworks, to DTrace, to ZFS, to Joyent and Node.js, and now working to ...build on-prem cloud servers as they should be at Oxide.
Transcript
Discussion (0)
What's up friends?
This is the ChangeLog.
We feature the hackers, the leaders, and the innovators in the software world.
I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog.
And today, I'm joined by Brian Cantrell, co-founder and CTO of Oxide.
From Sun to Fishworks to Dtrace to ZFS to Acquisition, Oracle, Joyent,
and now building servers as they should be at Oxide.
Brian and I talked for a little over two hours.
I hope you enjoy it.
A massive thank you to our friends at fly.io.
Launch your apps, your databases, and your AI near your users all over the world with no ops.
Learn more at fly.io.
Okay, let's do this.
What's up, friends?
I'm here with Sama Alamnaylor,
Senior Developer Advocate at Sentry.
So we've been working with Sentry for many years now.
And I love Sentry.
We use Sentry.
It's so helpful for us.
But we don't write many bugs here at ChangeLog.
We're just that good.
But I do say often, how many teams use Sentry?
And that number has grown over the years.
It was 40,000.
Then it's 70,000. then it's 70,000,
then it's 90,000, and now 100,000 plus teams use Sentry. Numbers don't lie. Check the NASDAQ
scoreboard. Can you believe that, Salma? What are your thoughts on Sentry's impact to software teams?
Do you know what? I'm not surprised. It's a quality product. And I'm not just talking about that because I work for Sentry, but because I've used Sentry. And I think its success is Whether it's an official maintained SDK or whether it's
a community SDK, there's a way that you can implement Sentry in your projects with a few
lines of code. You don't need to really do much to get its benefit. And I think that's really
powerful also in showing that people want to make Sentry work for their frameworks or their
languages of choice because it works. And the fact that you
can self-host Sentry as well, it shows how valuable it is and shows how valuable Sentry knows it is to
people. The fact that it's open and out there and you can use it and configure it to your
specifications at the code level if you want. And if you want to not bother about that and pay for
it, then you can do that too. I'm not surprised and I'm not surprised that it's growing. I sound biased, obviously,
but it's the best error monitoring solution I have used in my dev career of many years.
And as a front end dev, it feels intuitive. I think a lot of these error monitoring solutions
are very backend focused. They're very stack Tracy and not really geared up with a good developer
experience. Like here are some logs, here are some things to spit out. You can read them if you care.
But with Sentry, it seems to appeal to more developers because of the way it's been engineered.
The amount of SDKs that are available makes it appeal to more developers. And you can get started
in Sentry in so many different frameworks in less than a
minute. And all the instructions are in the app and they point you to documentation if you need
it. It's, you know, a joy to use. And so I'm not surprised that that many people use it.
I'm glad you're not surprised because I'm not surprised either. It's an amazing tool. We love
it. We use it. Go to Sentry.io, use the code changELOG. That will get you $100 off the team plan.
It's basically three and a half months free or almost four months.
But code CHANGELOG will get you $100 off the team plan.
Use it.
Love it.
Sentry.
We love it.
Sentry.io.
That's S-E-N-T-R-Y.io. Brian Cantrell is here on the show solo with me, which is, I think it might be, honestly, I've interviewed lots of people.
But this might make my entire career.
I've been a fan of yours for so long.
For real.
Really?
And I'm just such a fan of what you're doing currently. But also we have to talk about where you've been
to where you've gotten to. So I think we talked in the preamble of like
are you comfortable? Can we take pee breaks? Can we go long? So we'll see what we do
here. But the plan is to don't stop, don't quit
until it's all out.
Let's do it.
Let's go deep.
Yeah, absolutely.
Do you have a lot of fans?
Do you think you're, do you have a lot of popularity?
Do people like you a lot, Brian?
You know, my kids say that, dad, you're not famous, but you are nerd famous.
Yeah.
And I have to say, it's kind of like the perfect level of fame.
Yeah. And I have to say, it's kind of like the perfect level of fame. I actually, it just happened a couple days ago.
I was in Chicago and someone came running up to me in the airport and stopped me to get a selfie and explained that they loved what we were doing.
That stuff is great.
I think it's terrific.
I love that what we're doing is connected to so many people.
So I think it's terrific.
But I don't know that I would want more fame than this.
I think this is just fine with me.
Were you wearing an Oxide t-shirt or something?
Or were you wearing a joint t-shirt?
Oxide, yeah.
Oxide.
Well, okay.
So they were like, looks similar.
I know that brand.
That must be.
Exactly. That's the guy.
Okay. Let me run up to that person because I feel comfortable enough to embarrass myself just enough to get the selfie.
I think it's great.
I don't have a selfie with you. Now I feel like this is like, have we met face to face? I don't think we have.
I don't think we have. I think it's just been in the conversations. Yeah. I don't think, I mean, it feels like we have, but I don't think we actually have. I think it's just been in the conversations. Yeah, I don't, I mean, it feels like we have, but I don't think we actually have. Yeah, it's such a weird thing in our world, isn't it? Where you, you know, somebody in
quotes, know somebody.
Right.
For so long, seemingly. And here I am just a podcaster. Here you are a world dominator
changing computing forever, so you say. And I just can't compare, you know, I just can't
compare.
Well, we've never seen one another's legs, right? That's the always looking, you see someone, you see someone's whole body.
There's always that little bit of a shock of like, wow, this person is taller than I would have thought or shorter than I would have thought.
Or this person is more average than I would have thought.
I mean, I've always, I've always felt that shock.
You came up recently.
I think it was in a post show.
And I mentioned, this was on Tuesday.
I mentioned I was talking to you.
And they said, have you seen the video of Brian yelling into the server?
Do you recall this where you were yelling into the server?
I haven't found the clip yet.
And you were trying to like predict latency when you scream.
Like, do you recall this?
Oh, for sure.
And so it's actually not me.
I am the videographer.
It's my colleague, Brendan Gregg, that's actually shouting at the drives.
Okay. And oh, for sure,
I recall everything about that. That is very present in my memory.
That was in 2008, and that was between Christmas and New Year's 2008.
And Brendan and I were solo in the office. Actually,
the full backstory there is that a couple weeks earlier, we were trying to debug a latency outlier in a JBOD, just a bunch of disks.
So we had a single disk that was an outlier and was very perplexing.
We're kind of staring at it, trying to understand, like, why is this drive exhibiting worse latency than every other drive?
With my colleague, Eric Schrock at the time,
Adam Levithal, a couple other folks.
And we were about to
head to lunch, and Eric
said the actual lab
which was running the JBOD was next door
to our office. And it's like,
well, why don't we go look at the JBOD before we go
to lunch? And I'm like, what do you
mean look at the JBOD? Like, we're going to go like,
wait, what are
you possibly going to see?
Just disks sitting there doing their thing, right?
They're plugged in, they're on.
Are you going to see a raccoon chewing
on one of the drives or something?
I don't know what we're going to see in one of the drives,
but sure, why not? Let's go look at the drives.
And actually, it's one of these
interesting object lessons that when you're debugging a
problem, you should always be open to asking questions that you think you already know the
answer to, because you may learn something. And so we went in, and we were looking at the drive,
and in particular, Schrock actually pulled the drive out. And when he pulled the drive out,
the drive was actually, all of the screws in the bracket had worked their way out
and the drive itself was just hanging on the connector. And so what we were actually seeing
was vibration in that drive. And so the drive was bad for a drive. It is bad. And in particular for,
this is a spinning media. This is, so this is not a flash drive. This vibration is bad in general,
but it's extremely bad for, I mean, a rotating media.
Spinning disk drive is a miracle of physics.
It's amazing that it works at all.
It is, yeah.
They're very sensitive to vibe, very sensitive to thermals.
And it was just very eye-opening because the drive was not reporting any errors.
The drive was merely reporting these latency outliers.
So we kind of did,
and we were all kind of marveling at this, you know, we replaced the screws and the drive worked
fine and kind of marveling about it. And Brendan was reflecting, it's like, it would have been
really nice to actually, let's recreate that problem so we can actually show it again.
And he was trying to experiment with ways to induce vibration. And so one of the
things he thought about was like, well, maybe I can emit some sounds. So he was running like a
Python program to emit various sounds to try to hit a resonant frequency of a drive or something.
So by this point, it was between Christmas and New Year's, and he and I were alone in the office. And Brendan comes running in to the office from the lab next door and says, you've got to see this, and runs out.
And Brendan's the kind of guy that doesn't really run very frequently.
If Brendan's running, you should run in the same direction that Brendan's running, right?
Okay.
So I'm like, I don't even know where I'm going, but I start to run after him.
And he says, watch this.
And he has our analytics up that shows the real-time latency.
And he screams at the drives, and you see the latency all spike up.
And so he hands me the video camera
and I'm like, we've got to video this.
And I video him screaming at the drives.
And two minute long video.
It was, I was laughing during it
because I was basically seeing it for the second time.
And we uploaded it to,
there was a relatively new website at the time,
YouTube, right?
It'd been around for maybe a year and a half,
something like that.
For sure.
And I'm like, let's upload it to YouTube.
So we upload this video to YouTube
and thinking like,
this will get like a thousand views.
It'll be kind of fun.
You know, we'll kind of show
some of the things we've done
in terms of the storage appliance.
And I mean, I believe that thing
has got like 4 million views at this point.
I mean, the views are
kind of crazy, and every
so often it gets another
huge wave of
views as it starts. The YouTube
recommendation algorithm recommends it to people.
So it's a video that's endured
and it was seen. So this is
when we were at Sun Microsystems together.
That video has been watched more than any other marketing material that Sun put together, that Sun developed.
If you take the next, I believe, 10 put together, that video has been viewed more than they have been.
It's only Sun's Super Bowl ads.
Sun had some very ill-advised Super Bowl ads.
The Sun Super Bowl ads, we had two Super Bowl ads
that were viewed, obviously, by more people.
Sun was the kind of company
where it didn't even occur to me to
ask permission before we did that.
Fortunately, the marketing
folks were pretty forgiving.
One thing we
do wish, though, is you didn't actually
mention the name Sun
Microsystems at all. I'm like, did we actually mention sun the name sun microsystems at all
i'm like oh did we forget like no company name no logo it's like oh sorry about that right we
just mentioned fishworks but yeah i definitely remember that that video very very vividly and
then youtube was convinced that i was like some you know early youtube star and i was trying to
tell it that like no like this is we're a one trick kind of pony here.
This is accidental, not on purpose.
It's accidental.
Exactly.
Brendan was the talent.
Just showing the world how we work.
That's exactly right.
Just showing the world something
that we thought was pretty interesting.
Well, I think, so I went and found it.
I was thinking to myself,
geez, Adam, why didn't you, you know,
search YouTube and prep for this
and find this video?
So I searched screaming into the server, which seems
accurate, but the title of it is shouting in the data center.
And this is on your user, your personal user on YouTube.
So this is the original 15 years ago.
4.2 million views. Does that sound accurate?
Yeah, that's it. That's the one.
And my kids are like, you idiot.
You could have been a YouTube star.
You could have been like an early YouTuber,
and you fumbled your own future.
My kids view this as just like, I'm a dismal failure because I didn't actually capitalize on the story's success.
But I try to point out, like, look,
Brendan was the on-screen talent.
I was just the one holding the video camera.
Right.
And it was very much just reflecting something But I try to point out, like, look, Brendan was the on-screen talent. I was just the one holding the video camera. Right. Right?
And it was very much just reflecting something that we thought was an interesting property of the system that we discovered.
So going in on the technicals of latency from vibration, I would just call that loosely, like the screaming and the shouting creates vibrations.
What exactly happens? You know, going beyond this awareness, now you're aware.
Did you go further to root cause? Okay, what is actually happening inside the spindles that
creates the latencies? Does the drive just spin less fast? What happens? No, no, no, no, no. So
the drive is spinning at a constant rate, 7,200 RPM, but the pressure wave of the sound is blowing
the head. It's moving the head out of alignment.
And it's getting what's called a non-repeated run out.
So the head is losing its track.
And the fly height of these things is actually extremely low.
It's really vulnerable to these kinds of things.
And these are before helium drives.
And there are other things that I that would arguably change the dynamics today.
But they're vulnerable to that kind of disturbance.
So it's pretty easy to knock the heads out of alignment and then they get retracted.
They're able to figure out, okay, I've gotten knocked out of alignment and I've been able to retract myself and then perform the read. It's kind of,
I mean, maybe a tragedy is a bit too strong, but
we don't really,
hard drives are fading pretty quickly.
They're being replaced by flash
because they are so mechanically, finically.
There are a lot of problems
with a hard drive. We
experimented for many years with making
that rotational velocity faster,
but that has gotten
really diminishing returns so we had like 15k rpm drives and they even made 20k rpm drives
but relative to the access of flash like there's no comparison flash is much much much better
skip the spinning just move to something different that actually has better technology because
at some point you got thermodynamics that are going to push back and physics that push back on like you mentioned helium i'm imagining that the reason for helium
and i don't know all the science behind hard drives but i know that the fly height of the
head that reads the data off the disk is minimal right like there's micrometers i don't even know
what the term would be to like nanometers so the the fly height and this is learned this from we deployed
a lot of hgst drives from their vp of quality because we're talking about the importance of
thermals in the data center for hard drives and the fly height of an hgst drive during a write
and i'm like i would have guessed i don't know what i would have guessed, I don't know what I would have guessed. I actually don't know. Hundreds of nanometers, I don't know, was 0.8 nanometers.
And I'm like, I literally fell out of my chair.
I'm like, do you mean 800 picometers?
And he's like, yeah, it actually is 800.
You know, I hadn't thought about it that way, but it is 800 picometers.
I'm like, what do you mean you hadn't thought about it that way?
It's like 800 picometers. I mean, that do you mean you hadn't thought about that it's like 800 picum
i mean that is so unbelievably i mean that is the accuracy yeah the precision is just how do you
even say the engineering behind a hard drive is like you said before basically a miracle that it
even does what it does and i'm i mentioned the helium because i think that probably does with
friction right you have to worry about friction with the speed of the drives moving, right? Like this is all an issue is like the heat created inside of the speed of a 7,200 RPM rotation of what are they six inch?
No, they're like three and a half inch drives.
Like the spindles around three and a half inch.
And you are flying through air.
And that's why I think that moving to helium, changing that kind of substrate allowed for much more precision.
But I mean, you have like, things like smoke can
destroy the head. I mean, so you can, I mean, God, if someone
were to do smoking in the data center, you could really do some damage. They're very,
very, very precise instruments. And it's amazing that they work at all.
And then, of course, all we do in software is complain about them.
Absolutely. I guess fast they're kind of a...
I guess fast-forwarding a lot, because I do want to go back in the past,
and we are in the past, because this was 15 years ago while at Sun.
But now Oxide only deploys to NVMe, right?
You only deploy to Flash.
So fantastic, right?
You've made it.
Congratulations.
That's right.
No more hard drives. No more hard drives.
And, you know, we
do not, we do not
yet have an object storage kind of a
cheap and deep product.
And I still think like when we do
that, that's going to be like QLC based
with our own FTL.
I don't think it's going to be. I don't even know what you just said there.
I know QLC.
Everything else, I was like, what did you just say? FTL's a flash translation. Was that English? Was that an acronym? What was it's going to be. I don't even know what you just said there. I know QLC, everything else. I was like, what did you just say?
Right, FTL is a flash translation layer.
Was that English?
Was that an acronym?
What was it?
Yeah, sorry.
Yeah.
So, I mean, QLC is a cheaper, denser, but less reliable flash.
And FTL is a flash translation layer.
And we would presume that we do our own software on top of QLC. But in part because the real challenge is that they have done an amazing job of improving the density of rotating media.
But you can't do anything about the access time or very little about the access time.
You can kind of like add a second head and do some other tricks.
And so you end up with this.
I mean, it ends up being a tape drive.
It ends up looking logically like a tape drive where you just, you literally cannot, it takes so long to actually extract all the information out of it that if you actually blow a drive, and I guess I don't miss it at some level,
but I think it is probably on the way out.
Although I've been predicting the demise of the drive also for 15 years.
It'll probably still be with us.
Yeah, I'm with you on that.
Hindsight a bit is 20-20 in that case,
but you've been in the game long enough to predict that future.
Let's go back then.
I mean, I think that I wanted to obviously touch the fact that you're at Oxide.
You founded Oxide.
You are, in quotes, creating servers as they should be.
But you began probably further back than Sun.
But I began there with Fishworks and this sort of common ground to some degree.
ZFS was born inside of Sun.
You worked and co-founded Fishworks, which was a group that really created an appliance, I think, based on ZFS.
And obviously, this is your life, so you can tell the story better than me than bumbling and mumbling through it.
But I didn't know that you worked at Sun.
I had always just thought you were a joint.
I never knew until I prepared for this conversation. I had always just thought you were a joint.
I never knew until I prepared for this conversation.
I never knew.
And here's me.
I've got this personal satisfaction and love for ZFS as a file system. I think it's just phenomenal, really, how easy it is to use and how resilient it is and how enterprise-grade it is.
And all it takes is just an app getaway
to have this thing on any given Linux box.
And I can just have the power of ZFS.
And here you are, this division, Fishworks,
building an appliance around ZFS.
You were inside of someone that was created, right?
Yeah, I mean, more direct than that.
More direct, take us there.
Yeah, so I went to Sun in 1996,
knew that I wanted to do OS development
and I wanted to work for a computer company.
Even then, I really saw the power
in combining hardware and software.
And every computer company in 1996,
with the exception of Sun,
had decided to mortgage its future to Microsoft and Windows. It was a really dark time. Wearing the Unix shirt today.
And Unix was on its deathbed. This is before, you know, Linux was still very much a hobbyist
project. BSD was mired in lawsuits.
Windows was, Microsoft was in the height of its Gates and Balmer era.
The world is very proprietary.
They're not really open source systems.
And every major computer manufacturer, SGI, DG, IBM, and so on,
were all moving to Windows as their future.
And it was really only Sun that saw its future as being its operating system.
And I had used Solaris a lot as an undergrad.
And I had actually, I'd had kind of, I would say,
a complicated relationship with it
because we'd used Solaris from the early days of Solaris when there were some real problems with it.
But it was a real – so when I came out to interview at Sun, I didn't really know what I was going to find.
And I interviewed with Jeff Barnwick, and it was just an absolute bolt of energy to talk to Bonwick.
I mean, Jeff was the one.
I actually, Jeff had a post to Comp Unix Solaris describing what was new in Solaris 2.5.
And I had reached out to him after seeing that.
And meeting Jeff in person was a transformational moment in my life because I met someone who was older than I was,
10 years older, roughly,
but had a real energy and zeal
for system software development
and realized that I was going to come work,
going to come join him and come work at Sun.
So, you know, he and I were, along with Kevin Clark,
were in the Solaris Performance Group.
And Jeff really had a...
I mean, Jeff had a couple of chips on his shoulder.
I would say that he and I, respectively, each had a big chip on our shoulder circa 1996.
I had a chip on my shoulder around the way the system was understood.
I felt that the debugging infrastructure in the system was really inadequate.
And I didn't understand why we had this entirely synthetic system that we had created.
And yet we couldn't ask ourselves what the software was doing.
It was really frustrating to me.
There was a real lack of observability of the system.
And I had had some ideas as an undergrad about how I wanted to go instrument the system
dynamically and for the idea for what would become DTrace. And when I was actually interviewing with
Jeff, this is one of these moments where I remember exactly where I was. We were kind of
coming back from lunch over the 101 from Palo Alto. And I was walking Jeff through my ideas for what would become DTrace and trying to understand why we,
why was dynamic text modification not being used for instrumentation of the system? I just didn't
understand it. And, you know, Jeff was like, yeah, sounds like you should do that. In fact,
you should come here and you should do that. And that was a really exhilarating feeling
to feel like the future was open
to someone who was so young.
So that was my chip on the shoulder.
And Jeff's chip on the shoulder
was really frustrated about file systems
and frustrated by, he thought,
UFS was a mess
and that all of these various layers, the layering on top ofFS was a mess and that all of these, these various layers, uh, the layering on top of
that was a mess. And, you know, I think he and I respectively both really wanted to take on these
problems and it took until, you know, it would be three to four years later before we could really
go start that. And I started the trace in 2001 And at roughly the same time, he started Pacific, which is what ultimately became CFS.
And we realized that he, there was kind of an aborted year where he had tried to kind
of take an existing organization and turn it into something that would develop something
much more novel.
And it was just not organizationally going to happen. I could personally see that like, what actually what Jeff needs is Jeff needs someone who is
a fresh grad, who is remarkably talented, who can who's energized and can join him,
and they can be bound together and go take on this problem together and not a large team.
And I remember vividly when I was recruiting at a school and we took Adam Leventhal and Matt Ahrens out to dinner and put the two of them in a cab back to school. And I said to my colleague
at the time, the future of the company is in the back of that cab.
Adam and Matt both came out to sun Matt to join Jeff on CFS and
what became CFS. Then Adam would join Mike and me
on DTrace. Adam, of course, works for me today at Oxide.
Matt very much still involved in CFS.
That was the beginning of a very, very, very long odyssey.
And it's, you know, it's kind of remarkable to me now that like, I'm, you know, I'm 50
now and, you know, I think that, you know, Matt and Adam were 22 at the time, you know,
they were only, only slightly older than my oldest kid.
And to, I mean, almost the, the, the kind of, you think about the,
the arrogance or the hubris about seeing that kind of potential in someone so young,
but that potential was real, you know, that was right. And we can do kind of extraordinary things
when we are given the power to do so. And, and then I, and ultimately Jeff and Matt building
a much larger team around them. It wasn't just the two of them, but it definitely started with the two of them really testing out some of these early ideas.
And then certainly it was really exhilarating for us on DTrace. And there were a bunch of other
things happening that same time, that time in the early 2000s. It was kind of ironic because
it was after the dot-com bust. I mean, people do not know what
an economic wipeout looks like. I mean, we have not seen anything since then. Yeah. That rivals
close, but not quite. Yeah. I mean, not quite. Definitely not. I mean, I think if you, if you
work in tech, because I think that what people don't realize is that like 2001 was devastating.
I mean, it was in a different era, right?
There was no remote work, really.
And companies slashed so much.
I mean, I don't know if you were in Silicon Valley at the time, but anyone who's in Silicon
Valley at the time will tell you the eeriest thing was the traffic disappeared.
Is that right?
Yeah.
In like April of 2000 was the traffic disappeared all right yeah in like april of 2000 the traffic
disappeared and you're like why is there no traffic right now and then the traffic like
never came back and you realize that like something is very large is happening and you know
it was you know there was a long shadow from that through 2002, 3, 4, 5, 6. It was only in kind of 2007, 2006, 2007,
things kind of began to crawl back. And then 2008 was, of course, a very deep recession for the rest
of the country. But I think it was actually less, I think tech was still recovering from this nuclear
bomb. So I think it was actually less acute in tech. 2008 was largely in housing. For sure. But
it affected everything, but it was largely in housing. For sure. But it affected everything.
But it was largely in housing and it affected everything by way of everybody has a house or lots of people live somewhere, right?
Whether they own it or not.
So there was an effect that waved through things.
But the dot-com bust was literally in tech.
Like the epicenter was tech, right?
It wasn't adjacent to tech.
It was in tech.
It was in tech. It was in tech.
And I think it just affected forward expectations so much where there was this idea that like,
okay, all of this kind of heady thinking of the late 90s was just obviously wrong.
And of course, it wasn't.
Most of it was just ended up being actually just way ahead of its time and not actually, you know, Webvan wasn't really wrong.
A lot of these things were not necessarily wrong.
They were just way, way, way, way, way ahead of themselves.
But the irony is that it was in that period that I think Sun was its most innovative, which is kind of crazy because it was when Sun went through
layoff after layoff after layoff
during that period.
But in part because of,
maybe it was the focus
that desperation gives you,
but it was during that period
that I think we did our most interesting work.
And that was when ZFS happened.
That was when Detroits happened.
That was when the Superhumanity Facility happened
and FMA and Fire Engine, a bunch of other really,
I think, interesting OS technologies. All that became open source then in 2005,
which was really important to me personally, because I began to realize that I've kind of
done this very important body of work for myself in terms of detracing. And I created the tool that I wanted to use.
And now I am actually worried that it's going to be trapped in a corporate vessel that's
going to sink.
And the last thing I wanted to do is to be the kind of person who was pining about the
tech that they no longer can have.
It's like, I've implemented this thing.
I want to be able to use it for the rest of my career.
And actually, it was great because just yesterday,
I had a really interesting bug at a customer site, Dockside.
And man, we were using D-Trace so much to debug this thing.
And the engineer, Robert Bustocki, who was working on this,
he and I were both reflecting on it, being like,
man, thank God for this.
Thank God for D-Trace.
How would we debug this without D-Trace?
That's so crazy. It's crazy. I mean, that's
gotta be like so crazy for you
to be using the tool
so long from the day it was created
to be the person who's like, something's
gotta change here to do the work, to be recruited
to do the work, to be in tumultuous
times in a company in an era
of tech that's never been seen before, at least we hope not,
to be still using the tool because of the open sourceness of it.
Otherwise, it would have been trapped, right? ZFS was almost trapped in
Sun. It's open ZFS. There's some history there licensing-wise
that makes ZFS a little challenging for folks to implement, like in Ubuntu
or Linux today. It could have been that.
Yeah, no, and I think that, absolutely.
And it would have been very easy
for all those things to be truly trapped.
And it would have been awful.
You know, it would have been really...
More than awful, yeah.
Yeah, no, I think so.
Catastrophic.
And I think that, you know, this is like,
there are many, many, many reasons to open source things, right? I mean, there are lots of reasons why open source is really important. But you know, one of the emotional things that we don't talk about in terms of open source, but to me personally, is something that I feel like deep in my heart, is open source is what allows engineers to achieve enduring meaning.
And to like really feel, you know, as an engineer, if you ask an engineer, when have you been unhappiest in your career?
It's always a variant of the engineers trying to make the right thing happen and management was preventing it from happening. right? It's like the same story. And I feel that like doing proprietary innovation is kind of a variant of that where you, you work so hard for something that now somebody else owns. And on the one hand, it's like, okay, they owned it because, you know, they into the side of the mountain and you no longer
have the thing that you created, that you poured your soul into. And like, that's really, that can
chip away at your soul, you know, I mean, to kind of lose that. And conversely, it feels, I mean,
bluntly transcendent to be able to have this technology that I am using, you know, 20 plus years after
its creation, and I'm still grateful for it. And, you know, it's so, I, you know, I love the fact
that it's so, it's something that we are able to use so easily, so readily, so robustly. It's,
that is deeply, deeply, deeply gratifying. So yeah, I mean, it changes one's
disposition to kind of have that longer view about the importance of open source for sure,
to allow engineers to have that deeper meaning. And we got very lucky because we got that deeper
meaning with the stuff that we worked on at the time. I love that story and that history.
I got a different perspective of this, I guess, same story in a way from Matt Aarons directly
so I talked to Matt Aarons a couple years back you may have listened to that episode I don't
know how much of a fan you are oh no yeah I did yeah yeah well you knew the story then I mean you
were probably just listening for posterity's sake for just sheer enjoyment because you were there
for the most part for all this stuff and he was just literally telling me the title of that show was Making the ZFS File System.
And I thought that was cool.
It was, you know, Jeff, how he recruited Matt.
He didn't share the whole, you know,
those two in the back of that cab there,
the future of the company perspective
like you did just now.
But he was like, I'm a co-inventor of it.
But really, Jeff was the real brains.
I was a brand new person to the company.
Hadn't really done anything substantial. Obviously, I brought a brand new person to the company. Hadn't really done anything substantial.
Obviously I brought a lot of knowledge to the table and capability,
but really this was the brain shot of somebody else.
And I was just there at the right place,
the right time to help make ZFS what it is.
So I think Matt is being characteristically self-effacing.
Is that right?
Yeah.
He's being self-effacing.
I mean,
I think that he's right.
So,
I mean,
he's being very generous to Jeff and you should be, but the, and Jeff's obviously much older than, yeah, yeah. He's being self-effacing. I mean, I think that he's right. So, I mean, he's being very generous to Jeff, and he should be.
But Jeff's obviously much older than I am, and I'm five years older than Matt is.
So there's an age and experience gulf for sure.
Matt also brought not just enthusiasm, but terrific acumen and insight.
And I remember one of the very first,
I can't remember if Matt talked about this or not,
but one of the very first things that Matt did
is a prototype of like the directory hashing.
And I just remember how great it was for Jeff
that he hadn't had to like micromanage this.
He kind of like pointed Matt in this vague direction.
Matt figured a ton of stuff out
and had a bunch of his own insights besides.
And I think Matt is discounting that.
I don't think Matt necessarily,
I mean, Matt's on the inside of it.
So he's not necessarily,
he didn't have the perspective that I had on it,
which is you have no idea how invigorating this is for Jeff.
I mean, yes, on the one hand,
Jeff's providing a lot of the energy and the seniority for sure. But Matt was providing so much of that energy that it was energizing for Jeff. So, I mean, it very much required,
my view on it was like Jeff, I said, Jeff needed someone to pair with on this.
And Matt was perfect.
So I think it very much required both of them.
It's crazy.
Kind of fast forwarding a little bit.
I was like, let me go command K your docs at Oxide and see where ZFS lives inside this stack.
Because I can't imagine someone like you would build what you have even without your history, without having ZFS based in there somewhere, right?
That's right. OpenZFS. And so, of course,
you're, you call
it Crucible. The Crucible storage,
yeah, Crucible is our storage service.
And so we, I mean, we're using
ZFS in a slightly different capacity,
but very load-bearing for us.
So what we are not doing
is using the kind of the volume management
piece of CFS.
It's not at the moment.
We are putting effectively a different Z pool on each of these U.2s.
But we are very much using the fact that CFS has got this, as you know, the idea of the file system as this lightweight abstraction that you can create many of.
And you can have different policies around.
I mean, there's so many different winds of CFS.
So while we aren't using the storage pooling necessarily,
we are using all these other features of CFS.
And it's been, yeah, it's been terrific.
So far, so good.
Mainly copy and write and snapshots, right?
Like that's what you come for.
You stay for everything else,
but you come for copy and write and snapshots.
Snapshots, the encryption data at rest.
I mean,
there are,
the fact that you've actually
got a proper file system
and you don't have to,
I mean,
there are a bunch of things
that are,
there's a bunch of value
on top of it.
And then,
and actually,
honestly,
from my perspective,
the fact that you are,
you actually have
the data integrity
guarantees.
I mean,
I have seen,
I mean,
and Matt may have relayed this to you. So, and I have seen, I mean, and Matt
may have relayed this to you.
And there were a couple of these, but one that
really sticks out, I can't remember
if Matt relayed this to you or not,
there was someone
who's reporting, ZFS was reporting
data corruption, right? So ZFS is not going to return
corrupt data to you. ZFS will indicate
like, I've had data corruption.
And ZFS is reporting data corruption. This is relatively early on in the development
of ZFS. And everybody believes
that ZFS is mistakenly reporting data
corruption, or ZFS is itself corrupting the data.
And as I recall, several weeks of debugging
before they realized that this particular DMA controller, silicon homage DMA controller, if memory serves correctly, was corrupting the last 64 bytes of a transaction under certain conditions.
And this is a bug in the controller itself, not in our software.
And this is one of the most widespread DMA controllers anywhere.
And they realized that like, oh my God, this thing's just been corrupting data everywhere
and only ZFS detected it.
And it was crazy.
It's awesome.
No, and it was like one of those moments.
Not awesome, but awesome that otherwise it would have been unknown if ZFS wasn't as good
as it is, basically.
Absolutely.
And it was one of these moments where you realize that these storage systems are really complicated.
And that when you have these kinds of errors, people don't necessarily have ways of observing them.
So, you know, a DMA controller vendor is not necessarily going to, they're going to be very hard pressed to be directly connected to the folks who are actually seeing data corruption.
Because software doesn't routinely validate the data that it's getting back. Or if it doesn't,
it doesn't in a checksum that is stored within the same block. And if you just give me,
if I gave you a block that you corrupted on the way down, you corrupted it on the same way coming
back up. It's like the block looks self-consistent. So like, I don't know the difference. It's like,
no, no, this block is actually corrupt. And it was very eyeopening. And it's like, I'm sorry,
I'm not living without that ever in my life. I'm just like, no, thank you. I would like to
know when my data is corrupt. Please don't return it to me if my data is corrupt. I mean,
file system corruption is really, really horrifying because it's very hard to trust
again when you've seen file system
corruption. And, you know, in all of my years with, with ZFS, it is all of our years, the
fishworks appliance, as I told people, we, that we had some data take some very long vacations,
but it always came home. We, we, we not actually have corrupt data. And that is, I mean, first and
foremost, data cannot be corrupt. So yeah,
definitely I'll be using ZFS for the rest of my career. Thank you very much.
Yeah, thank you very much. Thank you, I suppose, to the early days, like you'd mentioned,
even in your own passion for Sun to get the open source, those things. And I think it was only
because of those actions was what is known as EFS was able to survive Oracle's acquisition,
which you were there for, I think in 2010, right? Like you were there based on your LinkedIn
history. I assume, you know, the date range seems to stamp when the acquisition happens,
you were there. Oh yeah. Right. Very much so. I actually went into the acquisition with a very
open mind. I, I mean, this just sounds like, God, this is embarrassing.
And it sounds comically naive.
Well, I was just like, you know, this could be good.
This is going to be, we're going to, I know, right?
We're going to take the technical excellence of Sun
and we're going to combine it with the business savvy of Oracle.
And it was almost immediately after the acquisition
that I realized just how stupid that was.
And that Oracle doesn't actually have business savvy
in any true sense.
Oracle understands coercion.
That's what Oracle understands.
And the way that Oracle operates is they get the leverage
to be able to coerce their customers,
and then they lack the mirror neurons to stop themselves from just excruciating coercion.
And so, I mean, I felt embarrassed to work for Oracle, which is not something, I felt ashamed,
not something that I would have anticipated. But one of the things that I really learned very quickly after the acquisition
is that, you know, despite Sun's faults,
and Sun had plenty of faults,
Sun's customers, our customers,
wanted Sun to succeed.
And what I realized was like,
despite our faults, our customers trusted us
because we were always transparent with them,
like, worst of all.
Sun did try to pretend that it was something. I mean, again, every company has got faults.
But for its size and history and staying power, Sun had really surprisingly few – there were no – no one went to jail at Sun.
There were not these major ethical transgressions.
Sun's customers, by and large, wanted Sun to succeed because they trusted the company.
That was not true of Oracle.
When you get to Oracle, Oracle's customers don't trust Oracle.
Oracle's customers are—when I was there in 2010 like the state of California refused to purchase any more Oracle. Like the state I lived in, the government of the state
I lived in refused to purchase the product because of the, they had, you know, outstanding lawsuits.
And I mean, that's not what I want to do with my career. You know, I, I'm not, what I learned in
that whole process is that the, the, the meaning of an endeavor is really important to me.
Like, there's got to be a bigger why.
Why are we doing this?
And I have always believed that in computing, you know, there's this line.
I'm not a huge Halt and Catch Fire fan, but there is a line that I do love, which is, I'm a much bigger Silicon Valley fan than Halt and Catch Fire.
Oh, gosh, you said it first. Okay Silicon Valley fan than Halt and Catch Fire. I think so.
Oh gosh, you said it first.
Okay.
There you go.
We'll go there.
But there is a line that I do love in Halt and Catch Fire, which is, we're not the thing.
We're the thing that gets us to the thing.
And to me, what that means is like computing is a tool for humanity.
And it's a tool to allow people to do extraordinary things.
And I just feel really strongly that the meaning of being in computing infrastructure
is to unlock the potential that people can go create for themselves.
And that necessitates a relationship with customers that is based on trust.
And you can't have this transactional antagonistic relationship with customers that is based on trust. And you like, you can't have this like transactional antagonistic relationship with your customers because you lose the meaning.
It's like, you know, no longer is it about how can I use your thing to go create something larger
than the two of us together? It's about you trying to extract the most amount of, you know,
licensing revenue from me or whatever. And just like, no, no, thank you.
That's not for me.
And Sun was always about something bigger.
And I think that you can see many of Echoes
and many have seen many Echoes of Sun in Oxide.
And we've tried to recreate some of the good bits
without recreating the bad bits.
But certainly the good bits we've tried to recreate in Oxide
is that called emission.
All you have to do is not get acquired by Oracle.
I think you're good to go, right?
I think I'm pretty sure
we're going to be safe on that front.
Can you represent that right now?
I can represent that.
That's truth?
Yeah, I think so.
Okay.
I don't think Oracle's going to be interested
in acquiring Oxide.
Not yet. We'll see. I don't think so, no. be interested in acquiring Oxide. Not yet.
We'll see.
I don't think so.
No.
If you do your job right, maybe.
You know what?
If we do our job right, I actually – no.
I actually – this is another, I guess, belief of mine, but I really want us to be – like the exit for Oxide is a public company.
I think that I just,
I believe in the democratization of capitalism. I believe that it should be owned by the public.
And I know that plenty who are in a public company
will like roll their eyes at this
because it's a lot of work to be in a public company.
You've got a level of scrutiny
is obviously extraordinarily high and so on.
But I just think we see the greatest societal dividends
when companies are public. I think there's a lot of importance. And I guess this makes me, you know, it says
something crazy about our divisive age, that that view seems nuts. That it's like, oh my God,
like who is this socialist that they believe in the public companies? But I do actually believe that companies should be, I think there's a lot of value
in being owned by the public. When you say public, you mean publicly traded, right?
Public traded. Public traded, public owned. Which is actually not socialist, right? That's capitalist. That's my view.
That when you have, when people can, you know,
because it used to be that companies went public much earlier
and a lot of that kind of ascendancy, the public could participate in.
And, you know, you could have someone who, you know, was a barber who was able to put
away some money and buy some shares of Apple.
And, you know, they've got a nice, they've been able to enjoy some of that success, though
that we've democratized that success, I think is really, really important.
So that to me is the future for Oxide.
Big public company.
Hey, friends. Hey friends, you know how much I love Tailscale and I also am a big fan of Alex. He's awesome. He also does some really awesome videos on the YouTube channel for Tailscale. And recently
he did a tutorial, kind of a walkthrough of using Tailscale with Home Assistant and what it took to
set it up and access Home Assistant remotely via Tailscale. Take a listen. Under Home Assistant,
we're going to go ahead now and install another add-on. So I'm going to go ahead and install the
Visual Studio Code server add-on. Whilst that's doing that, I'm going to go ahead and go back to
the Tailscale add-on that we installed earlier and just grab the piece of
configuration that we're going to need from the documentation. In the
documentation page do a command F or control F and search the page 127.0
and there you go. We just need these four lines of code here. Home Assistant by
default blocks connections from untrusted proxies, such as the tailscale proxy. In this
case, we're going to add the 127.0.0.1 as a trusted proxy in the list here. So I'm going to
go ahead and copy this to my clipboard. I'm going to go ahead and click on start and then also show
in the sidebar. And you can see we're basically in Visual Studio Code, but in a browser. And this
is running directly on Home Assistant and has access to your configuration files and what have you underneath. All we need to do is paste those four lines into our configuration.yaml
file and restart Home Assistant. So I've pasted the four lines I'm going to go to the hamburger
menu up here click save and then settings and restart Home Assistant. We want to go back to
the add-on section and under tail scale we're going to have to go to the configuration tab for the add-on and click on tailscale proxy. This is going to
turn on tailscale serve. This is what will automatically generate you a TLS certificate
using Let's Encrypt for your tailnet.ts.net tailnet name. So if I click on save here it will take a
moment but it's going to restart the tailscale add-on. And so now I should be able to go to https home assistant velociraptor.ts.net and it's going
to load my entire Home Assistant instance over Tailscale with a TLS certificate using the name
from my Tailnet. And I can log in just as if I was using the IP address and port number that I was
before. And you can use this name from anywhere on your tail net.
So any device that's connected to your tail net,
such as a phone, for example,
that can now connect to Home Assistant
whether you're in the house
or whether you're at the coffee shop
or whether you're in Iceland looking at volcanoes.
It doesn't really matter where you are.
If you're one of the few out there
who have not tried out Tailscale yet for free,
you can do so today up to 100 devices
and three users
totally free at Tailscale.com. No credit card required. Just go there, sign up, get 100 devices,
three users, totally free. That's where I'm at. I use Tailscale totally free and you can too.
I'll link up Alex's tutorial in the show notes. Check that out. TLScale.com.
Do it now. that's uh those are lofty goals we're 55 ish minutes in and we have barely talked about oxide really we've just talked about your journey which i think is super interesting personally so i hope
the listeners also appreciate it i'm a a historian to some degree i would say over the last two years of sun and zfs in particular i didn't know about the story
behind the scenes of d trace and its origins i've used the tool before but i've never knew its
origin i didn't even know you created it i'm so embarrassed in a way to know you like my fame for
you i suppose began with joint as cto there not everything before that
which was you know really your true beginning and maybe your best work not that there wasn't good
work at joy and but you've left and you've done what you've done now to kind of recreate what you
loved most inside us on as you just said to sort of give on-prem this this full stack a new a new
life i mean i would say oxide pulls from a lot of different threads. Oxide pulls
a lot from Joint in the history of Joint. Oxide pulls a lot from, I mean,
Oxide is pulling in all of our history running a public
cloud, developing the software there. You know, we had the whole Node.js experiment,
which was interesting, but it was...
It was interesting? Well, you know, it was... You're laughing about it. It was interesting?
Well, you know, it was a whole different set of education, right?
Where one of the things that I... And part of the reason I went to Joyent
is because of my own enthusiasm for Node.
And Node was happening at the time.
So this is in 2010.
And we had used at Fishworks, at Sun, we had obviously, we'd done our own, we really wanted to
make this thing a tight product, which is one thing that Sun was not very good at historically.
So we wanted to take these great technologies in terms of ZFS, in terms of detrives the operating
system, and really turn that into a very tight product.
Part of that meant a great graphical interface,
which Sun was not historically very good at.
In 2006, when we started, that meant AJAX, right?
Remember AJAX?
Oh, yeah.
JavaScript and XML.
And so we pretty quickly drew the conclusion that all we are not all these like frameworks
were kind of junk at the time.
We're just going to do, we're going to write our own
JavaScript. And people are like, oh my
God, like you're going to write straight to the DOM?
It's like, we're a bunch of kernel programmers. Like this is
not a big, like this is, this is not
low-level systems programming. Writing to the DOM
is not low-level to the browser, but not low-level
to the system. That's right. That's right. So we
were, as a result, like we made it there.
I mean, it was, we did a lot of JavaScript and I had just not, never done anything with
JavaScript.
I had never, and I was only adopting it because it was the lingua franca of the browser.
And I began to realize that like, wait a minute, JavaScript is like a real programming language.
And this is like the same realization I think was happening everywhere in 2007.
For sure. Yeah, it was a boom.
Where people are like, hey, I kind of was blowing
this language off as like glorified blank tags
or whatever, but this is like,
we've got a real programming language here.
We can now do these actual like single page apps.
We can write a real app that kind of spreads
across the browser and the backend.
And there's a whole bunch of stuff.
And then, so I was like, this is really interesting.
And I'm really enjoying JavaScript.
Then we also extended that onto the server side.
And we were starting to, we did our CLI.
I re-implemented our CLI had started as this Perl monstrosity,
like there's any other kind of Perl.
And I realized I'm like, I'm down the wrong path.
And I need to rewrite this thing in JavaScript.
And that was really, really eye-opening for me
because we were able to re-implement this thing really cleanly in JavaScript.
And I'm like, I think this has got a lot of potential on the server side.
Node was a catalyst for that.
So I went into Joyent and Node full of enthusiasm for, for node JS in 2010,
a bit more jaundiced after a couple of years, because I had this idea that we would be able
to really bring robustness and rigor to that. There are things I love about JavaScript,
but there are things that, that where JavaScript needed best practices in order to make it rigorous. And
there was just a lot of resistance. And in particular,
and we had done a bunch of technologies, developed a bunch of things to make it
more debuggable and so on. But that was not the priority
of that broader, the broader community was a JavaScript community, not a
Node community.
And there was just some pretty deep fractures that were, that I had kind of wanted to bridge. And I began to realize like, these are not actually bridgeable. And I actually gave a talk
on this. So we, I mean, there was this fracture kind of famously, there was a fork.
IOJS.
IOJS 2014. You know, just to interject
a little bit here, I talked to Scott Hammond
when this happened. Did you?
He graced me with his presence.
I was a lowly podcaster,
green behind the ears in comparison,
maybe still green behind the ears.
This is 2015.
During the whole major shakeup. You talked to
Scott in 2015? Okay, yeah.
Scott's the best.
Yes, yeah.
So yeah.
I mean, he is a CEO.
I mean, he didn't have to give me his time,
but there was a lot of things, as you know,
happening in the Node world.
Node was very important and you had this fork
and it was forked by major players
and people within the community.
So it wasn't just this, we're upset,
let's do something different.
And there was a huge rift.
And IOJS was legitimately trying to truly fork and stay separated,
but then eventually it came back.
And Scott, we did a show, it's called The Future of Node.js.
And largely it was about this rift and the stewardship and the opportunity to guide and direct Node.js and to be good stewards.
And so that was largely what we had talked about.
And I don't know if you saw,
they just did a documentary on this.
Yes.
And then Scott was a part of it,
which I thought was great.
Actually, it was funny because he's like,
you know, I was actually,
I didn't want to tell you I was doing that
because I thought you'd try to talk me out of it.
I'm like, no, I'm not going to try to talk you out of it. I'm like, no, I'm not going to try
to talk you out of it because I think that, you know, I thought you would bring a different voice.
And he did. I thought he was great. I think the one thing that that documentary didn't really hit
on, which is not necessarily any fault of theirs, but IOJS was actually symptomatic of much deeper
fractures. And it was very easy for people to say, oh, this is like a joint versus the community fracture.
And it's like, no, no, no.
This fracture is actually a lot deeper.
And I will say I got some satisfaction in 2017 at Node Summit.
The organizer of Node Summit says, hey, I'd love for you to give a talk on Node.
This is in 2017. It's three years after the fork. And I'm like, bad idea. I got nothing nice to say.
This is not, I don't think that's a good idea. He's like, oh, no, no, no, that would be great.
If you've got nothing nice to say, that would be a great keynote. I'm like, oh, God, all right.
Fine.
So, fine. And actually, I'm like, you know what, I did the talks that I am most grateful for,
are the talks that people have asked me to give on a particular topic.
Because it forced me to kind of get out of my own comfort zone. And I'm like, okay, wait a minute,
how do I actually talk about this? I want to talk about what happened here. And I do want to do it
in a way that's actually like reflective and constructive. And when I begin to realize is
like, okay, actually, what happened here is not just it's like it's yeah you get the personalities got all that stuff
and that's kind of easy to gravitate to but there's actually something deeper there was a values
clash there was a divergence in values and once you see it that, you see beyond the kind of the joint V community aspect of it.
And so as I, in this presentation, software platform is a reflection of values, I think
is the name of the talk.
But I kind of walk people through the values that we had, that Node had for itself, and
the values that we kind of, we had Joyent and other people in the industry had
and how those values didn't match. I said that in hindsight, a rift was inevitable.
In 2014, that fracture finally happened. Does anyone remember what happened in 2014?
I'm thinking I am making this a little too obvious in this talk. Like the audience is going to realize what I'm doing and they're not going to like lunge
into this bear trap.
But it turns out the audience absolutely lunged into the bear trap and everyone, you can,
you can hear the audience say IOJS.
Like, oh, IOJS.
Like I did just now.
Like you.
IOJS also happened in 2014, but I'm talking about something that actually happened before IOJS. Like I did just now. Like you. IOJS also happened in 2014.
But I'm talking about something that actually happened before IOJS.
And I pull up TJ Hollowaychuck's farewell to the Node.js community.
So TJ Hollowaychuck was by far and away the largest Node contributor
and walked away from Node prior to the fork
for effectively all of the same kinds of issues
that we at Joyent had. And what you begin to realize is like this values divergence was
driving lots of people away. And a lot of those people got driven to go at the time. And if you look at the rise of go happens,
like that is happening just prior to that IOJS fork, as people are realizing, I like the idea
of being able to, there are things that I like here. I like the performance, but there are things
that I don't like here. I don't like how sloppy this is. I don't like how easy it is to have a typo that creates
a bug for me in production, for example. I want something more robust. And they're beginning to
gravitate towards Go. And TJ Hollowaychuck, when he walked away from Node, no one said anything
about it. I'm like, it's kind of amazing to me. This is like, you know, I don't know,
Brian Kernighan walks away from C, like no one's going to say anything about it, you know i don't know brian kernigan walks away from c like no one's gonna say anything about it you know or the you know russ cox walks away from go is anyone gonna say bjarni strostrup walks away
from c++ feels like to me like the the maybe that's that's probably a bit too far because i
mean tj did not was not a creator of node but he was very prolific and to me it was a it was a time
for reflection but people were so caught up in the kind of the personalities of the moment and the conflict of the moment and the kind of the corporate V community kind of narrative that they were losing track of the fact that there was a bigger values divergence here that was driving away a lot of folks in the community.
And I did, I was actually pretty satisfied.
The audience like gasped.
If you turn up the volume on that, you can hear people gasp.
Is that right?
Yeah.
What was the line that made them gasp?
Can you recall?
Yeah, it was just me showing the TJ Holloway, Chuck's farewell message.
Right.
And that's what I'm referring to.
And I had a couple of people come up to me afterwards being like, oh my God, I had forgotten about that.
And it was so gutting.
I remember when I read that and I was
just like gutted that TJ had left and kind of no one talked about it and everyone kind of went on
to the next thing. But you're right. Like we didn't, that there was something bigger that
was happening there. And, you know, I think some of that has been resolved by TypeScript to a
degree. And I think things have just kind of settled
in that I think that folks
that are looking for something else went elsewhere.
And that included me.
It's just like Node is not a good fit.
It's a great fit for a bunch of software in the world,
and it's not a good fit for the software that I want to write.
And it's not a great fit
when you've got load-bearing infrastructure software that's got to be right all the time that needs to run forever.
I want something else. And to me, like Go is not the right fit either. And what I realized in 2018
was that Rust is going to be the next thing for me. And Rust is the language,
the values of Rust very much aligned with my own personal values.
And, you know, I think that the whole Node experience, so I mean, it's easy to see the
echoes of Sun in Oxide, but there are arguably just as many echoes of Joint in Oxide, because
that experience was so educational for me with respect to node and being so i wanted to be much more mindful about the way that i like looked to platform and how platform reflected those values
and then the kind of technologists that you're going to attract i mean the name oxide is very
much a tip of the hat to rust because we knew that we wanted to go build in Rust. And I knew that we would be able to attract
technologists that kind of shared worldview on how things should be. And that's been even truer
than I could have possibly imagined. You know, just like the future of the company being in the
back of the cab, the future of software being in Rust, at least for the software that I want to
write, has been even truer than I could have imagined. And the technologists we've attracted to Oxide are extraordinary and very
much share that worldview and Rust being a very concrete embodiment of that expression.
Let's go to where Oxide begins, not so much as a company, but as a convictionable, I'm not sure
what the word might be, but something where you have conviction, this idea, you have enough conviction in the idea to begin to take the steps.
Whether it's a phone call, a simple text message to Steve or your co-founders or Adam Leventhal, like whomever, however it first began.
What was, take me to that moment.
What was the conviction?
What was the initial idea?
How did you get there?
Yeah, so the origin of Oxide first,
like long before we had a name or an idea,
is that I wanted to do something with Steve.
So I actually, Steve and I had been together at Joyent.
I went to Joyent.
I loved the technical positioning,
but wasn't necessarily in love with the folks
that were running the company at the time.
And I didn't meet Steve until after I came to Joyent.
I met Steve on my second or third day at Joyent.
And it was, I remember exactly where I was when we had that first conversation.
We're just like, wait a minute, who is this guy?
This is great.
And Steve and I had a great run together at Joint. Ultimately, we both
partnered with Scott, we hired Scott, and Steve and I both learned a lot from Scott.
And after the acquisition of Joint by Samsung, as we were beginning to realize that, okay,
Samsung is not where I'm going to spend the rest of my career, and what's next for me,
Steve and I wanted to go do something together. And, you know, the environment,
such as it is at Samsung,
there was always a little bit of a concern
that one of us would get axed
before it was time to go start a company.
And the kind of like the unstated belief there was,
it would be Steve who'd be axed.
Just because Steve was running,
he was running our go-to-market on the commercial side for Joy-In. And it just felt like that's the thing, that's a
function that Samsung might get rid of. He just didn't know. And I remember we're at coffee
together. And it was just really important to me that Steve and I do something together. I had no idea what it was at this point.
I'm trying to make, and he and I are both concerned
reasonably that Steve will be on the street.
I'm like, I need you to go consult for a year or something.
I don't know. I need you to wait for me, basically, because
we were all going to be at Samsung for three years, kind of the term of the deal and so on.
So as we're saying this, again, with this kind of implicit assumption that, like, that if anyone is going to be axed, it's going to be Steve.
I get a text message from Scott, my boss, saying, you've got to get back here right now.
Like, the folks at Samsung want to talk to you.
I've got no idea what this is about.
And Steve's like, maybe you're the one that's going to get axed.
And I'm like, Jesus, maybe I am going to.
And as I remember, I had a very weird conversation.
As I'm going up the elevator thinking, we're kind of laughing about it.
And I sit down with the folks from Samsung.
And Scott's in the room. And he says, you know,
the CEO, pointing to Scott, the CEO manages HR, they manage marketing, they manage sales.
But what does the CTO do? Pointing to me, and I'm like, I am getting fired in the next sentence.
I am just like, I am completely convinced that I'm getting fired in the next sentence. Like, I am just like, I am completely convinced that I'm getting fired in the next sentence.
And I just remember thinking like, no, like I can't, this is going to really, I want to live.
I haven't done, like we, but I was not getting, I was not getting fired.
It was fine.
It was just very, it was the one in many, many, many, very strange conversations.
What did you say? You can't go on with this story without telling me what strange uh conversations what did you say whether i
didn't go on with this story without telling what you said what you how did you respond
oh well i didn't it was a rhetorical question i was like okay i'm like do you actually want me
to enumerate what i do here no they this is yeah this is very they said well um
you well they actually said samsung Samsung has no Nobel prize winners.
That was the next sentence.
And I'm like,
do you want me?
I actually did say,
do you want me to pause it?
Why that might be,
am I supposed to like expand on that?
Like why that might be the case.
And,
um,
you have done Detroit's,
they said to me,
and I'm like,
where are we?
And what is going on? I'm so confused. Where, where, like, I do not say to me. And I'm like, where are we and what is going on?
I am so confused.
Where, like, I do not know what's happening.
They're like, we want you to do Cloud DTrace.
And I'm like, what the hell is Cloud DTrace?
I'm like, okay, fine.
And like, when do you think it will be done?
I'm like, when will what be done?
The thing that you just invented in the last sentence,
like, I don't, I mean, it actually was an interesting opportunity to be like, okay,
well, actually, let me walk you through the full history of D-Trace, where the original ideation
for that was in 1995. And it wasn't really done until 2005. So 10 years.
A decade. So if you want me to put a schedule estimate on the thing
you just invented in the last sentence, a decade, next question. I didn't say it exactly that way,
but captures a lot. So there's a lot of that's endearing about Samsung and a lot that's,
that's frustrating about Samsung. But in that whole experience, it was just a, for me, I remember it was a very emotional reminder of how much like, no, I want my next thing to be with Steve.
So we knew we wanted to do something together.
And it was just, we were contemplating bad ideas, basically.
And we had a friend of mine who had been a early CTO when it
had become a prominent public company and was, you know, employee number four or whatever it was.
And he was very good, asks very thoughtful questions. And he's like, why, why do you want
to do this? Like, what do you mean? Why do you want to start a company? Starting a company is such a,
you know, it takes so much emotional energy.
Like, what is it that you want to create?
I think it's a really good question.
Yeah.
You know, I mean, because I think it's
when you think about like,
what am I trying to do?
And there are a bunch of answers to that.
Like, you know, I want to create things
that are valuable to people
and I want to create a sustainable business.
And I want to, you know,
there are a bunch of things that I wanted to go do.
But as I began to kind of like think about this in terms of like prioritizing them and
thinking about in particular, when have I been happiest in my career, which I think
is a really important question to ask people.
We ask people, this is part of the Oxide application process.
And it's very revealing to ask people, when have you been happiest in your career and
why? And as I was reflecting then, I'm ask people when have you been happiest in your career and why and as i was reflecting then i'm like when have i been happiest and the moments that i've
been happiest were you know we talked about those moments in the early 2000s with the cfs and
detrace and this like exhilarating sense of creation from this team of folks from not just
it was not any one of us alone.
It was Matt and it was Jeff and it was Adam
and it was Shrock and it was, you know,
all these people together,
working together to create something bigger than themselves.
And when Adam, Mike, and I did D-Trace,
I remember so many times thinking,
this is so much bigger than any one of us.
It requires all three of us at the absolute limit
to create. When at Joyent, there were a bunch of times when coming together as a team, whether it
was Manta or whether it was working on Triton, a bunch of technologies where we felt like we were
creating something beyond ourselves. It's like, that's what I wanted. And I'm like, I want a team coming together to create something bigger than themselves.
Like, that's actually what I'm happiest. And the other stuff is still important. But like,
and it was a really helpful to kind of have that at the back of my mind of like, whatever it is, it needs to be big enough
to attract that kind of team.
I knew that it needed to be deep enough
to be able to hold my own attention for a decade.
And when Steve and I were kicking around ideas,
they were all kind of small ball ideas
that weren't going to necessarily have that staying power.
So he and I are both rejecting idea after idea after idea. didn't really have that, that weren't going to necessarily have that staying power. So we,
he and I are both like, rejecting idea after idea. And it was really when we, again, we knew we wanted to start something didn't know what it was going to be. I'm like, well, look, I should start
mailing venture capitalists at least. And like, we should at least start having these conversations.
And I, you know, I talked to VCs throughout my career, but I had never started anything. And so I actually went back to the email of a pretty prominent VC who I used to meet with reasonably regularly.
Like, you know, every year we would get lunch or something.
And I wanted to just remind myself of his email address.
And the email he had sent me is like, hey, love getting lunch with you today.
Just want to remind you, I will fund
literally anything you put in front of me. I remember thinking like, okay, this is maybe
the catalyst that I need to go big. And I really think it's important with new company formation
that you got to go big. You got to go big. And I think that, and certainly for the kind of company
we wanted to go build. And I'm that, and certainly for the kind of company we
wanted to go build. And I'm like, you know, Steve, and this is at a moment, wait, we are at Samsung
just suffering from the problems of deploying at scale. And in particular, we've got a bunch
of Dell machines that are dying with these uncorrectable memory errors. Dell is not, the whole experience is awful.
And the Samsung is extremely upset.
Dell is extremely useless.
Dell is like, this has been escalated
to like Michael Dell's level.
So this is like one of these things where you're like,
Dell's solution to the problem
is to put more and more people on the call.
So Dell will be like,
okay, we've got like 35 people on the call.
So let's go around and do intros. And I'm like, no, no, no. Gosh, no, we'll be here all day. No, I exactly. I don't
want, I'm not here to like to run out the clock. So I don't want to hear an introduction from
anyone that's not going to solve my problem. And everyone's like, okay. And what I began to realize
was Dell themselves did not understand the origin of this problem.
They were trying to tell us, like, oh, we've never seen this before,
which is like, oh my God, this is just such garbage.
It's kind of gaslighting from Dell that I'm the only guy.
Dell was spending their time trying to find something else they could blame.
And for, again, for a long time, I thought this is like,
this is out of laziness. And what I began to realize time i thought this is like this is because out of laziness and what i
began to realize for this particular problem is like oh no wait a minute they don't actually
understand how their own systems are built they actually don't get they they themselves they're
doing this not out of laziness this is out of fear they don't know how this system works and
they don't understand how it fails and i were some comical moments. There was one moment where they were like,
what we need you to do is we need you to
reflash the ME, the Intel Management Engine,
which is this off-chip part that is basically the brainstem of an Intel
machine. I'm like, what do you mean? Do you think this is an
ME issue? They're like, no,
no, we need you to take the same image that you've already flashed on there and reflash it.
I'm like, do you think this is like program text corruption? And they're like, well,
we can't actually tell you any other details because our NDA with Intel prohibits us from
going into any other details. I'm like, what are you talking about? And this is post-Spectre meltdown,
when Intel, to their credit,
was, I mean, we had a very close
relationship with Intel after Spectre meltdown.
We found out about Spectre meltdown on the
internet, so Intel was a bit apologetic about that.
I'm like, I'm just going to call up Intel right now
and I'll, like, let's get, we'll get Intel
on the call. And, like, we've got our folks
at Intel, and, like,
if this is an Intel ME,
because I'm actually thinking, I'm making the mistake
of taking Dell with a word,
I'm like, oh my god, there is
a very serious ME
issue that
Dell has read in on that I don't know about
that's affecting all these servers.
Let's get Intel in here right now.
And they're like, oh, actually, no, no, don't do that. It's not
Intel. I'm like, what? They're like, no, no, don't do that. It's not Intel. I'm like, what?
They're like, no, no, no, we don't think it's Intel.
I'm like, so you're just making stuff up.
You're just making stuff up.
And again, it's like, I was kind of at this moment
where it's like, this is not out of laziness.
This is out of like, they don't know what's going on.
They actually don't know what's going on.
They don't know what's going on.
They are not doing first principles engineering.
And when they say, you're the only customer that's seeing this, it's like, say, you're the only customer that's seeing this,
it's like, yeah, I am the only customer that's seeing this
because I'm actually the other customers that are running at scale
that are at Google, at Meta, at Amazon.
They're not Dell customers.
You're right.
They're not seeing this because they're not your customers.
That was all of these things together.
The mail back from the VC saying that they would fund literally anything we put in front of them the desire to have a a large invigorated team doing something that beyond themselves the desire
to work with steve the very clear problem we had in front of us it's like all this kind of came
together i'm like we got to start a computer company. That's what we got to go do.
And I remember sitting in Steve's office being like,
we got to start a computer company. And Steve's like, I mean, yes, but like,
you know, what's required to do that. No, no, no, no, no, no, no, no, no,
none of that. I mean, he may say now that like, God, I should have, but no,
no. He was like, do you think we, like, can we?
Like, are we allowed to?
Do you think we can get that funded?
Oh, like permission in a way?
Yeah, like permission.
Exactly.
Do you think like, can we?
And I'm like, yeah.
I mean, the thing is like, I'm like, Steve, that's what's in our heart.
And it is.
Yeah.
Like that is like, if you look at like our, and it just felt so right because that's like,
that is what our entire careers have been pointed to with Steve having come up at Dell
and having been at Dell for 10 years.
And I had been at Sun.
It's like, that's what it was pointed towards.
And, you know, when you've got that level of conviction, then of course, you know, as we began to do our own homework, we began to realize that
like lots of people in the industry had come to the same conclusion. But we, and we talked to
actually plenty of people who were like, great idea. You're never going to get it funded.
And what we discovered is that it's a challenge to get it funded. It's definitely contrarian,
but in venture capital, you want to be contrarian and right. Those are the big wins.
Yeah, those are the big wins.
The big, big wins are when you are contrarian and right. And we got truly blessed to fund Eclipse Ventures. And we felt very lucky to find the team at Eclipse and very much shared our worldview.
And what I learned later, you know, it's very interesting because we did this kind of raise.
And they tell you when you're a, you know, you're an entrepreneur, you want to kind of start, you want to move everyone through the process at the same pace.
Because you want to have every entrepreneur's fantasies.
You've got different VC firms that are fighting over a term sheet.
Great. What a terrific fantasy.
In practice, you don't control the timeline at which VCs move. And in particular, we found is that VC firms varied wildly in
their appreciation of this problem space. So what we would have happen over and over and over again
is we would hit a firm where one partner would really understand it, but they would struggle. The rest of their partnership wouldn't understand it
or didn't see, you know,
I've had one VC firm
that reached out to me.
They're like, hey, you know,
we hear you're raising
and we want you to come in
and pitch the firm.
And I'm like, terrible idea.
You guys are a terrible fit.
You're, you don't know
because you're-
You just tell that like point blank.
Oh yeah, yeah, absolutely.
But your response via email is like,
no, it's a terrible fit.
No, terrible fit.
No.
You got to be honest in those situations though, right?
Like you can't waste people's time or your time.
A hundred percent.
A hundred percent.
So I'm like, we're a bad fit.
Like we are, this is hard tech.
This is like, you're not a hard tech firm.
We're doing hardware software cloud design.
You know, you are,
every one of your portfolio companies
is on a public cloud.
Like,
you're a bad fit.
But this is like total shades
of HBO Silicon Valley,
right?
In the sand tail shuffle.
Yeah.
That like,
you tell them these.
Negging you and stuff.
Were they negging you?
You were negging them?
Oh,
yeah.
So like,
you tell these.
Or was it fake users?
Which one was it?
Exactly.
No one was negging them,
right?
Is it the,
and obviously I'm not using this deliberately,
but when I'm saying like, look, you're a terrible fit,
they're like, oh, we're now, we're very interested.
I'm like, no, it's like, no.
We're more interested because we're not a good fit.
Exactly.
And so, and actually with this particular firm,
I remember telling them that the,
so in particular, like on our first slide,
it's like Jeff Bezos at the time, now Andy Jassy,
this is the company.
Jeff Bezos is not going to own and operate every computer on the planet.
Not every server is going to be a public cloud machine.
There is going to be on-premises computing.
To folks that run on-prem, whether that is for economic reasons, as Samsung had to, whether that is for risk management or security reasons, as
certainly you see in federal spaces, government spaces, and so on, whether that is for latency
reasons, as you get people kind of co-locating near their data and so on.
There are lots of reasons to run on-prem.
And they're like, okay, yes.
But then you get to the next slide, and their first objection is like, but won't it all
go to the public cloud?
I'm like, okay, this is why we shouldn't be here because you only see public cloud computing.
We are very pro-cloud.
I love cloud. I think it's great.
I think for startups, public cloud is perfect
because when you are starting from zero,
of course you should not buy your own hardware.
Go spin it up in the public cloud. That's awesome.
But you get to a certain scale
or you have certain constraints,
you will need to own and operate
your own computer in your own
data center or co-located data center.
They just don't see that.
They don't see that world.
Every time I'd be like,
you know what, this is why we should leave.
Kind of like, we should leave. No, no, no, no, no, no, no, stay, stay, stay, stay.
No, no, we're interested. You're making us more interested. But you're like, oh, God, what are we
doing? Why are we doing this? And the real
problem is that they would kind of convince us to stay. We would stay. We shouldn't. We should just
walk out. At one point, I did tell that particular firm, I'm like,
you have to understand, we are going whaling.
You should be fishing from the pier. You should not come whaling with, it's like,
go like, go stay here. You don't want to come out on this whaling vessel with us. You do not want
to be on this, on this multi-year dangerous voyage. But again, that did nothing. And of
course, they spent a ton of time and then ultimately
they realized
like, I think we should fish from the pier.
It's like, yes, you should fish from the pier.
Thank you for the education.
Yeah, exactly.
That is hilarious. Yeah.
Negging though.
That is a lot of what happened
in obviously the show that we keep
mentioning and I'm resisting going deeper.
But you live this stuff, man.
That's crazy that not only you're a fan of the show, but you literally live it.
I mean, I live it too, but not quite the same way that you live it.
You've lived it from the halt and catch fire kind of ideas to truly Silicon Valley satire TV show ideas.
You've been in the room where you neg you neg yourself. Like that's crazy.
Oh, it's great. I mean, I, what a blessing, right?
Oh, I think it's great. I love it. Actually. I,
and I just think it's extraordinary because it's so well written and so funny
and it's actually a way that I can certainly with my, you know,
my teenagers, I can actually kind of pause it and
explain like, no, this is like a real point that they're raising here. Right. And this is like a
real issue that, you know, we actually have seen this kind of issue. It kind of explained how,
because it's so well-researched. It's really, it's really well done.
It is. Well, even early in the show, I think I almost mentioned it, but I'll kind of mention it briefly here is whenever you were talking about Sun and the way Oracle bought it and the acquisition process, it made me think of when Jared had said in episode one, you take an application like this and make a business facing like that literal line there is like was like Oracle's somebody inside of Oracle said that line to somebody and said we should acquire Sun.
You take an application like Sun and all they're doing over there and you make it enterprise facing and business facing.
And then you see what happened.
Like they have no idea what they built, which is kind of what Richard Hendricks dealt with, which is he built this music search app that had super fast compression and he was
just trying to build this music app he wasn't trying to build the compression algorithm but
he kind of stumbled on you know what we had become in the show middle out and all this fun stuff and
like he was largely unaware of the true diamond in the rough that he created and that was hooli
trying to give him the money and then the whole entire show spun off from that day one, basically.
Yeah, totally.
I think in terms of actually what motivated Oracle with respect to Sun, I think it actually is Java for a difference worth.
I think that what they saw was the opportunity to go really exploit Java and use Java to coerce rents out of us. I mean, in many ways, Oracle v. Google
was very predictable. I think we've kind of forgotten how important that battle was.
And I guess, I mean, I guess you could argue that it kind of ended in a stalemate. But Oracle,
I think, wanted to acquire Sun because of its ability to go extract monopoly around side of Java, which
is a little bit ridiculous. I mean, I think it's probably accelerated the adoption of alternatives
for sure. Well, I sidecarred us there for a second there, but Oxide is very much an ambitious play.
How many years are you into this? Is it three years? Four. Four years. Four years in, long way to go.
You know, this is kind of always the belief is that we, the ambition here is pretty,
is substantial. What we're really trying to make up for a, you know, multi-decade rift
between the kind of computers that people could buy
and what the hyperscalers were making for themselves.
And what we wanted to be able to do
is really develop the RackScale machine
that includes both the hardware and the software
required to create and manage elastic infrastructure on-prem,
cloud computing effectively on-prem.
And there's a lot there.
There's a lot of surface area there
because it's not just, you know,
it's virtualization, it's compute virtualization,
of course, but it's not just compute, right?
It's storage virtualization, it's network virtualization.
We've done our own hypervisor.
We've done our own host operating system.
We've done our own embedded operating system
for the service processor.
We've done our own switch. We've done all of our own software there. We've done our own,
we talked about Crucible earlier. We obviously using CFS, we've done our own. So we've done
all of that stuff. We've opened it all up, which has been fun. So it's all open source.
Then our own control plane, our own distributed system. And we're not trying to, wherever possible,
we will take components off the shelf.
We're not trying to reinvent everything
for its own sake, certainly.
But we believe very strongly
that we need to take full responsibility
for the system.
And what we will never do
is what Dell
had done to us so many times,
is effectively blame the customer for their problems.
And when you've got an oxide rack
and if you've got a stock provision
or if you've got something that's not working
or not performing or what have you,
it's like, we want that to be an oxide problem.
And even if the root cause is not an oxide issue,
we want to be able to fully understand
all aspects of that problem.
So owning every aspect of our stack has allowed us to do that.
And it's extraordinary that we've got this thing working and we got it shipped.
We got it passing regulatory compliance.
We got it running in production now in customer data centers, which is really exciting.
But there remains a long way to go.
So this is not, the journey is nowhere near at its end for sure.
Well, I think you came from a good place really with,
at least based on your story,
with this animosity with your personal experience with Dell, right?
Not knowing the system, you don't want to build everything.
I'm assuming I'm paraphrasing to some degree.
You don't want to build everything, but you do want'm paraphrasing to some degree. You don't want to build everything,
but you do want to,
if you have to,
because you want to know your system so that whenever the future Samsung's
that the future Brian's work at come to you with a problem,
you don't tell them to reflash the EM.
That's exactly right.
Yes.
You have a different answer to get to the customer's problem and get to root
cause.
And I think they couldn't, based on what you shared with your story and your history and your experience, was that Dell could not own the problem.
Because they could not realize that they were the problem in the way they treat their systems and first principles, actions, etc.
You have very much rebuilt what could be based on first principles so that you
don't have that same scenarios in the future.
So you're kind of like protecting the future Bryans and whomever else's that
might do what you've done from having that same experience.
And that's exactly right.
And actually,
I think the other bet probably is,
is the on-prem like not so much that cloud is bad,
but public cloud is not the only way.
And we're seeing
permutations that right now in the marketplace you got that with david hanemar hansen dhh famously
with rails and ruby on rails pushing back hard ejecting from the cloud now that's one path
and probably way more that you're personally familiar with but that's what i'm most familiar
with is there's a lot of pushback on on that and i actually have a friend and we talked you know loosely in dms about this like i have a friend
that works for aircot here in texas and he's like we only do on-prem we don't have any public cloud
stuff we only do on-prem we only do rack and stack our own thing and we only have our own colo
so they're like that's your future customer you're welcome kind of thing absolutely i mean
you did the work i'm just like just hey do you know about oxide okay cool check right and i sent
the url like right away via via text i was like check them out all that good stuff but like you
had said before there's a lot of organizations out there that just the public cloud will never work
for and the lack of visibility into all the permutations of the system, whether it's storage or computer, whatever,
that invisibility with other systems like Dell, like you'd mentioned, that's going to be a challenge for those people.
And it is a challenge because they're working around it with different systems like Kubernetes or whatever to get observability and at scale and all the resilience you want in a stack.
That's exactly right. No. That's exactly right.
No, that's exactly right.
And I think that, you know, and I, Kubernetes is obviously great.
And, you know, we actually just got back from Red Hat Summit, which is really exhilarating.
I saw you, man.
I was like, man, there's, there's you next to that big old rack.
Oh, I loved it.
I was, I was watching some videos on LinkedIn, so I appreciate the coverage there.
Oh, it was great.
And the, a lot of supporters of Oxide inside Red Hat,
which is really exhilarating.
And the open shift on top of Oxide
makes a lot of sense, right?
And I think that so if you look at Kubernetes,
it does a great job of kind of solving that problem
from the VM up.
And it's from the VM down that's been the issue.
And there's been a lot of challenges with respect to integration.
Again, the whole thing goes sideways. If you're running on-prem, the whole thing goes sideways
and everyone's pointing fingers at one another.
Ultimately, as the person deploying the infrastructure, you feel like
you're trying to manage divorce proceedings.
There was a great tweet.
I feel like I'm dealing with a support issue with Dell and VMware.
I feel like I'm dealing with my divorced parents.
I think that for end users, that's a really frustrating feeling.
And I think we're excited that we're able to help people deliver high-quality, robust infrastructure.
And we can help them delight their customers
and help them, because again, it's like,
we're not the thing.
We're the thing that gets you to the thing.
We are, our job is to enable other people
to do great work with this incredible device,
the computer, and enable one another
to really innovate and so on. But it's like infrastructure should be
out of the way to allow you to do that. You want that infrastructure to be... It just is like,
honestly, AWS, how much innovation has the public cloud unlocked? A lot, right? I mean,
the public cloud has been really important for a lot of new company formation. And I think that a lot of that same kind of pace of innovation has arguably been deprived
of folks on-prem.
So we're really excited to be able to extend that. What's up, friends?
This episode is brought to you by our friends at Neon.
Managed serverless Postgres is exciting.
We're excited.
We think it's the future.
And I'm here with Nikita Shamganov, co-founder and CEO of Neon. Managed serverless Postgres is exciting. We're excited. We think it's the future.
And I'm here with Nikita Shamganov, co-founder and CEO of Neon.
So Nikita, what is it like to be building the future?
Well, I have a flurry of feelings about it.
Coming from the fact that I have been at it for a while,
there's more confidence in terms of what the North Star is.
And there is a lot more excitement because I truly believe that this is what's
going to be the future. And that future needs to be built. And it's very exciting to build the
future. And I think this is an opportunity for this moment in time. We have just the technology
for it. And the urgency is required to be able to seize on that opportunity.
So we're obviously pretty excited about Neon and Postgres and Managed Postgres and Serverless
Postgres and data branching and all the fun stuff.
And it's one thing to be building for the future.
And it's another to actually have the response from the community.
What's been going on?
What's the reaction like?
We are lately onboarding close to 2,500 databases a day.
That's more than one database a minute of somebody in the world coming to Nian either
directly or through the help of our partners.
And they're able to experience what it feels like to program against database that looks
like a URL and then program against database that can support branching and be like a good
buddy for you in the software development lifecycle.
So that's exciting.
And while that's exciting,
the urgency at Neon is currently is unparalleled.
There you go.
If you want to experience the future,
go to neon.tech, on-demand scalability,
bottomless storage, database branching,
everything you want for the Postgres of the future.
Once again, Neon.Tech. Aside from Oxide, has there been or has there been any particular push for being very on-prem friendly?
Like you're building the full stack, the full rack, you know, hardware and software marriage.
What Apple has done in a lot of ways for the Macintosh where you marry this beautiful end-on product.
In a lot of cases, there's beauty in everything that Apple has really made,
not just literally beauty in its physical form,
but the marriage of great hardware paired with great software
has unlocked a lot of creativity for creators.
I'm on a Macintosh right now.
I assume you probably might be as well.
I don't know.
Maybe the year of Linux desktop is finally here for you.
Who knows?
I'm on a framework.
So similar though.
Framework, frameworks that were in spirit.
Gotcha.
So, yeah, I'm just thinking like, who else is for on-prem?
Is there anyone else doing any version of what you're doing?
Oh, in terms of other companies?
No.
I mean, no. And I think I kind of assumed that there, I mean, there have been companies that have taken swings at it.
But part, and we, when we were raising, we went into all of them in detail.
And so there are so many companies that you have not heard of that I dug up the graves for, talked to the entrepreneurs, talked to the VCs,
like what went wrong?
And a bunch of them had hit on kind of the need
for the same thing that Oxide had done.
The challenge is that they didn't go big enough early enough.
So by the time they realized the actual true enormity
of the problem in front of them, they were really low on
capital. And from all of that, I realized that you really need to take, it's very technical,
and it's very big, and you can't underestimate it. You've got to be really clear-eyed about how big
this thing is. And you've got to take the right approach to the market. You can't try to take
a shortcut here.
You said earlier about first principles. First principles
are very important. You've got to do it
from first principles.
And when you're
taking venture money,
it takes the right kind of investor to not take shortcuts.
Well, yeah, because they want to return, right?
They want to return.
That's the whole point. For the most part. I mean,
the right venture capitalists will be in it for the long haul because they see the long haul journey and they
understand the road required to get there. But in the end, the goal really is
return. Return is the goal. And also,
if you're uncomfortable with that goal, do not take venture money
because that is their goal.
They have responsibility to their LPs.
And it takes the right kind of investor to not take a shortcut.
And it takes the right kind of investor to like...
And as a company, you're constantly trying to figure out,
what is the most expedient path we can take that is not going to result in work
that is throwaway or going the wrong direction. So we were constantly doing that for the lifetime
of the company. We are constantly trying to figure out like, what can we do that is a,
that gets us towards the problem is the minimum viable product is very large.
So I think, you know, I had kind of assumed when we raised that another company would see that we had raised and realize that, oh, you actually can raise for this.
And I assumed that we would have another startup that would raise behind us.
But it turns out it was a little harder to raise than I thought it was going to be.
Obviously, we raised and found the right investor.
But I also really realized that it's going to be really hard for a company coming behind us.
And I want to make it for anyone who would have that kind of desire, anyone who's got
the kind of entrepreneurial spirit who would want to raise behind us.
I would want to make sure that coming to Oxide is a much better approach for them, that they
would much rather just like, actually, I'd rather just join Oxide.
So we really tried to make ourselves appealing
to anyone who had that kind of, that desire.
And I think we've done that.
I think we've got a lot of people at Oxide
who have that, who certainly share our mission
and have that kind of that mindset
of creating something new.
And as a result, like, you know,
I think through a combination of factors,
there's not really another startup raising behind us.
And which, you know, there are some investors
who are just like, well, this is such a great idea.
Like, why are you the only one?
Because it takes too much.
It's hard.
You have so much, I mean, to speak for you,
hearing some of your history and then reading your resume, you have so much history in what has gotten you particularly here today.
Sun, D-Trace, the sheer desire to build the best system, the morals and desire to build the best thing, not so much just the best, but the thing that you understand how the system works.
Observability was your original concern into the system. And that's
probably from just a psychological perspective is also your
same demeanor currently. It is. Observability into
the thing you're building with the right kind of people. Your conviction for rust.
Just being on the call with 35 people asking you to do intros
when all you want to do is solve your problem.
Like, piss me off enough, and I'm just going to create a company around this thing, okay?
Right?
That's how I look at your journey based on what I've heard to get to here.
And compete with that?
No, thank you.
No one has the acumen and the history that you have to be where you're at today enough to to not want to just
join like you had said you're kind of like uh the positive side the positive effects i guess of a
black hole with gravity you can't get away from it oxide does visually beautiful technically
beautiful uh organizationally beautiful even from your principles, beautiful. You've created a beautiful thing to not want to compete with
and in a lot of cases to go on-prem.
So you're in a great position from what I can tell.
Yeah, thank you for saying that.
That's really been our intent.
And it all goes back to that question that a friend of mine asked me
in terms of what made me happiest.
How do you gather? All of that is in the spirit of gathering that team.
And we've done some things along the way that have been really surprising and that I would not have anticipated.
And, you know, as we have formed the team, gathered the team, built the team, the team's reinforced itself. And it's just absolutely extraordinary and so exhilarating
to watch people create not just for the customers that we want to serve, but create for one another.
And really being inspired by one another is just really singularly satisfying to watch.
Are you by nature individually or or corporately, contrarian?
That's a great question.
I don't feel that way.
But I also think that there may be some people who know me well
who would feel it's ridiculous that I would not say that I'm contrarian
because there have been so many things where I have thought differently than many people.
And I think I'm unafraid to go my own way.
I'm unafraid of my own conviction.
And I think that that conviction doesn't necessarily need to be contrarian. Like I'm not, I'm definitely
not uncomfortable when I agree with a lot of people. Like that's fine with me. You know,
I actually do. I think there are, you do get folks that are like so contrarian that like,
no, I'm like uncomfortable if I'm doing the same as everyone else. And I don't feel that way,
but I definitely have the strength of conviction. And I think the older that I've gotten, the more that the back of a cab that I feel kind of in hindsight, you're kind of like, you're a bit over your skis.
You've got no basis for feeling that strong a conviction.
But that strength of conviction is really important.
You get kind of this point in like, I think mid-career where I began to feel that maybe
I was too strong in some of these convictions and beginning to be, and some of that amenability
can be really good. But in doing some of that,
I've also, I kind of rediscovered the importance of some of those deeply held convictions. And I
would say that like the experience with No J.S. Adjoint is an example of that. We're
kind of rediscovering like, no, no, like this is why, and kind of finding different ways to
phrase that conviction too, where it's not just like, okay, this is not just like in my gut,
let's actually take this apart a little bit. And being very explicit about values is a great
example of that, where it's really important. At Oxide, we are very explicit about our values.
And we differentiate that from the principles. And that's been really, really important for
the company because it has given us a lexicon by which we are able to kind of express what is in our kind of collective gut because we do have shared values.
But then especially as those diverge a little bit, being able to kind of express that divergence.
So I do think I've got – I don't think I'm contrarian, but I think that I have the courage of my conviction.
If that's, and I've also been right, and maybe this is a little bit dangerous, but I've also
been right plenty of times when lots of people thought I was wrong.
And I think the reason I say that that's dangerous is because I think that you do see this in folks that if this happens to them enough times where they've been right when everyone else has been wrong, they do become just like abjectly contrarian.
And they start to ignore preponderance of evidence because they're so used to be like, well, I've been sorry, I've been right before when all the evidence has been wrong.
And so now I'm going to be. Yeah. Anti-data, anti-evidence, whatever it might be.
Anti-data, anti-evidence. And the, and I, you know, I think, I think there's a whole generation
of entrepreneurs that you, you, you see this in that, that is really, it's dangerous. And so I,
I would like to believe that I am nowhere near falling into that, where I'm always willing to listen to evidence and the data, but I'm also willing to have the courage for my convictions for sure.
I guess zooming out, maybe big picture to some degree, you're four years in, you said you have a long road to go, you've taken a long road to get here, and you're not done, which means you're not, maybe you're not exactly at product market fit
or what they might call product market fit.
Oh.
That's not my question, but I'm curious about that piece there.
No, I think we're, no, I think we are, we're a product market fit.
I just think that there's like.
You're getting close, right?
Yeah, I just think.
The idea's there.
For sure.
And I just feel that there's so much more to go do to execute on the vision. Just like,
I mean, you look at AWS, you know, where AWS was 2010, whatever, 2014, you know, it's like,
there was so much more to go flesh out to execute on their vision. And it's not, I think that we've
got product market fit. It's just that there's so much more we can and will go do that because
this rift,
because no one else has addressed this rift.
There are now a bunch of things that we uniquely can go do that no one
else has been able to go do.
And that's the stuff that is like super exciting that when you go think
about,
like now we can actually begin to think
really truly holistically about the system and we can begin to look at you know do things like
true holistic power management and be able to correlate like software workload to the actual
like to the co2 that it emits you know which we've never been able to really do before
there's a whole bunch of stuff that we can go do, and we guess that gets that observability that's deep in the marrow.
And I feel that there are years and years and years and years and years worth of work to go do to fully execute on that vision.
Okay.
So given that, that was not exactly my question.
I wanted to understand product market fit and what your position was on that currently.
If you're successful, I imagine
you're a version of success today, so I'm not going to say
you're not successful by saying if successful.
If you're truly successful
to where you're trying to go,
what will happen to
on-prem,
to compute? How will
the market be divided when it comes to
on-prem, private cloud to public cloud? What will the market be divided when it comes to on-prem private cloud to public cloud?
What will happen? That's a great question. So I think that public cloud, I think, will always be
important. I think that it will just as like, we will always have hotels, right? I think public
cloud will be important. I think that the presence of a viable on-prem option that gives you the same developer velocity, for lack of a better word. I'm not a huge fan of that term, because it makes a developer sound like a projectile, but I'm going to deploy infrastructure quickly.
If the on-prem user has that same power,
I think you are going to have different people making different decisions.
I think you will have people
who grew up on the public cloud,
who grow to a certain scale in the public cloud,
and then realize, actually,
I want to control my own fate.
I want to get on-prem.
Economically, I want to get on-prem.
I think you will get people who want to do... Actually, I don't want this to be OpEx. I want this to on-prem. Economically, I want to get on-prem. I think you will get people who
want to do that. Actually, I don't want this to be OpEx. I want this to be CapEx,
which is reasonable, right? I mean, if you look at, we all know Moore's law is slowing down,
right? Everybody knows that. What does that actually mean though, if Moore's law is slowing
down? That means that the life of a computer should be longer, right?
And if the life of a computer is longer, that means that there's more of a reason to own that
asset if you know you're going to use it for the lifetime. I mean, and not that long ago,
Amazon changed their depreciation schedules, right? This is one of these little details
that's happening in the financial disclosures
of a bunch of companies.
They are changing their depreciation schedules.
This is financial engineering.
This is making their numbers look better.
But let me tell you what they're not doing
when they're extending those depreciation schedules.
They're not cutting prices.
Right, they're making more.
They're making more.
And if you are going to run compute
for years at scale, you should own it.
And I think that you're going to have, I believe, that in this arbitrary future,
the choice between on-prem and cloud computing should not be the choice between modernity and antiquity the way it is today. It should not be the choice between a product and a DIY mess the way it is today.
It should not be the choice between, I've got actually a vendor who is responsive to me,
be it Amazon or GCP or whatever, versus this finger-pointing exercise I have between.
It's like, that should not be the choice.
The choice should be purely around economics,
around risk management, around latency.
And I think that for a bunch of people,
they will remain on-prem, they will choose to be on-prem.
And then I think you're also gonna have people who are gonna always have to be on-prem.
And I think in many ways, like the thing that is most satisfying in Oxide is bringing those folks maternity.
Because they've got no choice.
They've got to be on-prem.
I'm a naval vessel at sea, right?
It has to be on-prem.
Sorry.
We're not using the public cloud on an aircraft carrier.
And it's like, but what is that using for its infrastructure?
And shouldn't that infrastructure also be elastic infrastructure? And I think that that's where you're allowing people to do things that they fundamentally couldn't do before.
So I think in terms of how it settles out, I think it'll be interesting.
I mean, I think that it's hard to imagine that public cloud will be any less than a third of compute, maybe even two thirds of compute kind of somewhere in there. It's hard to imagine that
on-prem is going to be less than a third of all compute. So I think it's going to be kind of a
third, I would say a third, very strict on-prem, a third probably very
strict public cloud, because it just makes sense. And then you've got a third that will be kind of
swinging a little bit in the middle, but in a world where people are not having to choose
between modernity and antiquity. They're able to get that modernity wherever they are.
How do you quantify, I suppose, the total addressable market, like the TAM
of this? Not that you'll capture it all, but how do you address
I mean, obviously, any company is like the TAM, right?
Anybody who ever has bought public cloud is in the TAM.
Anybody who's ever done anything on-prem at any scale is some version of TAM.
But at some point, you look at who you can truly capture.
What is the capturable in the next,
I don't know, year to five years?
It quantified in dollars to some degree.
Yeah, all right.
Are we looking to, should I get out my deck?
Are we looking, are we raising here?
All right, there you go.
Off the cuff, you know, no true numbers here.
No, I mean, I think that we, I mean, if you could,
and this is where like, you know,
the TAM numbers are always, the TAM is, I mean, you're talking we, I mean, if you could, and this is where like the TAM numbers are always,
the TAM is, I mean, you're talking.
For sure, yeah, that's the fictitious one.
I want the closer to real.
150 billion easily, and you've got to get a fraction of that.
I mean, I think that we have got a,
I'm less focused on that and more what our,
in terms of like what that total market is,
because we know the total market is extremely large.
So to a certain degree, it like doesn't hugely matter
because it's like, it's very large.
It's basically everybody.
The total potential addressable market
is certainly large enough to allow for a viable oxide, right?
No question about that.
So our question is not that.
Our question is more, and this is what we've been
engaged in, you know, kind of post our launch. And it's where, as we are getting customers into
production and getting new customers online, is finding those customers that represent those big
verticals and making sure, we talked earlier about product market fit, making sure that those customers are absolutely delighted
with their experience and just getting totally bolted down.
Because what you don't want to do is go overly broad
while you're still figuring that out.
Because the TAM is very large, but also very varied.
And one of the ways that companies have perished is that they go
too broad, too early. They get pulled in too many different directions. They actually end up truly
satisfying no one and they drift beneath the waves. So our focus has really been on getting
our earliest customers, identifying the right customers, customers for whom Oxide is really
strategic and represents a big part of their vision for themselves, and then getting those
customers just delighted with what they're getting. That emphasis on customer delight
is really important to us as a company. And it's a stark contrast, I think,
from what people have had in the on-prem space.
Well, that may be true.
And maybe as you scale, it gets harder and harder to do that.
I would argue that Dell is very big.
And so to use them as the example,
because that's where you began,
I think as you get to that scale,
it gets harder to give every customer the most.
But I guess at the same time, it was Samsung. So it's not like they didn't have a reason to be very helpful, but I guess to
be the best at service when you're as big as Dell may be the actual challenge. Maybe the ability to
execute is being smaller or more nimble, or I don't know how to describe that necessarily. What do you think?
Okay, so challenge accepted.
So I think that,
so great customer experience
comes from a couple of different things.
It has to start from a great product.
If you don't have a great product,
it's really hard to paper over that
with great customer experience.
And you really need, and that's Dell's problem.
Like for a long time, I thought Dell's problem was they don't care about me.
Then I began to realize that like, no, Dell's problem is they don't understand their own
product.
I keep waiting to get through to the right person at Dell.
It's like there is no right person at Dell because-
Because nobody knows.
Because nobody knows. And I think that if you can develop the right product
based on first principles,
it's like being a healthy person.
If you're a healthy person, it's easy to stay healthy.
If you start having many things go wrong,
it's very hard to pull all that back together.
And if we can i think as we
scale and it's certainly a challenge for the company but i think if you are if we remain
deliberate about how we add to the company unified by our values and shared mission i absolutely
think that we can continue to deliver an extraordinary customer experience.
And I think because these things build on one another. And I think that when you,
because we are so focused on, because we have built this thing on first principles,
magic does not happen in this system, right? So if we see anything go wrong,
we want to completely understand it. And we've already had this happen a couple times with customers where something happened that didn't even really affect them, but it bothers us.
So we'll be like, actually, so that issue, I know it's not an issue for you.
We've actually taken it to root cause. And here's what we have discovered. And being
kind of completely transparent with them about that whole journey.
I think actually being, in terms of that customer experience, being open source is actually huge.
I mean, we had a customer we were working on.
We wanted to reassure them that they had hit an issue that we were working on.
And we wanted to reassure them that we'll continue to work on your issue.
They're like, no, no, no, we know.
We're actually watching the GitHub issue.
We've been watching the conversation back and forth this morning between two engineers
on it. That level of transparency is something that I think
most companies, I'll just be totally blunt, most companies don't have the
confidence to be that transparent with people.
Most companies are afraid that they, I mean, most companies are absolutely terrified
of saying what Dell was never able to tell me. We don't know because fundamentally a company
doesn't, that kind of punctures this myth of the company. And we take in the opposite approach
of like, we're going to be first principles.
We're going to be completely transparent.
I'm happy to tell you when we don't know. By the way,
if I tell you we don't know, I'm also going to tell you
here are all the things we're doing to try to figure this thing out.
We don't know,
but here are our hypotheses.
Here's all what we're doing to actually go
understand this thing.
Well, that's because your I don't know is got a parentheses
after it with the word yet. We don't know yet. That's right. Because we're on the pursuit of knowing you're i don't know is is got a parentheses after with with the word yet we don't know yet that's right right because we're on the pursuit of knowing but
we don't know yet and i'm okay with telling you that that's clarity right that's that's setting
clear expectations clarity that you do not have the answer just yet but you're trying to get there
i think that's a healthy relationship so many many people hide behind, I guess, maybe their pride to some degree.
Either a corporate pride or a personal pride where I have to know this person.
I mean, their head could roll too. So the person who couldn't give you the answer
couldn't tell you that because maybe they'd be fired. And that's an awful thing to have
happen to you, obviously. So there's many reasons why they can't fully say that.
But I think from the top down, building a company that can say, I don't know, yet, is the way to be. Because
you enable everyone else to say, and you empower them to say, I don't know. But we're going to
find out. We're going to find out. And this is how we're trying to find out. That's right. And
I think we've taken that transparency to somewhat new extremes.
One thing that I love is that we, I mean, obviously you and I are both fans of podcasts.
And one thing that's been super fun is that we have taken the engineering team and we have talked about things that engineering teams never talk about.
Things like bring up or compliance.
Folks don't talk about it because, again, they feel embarrassed.
They feel that they don't want to tell you about all the things, all the terror of bringing up a new board and all of the things. And it is terrifying, right? Because, by the way, if the
board doesn't, if you can't boot, you don't have a product. The stakes are high.
There were some moments that were really, really terrifying.
I love sharing those moments.
Maybe just because I'm
an oversharer, but also because it's
a great story.
Who doesn't want to hear
a story of
the time that we didn't know if we were
going to live or die because
we had a 1k ohm resistor
where we should have had a 499 ohm resistor, which was
as a concrete example, where 500 ohms of resistance
was the difference between life and death for oxide.
We've had lots of variants of those. I think they're really interesting.
I think they're really instructive.
And I think that more companies
should describe more of this stuff in detail
because they would find
that they build better relationships
with their own customers.
Certainly, if I've got a company
that's being that transparent with me,
I love it as a customer of theirs.
Like, this is great.
This is making me,
this is giving me the reassurance.
You telling me what you know, what you don't know,
gives me more assurance because I also know like,
hey, if you tell me you know something,
I actually know you know it.
Because you're so candid with me
when you don't know something,
I actually have got a lot more confidence
in the stuff that you do know.
And to me, that's been really, really important.
It's been really important with our earliest customers and it's been really exciting, I think.
But it's also terrifying for a company to have that much transparency.
Yeah, absolutely.
I guess, where are you on this journey since we began kind of the funding method to get here and the craziness that you did with the conviction to actually do this idea?
Are you planning to raise more money anytime soon?
What keeps you alive?
I know you've got a launch that you've done. You're in customers' data centers. Where are
you at as a company in terms of when you need to raise again? Healthy, not healthy, what?
Yeah. So as a company, I think we absolutely need to raise again. So we had very much raised with
the intent of our next milestones were getting into market, getting our earliest customers, getting them successful.
We're really excited about our progress there, and we will be raising again for sure.
I think that we're starting that towards the end of the year.
We're excited because I think that there's some things that we have done that
look a little less idiosyncratic
now. I mean, it was really
frustrating, I would say, that
so many firms
were puking at
the integrated hardware software
that there were. I mean, it was amazing.
We talked to, like, I mean, I'm just, I'm going to
calm out because, like, they deserve to be called out
for it. NEA is like, we will not fund any company that has a hardware component.
And it's like, okay, really?
First of all, I'm not even sure that's right.
I'm sure you could find hardware companies in their portfolio.
I'm sure this is just an excuse they're giving us.
So you do not want to fund the next Apple, the next NVIDIA, the next AMD.
You're not going to fund those companies.
Those companies are not interesting to you.
And I think that has changed, I'm sure,
in the last four years.
I think that this is where the AI boom
has had such a hard work component
that it has reminded people that actually...
It's not just software.
It's not just software.
It's not just software. So we've just software. It's not just software.
So we've got, I think, more,
which is exciting that we've got more and more investors
that are not puking on that,
that are excited by that,
that see the potential,
that see that there's not a lot of other folks doing this,
which is really exhilarating.
It gives us kind of a... So yeah, we will raise again, but we're definitely confident that we're going to be able to do so because we've got a lot of wind at the back at the moment.
Since you're such a Silicon Valley fan, and I think we can both agree that we're big fans of that show. When I was talking to Resend's founder,
Zena Rocha recently,
towards the end of the show,
I said,
so,
oh,
so you're in season two of,
for,
for Resend.
What season do you think you're in,
in terms of a business?
Like,
if you can align the placement of your business to a season,
juxtaposed against Pied Piper,
are you,
have you just landed Dan Melcher's data play?
Oh my God.
Where are you at?
Oh God, thank you, Ricky.
We have, okay, so no, that is so funny.
What a great question.
Oh my God.
I love the Dan Melcher question
because we have not, it doesn't deliberately.
So I think that that's a really interesting
like object lesson.
There's so much in Silicon Valley,
the student conference.
So for those who have not watched the show,
the Richard in Richard, in a
kind of a moment of desperation,
lands this insurance
company for its data.
And there's a really
interesting object lesson there
is it's a bad customer for them.
Right? It is, they
go in purely as
a cost savings play. Like, that's it.
And there's no melter, I mean, aside from the other, all of the personal partial details.
Season one's debacle, yeah.
The season one debacle, their proponent at the insurance company is only looking to save money.
That's it.
It's not really any strategic value.
So there's the episode where there's the cold, the uncharacteristic cold day
in Silicon Valley,
which of course is like a warm day
in most other places,
like 50 degrees or whatever.
And it's causing a bunch of data transfer
that Richard's not anticipating.
And it's like a financial crisis
for Pied Piper.
And it's an interesting object lesson
in not being transparent, right?
Milcher's got no idea
what's going on behind the scenes,
doesn't have any appreciation
or doesn't care at all about the technology and is really only interested in money.
So we have actually wanted to, I've actually used that example a couple of times of like,
because we have had customers that come inbound that are not a good fit for us because we're
not sufficiently strategic to them.
And we don't want them to be in a situation where you want to take us in a direction
that is actually not where we want to go as a company.
And you want to be in a position
where you've got some discretion
over the customers that you take.
Now, I mean, I think what's also great
about that particular kind of storyline
is like they don't have that discretion, right?
They are – they're desperate.
Yeah, I think they couldn't give venture capital, right?
That was part of their – if we can't give VC money, the murder – I think the one analogy was Jared was on Zillow, and he like looks at – like look at property that he can't buy, just a dream because that's what he's a dreamer.
And I think it was like, well, that would be a Ponzi scheme.
This isn't a Ponzi scheme.
We need to make something to sell something so we can live basically they needed a deal
yeah and that's kind of where they went into that uh and not suggesting that's exactly one-to-one
true for you but i think you know where are you at in seasons is like will you eventually kill
yourself too not yourself as a human being but like that the, does oxide have to, you know, fail disastrously to, to actually not take over the world.
Like happens for everybody,
you know,
kind of thing.
Yeah.
No,
no,
thank you on that.
Yeah.
This is where I would like to not reach.
There are many seasons of Silicon Valley that we will not be retracing.
There will be no Piper pulse in oxides future.
There is,
you know,
we're,
we're not,
I,
I absolutely love
the late seasons of Silicon Valley when they get much
larger because there's so many
large company just
shenanigans that they pull in that are just genius.
But we're in those early days. We're in those
early days with the early customers and
getting those customers,
unlike Moj, getting those customers
really delighted and
beginning to scale out as we
scale out the number of customers as well.
So we're definitely, it's early for the company, for sure.
But we also have a lot of technical risk that has been de-risked.
I mean, this is an audacious thing that we did.
And the fact that we've got this, again, our own switch, our own computer, our own operating system, our own control plane,
our own everything. The fact that this is working, let alone running in production,
let alone working well, it puts a lot of tech risk
behind us and a lot of opportunities still in front of us.
So that customer you mentioned that called upon you in that case where it wasn't a
strategic alliance or in alignment, did you tell him no?
No, we can't sell it to you?
Is that what you do when there's no alignment?
Do you say, we're for sale, but we can't sell to you?
Which is not a good fit for you right now.
And you can feel when people are, and you've got to love these champions who are, you know, they're, they're their own internal champions and they love oxide and that's great. But we really have to kind of
explain to them why we don't think we're a good fit for where you are right now. And there are
often a bunch of factors, but one of them is just like, look, like this is, you want all of the
compute that you need is going to be on one rack.
You never are going to need 10 racks or 20 racks.
You are going to be,
the beginning and end of your deployment is going to be as a one rack customer.
And that just immediately lowers
the strategic value of Oxide for them.
And we just don't want them to be in a situation
where it's like, yeah, you've got,
you literally have no other computer
and you are solely dependent on Oxide.
Obviously we'd make that successful,
but that is just not kind of the right customer for us right now.
We're really looking for those customers
for whom Alaska infrastructure on Plurim
is strategically very important.
And one rack is not like the end of where they're going.
It's the potential beginning.
So if you know who's not your customer, where are you the best fit?
Even right now, in the next year-ish of where you want to be at to understand your fit for the customer,
see it deployed, understand the permutations of what they're using it for
and how it's being used.
What is the best fit for Oxide today?
So, I mean, and we've got a couple of these, which is great.
I'm just blessed with any CDs and a bunch of different verticals.
But we are folks that are either significantly on-prem today
or want to do big on-prem experiments that are like,
I'm currently, I'm public cloud born,
but there are some big strategic reasons
why I may want to go on-prem in a very big way.
Cloud educated, for sure.
Like I understand cloud computing.
I know the public cloud,
but it's not a fit for this use case.
Deeply technical is extremely helpful,
not because the product itself,
I mean, we pride ourselves on the fact that, you know,
API-driven, CLI-driven, UI-driven,
teams have done a terrific job
making the product easy to use.
But I think it's important for teams
to really appreciate the kind of transparency
that we bring.
It helps for a team to be technical.
And also, if a team is like,
I don't understand why you guys keep disparaging Dell
because they're fine for me.
There's a certain degree where it's like,
you're just not deploying enough of it.
You know what I mean?
Where it's like, if you have had nothing
but great experiences with Dell or HP or Supermicro,
you just have not deployed it at sufficient scale
or at sufficient depth
or with sufficient bluntly attention to
availability that you just haven't hit these issues. So you kind of, you need a team to be
technical enough to appreciate Oxide's differentiators. You know, and fortunately,
we've been blessed by a bunch of those, which is great. This is, you know, the real, the value of
kind of connecting with people at kind of a grassroots level is that there are a bunch of
folks out there that,
that fit that bill.
Well,
not that you need our mouthpiece to connect with those folks,
but we have a lot of engineers and a lot of big places and small places.
I sees everywhere that listened to the show.
I'm always surprised by like literally who we reach across the globe and the
people who write into us and say,
I've been listening to you for a decade or more,
you know,
or I listened to you doing the dishes,
sometimes in the shower,
which I debated with Jared.
I'm like, seriously,
did somebody listen to us in the shower?
And it's true.
Somebody, at least one person,
has admitted to listen to us in the shower.
You know, I love podcasts,
but not in the shower.
Not in the shower.
Yeah, me either.
Although I've never tried it,
so I can't say I really wouldn't like it.
I might try it. A podcast doing the dishes, though, I'm 100% on that. can't say I really wouldn't like it I might try it
podcast doing the dishes though I'm 100% on that
yeah for sure
I can't be productive in my house
I can't do my daily chores
as a human being in my household
unless I throw on my noise cancelling headphones
and just jam on a book
or jam on a podcast
or a sermon or something like that
that really just like is a good hour-ish of
something, you know? And so thankfully we've recorded almost three hours, well, two and a
half hours now here for somebody doing multiple days of their chores. So you're welcome.
Yeah. We got you doing the dishes. We got you doing yard work. You're walking the dog a couple
of times.
That's right. And then if you're done with all that, try us in the shower.
All right. Now you made it weird.
So let's say that there's an IC out there or somebody that has been listening deeply to this.
They understand your personal journey.
They understand what you're trying to do with Oxide.
And they've got your problem.
Like, point out some specific companies you would love to work with if you want to.
And if you don't want to, then don't.
But is there any particular name that you can drop oh i guess specific names
yeah like not like a person's name but like a brand name like oh man where do you think you'd
be perfect at you know they've got problems maybe even a customer of theirs and you're like yeah
i've seen this latency omg i can fix this for you if you just buy outside. If you just, yeah, that's funny.
Or where you think you'd like really do a good experiment at even.
Like I mentioned my friend at ERCOT, for example,
like they only do on-prem because they're ERCOT, right?
They run the energy grid in Texas.
You know, like it's a big deal to like,
obviously have a strong, resilient energy grid
and any systems deployed for a company like that,
you're going to, or I guess an organization like that,
you're going to want to have, you know, great on-prem hardware where they probably are racking and stacking Dell.
You can pause if you need to think about it too.
Yeah. So part of my challenge here is that we've
got so many conversations going with companies that already kind of fit this bill
for me. So the, I will say
that I'm just really interested by how people use compute kind of industrially, right? So
whether it's financial services, healthcare, energy, I will say that like a place that I'm
super interested in that I think will be a big deal for us in the limit is in biotech, biotech pharma.
But I just feel like that there's so many interesting problems that computation is going
to, computation has got the potential to really change lives in the kind of, in the limit.
And I mean, and I think that like a lot of those, if you look at biomed or biotech in general, the compute needs are just make the cloud economically a non-starter.
And they really need to be on-prem, but they often don't have, they really do not want to be spending a lot of time in energy managing infrastructure.
So that's kind of a, I think that will be a big space for us in the limit that we actually don't,
that would definitely be new for us.
But I think,
you know,
financial services folks,
like if the,
certainly,
and this is where,
honestly,
the Broadcom acquisition of VMware has definitely served to accelerate things.
Yes,
it turns out like that's not going so well.
If you're a VMware customer,
yeah, Broadcom is not so great and so we've got a bunch of folks
who are using
this as an opportunity to really reconsider
a bunch of things and so we've seen
certainly a bunch of that in financial services
I think we've seen a bunch of that
in federal spaces
I think we are
I think we will see a bunch of that in energy and oil and gas I think we will see a bunch of that
in energy and oil and gas.
I think we will see a bunch of that in biotech.
But yeah, you know, it's funny,
because in terms of like the,
like, boy, what is like the dream customer
that I want to land?
You know, I am so,
it's so interesting to me
when people are using compute
as a differentiator,
information as a differentiator,
which so many companies are,
that I find, I think I'm always intrigued by,
and I think we've been,
I feel very lucky with our earliest customers
in terms of what they're doing
and just total reverence for their technologists.
So I know it's not like I can't pick among my children,
but, you know, I can't.
I love them all.
I love all customers, Adam.
I just, I love them all.
Yeah.
Now I'm actually worried if, well, I guess worry is probably the best word,
slightly concerned. Can you handle the influx of need? Because when you're at the innovator's
dilemma, which you've been at for the past four years, you've launched, so you've crossed that
chasm, so to speak, and now you need to begin to experiment in production in ways and you need to do that with
confidence and you need to do that without screwing up everything basically so you can only take on
so much burden as i guess deployed product can you handle the demand that's coming to you right now
that's right and i think that i mean this is steve my co-founder likes this metaphor that i think is
very apt of the gas and the clutch, right?
And you're really trying to hit the gas and the clutch at just that right friction point.
And if you pop the clutch too quickly, you're going to stall the engine.
And if you gas it without pulling the clutch out, you're just going to rev the engine.
You're not going to do anything.
And so you need to be right at that friction point. And I think in every domain of the company.
So there are a bunch of things that we're doing.
I mean, actually, to make it really concrete,
looking at just like manufacturability,
one of the things that our AA and MECI teams
have been working on is how do we get to a,
how do we significantly up our ability to produce racks
by developing better infrastructure
for things like testing?
So, you know, in software,
we love to talk about automated testing.
And we will for sure do this
on an upcoming Oxide and Friends,
but some of the hardware that we've done
for automated testing
and to improve our ability to test the rack,
because that improves our ability to test the rack because that improves our ability
to actually deliver the rack.
Like the faster we can test it,
the faster we can find manufacturing defects
such as they exist,
the faster we can actually ship the thing
and the more we can scale that out.
We develop our own manufacturing software dockside
and we develop a lot of our own jigs and so on.
So that's going to be super important to,
you know, that's a big part
of being at that friction point
is developing that apparatus
to allow us to go faster
from an operational perspective.
You're making your own hardware?
Yeah.
Self-manufacturing.
Yeah, we've got a,
so we've got a manufacturer,
Batchmark Electronics,
that is our
contract manufacturer but we very much like i mean they do the assembly and they but we provide
the infrastructure for things like system test and component test and so on so it's very much
a collaboration well that's a whole three more hours we can't even talk we're out of time brian
gosh we'll have to we'll have to do more of this.
I mean, I'm not denying you talking by any means,
but I feel like there's just so much more to cover
that we can't cover in one session.
Yeah, there is.
And I think it's like that stuff,
that stuff is super important, right?
Because it's like you want, I mean,
we haven't had this happen yet,
but if we were to have an issue where we had a defect in the field, we would want to be able to trace that back through manufacturing to be able to understand.
It's like, is there something happening on the manufacturing line?
And we've got to own that whole thing end to end.
And so, yeah, it's extremely important to us that we're able to actually own that whole thing. And Venture Talk's been a great partner in that regard.
I'm impressed. I was impressed before this conversation. I'm even more impressed during
the conversation. I'm going to listen. I'm going to go in the shower, okay? I'm going to take a
two and a half hour shower and I'm going to listen to us, mostly you talk, and me ask a few decent questions and listen back.
As a podcaster in the moment, you can only get so much from the conversation.
There's a second layer that I get after the re-listen, so to speak.
So I'm going to take a long shower and listen to this conversation.
And maybe stop there just because I feel like, gosh, you're self-manufacturing.
It's just so much more.
You're,
you're really committed.
I would say it's probably the,
if I had to summarize your,
your standpoint is committed,
very committed,
right?
Very committed.
All in conviction and committed.
That's,
that's wild.
Good for you.
Good for Steve.
Good for everybody else.
I'm part of the team,
Adam,
the other Steve that I know,
Steve Klabnick,
who's been a long time friend of ours.
So when, uh, when we think about all that I know, Steve Klabnick, who's been a longtime friend of ours.
So when we think about all you've done,
Jesse Frizzell, like everybody that's involved in the origination of what you're doing,
such a fan, really, I'm such a fan.
Thank you so much.
It was everything I could do to not talk more
about literally Silicon Valley, the TV show
and this conversation is like, we could.
We could, I know.
I just sprinkled a little bit in there
because we had to.
It was necessary.
But maybe, maybe sometime we can just like literally just talk about only that.
Oh, that'd be great.
Would you like that?
I really did reach out to Gwartz, the actress that plays Gwartz.
Did you really?
And I talked to her handler.
Listen.
So I talked to her handler.
Oh my God.
Or whoever is answers her emails.
I assume a handler.
I don't know the title to call a person
just for sake of not butchering their name and because i had dm'd you mean just give people a
little bit of the backstory here in terms of court because i dm'd you about this right well you told
me you left me hanging for a bit there you told me that gourd had a cameo in season one and then
you didn't respond to me for a week and And I'm like, are you being for real?
And then so then you're like, yes, it's episode one.
And then it's at the party, in the toga party.
And sure enough, she was there.
And I asked, I had to like confirm.
I saw Gwart.
I saw the actress who played Gwart.
Her name is Nandini Bap.
Okay.
Bap. I don't know how you say it last. I'm sorry is Nandini Babat. Okay. Babat.
I don't know how you say it last.
I'm sorry, Nandini.
It's B-A-P-A-T.
And so, in disbelief, I watched the episode, saw the actress who plays Gwart, went to,
searched for who the actress was on IMDb, of course.
That's like my Bible for all things when it comes to movies and cinema and whatnot.
Yeah.
Found her name, found her website,
reached out to the contact form.
Like this is probably a half hour
after you finally told me which episode.
I immediately went and watched.
Oh, this is awesome.
Reached out and said,
hey, let me see if I can get the email up too
because I'll verbatim what I exactly said.
This is awesome.
My 16 year old, myyear-old and I,
I'd like to pour over this together.
It was when we were doing a rewatch
where he was like,
oh my God, that's Quart in season one.
And that's so cool.
So I talked to Nandini's assistant, Marjorie, by the way.
I don't actually have a copy of what I said,
but she said,
thank you for reaching out and supporting Nandini
and Silicon Valley.
Yes, she did have a cameo appearance in season one episode four
it was an unexpected bonus that nandini happened to be working as a background actor on the show
before she was hired to play gwart in its final season so technically it's gwart but not gwart
not court oh yeah all right technically, closing the loop,
she was a background actress at the time, right?
That's amazing.
And so she was there as a background role.
I don't think she even had the glasses of Gwart on,
if I recall correctly, going back to...
No, that's right.
Yeah, it was definitely like...
And Gwart wears those thick Coke bottle glasses, right?
Signature of Gwart.
So you're right, but technically not right.
The actress who plays Gwart
was x-raying at the time,
got a gig, and then came back for season
six as true Gwart.
I'm calling it Gwart. I agree.
And so I just said in response,
so cool. And I, this
is the salesman in me, okay? This is how I am
as a person. If I have
an opportunity to talk to you, I'm going to ask you a couple more questions and get not so much more out of you, but get an opportunity.
Right.
I love opportunities.
I said, so cool.
Thank you for closing the loop for me on that.
Do you think she'd be down to pod with me and some big fans of Silicon Valley?
I'm dropping you.
You're dropping me.
I like it.
I like it.
I'm in that sentence.
From a shade perspective. I'm like, this is you dropping me. I like it. I like it. I'm in that sentence. And from a shade perspective,
I'm like,
this is,
you're my friends of Silicon Valley.
And I looked at her podcast and she says,
of course,
my pleasure.
She's always open to talking to fans.
So I like it.
At some point we may be talking to the actress who played on a podcast about
our favorite show in the whole world.
I love it.
I do love it. What a great way to close the show, right? That's the whole world. I love it. I do love it.
God,
what a great way to close the show,
right?
That's the way it is.
All right.
It's the proper way.
It is the way talking about Silicon Valley and the future.
It's been a lot of fun.
It has been a lot.
I mean,
it always is.
It always is.
I'm obviously,
I'm a big fan.
So I always,
always appreciate nerding out on Silicon Valley.
Yeah.
Both.
Yeah.
But both the, the literal and the metaphorical.
I'm glad that when I dropped the Dan Melcher reference, you were able to not just go there with me.
You were also able to reiterate for the audience exactly what was happening in the season, in the episode, to kind of give context, which was.
I think it's so good.
That's how you know we could be true friends, Brian.
That's how you know.
That's it.
Right?
That's it. That's it. If nothing else, we always have Silicon Valley.
We will always have Silicon Valley, Adam.
Thank you, Brian. It's been awesome. Thank you.
Thank you so much for having me.
Okay.
That was epic.
What a story.
We went to Sun, to Oracle,
to Joyent, to Oxide, to literally Silicon Valley, and then
literally Silicon Valley, the TV show. We had to do it, right? Lots of fun. Maybe some things
happening in the future. Who knows? But I'm a big fan. Love what Oxide is doing. If I had a reason
to buy one of their racks and I was that ideal customer we talked about,
I would be buying the rack right now.
So for my friends out there in the world that Brian painted a picture of,
if this is for you, reach out.
They may be ready to talk to you.
Check them out, oxide.computer.
Love the domain name too.
So cool, so cool.
Okay, so Jared and I are literally
at Microsoft Build
right now. If you're listening to this right now, we're at Seattle podcasting, hanging out,
meeting people, seeing what the future of AI is. There were some announcements from OpenAI
and some announcements from Google. So I imagine at this conference, we'll also see some more,
some more announcements from Microsoft. It's getting crazy
out there. Big thank you to our sponsors today. Century.io. Use the code changelog, get a hundred
bucks off the team plan. Almost four months. Enjoy it. Once again, changelog and check them
out at century.io. Our friends at Tailscale love Tailscale. I'm using Tailscale right now at Microsoft Build because I travel nowhere without my Tailscale.
My tail net is one ping away.
And also our friends at Neon.
That's the home of our Postgres.
Yes, our Postgres lives at Neon.tech.
And of course, a massive, humongous thank you to our friends and partners over at Fly.
Put your apps, your databases, and your AI
closer to users all over the world with no ops.
Check them out at fly.io.
And to the beat-freaking residents,
Breakmaster, selling those beats are banging.
Love you, BMC. Thank you so much.
Okay, no bonus today because it was just that long
and just that good, hopefully. We'll see you on Friday. Thank you. Game on.