The Changelog: Software Development, Open Source - Rebuilding DevOps from the ground up (Interview)

Episode Date: June 22, 2023

This week we're joined by Adam Jacob and we're talking about his mission at System Initiative to rebuild DevOps. They are out of stealth mode and ready to show off their transformative new power tool that reimagines what's possible from DevOps. It's an intelligent automation platform that allows DevOps teams to build detailed interactive simulations of their infrastructure and use them to rapidly update their production environments.

What's up, welcome back. This is the changelog. This week we are talking to Adam Jacob, co-founder and CEO of System Initiative. They are out of stealth and ready to show you exactly how they plan to rebuild DevOps from the ground up. Of course, we had to have Adam on this show to go deep on this subject. So that's what we did. A massive thank you to our friends and our partners at Fastly and Fly because this pod got to you fast because Fastly is fast, super fast globally. Check them out at fastly.com and our good friends at Fly. Well, they help us put our app in our database and they'll help you all over the world with no ops.

Check them out at fly.io. Okay, before the show kicks off, I'm here with one of our sponsors at DevCycle, CTO and co-founder Jonathan Norris. So Jonathan, my main question, I guess, if I'm handing off my feature flags to you all, is my uptime dependent on your uptime? Like if you're down, am I down? We've designed into all the SDKs and all the APIs. APIs fail, right? That's a cardinal rule of the internet.

Exactly. It's slower. It's something, right? So all the SDKs have been designed with kind of defaults and caching mechanisms and all that stuff in place so that, yeah, if our CDN is down or our APIs are down,

it'll sort of fall back to those defaults or those cache values in those SDKs. So that handles for those blips pretty easily. And then we rely on Cloudflare as our sort of main high load edge provider. So all of our edge APIs are through Cloudflare and they're also operating as our CDN for assets. So obviously relying on a large provider like that, that runs such a large percentage of the internet means that, yeah, you're not relying on our ability to keep AWS instances running properly. You're relying on sort of Cloudflare and ability to sort of make sure the internet still works

as they control such a large percentage of it. So yeah, we've architected it in a way that it doesn't sort of rely on our APIs to be up all the time and our databases to be up all the time to have that good reliability. Well, that's good news. Okay, so how do you accomplish that? One of the core sort of architectural decisions we made with our platform when we designed it was trying to move the decisioning logic of your feature flags as close to the end user and end device as possible.

So we did that with those local bucketing server SDKs that are using sort of a shared WebAssembly core. And then we have edge-based APIs that are also powered by WebAssembly to serve sort of those client SDK usages. So things like web and mobile apps. So that's one of our core principles is to try to get that decisioning logic

as close to the end device as possible. And this is probably one of the only use cases where performance really matters because you want your feature flags to load really, really quickly so you can render your website or you can render your mobile app really quickly. And so, yeah, we definitely understand

that your feature flagging tool needs to be fast and needs to be really, really performant. So if you want a fast feature flagging tool that's performant and is not going to impact your uptime, check out our friends at DevCycle. That's devcycle.com slash changelogpod and for those curious they have a free forever tier that you can try out and prove to yourself and your team

this is going to work for you so check it out devcycle.com slash changelogpod and tell me sent you We'll be right back. All right, Adam. So in September of 21, at the end of the last time we had you on the show, we said, come back when System Initiative launches. And you said, I'll dive as deep as you want. I'd love nothing more than to spend however long you want to talk to me being nerdy about it. Believe me, no one wants to tell the story of what that thing is and why it's amazing more than I do. That's super real. Yeah. Yeah. And so

here you are, nerd out with us. Tell us the story. It's a couple of years later and you were already toiling for some time when we said that. So yeah, it was not, not an easy thing to build. What's been going on, man? Yeah, well, so yeah, system initiative. Now, like a public thing, you can go learn what it is and watch a demo video and you can come in, like join the community. We're going to open source it soon. There'll be a little tracker that can tell you like how close we are to open sourcing it.

Basically, we need to add a couple of features before we're ready to open source it, but soon. And yeah, I'm sourcing it. Basically, we need to add a couple of features before we're ready to open source it, but soon. And yeah, I'm stoked about it. And it's basically, we're trying to rebuild DevOps from the ground up and change the outcomes for what I think are kind of mediocre outcomes that the majority of us have sort of come to experience.

When you rebuild, do you demolish along the way? Do you obsolete things along the way? And if so, what dies? Well, yeah, well yeah okay look so i suppose we should back up maybe i maybe i would trade straight to incendiary too soon but like you know wait a few minutes but i want to make it cool you know rebuild from the ground up it is rebuilding it from the ground up here's the thing so i went back and watched john allspaugh and paul hammond talk from 2009, the 10 Deploys a Day at Flickr, that was basically the moment that DevOps started, right? Patrick Dubois, I think, was in the audience, or at least saw the talk shortly thereafter. I think he was there.

And that's sort of what led to DevOps. And, you know, if you watch them give that talk, it's an amazing talk today. And they describe how they deploy Flickr 10 times a day. And the way that they deployed Flickr in 2009 is essentially exactly the way that we tell people to do DevOps today. Like the tools are completely different. We've replaced the tools 50 times. You know, you've got a ton more options

about like which thing slots into which, which part of the workflow or whatever. But that workflow unchanged since 2009. You like put in some code, it goes into CI, which part of the workflow or whatever. But that workflow, unchanged since 2009. You like put in some code, it goes into CI. You know, at some point someone wangs a button, you do some feature flags, you do a little dark launching, right?

Like you put it all in a blender, get some monitoring and observability. They were using Ganglia. Now you'd be like, I will use Honeycomb because observability is better than monitoring or whatever. You know, you got some Datadog. But like essentially it's identical to what we did in 2009. And the last DORA report talked about how there was like 88% of all the companies reporting can't deploy more often

than once a week to once every six months. And like, that's insane. 14 years later, like 88% of us are stuck, can't deploy more often than once a week. And like, it's not because we didn't optimize each individual tool. It's not like you can point at a single tool and be like, oh, it's, you know, whatever.

It's Jenkins's fault, you know? Or it's like, or it's like chef's fault. We try to do that though when we're mad, don't we? Yeah, you know, or it's like, or it's like chef's fault. We try to do that though, when we're mad, don't we? Yeah, we try, but it's like, but it's not right. The truth is that what worked for us in 2009, in the small, when you say, now I want to extrapolate that same way of working to solve the problems of every enterprise on the planet. The problem was the system itself, right? The way that we interacted with it, the way we conceptualized

how all the pieces come together, that's the reason that we're not getting the outcomes we want. It's not because we're not like DevOps-ing hard enough. Do you know what I mean? Like nobody wants really, I think, people actually do are now saying that they want to go back to a world

that was more like devs do devs and ops do ops and never the twain shall meet because the experience of doing dev ops the way that we conceptualized it in 2009 when you extrapolate it out to 2023 is kind of miserable and so when i say we have to rebuild it from the ground up i i mean it in terms of like i think the shape of the system as a whole is actually what's holding us back and it it's not, and our ambitions were the right ambitions to begin with. But our ability to actually achieve those ambitions, we now know that if you just sort of DevOps a little harder, you know, swap one tool for another, but keep the basic shape the same,

you're probably going to wind up deploying once a week, once every six months. It's going to kind of suck and you'll be kind of miserable about it. What was the reason for this once a week with the Dora report? Did they cite why it was once a week? What were some of the bottlenecks that made it once a week? I mean, we can deploy. I mean, we're a smaller team though, but what would make you not be able to deploy more than once a week? Yeah, I mean, sometimes it's the complexity of the deploy. So if you think about how you deploy the software, if you think about how the build pipelines work,

if you think about how, do you need to make changes to infrastructure? How do you collaborate together on the infrastructure changes? How do you coordinate them together with the application deployment changes? What about your QA team? Maybe you have one.

Does that mean there's a lead time where they get to bake in the software? There's like a lead time that where they get to like bake in the software there's a million questions like that where like we have answers but once you glue it all together it's quite complicated you know if you've ever seen people use like like an enterprise grade terraform repository that's like a lot of code you know what i mean it's not like an easy peasy thing that it's like a whole giant computer program that's like factored in a particular way slack had a post not that long ago where they have like i think they were it was like a thousand plus state files that they were tracking for their

how they use terraform at slack and so if you imagine as a developer trying to like influence you wrote some code and it needs to change the way the infrastructure at slack works what do you do like how's that work do you load that all into your head like where's the collaboration and and the answer is in pull requests right right so maybe somebody told you well you have access to the repo so you know maybe go figure it out man or they didn't give you access to the repo. So, you know, maybe go figure it out, man. Or they didn't give you access to the repo at all. And then you just had to go find somebody else to do it. But mostly we do it in code review.

And like code review is great, but like it's not very collaborative, right? Like you do the work and I review it. It's there in the name, you know? I submit it. I wait for you to review it. And then you tell me if it's good or bad. You're probably wrong. You know, you were busy when you reviewed it. Maybe you got it right. Maybe you got it wrong. How dare you? No, obviously your pull requests are always correct,

but like, it's a real thing. And if you ask people how they feel about it, you know, very few people will tell you that they love the experience of doing DevOps work, the user experience of it, the actual human, what's it feel like to do this? We're good at it and we're proud of it. And it's unequivocally better than it was in 2009. So don't get me wrong. Like if your choices are do it the way we do it today or do it the way we did it in 2008, all day, do it the way we do it today. All day. But like, is it good? I don't know. Like, not really. Well, you're the kind of person, Adam, that looks at things from so many different facets.

And I think you have a way to express your ideas well. And then you're also the kind of person to examine a system to say, okay, is this the right way the system should be? Obviously, system initiative makes a good sense in the name of the biz. Yeah, suddenly the name makes sense. Right. You know, I think that's a good thing. You know, so if we're rooting for somebody, it's you.

It's you we're rooting for. Thanks, man. You know, but I think actually we need to be rooting for us. I mean, more than anything, if you think about it, like what's amazing about DevOps is that it happened at all. Basically, the message that said, operations people and software developers are the same, and we need to work together to create these outcomes that are better, not only for our day-to-day work and how we feel about each other and our personal experience of the work, but also for the businesses that we serve.

And it's kind of amazing that that cultural transformation is so widespread. Very few people, you know, it's like what happened with Agile. Lots of people are maybe doing Waterfall, but if you ask them, they'll almost all be like, oh, we're definitely doing Agile. No, we wouldn't do Waterfall, you know, because the culture of Agile is so dominant. Right. That even if you're not doing Agile, you're doing Agile. Because it's embarrassing to say what you're actually doing., you're doing agile. Because it's embarrassing to say what you're actually doing. Yeah, because it'd be embarrassing to do anything else.

And DevOps is kind of the same, right? Like you wouldn't say, no, we don't allow the ops people to talk to the devs. You know, like that's insane now. And it definitely wasn't insane in 2008, you know? But I think what needs to happen in DevOps, I think, is we need this second wave of DevOps tooling where people realize that the problem was systemic and that we need to actually build new solutions that change some of the fundamental rules of how these things worked. We made choices. John and Paul, I love, they're incredible, but, and so is everybody else that like influenced how we work. But like, we just made choices based on what we knew and what we had and built the systems that we thought would work. And it's okay

to realize that they didn't work the way we wanted them to in the end. And that what we need to do now is experiment again. Like now we need to actually go try new ways and be like, well, what if we did explore new ways to deploy applications collaboratively? What would a world where we could deploy applications, but we didn't require code review, but we still had all the same safety properties look like? I don't know the right answer to that question, but I want to, right? Like, I think that could be better, you know? And I think once people realize, I hope people start to realize that we shouldn't give up on those aspirations that we had in 2009, that we were right. And the evidence that we were right is how pervasive that culture has become. And what we need to do now is live up to our own

aspirations by being willing to do it, like being willing to actually do the work and create something fundamentally different than what we have now. Cause right now we kind of know what happens, right? Like if you adopt all these tools, whichever ones you choose and you put them together roughly the way we told you to, you're probably going to wind up 88% of you deploying once a week, once every six months, right? Not collaborating super well. Like that's what's going to happen. And we can do better. We should do better. We deserve better. So when you started down your path to where you are now with something to show and you thought to yourself, we can do better, we should do better. I'm going to try something fundamentally different.

What was your starting point? What was your base premise or that you built from that said, this is going to be different. Here's where I'm going to start. Yeah. So, I mean, I had two co-founders, Mihiro Lupinacci and Alex Etieh. And I think my starting place was that there was something wrong with the system as a whole. So that like insight that what was wrong was the way we did it. That was the whole thing, not just a single tool that was wrong. And I had some ideas around making the automation smarter, basically using like relationships and inference and constraints theory to like think about how we could infer more of the configuration

instead of so much repetition. One of my co-founders had a background in visual effects. And so he had a lot of ideas around how the interface itself could be changed to serve us better, right? That if you thought about the way we deal with really complex, the level of complexity that you see

in like big video games, AAA video games, or those sorts of things. We have really powerful tools that are very collaborative with people across multiple disciplines working together to produce those AAA games. And the user experience they're having versus the experience that DevOps people are having is pretty dramatic, right? You're talking about the people building the games, not playing the games, right? Yeah, totally. Yeah, building the games, not playing the games, right? Yeah, totally. Yeah, building the games.

I guess I'm not familiar with what tooling they have that's better than what DevOps has. Well, if you think about Unreal, where you think about the way that you can collaborate on building scenes or building shaders or rendering, and you can go from a high-level interface where you see all the relationships of all the different objects in the scene,

but then you can also get down to the C++ code that actually makes that particular thing work. And you can do that dynamically in the same IDE. And you can't do anything like that in DevOps work, right? You're in your editor writing code, then you run a plan, you hope it works, you apply it later. None of that is possible. And so a lot of where we wound up was really focused on the user

experience and saying, Okay, if we want it to be collaborative, if we want it to, to be more intelligent, how do we build a system that actually enables that to happen? And there's a bunch of stuff that sort of gets in your way. So some of it is like the feedback loops. You know, if you have to talk to the real AWS API or actually create the infrastructure to do something that takes forever, right? So suddenly everything's slow. So if we want to speed them up, then the only way we could figure out how to do it

was to basically build a digital twin and say, hey, even though this asset is digital, let's build a model of it that's like one-to-one and then simulate that model the same way that like a Formula One race car gets simulated because we don't like tweak, you know, you don't like tweak the race car and then drive it and then tweak it and drive it.

You like put it into a big model and you tweak it and you do a bunch of airflow simulation and like- We saw this in Cars 3. They did that in Cars 3. It was a great simulator. Exactly. The inspiration was Cars 3. That's right right there'd be no system initiative without what was that little car's name that with the with the red race car with the bink you know i'm talking about italian

one yeah i'm talking about the guy who was working on stuff i don't know anyway i forgot all of your names but wendy mcqueen is the main mcqueen well yes oh that's who you're looking for was the main character okay he said kachow kachow he said likelink or something yeah no ka-chow he's right it's ka-chow i'm saying what you said adam was yeah it wasn't right because it wasn't actually i was like ka-chow it was yeah but it wasn't ka-chow it was a caricature of what should have been true yeah yeah right yeah it was it was a poor simulator and so we could have given me see you gave me immediate feedback that my ka-ching was wrong and that it was in fact,

right. And so now I can adjust and do the right thing. And so system initiative does the same by basically building this really full fidelity model of all the stuff that you use. And then we use that model to infer configuration. So we can say, Hey, your model of a Docker image knows that this Docker image exposes a port

number. Therefore, if you wire that up to, you know, a butane configuration to run on a Fedora box, then it'll automatically expose that port when it writes the systemd unit file for you that you need to boot that service. And like, you don't have to remember it anymore. And if you change the port number, it'll automatically get changed everywhere and you can watch it happen and sort of flow through this visual graph of all the configuration.

It's super cool and much faster than doing it yourself. Can we dive into the digital twin thing for a moment, into the weeds? Yeah, man. How do you build a digital twin of a moving target? I mean, it's gotta be difficult when your twin diverges from the real thing because AWS is not a solid state service. What a great question. Yeah. I mean, the thing is it moves

slower than you think it does. Okay. So this, this comes back to some of the reasons that people tend to be less ambitious than they should be. You know, like when you think about like, when you look at, at something like AWS as a target, you're like, how could anyone ever cover off on all of these services that are in AWS? And you're right, there's a lot of them. And also, they're not that hard. It's mostly documented. It's quite detailed.

You can understand them. And yeah, it's going to take a minute to write all the models. And also, you can then refine the process of writing models. So like, for example, in system initiative, one of the things we know needs to happen before we open source it is that you need to be able to write the model in system initiative itself, so that you can immediately just write the model, tweak it a little play with it, tweak it, play with it, tweak it, and like never have to leave the flow. And that makes creating models

both fun, and also not that hard. And the same thing's true with sort of extending them. So you know, yeah, it's a big list. But also, it's kind of straightforward. And they don't actually change as much as you think they do. So you know, if you look at like the EC2 API, it's only really had a handful of dramatic changes over the years, right? We had like new instance types, we had new, new stuff, but like, you know, mostly it stays the same. You know, if you look at the S3 API, it's gotten more complex, but mostly because they've added more objects and more things you can do around the core of the S3 API, right? That again, that core API kind of the same, you know? So like, yeah, there's a lot to do there, but lots of people,

some venture capital dollars. Right. Throw money at the problem. Next thing you know, you've, you know, gotten a lot of AWS coverage, right? Right.

How do you keep these digital twins in parity? Like when there is a change, how do you, you know, institute that new change? So you bake it

into the system. So basically, the idea, we have like a schema that defines the digital side of the model. It also defines the sort of resource data. So the real world information about it, you track both, they're both true. So there's not a single source of truth. There's actually two. There's the simulation and then there's the real world and you track both. And then what people do, what we do is let you reconcile between them, right? So choose which one is actually correct right now. And so you can go from, you know, you can go from the world to the model or the model to the world, right? But you basically just make a new variation.

So under the hood, the way this thing works is you can just imagine every configuration option and every action you can take is a function that's sitting on a graph and it takes inputs from other parts of the graph. And then anytime the inputs change, the function runs. And then we take the output of the function and we store it as the output on the graph. So when you like set a configuration value, there's like a, you know, what happens is we calculate all the inputs, then we take the output and write it.

And so when you think about how you would then change over time, you're going to have all these schemas, which then people build variations of. And some of those variations might be because the upstream API has changed. But it could also be because you've been adding customizations on top of the existing thing, Cause it's one thing to have a cool model. It's another thing to be like, Hey, you know, here in my company, we're only allowed to use these three AMIs. And if you can't even launch a system with another one. So that's something you can encode in system initiative and a qualification, and it would make a new variation of AMI, right. That is like the upstream AMI, but is now yours,

right? Sure. And so that like shape of easy, quick customization is built into the system so that you can just evolve the model as it makes sense in your environment over time. Sounds like Lisp. Are you using Lisp? It's not Lisp, but it does sound like, but I'm so glad you brought it up because it's actually probably even closer to small talk. Okay, cool. So if you cast your mind back to the old like Alto demos where they would like tweak the way the Alto worked in the demo, right?

There was that one where they had the like bouncing ball or whatever. And then he like- You're going way back now. He like edited the window and changed the code for the streaming thing so that now he could take single stills of the bouncing ball or maybe to animate it.

I don't remember what the order was. The point was he changed the operating system in real time by clicking a button and being like, let me edit the source code for this part of the operating system. And we just forgot that that was a thing you could do. And so a lot of what's in system initiative actually draws a very direct inspiration from that moment where I'm like, hey, you know, it would be great in DevOps work.

If anything I see that I need to change, I could just change the code without having to stop, which is exactly what it lets you do. And it's better, you know, like turns out that's super good. At least I think it's super good. So I guess, I mean, thinking about just the surface area of some of these APIs, I now know why you've been working on this for so long. I mean, there's just a lot of work there or was that a small percentage of the overall effort? It's actually, so there's not very much coverage right now. Okay. So you're still in system

initiative. Okay. A lot of what we have spent time on, like everything I just said, no one had ever built before this way, you know, like, like I didn't talk about the fact that like that graph I just described, it also has built in change control. So when you decide you want to make a change, what you're really, it's not actually a graph, it's a hypergraph because it's N dimensional. And so you can have a different function at a different moment in time, depending on your perspective, which is what you need because part of what you do when you collaborate with other people is just like, put up a whiteboard, you know, and be like, well, I think it's this, or what if it was that, or what if it was this way? And sometimes you want to keep that and sometimes you don't,

but you want to be able to do it in the same environment, right? As the one that you'll finally do get the right answer in. And all of that is wickedly complex like how do you build change control into this like hypergraph of functions that are all dispatched in real time you know it's like a lot just it's complicated just fork git or something show some git in there just fork git except you can't because part of, part of how deep this rabbit hole goes is that once we decided, like gets beautiful for source code. But if you look at what the root cause is of a lot of the bad user experiences

we suffer in DevOps work, it's source code. It's that it's words. It's that it's factored the way that it is. It's that it's not a digital model that I can ask questions about or interrogate or programmatically update. I can't do any of those things.

All I can do is declare them, write them in some files, and hope. I can see a textual diff of, you know, Git will give me a textual diff of whatever it is I'm looking at. But it can't tell me the impact. It can't tell me, hey, you don't want to restart the production database 2 p.m. on a Tuesday. Because it has no way to know that that's a production database. It just knows that it's a block of code. You know what I mean? Yeah.

There's no intelligence. Like a connection string to get. It's like, here's your connection string. Yeah, exactly. It's just a string. I don't know. What do you put in there? Whatever you want.

Did you get it wrong? How do you know? Well, I deployed and nothing connects anymore because I foobarred the connection string. But even that you have to know, like, well, what files it in and how do I find it? And so like a lot of the effort that went into building system initiative was really just the effort

of just continually pulling on this thread and being like, oh man, you know- You have to build everything. Yeah, if we actually want this to be better, I have to just, you have to just keep tugging and be like okay yeah well nope we can't make that feel good because this other thing is in our way so i guess we got to rethink how that thing works so that we don't like lose all the good parts you know like if there was a if there weren't integrated change sets in system initiative and instead it was just like YOLO mode all the time,

you would reject it immediately as a toy. Yeah. Right. You'd be like, Oh no, we can't, I can't use this thing. Right.

We get fired. Yeah. It'd be insane. So like, that's why it's taken so long is that it can't just be one piece where you're like, Hey, look at this cool toy I made where you can like,

you know, build a little diagram and it like does some infrastructure stuff for you. No, it has to be like a full fledged power tool that actually like proves that you can do some complicated, hairy shenanigans with it. This episode is brought to you by our friends at Drada. Automate and accelerate your SOC 2 compliance, your ISO 27001 compliance, and many, many more compliance frameworks with a suite of more than 75 integrations. Drada easily integrates with your tech stack through applications such as AWS, Azure, GitHub, Okta, and Cloudflare, and countless security professionals from companies including Lemonade,

Notion, and Fivetran have shared how crucial it has been to have Drada as a trusted partner in the compliance process. They have deep native integrations that provide instant visibility into a security program and continuous monitoring to ensure compliance is always met. Drada allows companies to see all their controls, easily map them to SOC 2, ISO 27001, and many other frameworks to gain immediate insight into framework overlap. They are the only player in the industry to build on a private database architecture from day one, meaning your data can never be accessed by anyone outside your organization.

It is time to say goodbye to manual evidence collection and hello to automated compliance by visiting drada.com slash partner slash changelog. That's drada.com slash partner slash changelog. They are bringing automation to compliance at Drada speed. We're getting to the point where we're talking visual because you described diagram.

We've had the pleasure of watching your five-minute demo, which is essentially you're laying out a diagram that seems like a model of what should be infrastructure. And you're connecting Docker images and you're correcting them along the way. And you're choosing Fedora Core OS and you're choosing EC2 and deploying. And this is a model. Can you describe as best you can what that diagram is? Do you call it a diagram?

What is the interface for a system initiative? What was that demo? Yeah, so it does sort of center around the idea that there's, we struggle with what to call it. So yeah, we call it, we call it the diagram. The diagram. Okay. But I'm sure it's going to need a better name. I'm sure there'll be other diagrams, you know, we've called it a schematic in the past, but like, you know, what's interesting about it is that essentially you need a way to think about the relationships between the models.

So if you're building, we're building these full fidelity models. So we're not abstracting things from you. We're not translating them into a different domain. So like, if you know how Docker works, you know how the model works for Docker. If you know how AWS works, you know how the model works for Docker. If you know how AWS works, you know how the EC2 model works. And then visually, we're letting you create sockets, essentially, that have data that flows through them. And so you can connect the data about a Docker image to things that can use the data.

So an example here is like Docker images have a socket that exposes their port numbers. It just, if you have port numbers that you've exposed in your Docker image, images have a socket that exposes their port numbers. It just, if you have port numbers that you've exposed in your Docker image, there's a socket that will take that data and emit it. And then visually you can just connect that socket to something that takes it as an input. For example, an ingress rule in AWS that sets up a firewall rule that lets that traffic flow through. And then it will automatically translate between the shape of the data in the Docker image and the shape it needs for the ingress rule. Yeah. So in the Docker image, it's like 80 slash TCP is the syntax. And then in the ingress rule, you need the port number, but they don't care about the protocol, blah, blah, blah,

blah, blah. Right. And so when you think about doing all that stuff in code, it's actually kind of tough because you'd have to like describe the relationships. How do you do it? What's the declaration look like? But if I just take all of what is fundamentally code in the end and I slap a visual interface on it, it's suddenly quite straightforward because you're like, oh yeah, I can just, I can click on this thing and drag to the other thing. The fact that behind the scenes, it's all software, it's all JavaScript functions, and you can just call them and manipulate them. That's

true. But it's a lot faster to just do it by putting them on the screen and connecting them. And it's certainly easier when you think about collaborating with other people, because you can visualize it. You can see it and be like, well, this is what the architecture is, and this is how it works. And it's kind of laid out that way. But it's mostly a diagram of configuration more than anything else. Yeah. It's easy to go from that visual artifact to the code, which is sort of in the sidebar,

but it's easy to go from, you know, what would normally be YAML files, essentially, or some sort of config files, pick or battle. Like, that visual interface is the natural way humans think anyways. It is. And if you think about how that, then the YAML file gets generated,

it's just another function. Its inputs are the object, right? As the model. Then we have a function that emits code and you can generate it and write it yourself. So if the model needs to emit some YAML, you attach a function to it that emits it as YAML and maybe manipulates the data and gets it in the right shape or whatever in the middle.

And then every time you change the information on that model, we automatically regenerate the code for you, right? So this is how we do things like populate user data in an EC2 instance, right? So you can go from a Docker image to the Butane configuration, which then gets translated into a thing called ignition, which is a JSON syntax for configuring a Linux instance running Fedora core OS. And then that needs to get packaged up as user data, which is base 64 encoded to get to the other side. Then all that just happens automatically for you because we've just written some functions that are like, yeah is in user data base 64 encode it pack it in yeah and then you don't have to remember

to do it anymore right it just sort of happens because it's in the model it's pretty cool sounds cool sounds like the opposite of what we have so you know we've been toiling on our system and gerhard has spent years experimenting changing things you know swapping out different component parts and when he gets a nice system that he thinks works really well you know he goes out and he painstakingly creates a diagram that shows the system and it's like this is just starting from the other way you just start with the diagram yeah right or the schematic or the blueprint maybe blueprint's a cool word i don't know blueprint's kind of overplayed like a lot of people say blueprint exactly that's why i didn't call it a blueprint i'm like if i call it a blueprint everybody's gonna be like oh man that's

not a blueprint and if i call it like and like it is kind of an architecture diagram but it's not like it is really a configuration diagram it turns out that that tends the the configuration relationships also tend to be architectural do you know what i'm saying yeah like they're those relationships tend to mimic the architectural layer but But one of the things that I think is most promising about the approach is that once all the data is in a model, that's like a living thing. So instead of it being locked up in code, now we can start to add layers on top of that data. You can start to like, imagine, oh, well, if we want to show an architecture diagram, why wouldn't we just allow you to express

architectural relationships on the same components? And that could just be a different layer. It could be a different diagram. It could be a different way of viewing the same information, right? You almost need a brand new word that's not currently used, like maybe something like ka-chow.

It's the ka-chow. That's what I'm going to call it from now on. There you go. Yeah. Until you think of something, there's a good placeholder at least. I'm CEO, so I get to do what I want. That's the ka-chow. That's what I'm going to call it from now on. There you go. Until you think of something, there's a good placeholder at least. I'm CEO, so I get to do what I want. That's true. I'm kidding. Ka-chow. It's not at all true. I have a team of incredible

experts. I was just agreeing with you because it sounded like it might be a good idea. No, they'll 100% stop me from calling it the ka-chow. Okay. Well, when I was watching this demo and I was seeing you kind of lay this system out from start to finish and then you said, okay, this is a working model because you can see the relationship between okay this would work on EC2 or this docker image would

not because it doesn't support X and then you made some changes that all seem very logical yeah as you were going through it and I think at the end if I understood it correctly you clicked some sort of button and it was like just like the button just basically said make it true true. And then boom, it was in real life. Yeah. And you went to cat website and there you go. Yeah. That's pretty much it. That to me seems very logical, but what are the components in there that make it so hard to pull off that visual to reality, like the relationships and the communication and the interoperability, et cetera. So, I mean, the big one is just, is separating out the sources of truth.

So infrastructure as code taught us that there should be a single source of truth and that the source of truth for that should be the code. And there's a bunch of reasons why it does that. But the biggest one is actually that there's no way to do it the other way anyway, because when human beings write code, we write code and then we refactor it. We structure

it differently. We put it into modules. We add all these semantics to it that you can't just serialize and deserialize, right? It's more than just the YAML. It's where's the YAML on disk? It's how's that YAML relate to the other files on disk? How's it get committed to the repository? There's like a lot of information in there. And so when you imagine trying to go from a real world thing to the code, you can do it. Like there's, there are projects that like generate you Terraform or generate Pulumi or whatever, but you don't live in those projects. You do them once and then you tweak them so that they make semantic sense to you. Right. And so when you think about how you apply that stuff to the real world, you think of it as a reconciliation process where you say, well, I have this model

that's trying to be as full fidelity a thing as it can be of the way that whatever the thing I'm dealing with, the AWS instance, the EC2 instance I'm dealing with, for example, thinks about itself. And the closer those two things are together, the easier it is to go bidirectionally between them, right? It's easier to say, well, I don't have a resource that exists in AWS that I can find that looks like the one you've described. Therefore, the action you should take is probably to create the EC2 instance. And those can just be, again, functions on the graph. So like if the model, for example, doesn't have a resource attached to it, then most likely the action to take is creating it. But in the case of an AWS instance, if you update the tags, for example, on that EC2 instance,

what you probably want to do is call the tags API to just update the tags. You don't probably want to destroy the EC2 instance and create a new one just because you wanted to update the tags API to just update the tags. You don't probably want to destroy the EC2 instance and create a new one just because you wanted to update the tags, right? That feels wasteful. So that's a different kind of function that would then look at an existing resource and go, oh, hey, I see that you do have a resource.

If I look at the model and I look at the resource, the tags aren't the same. Now I have to let the user choose, what do you want to do? Do you want to update the tags in the model so they Now I have to let the user choose. What do you want to do? Do you want to update the tags in the model? So they're reflective of the real world because maybe somebody logged into the AWS console and just fixed the tag because they had a problem. And, and that's actually the right set of tags. So in system initiative, you can just say, well,

update the tags on the model. So now the model matches and we're cool. Or it could be that they're the wrong set of tags that like you in fact have updated the model and the model. So now the model matches and we're cool. Or it could be that they're the wrong set of tags that like you in fact have updated the model and the model is correct. And what you need to do is change the world by updating the tags. And all of that basically can happen relatively easily because we've turned it into a problem of reconciling the data and then taking action as opposed to having to declare the correct state up front. So in system initiative, where does this graph live? I mean, in system initiative? In Postgres?

Yeah, but what's that mean? Okay. It's like, yeah, in a lot of ways, it's inside of the database and then in some Rust code. And then if you actually dig into the architecture, a lot of system initiative is like old techniques applied in new ways. So like, there's a lot of very like dope stored procedures that like actually work on a lot of this data, or like the way change sets work are baked into the database. And through stored procedures, that's actually a kind of a cool tagline, like powered by dope store procedures on the hood you know kind of you know like we built we built versions of it in every possible way before that was the one we built and it turned

out that it was so it was just too hard to work with until you made a database that was essentially feels custom from the outside because you interact with it mostly through these stored procedures that handle all of this bookkeeping of like, okay, you want to run this function because these inputs have changed. What's the calculation for what the inputs are? How do I know they've changed? How do I decide which other functions to run? They're all in a database

and you can query the database and it'll tell you. So you tried other databases than Postgres is what you're saying? Yeah. Oh, I tried every possible version of not putting it. The last thing we tried was putting it in stored procedures because like right yeah that's not a thing you normally want to do

you can avoid it yeah but like it's the one advantage of being 45 is that like i was working when we did that you know like i worked with dbas you're not afraid of it you're just like we can do this yeah i was like you know what would work? Right. It's like, what if we just put it in the database and then did it that way? And it did work and it was great and go figure.

But, you know. I actually think that, I think we're going to start trending back towards put it in the database. I'm starting to see it already so this might help in that way.

Like, hey, Adam Jacob got away with it. Because, you know, we go away from it and we come back. That's what we do in software. We swing the pendulum. Wasn't there something

recently we were going to do to put it in a database? Was it like a cache or something like that? Yeah. Is that worth sharing a micro story version of that? Like just to commiserate with Postgres? Well, I'm just putting some data in there that is probably unwise.

He's putting code in there. Right. So a little bit different. Well, the code's in there, but it's also data. Yeah, sure. So like the actual execution of the function is not in the database. It's like, that's out.

There's like a whole separate subsystem that knows how to do that for you. But all the bookkeeping about like what to run. Just one big eval, basically. Yes, exactly. It's just- You're just evaling our JavaScript at the end of the day. Yeah, I'm just running it all in the browser through eval.

That's the answer. Don't put that on the tagline, yeah. System initiative powered by stored procedures and eval. And eval. That's the answer. Don't put that on the tagline. System initiative powered by stored procedures and eval. And eval, yeah. I mean, you get some hacker credit. Dope stored procedures. It's not wrong. All right, so let's imagine that I'm coming to system initiative. I go down to Circuit City, I'm going to buy one.

Yeah, you're going to buy a system initiative. I'm going to buy myself a system initiative. What part of my infrastructure am I replacing? Or what do I not need? Because there's still a lot of components that you're talking about Docker. I know you're talking about Kubernetes. You're talking about blah, blah, blah. So by design, we're trying to meet people where they are.

So one of the things that is hardest about adopting new DevOps tooling is that in general, it sort of forces you to throw away what exists. So today, it's still pretty early on. And so some of these features we've had working in earlier prototypes, but we sort of stripped out in order to get it shipped. But one of the great things about being able to go bidirectionally is that you can start from your existing infrastructure and build a model from there. So rather than onboarding being, hey, rebuild all your stuff in system initiative in a not too distant future, we want that to just be give us your AWS credentials. We'll look at everything that's in AWS and then we'll build a model backwards based on

what you have. And then if you think about the bidirectionality of it, if that model, if those resources were built by Terraform or by Pulumi or whatever, CDK, like you can keep doing it that way. And what system initiative is going to do is notice that they changed every time you run it. And so it's just going to be like, Hey, you want to update the model? And you're like, yeah. And now you can visualize it and see it and you can keep doing it. But then you're probably going to be like, you know, it would be easier than futzing around in this Terraform module. Like I could just maybe click on the instance that I want to reboot and say reboot and then hit the apply button and then it would do it. And then you could see it happen and it would just happen visually. So the goal

is more that than it is like rebuild all your stuff around system initiative. I think the number one thing that it changes for now is that sort of infrastructure as code layer. So if you think about all the things that go into doing infrastructure is at code well and at scale. So, you know, when you think about how do we actually define it, how do we collaborate on it? Do we need deployment pipelines? Do we need, you know, reviews, all that stuff. We're streamlining all of that stuff first, partly because it's the world that I know best.

And also because when you really look at where the pain is in the DevOps workflow, a lot of it's right there, you know, like that's actually where, where a lot of the paper cuts live right now. Not because those tools are garbage. They're not. But just because the way they're, how they fit into the systemic workflow, the overall arc of what we do, it causes a lot of downstream problems. Is this the death of GitOps? I mean, is it? I hesitate to say it's the death of anything. Because like, I'm full of hubris, right? Like I got plenty of ego and I even have some aspirations around like feeling more confident in my own ego, if that makes sense. So like I'm more than willing to be like,

no, I'm great at this. But like, is it the death of it? Like- Well, let me rephrase. If this is successful, is this the death of GitOps? If it's successful, you won't need it. Okay, it's the death then. What else won't you need?

Well, you won't need infrastructure pipelines anymore, it's the death then. What else won't you need? Well, you won't need infrastructure pipelines anymore. You won't need continuous integration pipelines. None of that for your infrastructure you won't need anymore. It'll all be done real time. You won't need pull requests and code reviews. Instead, you'll replace them with

like, think about it this way. If we know that the model, none of this is like, this is all stuff we're going to build, yeah. We're aspirational now. This is all aspirational. So just, just so we're clear, I'm not saying you can come do this stuff right now. If you think you can, you're going to be disappointed, but this is what we're building for. And we know it will work because we've built prototypes where that worked this way. So like, you know, you can say like, imagine that you need a DBA or someone, let's say I need a DBA and a principal engineer to approve any change that requires rebooting the production database.

And so Jared and Adam go and whip up a change that requires touching the production database that's going to require the action rebooting it. So rather than just being able to apply that action immediately, instead of pulling it into a PR, we can just use the data about the impacted objects and say, in order for this to be applied, you need the real-time permission from these four people. And then they can just log into System Initiative and in real real time join your change set and see the screen that says hey adam and jared this is what they want to do here's all the things

they said like they want to do it right now say yes or no and it's a little more like submarine captains launching nukes which has a weird analogy and i'm sorry i used it but i like don't have a better one and it feels really evocative from you know like one ping and only vasily you know requires like two keys right it's like two yeah you turn at the same time right it requires synchronization right star trek self destruct sequences or whatever it's not async no this is a synchronized action because it's about collaboration again because right because if the dba is like why do you got to reboot my database? Like you actually don't want him to ask you that in a PR. You want him to just say it to your face, right?

And so you can just literally look at the interface and be like, say it to my face. Just- This is why. This is why. And this makes sense. So yeah, sure. And so like all those things could start, you can imagine all those things sort of fading away.

The same thing around like, when you think about like continuous delivery pipelines. So the way we talk about application delivery today is often through a series of activities we take on infrastructure, right? You're like moving something somewhere, you're building a new thing, you're doing that kind of stuff. Like what if we turned those into workflows that you could collaborate on and then trigger with inputs as part of the API? So instead of having like a continuous delivery pipeline that defines all those things outside

the flow, what if they were just calls to models that said, find all the models related to this thing and call this action on them, and then we'll trigger it under this trigger, right? So like over time, the ambition is to rebuild the whole, the whole shoot and match. Like the whole thing could work differently. And if it worked differently, it would be better. And it's ambitious and nutty. And also we're not wrong. Like it is in fact better. And you can see that it's better already. It's just going to take a lot of people and effort and time to understand like all of

those suggestions. They might be better, but no one's tried them. Like we know all the ways that code review is bad. We know all the ways that the existing process fails us. We don't know what the alternatives are when you really try to use them at massive scale. And so we got to figure that out together. Well, you said together, and I've been thinking a little bit about community because you have a long tradition of open source and open things in community. You have Chef in the back pocket,

you're 45 now. So you have lessons learned the hard way, the easy way, and you get a chance to restart, right? Like you're launching a new thing now. What are you doing differently? Or what are you thinking about differently that hopefully will be better or successful with System Initiative? I mean, in terms of open source, like a big one is that it's not open core and it won't ever be. Because A, I think that that model sucks to execute.

It's just no fun on the inside, and I don't think it's better for your community. But B, I just went on this whole ambitious rant about rebuilding the entirety of the DevOps workflow from scratch. And that's not a thing you do alone, right? That's only a thing that happens because other people are like,

yeah, I agree with you. I think that it could be better. And I like your groove, right? And whether they like system initiative or not, I hope what it inspires people to is to like break a bunch of rules. Like we need to build more wacky stuff because we need to see if it works. And we're kind of not doing that right now. We're building a lot of variations on the theme.

And like, I want to see people, I want to see a second wave of DevOps that's full of new, interesting stuff. And so part of what we're doing is open sourcing all of System Initiative. So it's the Red Hat model. It's the same as what Chef's model was in the end. So every single piece of software that we use in System Initiative, all of the System Initiative software is open source. And what you're not going to be able to do is make a distribution of it and call it system initiative. Only I get to do that. And that's, that's it. And we're doing that because I can't invite people to explore what's possible with me

and to see if this new way is better while simultaneously telling them that I'm the only person who's allowed to make money off of it. Or I'm the only person who's allowed to better my life and the way I want to with it. Like I can say that, but what it means is it's not about actually about the outcome. It's about me. Does that make sense? Like it means it's about whatever the money in my pocket,

which it is like, I want money in my pocket. I think I said that on one of these podcasts earlier. Like, yeah, you have, I'm not, I got no,

I got no shame about it. That's totally cool. Right. It is great. It's a lot better to have money than not to have money like i think anybody who tells you otherwise is wrong but it's not the only reason to do it and my ambition our ambition for it is a big one and you can't really think about how to get there it's impossible really to me to imagine how to get there without building a giant community of people who share that point of view. And if you want that to happen, I can't simultaneously believe that and also say,

but here are all the ways you can't make your life better, right? If it involves you making money in the way I want to make money, I don't like that. Or if it involves you doing something, anything I don't like, you're not allowed. Or if you disagree with me in some fundamental way, and you're like, Adam's totally wrong. What system initiative needs, what the system initiative software needs is to work with Git. Then, you know, you got to be able to fork system initiative, make the Adam initiative and like, which you shouldn't call it the Adam initiative. Cause I'd come find you and be like, that's too close to my trademark, but you know what I mean? You can call it LaCroix or whatever, let them sue you and make it work with Git, you know? Yeah. Get more specific with the Red Hat model. What do you mean when you say,

like, for those who don't know the details of the Red Hat model, what does that mean for how the software gets distributed, used, deployed, how you make money from it, how you hold a trademark over it, the name? Break that truly down in practical terms. All right. So there's three levers in sort of open source business models. There are no open source business models, open sources in a business model, blah, blah, blah, blah, blah. Let us just admit that that's true. And then we're going to go on calling it open source business models because that's just

what you have to do. So open source business models, three levers, copyrights, trademarks, patents. All three of those things are like, you can think of them as just ways that you create scarcity. So, and then in order to get people to pay you for things, there needs to be scarcity. So you don't tend to pay for things you can get for free is the way to think about it, right? So the one people are most familiar with is copyright.

So we say, hey, I made system initiative, therefore you can't have it unless you pay me money and if you think it's valuable you pay me if you don't think it's valuable you don't no harm no foul that's the copyright lever the trademark lever says only i am allowed to make system initiative and call it system initiative and if you want to build something similar and also call it system initiative you can't because that's my brand that's the thing i do that's how i make money that's the way that's my brand promise and then patents are the third where you can't because that's my brand. That's the thing I do. That's how I make money. That's the way that's my brand promise. And then patents are the third where you can be like, Hey, we do this

in a novel way. And anybody, nobody but us are allowed to do it in this novel way. Because if you try, we'll find our little patent and we'll be like, waggle our finger at you and be like, you can't do it because you know, my patented algorithm says you can't unless you pay me money. So most open source business models are hinge on copyright. They say by keeping some amount of features proprietary, and so you can't have them without paying me money, then I find someone who's willing to pay for those features, and they do, and therefore we're happy. Or you say, Hey, here's my SAS, you know, and I'll run that here, but I'm the only one who's allowed to blah, blah, blah, blah, lots of shenanigans sort of around copyright. So the red hat model says, okay, the actual value, the value people get is in products, not in software. So when you buy a product, like for

example, when you record this podcast, y'all want us to record a separate audio track just in case the one we're doing right now doesn't work. And I'm using audacity to do it and it's not a very good product. And so if, I mean, it is, I'm so sorry, audacity people. Gosh. Oh no. I just really made a boo-boo. Provided a lot of value for a lot of people. It's providing a lot of value for a lot of people, but I really struggled to get it to work. Yeah, because I don't use it very often. It has challenges. It has some, right.

I'm so sorry, Audacity people. It's great. And it's working for me, God bless it. But like- Right now. So if you think about that software, it's valuable to me because I could get it and just use it.

But my mom, not so much, right? If I threw Audacity in front of my mom, not good. But my mom, not so much, right? If I threw audacity in front of my mom, not good. But my mom could use Riverside, which is the platform we're using that records this podcast, because you basically log into Riverside and talk. And then it's all done, because the product is better. And it's not just the software. If I gave my mom the Riverside software, that's not better for my mom. She's like, it should be like, how do I run Riverside? And you're like, well, you launched this service and this one, you got to don't forget to configure the Side software, that's not better for my mom. She's like, it should be like, how do I run Riverside? And you're like, well, you launched this service and this one, you got to don't forget to configure the S3 bucket you stream to and like all this stuff you got to do.

So products are the full experience that someone has around getting the value that they desire, right? And it's not just the software. The software is a big piece of it, but it's not just the software the software is a big piece of it but it's not all of it and what red hat does is say yep we sell enterprise products for money and we build them primarily out of open source software so red hat makes a billion dollars in arr on top of kubernetes selling open shift every single thing you get in OpenShift is open source. You can have it for free right now. They make a billion dollars a year with a B. A billion dollars a year. One billion dollars.

Whoa. Right? One billion dollars. And like HashiCorp, they don't make a billion dollars a year. That's a public company that like is the incredible success story. Nowhere close to a billion dollars a year in ARR. Nowhere close. Why? Well, Red Hat takes software and sells it for money exclusively.

If you would like the product that is OpenShift, the vetted experience that comes with Red Hat, there's only one way to get it, and it's to pay them. Feel free to take all the piece parts and try to cobble together some OpenShift yourself. God bless and keep you, and some number of people will. And that's great for them because it keeps you in Red Hat's ecosystem. You don't leave Red Hat's ecosystem even when you don't want to pay Red Hat's ecosystem. You don't leave Red Hat's ecosystem, even when you don't want to pay Red Hat, you're still there. Because they provide so much value in the software that even when you don't want to pay them, you're still hanging out using their stuff. And that's such a better

competitive advantage, right? Because now you're not competing, you're not competing on a closed source basis, you're not competing on your software specialty. You're competing on the experience of what it's like to receive it. And the people who have the most money and will see the most value in what we build tend to be people who need the experience to be better and they're willing to pay for it. And so that's the business model. And it's better because you don't hold anything back from the community. It's better because it's more straightforward to execute because it's the business model is really easy. Like we build

system initiative and we sell it for money. That's the business model. And if you want it for me, depending on how you, what you need, you have to pay me. And if you don't want to pay me, that's fine. You can use the software. Now, you might have to get that software from somebody who's not me because I produced that software to sell it for money. But that's cool. You don't mind because you don't want the product

for me anyway because to get it for me, you got to pay for it. Yeah. So in the future, when it's open source, I'll be able to download or do something with- All of it. System initiative. And I put it on my own server and I have to orchestrate it

and fine tune the database and make sure the database works and the APIs are all working. And you've got to call it something else, you've got to remove all my branding. Well, no, I don't have to if I'm just using your thing. I'm using it for me. Just to run it? Nope, you can't get it from me. If you want to run system initiative, you have to get it from me

under my commercial terms. Now, my commercial terms will likely be incredibly easy for you to get it for $0. So for example, you do this every day with VS Code. VS Code, the thing you download is not open source.

That is a commercial piece of software that happens to be $0 and you don't even notice. So for some huge number of people, system initiative will be basically $0, right? But then for other people, it won't be. Because VS Code that you download is packaged and wrapped by Microsoft and therefore, quote unquote, sold by them. Using the VS Code trademark and a distribution license to change the terms on which you get that distribution,

even though every line of code in VS Code is open source. Right? Right. Okay. And so there are people that will take the VS Code open source stuff and they'll repackage it with a name like... VS Codium. Yeah, exactly.

And they'll do what you're talking about, Adam. And they totally get to. What they don't get to do, though, is use Microsoft's extension store. Because Microsoft builds services that make VS Code better, and the only people who get to use those services are people who use VS Code.

And that's likely the same thing that happens with System Initiative. There's services that make System Initiative work that everyone who's using System Initiative uses. Those services are open source. You can take them, you can run your own versions of them, you can do whatever, but the data that's in there, it doesn't come with them. You know what I mean? Like the, those things aren't, but we'll work with people to do it. Like if that's what needs to happen, like I'm down to collaborate, like I want system initiative in the world. I want it to succeed. I, I want that software to win. I think it's better. So, you know, when those things come up, we'll cross those bridges when we come to them.

When you explain it, it makes total sense. But then when I walk away and try to tell somebody else about it, I'm always like, you see, it's free, but you still pay for it. You see. And they're like, why? Because you like what they're putting out there. The struggle is that you start by saying it's free. It's not free. Well, you know what I mean? I do, but that's what I'm, this is why I'm telling you, this is how you solve your problem, man.

I'm helping you get through this as a brother. Okay, I appreciate that. And what you got to do is just say, like, it's not free. The software is open source. Like the software. But the product. But the product's not free. You have to differentiate between the product and the software.

That's right. And we buy products. And it's important that the software product, but the product's not free. You have to differentiate between the product and the software. That's right. And we buy products and it's important that the software is open source. It's important that the software is free because this is software that if it's successful runs the world and it's better. Everything I know in my career happened because people open source their software and I got to understand how it worked by looking inside of it and figuring it out and, and, and munging it and doing all of that. It's incredible. And like, it's a unique thing that software can be not a zero sum game

in this way. And the thing you just have to believe as a business person is that sometimes the products we build are best served by not being zero sum games that like the, that the path for them to become most successful in the world. And for them, therefore my business to be as successful as it can possibly be is actually that it go out in the world and just be valuable to whoever it needs to be valuable to in whatever way they find that value. And the more you try to constrict it, the worse you make it. And I think it's better. And the hard part is that most people don't actually believe what I just said. In the end, if you push them, they believe that because they built it initially or because it was their idea, that they deserve more.

And I don't think I deserve more. I think I'll wind up with more. I think that's probably real. If I'm a good steward of it and I take good care of it and I do the work of building a great company and finding incredible people and putting them against that problem and building a big community,

all that stuff will bring good things to my life, unquestionably. But that doesn't have to be at the expense of other people's ability to do the same thing. Does a license matter in this scenario? Like, do you choose a particular license to ensure the war for the soul of open source survives this initiative you're on?

Thank you, Salil. You know, I think that was a joke. That was a deep cut. I didn't catch it. MongoDB. Okay. Wrote that blog post with the soul for the war of open source or whatever.

Okay. That's totally going to get edited out of this podcast. Okay, so. Nope, it's in there. Nope, it's open source or whatever. That's totally going to get edited out of this podcast. Okay. So it's in there. Nope. It's in there forever now. It's who we are. Okay.

So I think, I think one of the things that happens here is you, I've totally lost the plot now because I'm thinking about how it's not being edited out. Licensing. Yeah. So the Apache license is particularly good for this model because it's very explicit about not granting you any trademark licensing. And so when you look at licenses like the MIT license, which is also a good liberal

license, it doesn't say anything one way or the other, which most lawyers will be like means you don't have those rights. But lots of humans on the internet will be like, but you don't know, you know, and then you just wind up in all these ridiculous arguments about what people can and can't do. So I like the Apache license quite a bit for this I think the MPL would also work just fine for it I think if there's a piece of you that really loves the GPL then get on with your bad self lots of GPL software and Red Hat products

so if that's what grooves you go for it so do you Apache license across the board then in this case since this best fits? So all the open source system initiative produces will be AGPL. Yeah, not AGPL. Apache license. Sorry, Apache.

Yeah, and I mean the only things that maybe won't be are the code. We're going to run managed services. We'll run SaaS services. We'll do that stuff. So like the code for us to like run those services, we probably won't open source it, but it will be if we need it to be, you know what, like if it's better for our engineers, if it's better for the way we build it, then like I would, I wouldn't necessarily hold those things

back. I don't believe that I have to hold them back in order for the product to be valuable. I just think it's also, it's just weird to imagine how you would in a way that allows you to do the things you need to do safely and securely. And like, it just gets weird. I'm Jared, and this is a Changelog Newsbreak. DeviceScript is Microsoft's new TypeScript programming environment for microcontrollers. It's designed for low-power, low-flash, low-memory embedded projects, and has all of the familiar syntax and tooling of TypeScript, including the NPM ecosystem for distributing packages. This project has a lot of devs excited. Jonathan Berry says,

quote, dope. TypeScript for hardware. Always glad to see these attempts at bringing web technologies to embedded systems and see what sticks. Even when they don't, they inspire innovation. Zach Silviera says, quote, this is so much better than MicroPython. And Andrea Ghiomarchi says, quote, this is the first Esperino competitor, and I think it's going to be huge. You just heard one of our five top stories from Monday's ChangeLog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com slash news to also receive our free companion email with even more developer news worth your attention.

Once again, that's changelog.com slash news. Does things like development environments in the cloud play a role in system initiative in the future? If you're doing more things in this sandbox, this toy box, this hypothetical that could be true, that maybe mirrors true or gets sucked in as a mirror of a true, is that a thing in the future? Yeah, it is in many ways a dev sandbox in the cloud already. The IDE is built into the product.

Like that stuff is kind of already there. So it definitely draws inspiration from those things. We'll see what the future holds in terms of like bringing your own IDE or like some of those sorts of pieces. It's hard to imagine how you make that user experience as good as you want it to be because so much of what you're doing is this like tight iteration between tweaking some code and then seeing it happen in the product good as you want it to be because so much of what you're doing is this like tight iteration between

tweaking some code and then seeing it happen in the product and then like how tight that loop can be is really important so i don't know like we'll see what that is i think it's going to be like the number one thing people bring up they're going to be like can i can i write all this code in my own ide and it's the answer is going to be like maybe? Well, are you constraining this to infrastructure code or is this all the code in your system? Infrastructure code for today, yeah. Because it feels a little nutty. And I'm not sure that as an application developer,

I actually find the experience of writing modern applications quite delightful. As an application developer, I'm not suffering like infrastructure operations developers are suffering. It's pretty good. My IDE is kind of great. I'm getting incredible AI-assisted search.

Things that are happening for software developers where the outcome is software, that experience is pretty on point right now. But boy, the infrastructure side isn't. I was just trying to see the the scope of your ambitions there because as you're talking about like a new change control system collaboration built in all these other things you start wondering like well if it's source code at the end of the day yeah yeah and all these things are like great then maybe they're also eventually can replace other things that we know and love.

But how weird does it get? I don't know. Yeah. But I really want to find out. But like one way to think about it is you could think about the application source code as an input to the hypergraph. Well, it's got to get in there somehow, right?

Like that's what ultimately gets deployed, right? Yeah. So if you think about it as an input though, then you don't have to think so much about, well, I have to alter my tool chains or any of those things. It can just be an input that then informs the behavior later on. So rather than thinking about modeling, like writing all the source code in a system,

it should keep it there. Maybe it's more understanding what the outputs of that process are and then using those as inputs to how the system needs to change when the when those inputs change. Cool. So what does it look like today? You described it tomorrow, we've described it a little bit today, but let's get like the exactly what you have on offer. Yeah, so today, it's really ready for the like, most DevOps builder of builders, right? So if you're a person who like is an experienced open source developer, you are experienced in building and managing DevOps tools. And you think what I'm talking about is cool. It's ready for you to come and check it out and see if it's as cool as we

think it is. And then to talk with us about what's possible and imagine with us about what's possible and write code if you want to, to make the system do the things you want it to do. Right now, it's mostly good for like deploying containers to EC2. But that's going to change pretty rapidly. The goal is like, we'll be increasing lots of that coverage, we'll be open sourcing it right as soon as the customization features are in. So one of the last features we want to add before we open source it is the ability to basically package up your customizations and share them. So if you make a new asset or you change it, add a function to something that already exists, then you can package that up and share it. And then from within system initiative, you can just install those directly. So once that is landed, we'll open source it and

it'll be ready for those early builders to come and dig in. I don't know that it'll be ready. It'll be a minute before I think it's ready for people to just show up and consume it. Because to your point from earlier, there is a lot of coverage that needs to happen. There is a lot of exploration that's needed. But it's ready today and super fun to explore with it and see what can be built. Is the feature where you can have a production environment and mirror it into system initiative, is that close or is that near? Is that today? How far away is that? Yeah, no, it's going to be, I think it's probably going to be more like toward the end of the year.

Okay. So for now it's Greenfield, brand new, created in system initiative, merge it elsewhere. Totally. And I think the work to figure out how to do that bidirectional discovery, it's in flight right now, but the user experience is really the tough part. So thinking about how it works in the under parts, do you know what I mean? Like in the graph is relatively straightforward. Thinking about what the user experience is

for letting you choose, that's where the hard stuff is. And we'll do that through lots of user studies and lots of testing. So is this a SaaS, essentially? Will somebody go there whenever it's launched today essentially to play with it? Do they have to put a credit card in?

Is it profit growth? Is it PLG? So right now, it'll be software you download and you can run it or source code you check out and compile. It will be SaaS software as well. So the first one will probably be like a bring your own cloud kind of SaaS. And then roughly when that discovery work lands is when we'll turn on full multi-tenant, show up, bring a credit card kind of SaaS.

Because the onboarding experience will be dope. So dope. Okay. Cool. System initiative. Wow. It's fun being here. Like we started so many years ago. Was it 2018, Jared?

2019 when we first had Adam on? I was like, I don't remember the first time we had him on. Don't ask me such difficulties. It's been a minute. 2021 was the last time, but I don't know what the time before that was. The war for the soul of open source was july 16th 2019

something with like june and july for you adam yeah well you know that's uh it's the time i think yeah i mean what i i was so pleased that you all wanted to talk to me again about this um in part because it's so fun to hang out with you and you're like your perspectives are always like delightful and i think the there is something about june and july i mean part of it is that it has to be june because if it's july everybody goes on vacation yeah forget it right forget it you can't do anything useful you want to launch in july it's like launching on a friday well i think oscon was in july wasn't it like end of july that year it was this 2019 when we first met you, at least via voice. Which must have been the last OzCon, right?

It was. It was, yeah. What a bummer. Yeah, it was. Rest in peace, OzCon. I know, I was so bummed when they said, they never even hesitated. Our whole conference's divisions just closed immediately.

I was like, they must have been waiting for this opportunity. It was opportunistic almost. And what a sad ending to such an important institution. And I feel like that void's going to get filled, but it does bum me out. Like I said, I was watching that John Ospaugh and Paul Hammond talk that they gave at Velocity in 2009. And there's so many examples of talks like that

and conversations like that that they just don't happen anymore. Now they happen on podcasts. You know what I mean? Right, yeah. Now they happen here. And that that's fun but it's not the same as going to oscon and i kind of think that's kind of almost better because talks are very one-sided they're not interactive in most cases not that they're bad it's great to put out like a hypothesis or a thesis or a big vision but i love these interactions because like you don't just get to rant at them

we get to like push back on We stop you. some of your ideas. Like, hey, hold on there. It is ka-chow. It is 100%. And I, look, I'm with you. Like, and I would rather prep.

I'd rather do this all day than write another talk. There's room for both. There's room for both. Well, I'm being biased here because I love this format. It's just such a great format. I mean, people are listening to us

right now in the shower. Someone is washing their hair. Yeah. And thinking that show i'm watching cars 3 like that about i hope they looked at their hair in the mirror and were like ka-chow you know i hope they like yeah i hope they like tossed it back with like luscious like luscious lock yeah right the point i was getting to was was the war for the soul of open source back in 2019. And then, you know, we had this, this actually predates the elastic search AWS kerfuffle.

Yeah. The business model of open source that was in 2021. I was like, just paying attention to your tweets. And I'm like, we've got to talk about this on this show. And so we're like, Adam,

get back on here and school us. And then here we are back now at the end of that show, as Jared mentioned at the top of the show, was quoting you as talking about System Initiative and what your plans were and your vision and like, hey, I'm going to come back on here when it's time. And this is that time. That's what I love about this show. Remember our word, you know, we said we'd have you on the show when System Initiative launched. And here we are. Dog nab it. We're going to do it. Yeah, let's get it done. The loop has been closed. The loop has fact been closed and yeah i i'm i'm so grateful for it if you couldn't tell from listening to me talk about it like i feel like i have like years of like pent-up experimentation and like thought that i just haven't been able to share yeah well i asked

you about it last time and you're like i can't talk about it I was like I can't talk about anything and like such a bummer and so like I but now I can and I'm yeah I'm a little I'm a little overly enthusiastic perhaps about talking about it because I just it's that there's so much of it there's like I could go on forever I guess to some degree on that note this stealth mode idea it's not uncommon for a startup to be in stealth mode it's less common these days what what did you like and dislike about this stealth mode you startup to be in stealth mode. It's less common these days. What did you like and dislike about this stealth mode you had to be in? Oh, that's a good question. And what did it benefit you and what did it take away?

In hindsight, I think, I'm not sure how much it benefited us in hindsight, except if I started talking about the things we thought might be cool, it was such a big swing and there was so little knowledge. I think we probably would have over-rotated to the wrong solution too early because, you know, you'd have been talking about what you thought was right and people would have agreed with you and it kind of would have taken the art of it sort of off into a direction that resonated with an audience before the art had actually kind of achieved its like full greatness not that it's achieved its full greatness now but do you know what i mean like

yeah before the album was ready it's like describing a painting verbally and you're like this is an amazing painting there's hills in this painting and it's this great sunset but you can't see it you can't truly appreciate yeah the artifact i think the downside of it is that you do get really pent up about it and really like it the pressure kind of builds up on you a little because it's been a long time and you want people to like it and so a little i feel like axl rose and chinese democracy you know where it's like no i've got this album and some people have seen it and they think it's amazing you know and like maybe it's going to be amazing or maybe it's not. And that album was actually pretty good, but it was like 10 years too late. You know, like if that album had come out the first time you heard about it, it would

have been the greatest album you'd ever heard. And you'd have been like, Axl Rose is amazing. He doesn't need Slash at all. And then like, but you know, 10, 12 years on, you're like, well, you know? So like, there's definitely risks you run. I think, I don't know that I would do it again, be stealthy that long again, but look, if we win and we're right, then everybody will be like, they're geniuses because they stayed stealthy until you got to go stealth mode. That's how you win. You got to go stealth mode until all of the, until it's perfect. And then you'll win. And if we're wrong, there'll be like those dummies. If they'd gotten it to market sooner, they could have known they were wrong and pivoted and like,

eh, you're kind of damned if you do damned if you don't. Yeah, totally. But if we win, we're geniuses. And if we lose, we're dummies. And that's just an entrepreneurship in a nutshell. Along that line, the world can change in the meantime as well. In terms of the software world, you know, the, the, The game that you're playing can change while you're in stealth mode, especially a long one like you were in. Of course, you can pivot stealthily because no one knows what you're up to.

But was there anything that came out? Yeah, I mean, maybe some of this AI stuff, which I think I did notice in your pitch deck there was one reference to something. And I thought, eh, it's like machine learning somewhere. You've got to have that word in there. Was there any moments where you're like, oh, crap, we have to change something? Oh, there were a million, but they were all because we

didn't know how to build it. Like when I say it now. They weren't from external forces, though, from internal forces. Yeah, like something happened in the world where you're like, oh, well, that's good. But that's because it's kind of, maybe this is awful but like it's been a little boring in the devops world in the last couple years you know like kubernetes hit like a bomb what are we doing well you're either on the kubernetes train or you're not if you're not you're like i don't like yaml or kubernetes and if you are you're like this kubernetes and that's it forever and okay you know like but we haven't what's the most innovative thing you've seen in the last four years in in like infrastructure software like

variations on themes for sure but nothing close to docker or kubernetes yeah you know like nothing close to like nothing that where you're like whoa whoa you know, you know? Is this a whoa moment then? I think, I mean. Do you believe this is a whoa moment? He wants it to be. Yeah, I want it to be. But it's not up to me. Yeah, like it's up to everybody else.

What do you think is going to happen is what I'm asking. I mean, obviously I know you want it to be a whoa moment. You think based on everything you know, this is a whoa moment. Yeah, man. Yeah, I want to win. I want your affirmative, Adam. I want your full affirmative here. Yeah, dude, I want the whole thing. Okay. Like, yeah, and I think this is it. I think it's actually, I think we're right. Based on what I've seen, I think you could be right. I think, you

know, you need this visual interface. This visual interface is really a good thing, I think. We connect the dots, this configs update across the board when you make changes like who wants to go and do all that like in the ops world in particular no one does and even if i'm wrong let's say i'm wrong about system initiative as an alternate implementation of this system of doing devops work i am not wrong about what happens when we do it the way we do it today. We know what happens. 14 years in, we know this is what happens. And making small incremental changes or putting another layer on top of it or whatever, it's not gonna move the needle on the outcomes. It's just not.

We have 14 years of evidence that it won't. And so whether it's system initiative that changes the game or something else, there's somebody listens to this podcast and they're like, yeah, I've had a, I've had an idea. I think it, I think it should be like this. And then they go build it and they're going to change the game. One of us has to, someone has to, or we're just going to admit that this is as good as it gets. Yeah. And then 20 years from now, there'll be some podcast where somebody does the moral equivalent of what john and paul did

and blow everybody's mind and everybody'll be like oh it's that way you know and i don't know what they'll call it then dinosaur ops or something but would you mind if i read a quote from your pitch deck is that cool uh yeah that's cool i don't know who it's from it just it's just a quote it says way more intuitive with an exclamation point yes way better than having to write custom Terraform providers

way more powerful and intuitive than the very basic Terraform checks better than repo slash OPA

or OPA or OPA because they still have horrible syntax yeah we'll see how this evolves though yeah so very optimistic

and then like yeah we'll see but that's how everybody should feel about system initiative. Right. That's how you should feel. It's the truth. Yeah. And also it's going to work because you can smell it.

Do you know what I'm saying? Yeah. Like sometimes you're like, oh, yep, this has legs. And like, it's got legs. And I don't know exactly how that works. But the right posture to think about system initiative is to say, well, how could it scale? What could we do if we had all that information about large enterprises? Like what could happen? And once you start asking those kinds of questions, it's over in some ways, because now, now we're doing it together. Now you're in the game. Now

you're like, Oh yeah, you know, I actually do want to think about what that could be like. Could it be like this? And then it's like, yeah, it could. Cause it turns out the way it is was only ever just us talking to each other. What people forget about that conference talk with John and Paul was that I was there. Patrick Tabois was there. Luke Kinney's was there. Bunch of people who started Kubernetes were there, right? We were all there. We were talking to each other. It was like, we were feeding into each other and it's the same loop that's going to happen and so like

once we start talking about the future like that that's it right that's the game okay well that's exciting let's send some people

back to your home base then systeminit.com you also have a discord I believe is that true yeah yeah discord is pretty active well hopefully it's active

we'd like to become more active because we think we think people need to talk about this stuff with us and with each other so So that's what we hope that Discord community really becomes. Okay. Systemindit.com. Check that out. If you're down with Discord, be down with their Discord. And no more paper cuts, right? God bless us. Yeah, I hope so. That's it? That's it.

That's it. You're all great. Thanks, Adam. Thanks for coming back. It's been so awesome seeing you again. Hit us up when it's open source. We'll it? That's it. That's it. You're all great. Thanks, Adam. Thanks for coming back. It's been so awesome seeing you again. Hit us up when it's open source. We'll help share that as well. Oh, for sure I will.

For sure I will. We'll put that on news. Later, Adam. Thank you. Thank you. You're all great. I'm sorry I have to go. Just keep telling us that.

We'll put that on loop. That I'm sorry I have to go? No, no, no. No, that we're great. Oh, you're incredible. Yeah. Anytime you need it, just call me up and I'll be like, I'll give you soundbites all day. Thanks guys.

You're great. You're incredible. They love you. That's what Jared and I actually say in the mirror every morning before we get on these microphones and do what we do. It's just like, you know, Hey, you're great. You're wonderful. They like you. And that

kind of gets us going. I'm kidding, of course. But hey, when we hear this kind of stuff from Adam Jacob or a listener, like it really makes Jared and the rest of the team here at changelog.com just giddy, just excited to do this every single day. It's fun. But hey, this call with Adam was amazing. System Initiative out of stealth, ready to go. Systeminit.com. Check it out. The five minute demo that we talked about is online. You can see it, you can watch it, you can enjoy it. So do that now. And of course, if you have feedback or thoughts or questions or comments or whatever the link is in the show notes to comment hit us up on slack we'll answer on twitter we also like a comment on the website change.com slash whatever episode this is the link is in the show notes and of course a massive

massive thank you to fastly fly Fly, and also TypeSense. They got our back. You should check them out. And, of course, to Break Mastacilla and those beats. They're banging. They're the best in the business. We love rocking those beats. From Zelda to Castlevania to Mario Kart to brand-new renditions,

BMC helps us keep it fresh, and we enjoy it. But that's it. This show is done. We will see you on Friday. Outro Music