The Changelog: Software Development, Open Source - rkt, App Container Spec, CoreOS (Interview)

Episode Date: January 23, 2015

Alex Polvi, CEO of CoreOS, joined the show to talk about their new open source product rkt, their App Container Spec, and CoreOS - the container only server OS focused on securing the internet.

Welcome back everyone, this is The Change Log and I'm your host Adams Dekowiak. This is episode 138. Jared and I talked to Alex Polvey, the CEO of CoreOS. Great conversation today, talking about containerization specifically their awesome new open source product called rocket a competitor to docker specifically to standardize the app container spec great conversation around that as well um alex great guy today don't love this conversation we have some awesome sponsors making the show possible, CodeShip, TopTal, and for those who do not know what I'm saying when I say TopTal, I'm saying T-O-P-T-A-L.com.

I'm assuming their name. I have never asked Brendan this, so this is going off the script a little bit, whether or not their name is based on TopTalent. I'm going to assume that, but it's T-O-P-T-A-L.com, TopTal. Great support for the show. And not to mention, we also have the support of Rackspace. We'll tell you a bit about those guys

later in the show, but CodeShip is an awesome sponsor of ours. In fact, one cool thing I want to mention about CodeShip, recently they just listened to all sorts of feedback they got from their users and recently redesigned their entire application. Not only does the new design look better,

but it also has tons of new usability improvements to make things even easier than before. You can set up continuous integration for your app in just a few steps and deploy your code when all your tests have passed. No matter what language you use, no matter what framework you use, they have great support for lots of languages and frameworks. They integrate with GitHub or Bitbucket. You can deploy to cloud services like Heroku and AWS.

Setup takes just three minutes. You can find CodeShip at codeship.com slash the changelog. Make sure you go to that URL. Use the offer code the changelog podcast to get a 20% discount on any plane you choose for three months. Also, you want to check

out their blog at blog.codeship.com to get updates again the offer code to use is the changelog podcast and get 20 off on any plan you choose for three months and now on to the show all right today we're back hey the changelog here adam jared and alex alex pulvey and Alex, Alex Polvey from CoreOS, the CEO of CoreOS. We're here to talk about CoreOS, Linux, containers, Rocket specifically, maybe a little bit about Docker, who knows? But Alex, welcome to the show. How are you? Thank you for having me. Doing great. Doing great. Happy to share the story.

Yeah, I think we've been watching Docker closely, and obviously we're excited to see a new opportunity for not just Docker but what Rocket offers. And you seem to have a unique way you brought it out, I guess, into the ecosystem of open source. But before we go there, maybe give an introduction to who you are and what you do at CoreOS. Maybe from that we'll blend into a little bit more of what CoreOS is just for the audience who may not know. Sure. So I am the CEO of CoreOS. I'm one of the co-founders with Brandon Phillips. We started working on CoreOS about two years ago now. Before CoreOS, I was at Rackspace,

which I joined through the acquisition of my previous company, CloudKick. CloudKick built tools for cloud server monitoring management. And before CoreOS, Brandon was at Novell, working on SUSE as a Linux kernel developer. So, you know, when you put a kernel guy and a cloud guy together, at Novell working on SUSE as a Linux kernel developer. So when you put a kernel guy and a cloud guy together, you get a Cloudy OS. There you go.

And this is actually episode 138, right, Jared? Right. And so since you mentioned Rackspace, Rackspace is actually a sponsor of this show, so it's kind of funny that you have some rack space in your blood yeah i bleed red yeah austin texas well the you know those who listen to the show forever but you may not know this alex um the changelog's born in texas right so we're not very far from your your your previous mothership so to speak speak. Got it, got it. So maybe a bit of backstory on CoreOS itself, age, what is it, what is it for, that kind of thing. Sure. So we shipped our first release of CoreOS in August of 2013.

And at the time, CoreOS is a lightweight Linux OS that updates itself. And I didn't get into why that's important, but it's kind of a rethink on what a server Linux operating system should be. And we felt that the time was right to build something like that. And with containers emerging as a thing right around the same time. We've really kind of grown into this set of tools for helping companies build their next generation of infrastructure, kind of centered around containers and distributed systems and really getting security right as well.

That's one of the things we care a lot about. So you guys have a ton of open source projects up there at github.com slash coreOS, etcd I think being the biggest one. But is coreOS itself open source projects up there at github.com slash coreOS, etcd, I think, being the biggest one. But is coreOS itself open source, or do you guys have a proprietary core and then open source in the ecosystem? Sure. So the way we work,

first, our team is all open source, I'd say zealots, because there's really a better word for it. But our team is all very strong open source folks. I was before my previous company was at Mozilla. I was the 12th employee at the Mozilla Foundation. And Brendan and I actually started working together at this thing called the Open Source Lab, which ran the servers for Apache.org, Kernel.org, Mozilla.org, kind of all the big open source projects. So open is is definitely in our dna and the way that we go to market with all of this stuff is we build open source components which

are open source through and through we want them to be like the apache 2 web server of whatever they're trying to solve and that just ubiquitously used and no company really directly monetizing them and then to have a business, to make our efforts sustainable, we sell commercial products. And those products are more full solutions. They're end-to-end things that have dashboards and easy to set up. And it's a full-on product solution. And so there's kind of two types of software that we build.

There's all the open source components, which are individually useful and reusable and vendor neutral and use them however you want. And then there's software commercial products that you can buy from us that take advantage of the components that we're building. But they are, at the end of the day, products that companies go and buy. You mentioned Brandon in there, Brandon Phillips. I know he can't be here. I think we might have him on a different show. I'm not really sure, Kelly.

She's not really on the call. She's listening the scenes maybe you can say hi kelly hello hey so there's kelly so kelly helped us set up this call a little funny story there she put her phone number in the email i needed to talk to her right away a funny little side chat i called her right away and i said hey we gotta have alex on the show uh we couldn't wait until like late february and we had to have it happen in january so kelly is is uh is awesome she made magic happen for us so that's that's good there but brandon was supposed to be on the call at some point but can you give it like a brief intro of who brandon is and sort of what role he plays uh for the team sure brandon is our co-founder and cto um he really

is the kind of technical mind behind all this you. I'm a pretty technical guy, but Brandon was driving the architecture and overall decision making on the day-to-day technical details of everything you do. So when you look at, for instance, and we'll get into this, but in Rocket, if you look at app container, which is a specification, it's almost like an Rc for what a container should be you know that's brandon's design kind of coming through and really shining and and uh you know he owns the the deep technical side of of the company i guess maybe um to sort of give some premise to what this calls about um you know we like i said we've been watching docker fairly closely we've um we've had them on the show to talk about things and stuff like that. So we really wanted to sort of just kind of dig into talking about CoreOS, talking about containerization.

You've got your own philosophy on it, which is where Rocket came from, and just really drive into that. So what's the easiest way to open up that conversation? Sure. I mean, maybe we can talk about – I think first I'd like to give a little bit of background on CoreOS overall and why we started building it. And I think it'll help kind of paint the picture of why Rocket is what it is and why we built it as an alternative to Docker. So maybe we just start with that, a little bit of background on CoreOS. Sounds good. All right. Go for it, AlexOS. Sounds good. All right.

Go for it, Alex. Go for it. All right. Team might step up on that one. And so CoreOS, after the acquisition with Rackspace and helping Rackspace build out some different cloud products, it took a little bit of time off to figure out what to work on next. And when Brandon freed up, we've known each other for a very long time. And we looked at sort of what do we know best and what is something we could do that has

good social value as well as could be a good commercial value, have good commercial value down the road. And what we looked at was security. And we asked ourselves, what could we do to fundamentally improve the security of the internet? Okay. Kind of a big lofty goal, you know? Um, but we thought, Hey, if we could build something that could do that, then, you know, there's probably some commercial value there. And also it's something that, you know, like us and our friends are, are well positioned to actually go and sort it out too. So might as well go for it. Um, and so the key insight that that we

had is that security at the end of the day is all about updates, there will always be another vulnerability, another patch, another issue, you can't harden software to be perfect, but you can make it easy to update it when there is an issue. And on servers, I mean, servers are like notoriously, you know, get it running and don't touch it. I mean, some of the most fragile environments out there are these old server infrastructures that people just don't pay attention to anymore. But yet, that's where all the family jewels are. That's where like all of our data is. It's where our social security numbers, you know, everything is on the server. And so we thought, hey, let's build a server that automatically updates.

And if you talk to any sysadmin, this is like a crazy idea, okay? Like any sysadmin would be like, wait a minute, you can't automatically update my server. You're gonna break everything, right? And so we felt this is a perfect thing for a startup to go and try to do. Everybody thinks it's not possible.

And if it works, it unlocks a lot of value, all right? And that value is not just security, it's reliability, it's security. It's reliability. It's performance. It's like everything you get by running the latest version of software. And so that's really where we started. And that's why you might have, if you read anything about CoreOS, you see us message on the updates quite a bit. Because we think that that's kind of a basic requirement for good security.

I was going to say, that sounds like when I heard you say automatic updating a system. because we think that that's kind of a basic requirement for good security. I was going to say, that sounds like when I heard you say automatic updating a system. I just think it sounds awesome and terrible at the exact same time. So let's talk about how we do it, all right? It's not trivial, and it requires a lot of changes, and that's why CoreOS is quite a bit different than the existing server OSes out there today. So one of the first things we need to solve when we want to go update a server is how you package and deploy your applications.

And one of the main things that breaks when you update a server is inter-application dependencies. So you go do your patch to Heartbleed and fix OpenSSL, and it works for your Java app, but it breaks your other thing that's running on the server. And so our our solution to that is every application is packaged with all of its dependencies well what is that hey that's a container right and and so core os by design will only run applications inside of containers

containers are our package manager effectively um and so we got very lucky in that um you know docker came out around april of 2013 and and when that's we were in the middle of hacking on our first versions of core os then and we needed a container runtime that we either built ourselves because of you know just filling in the white space or um you know or use something off the shelf that that appears to be exactly what we would have built you know if we if we were left to our own devices and so that was docker and we released um you know core os with docker in our very first release we were the first like kind of docker native operating system uh to come out uh the first kind of container native um thing uh to come out and and we did we did our updates from the very beginning so the very first release that

we shipped was the one that we could update because we could just make the os better over time it's kind of like a software as a service but for a whole a whole os and that's still what we do today when you know when heartbleed and shell shock came out our little sliver of the internet uh we had patched and fixed you know hours after the patches were available and not just like packages available for download like the running servers out there on the internet were upgraded and no longer vulnerable to the issue one so it it's working the system's working containers are one of the things that that really enables that um and and that's how we got started with docker so early on so help me out with this infrastructure so core os is a linux distribution which is a Linux distribution, which is a kernel and some supporting software that only runs containers. And as long as I have a container that you can run, you're going to keep that underlying

infrastructure updated for me. My goal for CoreOS is for you to not have to think about your OS anymore. I just think that when a patch for Shellshot comes out and we have all these individual ops teams around the world scrambling, patching their servers, that's just like a redundant effort that the world doesn't need to do. That my team of OS guys can centrally patch the kernel and patch your OpenSSL vulnerability and deploy it to every server that is sort of opted into our platform. And we can take care of it for you centrally. And the tools that we've designed allow us to do it that way and do it, you know, kind of ultimately scalably as well.

We can, you know, we can run millions of servers this way, you know, and I can have my small team of engineers developing the patches to do that. So what if I'm a user of the CoreOS distribution but not necessarily a customer of the CoreOS distribution, but not necessarily a customer of your guys' ongoing update service? How would I go about managing the upgrades? So just like the way Ubuntu and Red Hat or Fedora work, our gift to the world is that steady stream of updates.

So we don't commercialize the updates themselves. In fact, all of our open source projects, CoreOS, RocketFleet, we don't directly commercialize at all. So our gift is that we give you that steady stream of updates free of charge. It's almost a community service. Now, again, that's why we care a lot about our sustainability and why we sell commercial products is to keep those efforts going. Nice. So of course you needed Docker because you did not have the container environment when you guys wanted to launch. Is that fair to say?

Yeah, I mean, if you think about Docker, sort of originally, it's a tool to download and run a container. It's very much like apt-get or yum, but instead of an RPM or a.deb file, you're downloading a container image and executing it. And it was our package manager, effectively, and still is our package manager today. But that's how we were using it. Another component towards how you build a system that you can automatically update, another property that you want is you want any individual server to not matter.

So you want to be able to pull the plug on any server and your environment keeps running. Now, if you talk to your ops buddies, they'll probably all agree that that's what they want too. But if you ask them if they can do it, they'll probably say no. And that's because it's too difficult. And so that's why we started building kind of, again, at the lowest level, a tool called ETSED, which is a distributed key value store primarily intended for shared configuration among servers. As soon as you have more than one server, you need to start sharing configuration across those machines. And we built that because we wanted to make it easier for people to build these distributed platforms so you could run things in this way such that we can pull the plug on any machine and update it anytime without you taking a downtime. And really filling in the white space.

And etcd was one of the first sort of areas of white space that we saw needing to exist. And etcd itself has been adopted by, I mean, Kubernetes, Mesos, Cloud Foundry, kind of like every cloud platform-ish thing that's emerging right now is chosen etcd as their underlying key value store. Nice. Yeah, I'm looking at the etcd GitHub page now. It looks like you have 134 contributors, over 4,000 commits.

This looks like a really mature product, so that's really awesome. That's all open source and available and looks like it's pretty popular as well. Yep. So you started off with Docker and you still use Docker to this day

in your guys' core product. But there came a point in time where the Docker philosophy and perhaps the core OS philosophy about how to do containerization apparently those diverged can you tell us about that sure so docker originally came out it stopped it talked a lot about this the standard container this idea that we could have a unit that's an application that could be ran in many systems. And that, you know, it was decoupled from, you know, a particular implementation.

It was about packaging a container and, you know, stick the container on a boat or stick it on a train and it all kind of works, you know, like using the shipping container analogy. What's happened over time is Docker is clearly on a path to be its own platform now. So while Docker started as a great tool for building a platform with, and that's why we saw it inside of Kubernetes, inside of Amazon's cloud product, inside of VMware's products, because these are existing platforms that wanted to add a container to it. It's becoming a platform like those things I just listed off now by adding its own clustering and everything. And I get the product decision there. That's fine.

And I have no objections to it. They should go and build their platform. That's great. I mean, we'll probably build a platform at some time too, you know. So it's a fine business idea. The issue is we still want that standalone component that is not directly commercialized at all, that allows you to download and run a container, essentially like a package manager for containers to exist. And so when it was clear that, you know, Docker was not investing in things like standards around what a container is for interoperability or, you know, just getting some basic security and composability issues right in the architecture of a Unix tool, you know, we said, hey, it's easier for us to go build a new thing that serves the needs of what we want versus send some pull request to Docker that rewrites the project.

And this culminated in a launch of a new tool along with some specifications that you're trying to get, I guess, formalized and community-driven around containerization, the tool we've mentioned called Rocket. The announcement for that came back early December, and it seems like it caused a bit of a stir. Is that fair to say? Yeah, you know, that one got away from us a little bit. If you look back on the whole thing,

the only messaging we've put out on Rocket at all so far is a blog post stating a couple technical reasons of why we built Rocket. All the press and all the excitement and all the Hacker News threads and everything was just fallout. The blog post is what stands there, and if you read it with a technical lens, you'll see some very specific technical issues

being addressed. This is not politics, nothing. We're just fixing some technical issues. And all the fallout and Docker's response and all that stuff was just kind of extra, extra stuff that got away from us. And kind of our lesson learned in all this is people are watching.

People are paying attention. Putting words in your mouth too. Yeah. I mean nobody can put words in my mouth because it's just written on the blog post. And if somebody is saying something that is not what's in the blog post, then they're just, I guess, making things up. It's probably an important point to say too that we're not here to throw stones at anybody. Like I said before, we got on this call, and everyone who listens to the Change Log knows that open source is hard. Open source is hard enough as it is without trying to call your buddy or your competitor or whomever to a degree opposes whatever you're building bad or not right or whatever.

We're not here to do anything like that whatsoever right um and i and i don't think you you definitely aren't because that's not what you wrote and that's not the point um maybe to put a timeline on what's happened like this is transposed over the last let's say i'd say 45 days december 1st was that original blog post and uh the tech crunch article that put words in your mouth like fundamentally flawed was posted later that same day so um what exactly happened inside of core west inside of the team when you released rocket put out this blog post and what was sort of the i guess the press frenzy how did that impact internally and did did anybody get how did the team react to it i guess is more or less the what i'm trying And did anybody get, how did the team react to it, I guess, is more or less what I'm trying to ask. I'm really proud of the team and how they reacted. I mean, there wasn't a lot for us to do. You know, we did get a lot of calls from press and such,

and we essentially just read the blog post back to them. Did you read the post, by the way? Yeah, exactly. So the team didn't really react or do anything there was a lot of sort of you know drama in air quotes on you know hacker news and tech crunch and things like that but the team um you know even internally us deciding to ship it um something that we've been working on for three weeks by the way um uh you know go-no-go based on technical merit. My engineers are not at all. They're hardcore open source developers, and we don't want to create redundant efforts or anything. So we saw a clear kind of white space, and that white space was a tool that was secure

and composable and had an open standard for running a container. And so we built the thing we wanted to exist for that. And the team, I think, has stuck by it, and we're continuing to invest in it. And it's coming along quite nicely, actually. And now a word from our sponsor. TopTile is the best place to work as a freelance software developer. If you're freelancing right now as a software developer and you're looking for ways to work with clients

on projects that are interesting, challenging, and using the technologies you want to use, TopTile might just be the place for you. Working as a freelance software engineer with TopTile, your days of searching for high-quality, long-term work and getting paid what you're worth will be over. Let's face it, you're an awesome developer and you deserve to be compensated like one.

Joining TopTal means you'll have the opportunity to travel the world as an elite freelancer. On top of that, TopTal can help provide the support for software, hardware, and all the support you need to work effectively no matter where you are. Head to TopTal.com slash developers. That's T-O-P-T-A-L.com slash developers to learn more and tell them the change law was sent to you.

So was the lack of that open standard, was that kind of the crux of the matter that made you guys finally decide to do this as opposed to trying to contribute to Docker or steer Docker? Yeah, I mean, we've contributed and tried to steer the project for a while because we've been involved

with the community for quite a while. And so, it's not that we aren't. In fact, I think we are steering the project the most we ever have now. It's just a very heavy-handed way to do it. And so, I think, look, nobody argues with,

we want a more secure tool. We want a more Unix-y tool. Follow the Unix philosophy, right? And we want open standards and shared open standards across projects, right? Like, who doesn't want that, right? So it was kind of an obvious thing to put out there. And it was those three things in combination.

I wouldn't say it's any one of them. Again, if our goal is to secure the internet a obvious thing to put out there. And it was those three things in combination. You know, I wouldn't say it's any one of them. Again, if our goal is to secure the internet and have 1.0 products, you know, they don't even do signature validation yet on the thing that was downloaded. It's kind of like, well, at some point we have to do something about that. And if, you know, we are building a system

that we want to add containers to, we're not trying to just build a... We're not trying to help users download and run the Docker platform as is. We just want to add containers to the stuff we're building. Then we need the composability and the Unix-y kind of philosophy there. And then on the standards front, for me, as an open source guy, we have a shot at cross cross cloud interoperability for the first time. And it's about packaging an application in a container, like we actually have a shot at

Amazon and Google, and folks are sharing a standard around the container as a unit that is actually movable. And in order for that to be actually widely adopted, we need it to be written down so other tools can interop with it. And so we just wrote it down. We wrote down the ideal thing that we wanted. And again, I believe what this will cause over time is, I hope, a shared standard between Rocket and Docker. Essentially the

Firefox and Chrome of containers. And the Firefox and Chrome created a better internet. It made JavaScript way faster. It made open standards stronger on all these things. So even if they share a standard, it's still good for everybody if multiple implementations of a standard exist. It's funny the way your perspective, because it reminds me, Jared, a little bit about our call with Tom Del New Hood of Cats with Ember.

They weren't playing the short term. They were playing the long term when it came to the tech they were building. me jared a little bit about our call with uh tom del new hood of cats with ember like they they weren't playing the short term they were playing the long term when it came to the tech they were building so completely different animals in terms of the tech but similar philosophies in the way they approached it um alex for you when whenever it said that um you're contributing back or in a more heavier hand obviously with rocket out there now, kind of guiding or steering the Docker ship. What was the relationship, I guess, between CoreOS and Docker prior to and then post-Rocket release, and how is that ship being steered collectively? Yeah, I mean, I hope that just the technical stuff shines through.

Again, there's been a lot of like not non-technical focused fallouts from this um but i just hope that the technical merit shines through i think that rocket will create a better docker just in the same way that chrome created a better firefox um and and that's what i want best i. I just want containers to win and to be successful. Okay. And for certain classic customers to adopt containers, they need security to be taken seriously. They need to integrate it with existing environments. And then they don't want to be locked in. And so they want to have the ability to build their own implementation of the thing that runs a container if they need to. And so that's why we addressed, you know, those three things.

And I think containers overall are better for this in the long run. And while there is a path of, like, doing it by collaborating around one project, we tried that around a year and a half, maybe almost two years, and effectively to be able to guide the project in a way that we thought mattered. And so we just built the thing we wanted to build to solve the needs we wanted to solve. So do you all have collaboration? Do you have sort of channels open up between CoreOS and Docker to sort of talk about the direction of containerization?

Brandon is on the Docker governance board, so that's good. And then as CEO, I talk to the ceo of docker around ways to collaborate um you know but it's it's a little bit difficult given the current circumstances of how everything sort of played out but again that's just why i hope that the uh the technical merit of everything shines through and containers are better overall that's that's really what i want from this one more question for you on that note. And hopefully the way I ask it and the way you get to answer it is the best possible way. I don't think it's – I guess what I'm trying to ask – sometimes I hem and haw over certain questions.

I don't want – because I never want to seem like we're trying to position a guest or this show in a way to throw stones. And I guess what I mean by that is that – crap, man. I totally forgot my question now. I'm trying to explain it. Jared asked a question real quick. I'll come back to it. I totally lost it. No problem. I want to just kind of talk about this issue of redundant efforts versus competition and i think uh you know as

engineers and developers like we're trying to squeeze like as much efficiency out of everything we do as possible and we see redundant efforts and it's like oh why you know why can't we all just work on one thing and and put all of our efforts together but i think chrome and firefox is a great example. Time after time, we see a competition in a marketplace, whether, you know, any kind of marketplace, even an open source, actually just makes all the projects better. And so you see the redundancy, and you're like, why can't we have a single effort? But historically, every time we have competition amongst a diverse ecosystem, it

raises the tide and everybody has to get better. So first off, in my lens of the world, I see Rocket and Docker as filling different things. Docker, when you think about it as a platform, is much more like a Mesos or Cloud Foundry or something like that. They use containers, but they also like have all this clustering and sort of other stuff built around it okay and in my eyes rocket is just the container runtime essentially what people originally thought of docker as the thing that downloads and runs a container and that's it so the companies that will use rocket are the ones

that have existing platforms like some internal environment that they want to add a container to, or the platform product companies themselves. These are things like Cloud Foundry and Mesos, or even Amazon and Google, that want to add a container to an existing thing. They already have clustering. They already have other stuff in the environment that they want to integrate with. And Docker is, in my eyes and just like how i see it going and this could have changed because of rocket but when we made these decisions um the way it was going is more of a soup to nuts platform in and of itself something you could take off the shelf to run your infrastructure which is meant for companies that want like they need in a platform

of some sort to run which is fine and again a lot of companies need that so it makes a platform of some sort to run, which is fine. And again, a lot of companies need that. So it makes a lot of sense. Okay. But the container piece that we want is to have containers interoperable between a bunch of different platforms, you know, not just have like, there's the one container thing and the one container thing you can do with it. And so that's the I guess my point in this is I see them as actually distinctly different things. You know, the Firefox and Chrome analogy, I think, applies for like, let's just step up security on the way that you download an image and validate it before you run it, and so on. And those things apply.

But Rocket is a component. Docker is a product. And that's like the core kind of difference. Aren't there two pieces? I mean, isn't Docker both? Isn't there still that idea of a container, and yet then there's services around it? It seems like, in my mind, I see the two,

and maybe they're merging or becoming one. Do you see it as just the platform? There's no such thing as a Docker single container that can be used as a component? There's the Docker single container in there. But it's a, again, there's a component in there. So the way, we can get technical, right? It's okay to kind of get into the weeds a little bit?

Okay, so the way that Docker is architected today is there's a central daemon that runs on every server. Okay, running as root on your server. And then when you type docker run, it's actually an HTTP client talking to this daemon locally over HTTP. Or the daemon can be remote too. And that's one of the like more clever things of Docker

is you can easily like have your OSX go binary push over to a Docker daemon that's on a remote host or on a virtual machine on your laptop, okay? So that's clever. And that's on a virtual machine on your laptop. Okay, so that's clever and that's nice from a ease of use perspective. But the problem is is when you have a daemon running as root on your server, that one has an HTTP interface,

like I think again sysadmin 101 here is, do we run our web servers as root on a server? You know, like no, okay. And then the second piece is anything that talks to the internet, should we run that as root? So something that like downloads an image and runs in it you know downloads an image or uploads an image so that thing be running as root no and kind of the whole architecture of the way docker's built is around the central daemon that kind of has all the functions of docker in it so yes part of that daemon that's running has something that the downloads and or that runs a container, and that's great.

We just being like Unix guys that care about security need it to be refactored such that those are individual actual applications that run so we can invoke them with different privileges and different users to get the security model more correct. And to do that, you effectively have to rewrite Docker because you have to break apart this whole HTTP client to like Damon thing, um, that's going on. Uh,

and so that's, that's why we're like, Hey, let's just start from, from scratch. Uh, because it, it's just,

it's actually like the model is totally different in our world. What you would do if you wanted an HTTP interface, so you'd write a little service that's like an HTTP kind of, uh, it's an alternative to SSH on your server that speaks HTTP and JSON. And when you hit that, it probably talks to like Dbus and tells system D, the init system, to invoke a container and run it.

Or if you want it to download an image, it would tell system D to download, you know, to run a process as an unprivileged user and download an image. Maybe it just uses curl to download. It doesn't need to have a fancy go binaries. It's like composability is the way we architected Rocket to be so you can use it to build systems. But anyway, yes, if

Docker was, for instance, to clean up their security issues, refactor Docker into a bunch of individual components that could be used differently, and then have an open standard that was interoperable with projects outside of Docker itself. Well, now it starts to become a lot more like Chrome and Firefox because they're roughly the same than in that case.

Yeah. And hopefully, bringing up issues like you are here allows them to bring those problems to light and then address them in their software. Right. And I think they got the message loud and clear with Rocket. And we've seen them kind of start going down the right path. And again, what we feel is the right path is all just an objective opinion. But I think they're going down the path of making it more composable and really fixing the security thing. And I do find it unfortunate that we weren't able to do this sort of as an effort together. That's not without trying.

It's not like we never tried. We tried for multiple years, actually, and then eventually decided, well, we just need to go our own way to get it the way we need it, which I think is also a very hacker way to go. Yeah, well, in the hacker world, we would tend to fork, but it sounds like forking in itself was another decision you guys didn't want to make. Again, because the model is so different, we would have had to essentially rewrite the fork, at which point it's like, just write a new thing and might as well get some of the security primitives right because that's also pretty core to the architecture too um so we just we we built it like from scratch because it was easier than forking

gotcha i guess it's kind of where my question that i lost by the way uh on there was more like you know docker was is sort of synonymous with containerization. They sort of coined the term or coined the name, not so much the term, but kind of made it popular. They popularized it. For sure. Right? And so you've got this non-standard way to make a container

and CoreOS sort of being built around this container world, not running anything that's not a container, basically. And I was just wondering, I guess, what your thoughts were, and this may be sort of awkwardly placed in the conversation, but what your thoughts were on their change of business model when they went from DocCloud to Docker and sort of built their new business model around it, whereas with Core os you know

you started off with the idea of how you were going to build core os from a monetization standpoint how you were actually going to build a company around it and as you mentioned before providing you know free update services and community service as part of your business model like sort of buffered in what what the difference was their model versus the way you and how that might have could have played differently to to containers as a whole i'm really not sure how to answer that i mean their business model is their business model i don't think it's even that's why i had a hard time phrasing the question yeah i don't think i don't think it's played out yet on what their business model is it's not that clear right now it appears to be kind of a github like thing for

hosting containers um i assume there'll be more um you know down the road on that um so i don't know i can't really comment on their business model all i know is the way like our company wants to build open source software is we want to build open source components that are freely reusable and that helps companies sort of run their infrastructure in this new way. And then we intend to build commercial products that take advantage of this transition to this new way of running infrastructure and help companies get there faster by buying our commercial solutions. So let's change focus over to this app container specification.

This seems like a call to arms for the community anybody who's interested and invested into containers and rocket of course is the command line tool that implements the specification can you speak more about specification what's in there what's not in there who owns it that kind of stuff app container is an awesome piece of tech if if you're into in all this container stuff and want to get really nerdy with it definitely go read the specification it's really cool it's a really really cool piece of tech um and uh and so what we did is we talked to kind of everybody that's in the container space and got their feedback on what would be ideal and then brandon who's's a very talented engineer, spent a lot of time refining it. But there's three components to it. There's

an image format itself, which is essentially a GPG signed tarball with some metadata. I mean, it's simplifying it, but that's what it is. And then there's the runtime itself. So you can't just define the image. You have to define the environment that the image runs in in order to have real consistency and portability. Some of the things that I think are really cool in the runtime are, you know, one problem with containers is how do you give them state, like to start up? How do you essentially give your containers arguments?

And there's kind of three different ways to do that. There's environment variables or a config drive where you have a directory that has config variables written down to disk. Or the third way is a metadata service. And that's what Amazon and the cloud providers use. App Containers Runtime specifies a metadata service for doing that, which is, again, how the cloud providers kind of done it. And then the thing where we moved everybody forward that no cloud has done it all the day is we give every container that runs an identity, which means on the metadata service, there's an endpoint that you can post data to and get a signed version back. So it's like every container has a little

mini HSM built into it. And again, from a security perspective, the key to good security is giving everything running in your environment a strong cryptographic identity. And just like etcd, we want to make these more complicated topics easy. You know, we essentially built a tiny little HSM into the metadata service for the runtime and things like this. It's like, yes, let's just move state of the art forward and help people more easily build secure systems. So there's the image format, the runtime, and then the image discovery specification. So one of the novel things of Docker is how tightly integrated it is with the hub, which is the place where you host and share your Docker containers. And that's a Docker Inc. RAN service. The way we did the image discovery and download for a rocket and an app container is, it borrows some concepts from the Go programming language,

where essentially you can federate it across the DNS namespace where the image is hosted. So if I said rocket run coreOS.com slash etcd, there's a convention for discovering using DNS how to find and download and run that image, which means it's truly distributed and federated because everybody can do DNS however they want. And that we think is also a pretty kind of clever, novel piece of tech in there borrowed from the from the go world. So definitely check it out if if you haven't already and you're interested in these sorts of things. And now a word from our sponsor, Rackspace, Rackspace, Rackspace. You know, I thought about actually saying nothing but rackspace for the whole spot but i didn't think that would be cool and i don't think you would either and when i told rackspace about it they were like nah you can't do that but what they did want me to

do is tell you about how much they love open source and how much they appreciate you listening to the changelog and they want to give you and everyone else who wants it 50 a month in credit I really like that identity piece. I've enjoyed that in Go as well. I think that's a great addition to containers. What's the state of the specification? Is it pretty much written? Are you looking for feedback? How do people get involved?

Right now, we're between a... The very first thing we released was a 0.1.0, which is essentially prototype. Here's our ideas. We wanted to put enough rails on it that the conversation could move forward, but we didn't want to define everything. I think we cut yesterday's 0.2.0, which is getting pretty good, but still moving.

And we've been keeping Rocket in track of the spec the whole time, so we are forced to think through the spec with an implementation. And then our next major one is around a kind of, we think it's good. So I'll, you know, outside implementations, like go for it. Let's start doing the interoperability thing. Folks that want to help kind of show that the standard works. And then once we have a number of sort of outside implementations, then we'll call it 1.0 because that should just prove that the spec is pretty solid if we're able to get outside folks to contribute to it and build their own things. And we're starting to see it happen. There was a C++ version that was released of the app container spec. There's another one I can't recall off the top of my head.

But even that, before we have a stable spec, is pretty solid. For a project that has been out for about 45 days minus 15 days of holidays in the middle there. So it's moving along pretty quickly. So no doubt you eventually want to get Rocket involved in the CoreOS product. Got a timeline on that transition? And will you continue to support Docker into the future? So we'll definitely continue to support Docker. The Rocket timeline depends on how quickly Rocket is production ready.

And so it's a little bit TBD just because we know we don't even try to set timelines on our open source projects. It's just kind of like when it's ready, it's ready. So it'll take a little bit of time to get 1.0, but it's moving along very quickly. We will, at some point, have a Core OS with Rocket in it. How those all kind of play together, we haven't really talked too much about. I will say, though, the original motivation of Rocket and our use of containers is to treat it like a package manager.

But our packages are different in that our packages are always up to date for you. So you could imagine building a package manager, hint, hint, wink, wink, that also does auto updating, you know. And that's something that we would want to do for the docker platform itself you know as they ship new features like constantly and ship uh you know security fixes and everything we would love to deliver those extremely quickly to the user as an entire platform not just as treating it as our package manager you know just like how we would love to help the mesos community run mesos on top of core os um but to do that we

would package it you know we would package those things we wouldn't make it the primitive on core os for how you download and run the package if that makes sense yeah cool sounds really cool um let me ask you this say i'm interested in core os because actually I am kind of interested in CoreOS. Say I am. Hypothetically. Can it run pretty much anywhere these days, like Amazon, DigitalOcean? Is it just like any other Linux distro that I can go install onto a VPS? Yep. So we're on Amazon, DigitalOcean, Google, OpenStack, Eucalyptus, on-prem, bare metal, ISO, USB stick, VMware, you name it,

and you can run CoreOS there. And the really cool thing about CoreOS is when you run us on a bare metal server or you run us on a cloud server, the root file system is bit for bit identical we can pass a signature validation on the on the entire root block device that says they're cryptographically identical which is great from a security perspective like forget about ids it just doesn't matter anymore um and it's also great from if you're at you know a developer and you want to target a consistent platform in different environments, we are 100% consistent. So if you want to use Ubuntu on DigitalOcean and Amazon, that's cool.

You can do that, and they're pretty close. But they're not like bit for bit identical, which is kind of a requirement if you want actual portability between these things. And so we put a big emphasis on CoreOS to really nail some of these things home as we get distributed across all the different cloud environments. I guess one closing question before we tail off to our super awesome end of show questions. What role does Quay play into, if that's the way you said Canadians say it,

maybe they're French Canadians, Quay play into, if that's the way you said Canadians say it, K? Maybe they're French Canadians. Key. What role does that play, I guess, into the future of CoreOS and this open standard for the app container? Sure. So first, that's a great example of our commercial offerings. You could go and use an open source Docker registry, or you could use Docker's hosted registry. But we build an enterprise-ready, on-prem version of a Docker registry that companies can go and buy if they don't want to piece it together themselves. And there's no alternative to that right now on the market.

We have a complete monopoly on an on-prem, kind of commercial-ready version of a Docker registry. So that's a perfect example of it's like, hey, you could go replace it with open source by your teams piecing it together if they want, or you could buy it off the shelf from us and you choose. And it incentivizes us to be interoperable with standards, but also just do a great job of piecing those things together for our customers. Now, features of Quay that we might add as they relate to Rocket and App Container, I

think it's only natural to assume that we will support App Container and Docker. Just like all these other projects that are trying to target App Container, it's moving right now, so we can't just ship it overnight. We have to get the spec firmed up before we can have our tools support it as well. So that, I think, will only be another value prop of enterprise registry is you could choose the best container technology for you. If you want the one that Docker has put together, that's fine. If you want ours, you could do it, and we'll make sure they're all interoperable, and you can kind of choose which one is best tool for the job. And having CoreOS Power all that has got to help the development team sort of bug fix

across the spectrum too exactly exactly well um alex has definitely been fun talking about uh app containers the standard uh rocket docker core os uh i think jared's excited about it. Those who know the show well know I'm a front-end designer person who plays hacker for fun on the radio, as Wynn used to say when he was co-host of the show. He used to say that a lot, so it was kind of funny. But I've actually done several server builds over the last couple years, and I've gotten more and more into my DevOps space. But if Jared's excited about CoreOS, I'm excited about CoreOS.

I hope you're excited about it because you don't want to care about it. I want you to say, I want to use CoreOS because I don't ever want to have to worry about a security patch. I'll just let the CoreOS guys take care of it for me because I think they can do a better job than I can. See? That's exactly probably what you want, right? That's exactly what you want. Exactly. When Hartley was around, I was like, oh, man, what do I got to do?

And I'm not obviously as the non-DevOps, non-server builder person but does it part-time when he needs to sort of person. I was thinking, what the heck do I do? I don't even know what the problem is exactly at the moment. And then I'm sort of playing catch-up because I'm less in the fringes on that stuff. And while we pay attention to open source and keep our finger on the pulse, open source is big. Technology is big. You can't grasp it all. And I was like, what the heck do I got to do? And it would have been nice to have a CoreOS-like thing where I can trust that you're going to auto-update it on my behalf with security.

But then you do have the fear side, which Jared pointed out earlier. So you sort of have this double-edged sword that so long as you keep doing your job right on security and non-breaking, I guess the containers sort of take care of that, right? Well, I can tell you what I did on Heartbleed is I went out and patched double-digit servers for my customers. I spent the whole day patching servers. So I could definitely get on board with somebody else pushing those security patches onto my os that'd be awesome we patched tens of thousands or hundreds of thousands of servers is it like the uh the u2 song for apple you you push it out to 5 million or 50 million

or 100 million people at once that's right right into their into their music. Right, exactly. Our users opted into it. I was going to say, that backfired a little bit. That did. Well, Alex, we have a couple questions we'd like to close with. I know that Kelly probably helped out by feeding those questions to you. Jared, which ones should we ask? Should we ask them all? We've got a few minutes.

We've got some time. We've got to do Programming Hero. Yeah, okay, so let's start there. Who's your Programming Hero?'ve got a few minutes. We've got some time. We've got to do Programming Hero. Yeah. Okay. So let's start there. Who's your Programming Hero? That's a great question. There's a number of them.

I think the one that takes the hat, though, is John Gilmore. Do you guys know John Gilmore? I do not. So John is an early Unix guy. He did the original public domain implementation of TAR. He also founded a group of cypherpunks. He was the creator of the Electronic Frontier Foundation. One of the coolest moments, I met him really early in my career when I was in high school, actually. A mentor of mine in high school said, hey, you should write a paper on the digital millennium

copyright act uh that would be a great senior paper to write and i'm like okay having no clue what that was about and then my friend was like and i have a buddy coming over that can help review it for you i'm like okay so i write i write a paper on the digital millennium copyright i could john gilmore founder of the eff reviews my senior paper on the DMCA. It's like, uh, what do you have to say about it? I mean, the,

the conversation was fine. I mean, I was so green and had no idea about any of this stuff. Uh, I didn't even know, like, yeah, you know, I,

this was pre Mozilla and pre everything. Um, and you know, I think it was just a nice conversation. He wasn't too hard on me. Um, but it, that he definitely has a lot of respect of

mine because of his stance on online civil liberties as well as he's directly contributed to really core technology in the Unix world. You know, TAR, it doesn't get much better than that, right? So I would have to say John Gilmore. Awesome. Well, for CoreOS, the App Container Standard, anything that you're working on, what is a call to arms? What is a way that the listeners of the show, they're either professional open source developers, enthusiasts, hackers, whatever you want to call them. How can they step in? Where could you best use the help, I guess, from the listeners of the show and the crowd that collects around open source? Sure. I mean, I'd say overall it's like, hey, all of this from the CoreOS perspective is about our goal to secure the Internet. And so there are a number of different ways you can contribute to that.

One key part of that that's very timely right now is about application interoperability between platforms, and that's App Container and Rocket. We could use help on Rocket itself as a tool, but also and more importantly, we need third-party implementations of App Container to exist in different languages. So that means, let's say you're running an existing configuration management system and you want that to output a container instead of manipulate a running host, that would be a great way to build an app container image. Or maybe you are working with language-specific stuff. You're a Node.js guy. We should be able to build tools that in the native Node.js tool set

output a container image to allow portability and things like that. So I would say today we use the most help on is is our third-party implementations of of the specification to know that we truly have built a interoperable spec that works well for people um and the spec is just getting to the point right now where we're about ready for that um so i would say that's the most timely one but hey if you're up for securing the internet, we got systems programming for you with the operating system.

We have Rocket container stuff. We have ETCD, distributed database. I mean, all of these things are components toward this bigger vision and we can use help on all fronts. Any chance you got a link to third-party spec for what you mentioned there? Yeah.

We can share in the show notes. It's on, yeah, it's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com.

It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com.

It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com.

It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. It's github.com. it's github slash app c appc uh for app container so uh github slash app c we'll uh we'll trudge through there and figure out where it's at and throw a link in

the show notes so if you're listening check out the show notes for that link um and i guess the the last question is kind of fun since we do have uh we do have oh we're past time 21 seconds i'm just kidding. The last question is sort of fun, so have fun with this one. What would you be doing if you weren't doing X, and that X being whatever you're doing now? So right now you are CEO of CoreOS and what you're doing now, but if you weren't doing that, what would you be doing? Well, the short answer is what I'm doing a lot of CEO core OS is not able to actually

hack on the products directly and doing a lot of other things. And so I would not be doing those other things and just working on the product. But that's the more short term thing. I think overall, you know, I spent a lot of time after Rackspace and the acquisition trying to figure out what to what to work on next. And this mission and what we're on right now is where we ended up. And I couldn't be happier than than what we're doing right now is, is where we ended up. And I couldn't be happier than, than what we're doing right now in the,

in the work that, that this team is doing. So I'm right where I want to be, which is, you know, that makes us pretty defensible towards ever being like, you know, acquired or being killed or something because we're,

we're building the exact things we, we want to build and, and getting good traction on it. Um, so really happy, but in, in the micro sense, I would,

I would die if i could work a little bit more on the actual products and tech and red code and that kind of stuff uh because that's really what i really what i love any fun hobbies come to mind that aren't exactly job or tech related um yeah i there's kind of we've gotten windsurfing and things like that in the past by the way yeah i have these uh these little inflatable kayaks that I take out everywhere. And so there are these little kayaks. They're actually for like whitewater rafting in Alaska when you need to hike up a big, you know, you're hiking somewhere that requires a river crossing. And I take those things out.

Even last night I was out on the bay with my little kayaks. But they're so small you can put them like in your carry-on luggage and just take them with you wherever you want to go uh so i really uh i have fun playing out in the water and those things you know on the surf or out in lakes or in rivers and that kind of stuff so i don't know i enjoy the outdoors yeah well good deal alex again thanks uh for coming on and And Kelly in the background, I know you're still there. Thank you for making this time possible for Alex. I also want to thank, we mentioned Rackspace already as one of the sponsors for this show, but we got two other sponsors, fantastic sponsors, by the way. Codeship, TopTal.

People have emailed me and said they cannot understand when I say TopTal. I can't help it. I'm sorry. Their business is called TopTal, T-O-P-T-A-L. I'm assuming they named the business name short after TopTalent. So, T-O-P-T-A-L.com, by the way, if you're that person or if those several people who have emailed me and said, dude, what are you saying?

That's what I'm saying, TopTal. And still, it's still unrecognizable. And of course, Rackspace, we thank them for their support for this show and whatnot. So let's say goodbye, fellas. Great show. Goodbye. All right, thanks. Goodbye. you