Screaming in the Cloud - Episode 67: Infrastructure as Code with Terraform and Mitchell Hashimoto

Episode Date: July 3, 2019

About Mitchell HashimotoMitchell Hashimoto is Founder and CTO of HashiCorp. He is the creator of Vagrant, Packer, Serf, Consul, Terraform, Vault and Nomad - a set of open source tools that each individually are downloaded and used millions of times per year. At one point Mitchell was in the top 5 most active users on GitHub. At HashiCorp, Mitchell is helping define a remote-first culture with over 500 employees spanning dozens of countries. He loves open source, automation, and working from home.Links Referenced: Twitter: @MitchellH

Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This week's episode is generously sponsored by DigitalOcean. I'd argue that every cloud platform biases for different things. Some bias for having nearly every feature you could possibly want as a managed service at varying degrees of complexity.

Others bias for, hey, we heard there was money in the cloud, and we'd like it if you would give us some of complexity. Others bias for, hey, we heard there was money in the cloud and we'd like it if you would give us some of that. DigitalOcean is neither. From my perspective, they bias for simplicity. I wanted to validate that, so I polled a few friends of mine about why they were using DigitalOcean for a few things, and they pointed out a few things. They said it was very easy and clear to understand what you were doing and what it took to get up and running when you started something with DigitalOcean. That other offerings have a whole bunch of shenanigans with root access and IP addresses and effectively consulting the bones to make those things work together. DigitalOcean makes it simpler. In 60 seconds, they were able to get root access to a Linux box with an IP. That's it.

That was a direct quote, except for the part where I took out a bunch of profanity about other cloud providers. The fact that the bill wasn't a whodunit murder mystery was compelling as well. It's a fixed price offering. You always know what you're going to wind up paying in a given month. Best of all, you don't have to spend 12 weeks going to cloud school to understand all their different offerings.

They also include monitoring and alerting across the board, and they're not exactly small time. Over 150,000 businesses and three and a half million developers are using them. So give them a try. Visit do.co slash screaming, and they'll give you a free $50 credit to try it out. That's do.co slash screaming. Thanks again to DigitalOcean for their support of Screaming in the Cloud. Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time in 2014, I was a traveling trainer for a configuration management system that need not be mentioned here. I was on the way back and the person next to me on the plane saw my shirt. Oh, great. A plane talker. Doesn't everyone love talking to one of those? He started talking about a system he was

in the process of building and about to release that would handle infrastructure management at scale programmatically. Cool. Great. We've all heard those stories before. I didn't see it going too far, but wished him well. Now we're having this conversation almost five years later. Mitchell Hashimoto, founder of HashiCorp, author of Terraform, and a bunch of other things. Welcome to the show. Hey, Corey. Nice to sit next to you again, virtually. Oh, yeah. I will freely admit I was wrong on that one. It's certain that you can have a 98 or so percent success rate by talking to people who are building something new and

saying, oh, that'll never work. And then just, you're going to get it hilariously wrong for conversations like this. But by and large, it always seems like being a pessimist is a viable strategy or a path forward. It's a great way to lose

nothing, but also to gain very little too. That's the fun thing of being a futurist too. You can always make outlandish predictions. If you happen to be right, you'll be lauded. If you're wrong, well, everyone's wrong and no one really holds it over your head anymore. Yep. Yep. To be fair, I wouldn't have, uh, I wouldn't have been optimistic about me either from the third person. It seems to have worked out, though. You folks have now become a name brand in the space. It's been years since I had to mention what Terraform was when giving a talk around this stuff.

Nice. That said, for those who are unaware, because not everyone knows everything, as it turns out, can you describe what Terraform is for folks who may have never heard of such a thing? Of course, yeah. So Terraform is quite literally infrastructure as code. So you describe servers, switches, DNS records, anything you would imagine. I like to say anything that would be in a

quote-unquote data center to run an application, you put it into a text file, you tell Terraform to make it for you, and it does that by stitching together a variety of APIs from cloud providers and SaaS providers and so on. For those of you in the AWS ecosystem, you can think of it almost like CloudFormation done properly, which is, I'm sure, a battle that we will get into later

and no one would ever disagree with. Meanwhile, you can just hear people starting to seethe in Seattle at this point. Yes, yes. I won't say you're right, but let's see where this goes. Absolutely. But first, let's start with something else that'll irritate people. I noticed a while back that you had a post on Reddit, which is always a great place for a conversation to

start, talking about multi-cloud. And given that I've spent roughly 18 months now talking about why multi-cloud as a best practice is almost always the wrong decision to go in, I would love to get your thoughts on this. Yeah, yeah. So multicloud has been, let me start by saying that HashiCorp's sort of core mission or vision has been enabling people to adopt multicloud. So that has the underlying assumption that multicloud is real. And for a number of years early on in the company, that was pretty laughable, I would say. When we founded the company

in 2012, I think that it was pretty common for people to think that there was no possible future reality other than a fully Amazon world. And that has changed over time, at least from my biased point of view. But yeah, what I always tell people is sort of like, I've learned, I bet this early on, but I've learned in practice since then that multi-cloud is somewhat inevitable. It's kind of like entropy in my mind. It's like, you could fight it and you could fight it. And there's good reasons to fight it. I won't, you know, I won't disagree with you there, but at a certain point it's inevitable. So if it's an, if it is an inevitability, then how do you best equip your business to or organization to be multi-cloud ready, even if you're not? So I want to make it super clear that I never tell people and I don't advocate for people being multi-cloud early.

I think you should 100% focus on a single cloud, get really good at it until you're just forced away from it for a good reason. But while you're a single cloud, you should just try to adopt tooling that will make that a little bit of a smoother path. Before we dive into this and potentially start tilting at the same windmill from opposite sides, it feels like we should define terms. When you hear the term multi-cloud, what does that mean from your point of view? Yeah, that's really important. So for me, it means what I like to call workflow portability. I define multi-cloud, in my mind, as four different things. It's workflow portability, workload portability,

data portability, and traffic portability. I think I've experienced when most people say multi-cloud, they immediately snap to workload portability, which is sort of the idea that any application you write could run on any cloud. And for me, that's absolutely not the case. I think you should take advantage of all the best in class, first class services that make applications explicitly not portable between clouds. And what is more important to me is workflow portability, which is the process by which you get infrastructure, deploy applications, secure them, network them. Those sorts of core things should be multi-cloud.

I would agree with you that on a per-workload basis, it tends to be something of a red herring. I would also say that, oh, you should never use a second provider for anything. That's nonsense. I mean, easy example of this is PagerDuty. There's not a whole lot else like that in the market, and certainly none of them from any one large cloud provider.

I would also strongly suggest people use, even if you're on AWS, I'd still recommend you use GitHub, which is a Microsoft product, or Office 365, or G Suite. There's always going to be higher-level differentiated services that make sense to consume.

I think that people tend to hear multi-cloud in conversations I've been in, and where their minds tend to go is almost a VM-centric, where you're coming from an on-prem environment where you've got this thing sitting, running in VMware, most likely. And I should be able to seamlessly have that flow to AWS,

to Azure, to GCP, to Oracle Cloud in some very strange conversations. And that nothing else should have to operate around that in any way. And I just haven't seen that in the real world. Exactly. Yeah. And I've seen, I think I alluded to this in that Reddit comment, but I've seen the opposite now, where it's been long enough with AWS in existence that I've now seen a company go from startup to public, pure AWS, like actually AWS sort of poster child in a way, and then be forced to adopt another cloud.

And sort of the almost tragedy that fell out of that. And I think that's very interesting. And the abridged form of that story is really that there is a sort of darling startup, which I won't name, but it's real. And they went 100% into AWS, blogged about how they automated the heck out of it and, and really good, you know, sort of thought leader at AWS, um, became a multi-billion dollar public company, full AWS. Again, they ended up acquiring another company for a couple hundred million dollars. And this other company was pretty large themselves. I mean, if you're getting acquired for that much, I mean, they're pretty large themselves, pretty successful, 100% on GCP. It became this thing where I've talked to the executives, and it was one of those things where you're not going to block an acquisition that's good for the business of a public company based on their tech stack.

It's just sort of not going to happen. If the strategy makes sense, if the partnership makes sense, sort of not going to happen. If the strategy makes sense, if the partnership makes sense, it rarely is going to happen. So they decided to acquire this company no matter what. They bought them. And then what ended up happening was they basically churned their whole infrastructure team. Pretty expensive. It was either convert everyone to AWS, which they were violently against and the people internally were sort of against from a cost perspective, or just figure out how to do multi cloud. And again, this wasn't workload portability, they didn't want to move anything over to AWS. But just how do you do, you know, governance? How do

you do cost reporting? How do you do like these higher level things across clouds? And they ended up basically rehiring an entire sort of infrastructure ops team to handle us, and it was messy. It's a form of lock-in that people generally don't tend to think about. It's not just technical or architectural, but if you have an entire team of people that are great on cloud A, and then, oh, by the way, we're migrating things to cloud B as well, very often people don't want to do that. They'd rather continue to focus on their area of expertise rather than start to dilute that across multiple providers and confuse themselves.

There's also any higher level service story starts to break down. I think that when there's a compelling business reason to go multi-cloud, I'm all for it. What I'm agitating for has not been, oh, pick a provider and go all in at the cost of your business. That makes no sense to anyone.

But building something Greenfield on day one with this idea that you're going to run it on AWS or GCP or Azure or wherever else you want to put that, maybe don't do that. Maybe pick a provider, I don't care which one, they probably care which one, and just hope for the best until you have a compelling reason to move. But trying to build something that can be seamlessly deployed to everything,

it doesn't work. And if you do go down that path, you're going to find that you're constrained to some very low-level primitives, because nothing beyond more or less a bunch of VMs and maybe some databases and block storage tends to be the same across providers. Even database provisioning calls tend to start breaking down. Yeah, I think that's an important part of the way we view multi-cloud is, you know, one of the first things I tell anyone that is adopting Terraform for the first time is they read, you know, CloudFormation, but for anything, or they read something like that somewhere on the internet, and they think, oh, this is like the write once,

run anywhere dream. But I make it really clear, and I'm going to make it clear right now that it's not that. The Terraform you write is AWS-specific or Azure-specific or Google-specific. You cannot rerun that and target a different cloud platform. That's not what we're doing. That would be workload portability. What we're doing is workflow portability.

Any config you write for any cloud, it's always the same command to create it, the same command to see what would happen, the same structure to investigate for cost analysis, things like that. It's higher level than the actual tech itself. I spoke to a single tenant customer that was trying to build out a deployment on AWS and GCP. They were already a Terraform shop, so they were very conversant with the tool. But getting the networking and the security groups to coexist between providers was massively painful.

And that was not anything I would consider having added differentiated value. For their business case, it was worth the experiment. It didn't really pan out longer term. But the challenge that you'll see is, at that point, when you try an experiment, it doesn't work out. You have to sort of stop riding two horses. Which one do you pick? Invariably, I find that data gravity tends to carry that argument pretty quickly.

Yeah, yeah, I would agree with you completely there. I think that's why applications tend not to move. I mean, it's very rare for even our multi-cloud believers, so to speak, to move an application or especially because of the data associated with it. What's much more realistic is a new application or a totally differing team or business unit decides for really practical good reasons that they want to use a different platform. When I'm doing greenfield architecture and looking at something that, okay, I want to go ahead and build this on a particular provider, I generally don't bake multi-cloud requirements into that.

But I will say that I do try to avoid, I guess, strategic architectural lock-in. There's a whole bunch of stories around that, but an easy one might be using something like Cloud Spanner or DynamoDB Global Tables. Just because those tend to be things that don't manifest in the same way as anywhere else. So migration isn't just a painful move a thousand things over, it's also and re-architect your entire application's data model. Yeah, but again, I don't worry too much about that

because I don't think that your applications necessarily need to move or be portable. It's going to depend on your use case. But Cloud Spanner is quite expensive, but if you look at Cloud Spanner and it's going to save you a ton of time in order to get a technically correct application to market, then I think you should do it. I think that if your application is successful,

it's unlikely that it'll have to be across multiple cloud platforms. So it's worth it, but at the same time, it's just a question of whether you want to be locked in technically to a provider or not. I would agree with you, but this question often comes from higher-level strategic business decision makers. The question of what happens if Amazon decides to compete with us? To which my response is always, what do you mean if? Their product strategy is yes.

Yes, yeah. We've seen that more in the recent years. We've seen that more and more, for sure. It's been really, really interesting watching entire industries basically create plans to shift providers overnight due to major acquisitions like that. It's always challenging watching people go through that process. We've decided that we're not going to put anything on AWS because some of our customers demand it. You take a step back and you look at some of those customers who are demanding that, and those customers themselves have workloads living on AWS that aren't small.

It's one of those fun explorations of, do what I say, not what I do. It feels, to some extent, like a little bit of tail chasing going on in our industry. It also feels to me, to some extent, like multi-cloud on a workload basis is actively being pushed by a number of vendors in this space, where if you take a step back and say, we're going to go all in on any given provider, well, at that point, there may not be a story for those vendors who benefit from the multi-cloud approach to have something to sell you. I'd like to ask, from your perspective, given that Terraform's entire, one of the reasons that Terraform exists

is to support infrastructure across all providers. If someone goes all in on a particular provider, do you see that you have something to sell them? Yeah, so I think we're different because we sit at that workflow level that even these passionately single cloud companies, we're seeing them use things like Terraform in many cases, not always, but in many cases.

And that's because Terraform is a lot more than just the infrastructure. And I think you brought this up. It could glue together so many different levels of things. It's DNS providers, it's GitHub source control, it's CDNs, it's DNS providers, it's GitHub source control, it's CDNs. It's things like that that, you know, a cloud provider alone

doesn't often provide all of them. And so Terraform is a good way just to do that. Even if you say all our all our compute networking storage, like the traditional things are all on a single vendor, but we want to pull

the latest commit from GitHub and use that as a way to tag things or something. I don't know. There's all sorts of use cases sort of like that where Terraform is super useful. So I think that it doesn't affect us as much. And we explain the benefits we view of multi-cloud from a workflow perspective. But the ironic thing is we have a lot of paying customers that are both single cloud, and we have a lot of paying customers that are both single cloud, and we have a lot of paying customers

that are actually not cloud at all. They're all just private data center. And so I think we build technical tools that work with both, even though we think that we use multi-cloud as a way to basically explain how workflow portability is extremely important. I think that's the narrative that makes sense, and I think that's what we're starting to see resonate in the space.

I also do want to special case the idea of hybrid cloud, where you have an on-premises and data center environment that tends to be there for a while. Having a hybrid strategy makes sense. You can also see it as a transitional state, and for a lot of these companies, that transition looks like the better part of a decade through no fault of their own. That's okay. One thing I want to make very clear is that I'm not trying to be architectural shaming

of anyone here. The idea of, well, we built a thing for certain constraints. The constraints have shifted and changed. 20 years later, we have this thing that makes money. We can't just wave our hand and turn it off because something on a whiteboard looks more appealing now. I have a lot of sympathy for that. I don't think anyone has intentionally gone out to make a series of poor decisions on this. Yeah, and I mentioned it in our keynote at our last conference that

I'm sorry if this is a negative thing, but I think we've made those transitions a lot longer for a lot of people because our tooling has made the quote legacy environments a lot more productive. And so Terraform is actually one of those places where you see it pretty often where people adopt Terraform for their cloud adoption story and then realize like,

oh, hey, Terraform works pretty great with something like vSphere. And oh, wow, we can automate our entire on-prem setup end-to-end much better. Let's just actually move that fence post of decommissioning this off another couple years. And so, again, one of those ironic things, we've signed a lot of customers where they're saying,

okay, we're completing 100% transition to the cloud in, let's say, three years. And now we're talking to them two or three years later, and they're like, maybe we could have done that, but we're happy, and we're going to push it off another three years. And so I'm either sorry or kind of excited that people could get more value

out of what they already paid for. Oh, I think the idea of being able to stretch an existing investment further is compelling to everyone. I don't think anyone has any particular problems with that. You see people moving from on-prem to public cloud as well, and there's just almost a comedy of strategic errors that you see getting made along the way, where first they'll spend eight months figuring out what it's going to cost them. Spoiler, they're going to get most of it wrong, and it's effectively an educated guess.

Then they're going to start the process. wrong, and it's effectively an educated guess. Then they're going to start the process, it's going to take longer than they thought, it's going to be more difficult, and they're not going to save money during the initial phases until they start teaching applications to dynamically scale, which many of them don't. But it's worth doing in most cases anyway from a time-to-market and feature velocity story,

even if the cost savings won't be captured for a longer timeline than they would expect them to. It feels like there's almost an entire cottage industry that's sprung up around making those migrations take longer than they need to. Yeah, we're still super in the, it's kind of sad in this way, but we're still super in the beginner cloud usage phase for, I would say, the majority of large companies right now. It's like you brought up auto scaling. I mean, auto scaling is still so rarely used. Actually, Microsoft Research published a really interesting paper about doing an analysis across.

I believe it was Azure, I'm sure. But it's sort of an academic paper. I mean, you could read it publicly, but they did this study on how many people are doing autoscaling and what the benefit would be if they did it theoretically, and the number was laughably small. And if you exclude people that use autoscaling

as a simple way to just make sure a single VM stays up and running or a fixed count of VMs stay up and running, it's basically zero. So I think as we get closer and closer to actually getting to more advanced resource usage like that, then it'll get better. But really, the conclusion of that paper and the conclusion I'm getting to is cloud adoption is more or less lift and shift still at the moment.

And one of the problems I have with it is the idea that that's somehow a terrible pattern. Because the alternative is you transform as you go, okay, we're going to take this application in the data center, rewrite it to take advantage of cloud-offered primitives, and then deploy it, and then it doesn't work, or

heaven forbid, it's intermittently bad, and then you get to start going through this entire analytics process. I'm a fan of lift and shift, but you have to follow through on the second phase and expect that to take longer than you think it will. Yes, it's going to take a very long time. I'm saying that optimistically. I always am super clear about, if it weren't for, and I always am super clear about, like, if it weren't for cloud existing, then companies like ours wouldn't have even had a chance because the existence of cloud is forcing companies to open their minds to the idea of maybe adopting a new way to do infrastructure or a new way to think about security or networking or things like that.

And so if cloud didn't exist, then I don't think something like Terraform would be nearly as popular, or Vault, like the way we do security, would be different. And I think software is getting better and the infrastructure around it is getting better, even if we're still in this beginner phase

of using what the platform actually gives you. I think that you're onto something there. The idea that you can take wherever you are and have tooling that winds up meeting you there and helping you get to a better place or get more out of what you've already bought is compelling. Now, not to get too annoying for people on this, but let's go to the exact opposite direction

of making everything better and talk a bit about CloudFormation. So when Terraform came out, CloudFormation already existed. Was it just the multi-cloud story that inspired you to start building something that stood in its place? No. So, okay. So let's talk about the history of Terraform because there's a truth in the history of Terraform, which I don't know. I'm sure I've said it somewhere publicly, but I'm not sure if I have, which is that when we first made Terraform, we actually didn't see the multi-cloud use case. We built Terraform as what we felt was just a

better way to do infrastructure as code on single providers at a time. It wasn't AWS only or anything. It was either you wrote Terraform code that worked just with AWS or just with Google and like, couldn't, couldn't overlap them. Basically they multi-cloud like multiple providers in a single configuration, sharing data and stuff. That was a coincidence prior to the release of Terraform, which is fun. And I think that coincidence was the thing that made me actually realize, oh, I think we created something really cool here. But that wasn't actually the original motivation. The original motivation was that I felt

we could do a better job. And taking a look at it now, I would argue that by any objective standard, there's a strong argument to be made that you had. We still see scenarios where Terraform providers for new services come out before CloudFormation does, which is just ridiculous to me. And I don't pretend to understand what goes on behind that. But very often people I've seen go directly into a Terraform world for pure AWS environments just due to a wide variety of factors that all distill

down into it offers things that CloudFormation simply doesn't. Yeah, I mean, Terraform definitely offers things that CloudFormation doesn't. On providing resources, I think the open source community is a big help in making us go faster. But at the same time, someone did an objective analysis of CloudFormation versus Terraform and found that it was more or less a wash in terms of resource speed. There was like, there was a bunch of stuff CloudFormation was first at, there was a bunch of stuff Terraform was first at, and I don't know, it's kind of a wash. But yeah, I mean, it's, it's, there's a bunch of other features, right? Like the

ability to run it as a CLI on your own machine and not use a dedicated service. For a long time, Terraform plan was completely unique. I think CloudFormation has something like that now. But I think the biggest thing overall is I think the ergonomics of the language are better in a lot of ways. I think that the fact that you could glue together different providers and enhance it by writing your own providers is extremely attractive. But I'm speaking from the most biased point of view. Well, absolutely. I mean, you've even got support for a CloudFormation provider, to my understanding, where you can take any arbitrary CloudFormation

template you want and do that within Terraform. Yeah. I mean, there's practical, pragmatic reasons for that, which is you're transitioning to Terraform and you've converted a bunch of it, but you still have that one CloudFormation template that you have in full. Just, we'll run it for you. Oh, it also absolutely empowers my preferred developer model, which is copying and pasting directly from Stack Overflow,

which is why I'm a full Stack Overflow developer. It winds up making a fair bit of sense for me. One decision you made early on that I would love to ask you about is you picked writing your own domain-specific language for Terraform. I knew it was going in this direction. At the time, CloudFormation was only JSON. Now it supports JSON or YAML, but I'm holding out for XML just to ensure that I'm irritating everyone with this question. Why a DSL and would you do it again?

Yeah, so this always hits different people with different chords. So one thing to, I think, put into context about the history of our company and our tools is that we've tried both ends of that spectrum, right? So Vagrant was the first thing we made. It's almost 10 years now at the time that we're recording this. And it was all Ruby DSL, like Ruby config. So you can do anything if you're proficient with Ruby in a Vagrant configuration. And some people love that and some people hate it. And then, you know, before we go deeper into that,

like there's, then we did Packer after that. And because of what I learned from Vagrant and sort of the sharp edges around Vagrant, I decided that the obvious solution with Packer was to use a pure data structure language. So I used JSON. And JSON, a lot of people love that. And the idea behind JSON was, you know, any modern language can generate JSON.

So if you wanted to use Ruby, you could actually use Ruby and just generate JSON. There you go. Like you have your DSL, right? That's not strictly true, but it solved a lot of problems. JSON was good. But then I realized that there's a lot of people that hated it. And the joke I tell people is that there's about once or twice a year, there's a

single week where I'll get two emails of someone saying, I love HCL, which is the name of our config language in Terraform. I love HCL. And then there'll be someone who says, I hate HCL. Or someone that'll say, I love Ruby. I love Vagrant's Ruby config. And then someone someone who says, I hate HCL. Or someone that'll say, I love Ruby,

I love Vagrant's Ruby config, and then someone else that says, I hate Vagrant's Ruby config. And I think when you get these totally split spectrum points of view, what it represents to me is sort of that it's an opinion. There's something here that you can't really win. And when we took a look at HCL, which was Terraform's language, we were trying to balance these two things as best as we could.

So what we did with HCL was the actual native HCL syntax is as human-friendly as possible. It's meant to be human-readable, human-writable. It's not meant to be machine-friendly in a big way. But what we did was make it a one to one mapping with JSON. So there is an exact JSON syntax that every HTML config maps to and vice versa. And that is super useful for machines. And we did this thing, I think, you know, you can't really win, we still we get some people, like I said, that still love

it, we actually ran some like independent studies across users who had never heard of us, users who had used some parts of our tools, users who have used all of them, and HCL performed the best in terms of what people liked the most. But yeah, it's a hard thing to say right or wrong. And if nothing else, you've given people something to focus on that isn't core and central to the ethos of building the tool. It's, no matter what you pick, someone is going to have a problem with it somewhere on the internet, loudly and usually not particularly coherently. Yeah. The problem is even if they direct their hate or whatever it is towards me or whoever for doing this abominable thing, I still can empathize. I understand the frustration because I prefer some things to other tools, and it's there, but it's hard to satisfy everyone.

Well, last question around the, I guess, glorious dissatisfaction of various people. Originally, when Terraform came out, it was coupled fairly tightly with Packer, Console, Nomad at the time. And this was all wrapped together in an enterprise offering called Atlas that you folks offered,

which was fantastic having played with it a couple of times. The challenge that I saw with adoption when I was talking with various clients back then was always that it required an all-in investment, where suddenly you're using all of the HashiCorp stack or else you're sort of on your own. And given that you no longer sell or reference Atlas, I have to assume that that wasn't just something that came out of my own experience. Can you talk a little bit about what that tight coupling looked like and I guess why it isn't around anymore? Yeah, it's gone. The idea was that what's the best way to build a business out of a product? It's like ask the customer, right? What do they want? And if you give them what they want, the theory is that they'll pay you for it. And, you know,

that's the dot, dot, dot, and then you're on step three profit, and you're happy. So we did that, right? Like, we asked the customer what they wanted. The problem was that the customers, quote, unquote, we're asking, were HashiCorp fanatics. And when you ask that segment of user, you know, what you want, they're going to say, we have all these Hotshcrub tools. We love them. We would love if you just tied them together for us to make it work. And so we did that, and we had a small group of pretty happy users of what we built. The issues really came around with the not fanatics, with the people who either didn't know who we were or adopted like one or two things because exactly like you said like they there was no incremental adoption and even worse there was

not really incremental pricing i mean that's a fixable thing but it just made it hard to reason about like things were expensive because there was a lot more value that they didn't want out of it and things like that and so what we realized is what we built as an enterprise product as our business was sort of counter philosophical to how we viewed the world in open source. And that was a problem, both philosophically and in practice, because, you know, our open source point of view has always been, you know, we have six open source tools that do dramatically different things. And whereas one open source vendor might have put them all in a single mega tool, we've always been really clear that we split them up because it

makes them easier to adopt, easier to learn, easier to interface with other software, things like that. And we like completely threw away that philosophy originally for our enterprise products. And yeah, I mean, saying it out in hindsight, it's so obvious, but what we ended up doing was just sort of breaking it down in that same way so we could let people adopt an enterprise version of Vault separately from console, separately from Terraform, et cetera. No, I think that it's one of those lessons that only really makes sense after you've either gone through it once or seen someone else go through it, and I don't know if it would have been possible to predict that going in.

I mean, sure, but the benefit of hindsight. It seems like it, yeah. Yeah, Everything feels so easy to predict in hindsight, but in the thick of it, you never know how the market's going to react, how the market's going to shift. At the time we met, I was speaking largely around configuration management. It seemed like that was a full steam ahead business unit. Now it seems like the entire industry is taking a turn toward immutable infrastructure and I can either sit here and shake my fist at this

and refuse to change, or I can find something else to be good at that isn't imperative or declarative configuration management and instead start looking at something else. Infrastructure as code has clearly carried the day. Yeah, and I was actually, coincidentally prior to recording this, I was just talking to another founder

who has pretty cool, very hip, very edgy technology. And one of the things I was telling that founder was, don't try to be revolutionary in too many things because if you attach yourself to something that's already like rocket ship rocket shipping up you know like it their marketing dollars lift you up too but if you fight it and you say like now our way is the true way then you're just fighting something that you don't have the resources to fight and i think we didn't try to do that but that's what we were

happening to do was saying you know oh yeah oh, yeah, you adopted these three tools, but really the one true way is our mega platform. And, you know, cast aside the fact that it worked and it was pretty cool and things like that. Like for us, a lot of people didn't work, but, you know, for the right type of user, it did work. But even if you throw that aside, like it didn't matter because we're fighting what the rest of the industry was telling people to do. And you kind of have to be pragmatic about that and say, okay, they're going to adopt this one thing we have because they see the value in that. And even though we want them to use something like Terraform, let's say, they're going to use something like Azure Resource Manager or something. And we just have to be

okay with that. We switched to that sort of point of view. One thing that continues to stick in my mind a year and a half after I did it was, after the first year of running my newsletter, I wound up conducting a reader survey. I got almost 1,000 responses and asked people, what are you using to manage your infrastructure? CloudFormation and Terraform were within the margin of error of being equal to each other. Now, in hindsight, I could have constructed the survey a bit better,

because let's not kid ourselves, an awful lot of shops are using both in different ways. But it did turn into a very clear, this is as widely deployed among an AWS user base, that it is large enough to be statistically significant, that this is something that's real. This is not just some random project on GitHub that someone stumbled over. You've tapped into something,

you're seeing adoption across the enterprise and in startups as well, and everything in between. Yeah, and I already shared this publicly, but I won't name the exact cloud, but there's a major cloud provider where they've shared with us that Terraform drives a double-digit percentage of all infrastructure that they spin up. And that's not just compute. That's the databases, the load balancers, things like that. And so it's not just statistically significant.

I mean, that's financially significant. That's industry significant. That's industry significant. That's really cool, but I think that goes to show that there could be an independent infrastructure as code thing that can rival the first-class, first-party solutions that the cloud providers provide. And that's just a fantastic story. It's nice to see that even in this world

where it feels like the cloud providers are eating everyone's lunch, that it's probably not true. You just have to offer something that the market needs. Yeah, yeah. And I think we did that, and I hope so. It would seem that the entire industry has agreed with you at this point.

Mitchell, thank you so much for taking the time to speak with me today. Yeah, nice talking to you. If people want to hear more, where can they find you? I don't hide. Just, you know, Hashicorp.com for our company or Mitchell H on Twitter. I tend to respond to as many people as I can. I don't hide my email. So you'll figure it out through those avenues. Sounds good to me. Mitchell Hashimoto, founder and CTO of Hasha Corp. I'm Corey Quinn.

This is screaming in the cloud. This has been this week's episode of screaming in the cloud. You can also find more Corey at screaming in the cloud.com or wherever fine snark is sold. This has been a HumblePod production. Stay humble.