Screaming in the Cloud - Episode 41: Open Source is Not a Business Model

Episode Date: December 19, 2018

Have you ever had high expectations about a new software product? Did you think it was going to be spectacular? Instead, did it become less about solving a problem for you and more about reaching a bunch of billable consultants? The dynamics of open source communities and the Cloud platform can make or break software products. Today, we’re talking to Andrew Clay Shafer, who was a notable voice during the days of OpenStack. He had high hopes for OpenStack, which was an effort to bring a democratized solution of Cloud computing to anyone’s data center. He describes the importance of understanding the challenges associated with open source projects in order for them to be successful. Some of the highlights of the show include: Open source is not a business model; capture value for customers, or they’ll go with a different solution Openness/Closure: Every open source project has its own community dynamics Losing sight of level of expertise for profitability and easy path to useage Whether to become a product or service company - difficult to be both effectively or go from being one to the other; build partner relationship, focus, and say “no” Lack of awareness about AWS Outposts admitting public Cloud is no longer a viable business model Amazon relentlessly focuses on what its customers want and tries to keep promises about what it can and can’t do Cloud Native: Not where you run, but how you run; confining variables Self-fulfilling prophecy to under deliver when you make the bad decision to under source IT across the board Cloud Native, DevOps, SRE: Buzzwords that equal one thing and work together Dilemma of not building everything and buying some things, but you can’t buy everything; humans like to shop and go with the easiest option Links: Andrew Clay Shafer on Twitter Andrew Clay Shafer on LinkedIn Puppet Re:invent OpenStack Eucalyptus Docker Redis MongoDB Confluent Kubernetes AWS Outposts AWS Ground Station AmazonBasics Simon Wardley Maslach Burnout Inventory Datadog .

Hello, and welcome to Screaming in the Cloud, with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This week's episode is sponsored by Datadog. Datadog is a monitoring and analytics platform that integrates with more than 250 different technologies, including AWS, Kubernetes, Lambda, and Slack. They do it all. Visualizations, APM, and distributed tracing.

Datadog unites metrics, traces, and logs all into one platform so that you and your team can get full visibility into your infrastructure and applications. With their rich dashboards, algorithmic alerts, and collaboration tools, Datadog can help your team learn to troubleshoot and optimize modern applications. If you give it a try, they'll send you a free t-shirt. I've got to say I love mine. It's comfortable and my toddler points at it and yells, dog, every time that I wear it. It's endearing when she does it and I've been told I need to leave their booth at reInvent when I do it. To get yours, go to screaminginthecloud.com slash datadog. That's screaminginthecloud.com slash d-a-t-a-d-o-g. Thanks to Datadog for their

support of this podcast. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined today by Andrew Clay Schaefer, who is himself famous for screaming at clouds. So having him on the show was really just a matter of time. Andrew, welcome to the show. Do we scream now? We absolutely can. It's fun. There are a couple of companies I wound up encountering at reInvent

who wound up casting their company names entirely in capital letters. So at some point, I feel like there's going to be a gag where every time I mentioned their company name, I actually scream. And then I realize exactly how unpleasant that's going to be for someone listening to this on their commute. If you wind up just screaming mid-sentence out of the blue and someone rams a bridge abutment, you kind of feel bad for that. You can also level the volumes. That sounds like something smart people might do. Post-production.

I'm told that's a thing. So you have an interesting history, historically. You were a notable voice in the days of OpenStack. Is that something we can talk a little bit about, or is that still too painful of a memory? I had such hope for OpenStack at one point. What do you want to talk about? I think OpenStack was an effort to do something interesting to try to bring this democratized solution of cloud computing to anyone's data center. Unfortunately, one, that's a very, very hard problem. And two, the dynamics of the project

just kind of got turned over to essentially marketing way too early in its evolution. And the other byproduct of that was it sucked the oxygen out of a number of other projects that probably deserved a little bit more oxygen than they ended up getting. I don't know. What's the specific thing to talk about OpenStack?

I loved what it was. I thought there was something spectacular going on with it. And I started playing with Eucalyptus, which was a Java-based competitor that was nowhere near as open. And that worked a bit better. The problem is every time I tried to get into it, it felt like there were 800 ways to do it and everything was built around a construct that seemed to be less about solving the problem a customer had and more about building work for an army of 500 billable consultants for the next 18 months. And that meant that when I was trying to stand things up in my test lab, it didn't take long for me to go from this looks interesting to nope, nope, nope. And that was the end of it. I mean, there's a lot to unpack. So one is there's these dynamics of open source communities, which might also be interesting to talk about. But you mentioned Eucalyptus. And Eucalyptus essentially had the mantle and the right to be this open

source cloud platform that everyone sort of made OpenStack or centered on OpenStack. But they didn't understand a few things. One is the real distributed systems problems you need to solve to do this the right way. Eucalyptus was, for the most part, a Java monolith. And that was not the problem. I mean, it was a problem as you start to scale and try to build these things but when they were in that early phase and interacting with some of these organizations that were interested in having that be a solution that were running up to the scalability that that solution you know as a java monolith could handle then they did a very poor job or they didn't understand the dynamics of open source such that they essentially created OpenStack by not fostering that community.

So I know a number of cases and we won't need names, but there were people that were trying to contribute to Eucalyptus that were prevented from doing so because the Eucalyptus mindset was, oh, those are features that will be in our commercial product. And that ended up essentially creating the gap that OpenStack stepped into. That seems very similar to almost a retelling of what we're starting to see now in a different form, in that you're seeing a number of open source projects that are having a bit of existential challenge as far as they build something, they make it available for everyone, and their business model is either to sell an enterprise version or support around the thing that they released. And then large cloud providers like Amazon come in and build an entire package offering around that and wind up effectively

selling the thing that a bunch of people have contributed to and making money out of it and not giving anything back. That's completely permitted by the license. So we're starting to see things like relicensing and the existential question of how am I going to make money off of this when large vendors will take what I've built, repackage it as a service, charge money for it, and never contribute a single thing back, be it code, be it money, be it support, any of it. Where do you land on that? Well, for the most part, I've made my living for the last decade off of open source in some form. And I've been on both sides of this. And I think that you just have to understand

the open source is not a business model. And that it doesn't matter what the quality of the open source is or what you're creating. If you don't solve a way to capture value in a way that is conducive to your customers getting it, then they're going to get a solution from someone else. I mean, there's just so many nuanced things that are happening and people, especially once they've gone and taken a lot of investment with the expectation of an outcome for their investors, then you end up really, really struggling to find that balance where I think it's actually worth backing up. When someone says open source, that's not a thing. That's their openness with regard to their community? What's their openness with regard to the way that they accept contributions or that they help those users? So in the last six months, we've seen a number of these things

re-licensed, like you said. You've also seen an interesting dynamic in the Clojure community. I don't know if you're following that, but there was a, a post from, from the main author of closure about how essentially, you know, it's not his responsibility to care about the community. At least that's my, that's, that's my short version of interpreting what he said. I don't know. There's like so many threads that we could pull apart here. Maybe it's worth refocusing on one of them. I think you're right. It's also one of those areas that I am

acutely aware that if I put a foot wrong, I will wind up with 8,000 people in my inbox and four at my house. I'm not in this to slaughter anyone's sacred animal, regardless of what that tends to be. I guess my concern is, it's challenging. We see this with Docker, for example, where they released an amazing transformative technology. They were never quite clear on what their story was going to be around monetization. And it turns out VCs generally want to see a profit story come out of this. I never saw a container story that involved step seven, give money to Docker in any time I wound up building things around that particular technology.

So when you take a look at things that have relicensed, for example, aspects of Redis have done that recently. I believe MongoDB did, but don't quote me on that one. I haven't been following this with the rapt attention it probably deserves. And I understand where they're coming from. I take the Redis story. They watched their service from Redis Labs, so the product from Redis Labs, be turned into an ElastiCache service. Click button, receive a relative baseline Redis environment that just works. Suddenly that need for that level of expertise in many shops

is either not there as strongly or is perceived not to be there as strongly. And that's a scary thing. But I think you nailed it when you said that open source isn't a business model. It's a way in many ways of achieving certain outcomes and getting other people who aren't just you involved in something you have going on. And it's one of those things you lose sight of at your own peril. More threads to talk about. So I mean, you can't talk about OpenStack and some of these things are happening now without also talking about foundations. I'm not sure that's worth totally dissecting right now, but the foundations end up in this kind of weird standoffish

relationship where some of them are kind of pay to play and they try to have this blanket umbrella, you know, neutral Switzerland approach this blanket umbrella you know neutral switzerland approach to things but that's not always entirely true and then on this notion of a business model open source that comes out of a company like redis like confluent like docker where you have a single company that's kind of trying to make go at profitability and even has taken investment, is fundamentally different dynamic than what you see coming out of Google or coming out of Netflix or these other things where they're getting leverage from open source as an effort, as an investment that is different than the profit. They're not trying to generate a profit directly

out of TensorFlow, out of Kubernetes, right? It's essentially in some ways a marketing exercise as much as a, as much as a business. And then to go back specifically to Docker, I mean, I think Docker is interesting. There's a couple of good lessons in Docker in some sense, you know, use the word technology and some people will take issue with what I'm about to say, but I don't feel like Docker was really technology. I feel like Docker was unveiling or revealing fundamental features that had been in the Linux kernel for a while. And as a consequence of that, what they basically did was fix a usability bug with the way that, you know, C groups and namespaces were managed and gave you sort of had to be a system-level person,

and you kind of need to make sense of these really long, convoluted command line arguments to set up the root file system and all the flags or whatever to get the right behavior. But the Docker kind of wrapped that up in a usability layer that gave you access to it. So it turned the world on to what these possibilities were, but didn't necessarily create a new technology. And so they also had almost no moat to protect their business. Right. And past a certain point, when you fix a usability problem, it's one of those challenges where it's very hard to turn. We surfaced awareness of this thing and gave slightly better tooling

and visibility into what's going on, then turn that into something that's reliable, sustainable, and as you say, has a moat that is a defensible business strategy. Well, it kind of goes back to this comment you made about the consultants for OpenStack or the cloud stuff, because what happens in these dynamics, and I experienced this with Puppet to some degree as well, where you create something that is intrinsically valuable and easy to use you almost get less less leverage to capture value so you have to be very very careful if you're trying to build these businesses on open source to you know

assuming you're not printing money because you're because you're google to to do something where you have something that is is going to enable those transactions that you ultimately hope happen. Yeah. And that's the sort of thing where you also wind up with companies who are trying to navigate the question of, do they become a product company or do they become a service company? It's very hard to be both effectively. And it's even more challenging to go from being one to being the other at any sort of scale. I think that there's a little bit of a false dichotomy in that. This is a quick aside. But if you look at the way that that's been projected, and you also

look at the reality of enterprise IT, there is no successful enterprise product company that doesn't also do a massive amount of services. The question is, and really the bright line in my mind between a consulting company that is only a consulting company and a product company, especially in enterprise IT, is being able to focus and say no. So in the mercenary for hire, anything for anyone, IT business, then you're chasing the shiny coin. And that's ultimately what prevents you from capturing value from a product. But there is no product company. There's definitely consulting companies that don't have products, but there's no enterprise IT that doesn't have services. Where would you put enterprise IT companies

that handle the service aspect exclusively through partners, or don't those exist? I think that those definitely exist, or the proponents of their work will be through partners. That's certainly the way that the empire was built at Microsoft to some degree, Oracle probably as well. I think that the case there is you're basically choosing to do things through the partners. My point is not necessarily that you can't be more pure as a product company, but you have to enable that partner thing. And you couldn't do that until you get to a mature market. So what I see time and time again, and some of my smart friends made these mistakes before, is you want to be a product company or

you made these promises to your investors to be a product company. So you're trying to be a product company, which means you don't want to do services. But what that actually means is you don't have a relationship with the customer. And so you don't have good feedback loops into your product. So I think at the point where you have mature products that are at product market fit, then you can leverage these partners to grow that into world domination strategy. But if you don't actually understand that life cycle of how the partners are going to execute on this thing that's mature, and you're still in this nascent phase trying to build it, and you don't own a partner relationship, you end

up in a really bad place. And your product development suffers as a consequence. I think that's probably one of the more astute assessments of this that I've heard, which sort of makes me wonder why so many companies seem to stumble in this area. People are terrible at product management. If you want to have something engraved in a tombstone early, I think that might be a great epitaph. I'm hoping for better, but I'll put it on the list. Absolutely. So taking a bit of a different tack for a second here, AWS outposts has caused a bunch of kerfuffle. I sort of expect to see, for those who are not familiar, AWS outposts are effectively sealed racks that AWS will put in your data center if you pay them enough money and run AWS hardware inside of them and AWS services float on top of them. And now you have the AWS APIs on-prem. And a take that I saw around this was, aha, they're admitting that only public cloud is no longer a viable business model. And I'd expect that on Twitter, but I didn't expect it in headlines at Business Insider, for example. So I'm a little disappointed at the lack of

awareness a lot of journalists are bringing to this. What's your take? I'm just going to laugh. I mean, journalists are going to be journalists. They don't necessarily know what they're talking about. The reality is that cloud, to me, cloud native or whatever you want to say this thing is, is not where you run, but how you run. And you're literally getting the APIs in your data center. It's essentially what you wanted from OpenStack in the first place. And I think from the other side of that equation, if you look at what Amazon is really trying to do,

and this is something I've heard you say and other people are fond of saying, you can meet people where they are. You're giving them the same workflow on the same APIs in this data center that they manage and you know at some point that they're going to put that stuff in in your cloud in your public cloud because you're going to manage it better than they can guarantee absolutely so i think it's a long bet i mean they've already got some investment and and practice with some of the other things they did earlier snowball or whatever where they they have started putting hardware in places.

And this is just expanding. It really comes down to, and this is in some ways dovetails back to the comment I just made about product management. But one of the things that you can, you know, you could fault Amazon about a few things, but you cannot fault them for their relentless focus on the customers.

And that customer focus is really what drives all these things. So if you have a bunch of customers, they're saying, give us this thing in your data center, then yeah, it might take a while, years. But here we are, and they delivered it. And they focus on what their customers want, but they also try to keep promises about what they can and can't do. So at the point where they felt like they could keep those promises, they shipped those racks. Absolutely. I think there's a story here about how they also that same week launched ground station, which provides radio telemetry reception

for satellites in orbit. If the total addressable market of people with satellites in orbit is a large enough market segment for them to focus on. I'm going to go out on a limb and hazard that the people who are not comfortable with public cloud for all of their workloads is a larger total addressable market. And meeting people where they are and solving customer needs is absolutely what AWS has historically been about. Not to meander too much, but the fact of the historical evolution of Amazon, building a business, selling books on 1% margin gives them an advantage to explore all these things, right? If you have a margin and Amazon thinks they can get something

similar to market for half of that, they'll do it. You know, and it's not just about the cloud, right? I get 10 bucks a day from Amazon prime some days, thanks to my wife. But the stuff they're doing with like Amazon basics is basically an Amazon generic brand where they're like, Hey, we can make these things. We can do these things. You know, we can, we have whatever understanding of that market. We have whatever understanding of the manufacturing. We can basically put in a spreadsheet expecting to sell this much, and it's a done deal. It also, and I thought it was very astutely delivered from a marketing perspective, where they didn't have to say, the reason on-prem environments tend to suck is because people cheap out on the hardware, and then they mismanage the crap out of it. By shipping a sealed rack to customers,

they're effectively reducing the customer responsibility to more or less power and cooling. And that's not something Amazon has an offering around this year. But they're really moving things up the stack, even in environments that they can't control. They're still trying to eliminate as many variables as they can. They're not offering their control plane on your own custom crappy hardware, for example. I think that's the key to understand here. And this to me defines cloud native. And I don't confine cloud native to just Amazon, but it's really about confining those

variables. So if you look at the way Google's built data centers for the last 15 years, you're talking about football field size data centers with the last 15 years, you're talking about football field sized data centers with racks and rows of identical gear, right? So by collapsing those variables at the bottom of the stack, you can invest more of your complexity on the higher order features and applications. Oh, yeah. There's a story here about economies of scale. I used to rent from an individual landlord when the refrigerator broke. It would be a week until the repair person came out. I moved into a corporate building that had hundreds of units that were all identical, and I reported a broken refrigerator. They were there with a brand new one within an hour. And they have a storeroom downstairs with five new refrigerators, another 10 of them in spare parts. So it's at some point when everything winds up looking identical, it really starts to be a, oh, you just need a number seven. Got it. As opposed to everything being bespoke and difficult because it's the first time or the only one of anything you're running. It's the hardware equivalent of pets versus cattle. Another great point to tease out,

and this is probably the take home that drives a bunch of the bad decisions in a lot of these places. You said that they didn't have the right hardware, they under-invested in their hardware. I think that the reality is for enterprise IT in most places, especially where they told themselves they weren't a tech company, that they've under-resourced IT across the board. And in the worst cases, they're even managing IT, CTO reporting to the CFO. So when you treat these things as a cost center and your only lever is cost management, then it's a bit of a self-fulfilling prophecy and you're always going to under deliver. So it's a little bit of a misnomer or I think it's a fallacy when people think that these IT

departments, which are under-resourced and calcified for decades, are going to become this digital transformation center overnight because they decided that they want to become more savvy, they want to compete, whatever. Until you reframe that mindset at the highest level, and that has to do with comp, that has to do with this investment in the hardware, everything to do with how that money gets spent, then you're going to have a bad time. And especially if you're competing with the Amazons, the Googles, the rest of the world that has sort of made that leap and views IT and technology as a weapon. I think you're right. I think that it also gets a bit into what Simon Wardley has been talking about with value chains, where the differentiation that makes your company succeed is probably not going to be in stuff that's already become heavily commoditized, which is why it always frustrates me to see

companies that sell socks more or less, focusing all their energy on building their own customized container orchestration system. It tends not to be the path to victory in anything other than the extreme long-term, which is not the planning horizon most companies are operating in. This is hard to navigate. I'm a huge fan of Simon Wardley. And if any listener is not familiar with Wardley Maps and Simon's work, then I encourage you to go waste a day familiarizing yourself. But the reality is that you have not just okay to decide what's commoditized, but also what will be your thing. So I do think that there's a need to focus on the higher order functionality

and certainly Amazon, the other cloud providers and some other vendors are trying to help people do that. But the reality is not everything is off the shelf right now. And what is a commodity today will be a commodity tomorrow, but what is not a commodity today will probably be a commodity tomorrow, right? And so it's like figuring out when you want to make that jump to make those investments or not make those investment sets. That's a hard choice.

It's not easy, as I understand. It really is. Out of curiosity, where do you see cloud native fitting into the landscape along with things like devops sre and other buzzwords that wind up causing people to get all hot and bothered well i'll give you i'll give you the the short version well there's a much longer version this will still probably seem a little long but the cloud native as a moniker was something that i specifically used as as part of pivotal messaging for a long time. Before there was a cloud-native computing foundation, Pivotal was using cloud-native as a term. And to me, it's really subsuming

all the buzzwords. I think that when you look at DevOps, it's a focus on the interface. I mean, I think of it as a much bigger kind of systems thinking thing. But in terms of how you can explain it to someone and not end up sounding like a new age guru, it's really just this interface between developer and operations and trying to smooth the deployment and operation of software. When you get to cloud native, I think you open up into a much bigger story around architecture, some of these other patterns. And then I also think that there's a huge culture element. One of those things I just mentioned around the way framing of IT ends up being a

cost center versus a competitive advantage. So to me, the cloud natives, they definitely do use DevOps as a way to do things, whether they want to call it that or not. Certainly you have SRE as a development and some people have a fun time making these link-baity arguments about SRE versus DevOps. But in reality, it's really one thing to me. And I would also note that these are not things that people set out to do. People didn't set out to do DevOps. People didn't set out to do SRE. It was a function of the Darwinian pressure to deliver these things at scale, with reliability, with speed, reliability, scale, whatever, all the ilities and idities that you get when you're a Google or an Amazon. And that Darwinian pressure had a solution.

And that is very, very common. So you can find common patterns, not identical patterns, but certainly common patterns across all these companies, right? So Google, Amazon, certainly, when you get into the Twitters, the Facebook, some of these others, they manifest these same patterns, right? So to me, that evolutionary cohesion that gives you the ability to deliver software reliably at scale, you know, rapidly iterating to serve the customer. That's what cloud-native means to me. And that's tech, and that's process, and that's people, and that's everything. I think that there's a serious misunderstanding in the larger industry, sector, world, whatever you want to call it, around the idea that the solution to your problem is something you can buy,

where you just cut a check to someone and magically the problem goes away. Cultural change never works that way. Now that a process improvements, it's something that you have to internalize as an organization. Believe me, I wish it were otherwise. I would love to sell people the magic solution in a box, answer to all their problems, but I've just never found it. So this is reinforcing the point I was making a minute ago when you brought up Wordly Maps, because on one hand, you want people to realize that you don't want to build everything and that there's some things you should buy. On the other hand, you can't buy everything. And people like

buying things because it's more concrete than this abstract notion of culture, right? We start to talk about true transformation, true digital transformation, especially an organization that's done something for decades. Then the idea that all of a sudden you can buy this tool or this certification or implement this thing and you're going to be done, it's absurd. But you see that happen literally every generation. And to me, it literally reminds me of the lose weight with one weird trick, click ads that you get on whatever website, right? So people are always somewhat susceptible to the easy answer.

And I think that's just a dynamic that's developed, unfortunately. It's human nature. I'm not sure there's any patch for that that people would find even remotely acceptable. So last question for you before we wind up calling it an episode. You've given a number of talk ideas. Some years you've given a number of talks, others you've gone suspiciously quiet. And I guess I'm wondering, where do your ideas come from? What gets you excited or fired up about, I need to go on stage and quote, give a talk about this, which similar to me often translates into,

I'm going to rant into a microphone for 45 minutes, which frankly, I appreciate. So sometimes I get focused on tech and I'll get, and those are easy talks actually, because you want to just teach people and share how this works. I gave some talks about cgroups and namespaces, whatever. The ideas that I get most excited about, I often get them connecting things that are maybe from outside and then I just let them marinate and I try to do whatever kind of real-world experiments I can in my organizations or the people I'm working with at the time.

But the talks that I'm most proud of, they came from these organizational learning researchers, reading some paper, some psychology paper, the Maslach's hierarchy or Maslach's classification around burnout. All this stuff gets me excited reading the psychological or the management theory or whatever, and then trying to do those experiments in vivo and see what comes out of it. That tends to be a much healthier perspective than where my talks come from. It usually starts as a random story I'm telling someone and they're laughing and they say, that's great, but you'll never turn that into a conference talk.

And then I take that as a personal challenge. So- Challenge accepted. Exactly. Did you build that entire conference talk around that awful pun at the end? If someone walked up to me and said, I bet you couldn't turn that into a conference talk, I'm pretty sure it would be a conference talk. So we'll just throw that out there. Exactly. So if people are excited by the things you've been saying and want to hear more, where's the best place to find you? It's really easy to find me on Twitter, little idea on Twitter, and then Andrew Clay Schaefer,

LinkedIn, whatever. Don't just say you want to connect. If you send me a note, you want to introduce an interesting topic, something you want to throw back and forth, I'd love to discuss ideas, especially little ones. I'm right there with you. If someone wants to connect with me on LinkedIn, I should at least have had a long enough conversation to remember your name. If you saw one of my talks on video, perhaps, and then send me a LinkedIn request, I'm thrilled to connect with folks, but it's also, give me some context. What do you want to talk about? Not to be unkind, but convince me you're not a recruiter who's just really trying to scrape my network

because somehow, if you're connected to me, you're somehow perceived as being more trustworthy. No one should ever trust someone who claims to know me. Just the opposite, in fact. Honestly, you never have to talk to me, but just give me some context. Show me that you're interesting or interested in something. Exactly. You didn't just wind up on the grid of people you should connect with clicking until the API limits stop you. They'll never stop us. Exactly. Thank you so much for taking the time to speak with me today. It was a pleasure. Hopefully we'll do it again

sometime. Absolutely. This is Andrew Clay Schaefer, famous for Screaming in Clouds. I'm Corey Quinn, who's screaming at the cloud as we speak. This is Screaming in the Cloud. Stick around. This has been this week. This is Screaming in the Cloud. Stick around. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold.