a16z Podcast - a16z Podcast: All About Microservices

Episode Date: August 31, 2016

"Incremental change may be good theory, but in practice you have to have a big enough stick to hit everybody with to make everything move at once". So shares Adrian Cockcroft, who helped lead Netflix's migration from datacenter to the cloud -- and from monolithic to microservices architecture -- when their streaming business (the "stick"!) was exploding. So how did they -- and how can other companies -- make such big, bet-the-company kind of moves, without getting mired in fanatical internal debates? Does organizational structure need to change, especially if moving from a more product-, than project-based, approach? What happens to security? And finally, what happens to the role of CIOs; what can/should they do? Most interestingly: How will the entire industry be affected as companies not only adopt, but essentially offer, microservices or narrow cloud APIs? How do the trends of microservices, containers, devops, cloud, as-a-service/ on-demand, serverless -- all moves towards more and more ephemerality -- change the future of computing and even work? Cockcroft (who is now a technology fellow at Battery Ventures) joins this episode of the a16z Podcast, in conversation with Frank Chen and Martin Casado (and Sonal Chokshi) to discuss these shifts and more. The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/. Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.

The content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. For more details, please see A16Z.com slash disclosures. Hi, everyone, welcome to the A6 and Z podcast. I'm Sonal. Today's podcast episode is all about microservices, and I've been super eager to focus only on this topic on the podcast, since we mentioned it a lot in passing. And I'm really excited because we finally get to do that. Our special guest for this topic is Adrian Cockcraft, who helped lead Netflix's migration to a large-scale, highly available public cloud architecture a few years ago, making Netflix one of the originators and early adopters of microservices. And Adrian is widely credited for helping pioneer microservices at WebScale. Also joining the conversation, our A6NC partners, Martine Casado, and Frank Chen, who will be

moderating the discussion. And in this episode, we cover everything from what is microservices. to the evolution of the architecture, to how it changes the shape of organizations, to operations, to changing the role of CIOs. And finally, and this is actually what really excites me the most about this topic, is what new opportunities come up when you have these extremely ephemeral systems that are just like ghosts in the machine, from containers to servers on demand to serverless and what's happening there and some really interesting trends on that edge. The conversation begins, however, with the story of how Netflix got into microservices. Take us back to the days when

Netflix had decided they were going to move to Amazon and commit to a microservices architecture. Let's pick up the story there. So what's it like inside? We started off basically running away from a monolith. We had over 100 people every two weeks trying to get all the code they'd written in the last two weeks, jammed into one code base, get it through QA, and get that out into production. And that was just getting more and more painful. And we basically decided we had to break it into pieces. You want to. wanted it to be the work of one developer, basically controlling what they had deployed independently of everybody else. And at the same time, we were looking at moving to cloud.

Did you make both big moves at once? In other words, monolith to microservices and then private data center to Amazon? Everything together. And sometimes you find incremental change a good theory, but in practice, you have to have a big enough stick to hit everybody with to make everything move at once. And the big stick was we didn't have enough data center. capacity to support streaming. We were running the DVD business in the data center on a system that was growing at a respectable rate, but the streaming business was exploding at a much, much higher rate. And because of that, we knew we would have to either build lots of big data centers or get onto something else. So the bet was, okay, we need to go on cloud, then what's the

right architecture for doing that? What's the right organization for doing that? The developer group is getting bigger and getting less productive, and we wanted to unlock the innovation. So we were simultaneously trying to get better developer productivity, better time to value, which is one of the key things we're trying to optimize for generally. And then there was a whole bunch of other cloud transitions bundled in. As you went from the monolithic application to microservices, what did that entail? What's that mean? What is a microservices architecture? Well, originally, I called it fine grain SOA, service-oriented architecture. And there's a lot, some people get negative reactions to SOA because they were out there trying to do it 10, 15 years ago.

That's right. So it's all the same ideas over and over again with new dressing. Yeah. There's a question like, why now and why didn't it work then? And if you look at it, what we were doing was on relatively slow CPUs compared to what we have today, on relatively slow networks, we were processing big fat lumps of XML and passing it around. And we were really only able to break the application into a few large chunks because the overhead of all of the message passing was too high. If you come to today, you know, you can break it into, you know, maybe one hundredth of a size and a hundred times as many chunks because the overhead of the communication is now very low. We've got binary protocols. We're not trying to sort of make

everything conform to the big SOPEX-ML messaging schemes. So it became possible to build a fine grain SOA architecture and that ended up being called microservices by I think Fred George was the first to use the word, but that got written up by Martin Fowler and then everyone said, Okay, we'll go with that. Yeah. So, Big Bang moves. This was a bet the company set of technology decisions. Looking back at it, what are some of the lessons learned?

I think one of the ways to approach this is to basically create kind of a pathfinder or a pioneer team. There was a lot of controversy inside. Half the company thought this was stupid. A few of us thought we could make it work and other people a bit more gung-ho. So we got the people that thought they could make it work into a room and had a one-day project. where we all built a thing in the cloud to see if it would work, based built out of the kind of technologies we'd need to use to build this. That team then sort of knocked down a bunch of the straw men arguments

that everyone else was holding up against us. A lot of the time it is just straw men arguments, but you have to actually go and build something to actually feel that, find out what are the real arguments. Then you discover things you didn't even know which are hard. You run into the real blockers as opposed to the imaginary ones. So I think the trick is to get a small team go very deep, discover what you can and run a whole bunch of these little projects where you're trying

to learn as much as possible with the smallest possible input. You had this cultural aha, which is let's get the people who are gung-ho about this and let's let them go deep, knock down the strong man arguments, sort of zoom up to the 30,000 foot view and sort of describe the organization at Netflix sort of before and after. What did it look like before and after from a skill set point of view from an organizational design point of view? This is actually one of the big things that makes a difference. Some organizations are set up already to do microservice-based architectures

and others have to go through a re-org. At Netflix, it emerged naturally out of the way we were structured at the time. We were already structured of small cells that owned things. A lot of responsibility. Each team had a very clear idea of what it was building and how it related to the other teams, but it was assembled as a monolith at the end of the day. So breaking it apart was a fairly natural thing.

for us to do. What you see with traditional enterprise siloed organizations is they're actually having to do a reorgan set up teams that own that are responsible for services. And that's it's somewhat unnatural for the way they're currently set up. But I'm seeing an increasing number of people go through that transition. And sometimes you can see it as a replacing project based work with product based work. So every team becomes basically a product team for their microservice. And you have the product management aspects. the operational aspects within that team. And did you find that the people who were used to working on the monolith could be

retrained, or did you have to have a new crew come in? The culture at Netflix is interesting. Most of us had been around before. A lot of us had worked on SOA. We'd, you know, the gray-haired people that have been, there's a few people that were at Xerox Park in the 1980s and you could go and have arguments with them about object-oriented programming. We had some younger people, but it was a lot of very experienced people taking all the

stuff they'd learned and synthesizing it together. It was very collaborative experience, and we came up with things that made sense based on this series of transitions we were going through. The other transition was from a single centralized database. We had this enormous Oracle machine with a really complicated schema to a distributed no-SQL database in the end based on lots of different Cassandra clusters. And that was the third transition, and that was probably the hardest transition, was getting all of the SQL code and transactional stuff out of the system. It's actually breaking apart the database is probably the hardest thing to do. And then splitting chunks of code off is also difficult if you're trying to pick apart a monolith. And it turns out if you don't break apart your database, back end, and you just create lots of services to talk to it,

you've actually created what's called a distributed monolith, which has all the same fragility of the monolith, and you can't update things independently because you're tied by the database. You can't just take the Oracle database and break it up into little pieces. You have to think about it differently. Now, the same thing is true for the rest of the architecture as you migrate to microservices. Yeah, so I think what excites me about microservices in general, it moves all of infrastructure up to an application layer. So if you think about what you normally do in infrastructure, you've got these basic abstractions like compute and network and storage, which are pretty low level and they're semantic free, right? You don't have structured data.

One of the huge advantages is going up to a microservice architecture is you can do infrastructure and service. and things like, for example, security, things for like, even debugging, basic operations and management. And you can do it in a way that has the deep context and semantics of the application. The point here is that not only are you going away from the monolith, which is really important, and I think it's great, but also, like, you've got more semantics than you've ever had before. I mean, this is actually meaningful stuff when you're dealing with not IP headers, for example, not blocks, but actual, like, structured data. And I think that we can actually reimagine a lot of these tools in ways that we've never thought of them before because we've never had the ability to have this type of semantics in these tool chains. We're seeing this burgeoning area of microservices where you almost have like a function per company coming up.

And now I believe that all of the old stuff that we had in the Internet, whether it's naming or service discovery or routing or whatever. We've got an opportunity to bring this up in kind of a much deeper, richer level, which is really cool. Right. So we were going to the marketplace or the bazaar away from the cathedral, Which is any individual function can be provided by either an internal or external provider. It could be a cloud service. But then the challenge is now it's up to every organization to coordinate.

And so what are some lessons that you guys have learned along the way of picking best of breed and then making sure they work with each other, getting the version control to work? When you've got a monolithic app, everything is in there. If it gets broken into, you have all access. It's connection to the database, lets it basically say anything to the database. When you break things into microservices, you've got the ability to have some parts of your system be low security risk and other parts be high security risk. You can innovate really, really quickly in areas of sort of personalization and user experience. And then you maybe have a much more tightly controlled thing for, say, the sign up flow and where you're storing personal information. So the great news is you have a lot more agility.

The price that you pay is you're doing a lot more coordination. with a monolith, it's easy. You put all your eggs in one basket, and then from a security point of view, for instance, you basically just pile a bunch of appliances in front of it. Easy, right? Because it was a monolith. You knew exactly where it was. Now that the permitor is distributed across many machines, you have to be a lot more mindful of where the attack surface has gone and which security service you need to put in front of that part of the microservices architecture. So you cannot have the privilege escalation of because there is a little bit of PCI, compliance needed in one tiny corner of this monolith, the entire monolith is now subject to

PCI compliance and SOX compliance and all these things. And by splitting it up into pieces, you can have most of your app be extremely agile and very innovative and then have the bits that need to be safe, be extremely safe. And then if you look at the attack surface, you're basically keeping a very tight control over what can do what. And if you connect into the databases, you've got very single-purpose connections into the database that are doing one thing, and you can start to control at the access level there as well. What used to be policy controlled by the operations people, what they felt was a safe sandbox for the developers, is now really being driven from the other end down. So this idea of developer-driven infrastructure is something that is

turning things around. And a lot of what I'm seeing is at big banks and people like that, they have their existing policy frameworks and rules, and they're trying to apply it in the new world. And it looks the same. So they're happy because they're compliant, but they don't actually have the real policy separation that they think they have, because it's all totally reprogramable, and it's like you have the illusion that you're still conforming to the policy. A lot of these things were ops controlled. So the ops would control the data center and then the networks in the data center. And now it's all developer defined and software constructs, which are controlled by your cloud APIs. But if you're updating it 10 times a day,

there isn't time to have 10 meetings a day with operations to do the handoff. So what we've been seeing is that people just running it themselves. The only person that knows the exact state of the system is the developer that just updated it. That sounds scary until you realize that each of them is controlling a very small piece of the system, and the aggregate behavior of the system turns out to be really robust and reliable, partly because if you put a developer on call, they write really reliable code and they don't release code on Friday afternoons because they want to quiet weekend. and, you know, they learn a bunch of practices about having what it's like to be on call and how not to break things.

So we went from an in-person change review board infrequently, right, to vet the changes, to continuous change. And, hey, let's coordinate over Slack. Pretty much. Yeah, you have to tell people what you're doing, but you don't have to typically ask for permission and go and have meetings and things like that. So this is part of unlocking the innovation. And the people are most interested in this are large teams of people trying to build complex. products, typically enterprises, and they are worried about getting disrupted by the latest Bay Area startup or whatever. There's an existential threat here that if you're doing quarterly releases and your competitors

doing daily releases and continuous delivery, you're going to fall so far behind in the user experience that you're just going to suffer, right? So that's the big driver that is making people say, well, how do you get there? There's a whole bunch of things tied together. You're bringing in cloud. DevOps is a whole other area. and microservices as an architecture, all these things tie together

and some cultural change as well in the organization of the company. The companies that are doing well at that are really starting to accelerate off into the distance. It's also worth teasing a part two trends. And one of these trends is, you know, a single company, instead of building a monolithic product, wants to build a microservices product,

and it gets all the efficiencies of doing that as far as the development process and the OAM process and everything else. But there's kind of a broader industry trend where companies' products are basically microservices, right? There's companies out there that basically the only way to access to product is through a fairly narrow API. I mean, you know, there's so many of these now

that there are other startups that will just basically stitch them together and you could build full applications without writing much code. So I think that in addition to a single company getting a lot of advantages, I think the entire industry is going to get a lot of advantages and see a lot of innovation as a result. Yeah, a few had said five years ago that there would be multiple independent public companies that all they do was offer an API,

you would have been laughed out of the room, right? And now look at us, Twilio and Stripe, and on and on. I like to do the mental exercise of kind of where this is all going. And I still love Chris Dixon's quote of, you know, every Unix command becomes a company, is like, crap becomes Google or whatever. Like, I think, you know, we may be having an analog here,

which is every function becomes a company, right? It's like even more granular than a command line tool. Every single function or logical function becomes an independent company. And I do think there are implications on things like ownership and dependability and stuff like that that we haven't grappled yet as an industry, but it's a very exciting direction. Yeah, you're able to build something now that pulls in things from APIs and pulls in some containers and you just have your little piece of code in the middle that stitches it together and build a completely new service from that. So it's just much easier to get things built. It's more efficient for the big companies, but it has democratized all the way down to pretty much.

anybody with a laptop can go build, build something interesting. And if you go back five or ten years, you're doing things that would be just totally impossible to try and get together at that point. There's much more room for innovation. It also makes it harder to compete in some ways because now it's hard to build, you know, a billion dollar software company on top of these things because they keep changing underneath you and they're cheap to build. So you've got lots of disruption coming.

And it's actually, you know, GitHub and open source is another big. player in here that's just making it much lower cost to get things done. So what you're seeing now is Twitter and Facebook and Netflix and Google and LinkedIn producing the stuff you actually want to use, which has already been tested at volume. And then it's actually much harder to build a proprietary software company because you're competing with these big end users and you've got this thing you've just built and it's flaky and doesn't quite work right. We've talked about this, but it seems like close source shipable software is on its way out or dead. And there's a number of reasons for this. One of them is just the enterprise buyer likes open source software. But another

one is it's a real burden on the company to ship software, right? I mean, especially if that software is a distributed system, right? I mean, like you don't have skilled operators often. Every environment is different, right? So you've got these heterogeneous deployment environments. You end up with this like the mother of all cash consistency problems where you've got a bunch of versions out there of the product, you've got to maintain a bunch of versions and etc. The QA matrix from hell, right? The Oracle's a version multiplied by the flavors of Unix, multiplied by whatever Windows versions you're supporting, right? Yeah, that's right.

Your poor QA manager. Yeah, that's right. And then distributed systems generally, like, I mean, like a real trick if you're running your own operation is you have skilled administrators that know how to manage a cluster. And there are very, very few companies. And I think maybe one that's actually managed to ship a distributed system that was manageable with a non-skilled operator. It's a very, very difficult problem.

And so a great thing about if you offer something as a service is like, okay, you don't have any of these problems. And so, like, basically, your post-sales operation budget is way lower. It's much easier to start a company now. But at the same time, there are questions about, like, okay, so, like, what are the size of these companies are going to end up being? Like, I mean, like, how big is the market for a single function? I think it's still to be seen, like, how big these companies are going to become.

Yeah. Big challenge from an investor's point of view, which is if the essential argument is, there will be no more cathedrals. It's all bazaars from here on out. It's a little harder to make money, right? Because they're investing in the food truck. and that's as big as it's going to get. Yeah, yeah, right.

So put yourselves in the shoes of the enterprise CIO. The pace of change is accelerating, right? The ink just dried on her team getting VMware certified. And now we're on to containers. And then people are talking about serverless and functions as a service with sort of Lambda architecture. So talk a little bit about what's coming. And then the ability of an average organization

to sort of absorb these changes. Containers came along really over the last two years and one of the fastest takeovers of enterprise computing we've ever seen. It's quite remarkable how quickly they were able to colonize the enterprise space. It solved a real problem. What role did containers play in moving away from the monoliths to the microservices architecture? What happens with the containers, all that stuff is packaged into a bundle, which has all the right versions of everything inside it, and you can download it and run it. It also abstracts you away from the particular version of what you're running on. There's now containers

for Windows as well, but originally this was a Linux-based concept. You have the same container a format. If you want to run in-house or on a public cloud, it doesn't really matter. That container can run on VMware or KVM on OpenStack or on Amazon or Google or Azure or wherever, right? You've just abstracted yourself up one level. It gives you that kind of portability. If you think about machines used to sit at the same IP address for years. People would know a machine, they'd actually know the IP address off by heart if they wanted to do something to it. And then you had VMs came along and other VMs are more transient. and they you know this thing would come and go maybe in you know order of weeks or something

but by weekly update of your VM and then with containers it's perfectly reasonable to have container that runs for less than a minute you can create an entire test environment to set it up run it run your tests you know automatically test it strip the thing down again and and the size of the things have got much smaller if you just take it to its logical conclusion we basically fire up effectively a container to run a single request and have it to sit around about half a second and then have it go away again. And that's really what the underlying technology behind AWS Lambda. It's a server on demand that just isn't there most of the time.

And this is the bleeding edge right now. We have to figure out how to extract these sort of ghostly flickering images that are sort of coming into existence for short periods of time. How do you track what's going on? You end up figuring out how to do end-to-end tracing as the only way you can monitor things rather than being a special case like it is now. So there's a bunch of interesting problems here, but what's really been happening is just this trend to more and more ephemerality and these extremely ephemeral systems.

And then the charging used to charge by three years worth of machine. And then it became, well, you can rent a VM by the hour. And then containers, you know, that's lighter weight. And now you're paying by the 100 milliseconds, right? It's perfectly reasonable to run for half a second, which means that the setup time to create that half second worth of machine needs to be radical. less than half a second. And the time taken to bill it, bill for it needs to be less than half a second. If you remember the story of SMS, the SMS record for 140 characters. The billing record is much bigger than that. It's more like a kilobite. So if you actually take a telco and rip out

all of the billing stuff for their SMS things, you know, it's 10th, you know, it would cost a 10th of the amount to run if they didn't bill for it. Right. So you get this effect that the overhead of doing the thing is actually vastly more than the thing you're trying to do. So there's actually, it's a really interesting challenge is how to create monitoring and billing and scheduling systems that work so quickly that you can afford to build things in tiny increments. Our portfolio company 21 is right in the thick of this, right, which is how do you stand up an ad hoc agreement between an API and an API and like have the billing all work and, you know, Bitcoin might play a part in that. Also to your question, going back to the CIA, I mean, it seems to me in general with

disruptive technologies. It's like the disruption happens first and then all the day two ops happens second. I mean, whatever that is. And I think in this case, you know, the disruption is around delaminating the app and breaking it apart. I do think that CIO should not despair and ops team should not despair because what happens very quickly in the vacuum being left from kind of the, you know, this sprint on these new technologies is whole, you know, ecosystems and whole industries arise around them to provide visibility, to provide security, to provide ops. And we're seeing that now. And so, I mean, I think that it's quite possible to decouple the disruption, which is this velocity around development, and then, you know, the basic operations. And that tooling

was definitely going to happen as well. And understanding that ecosystem and understanding the players is very important if you want to stay on top of this kind of big change. Leaning forward into the change, assuming the tooling will meet you halfway. Exactly. And then you get the benefit, the big benefit from the CIO's point of view, in my opinion, is that you don't have this loop where the business user asked for something. It took you 15 months to build it, only to discover that's not what the business user really wanted because the requirements were poorly specified. In these days, right, no problem. I've got a change for you.

We'll put it live this afternoon, right? So the rapid experimentation that happens in startup plan can now migrate into the big organizations, and you don't have to get your requirements perfectly specified at the beginning of a waterfall process anymore. Let's run the experiments. It's actually even better than that. What the CIOs are providing now is a set of APIs for the development. team that is part of the business to automatically provision whatever they want, with certain policy constraints around it for what they can and can't do.

But fundamentally, you're providing APIs. Operations has moved from being a ticket-driven organization to be an API. They are now no longer a cost center. That is a very profound move. And I'm seeing a lot of the CIOs buying into that. They want to be part of the product. They want to be, how do you support the business? And you provide APIs so that they can just get business done.

at a rate that you're not slowing them down. We're actually seeing the creation of a new buying center in the industry. I've heard it call platform engineering. I've heard it call the DevOps team or whatever. This is like budget allocated. It's actually viewed as a profit center. It's product aligned, but it's core infrastructure and operations. And these are very technical buyers, so it's not the traditional enterprise go-to-market.

This is also moving across industries. We've seen, obviously, media and entertainment, and to some extent retail were early movers, mostly because of the threat of Amazon themselves, causing retailers to step up to re-engineering. We're now seeing FinTech, Wall Street, is really paying attention to some people are way down the road. Some people are just starting. Manufacturing that whole industry is just starting to think about this. There's definitely a sort of industry by industry sort of domino effect as people are figuring this out. So we're a decade on or so into this revolution, right? Many strands. What excites you now? For me, what's really exciting about this,

and I've said this before, is we just have the ability to reimagine all of infrastructure. You can now reimagine tooling and reimagine security and reimagine operations and management. We get to reimagine it with more semantics and context than we've ever had. You know, so what does it mean to have a firewall in a world where everything's microservices? What does it mean to have operation management and debugging? Things that were traditional boxes that were stuck on perimeters now also become functions. And actually managing your infrastructure is almost like looking at a debugger, a context debugger. It's like you have a symbol table with you.

It's like this whole thing is in one large IDE, and you can do that for your operations. I think it's going to push the state of the yard on how we even think about ops in an entirely new areas. I'm really excited about that change. I think that the whole serverless area is the bleeding edge right now. The monitoring tools industry is right now being disrupted pretty heavily by serverless. There's only one or two tools that have really come into existence in the last year or two that have effectively a way of processing stuff that is this ephemeral and dynamic. So there's some interesting products coming out. There's just a better way of living.

If you're a developer and you're working in the waterfall siloed organization, it's kind of soul-destroying for a lot of people. Indeed. And when you get ownership of a product, distributed teams, you give each distributed team their own product ownership, and they get to define the interface and manage it and run it, yeah, you might be on call, but you're much more control of your destiny, and it's much more rewarding and it's more productive.

and the ability to get more stuff done as a developer is just rewarding anyway. It's a better way of working for people. Well, that's great. Well, thank you, Adrian. Thank you, Martin. We'll see a lot more unfold as the architecture shifts.