The Infra Pod - Transactions don't belong to databases anymore, let's discuss the Modern Tx Stack (w/ Yoko from A16z + Tony from Inngest)

Episode Date: July 31, 2023

Transactions is what makes critical applications with state works in production, and it has been decades of research and iterations of having developers leveraging the tools that exists to implement transactions in their applications. Ian and I are sitting down with Yoko (investor from a16z) and Tony (CTO from Inngest) to chat about the new Modern Transactional Stack (https://a16z.com/2023/04/14/the-modern-transactional-stack/), and how it's looking to shape up the industry in the near future.

All right, let's do this. Welcome back to yet another Infra Deep Dive. That's the intros, right? Tim, I'm from Essence VC. That's Ian, let's kick it off. I'm Ian Livingston, been helping Sneak figure out how to become a platform. And today we're going to talk about modern transactional stack. I'm super excited to be joined both by Tony from Ingest and Yoko at Andreessen Horowitz, who both contributed to this most recent modern transactional stack blog post that Andreessen published. But in order to figure out who these folks are, Tony, why don't you tell us a little about yourself and about what you've been working on?

Hey, I'm Tony. I run Ingest and we make it easy to write reliable code. So you can drop in RSEK and write workflows as code and deploy them anywhere because they're backed by serverless queues. So zero infrastructure. We manage function state. We push function state into your functions and everything kind of just works. Yoko, let's go. Hi, everyone.

My name is Yoko Lee. I invest in infrastructure, developer tools, AI, and all the other things at ASICC. But I'm a cartoonist and I'm a developer by trade. So I worked at a few companies most recently, like a product at HashiCorp. So super excited to talk to you all about transactions. Awesome. I would love to start, like, let's take a step back from the blog post and maybe Yoko, you can help us out. Like, what is it about the Modern Transaction Act? Why did

you at End Reason start writing this? Like, what was the like insight or the realization that started you thinking about that? Hey, there's a problem here. We don't understand it, but there's something going on. We feel like we can do a whole bunch of research. What was the starting point? Like, where'd this come from? And what got you so interested in it? Yeah, so I mean, the background of this is that for many years, actually, since the microservices came out, remember all that rage, you know, 10 years ago, the team at A16Z, we have been just looking into the space. And every time a major platform shift happens, we observe behaviors and the new tools adopted by developers. And this time around, it is interesting that when microservices

came out a couple years ago, people started to find that it's actually very hard to build transactional systems in distributed system and across different microservices. So back in the days, if you just have a front end and one back end, and all transactions can be handled by databases, there's no questions there. But now if a request has to traverse between different microservices, say, I want to check inventory, and then I want to make a payment, you have to manage a state in between these microservices. That gets very difficult.

And people usually throw queues and caches at the in-between state storage between these microservices. So we were wondering, how do companies at scale manage this? And then a couple of years ago, of course, there are more novel solutions coming out in the market. So that's what we call now workflows. So a workflow is just a block of code that execute based on events or timers.

But the upside is that instead of using queues and caches speaking microservices, you can just use workflows like a programming language level to write an important code that execute across these microservices and workflows will take care of it.

So that has been an interesting platform shift transition for us. And then when we dive deeper, we realize that there's a lot more nuance to workflows. So I'm going to stop here before I dive into nitty gritty technical details. So curious about which class you want us to go down. Tony, maybe you could you know we have yoga's introduction why she's interested you've started a company that's kind of solving a problem in the space maybe you could tell us what what made you start ingest and then we could talk a little bit more about okay well what's the world actually like yeah yeah yeah so prior to ingest

I was running engineering at a healthcare company um vertically integrated in this line called uniform teeth healthcare is crazy and I kind of wanted to get into it to fix it which is naive I was running engineering at a healthcare company, vertically integrated in VizLine called Uniform Teeth. Healthcare is crazy, and I kind of wanted to get into it to fix it, which is naive. And we had a few problems. We had to build a medical record suitable for orthodontics. We had to build a treatment planner, and we had to build a patient app. And ultimately, the theory was, if we can design world-class treatment planning software, which accurately treatment plans better than any other company,

we can lower the total cost of treatment with software. That meant that we had to build all of these really complex flows. Automating patient intake seems basic, but there's a lot of nuance to it. So when a patient comes in, you need to email them two hours before. If they actually don't sign off on treatment and don't allow you to hold that treatment data, you've got to do stuff with it. So there's all these edge cases around what you need to build as an engineering team, which is really important for compliance, but also just really important for the product. And this is kind of a more extreme version of what you see in normal products. Like for example,

sneak, you've got a bunch of workflows around security. So building this just generally sucks. It's never easy, whether you go down the microservices route or even if you've got this monolith. If we talk about like Yoko's microservices part, it's kind of composability. You still have all of these branches in your workflow that you need to put together. You need to make sure that they all work and they're reliable. And you need to handle failure states.

Not only do you need to do that, but your processes is change you need to handle versioning between the two so building all this stuff as a developer takes a lot of time hugely error prone debugging sucks and often there's only a few people in your company that can do it it's not like you can take a front-end engineer or a bootcamp grad and then chuck them into this problem because if you want to make it reliable ultimately that's distributed systems as much as people don't like it and so it's really difficult further kind of annoyingly we were event driven which i think is the right way to do things events are great because then we can react to what happens if a provider doesn't submit the form and we don't see the event what happens if the intake is slow or a patient turns up late? Kind of events in the real world. But if you're modeling events and trying to add that into your workflows, it just sucks.

And so there was never really a nice way to do this. Nights and weekends, I'd stay up until 2 a.m. prototyping and just to make it easier for the engineering team and which is kind of how it got started so generally just feeling this pain over the course of five years as we were building out this healthcare company so yeah that that's the story behind it the theory is everything there you know workflows distributed systems event driven systems and generally engineers need to build this whether they know it or not, being really difficult. Got it. I think we'll have to really jump deeper into this modern transactional stack. And what I think will be super interesting is we kind of describe what a good modern transactional stack even looks like. Because I think in the end of the day, you ask any developer, what's the best ideal stack or tool or abstraction they need?

You come up with like 50, 100 different political, religious answers that they're willing to fight each other with. So depending where the layer sits, we kind of give it up on some level. Like, okay, that's not my problem anymore. I'm willing to give it up to somebody else to solve it. But once you jump out of the database, like, okay, applications start having workflows. You know, we should all use events in some way.

Like that becomes a very big debate. And we see this happen so often since the problem we're trying to solve, from what I can understand, is the growth of the services, right? The growth of data inconsistencies and everybody trying to solve the same problems. Well, how do we ever get to the same agreement?

This is the kind of thing abstraction should be. And this is the kind of thing developers should start doing. And we'll start bringing my knife, you bring your knife, and we had to figure out how to get to some middle ground, right? So Tony, let's start from you. How did you decide what is a good stack for apps developers to think of? It's super interesting because there's so many facets to this. Taking a step back, if we're building a new platform to allow you to build transactional

systems, it has to meet the standards of existing technology. For example, Qs, right? SQS came out in, what, 2004? It hasn't really changed in 20 years. Sure, there's been additions to it, but that's kind of crazy. We're still working with technology from 10, 20 years ago. Whenever we're building these new platforms, they have to meet standards that exist. So what's that like? Concurrency toggles, prioritization, back off. You need to meet these standards. But realistically, there are a few things that are really

important nowadays. Developer speed is huge. We have to build really complex products. I think the table stakes has changed in the last 20 years. We're not building crud apps. We have to build really complex products. I think the table stakes has changed in the last 20 years. We're not building CRUD apps. We have to build complex things that integrate with a bunch of different services. And it kind of has to be reliable. So you want to build these fast because you have to build a lot and there's a lot of pressure. And I think a lot of clouds for sale on Netlify have shown that developer experience is really important. So you want to also make something that's really easy for developers to get started. So those are two fundamentals that are guiding paths.

They're not engineering trade-offs, but they're principles that you need to live by. And then when it comes to actually choosing the right thing, I guess power, usability, and flexibility are really important. You don't want to be beholden to something that makes it really difficult for you to actually implement what you need to implement or make it really difficult to change because then you're sort of swapping out one set of problems for another. That becomes really, really, really hard. So realistically, the overall premise of just write code, don't think about the infrastructure is appealing for a lot of developers because they can focus on the problems that they need to focus on rather than the kind of auxiliary baggage.

And distributed systems has been auxiliary baggage that you need to think about. up. So I guess my point of view is if we can abstract a lot of that away, make it simple, but still powerful, then that's the ideal end state. And looking at that as well, like there's a whole lot of decisions that you need to make. Kind of serverless is the lowest common denominator in the tech world. You know, there's no state, there's no reliability, there's no retries, it's push-based model. And if you take a look at something like existing queues, it's pool-based. So changing that on the head is really important so that we do push-based versus pool, then you can

deploy anywhere. Meet serverless edge servers as well. It just kind of works. So I think high-level developer speed is of use. Low-level there's a ton that goes into these specifics to kind of meet everyone where they are. That's super interesting. There's all these different models, different ways to think about

it. I'm really interested, Yoko, to hear from you as you research for this blog post, which knowing what I know about Andreessen and how you have a very research-focused way that you go about thinking about what you invest next and thinking about trends. What are some of the different options you've seen in the world? You know, we've talked a little bit about workflows. Tony's talked a little bit about events. What are you seeing as the different trends that are emerging to try and basically at the end of the day, try to solve this problem?

Yeah, I guess like to set the stage at a very high level. So Tim, previously you kind of asked like, what's a good stack for developers? And how do people even think about trade-offs? To get started, I think when we kind of made the modern transactional stack, like the Diagweb, we were thinking about what does modern mean and who are the developers? And then you kind of see there are two personas. There's one kind of persona, kind of like the microservices developer I kind of mentioned. And And the pattern they follow are different from this other type who are mostly going to start developing apps in the past couple years, mostly on JavaScript. And the tools needed by the microservices kind of app developer and infrastructure developers

are different from the ones who are developing on JavaScript. So when we think about modern transactional stack, we're thinking about, okay, now, like this year, if you are a developer in a great startup company or a mid-market company, you're thinking about starting a new project. What is a stack you will reach for? So if you have legacy, kind of like older architecture, they'll always be there.

Martin Casado likes to talk about, like, infrastructure gets layered away. You will never replace them. So what we're seeing is, like, older architecture, they'll always be there. Martin Casado likes to talk about infrastructure gets layered away, you'll never replace them. So what we're seeing is what are the emerging trends that kind of necessitate handling of these transactions and what are the best patterns? So from this point of view, when we're thinking about the modern transactional stack, when we talk to a lot of developers from like either larger mid-market companies smaller enterprise kind of like tech focused companies what we consistently hear from them is that when we started a feature we use javascript when you use javascript transaction

is still a problem javascript does not solve any of that probably made it worse too unfortunately and then the question becomes okay like do I use my database for it? Which is like, usually, in bigger companies, as you all know, it's not always that application engineers will handle this invite. And there's always like a distributed systems team who handles like writing things to your queue at scale. And the question becomes, what's the right interface for the application developers to be able to solve transactions. And it just happens that from what we've seen, there are two approaches.

One is you can use database, which are starting to implement workflows too, like things like Supabase, they have workflows, convex workflows. And then there's the other approach, which is more workflow native, kind of like what Tony is building here in Jazz. And then we also see temporal supports like TypeScript.

There's two other ones, Trigger, Different, and we've seen a lot of companies in that area. So from this point of view, we kind of grouped components together. There are databases, what we call ALTP application, logic transactional platforms, workflows, and then how they interact with business logic. Amazing. You talked a little bit about JavaScript developers don't have the problem solved, you know, you have legacy.

Are there specific technologies that we would consider being layered away today that you think you're seeing in the stack that, you know, five years ago or three years ago, or even today for some engineers, you're like, oh, the solution to this problem is that? Have you developed opinions on what is going to be layered away and what the new emerging layers will be as a result? We're going straight into hot takes. So it is interesting.

At first, my intuition was, oh, is the database getting layered away? And now, as we research more, the answer is actually no. Because this market is also evolving very fast. So we're seeing a lot of the database players kind of innovating, adding new components that can make this new architecture work. So the part that I would say is we're seeing new patterns that like people still use microservices, but it's not as hot, I would say, as maybe five, ten years ago when I was writing code. They were like, oh, move from monolith to microservices. Now we're kind of moving past the early innings of transit, moving back.

And I think everything is kind of starting to become a function instead of kind of like an API that's abstracted away. So that really speaks to what Tony mentioned, like the developer ergonomics, the taste, the feel, like how you want to use the tool. Those are changing. So I don't know if it's as hard of a take as what you had wanted, but that's what we're seeing. I have a hot take on this one. Go for it, sir.

I personally hardcore belief that I don't think queues are developer friendly. And I think they're going to get laid away. So everything from creating subscribers, capacity hardcoded into your workers is absolutely bananas. Like what happens if you change the logic or change the payload and you've got things enqueued from last month? It's super hard to get any visibility into things that have existed for 15 years. And then retries is like, again, bananas, like dead data queues are crazy and they should die just like the name. It's kind of insane. If you took a look at what exists in the world now, like ClickHouse is pretty cool. You know, BigQuery has been out for a long time. Why can't we just store everything in an OLAP database and then allow you to replay based off

of that, which is kind of what we're working on at Ingest. Just replay failed functions without having dead data queues and have all of the state visible so that you understand what happened. Replay it locally. Queues are kind of crazy and generally shouldn't necessarily be exposed to people, I don't think. I think that it's too much to consider. It's like working with the pipes when you really just want to care about what you're actually building. That's number one.

I'll stop. One hot take at a time is maybe enough. Well, I think we sort of already agreed as a group. We should just do hot takes all the way to the end at this point now. So how about we just do a spicy future? Because I think this spicy future will be quite lengthy. There's endless things we can probably jump on.

Spicy futures. I think you already started, Tony, right? Like about what you think should happen. And we can maybe go even further. Like, okay, what do you see people actually do in five years or so, right? Like Q's going away is more like a removal. But next three to five years, what should all developers be writing on? And how should they think about the trade-offs of writing new servers and apps anymore?

Yeah, yeah, yeah. So regardless of Ingest's approach, I still think that it's really good that developers need to focus on code. You know, that can live anywhere, but you need to just write code. It's important to meet people where developers are at. I don't think anyone's going to want to learn a language that works with distributed features. Otherwise, Erlang would be amazing and everyone would write Erlang. So we're not asking people to change languages. Write JavaScript, write Go, write Ruby. It has to be where developers are at. And you should just focus on code with the abstraction of function

state queues retries baked into that code ideally you would have a layer a shim middleware sort of like a distributed vm that runs your code that manages this for you and organizes function state kind of what we do but also like anyone can do it just takes time to build that way developers can just write code and they can do things like, when a user signs up, send this email reliably. Email services go down. Stuff fails. Whether or not you put this in a transaction is okay. But still, to some degree, you're going to need retries.

At some point, something's going to fail. You're going to get a wrong payload. You still want to be able to replay that and adjust the payload. Because types are great, but they don't catch everything. So, yeah, code retries, super important, baked into the system. One thing that actually we see a lot, I kind of brought this up a little bit earlier, events. I think events are the future. They're kind of immutable facts about what's happening.

And you might see this externally when you're building a workflow. You need to wait for some external state to happen, some external change. Typically, people bundle in Kafka and RabbitMQ, or they bundle in Kafka and Celery, mix these two together, have all the state flying all over the shop, which is really hard to maintain. And I think if you can bundle in events, listening, and sort of like pauses directly into your functions, then things get really, really, really easy to build. Like basic examples that every company deals with, even important for like a growth churn workplace, like a user signs up, they don't do something, which is really hard to do, then send them an email, like wait three hours for them to, you know, do something on your platform. User adds a product to a cart and they don't check out within an hour,

then send them an email reminder. That stuff is a basic workflow, but driven by events. And I think not only do you want the steps in that function to be reliable, you also want to be able to listen to outside state by events in a really, really, really easy to use manner. And I think fundamentally paving the abstractions over queues and events is a really good really, really easy to use manner. And I think like fundamentally paving the abstractions over queues and events is a really good idea because then developers can build anything realistically in a really easy manner, you know, in like a few hours.

Right now, that's something that you have to pay specific systems for, like braze.com do all of these churn workflows for like five, six grand a month. And that's one platform for one particular type of workflow that developers can't build because it takes a couple months. But if you abstract this away, then it becomes really easy to do that. I'm very curious,

if I could get your take, maybe this will end up being spicy hot or maybe it won't be, but I'm very curious to get your take on when I step back and think about trends, I'm like, well, what's that moment that makes this trend real? What is that defining moment that like, oh, this is happening, right? And when I think about trends, I'm like, well, what's that moment that makes this trend real? What is that defining moment that like, oh, this is happening, right? And when I think about

AFCA, EventQs, RabbitMQ, and I think about what type of organizations are running these types of infrastructure, I'm thinking Bloomberg, right? Hundreds of thousands of events, millions of events per second, highly transactional data systems. Is there something that's occurring, that's driving, that's pulling, if you will, people this way? Do you think there's a broadening out of some trend? Normally we'd see this only in the biggest of enterprises, maybe because they could afford it, or maybe because they actually have the use case. I'm curious, from your perspective, is this a thing that's going to continue to only live in the biggest of the enterprises or the most complex use cases, or is this really a thing that only is going to continue to only live in the biggest of the enterprises, the most complex use cases, or just really a thing that's being broadened out?

Here's to get your take. That's super interesting. Like high level, that's sort of like, you know, when you look at pricing, you're like, okay, cool. Is this working? Is our pricing working? Or is the data because our pricing is bad? Like, is Kafka working?

Is it only used in big companies because only big companies need it? Or is it only used because only big companies can afford to operate it because it takes so much time and effort and infrastructure knowledge to manage? Maybe it's the latter, is my take. I think that events are an interesting and correct way to do things. But I think that traditionally, it's been way too hard to do that. And if you look at any sort of theory, you know, like Nats is great. NSQ is cool. RabbitMQ's, you know, PubSub stuff is awesome. Google PubSub is amazing too, but still

you have to learn the sort of paradigm of development and that takes a lot. And so I think like it's really, really difficult, just like the edge. Edge computing was traditionally for, you know, maybe some VoIP providers. Maybe it was for phone networks. Maybe it was for CDNs. And then you get Vercel and Fly.io and Netlify. And it's like, oh, I just want to push to the edge by pushing out a button. Like Fly deploy and say which region you want is pretty amazing.

Not any developer can do it. And the question isn't necessarily about like, oh, is edge correct? It's like edge improves latency, makes your performance better, makes the experience better. So can we make it easy? I think events are a really good

way to work. Basically, modeling what happens in the real world, which is kind of what we do as engineers. And if we can make it easy for people, then yeah, it should be something that people consider, especially if it only takes an hour to use. You don't have to set up

stateful subscribers and work for workers and different subscriptions to each particular topic and take all of that away. Just write a function that reacts to events. Like that's, that's pretty cool. That's pretty powerful. So I think, um, number one, been too hard and too costly, too time consuming. I think number two, things are getting a lot more difficult to build as, as, as sort of products as, as classes of things.

Like if you look at real time, even like integrations, webhooks or events, real time message passing, which are events, you know. Think like if you were taking a look at like up on the bandwagon, LLMs, chat, you know, each message is in theory an event that you react to. And so I think depending on how you view things and model things, basically everything is an event. And you can choose to model it as an event or you can choose to model it in a specific niche way for that particular use case that you're working on. But events are the most general form of working.

And so broadly applicable, just too difficult. Yeah, to chime in on kind of what tommy said it's yeah since your question is also around kind of like what is actually pushing all these technology forward and why did the architectural pattern kind of start at this moment now i would say we're still in the very early innings but if i have to guess here are the hot takes. So the first hot take is that I think the cloud should be abstracted away. You know, having working in infrastructure for a long time, you know, provision a bunch of things, like, you know, manage lots of things with all these tools. But it was just very painful. Like, there are tools that make it less painful.

But to be honest, as an application engineer, it feels like a technical detail. Someone else should probably abstract. But then recently, we are seeing a lot more trend on people moving to serverless. And by serverless, I don't mean like AWS Lambda or things like that. That happened 10 years ago. By serverless, I mean the same technology, for example, Kafka queues or caches, the kind of thing that scales down to zero, you can consume as a service and then scale up, hopefully, infinitely. It's not infinite, of course, but scale up to the extent that you need it.

So very dynamic, great ergonomics. I'm seeing more and more serverless tech coming out and developers will just use that. Because of that, events is a natural kind of like abstraction to working with systems like this. Another thing we are seeing is apps are global ready by nature now, not only apps, but different forms of SaaS, like for example, payments now need to be global ready. If you're like an AI developer, you know, writing an app, you want to sell globally, your payments has to serve everywhere. So of these two things i actually think there will be probably less and less movement on

the lower level infrastructure layer like the traditional what like sre's devops writing shell script you know provisioning stuff i think there's going to be less of that but more on the driving consumption layer like more and more providers will consolidate this. They will still need DevOps to manage this very well. But the rest of the world probably needs less of that since they won't have this problem. But more and more people will just be writing application on top of this. And my other hot take is you either have to use workflows with a persistent storage, or you have to use a database with workflow kind of functionalities.

We don't know which way it will go. It's something that we're definitely deeply interested in. But at the end of the day, the workflows will be needing to tie back to the business data because that's your source of truth when you're querying. Who are the users? How much inventory do you have? Who paid? Who didn't pay? All of that is stored in, you know, like the traditional database storage.

So it's either that the workflows need to tightly integrate with all the databases and then make sure that layer can be conducted like atomic transactions so you can write back with no problem, which is what kind of a lot of today's workflow solutions do. Or as databases, you need to kind of stretch yourself to the other side of providing, hey, here's developer, this is how you use microservices or serverless

with the workflow engine, but with our database. I think there's so many possible topics to even ask questions in this sort of a general theme. There are actually lots of explorations and products or research that are going on in this space as we speak, right? And I think it's kind of the exciting part, you know, seeing what can actually work in the long run is quite hard. It's almost like predicting which LLM will win in the future. Like, I don't know. know. So I want to maybe figure out what do you think the maturity needs to be? Or what are some of the approaches here

that will win what kind of developers? This is my observation, at least. I'm seeing two kinds of transactional or user programming stack. One is a more opinionated, wrapped around stack where you can just write some code. It has built-in assumptions of how transactions will work. One is that my thought was modal, right? You know, like simplicity, I can just run my functions. It will just run somewhere and add

little primitives here and there to start to support your stack upwards. If you look at Yoko, your blog posts, the companies or the approaches you mentioned are more building blocks, right? They're building blocks like the listed, you know, Supabase, Upstash, Convex, a bunch of these. They are building one more layer

above your database, some level. And it's never clear what is going to be the next state of the world, you know, moving forward. Are we going to be the next state of the world you know moving forward are we going to have an event definition that we all use across the whole board and this is like a standard a w3c kind of craziness whatever and then everybody will build on top of that i'm not sure that could ever happen as a developer there's just already so much complexity and now you say here's a way to define events then you have to go and figure out everything else.

Nobody wants to do that, right? But I don't know what is that everything else, right? The cues are just part of it, but there's just so much work involved. Tony, because your world is like, I'm going to wrap around a system for people to be able to program and Yoko is seeing everything. So I'm curious about your takes. Like, what do you think is enough of a product that people will actually jump over they rather use you and don't have to think about other trade-offs because i feel like

queues or events is just like one part of it where it has to be some some steady state like actually kind of able to handle certain scale you can handle this kind of thing or the programmability or something like that there's a maturity that needs to get to i don't know what that looks like yet to be honest yeah there's a lot that goes into these systems you know like they're both broad and deep which is really annoying because like you would like to choose one or the other but unfortunately you have to do both realistically where we're at now if you take a look at db versus application layer implicitly there's a choice. Are we going to migrate everything to a new DB or are we going to be able to stick with what we've got? So choice number one is, where are you?

Greenfield projects can probably afford to try something new. But again, you have to try something new, which means learning a new paradigm and learning how to put all of this in your database. What is the debugging story? What's the scale story? Like you mentioned,

if you've got a database that does literally everything for you, are we talking about an IBM 90s mainframe? How does that work in the future? And so definitely individual decisions that would need to be made to go into that. And I sort of view the, similar to how you build things, like the lowest common denominator of what we do is push-based so that we can work with serverless edge. Serverless is unreliable and there's no queues. It's good for APIs, but you can't write business logic because it's going to fail. So minimum standards are push-based.

And again, the minimum standards to kind of meet everyone where they're at is like application level, write standard code, TypeScript, Go, whatever you need, and sort of provide people the building blocks to allow them to write their own business logic just works. I don't think people want to understand the details of how things really work under the hood to a degree. Largely when it comes to implementing things, I think people just want to implement being able to give them a small but powerful API to say, here's how you run a particular part of a transaction in a function.

Here's how you wait for something else to happen. Here's how you pause and schedule future work. Here's how you cancel jobs automatically based off of some event coming in or an API to do so, so that you can declaratively write things and deploy it anywhere, like Verc like Vercel or AWS or GCP is kind of an ideal for us end game, largely because it has the most surface area and it's the most applicable to every developer.

So there's definitely merit in exploring new technology, you know, like what does it look like if we could build all of this inside a database? It's a super appealing idea and fun to think about but then again like are you going to be able to take somebody that's got like you know terabytes of data or hundreds of gigs in postgres and written all over there if this microservice is on monolith and transition that that's a little bit of a tough ask so that's my take on things yeah i think that's an extremely good question.

I guess to get started, coming to your point on events-driven, so my view of the world is that a lot of the workflow systems that we see today, the inner workings are already events-driven. It's just a matter of what semantics they want to expose to developers. So there is a distinction between how it's implemented versus how they want the users to use it. So how it's implemented is almost always just events.

Everyone uses a Kafka queue or some other queues to implement these things, but developers don't need to know. So the right interface, to be honest, the market is still really deciding what's the best way to work this out.

But kind of to transform your question a little to, like when I was thinking about it in my head, I thought there are three concepts that any system will need to take care of in order to ship an end-to-end solution. One is like application state, like what handles the state? What moves the finite state machine forward?

That's one thing. And then two is like business data, kind of like what we said, like the OLTP where business data source of truth lies, where do you store that? And then the third part is the business logic. Where do you write that? Is it in a microservice?

Is it in like serverless? Like what have you? Or is it as part of the workflow? So to say like you want like an end-to-end solution you basically need a solution that covers all three of these things and i think for a subset of developers the three things could coexist in the same system and in fact today we're seeing it right like super based convex they're all doing it very well and then depending on what you're developing you could just run on it end to end

but then there's this other different persona the question is do we think these are different layers are going to be the same layer and then my answer is i think there's going to be different layers at the end of the day because the distributed system is just so hard to crack it's hard to have distributed system as a service you know like you can have a component of the distributors system as a service that i can believe so i think the steady state of the world is like each one of this layer they will stay separate layers but for a subset of audiences you could say you know like we could use one solution for all of them yeah that's interesting and also like you mentioned something really interesting like application state and business data. Basically, nothing here lives in silo.

It has to interact with whatever you need to do. If you've got a system that can only interact with 10 specific APIs, then it's useful in maybe 10 specific circumstances. And so it needs to be able to see your data, it needs to be able to interact with whatever you need to do as an engineer, but do it automatically and manage the kind of function state for you so you don't manage reshreds. I'm like, yeah, I totally agree.

Like, super important to actually be able to do regular engineering tasks, you know? I have a great question. I can't wait to hear some people's takes. We'll start with you, Yoko. You know, one of the things Tim and I talk about often when we think about doing this podcast, we'd love to talk about infrastructure trends.

This is obviously a trend. What apps are being enabled as a result of this? At the end of the day, I think what we're talking about is we're taking a thing that exists already in large organizations. We're talking about a democratization of that to make this more available. I'm really curious to understand, as you think about investing or what trends you think about are really important or where new TAM is or whatever it is, what type of new apps do you think are going to be enabled or new verticals that can be go after, new opportunities to go after? There's all the fact that we're taking what is relatively complex, difficult to do. I have built these things by hand personally.

I can tell you it's a miserable affair, would not recommend. What do you think the average impact on the developer will be? And more importantly, I think the question is, what types of businesses could we build that we couldn't have built before? Yeah, first, we don't really look at hands. I think as a developer, what we look at is if there's an entry point and then we try it out, hands off, to see if it's like an ergonomics that we like. But jokes aside. So an entry point, and then we try it out, hands-on, to see if it's like an ergonomics that we like. But jokes aside, so the entry point, I mean, off the bat, I think the latent demand it kind of unlocks

is that for serverless global-ready Greenfield apps, before, they will need a backend team to kind of implement this. So say if I'm implementing a global-first order processing solution, before, I couldn't of implement this. So say if I'm implementing a global first order processing solution, before I couldn't have done this. I have to be like a strike or like someone else who has a DevOps team. So today the latent demand it unlocks is that people can't function without as much of a DevOps team. And that makes them being able to build anything like global ready and that happens to handle a transaction. So the specific use cases, I don't think it's that different from what we see transactional apps are like booking tickets, placing an order, like you see this

everywhere in e-commerce. So in fact, when we look at like, what kind of apps generate the most event driven architecture, it just happens to coincide with transactions. It's e-commerce. It's kind of notifications. It's kind of like what you want to use Zapier for, but in a more powerful way. If I receive an email, send it to users. All the common everyday use cases, you couldn't have handled at scale in a consistent manner now you can

that just happens i think will happen first with the more greenfield stack just because adoption is easy you can move very fast but i have no doubt it will trickle down to kind of the legacy stack as because like people are building apps over and over again like new apps even in bigger companies that's what we have seen with microservices. How did everyone move from, you know, LAMP stack, haven't heard that term for a long time, to microservices like 10 years ago? And I think it's going to happen with how people are working,

kind of moving on to database with workflow, where workflows with persistent storage in future. That's great. One of the things you said, and then we'll pass it to you, Tony, but one of the things you said that triggered a thought in my mind was, most innovations of infrastructure is about, well, we've had all these people over here doing this really hard problem, a piece of infrastructure land, rebuild on top of the infrastructure. We no longer need those people solving that problem because it reduces, you know, makes something more efficient, makes it easier to manage, we have a service we can buy from someone, now they have to go focus on something else. The interesting question, okay, well, if you don't have to focus on dealing with distributed state management anymore, because you've adopted this pattern, you've adopted this new tool, you need to go build new apps, then what are those new apps? But Tony, I'd love to, the original

question is still the main question is, what apps do you think, like, what are we going to see? I think Yoko brought up a couple of really great points specifically around global ready is a great example. Yeah. Honestly, very much agree with all of the points that patient demand, all the stuff you need to build is just easier, way faster to build complex apps. So I think like,

more than we can see an increase in complexity in the stuff that we do because it's just easier to manage complexity. You don't have to worry about any info. You write the functions and the business logic and great, nice. So more complex things in general. But okay, this is crazy. And I saw this on Twitter, which is the world of hot takes,

taking a completely different view on things. Someone was like, I love Tailwind UI because the classes are declarative. And that's really cool because I can tell ChatGBT, yo, do X, Y, and Z and give me this thing with padding. There's never really been a nice declarative API for distributed systems, right? Like you still have to do subscribers, workers, backoffs, all the general stuff that people don't want to deal with.

And we talked about this before of like, what's enough of a product? Enough of a product is a really decent API that abstracts these things in just code. So if you have this decent API that just reads like code and it's really basic, just one line to do some really complex things, it's pretty cool because then anyone can do this. I'm not going as far as all the LLM craziness in the world, but like specifically, like you can just tell somebody,

hey, I want to write this particular crazy workflow that takes a lot of time. Please go ahead and write some pattern for me and it will just work fundamentally is the goal. That's the dream state. So I think everything becomes easier and realistically, yeah, you can kind of like bash out whatever you want.

So speed of development is the biggest one. Like you're saying, move people from infra to product. But then if it's a declarative API, everything becomes a lot easier to work with, which is pretty cool. So I don't know, the future is crazy right now in particular, you know, so much is changing. Yeah, quick thoughts after hearing all this. Because I think since I worked on like large scale games like Halo or worked at Mesos, I worked at lots of different kind of platforms. We basically have a dedicated team working on a platform team that has to build transactions from ground up. Always create an event system from ground up ourselves, right?

Some kind of actor system that can distribute all kinds of crazy stuff. Just be able to build what we want faster and cheaper and easier. And I think this is super cool. I don't know if we can actually able to chat GPT out of a Kubernetes the next day or chat GPT out of millions concurrent player systems from tomorrow.

But I think what's probably most likely is we can chat GPT out of an e-commerce store that's probably doing some kind of crud, plus some transactions. Your typical team that has to still shopify can't do good enough, but I need to have my own logic to do certain actions within certain transaction scale, certain safety. I really do believe that with the move of adding application transactions moving towards and it's more friendlier to define the events and friendlier to be able to describe the state

in a much more simplistic way, like AI can actually start generating much more accurate results here. And LLNs can start take over more generation of this. It's a beautiful state. It's a beautiful state. It's a scary state for some developers that just love to work in this middle and just kind of like,

I know how to run database and I know how to deal with transactions and commits and rollbacks. And that's your whole job. I'll be worried. Or pick up some other books. I don't know. But I think definitely there is this middle ground

where you're not the largest scale. You're not the lowest latency. You are a platform team that needs to build a decent scale because I need to handle multiple states with multiple outcomes when it comes to side effects. That should go away. We should be able to describe it much more quicker. And it should just happen. And we have some trust that this should happen within some time because right now all this stuff is hard how do you build a system like that right we can't have slas we can't ship anything right as a platform team you're

so figuring out how to do all the operations the trusts and the guarantees but can we get to that point where somebody built enough not just the the definitions, and also can start to guarantee the outcomes. I think actually there's some crazy thing that might happen in the near future. Yeah, I think it's important, especially when we talk about infrastructure that's like creating a new layer over a layer of complexity that we have today

is to remind ourselves that, we look in any at-scale organization or really any organization that's using a Pippo event or distributed transaction mechanism. What's the day-to-day of that experience like today? And for most of them, it's, oh, I improperly calculated a partition of a Kafka topic, and I lost all of the data in the state, and now we don't know where it went, and now there's been a customer outage. In theory, these things are fun to play with, but in practice, these things are terrible to manage and run. And anything that helps us spend less time solving with the problems we create ourselves and more time focusing on creating net good for the world, for our customers, or whoever it may be, that's better for everybody.

I always think about that when I think about infrastructure. It's like, well, why is this valuable? Well, it's because we're actually making the next generation of applications easier to build. And we're having people focus on more of net value add instead of just like managing what is unfortunately like the popsicle sticks that is the current state of most people's backend. This has been an absolutely awesome conversation. I think, I think both of you on that note, Tim has been messaging me time to go.

But this has been a great conversation. Yoko, you know, we love to give our guests like a at the end. So if there's any shout-outs, where can people find you? Yeah, yeah, yeah. Shout-outs to my partner, Statution Martin. We kind of wrote the piece together and did all the research together. And then you can find me

on Twitter at StaffYokoDraws. I draw cartoons about infrastructure, make fun of it, make fun of technology. So if you want to collaborate, let me know. The cartoons are excellent. I think, in fact, you drew one for the application stack. Oh, yes. So as our friend Ian put it, so I drew a

cartoon about... It's on the topic of I can't decide whether to bring the database to application logic or bring the application logic to the database. So come check it out. And Tony, over to you. Yeah, yeah. Grateful for the entire team that's helped build

this. And, you know, like the previous job there was this CMO who is amazing at specifying how everything should work and very, very, very helpful people that have been there in life. On Twitter, ingest.com. Two Ns, like, you know, a house for your data and functions.

And then, yeah, that's that's that's largely I I crap post less on Twitter than I should so maybe I should get back on Twitter awesome Timmy want to close this out all right well let's end thanks everybody thank you thank you