Invest Like the Best with Patrick O'Shaughnessy - Martin Casado - Entering Uncharted AI Territory - [Invest Like the Best, EP.381]

Starting point is 00:00:02 Hello and welcome, everyone. I'm Patrick O'Shaughnessy, and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. Invest Like the Best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at join colossus.com. Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of positive sum. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of positive sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc.

Starting point is 00:00:54 My guest today is Martine Casado. Martine is a partner at Andreessen Horowitz and first join me on invest like the best in 2022. So much has changed since then, and it was awesome to have Martine back to discuss all the different implications of this modern AI revolution. Before joining A16Z, Martin pioneered software-defined networking and co-founded Nikeira, which was bought by VMware for $1.3 billion in 2012. He has studied, built, and invested in digital infrastructure his whole career, which has primed him to go in-depth in this interview on the immerse opportunities and challenges AI presents among creativity, policymaking, agenic systems, real-world data structures, and beyond. Please enjoy this great conversation with Martine Casado.

Starting point is 00:01:40 Martine, I'm so happy to do this with you for a second time, and there's so much to talk about that's going on in the world of technology, most specifically AI. What do you think the 2030s will feel like? The 2030s. So I think the future is going to be a lot weird, wonderful, and more bizarre than anybody is imagining. I'm just talking about this AI stuff, I feel like one of the mistakes we're making as an industry is AI has been around for a long time. We've assumed it will do certain things like it'll do chatbots

Starting point is 00:02:13 and it'll automate the enterprise and try to cram these new advances into that old world. But it's not clear to me that's actually the best use of it. A lot of the problems that solved are problems we haven't solved in the computers like creativity. and the emotional connection. And actually, like the businesses, these are the things that are working.

Starting point is 00:02:32 And so I think we're in this free internet world where we can't really imagine what it's going to look like, but we're solving different problems. The glimpses that we see is just, I think we're going to have this creative social explosion as opposed to a, oh, we've solved an enterprise automation problem. I don't know if that's what you're asking, but I actually think we should all hold on

Starting point is 00:02:52 and be just ready for a trippy, amazing, wonderful ride. What do you think, zooming all the way back to today, is the most important next bottleneck on the path to that trippy awesome ride? What things, if they didn't go a certain way, would change your opinion and say, actually, maybe we're actually on a more traditional marginal incremental growth plan? I'll be honest. I think that for some subset of things, I think the cat's out of the bag, continues out of the bottle, whatever stupid effortism you want to use. When it comes to content creation, like any short of digital creation, the marginal cost has gone to zero. That's like asking after the creation of the internet what's going to stop its proliferation. And I just think the laws of physics are now that the economics are all going to shift.

Starting point is 00:03:33 We should be generating content and all this creative content with AI. And I think it's just going to naturally happen. But just like in the early days of the internet, I don't think we know what that's going to look like. And so we know that there's this economic dislocation. We know there's a new way of doing things. I think it's a question of timescales, not whether that's going to happen. I think the biggest existential threat to address your question directly is, There's always this type of exuberance where people see the economic benefit,

Starting point is 00:03:57 but the actual building the right apps and the demand takes a lot longer. And so you definitely have these big chill periods. Like, I remember very well the fiber glut. And the internet, we're like, yeah, everybody's going to be on the internet. We built all of this fiber. And then it turns out when the supply cycles are different than demand cycles. So the time it takes to actually build a data center, for example, in the AI, get GPUs to people versus like demand.

Starting point is 00:04:18 When they're out of whack, you often have these overbuilds, and then you have these gluts. and then prices dropped and you have it. So I think we'll go through some of those and that will slow stuff down. But I think that the long-term trajectory is pretty locked in from this point. And this stuff is going to change things in ways that we don't quite understand. I'd love you to lay it out as I've seen you do elsewhere where it's like the third marginal cost of X, asymptotting down to zero as the history of modern technology and why that's so important. Yeah, for sure.

Starting point is 00:04:43 And credit we're credit to do. A lot of this came out of a conversation I was having with Ben Silverman, the CEO and founder of Pinterest. and we've seen at least two cases in the past where Marshall Costs have gone to zero and it just changed the world. And so the first one was with the microchip. So the reason computers are called computers is actually that main computer came from a job of human used to have, which was to compute. And it's very tough to understand where ballistics are going to go. And to do this, you have to create these crazy logarithm tables. Three 1940s, the way that you would figure out ballistics trajectories, you'd have people.

Starting point is 00:05:19 people in rooms calculating these big logarithet tables. And then ENIAC shows up, and ENIAC is 5,000 times faster than a human. So almost four orders of magnitude faster than a human being. And so the marginal cost of computing goes to zero. And as a result of that, you have the entire compute revolution. Because if it's so cheap to compute, let's compute for everything, right? And this is IBM, deck, HP, all of that came out of that. And then starting in the 70s, but really in the 80s and 90s, the internet showed up,

Starting point is 00:05:51 and now the marginal cost of distribution goes to zero. I remember when I used to buy, like pre-internet, I remember pre-internet, when I would go whenever buy a video game, like I would grow a store and I'd buy a box, and that box probably had been shipped there for two weeks. So it just takes a very long time to get the software. So when marginal cost of distribution goes to zero, you had the internet wave. And again, it was one of these things where once it happened, And the entire world shifted.

Starting point is 00:06:16 And rather than you having less, you have so much more because it's so cheap and it's so much more accessible. And so one way to think about this current AI wave is the marginal cost of content creation is going to zero. And to some extent, natural language reasoning is going to zero. And a great example of this is, let's say that I wanted to create a likeness of myself that looks like a Pixar character. So like the Martine Pixar character. So the traditional way to do this is I go to a designer and let's say the designer is 100 bucks an hour and it takes them two hours, which having actually done this before, they're more expensive and they take longer. But let's just say this. So say it's 200 bucks.

Starting point is 00:06:53 If you compare that to the inference cost from one of these models, like the inference cost to create a Martinez of a Pixar character is like a hundredth of a penny. So again, we're four orders of magnitude, almost the exact same difference between ENIAC and a human computer we have between a model and creating a likeness of me. Just one last point on this, we can dig into any aspect that you want. If you want to have a mental conception how big disinflection point could potentially be, imagine a AAA video game. So a AAA video game can cost a billion or more create. Let's say it's $500 million. There actually isn't one aspect of that video game that you couldn't create today with a model,

Starting point is 00:07:32 whether it's the music or the sound effects or the textures or the textures or the video cutscenes or the images or whatever it is. all of these things. And so rather than being 500 million, the actual COGS cost should be something like 10 bucks. So this is just this massive. What we've learned in the past is when these cost drop, it's not like it wipes out cause. It's not a zero-something thing. It's just we end up generating so much more stuff. And we enable so many more applications because it becomes so much more accessible. So I think we're going to see this massive kind of creative explosion as a result. Do you think about it beyond content? Content is one thing. There already is lot of content because the marginal cost of distribution went to zero. So a lot of human brains created a lot of content, which still is much more expensive than these models. I certainly buy how much easier. That just seems like a foregone conclusion. What about beyond content? What about the robots loading the dishwasher type of stuff? The things that start eating outside of just the digital world or in different parts of the digital world to other kinds of work. So there's the content thing. Let's work our way to the robots because I think that's maybe the last bash. So there's the content thing.

Starting point is 00:08:37 then there's the emotion thing, which do feel like humans have some imperative for connection. And it just turns out these models are very good for that. I don't know the true numbers for this, but I believe that you draw a dollar at random being made with these current models today. At least a third of that are people that are just creating an emotional connection, whether it's through a virtual friend, whether it's through these new style of video games where you're actually really interacting, whether it's a virtual girlfriend or virtual boyfriend, like whatever it is. There are many companies that we see at Andreessen Horowitz and that are out there that are building we call it companionship, like these kind of companions. And so that's different than content

Starting point is 00:09:12 creation. It's actually emotional connection. It's not a modality that computers have been able to solve in the past than they do now. So I think that's very much another area. And then the third one is just basic language reasoning. So a way to think about this is I work with a number of lawyers, let's say I pay them 700 bucks an hour. And they do very smart things, but a lot of these are language reasoning things. Let's say I've got a new legal document PDF. So if I have a choice of giving that PDF for the lawyer to read. The lawyer spends three days reading it. The lawyer gives me answers some questions based on it. Some of those questions are right. Some of those questions are wrong. That's $2,100 bucks. Or I can do the same thing with an LLM. I think a lot of those language

Starting point is 00:09:50 reasoning tasks are going to go to these models because they do a pretty good job of the language reasoning piece. And so I think those three things we very clearly see in advancement. And I think that it's at the point that we know these are going to be big businesses. But you ask a very important question, which is when do the robots show up? Let me just give you one anecdote. When I joined Stanford for my PhD was in 2003. And around that time, Sebastian Thrun had just won the DARPA Grand Challenge. They'd driven a tonerously advanced through the desert for 1,200 miles. And we're like, oh my God, amazing, hooray, robotaxis are almost here. This was 20 years ago. And in that time, There's a very important thing for everybody to realize. We as an industry have invested $75 billion

Starting point is 00:10:37 in autonomous vehicles. And if you actually look at the unit economics of the cars that are working on the street today, they're still more expensive than Uber by quite a bit. There are certain modalities that AI is trying to tackle where they're better than humans, they're quite a bit better than humans. They solve very real problems, but the economics are very hard. And one way to think about that is, is competition is not another AI or another computer system. The competition is a human brain. And if the human brain is really good at something, it's like navigating a 3D space. It's not just a human brain. It's probably like the four million year old brain has been doing this, and it's very efficient at that. If you take one of these autonomous vehicle cars and you look at how

Starting point is 00:11:19 much wattage just the CPUs and GPUs burn, some of these are like 1.3 kilowatt rigs for the GPU setup, for all the planning and all the inference, et cetera. Like they're very like power hungry. the human brain consumes about 21. You're competing with the most efficient part of the human brain. Now, compare that to the three things I talked about before, the creativity, the language using, et cetera. That primarily comes from the prefrontal cortex, which is about 250,000 years old, that human brains. It's not very optimized. It turns out the computers are much, much better at the newer part of the brain, solving that. So if you take an economic view of this, I think you can say that solving the real world problem is hard for two reasons. One, I have not talked about it. One of them is just the human being is so good at navigating the real world, and you're competing with that, and we don't know how to do that economically, while solving the content-created language reasoning is easier because the human brain isn't optimized for that. Okay, that's one. There's a second point, and it's a subtle one, but I just think it's so important. Let's try to dig into it as much as we can. It's important.

Starting point is 00:12:16 There's a very reasonable view in all of this, that what these models are doing is all they're doing is tapping into human-created data, and they're able to reproduce more human-created data. That's what they're exploiting is data created by human. And that means that the creation of the data is the hard part. It's like a human brain looked at this world and decided that's a tree or that's a book and reduced it to something that's more linear and more structured. And then these models just learn from that structure, as opposed to actually learning about the world. So let's just talk about language.

Starting point is 00:12:50 As a human species, we've spent 3,000 years writing down language and creating a all of this content, it could be the case that these LLMs have just taken all of those tokens, those 3,000 years of tokens, and just learned all of the structure in those tokens, and then it can reproduce more tokens like a human being would, but there's nothing fundamental about the world in those. It's just the tokens themselves. It's a self-encapsulated thing. When you're talking about robots necessarily need to interact with the physical world. And the physical world, as we all know, it's chaotic, it's fractal, it's self-similar, it's nonlinear, it's just very complex. And it's not clear at all that.

Starting point is 00:13:26 we know how to rein in that type of complexity with these current models. We know that we can take human-created content and that human-created content, we can produce more human-created stuff. But we can't take real-world stuff and produce more real-world stuff yet. That is still an outstanding problem. These are such important foundational points. So what's the chicken and what's the egg here? You can imagine that if you had 100 million robots walking around, I guess there's cars gathering lots of data on streets, but if you had 100 million humanoid robots doing stuff, you would be creating this insane data set over time. There could be some sort of data training feedback loop or something.

Starting point is 00:14:01 But what do you see as the way to solve the chicken and egg problem that you just described? I just don't know if we even know if it's a data problem, meaning human-created stuff just has enough structure to exploit. There was a great reason where I don't know if it got accepted. I just saw it on Twitter, but I just thought it was such a brilliant piece of work by Rohan, by the way, at the AGI House, where he showed that if you have a data corpus and you G-zip it, use compression. If it compresses a lot, then the model learns a lot on the same data, which suggests that the models are just exploiting structure in the human-created

Starting point is 00:14:36 data. So it could be the humans that are actually adding the structure of the data that they're learning from. That's a very different statement than saying you get data from the natural universe and you train a model on that. And so it could be that the current AI wave is honestly just exploiting human-created content because we did the hard work of creating structure, and that's economically viable. And that's why we can make pictures that are economically viable and videos that are economically viable and Texas economically viable. But we have yet to see is how do we look at the universe and trade on that and then use that to build an economically viable machine, which again, we have yet to do even with Katana's vehicles,

Starting point is 00:15:16 even though we spent a lot more money on that. And so there's two questions. You asked a fundamental question, which is, can you just go with a bunch of cameras and look at the world and get enough data to train a model that's economically viable? We don't know the answer to that question, because you're competing with a human brain that's very good at that. It's almost become a cliche that anytime somebody says AV is going to be done, this is very famously something that Tesla's had a hard time with, for example. It just feels like the universe is very heavy-tailed. There's lots of exception cases, and it takes a long time. The basics is I would not look at a model's ability to read text created from humans and be predictive, I would not compare that to a model's ability to

Starting point is 00:15:56 look at the universe without human creation and come up with that. We don't know if that could be economically viable in the near term. We don't even know if it's possible. That's just the state. And so really the first three that you mentioned, companionship, creativity, productivity, is like this sort of known space. And then beyond that is this sort of, it's beyond the horizon. We just don't know when and if it's coming. I think that's exactly right. I would say the ones that are working are really working. These businesses are really working. They're solvent. They're profitable. They're growing. They're solving real spaces. But for anybody listening to it, for me, this is such an aha. Almost all of them, basically all of them are trained as human-created data. So humans create data,

Starting point is 00:16:35 whether it's like conversations for companionship, whether it's a high-created art. And if you train a model on that, it's economically viable because A, this is what humans are interested in. And there's already a bunch of structures so it can do that. It's very hard to come up with examples where models been trained with the universe, whatever, in streets or roads or any, and they actually end up being useful. And you could say that this is because the data is just too chaotic, there's too much entropy, it's too heavy-tailed, or you could say brain is just too good on the natural universe and we can't compete. And I don't know which one of those is true. My suspicion is probably both of those things are true. So it's just not clear at all to me. That problem is going to get

Starting point is 00:17:13 cracked in the near term. Maybe it will. I'm not saying it won't, but it's not obvious to me. But for sure, anything that humans can create, we know how to train model on. I've always thought about the bitter lesson idea that if we just throw more compute and more data at the problem, it'll just get better. As it being this like raw data, I've never thought about it as this like human created data structure is the thing. It's really a fascinating idea. So does that mean you think that maybe the bitter lesson stops applying? The solution might not just be, I think you've already said this, but might not just be more collection of data, more nuke-powered, you know, data centers running GPU clusters or whatever, there may be some other thing needed

Starting point is 00:17:51 to get to that next phase. Let's tease us apart for what we know. We know that if human beings spend a lot of time creating data sets and we train on those datasets, then we can get predictive models that are economically viable. We know that. But once those data sets are exhausted, what are our options? And in this case, we've literally exhausted the data created in the last 3,000 years, right? It's not like you can just spin some new ones up.

Starting point is 00:18:18 Yeah, yeah, right? So we just know that to be case. And so you can make an argument that what we're drafting on is all of the effort and the energy that human beings as a species have done to take the universe and make it well formed. And then we're exploiting that structure. In that case, the structure is like a reserve of energy, a reservoir that was created by humans that were exploiting. And then once that's gone, we've made this massive step function, but we either need to find new ways

Starting point is 00:18:43 to create data, which we're actually seeing now. A lot of these LLMs, the budgets are moving to post-training and creating data sets. So we're already seeing that happen. You have thousands of people out there creating code snippets or annotating sentences or whatever it is. So we're already seeing that happening. But you can't compare like a thousand people writing code with all the code ever written before.

Starting point is 00:19:04 So that will definitely slow down on that. When it comes to compute, so then everybody says, what about synthetic data? Certainly in certain domains, synthetic data makes sense. If you have an axiomatic system where you can fill out the state space computationally, like playing chess or arithmetic, or even some spatial stuff where you can simulate things, 100% you can simulate data. But that's not a general statement. What does it mean to create synthetic language?

Starting point is 00:19:33 What does that even mean? Language is not axiomatic. It's like this kind of ambiguous thing. What does it mean to synthetically do creativity? At the end of the day, we don't really know how to synthesize. theoretically recreate the stuff that human beings have created in the past. And so it's not even clear what you'd throw the compute at. Let's say you're out of data and you exploited all of the state.

Starting point is 00:19:53 You can create something out of data, which helps in certain domains, but it doesn't help in generality as far as I can tell. It certainly doesn't help with generalized language because I don't even know what that means. I think you actually have to either create more data or you have to come up with a way that we don't know how to do yet to take the universe and make it better structured. It could be we hit another plateau, which listen, I think the actual bitter lesson in computer science is that everything is a sigmoid and you have these exponentials and people get super excited, then it flattens off. And then you have another unlock and everything but he's excited and then

Starting point is 00:20:22 it flattens off. And who knows? I'm not saying we are, but it very well could be the case that things flatten off now. And that's great. We've got more problems to tackle. Let's go tackle them. I'd love to zoom in on this. I'm visualizing like an economic crossover point of viability of a business model for some product that passes that and one that doesn't. And I'm especially interested in the stuff on the line that you've seen, that you could maybe generalize about what you've learned about that economic viability line for AI products? I probably took my first AI course in 1999. This stuff has been around for very long time. And if you actually look, the progress for AI has always been incredibly steady. It's always been up into the right. We've solved many problems we didn't think that

Starting point is 00:21:02 you could solve. Everything from expert systems in the 50s and 60s to handwriting detection. We've solved all of these problems. We do it better than human beings and we have for decades have been doing many things better than humans. There's always been this open question in the investment circuit, which is, why have we not seen a platform shift or a transformation because of AI? Where are all the AI native companies? Why has all the value gone to incumbent? If somebody makes money off of AI, it's typically like a meta or a Google, it's not some startup. AI investing for startups has been a bloodbath. It's been very difficult. It's been this open question. Why is that the case? And the answer is because the economics have been crap.

Starting point is 00:21:39 you've just been tough. And he says, so why have the economics have been tough? So a lot of traditional AI, let's take perception, it's actually solving this very hard problem. It's trying to build a model that will see the universe and predict what happens, but the universe is very heavy tailed. So you have to bring humans in.

Starting point is 00:21:57 You've got diminishing marginal returns. You basically converge on having to generate new data or have human beings in the loop, and then you end up like ping where people, it says the economics have been very tough for startups and solve. So if there's one thing that's changed, changed with these new models is the economics are amazing, right? So I see some questions asking why are the economics amazing now?

Starting point is 00:22:16 And the reason is what I alluded to before is the models that are working are really not trying to solve these big, tough problems of trying to reason about the universe. They really are trying to just reproduce what human beings have always created. The closer you get to a model that's just trained on data created by humans, the more economically viable the company is because A, that data is what humans want. They spent the time to actually create it to begin with. So clearly that it's interesting. But also, the humans have done the hard work of creating all of the structure in it.

Starting point is 00:22:46 And I think the closer that you get to robotics or perception or trying to make sense of the real world, the harder the economics become. Even A, B, is robotics light, right? That's kind of 2D-ish. You're not like, what can a car do? If you can move into a 2D plane and go forward, it can go backwards, right? It's not like it's like dealing with like arms and fingers and this and that. So I think the closer you go to full 3D navigation in a real physical world and having to protect how the world is going to react, I think the heart of the problems get and the heart of the

Starting point is 00:23:13 economics get. And then the more you're competing with a more evolved portion of the human brain. So I think that really is the continuum. How do you think about with your investing hat on, I guess, the horizon to commercialization of a given idea? Are you willing to invest in a robotics company today that's building the foundation model for dexterity? I'm making something up. Do you have to be able to see a customer or buying a thing for an economic transaction in order to get excited about it as an investor? Or are you willing to invest in things where the commercialization is like over the visible horizon? So it depends on the domain.

Starting point is 00:23:47 I'm a firm believer that these models train on human data, creating stuff. The model is the app. They're so magic that if you do it, people will find value and they'll use it. And we've seen this. We've seen many of these. We've seen music. We've seen speech. We've seen video.

Starting point is 00:24:00 We've seen images. And all of these are these great growing businesses. and it's just so easy to articulate why these are great businesses because the marginal cost of creating something goes to zero and you're creating something people want, people will buy it. So for those, I think you want to invest in the best team to build a best model. And then I think you can do this well before. So for example, ideogram.

Starting point is 00:24:19 We invested in an ideogram. This was like the Google Imagine team. One of the pioneers of diffusion models, they went out and they created an image one. Like this is very obvious. We were the first money and it was just kind of a very obvious thing to do. As the economics become murkier, then it's harder to make these decisions. The one thing that I've learned looking at robotics companies over the years is that often when you're investing in robotics companies, you're investing in a very verticalized company

Starting point is 00:24:46 and you almost need to understand the business, the vertical that it's being applied to as much as the core technology. In the LLM creativity space, literally some Ph.D. researcher can create a model that consumers love, But if you're building a robot for agriculture, you better be an agriculture company, actually. And if you're building one for mining, you better be a mining company, actually. I've sat on the boards of these companies that are these core technologies that are building drones or robots, especially like it's like shipping or ag or it's forestry or whatever. And the companies end up becoming the specialist of what they are.

Starting point is 00:25:17 And so that's the mindset you have to go into those investments. This is an ad company. And I just learned that I don't know about these verticals. I don't know about construction. I don't get that sales process. I don't get the unit economics. I don't know about mining. I don't know about reforced.

Starting point is 00:25:30 That's not me. That's other people. This is one of the reasons we even have in an American dynamism team in Andreessen Horowitz, which is clearly AI is going to change robotics. It's changing robots. And robotics is our AI, but it's changing how we do things. It's going to change all of it. It's going to change defense.

Starting point is 00:25:45 It's going to change safety. It's going to change construction. But to invest in that, you better understand those markets. And so I do think you need special purpose teams to do that. So for my team, for those ones, we don't do them. But for other teams, Andrews and Reese, Horace, they do. But they're very much special. lists on the markets as opposed just the deck.

Starting point is 00:26:00 How do you think about and analyze the defensibility of these business models where pick a modality, I don't know, music or something? It seems every time some mind-bending demo comes out, five other companies come out, and there's like rough parity between what they're able to do. And I'm just curious where and how you think these companies will build defensibility and allow them to be huge enterprises. Yeah, that's great question. I've got a lot to say about this, but let me be.

Starting point is 00:26:28 say a few maybe unrelated things and then a specific thing. So one unrelated thing is first movers tend to have a massive advantage here. And it's because they're creating a new user behavior and they have a brand and they have a bunch of users and they have the feedback and whatever. So that's a very real thing. And I do think it breaks Pareto where, you know, the first movers have 80% early on and then the 20% is left for the rest. However, over time, all of that arose, right? And that's very clear. Now I've seen many of these model companies launch at this point. We probably have the largest portfolio of one of them. So I've seen many of them launch. And they all Look the following.

Starting point is 00:26:58 The company does nothing, and then they launch a model, and it's not very good. It's like a little blip, and then it goes back to zero. And then they go for a while, and they launch a bottle is not very good, and blip, it goes back to zero. And then they launch a model that's good enough that it's over some unknown bar, and then they get this great rope. They're like, amazing. We've made it.

Starting point is 00:27:15 However, it's mostly because it's so magical to get a bunch of user, and then it kind of looks like a TCP saw tube. People use it, and then they get bored, and it drops off. They have to lunch another model, and it goes up, and it drops off. They have to lunch another model. And so then you're in this long march, this is called the waves where you're writing these waves of needing new models and it's a treadmill and it sucks. And at that point, that's when the companies have to find a traditional moat. Now, traditional moat could be a community moat.

Starting point is 00:27:40 It could be an integration mode like an API. It could be differentiated styling and brand mode. There's a lot of modes, but that's when you have to create the moat and get off the treadmill. I think if you don't get off the treadmill, this all converges basically to the same model. Let's talk about the moats that actually work. Integration modes are a real thing. Open AI has a great moat and that many people use the API in order to do their thing. And APIs just tend to be sticky, especially as they evolve over time.

Starting point is 00:28:06 Now, interestingly, LLMs have this anti-stickiness property and that they're so unpredictable. You can't rely on them anyways. They have to be ready to swap them out. But APIs are so mistaken. So integration modes are a very real thing. Community modes are a very real thing. A lot of the, especially creative companies, move to a consumer experience that involves sharing and virality and community. and that has inherent stickiness, of course.

Starting point is 00:28:28 Workflow is a very real mode, which is independent of the model, many of these things still require human input. And so you have my dashboard, you own the access layer, and they're training a user behavior. So that's a real moat. Again, so if we're going to break this up, model companies will launch stuff. A lot of it will be failures. They'll have one that hits,

Starting point is 00:28:45 and then they're in this sawtooth pattern of the treadmill of the next big one. And in that time, they have to figure out a defensibility. It will not just be on the model. I'd love you to hear your view on the evolution of the competition between the foundation model providers, both we'll call them frontier or closed source model providers like OpenAI and the open source ones like Mistrawl. And this will be an excuse to talk about some regulatory stuff and some bigger picture stuff in a few minutes. But first, just on the models themselves, what is your way of understanding these things? How do you watch them? What do you think matters? What interests you just

Starting point is 00:29:17 riff however you want? So here's something. Everybody asks the defensibility question. And what the mental model people have in their mind when they ask the defensibility question, oh, long term, there's nothing really defensible and they end up having the same data and blah, blah, blah, blah. But it's so much worse than that. And the reason it's worse than that, and I think people don't realize to the extent that it's worse than that. Many people don't, which is there's a massive perverse economy of scale because distillation works so well. So what do I mean by distillation? What I mean is let's say you, Patrick, have your AI company and you release the world's best

Starting point is 00:29:52 model and it's an LLM, it just turns out that I can use the fact that your model is out there to train my model very effectively to catch up to your model. Maybe I won't get in there all the way. I can do it a lot cheaper than you did it. Distillation is so effective. We're seeing like a new model will come out and then everybody will catch up to it. But this is a massive perverse economy of scale. And I don't think we as an industry understand quite what that means yet. To me, it means if you're releasing a model in text, you better have a good plan to get to defensibility. It probably should be traditional, something that business had been built on for quite a while in software. I don't think being the best model is just enough of an enduring. That said,

Starting point is 00:30:31 it's also clear to me when markets expand, they tend to fragment, not consolidate. And so white space is created every day there's more white space. And so I actually think we're seeing so much fragmenting and fracturing, even if defensibility is a problem, companies are finding independent niches. Let's just take the image category. So one way to think about the image category is There can only be one image company, and that's going to be the winner, or they're all going to look the same. But that's actually not what's happening. What's happening is they're all finding independent niches. It's almost like when you walk into a bookstore, a bookstore, have different sections.

Starting point is 00:31:03 Here's the anime section, and here's the sci-fi section, and here's the reference section. Here's the horror section. It's almost like these image models are, like deciding which sections they are. Pika, which is video, like they're very good at anime. And Midrary is like the gritty fantasy realism one. And the ideogram is like the cartooning text one. And Adobe is like the photo real one. And so I think that when Marx expands, they fragment, there's a lot of white space to find niches.

Starting point is 00:31:27 We're seeing that happen as multiple viable companies. There will be a consolidation phase, I think, for a long way from it. And so I think we're going to see a lot of great companies being built in the near term. And people are too scared of the consolidation phase where when you have market costs drop this fast, I think we have a lot of runway where we need to worry too much about that. What do you hope to see from the big, anthropic, open AI, mistral, Facebook, like the big leaders that have provided the core models? What's the best case scenario in your mind about how that story plays out in the next three

Starting point is 00:31:56 years? If you run on that thesis, that these are a core CS primitive that can solve a very huge class of problems, I suspect that they're going to end up solving different modalities of those problems. I really do. I think it's amazing that Facebook releases. Lama, but when you release open source without actually having an enterprise sales force and go to market engine around it, the thing kind of lands when people would do whatever they want,

Starting point is 00:32:22 but you don't ever push and prod it and pull it in the way that an enterprise buyer would want. Mistral, on the other hand, it's doing open source and it's releasing it, but it also engages with customers. So it can very directly push and prod and pull it in the way that it's actually meaningful. Does that mean these two things are competitive? Not really. They're serving two purposes. One is, here's this model, do whatever you want with it. Another one is, I've got an actual solution for the buyer. And then open AITOthropic, again, I just think they're pushing the state of the art. I hope they continue to push the state of the art. I think that they will likely find somewhat overlapping, but also independent niches in different areas. And I think these things

Starting point is 00:32:58 are expanding so quickly. I think they're going to continue to be viable players. And so, I mean, I applaud all of these companies for basically changing the industry and launching these things. And again, I don't think it's a zero-sum game at all. I think that's actually the worst thinking in this AI wave zero-something. And it literally is the most detrimental thing. Somehow, it's a zero-sum game. And because you have open source and a road's closed source and because you have two people competing in the same area, like one has to win, it's not like that. This is like a hundred percent abundance. Marketing is growing by a hundred times. And let's cheer them all on. Can you lay out your position on all the important regulatory considerations that are happening

Starting point is 00:33:35 very real time that I think you've been very vocal about that you have strong opinions about? Can you like lay out your stump speech for what you think is right versus what's going on. I would like to start to say I'm actually not a political person at all. I never get involved in these discussions. And I just feel like now we're just hitting this very dangerous period that I've seen before that I almost feel like it's my obligation to come on before. But it's not my 10%. I just want to be clear. It's something I've thought a lot about. I felt like, okay, I've got to go out and do the right thing here. So here's the thing. I feel that in maybe 2013, 2013, 2000, He wrote this book.

Starting point is 00:34:12 He's a super smart guy. He's a great philosopher, and he has great things to say about us living in a simulation and how innovation can be the downfall of humanity and how AI can be bad. There are these great intellectual arguments, and they've been going on for a very long time, and I think they impact a lot of people. If you actually read the Walter Isaacson book on Elon Musk, one of the more interesting things is how these kind of early Bostrom arguments impacted people like Demis and Elon and Sam and Ilya, et cetera.

Starting point is 00:34:42 But what many people don't realize that this was all before the systems were actually built. And this was a platonic armchair thought experiment. And somehow that got conflated with actual systems that were being built. And so you had this, let's prepare for the Fumi recursively self-improvement AI that can end humanity and protect everybody from it

Starting point is 00:35:05 before we actually created like the systems that are working. And somehow that thread is being used to, in my opinion, tamper and dampen innovation to what could be one of the most life-changing, industry-changing, value-creating advances we've ever had. And I think that the best analog is during the internet. And I was there and actually very active in these conversations. So a lot of what my positions draw from that experience. So let me just talk about that because I think it's a great analog. During the creation of the internet and the web, we had the exact same conversations using a lot of the same rhetoric, which is, oh my goodness, this stuff, we're running our critical infrastructure on this stuff. Now you can take over

Starting point is 00:35:53 people's lives. You can have sovereignty. I actually went to think tanks in D.C. in the early 2000, on sovereignty ending events for the United States based on the internet happening. We would talk about digital Peru Harbor or cyberboro Harbor. There's all of this concern. and all of the same things, oh, we need to protect the people from it, we need to pass all this regulation, all of the same type of stuff. And at the time, it was actually a far more legitimate concern than we have today for two reasons. The first one is it actually changed the security posture of the United States because it introduced a notion of asymmetry. So before that, we were in this kind of Cold War, mutually assured destruction doctrine, which is if we have big

Starting point is 00:36:33 weapons and they have big weapons, we have it to talk. And maybe we'll ease tensions. but at least you have a stalemate, where in the internet, the more advanced society was the more insecure because you were relying on the infrastructure. And so if you were like at a cyber war with a country that doesn't use computers or isn't network, you're more at risk. So it introduced actually a new doctrine shift, which is this. But the second one is we had actual specific examples of the emergent threats that we'd never seen before, like the Morris ones. We had these worms that would take out things, right? So both were you armed with these very specific things that have shown a new threat and an as well. And yet, after all of the discussion,

Starting point is 00:37:14 the conclusions were don't hamper innovation. Open source is good. This is the way to keep the United States ahead. This is the way to have more secure systems. And that's what played out. Many of the top security contributions to the internet came from academia or open source. S.C. Linux is a very famous example of this. Listen, ignore, which became source fire of a great example. Mandi is a great example. Like the entire industry came out of open source. And so I just feel like we've had this discussion. Listen, we're in this AI time where the rhetoric is back, but hey, we don't have any of the proof points that we had on the internet. There is no clear doctrine ship. And a lot of the arguments people are using for this call to regulation,

Starting point is 00:37:54 they're on these kind of like non-provable science fiction views of like recursive self-improven or creating bio-weapons, which at this point have been pretty well refuted. And so, I'm afraid that this kind of Boestrom view of the world has captured the imagination. We've built these systems and we're using it to hamper innovation in the name of quote-unquote security and we have not learned the lessons from the past. And in the end, it'll hurt the United States. It'll hurt academia. It'll hurt little tech in the industry.

Starting point is 00:38:23 So that's why I've been this call to arms, anybody who's not in a big company who's going to basically benefit from something to wake up and get involved and make sure that we don't regulate ourselves out of a leadership position. So if we think about this as the role that small tech will play on all of this, what do you think people need to do? What do you hope happen? What needs to happen? It feels like a very different time than it did 20 years ago because academia has almost been entirely muzzled. I don't know why. So in the past, academia was a very vocal proponent of open source, even for critical infrastructure.

Starting point is 00:38:55 BSD came from Berkeley. This ran the internet. There's nothing more critical. If you take that out, the internet doesn't work. hospitals don't work. There was nothing more core to the safety of humans than this. And academia was on the forefront of this. The EFF came out of MIT. There was this kind of organic, individual, progressive, almost grassroots support. And now with this AI stuff, if anything, academia is totally complicit. The California bill, this horrible bill in California, which is literally, we need to

Starting point is 00:39:27 protect everybody from these models that don't even exist yet. It literally came from a Berkeley professor and her student. He's coming from academia. So academia is totally complicit. So I think that those in academia that are silent, I think they need to speak up. And I actually think if, for example, they called Scott Wiener's office, he would listen. And if they called the regular, they would listen. And this is a California specific one. I know this podcast goes to everybody. But I think academia needs to wake up and talk. I think little tech needs to wake up. And I do think that being vocal, I think political lobbying actually is quite important. I think making views known is quite important. I just think

Starting point is 00:40:02 the silent masses needs to arise. If there's one takeaway I've had, having lived in the 2000, has been totally silent, having lived now, is how different the landscape is. It just feels like back then big tech just wasn't as dominant. They weren't as involved in the government. They weren't as involved in academia. And I just feel like all of these things are not bought and paid for it. And there's so few neutral voices left. And so for those that care, I think just coming out and speaking for what's right is very important. Can you teach us conceptually a little bit about the importance and role of open source versus closed source? Those are words that everyone uses. But in a sort of brass tax way, describe the role that those two things play and what they both mean to you, just in general in technology?

Starting point is 00:40:40 Yeah, for sure. I'm going to be very clear about one thing. I'm in no way in open source is all it. My company had some open source and some close source. I invest in close source companies all the time. And so I'm in no way thinking that you should only do open source. But it's played an incredibly important role in the industry. and one way to view it is almost like this democratized or commoditizer of mature businesses. Unix was one of the first operating systems. It was a standard. It was used in the enterprise. Then the open source community created Linux, first starting from Linux Torvald. And then it followed the path of Unix. But the entire, all of the academics and the researchers, and the hobbyists and the companies evolved that they would add security and solve a number of problems.

Starting point is 00:41:26 And so because the closed source version had created this market, the open source version could come and create a flourishing ecosystem that everybody can participate. So it democratized operating system. We've seen this in many things. We saw this in databases. So like Oracle, of course, and Sybase for databases, MySQL comes, Postgres comes, and then it follows along.

Starting point is 00:41:46 Of course, the smartphone, then Android comes and follows along. So it's always been this, or historically been this almost democratizing, force that is a checks and balance for the close source version and allows anybody else who doesn't have access to it to innovate. And if you look at, again, the history of security and safety and innovation, so much of that has been in the open source. So if you remove that, we lose all of that. We lose all of those brains and all of that effort and all of that energy in it. If you think about the talent in all of this, how does it feel different to you, if at all, from the way you source, interact with, the types of people building where they're coming from,

Starting point is 00:42:28 like where you're looking for them versus the pool of SaaS talent or something like that. Does AI feel like a shift in that sense to you as an investor? Yeah, it's actually very interesting, which is you are seeing a rise of the H.C.E. CEOs, which is actually pretty new. Faculty at Berkeley, PhD guy, George Fraser, 5Tran. So you're actually starting to see almost these very deep tech. focuses. And I don't know if it's because we're early in a super cycle and they're just on the bleeding edge or it actually is a bit of a different discipline that it's not just engineering and

Starting point is 00:43:02 just business, but it's actually more a core two signs. I don't know the answer to this. That's one. There is an open question that we debate often in A16C. And the question is, do you back the core technical team or do you back the commercial team in a different modality? Is the tech democratized enough that any good founder can get the right team and use it or just the team have to understand the core tech. And I got to say, I personally go back and forth on this a lot. I think because I tend to come from this night, mid-grad school, I did research, I did a company based on that. I tend to err towards the deeply technical teams, but that's clearly not always right. And so I do think a lot is up in the air. I think commercial versus tech and up in the air. I think researcher

Starting point is 00:43:41 versus engineers up in the air in this realm. And it's an active dialogue. Maybe it's a great excuse, given that a lot of these companies are in this phase of going from zero to 10 plus million of ARR to talk about your concept of company market annealing. It sticks out my memory as a really interesting way to think about the early stages of companies and since so many are going through that now, could you describe that idea in that framework? I've read all of these books and they're all very useful about finding product market fit, but that's not how it ever seems to work.

Starting point is 00:44:09 And now, listen, I'm sat on many boards. I've watched this many times. And so the typical story in finding product market fit is you, the company, create some product and you go ask the market what it wants. And then based on its feedback, you tweak your product and you can A, B, test your way into finding the right thing. And so you get all these aphorisms that I find so stupid. Solve your own problem and then you have a company, which is just not true. Or solve a real problem with the customer.

Starting point is 00:44:34 And it's not helpful, right? None of these things tell you actually how to find product market fit. So here's my experience, having watched it played out multiple times, which is you both have to know your technology, evolve that as well as make the market pliable and educate the market. This is if you're doing category creation. Market in a mailing is for category creation. If you're in a mature market, just go build something better. People already know how to use it. But in new markets, like the customer doesn't know the value that you have. They have a clue. They don't know how to use it. They don't have the workflow. So there's a ton of work that you're actually hammering the market and making it so

Starting point is 00:45:08 they can accept it. And so the way that I think about is almost like this kind of annealing where you work a little bit on your product and then you actually work on the market. By work on the market, I don't mean do interviews. I think this is like the most broken way of building a company. So you go ask, imagine this. You're going to go interview a customer and ask what they want. They're sitting down there and they feel like they need to answer something. So they'll just make something up.

Starting point is 00:45:28 Oh, I need this. I want a pony, right? It's just almost like it's too much leading to win. This is an essential buyer. And so the most successful companies, they have a true norm. They're like, we know that this technology should involve them this way and we need to educate them on this, but there's going to be some real organizational constraints

Starting point is 00:45:45 and real budget constraints. We're going to find those at the same time. So they go pull and push on the market a little bit, and then they tweak their product to do that, and then they go pull and push on the market, and they tweak the product, and they do that. And this give and take this annealing. And that's how most companies do category creation, in my opinion,

Starting point is 00:45:59 not this, go ask the market what it wants, or go solve your own problem, the market will just understand it. If that works for you, you're very lucky, but I've never seen it. We've talked a lot about models and some of the apps built on those models. We've talked a little bit less of, first time I mean, you and I talk mostly about the modern data stack and just data in organizations. And we haven't talked much about bare metal, like compute. And I'd love to just spend a minute or two with your impression of what's going on in each of those two worlds, maybe starting with the data side. What are new things that you're thinking about that matter in the world of data and get exaggerated by the rise of AI?

Starting point is 00:46:37 Clearly, unstructured data now is very, very important. We come from this cloud data warehouse world where people use SQL. and data is very well behaved and it's in some relational database somewhere and we query it and whatever. That's still incredibly important. It's still growing. But that was the dominant thing. Now more and more, these data pipelines are, give me a bunch of random videos and a bunch of random images, and a bunch of random music and a bunch of random stuff. I'm going to sort of how like wrangle all of those things and use that to train this model. And oh, by the way, you have to be able to do this in a way that reproduced a little scriptible. So it's almost like the unstructured role in the structured world

Starting point is 00:47:12 are converging, which I think is great. So clearly data continues to grow important, like it has been doing from the beginning of compute, but the size for the unstructured has just been growing asymptotically. The world need new things that it doesn't have to handle all of that, new companies, new technologies, new ways of working. I think so. I don't think we yet know how to handle large pools of super unstructured data in a way that's efficient.

Starting point is 00:47:36 Imagine how much video there is out there. The ability to query across video, the ability to put it into the workflows, the ability to pull stuff out. I still think there's this kind of a lot of work. And there's a lot of companies working on it. And so I know right now there's a founder that, oh, but Bartin, my company does that. Yes, I know that. There's a lot of companies that are working on these problems, but they're not solved in the sense that somebody can't just take a big pool of video and then be able to worry it just like SQL right away. And then there's just a lot of edge cases for this. I do think that we need a mature as an industry, almost like the software industry has matured, just the tooling around managing data and getting it ready for this.

Starting point is 00:48:11 The collection of data is becoming this absolutely paramount in the labeling of data. And there's one lesson I've come away with in all of this stuff is the existing data that we have is great. But once you exhaust that by throwing a lot more compute at it, you've got to go get more. And nobody really knows how to do that other than maybe just pay a lot of money. I think that we need some real advances there to keep things living. What about on the hardware side? Anything interesting that you've seen on, obviously, Nvidia hogs all the attention in the AI world right now.

Starting point is 00:48:41 especially in public markets. Anything on just the raw hardware, semi's power side that has you curious or intrigued? It's great that we're compute down, right? I mean, the good news is these workloads are very different than cloud workloads. So we're going to see massive innovation all the way up and down the infrastructure. The communication patterns for training and the requirements on the fabric are very different than cloud workloads. And they're very different than inference.

Starting point is 00:49:05 It's not clear that you would even want to use the same fabric over time. We're back seeing like massive innovation network. The last time we saw that was when we were doing fat tree work for data center. So when the data center came out, we never had that amount of density of servers in the same space. We had to evolve networking to be able to manage that much bandwidth. And then we haven't seen a lot of innovation, but now this AI is pushing us to really innovate on the networking fabric, which just goes to show you how much we have to reimagine the entire stack. And it fills on the silicon, to me it feels still very early days.

Starting point is 00:49:36 Let me just give you just a very interesting anecdote. Let's say you're going to do a billion-dollar training run for a model. So how much does it cost to create a custom-ASIC? Let's say $200 million to create a custom-a-cum-a-custom-a. So if for $200 million, you could get a 50% improvement. A 50% improvement of a billion dollars is $500 million. The $200 million is well spent. It's like entering this space where it almost makes sense build a custom-asic for one training run.

Starting point is 00:50:07 The economics are there that it becomes very compelling to do hard for harbor work. Now, listen, of course I understand all the complications of that. Yes, timelines, et cetera. But I'm just trying to give everybody a mental model that it's been a very long time that you can justify building a custom piece of silicon for a single project like this. But the economics are actually there now, which is why you're seeing so much work in silicon again. So it's very exciting. I think the entire stack is going to be disrupted.

Starting point is 00:50:34 Can you say a word on the relationship between investors and, capital markets and pools of capital and whether or not it feels correctly calibrated to you, the needs of the marketplace to do all this stuff. You've seen relative to other parts of the economy some pretty lofty valuations relative to fundamentals or whatever, some huge checks being written by very large firms. What do you think? Do we have the right capital market set up for this explosion of innovation and ideas? Let me give you the two bookends. views of this, somewhere in the middle. So one view of this, a common view is, this is all a bubble, investors are stupid, valuations are crazy, it doesn't make any sense, markets are not efficient

Starting point is 00:51:13 in this sense, this will all go away. That's one view. There's another view, which is actually, we know that the economics work, actually, there's a bunch of value creation. All of this makes sense, markets are very efficient. I tend to be more of the latter one, but it's a very nuanced discussion because you can maximize the value in expectation. So the expected value is like the probability something's going to happen time the potential payout. The expected value of solving free energy is infinite, but the chances that it happen is almost zero. And so I think markets are very efficient. I think markets realize that this is disruptive and there's going to be a lot of great companies. They realize that, but they don't know where. And so I actually think the sizing is

Starting point is 00:51:56 correct. I think this is the same thing, by the way, with the internet, which is everybody said that go like the dot com was crazy and they overpaid, except for it was literally the growth driver of the world economy for 20 years, and way more than made up for the valuation. I think that will happen again. The question is how do you sensibly place those bets when you don't know for any given bet what the expected outcome is going to be? You just don't know these things. I do think this is one of the big ones. I do think this is a super cycle. I do think this is is going to be a massive debater. I think once it's all said, then you look back at these valuations, they're going to look very sensible. But that said, any independent bet may look silly.

Starting point is 00:52:34 And I think that's the nature of venture capital and I think that's the nature of investment. You've talked a lot about places where you feel the story is already being told, creativity, that the tools that are available and already possible today versus things that are uncertain. What area of uncertainty do you find yourself spending the most personal time reading about, thinking about, talking about? Do you know what the first video on the internet was. Of course, I'm going to guess like cats or something. It was a coffee pot. I think there's this guy in Cambridge, some researcher, and at the end of the hall was a coffee pot, and he didn't want to walk there if there was no coffee in the coffee pot because that would be a wasted trip, and so he put

Starting point is 00:53:11 a camera on that. And of course, it was on the internet, and so it became super, super popular. Right. Because, oh my gosh, if you can have that, then, you know, of course, you can have like that place. If you remember the big red button, this is a very popular app early on was the big red button on the internet. It was a web page with a big red button. And if you pushed it, nothing happened. It just pushed. That was it. That was the entire thing. It was super popular and people were super into it. Then you ask why. You're like, oh, this is just silly. Or you could be like, maybe it's because people saw the big red button. And then they imagine like you have one button, you have a bunch of buttons and then you have Salesforce, right? And so I will say,

Starting point is 00:53:46 having been through the web transition, I saw all the silly things and they were silly. It was all video games and goofy stuff and culturally marginal stuff, but those are the things that grew up to be the Googles and the Yahoo's and the Amazon's. And none of us knew what that looked like on the other side. And there was all of these half steps. Do you remember desktop as a service? The stupid thing, we're like, I've got a desktop here. I'm going to run a desktop in the cloud. There's all these half steps where you're stuck in the old way of thinking and you can't really do a new way of thinking. So where I am right now is this is the early days of the web. And a lot of things that look silly, talking to a virtual girlfriend or making an image of a cat, those I know are going

Starting point is 00:54:30 to grow up to be the thing. I know it in my bones because I've seen it before, but I don't know what those are going to look like. And so much of what I'm asking is, what is the big red button versus not? What is that going to become? What is the future going to look like? Let me just give you one anecdote, I think is interesting on there. So my daughter's 13 COVID kid, right, months are COVID. And that was very disruptive, and it was very tough. Basically, she went from being a very social kid to basically interacting online. And lately, she's been a big user of Character AI. So Character AI is from Shazir, who is one of the Transformer authors,

Starting point is 00:55:01 and it's just like these kind of characters that you talk to. It seems silly. But she really likes it. She's got these characters. And I've noticed recently that not only does she talk to these characters, but she'll actually invite them to group chats with her friends. And so it's entered the social dynamic. So it very well could be that we're entering a phase where the actual social dynamics shift.

Starting point is 00:55:21 And then I ask myself, okay, so for my daughter, in five years, she's 13, in five years, when she opens a bank account, is she going to go to Wells Fargo SaaS app or is she going to do it from her character? 100% from her character, right? It's almost like the access layer to the internet is changing, right? If she's going to do search, is she going to go to Google? Are she going to do it through her character? So I just feel like these very kind of almost silly, goofy, fundamental changes that are happening. right now may become the thing. And when the access layer to the internet changes or to compute changes, like everything changes behind it. So that's really what we spend a lot of time thinking about.

Starting point is 00:55:55 It brings to mind a question on agents. And right now you've got these chat models where it's like the smartest model you've ever seen that has no memory and can't really affect other systems. And it seems like the next step would be it has memory, it knows about you, or it has context, and it can affect other systems that can create new actions elsewhere. How do you think about it? Is overly simplified way of thinking about what an agent is. What is an agent to you? What do you think about them? Everyone's talking about agents. So one thing that I love working on a team like I do, I feel so lucky to have the team that I do. Everybody's so smart, so engaged and so knowledgeable, is we have a different set of opinions. And the reason I bring this up this way is the agents is

Starting point is 00:56:32 one where we've got very different opinions. We 100% want to invest in agents and in the agent space, and we look at all of these companies. But I personally have been very bearish on agents as a technical direction. And the reason for this, there's a number of reasons for this is, but I just feel like there are some very key technical problems that we do not quite cracked. Like, for example, the memory one that you've mentioned, erectus is another one that we haven't done. We don't really know how to do planning very well. So it's not clear to me that the advances that we've seen with LLM's models carry over to fully automation of agentic stuff. Now, that said, most of the people on my team disagree with me and think that I'm a boomer. Okay, boom.

Starting point is 00:57:13 or like the stuff is amazing. I've seen these things work and they're amazing. I'm very open to be totally wrong on this point. But it just hasn't been super obvious. There's actually another area where I'm like this, which is this kind of idea that AI is going to code for us. So this kind of one view, it's like coding is done. We're just going to tell an AI what we want and it'll do everything for us. And that's almost an agentic thing, right? It'll go do the coding. And many people on my team believe that is the future. And that's great. They should do all of those deals. But my view is coding is the correctness is really important. I think people spend like 90% of their time when they develop maintaining code. So actually understanding and readability is very important too.

Starting point is 00:57:52 So it feels to me that the human being still has to know everything that's going on. And so clearly coding will evolve and the UI will evolve and AI will help it. But I don't think that coding goes away. I think we just, we end up coding more. And I actually don't think that developers will have more time a week, but I have less time in a week because they're going to be so much more efficient. I think just low code, no code has always promised to get rid of programmers and never did it. I think AI will potentially hold the same false problem. But again, in my team, there's massive disagreements on this.

Starting point is 00:58:21 And so I'm a huge fan of company like Cursor. It creates a new IDE that's AI-Native IDE. And it's going to go ahead and change how people code and it doesn't replace developers. Or other people are like, listen, why would you ever need an IDE because you'll never need a programmer? And I'm going to invest in this company that does code. What is the definition of an agent to you? How would you define it? An agent to me has a control loop and state.

Starting point is 00:58:41 So you give it a high-level objective, and then it will do multiple steps along that high-level objective, and it maintains states between that, and then it actually has a controller of feedback loop to determine the next step. And I don't think we've solved the state problem. I just don't think that's necessarily part of the reasoning fabric right now. It almost feels like it's tacked on. I don't think that we've solved the control loop problem in a way that converges. I feel like we still don't really know how to handle the nondeterminism in these models. And so it's not super clear to me that this recent breakthroughs on these large generative models cracks planning, breakfast, agentic problem. Again, very open to being wrong, but that's my intuition.

Starting point is 00:59:27 In addition to the agent and the automatic coding, are there other areas where you feel like your views are the most divergent, either from your team or from the markets? I think one of the biggest mistakes that we're making as an industry is AI has been around for so long. It's all these enterprise use cases that we try to cram it into before. Customer service chat bots and basic automation and this low-code stuff. And what we're doing is we're taking this new thing and just cramming it all of these old use cases.

Starting point is 00:59:55 And we're like, oh, well, it's like this old version of it. But now with like generative AI, I think this is just massive lack of creativity and vision. I think that this is a new breakthrough, in my opinion. It's got its own set of properties. And those properties are amazing for certain problems and not amazing for other problems. I personally am very excited and attracted to the problem. I think it's a great fit for the emotion companionship. I think it's a great fit for the creativity problem.

Starting point is 01:00:22 I think it's a great fit for the language reasoning problem. It's a great fit for those. I don't think it's as good as a fit for it. I'm going to now use an LLM to replace some workflow that requires correctness over long periods of time where you don't have a human check in et cetera. I just feel in general, I can look towards the areas that are a bit emergent. But then as an industry, we just don't know them as well. They're just like the internet. Like, things look so silly. Whenever I talk about creativity and companionship, people are like,

Starting point is 01:00:50 oh, I thought you were a serious investor. But what they don't know, these are the things that become the new thing, I promise. These are the big red buttons. And it's the copy pot on the internet. These are glimmers. These are signs of how apps. will look in the future. And that's why I started when you asked me in the very beginning, I think the future is going to be a lot weirder and wilder and more wonderful than it is today. Exactly. If you could interview anyone in the world right now that have been given truth serum, who would you interview and what would you ask them? It's my clever way of asking, what do you not know that you want to know? I would love to know as if many of the people in the AI

Starting point is 01:01:26 safety debates, I want to know who's a Baptist and who's a bootleger. So do you know this kind of dichotomy. Explain it, but sure. I'm going to change your question. I want to be able to ask 10 people, and I want to create these are Baptist and bootleggers. So Baptist and bootleggers is an economic phenomenon where you have regulation, for example, in the time of prohibition, and the people that support it, you can often categorize is basically true believers and opportunists.

Starting point is 01:01:50 So in the time of prohibition, the true believers of the Baptist really thought that alcohol was bad. It caused to do bad things. You're going to go to hell. And they truly believe that the Baptist thought prohibition was good at alcohol is terrible. So the true believers actually believe it. It turns out that in any of these situations, you also have opportunists to access to the regulation is good. They're called bootlakers. So bootlakers, it turns out, really thought that prohibition

Starting point is 01:02:09 was great because then they could actually sell alcohol for a premium and they are put in business. I think that this is a game theoretic theme. I think this is an emergent property of these environments and it's not a value judgment on any single person. I just think that this will always emerge. And we're definitely in an area where there are regulations that are hampering innovation or will hamper innovation that are being espoused. And you have many people in the room. And I think some of them are true believers. They're true Baptists. They think AI is going to recursively improve itself and kill us all and take jobs. But I don't think there's very many of those. I think the vast majority of those are actual bootlegers. Listen, this is good for my big company. This is good for my

Starting point is 01:02:52 agenda, this is good for whatever it is, and they're driving that. And I honestly couldn't tell you who is who is who. And so what I would do is you're giving me one person to ask and I'd find some way to split it into five people to ask and I'll try to find out who's a backness and there's a bootlaker. I love the answer. To end on a high note, what thing, if you had to go really tactical and specific, everything we talked about, what excites you the most? Can I say two things actually. I think that excites me the most. I think it's a theft accompli. There's no holding this back. It doesn't matter how much bad regulation. It doesn't matter how much these large companies try to influence things.

Starting point is 01:03:27 This shift is going to be so dramatic over time. It'll win. The market will win here for sure. And I feel so good about that. I'm the most frustrated. I just feel very calm. I just know that it's done. We just have to wait.

Starting point is 01:03:38 It's just going to be above people on the way. And I do think that this, what has been, I do think it's going to change how we think about the universe and how we think about each other and how I think about the world and how we do it. I really think it's kind of that order of thing and it's going to happen. We just need to let it happen. and then we can ease away a pub. And then a very specific thing.

Starting point is 01:03:54 So that gives me a lot of peace that I just feel that's done. The very specific thing is I do feel that we're very close to being able to type a prompt into a computer and end up an entire virtual world that you can enter and is fully complete and is incredibly deep and has every aspect that we've ever wanted. We've had these notions in the past of the HoloTech or the metaverse. And all of these is you have to have humans. constructing it and designing it, et cetera. But I actually think that we've created enough content in the past that we can now auto-generate these things. And they're going to be as

Starting point is 01:04:28 rich and as interesting in the real world. And so I'll end on this. The one takeaway, people probably should have taken from Beostrom is not that AI is going to take over everything. I think the one takeaway is we probably are living in a simulation and we're probably about to create the next one. Martine, so much fun to talk to you every time. Thank you for your time. Yeah, delighted. If you enjoyed this episode, check out. out join colossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus

Starting point is 01:05:00 Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

Invest Like the Best with Patrick O'Shaughnessy - Martin Casado - Entering Uncharted AI Territory - [Invest Like the Best, EP.381]

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.