Invest Like the Best with Patrick O'Shaughnessy - Chetan Puttagunta and Modest Proposal - Capital, Compute & AI Scaling - [Invest Like the Best, EP.400]

Starting point is 00:00:02 Hello and welcome, everyone. I'm Patrick O'Shaughnessy, and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. Invest Like the Best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at join colossus.com. Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of positive sum. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions.

Starting point is 00:00:45 Clients of positive sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc. My guests today are Chaitan Putagunta and modest proposal. If you're as obsessed as I am about the frontier in AI and the business and investing implications. You will love this conversation. Chathen is a general partner and investor at benchmark, while modest proposal is an anonymous investor who manages a large pool of capital in public markets. Both are good friends and frequent guests on the show, but this is the first time that they have appeared together. The timing could not be better. We might be witnessing a pivotal shift

Starting point is 00:01:22 in AI development as leading labs hit scaling limits and transition from pre-training to test time compute. Together, we explore how this change could democratize AI development while reshaping the investment landscape across both public and private. markets. Please enjoy this great discussion with my friends, Chatham Poudaunta, and Modest Proposal. So Chathin, maybe you can start by just telling us from your perspective, what is going on right now that is most interesting in the technology part of the story of LLMs and their scaling? Yeah, I think we're now at a point where it's either consensus or universally known that all the labs have hit some kind of plateauing effect on how we perceive scaling for the last two years,

Starting point is 00:02:10 which was specifically in the pre-training world. And the power laws of scaling stipulated that the more you could increase compute in pre-training, the better model you were going to get. And everything was thought of in orders of magnitude. So throw 10x more compute at the problem, and you've got a step function in model performance and intelligence. And this certainly led to incredible breakthroughs here. And we saw from all of the labs really terrific models.

Starting point is 00:02:43 The overhang on all of this, even starting in late 2022, was at some point we were going to run out of text data that was generated by human beings. And we were going to enter the world of synthetic data fairly quickly. all of the world's knowledge effectively had been tokenized and had been digested by these models. And sure, there were niche data and private data and all these little repositories that hadn't been tokenized, but in terms of orders of magnitude, it wasn't going to increase the amount of available data for these models particularly significantly. As we looked out in 2022, you saw this big question of was synthetic data going to enable these models to continue to scale. Everybody assumed, as you saw that line,

Starting point is 00:03:34 this problem was going to really come to the forefront in 2024. And here we are. We're here and we're all trying to train on synthetic data, the large model providers. And now, as it's been reported in the press and as all these AI lab leaders have gone on the record, we're now hitting limits because of synthetic data. The synthetic data as generated by the LLMs themselves are not enabling the scaling and pre-training to continue. And so we're now shifting to a new paradigm called test time compute. And what test time compute is in a very basic way is you actually ask the LLM to look at the problem, come up with a set of potential solutions to it, and pursue multiple solutions in parallel.

Starting point is 00:04:23 You create this thing called a verifier. and you pass through the solution over and over again iteratively. And the new paradigm of scaling, if you will, the X-axis is time measured in logarithmic scale, and intelligence is on the Y scale. And that's where we are today where it seems that almost everybody is moving to a world where we're scaling on pre-training and training to scaling on what's now being called reasoning, or that is inference time, test time, however you want to call it. And that's where we are as of Q4, 2024.

Starting point is 00:05:03 Just a follow-up question on the overall picture. So setting aside CAPEX and all this other stuff that we'll talk about with the big public tech companies in just a moment, is it reasonable to say based on what you know now that the switch to test time scaling, where time is the variable, is like a who cares? As long as these things keep getting more and more capable, isn't that all that matters? And the fact that we're doing it in a different way than just based on pre-training, does anyone really care? Does it matter? There's two things that come up pretty quickly in test time or reasoning paradigm, which is,

Starting point is 00:05:43 as LLMs explore the space for potential solutions, very quickly as a model developer or somebody working on models, you quickly realize that algorithms used for test time compute might exhaust the useful search space for solutions quite quickly. That's number one. Number two, you have this thing called a verifier that's looking at what's potentially a good solution, what's potentially a bad solution, what should you pursue. And the ability to figure out what's a good solution, what's a bad solution or what's an optimal path and not an optimal path, it's unclear that that scales linearly with infinite compute. And then finally, tasks themselves can be complex, ambiguous,

Starting point is 00:06:35 and the limiting factor there may or may not be compute. So it's always really interesting to think of these problems as if you were to have infinite compute to solve this problem, could you go faster? And certainly there's going to be a number of problems in reasoning where you could go faster if you just scaled compute. But oftentimes we're starting to see evidence

Starting point is 00:06:56 that it's not necessarily something that scales with compute linearly with the technology we have today. Now, can we solve all of that? Of course. There's going to be algorithmic improvement. There's going to be data improvement. There's going to be hardware improvement.

Starting point is 00:07:12 There's going to be all sorts of optimization improvements here. The other thing we're still finding is the inherent knowledge or data available to the underlying model that you're using for a reasoning still continues to be limited. And just because you're pursuing test time, it doesn't mean that you can break through all previous data limitations by just scaling compute at test time. So it's not that we're hitting walls on reasoning or we're hitting walls on test time. It's just the problem set and the challenges and the computer science problems are starting to evolve. And as a venture capitalist, I'm very optimistic that we're going to be able to solve all of them, but they're who be solved.

Starting point is 00:07:59 So if that's the sort of research labs outview, modest, I'm curious for you to give us the big public tech companies down view because so much of this story has been the spend, cap-x, the strategic positioning, the quote-unquote ROI on all this spend and how they're going to earn a return on this insane outlay of capital. Do you think that everything Jathan just said is well reflected in the stance and the pricing and evaluations of the public tech companies? I think you have to start at a macro level and then get to a micro level. At a macro level, why this is so important is everyone knows the MagS7. a larger percent of the S&P 500 they represent today.

Starting point is 00:08:46 But beyond that, I think thematically, AI has permeated far broader into industrials, into utilities, and really makes up, I would argue, somewhere between 40 and 45 percent of the market cap is a direct play on this. And if you even abstract to the rest of the world, you start bringing in ASML, you bring in TSM, you bring in the entire Japanese chip sector. And so if you look at the cumulative market cap that is a direct play on artificial intelligence right now, it's enormous.

Starting point is 00:09:25 And so I think as you look across the investment landscape, you almost are forced to have an opinion on this. because most people in some form or another are benchmarked against an index that is going to be a derivative play on artificial intelligence. At the micro level, I think that this is a fascinating time because all of public market investing is scenario analysis and probability weighting different paths. And if you go back to when we talked probably four months ago, I would say that the distribution of outcomes has shifted. And at that point in time, pre-training and scaling on that axis was definitely the way. And we talked about what the implications were at the time. We've talked about Pascal's Wager or we've talked about prisoners still out. And in my mind, it was easy to talk about that when the cost of anti-in-up was a billion

Starting point is 00:10:25 dollars or five billion dollars. But we were rapidly approaching the point in time where the ante was. was going to be $20 billion or $50 billion. And you can look at the cash flow statements of these companies. It's hard to sneak in a $30 billion training. And so the success of GPT5 clats, gladly, let's apply that to all the various slabs, I think was going to be a big proof point

Starting point is 00:10:54 as to whether or not the amount of capital was committed because these are three, four-year commitments. If you go back to when the article was written on Stargate, which is the hypothesized $100 billion data center that OpenAI and Microsoft were talking about, that was a 2028 delivery. But at some point here in the next six to nine months, it's a go note. We already know that the three to 400,000 chip supercluster is going to be delivered end of next year, early 2026. But we probably need to see some evidence of success on this next model in order to get the next round of commitment.

Starting point is 00:11:35 So all that is a backdrop. I think at the micro level, this is a really powerful shift if we move from pre-training to inference time. And there are a couple big ramifications. One, it better aligns revenue generation and expenditure. I think that is a really, really beneficial outcome for the industry at large. which is in the pre-training world, you were going to spend $20, $30, $40 billion on KepX, train the model over 9 to 12 months, do post-training, then roll it out, then hope to generate revenue off of that in inference.

Starting point is 00:12:09 In a test time compute scaling world, you are now aligning your expenditures with the underlying usage of the model. So just from a pure efficiency and scalability on a financial side, this is a much, much better for the hypers. I think a second big implication, again, we have to say we don't know that free training, scaling is going to stop. But if you do see this shift towards inference time, I think that you need to start to think about how do you re-architecture the network design?

Starting point is 00:12:47 Do you need million-chips superclusters in energy, low-cost land locations? or do you need smaller, lower latency, more efficient inference time data centers scattered throughout the country? And as you re-architect the network, the implications on power utilization, grid design, a lot of the, I would say, narratives that have underpinned huge swaths of the investment world, I think have to be rethought. And I would say today, because this is a relatively new phenomenon, I don't believe that the public markets have started to grapple with what that potential new architecture looks like and how that may impact some of the underlying spec.

Starting point is 00:13:38 Jathan, I'm curious maybe to tell the story of deep seek and other things like it, where you see new models being built by small teams for relatively small dollars that are competing in performance with some of the leading edge models. Can you talk about that phenomenon and what it makes you think about or the implications for the landscape? It's really amazing in the last, call it six weeks of time, the number of teams we've met here at Benchmark that are two to five people. And Modis has talked about this in your podcast before, which is that the story of technology innovation has been, there's always been two to three people in a garage somewhere in Palo Alto doing something to catch up to incumbents very, very quickly. I think we're seeing

Starting point is 00:14:27 that now in the model layer in a way that we haven't seen, frankly, in two years. Specifically, I think we still don't know 100% that pre-training and training scaling isn't coming back. We don't know that yet. But at the moment, at this plateauing time, we're starting to see these small teams catch up to the frontier. And what I mean by frontier is where are the state of the art models, especially around text performing? We're seeing these small teams of quite literally two to five people jumping to the frontier with spend that is not one order, but multiple orders of magnitude, do less than what these large labs we're spending to get there. I think part of what's happened is the incredible proliferation of open source models.

Starting point is 00:15:20 Specifically what Meta's been doing with Lama has been an extraordinary force here. Lama 3.1 comes in three flavors, $405 billion, $70 billion, $8 billion. And then Lama 3.2 comes in $1 billion, $3 billion, $11 billion, and $90 billion. and you can take these models, download them, put them on local machine, you can put them in a cloud, you can put them on a server, and you can use these models to distill, fine-tune, train on top of modify, et cetera, et cetera, and catch up to the frontier with pretty interesting algorithmic techniques. And because you don't need massive amounts of compute or you don't need massive amounts of data, you could be particularly clever and innovative about a specific vertical space or a specific technique or a particular use case to jump to the frontier very, very quickly.

Starting point is 00:16:18 I think that is largely changing how I personally think about the model layer and potential early stage investments in the model layer. There's a lot of ifs here and a lot of dependent variables. And literally in six weeks, none of this could be true anymore. But if this state holds, which is that pre-training isn't scaling because of synthetic data, it just means that you can now do a lot more, jump to the frontier very quickly with a minimum amount of capital, find your use case, find where you're most powerful. And then from that point onward, the hypers, frankly, become best friends. Because today, if you are at the frontier, you're powering your use case, you're not,

Starting point is 00:17:03 particularly GPU constraint anymore, especially if you're going to pursue test time inference or test time compute or something like that. And you're serving, let's take 10 enterprise customers, or maybe it's a consumer solution that's optimized for a particular use case. The compute side of it just doesn't become as challenging as it was in 2022. In 2022, you would talk to these developers and it just became a question of, well, could you get 100,000 cluster together? Because we need to go train and then we have to go buy all these data. And then even if you knew all the techniques, all of a sudden you would pencil it out and say like, I need a billion dollars to get the first training run to go. And that just is not a model historically that's been the venture

Starting point is 00:17:46 capital model. The venture capital model has been, could you get together a team of extraordinary people, have a technology breakthrough, be capital light, and jump way ahead of incumbents very quickly and then somehow get a distribution foothold and go. At the model layer for the last two years, that certainly didn't seem like it was possible. And literally in the last six, eight weeks, that's definitively changed. I think it's important the point about meta, open source,

Starting point is 00:18:18 and the hypers. Open source pushing the frontier, smaller models being able to scale to very successful points is enormously beneficial, particularly for AWS, who doesn't have native LLM. But if you just take a step back and think about what historically cloud computing was, it was providing a set of tooling to developers and builders. AWS first articulated this vision.

Starting point is 00:18:46 I heard it publicly in September when Matt Garman was at Goldman Sachs conference. But their view clearly has been that LLMs are just another tool. that generative AI is another tool that they can provide their enterprise customers and their developer customers to build the next generation of products. The risk to that vision was an all-powerful, generalizable mob. And so again, this is where you sort of have to rethink if we're not going to build these massive pre-trained entities where you drive training loss down to near nothing, and that in some form or another builds the metaphorical God, if instead the focus of the industry is at test time, at inference time, and trying to solve real problems at the point of need for a customer,

Starting point is 00:19:41 I think that, again, re-engineers and re-architects the entire vision of how this technology rolls out. And I think we need to be humbled that we don't know what Lama 4 is going to come, out with. We don't know what GROC 3 is going to come out with. Those are the two models that are currently being trained on the largest clusters ever. So everything we're saying right now may be wrong in three months. But I think the entire job right now is to ingest all the available information and replot the various scenario paths. With what we know today, I do not feel like people have updated their priors as to how these pads may go forward if this is correct. I'm curious, Jathan, how the idea that maybe now you would invest in a model company because of this change,

Starting point is 00:20:34 I remember you telling me over dinner two years ago that as a firm, you just decided we're not investing in these companies. Like you said, it's just not the model that we do. We don't write billion dollar checks for the first training run. And so we're not investing in that part of the stack. We're investing more in the application layer, which will come back to. in a little bit in this discussion. But maybe say a bit more about this updated view on how that could work out what a sample investment could look like. And whether even if Lama 4 is the pre-training scaling loss hold, if that even changes

Starting point is 00:21:04 that because it would just seem like something like Deep Seek just only benefits from, okay, now instead of 3.2, it's 4 and like we're still doing our thing and it's still better and cheaper and faster and whatever. So yeah, what do you think about this new view you have potentially invest? in model companies, not just application companies. In Meta's last earnings call, Mark Zuckerberg talked about them starting Lama 4 development, and he said that Lama 4 is being trained on a bigger cluster than anything he's ever seen out there. The number that was quoted was it's bigger than 100,000 H-100s or bigger than anything I've seen

Starting point is 00:21:42 reported for what others are doing. And he also said, you know, the smaller Lama 4 models should be ready in early 2025. What's really interesting about that is that regardless of whether Lama 4 is a step function from Lama 3, kind of doesn't matter. If they push the boundaries of efficiency and get to a point where even if it's incrementally better, what it does to the developer landscape is pretty profound because the force of Lama today has been two things, and I think this has been very beneficial to meta, is one, the transformer architecture that Lama is using is a sort of standard architecture, but it has its own nuances.

Starting point is 00:22:26 And if the entire developer ecosystem that's building on top of Lama is starting to just assume that that Lama 3 transformer architecture is the foundational and sort of standard way of doing things, it's sort of standardizing the entire stack tool. towards this Lama way of thinking, all the way from the hardware renders will support your training runs to the hypers and on and on and on. And so standardizing on Lama itself is starting to become more and more prevalent. And so if you were to start a new model company, what ends up happening is starting with Lama today is not only great because Lama is open source. It's also extraordinarily efficient because the entire ecosystem is standardizing on that

Starting point is 00:23:13 architecture. And so you're right, as an early stage fund with $500 million, a capital, and we're trying to make 30 investments every fund cycle, a billion dollar training run is essentially you're committing two funds to do one training run that may or may not work. And so that's an extraordinarily capital intensive business. And by the way, the depreciation schedule for these models is frightening. Distillation as a technique. makes defensibility of these models and most of these models extraordinarily challenging. And it really comes down to what's your application on top of it,

Starting point is 00:23:51 what's your network effects, how are you capturing economics there and all of that. And I think what has now the case as of today is if you're a two to five person team, you can take something like coding as an example. And you could push your way into a model that generates better coding answers faster by fine-tuning and training on top of Lama and then offer an application with your own custom models that really produces extraordinary results for your customers, whether it's developers or something like that.

Starting point is 00:24:25 And so our particular approach and strategy here has been to invest heavily in applications, starting when we saw OpenAI APIs start to take off. We started to see developers talk about these OpenAI APIs. the summer of 2022. And a lot of our efforts starting then was to just find entrepreneurs that were thinking about leveraging these APIs

Starting point is 00:24:50 to go after the application layer and really start thinking about what are applications that simply could not exist before this current wave of AI? Obviously, we've seen some really incredible successful companies come out of that that are still early, but the kind of traction they're seeing, the kind of customer experience they're providing,

Starting point is 00:25:12 the kind of why metrics, all of that has been extraordinary. You had Brett Taylor on your podcast a couple of weeks ago, so Sierra is an example of this. In procurement, we have a thing called Level Path, many other examples across the portfolio at the application layer where you can just go through every single large SaaS market and go after it with an application layer investment and start to really think about what's now possible

Starting point is 00:25:38 that wasn't possible two, three, four years ago. I'm curious to talk a little bit about the big foundation model players that we talked about Lama, but less about XAI, Anthropic, and Open AI. Maybe Mata starting with you. I'm curious for just your thoughts on their strategic positioning and the things that are important for each. And maybe Open AI as an example, maybe the story here is just what a great brand they built and that they have so much distribution and they have all these great partnerships

Starting point is 00:26:08 and people know it and use it and they have lots of people paying them 20 bucks or whatever. Maybe the distribution is more important than the product in the model. I'm curious what you think about these three players that have so far dominated but seem through this analysis so far like it's important that they keep innovating. So I think the interesting part for Open AI was because they just raised the recent round and there was some fairly public commentary around what the investment case was. You're right. a lot of it oriented around the idea that they had escape velocity on the consumer side,

Starting point is 00:26:44 and that chat GPT was now the cognitive referent, and that over time they would be able to aggregate an enormous consumer demand side and charge appropriately for that, and that it was much less a play on the enterprise API and application building. And that's super interesting. If you actually play out what we've talked about, when you look at their financials, if you take out training runs, if you take out the need for this massive upfront expenditure, this actually becomes a wildly profitable company quite quickly in their projections. And so in a sense, it could be better. Now, then the question becomes what's the defensibility of a company that is no longer step function advancing on the frontier?

Starting point is 00:27:37 And there, I think this is ultimately going to come down to one. Google is also advancing on the frontier, and they most likely will give the product away for free. And meta, I think we could probably spend an entire episode just talking about meta and the embedded optionality that they have on both the enterprise side and the consumer side. but let's stick to the consumer side. This is a business that has over three billion consumer touch points. They are clearly rolling meta-AI out into various surfaces. It is not very difficult to see them building a search functionality. I've joked they should buy perplexity,

Starting point is 00:28:18 but you've also just had the DOJ come out and say that Google should be forced to license their search index. I can think of no bigger beneficiary in the world than MetaG. having the opportunity at marginal cost to take on Google search index. But the point is that I think there will be two very large-scale internet players giving away what essentially looks like ChatGPT for free. So it will be a fascinating case study in can this product that has dominant consumer mind share.

Starting point is 00:28:52 My children know what ChatGPT is. They have no idea what Claudez. My family knows what chat GPT is. They have no idea what Brock is. So I think for OpenAI, the question is, can you outrun free? And if you can, and training becomes less of an expense, this is going to be a really profitable company really quickly. If you go to Anthropic, I think they have an interesting dilemma, which is people think Sonnet 3.5 is possibly the best model out there. They have incredible technical talent.

Starting point is 00:29:28 They keep ingesting more and more of opening eyes researchers, and I think they're going to build great models. But they're kind of stuck. They don't have the consumer mind share. And on the enterprise side, I think that Lama is going to make things very difficult for the frontier model builders to try to grab great value creation there. So they're stuck in the middle.

Starting point is 00:29:54 Wonderful technologists, great product, but not really a viable strategy. And you see they raise another $4 billion. To me, that's vindicative that free training is not scaling so well because $4 billion is not anywhere close to what they're going to need if the scaling vector is pre-training. I don't have a good sense for what their strategic path forward is. I think they're stuck in the middle.

Starting point is 00:30:21 XAI, I will plead ignorance on that one. He is a one-of-a-kind talent, and they're going to have a 200,000 chip cluster, and they have a consumer touchpoint. They're building an API. But I think if pre-training is the scaling vector, they're up against the same math problem that everyone else has, only possibly mitigated by Elon's unique ability to raise capital. But again, the numbers get so big so quickly in the next four or five years that that may even be greater than him. And then if it's test time compute and algorithmic improvements and reasoning, what is their differentiation? What is their go-to-market when you have people who have staked their claim on the consumer side? And then you have an open-source

Starting point is 00:31:16 entity on the enterprise side that's every bit as formidable. So when you look at the those three, I think it's easiest to see what Open AI's path forward is. One thing I will say about Open AI, though, is Noam Brown, who I find to be one of the most effective communicators in the research world, he was on Sequoia's podcast recently, and he was asked about AGI. And he said, look, I think when I was outside of Open AI, I was skeptical of the whole AGI thing, that that was actually what mattered to them. And when I got inside of OpenAI, it was very clear to me that they are very serious about AGI and that that is their mission and everything else is in service of AGI. It's easy for us to sit on the outside and articulate the strategy that we might pursue if we were

Starting point is 00:32:11 in charge there. But I think we need to be cognizant of the fact that part of the reason they got into where they are today is because they are on a mission. That mission is to develop AGI, and we should be very humble about astriving any other end gain for them than that. And my personal belief is that AGI is very close by. Say more. And why is it not already here? These things are smarter than almost everyone I deal with. Yeah, I think.

Starting point is 00:32:43 So AGI as narrowly defined or maybe expansively defined depending on your viewpoint, is a highly autonomous system that surpasses human performance in economically valuable work. In some cases, it's very easy to argue AGI is here using that lens. I think what is pretty clear is that if you look at the announcements made by OpenAI and their execs that have given interviews in recent weeks, an example that's bought up is end-to-end travel booking as something where that's something we can expect to see in 2025, where you can prompt the system to book travel for you and it'll just go do it. And that is a new way of thinking, which is end-to-end task completion or end-to-end work completion.

Starting point is 00:33:34 That involves obviously reasoning, that involves agentic work, that involves using computers, as Cloud has come out with. And you're combining multiple ways of these large, like, language models interacting with the ecosystem itself, putting into a very nice package that then is just able to do the end-to-end work and fully automate it and do it better than humans. And in my view, from that lens, we're very, very close to it. And I imagine that we'll be pretty close to or at AGI in 2025. I don't see how, given the current progress and the current innovation, and now moving to test time compute and reasoning, AGI is not around the corner with that lens.

Starting point is 00:34:25 And it's funny because we sort of become the frog boiling in water where we passed the Turing test pretty easily. And yet nobody sits here anymore and talks about, holy crap, we passed the Turing test. It just came and went. And so it could be that this declaration of AGI is something along the same lines where it's like, yeah, of course the model can look end-to-end travel. that's not actually that difficult. Whereas two and a half years ago, if you had said, hey, there's an algorithm that you can tell them what you want to do,

Starting point is 00:34:57 it books it end to end and sends you a receipt, you would say no fucking way. So there may be some of this boiling frog to it where all of a sudden you wake up one day and a lab says, hey, we've got AGI, and everyone's sort of like, oh, cool. There is one particular reason, though,

Starting point is 00:35:15 that lab declaring AGI is interesting. in a broader sense, which obviously is the relationship with Microsoft. And Microsoft first disclosed last summer that they have the full rights to the IP of OpenAI up until AGI is achieved. And so if OpenAI elects to declare that AGI is achieved, I think then you have a very interesting dynamic between them and Microsoft, which will compound an already very interesting interesting dynamic, which is at play right now. So that's something to watch next year, certainly for public market investors, but also for the ramifications of the broader ecosystem.

Starting point is 00:35:59 Because I do think, again, if we're right about the path that we're pursuing now, there will be a lot of reshuffling of relationships and business partnerships as we go forward. Jathan, was there anything else in modest assessment of the big players and we'd love to hear your thoughts on Google since we didn't talk about them as specifically? anything that he said that you disagree with or would press further on? No, I think what we just don't know is we don't know the underlying discussions in all of these rooms. And we can speculate and understand what we might do. But I think ultimately every internet business or technology business ultimately has come down to either on the consumer side distribution,

Starting point is 00:36:43 then combines with some kind of network effect and lock in effect, and then you're able to just run away with that. separate from the field. And then on enterprise, it's largely been a business that's driven by technology differentiation and delivery of that technology with great SLAs, with great service, with very unique approaches to solution delivery. And so modest comments on consumer and how consumer is going to evolve, I think is exactly right. You have meta, Google, and XAI with consumer touchpoints, you have OpenAI with an extraordinary brand today with ChatGBT, GBT, and a ton of consumer touchpoints already. On the enterprise side, the challenge has been that these APIs have largely to date not been as reliable as what developers expect. Developers have

Starting point is 00:37:35 gotten used to, because of the excellent work of hyperscalers, that if you are out there with APIs for a product, that product should be infinitely scalable. available 24-7, and the only reason the API ever goes down is because some giant data center lost power or something. There's very few reasons why an API should fail has become the developer mindset to enterprise solutions. And over the last two years, the quality of AI APIs has been a huge challenge

Starting point is 00:38:08 for application developers. And so what's happened as a result is people have figured out workarounds and have solved all those problems with pure innovation, But going forward, in this, again, we keep going back to this if, if pre-training and scaling is not the way to do it, and it's all about test time compute, this is where, again, we go back to the traditional way of hyperscalers. And I think this is where AWS is extraordinarily advantaged because Azure and Google have great clouds, but AWS has the biggest cloud. It has really built for resilience in a way that's very, very differentiated. And even today, if you're running Lama models, you want to run Lama models on AWS, or if for some reason you have some very specific use case and you need to support on-prem customers,

Starting point is 00:39:03 you can at very large financial institutions that have complex regulatory environments or compliance reasons. You can run these models on-prem if you choose to. and AWS has even gone there with VPCs and dump cloud and all this kind of stuff. And so if we assume that retraining and scaling there is done, then all of a sudden, AWS becomes extraordinarily powerful and their strategy here to just be friends with everybody in the developer ecosystem over the last couple years and not pursue their own LLM efforts. Well, they are pursuing, but not sort of like in the same way that others have.

Starting point is 00:39:43 will likely end up becoming a pretty good strategy because all of a sudden you have the best API service. The other part I think is Google, which we haven't talked about yet, is their cloud is very good at certain things. So they have an enterprise business. That enterprise business is actually pretty scaled now, if you look at the latest earnings. And obviously their consumer business is dominant. and there has been a perception that they're getting disrupted today. I think these forces are very disruptive to them, but it's unclear that the disruption has already happened.

Starting point is 00:40:24 What are they doing about it? Obviously, they're trying, and it's pretty clear that they're trying very hard, but I think it's an interesting one to watch, and the one that I like to watch, because it's the classic innovative dilemma, and they're clearly trying to be on the good side of not being innovated away as an incumbent. They're trying very hard. And so there's very few cases in business history of the incumbent preventing the innovators attack.

Starting point is 00:40:52 And if they do defend their business through this era, that'll be an extraordinary achievement. Yeah, Google is so fascinating because you had a brilliant cell site analyst, Carlos Kertner, who unfortunately passed away, but in 2015 and 16, he spent many, many reports writing about Google's progress towards artificial intelligence and the underlying work that they were doing at deep mind actually was so fond of it he went to go work at Google ultimately, but sort of first exposed this idea of the underlying work that they were doing there in neural nets, in deep learning. And it's clear they were caught off guard by the brute force scaling of the transform that what advanced this wave of technology was literally throwing computer.

Starting point is 00:41:47 But if you read any of the interviews with people who foreshadowed this data wall, one of the things they talked about was that self-play might be a mode to overcome the lack of data. and who is better at self-play than DeepMind. And if you look at the pieces that DeepMind brings from before the transformer and what they bring together with transformer and scaling of compute, it seems as though they have all the pieces to win. Now, the question I have always had is not can Google win at AI. It's is winning whatever that looks like ever going to pop.

Starting point is 00:42:30 possibly replicate how good winning was in the current paradigm. That's really the question. To Chatham's point, it would be amazing if they overcome the dilemma and win, but I think they have the pieces there. The question really is if they can build a business out of the assets that they have, that in any way looks as good as what is arguably the greatest business model we have ever seen, which was internet search. So I'm equally as fascinated to follow them. I think on the enterprise side, they have incredible models and incredible assets. I think they have a lot of trust to earn. I think that over time, they've come and gone in that world. And so I think that's a harder axis of attack for them. But certainly on the consumer side, and certainly in the model

Starting point is 00:43:19 building side, they have all the assets in place to win. The question is just, what does that prize look like? Particularly now if it doesn't look like there may be one or two models to rule them all. Chaitan, I'm curious, as an investor seeking a return, what path you personally hope for? I personally hope for AI to continue for a really long time. You need big disruptions as a venture investor to unlock distribution. And if you just look at what happened in the internet or in the mobile and where value accrued, value predominantly accrued at the application layer, in those two waves. Now, obviously, our hypothesis and my hypothesis was that this layer, again, was going to be very receptive to distribution unlocked because of innovation at the AI application layer.

Starting point is 00:44:13 I think that's largely played out so far. It's still early days, but the application vendors that have come out with production AI applications for both consumer and for enterprise have found that those solutions, which can now only exist because of AI, are unlocking distribution in ways that was frankly not possible in the world of SaaS or prosumer SaaS or whatever. We'll give you a very specific example. With an AI powered application, we're now going to CIOs at Fortune 500 companies showing these demos. And two years ago, there were really nice demos. Today, it's a really nice demo combined with five customer references of peers that using it in production and experiencing great success.

Starting point is 00:44:59 And what becomes very clear in that conversation is that what we're presenting is not a 5% improvement over an existing SaaS solution. It's about we can eliminate significant amounts of software spend and human capital spend and move this to this AI solution. And your 10x traditional ROI definition of software is easily justified and people get it within 30 minutes. And so you're starting to see these, what used to be a very long sales cycle for SaaS in AI applications, it's 15 minutes to a yes, 30 minutes to a yes. And then the procurement process for an enterprise completely changed. It's now the CIO says something like, let's put this in

Starting point is 00:45:45 as quickly as possible, we're going to run a 30-day pilot. The minute that's successful, we're signing a contract and we're deploying right away. These are things like three, four years ago, in SaaS was just completely out of the realm of possibility because you were competing against the incumbents, you were competing against their distribution advantage, their service advantage, and all this kind of stuff. And it was very hard to prove why your particular product was unique. And so since 2022, and I'll call it since chaty PT, November 2020, that seems like a really good line of pre and post in this world. We've made 25 investments in AI companies. And for a $500 million fund with five partners, that's an extraordinary pace. The last

Starting point is 00:46:30 time we had that kind of a pace was surprised when the app store came out in 2009. And then the pace that we had that kind of pace was again in 95, 96 with the internet. And in between those, you see as in our pace being pretty slow. We average around maybe five to seven investments a year in non-disruptive times, and clearly now our pace is dramatically increased. And if you just look at, of those 25 companies, four have been infrastructure companies and the rest have been application companies. And we just invested in our first model company, which hasn't been announced yet, but it's two people, two extraordinary brilliant people that are jumping to the frontier with very little capital. And so we've clearly bet and anticipated there's dramatic innovation,

Starting point is 00:47:21 and distribution unlock happening at the application layer, we've seen that happen already. These products are truly, as a software investor, are absolutely amazing. They require a total rethinking from first principles on how these things are architected. You need unified data layers, you need new infrastructure, you need new UI, and all this kind of stuff. And it's clear that the startups are significantly advantaged

Starting point is 00:47:47 against incumbent software vendors. And it's not that the incumbent software vendors are standing still. It's just that innovator's dilemma in enterprise software is playing out much more aggressively in front of rise today than it is in consumer. I think in consumer, the consumer players recognize it or moving it and are doing stuff about it. Whereas I think in enterprise, it's just even if you recognize it, even if you have the desire to do something, the solutions are just not built in a way that is,

Starting point is 00:48:19 responsive to dramatic re-architecture. Now, could we see this happening? Could a giant SaaS company just pause selling for two years and completely re-architect their application stack? Sure, but I just don't see that happening. And so if you just look at any sort of analysis on what's happening on AI software spend, something like it's 8x year-over-year growth between 2023 and 2024

Starting point is 00:48:46 on just pure spend. It's gone from a couple of hundred million dollars to well over a billion in just a year's time. And you can see this pull. You can feel this pull if you're in any one of these AI application companies. It's like more of these companies are supply constraint than the mine constraint. We talk to the CEOs of these application companies and they just say things like, well, I see demand as far as I can look out. I just don't have the capacity to go service all the people that are saying yes to me. So I'm going to segment it and go to where they are.

Starting point is 00:49:20 And my hope as an investor is that it continues to play out this way and that we have stability to just pursue these angles. And frankly, the model layer stabilizing is a huge boon for this application layer. Primarily because as an application developer, you were sitting there watching the model layer take step function leaps every year. And you kind of didn't know what to build and what you should just wait on building. because obviously you want to be completely aligned with a model layer. Because the model layers are now moving to reasoning,

Starting point is 00:49:54 this is a great place for an application developer. One thing you know as an application developer is humans are not patient. And so you need to always build solutions that optimize on performance and quality. You cannot go to a user as an application developer and say, like, I'm going to deliver a high quality response. It's just going to take longer. That's not been a winning argument. Now, for certain use cases, is that possible?

Starting point is 00:50:20 Could you have it run in the background for 24 hours? Absolutely. But those use cases are not widespread and predominant. And people aren't going to be willing to buy that kind of stuff. And so if, as an application developer today, all of my board meetings over the last couple weeks have been these companies saying, in this new reasoning paradigm, we're really confident that we can invest in these four things that we've been super, super, hesitant to in the last year and a half. But now we're going to go all in on these bets and the

Starting point is 00:50:50 kind of performance gains you're going to see out of our system is going to be huge. Sorry, why is that the case? Why does the reasoning thing make it so that their confidence goes up? Just like spell that out? Well, if you are an application developer and you're looking at the models today and you're saying, I can see clear efficiencies for my use case, but I have to invest in these five infrastructure layer things and these UI things. But if a new model comes out in six months and blows all that investment away just because the model itself can do it, then why would I ever invest in these things? I'm just going to wait for the model to do it and then bet on that. But in this reasoning paradigm, if all the labs pursue reasoning, and reasoning is

Starting point is 00:51:32 intelligence on the y-axis, time on the x-axis, and that's where we're going, then any improvement that I can make in my own tool to make either that reasoning time dramatically compressed because of the way algorithmically I'm feeding reasoning and I'm able to take the data and manipulate it and all that kind of stuff, I should invest in it now if reasoning is now the new paradigm. And the last mile delivery at the application layer against these reasoning models means that I'm building technology and tooling that model companies are very, very unlikely to build. And as those reasoning systems, continue to get better, my last mileage and last mile delivery systems are still advantaged and

Starting point is 00:52:13 defensible. Do you both have favorite examples of this beyond coding and customer service, which seem to be the two dominant and incredibly exciting and cool use cases with lots of companies chasing after versions of that? Do you have other favorite examples that, you know, would fit the CIO of the Fortune, whatever company saying like we need this in our company now? Jason loves all his children, so he's not going to be able to give you specific examples. I can give you 20 of all. Maybe like categorically is my question. There's coding, there's support.

Starting point is 00:52:46 Essentially, top down, look at the biggest bends of enterprise software, and you could attack that with an AI-powered AI-first solution. And so we've got a great company called 11X that's going after sales automation. We've got a great company called Leia that's being used by lawyers to dramatically increase the efficiency of their. work. I think legal has been a very interesting question because people assume that lawyers work on billable hours. If you're automating billable hours, aren't their economics going to change? Well, now the evidence, two years into this, is that actually lawyers end up becoming way more

Starting point is 00:53:21 profitable by using AI. And the reason is, is that a lot of the work that was wrote repetitive and hard and done by junior people inside of law firms, law firms weren't able to bill for that stuff anyway. And so if you can take down the time to do document analysis from three or four days to 24 hours, all of a sudden, you free up all your lawyers to do all the strategic work that they can bill for and stuff that is extremely valuable for clients. We've got a company that's automating accounting as an example on financial modeling. We've got a company that's changing how game development is working. We've got somebody that's going after circuit board design, which has been a usually manual and human-intensive thing, and computer systems are particularly really good at.

Starting point is 00:54:06 And we recently invested in something going after an ad network. Now, that's been something that's not been touched for a long time from startups. But it turns out matching the people that have inventory with the people that want to do advertising in the AI world is just way more efficient. And so we invested in a company that's got a new document processing model. And And they're going after open text. When was the last time a startup thought about open text? It's been a long time where these huge incumbent SaaS markets were thought to be open to new startup. So you had to pursue more niche, more vertical.

Starting point is 00:54:49 And I often joke because, like, I saw this. It was payroll for field workers working in Eastern Europe was a SaaS company that you, like, legitimately had to think about in 2019. And now we're back to like large swats of horizontal spend again to say like, hey, there's an incumbent here that's worth $10 billion plus. The market here is $10 billion of annual spend. AI makes a product here easily 10x better, faster, and all the things that users want when they see it. And you need a new platform to come out with that kind of advantage. And that's what this is. And Patrick, you asked back at the beginning about the big deal.

Starting point is 00:55:30 debate on RLI and CAPEX and all this. And when you listen to Chip and when you listen to other investors in the application layer, when you listen to the hyperscalers, the big takeaway over the last three months is the use cases are coming. Yes, everybody knows about coding. Everybody knows about customer support. But this is really starting to permeate and get out into the broader ecosystem and the revenues are becoming real.

Starting point is 00:55:56 The challenge on the RLI question always was, okay, you put the capital in here, you then amortize it over the inference period. But meanwhile, you're then stacking the next quantum of capital for the next model. And so everybody could draw those extrapolations and say, oh, my God, it's not just that Microsoft is going to spend $85 billion in cash cap X, inclusive of the leases in 2025. It's what does it mean for 26, 27, 28, because the pre-trained models were getting so big. If, and again, it's an if, we are plateauing and we're spending less money on pre-training and moving that capital towards inferencing, we know that spend is coming. We know the revenue generation of the customer is coming. And so it becomes much more easy to say, this spend is warranted.

Starting point is 00:56:54 I think it is important that people remember the underlying clouds of these companies, meaning just the normal storage and compute, are still growing high teens to low 20s. So there's some capital that needs to be allocated towards that when you're a $100 billion business growing 18%. You're a $60 billion business growing 25%. It's the incremental capital above that, everybody was very concerned about six, nine months ago. My personal takeaway coming out of Q3 was, okay, I see it. There's use cases here.

Starting point is 00:57:28 The inferencing's happening. Technology is doing what it's supposed to. The cost of inferencing is plummeting. The utilization is soaring. You put that together. You get a nice rising pot of revenue and everything's good. Sachin Adela talked about this. The challenge is you spend the money for the model, you get it on inferenceing, but then we're

Starting point is 00:57:48 spending on the next model. If we can start to say, hey, maybe we're not going to spend the next $50 billion on the model, the ROI calculation looks a lot better. And something that you asked Chathen was, why is stability of the model layer important? I think Sam Altman gave the right answer on this, which was six months ago, he was on a podcast and said,

Starting point is 00:58:10 if you're scared of our next model being released, we're going to run you over. If you're looking forward to our next model coming out, then you're in a good position. Well, if the actual reality is the next model is going to be at inference time and not pre-training, you probably have less worry about them steamrolling. So I think everything that we're talking about in this one path is very conducive to a favorable economic reality for the entire ecosystem, which is all the attention capital being put

Starting point is 00:58:46 towards infancy. The real concern was, do we know? need to spend $50, $200, $200 billion to build these ever more accurate models in pre-trained. Where do prices most reflect extreme optimism or hype still? I've certainly seen my fair share of private markets companies, let's say Series A type companies, that price at extremely high valuations. They're often incredible teams and very exciting, but they're also playing in spaces where if something works, you could imagine lots of other very smart investors,

Starting point is 00:59:18 funding some competitors. So you see these scenarios where it's like great team, high price, high potential competition, really exciting, everything's moving fast. I'm curious what signals you both read from valuations and or multiples right now. In the private markets, one of the things that's happening is just the dramatic drop in prices of just compute, whether it's inference or training or whatever, because it's just becoming way more available. If you're sitting here today as an application developer versus two years ago, the cost of inference of these models is down 100x, 200x. It's frankly outrageous. You've never

Starting point is 01:00:00 seen cost curves that look this steep, that fast. And this is coming off of 15 years of cloud cost curves, which were amazing and mind-blowing by themselves. The cost curves on AI are just a completely different level. We were looking at cost curves. And the first wave of application companies that we funded in 2022, you look at the inference costs and it would be like $15 to $20 per million tokens on the latest frontier models. And today, most companies don't even think about inference costs because it's just like, well, we've broken this task up and then we're using these small models for these tasks that are pretty basic. And then we're like, the stuff we're hitting with the most frontier models are these like very few.

Starting point is 01:00:45 prompts and the rest of the stuff, we've just created this intelligent routing system. And so our cost of inference is essentially zero and our gross margin for this task is 95%. You just look at that and you're just like, wow, that is a totally different way to think about application gross margins than what we've had to do with SaaS and what we've had to do with basically software for the last decade plus. And so I think that's where you're starting to at and saying the entire application stack for these new AI applications, and it starts with people that provide inference. It starts with the tooling and the orchestration layer. So we have a portfolio company that's extremely popular called Link Chain and the inference layer we have fireworks.

Starting point is 01:01:31 These kinds of companies are seeing extraordinary usage by developers. And then all the way up the stack to the applications themselves, I think just the pace of innovation, pace of commercial success is driving a lot of excitement with, private investors, what is also appealing of model stability is now we can finally assume if this sticks that all these companies are going to be fairly capital light. Because if you're not having to spend a lot on pre-training, if you're not going to have to spend a lot on inferencing, because most of the hyperscalers are now going to present you with really reliable APIs at these kinds of costs, it's a great time to be in the application development business.

Starting point is 01:02:10 And it's a great time to be in the application development stack. Modest, what do you think on valuations? I think you have to start in general with animal spirits. If you go back to the week before chat GPT was released, if you go to the fall of 2022, that can probably just suffered its most brutal bare market since the dot-com collapse. It was arguably worse for the median tech staff than even the financial crisis. You had some of the very large growth funds down 60s,

Starting point is 01:02:45 70%, you had the hyperslayers laying off people for the first time ever. You had CAPX cuts. You had OPX cuts. It was a very different vibe in the tech world. And in the public markets at large. The release of chat GPT catalyzed the reemergence of animal spirits. And it's been a progressive process. But I think where you are today, you have the public markets trading at 24 times earnings. And again, this goes beyond just the Mag 7 at this point. I mean, Google trades it, I think, 19 or 20 times. So they're not one of the offenders here.

Starting point is 01:03:27 And so I think in general, there is a lot of optimism baked into the public markets, a lot of which is tied thematically to this idea that were in a new platform era and that the sky is the limit for a lot of various new, concepts. So there's that global overhang. If we are right, I think that what it really comes down to is understanding what does this new path forward look like. If CAPX and hyperscalor op-x is more closely tied to revenue generation, if you listen to AWS, one of the fascinating things they say is they call AWS logistics business. I don't think anyone externally would sort of look at cloud computing and say, oh, yeah, that's a logistics business.

Starting point is 01:04:19 But their point is, essentially what they have to do is they have to forecast demand and they have to build supply on a multi-year basis to accommodate it. And over 20 years, they've gotten extraordinarily good at. What has happened in the last two years, and I talked about this last time, is you have had an enormous surge in demand hitting an elastic supply because you can't build data center capacity in three weeks. And so if you get back to a more predictable cadence of demand, where they can look at it and say, okay, we know now where the revenue generation is coming from. It's coming from test time. It's coming from Chathin and his company's rolling out.

Starting point is 01:05:03 Now we know how to align supply with that. Now it's back to a logistics business. Now it's not grabbed every mothballed nuclear site in the country. and try to bring it online. And so instead of this land grab, I think you get a more reasonable, sensible, methodical rollout of this. It may be, and I actually would guess,

Starting point is 01:05:27 that if this path is right, that inference overtakes training much faster than we thought and gets much bigger than we may have suspected. But I think the path there in the network design is going to look very different, and it's going to have very big ramifications for the people who were building the network, who were powering the network, who were sending the optical signals through the network. And all of that, I think, has not really started to come up in the probability weighted distributions

Starting point is 01:06:02 of a huge chunk of the public market. And look, I think most people overly fix it on Nvidia because they are sort of the poster child of this. But there are a lot of people downstream from Nvidia that will probably suffer more because they have inferior businesses. Invidia is a wonderful business doing wonderful things. They just happen to have seen the largest surge in surplus. I think that there are ramifications far, far beyond who is making the bleeding edge GPU, even though I do

Starting point is 01:06:37 think there will be questions about, okay, does this new paradigm of test time compute allow for customization at the chip level much more than it would have if we were only scaling on pre-trade. But I think this question, whenever I have this in normal conversations, people overly fixate on Nvidia. I think people like to debate that particular name. But I think there's a lot of other derivative plays of the AI buildout where the distribution of outcomes have shifted, and that has not been reflected yet. I just think it's really important to think about

Starting point is 01:07:16 in the test time and the reasoning paradigm from an application layer, how many of your prompts actually utilize reasoning as a way to respond to those prompts? And yes, application developers, as this technology becomes more available and usable, will use way more of it than they are today, But if you just look at the current techniques and the wows you're getting from the application layer already,

Starting point is 01:07:44 what percent of prompts or what percent of queries are going to use reasoning? It's very hard to squint and say it's going to be 90 percent of queries. That doesn't seem like it's going to go that way because, again, your users are not going to wait. Humans are inherently impatient and you have a solution that's like just spinning and thinking your users are gone. It doesn't matter what sector they're in. they're just gone. And so, yeah, you can have a certain set of tasks that take a long time and deliver great accuracy, but speed is by far the most important consideration for these application developers.

Starting point is 01:08:17 And so are we just going to have a system that continues to just go back and through and back and through and utilize all this compute in what market share of queries use that? It's hard to imagine that being super majority of queries. And so then the implication, at least from a private market, early stage investor, which take huge grain of salt on what it means for anything other than my world. But the implication there is simply that you just don't need as much compute as you did with training. Training is just a constant exercise. You're scaling and you're just really hitting all your compute power all the time and is going. At the application layer,

Starting point is 01:09:02 it's extraordinarily bursty. You're going to have certain tasks that need a lot right away. and for a lot of it, you just don't need a lot. And so this is where, again, like hyperscalers and things like EC2 and S3 were incredible. And now in this new world, the solutions from hyperscalers are really terrific. I think AWS's training them and the TPUs from Google are really, really terrific. And they offer a great developer experience. I think part of what has been known for application developers is that GPUs are really tough to use for this use case. getting max utilization out of GPUs, chain together,

Starting point is 01:09:41 whether you're buying that from Dell or whether you're buying it from a hypers, it's just really hard to use. But with new software innovations, that's obviously going to get better. And then the stuff that's coming out from the hypers themselves, they're really, really terrific. And you just don't need to hit them as hard as you did when you were doing training when you're doing test time compute. I think it's a really important point in the utilization of the GPUs.

Starting point is 01:10:04 if you think about a training exercise, you're trying to utilize them at the highest possible percent to a long period of time. So you're trying to put 50,000 chips in a single location and utilize them at the highest rate possible for nine months. What's left behind is a 100,000 chip cluster that if you were to repurpose for inferencing is arguably not the most efficient bill, because inference is peaky and bursty and not consistent. And so this is what I'm talking about that I just think from first principles, you are going to rethink how you want to build your infrastructure to service a much more inference focused world than a training focused world.

Starting point is 01:10:58 And Jensen has talked about the beauty of Nvidia is that you lead. behind this in-place infrastructure that can then be utilized. And in a sunk cost world, you say, sure, of course, if I'm forced to build a million chip super cluster in order to train a $50 billion model, I might as well sweat the asset when I'm done. But from first principles, it seems clear you would never build a 350,000 chip cluster with two and a half gigawatts of power in order to service the type of requests that Chathen's talking about. And so if you end up with much more edge computing with low latency and high efficiency, what does that mean for optical networking? What does that mean for the grid? What does that mean for the need for on-site power versus the ability

Starting point is 01:11:53 to draw from the local utility? I think these are the types of questions I would be very interested to read about. But to date, a lot of the analysis is still focusing on what's going to happen when we lay up three mile island, because the new paradigm is really too soon to shake. Do you think that we still need and will see, though, tons of innovation in the semiconductor world and layer, whether it's networking, whether it's optical, whether it's chips themselves, different kind of chips? I would imagine this would accelerate. rate it even more because it was very difficult to foresee a world where you took on Big Green in train. The way I think about this over hundreds of years is you have a gold rush, a land grab, and everybody's just doing whatever they can.

Starting point is 01:12:46 But in technology, then as some stability sets in, you get an optimization period. You've already had that on the inferencing side. It's what Chathen referenced is people had time to optimize the underlying algorithms in compute and inference has fallen 99%. It's the same thing that happened with Internet Transit at the end of the bubble, which was, people said, no, you can never stream a movie. Do you have any idea how much that would cost? And the cost of transit has fallen 25% a year, like clockwork for 20 years.

Starting point is 01:13:22 The literal profit pool of that business is static. for 20 years. And so I think we've had this mannick demand search. And I think that if we get a little bit of stability and everyone can take a breath, there will be the two guys in the garage optimizing every single thing possible. And that's the beauty of technology over the long term is it is deflationary because it's an optimization problem. But you don't have time to optimize when you're in land grab. mode. I quoted this to you last time. The data center industry, they were power neutral. There was no demand growth in power for the entire data center business for five years. That was because you were in the fully mature stage of the cloud data center buildup. I don't know when you'll reach that point.

Starting point is 01:14:12 I mean, we know that these guys on a three, four, you, through at least 26 or 27 are pot committed to their buildout. When in that path will everyone have time to take a deep breath and say, okay, now let's figure out how to run these more efficiently. That's just the nature of things. Same thing on the compute side. I just think we haven't yet gotten to the point where technologists have been able to apply their optimization. They've been in the implementation. And I'll give you a couple of data points from my end. So my partner, Eric, is on the board of a great semiconductor company called Cerebrus, and they recently announced that inference on Lama 3.1, $405 billion for Cerebris is it can generate a great semiconductor company called Cerebris. And they recently announced that influence,

Starting point is 01:14:55 can generate 900 plus tokens per second, which is a dramatic order of magnitude increase. I think it's like 70 or 75 times faster than GPUs for inference, as an example. And so as we move to the inference world, the semiconductor layer, the networking layer, etc., there's tons of opportunities for startups to really differentiate themselves. And then the second thing I would bring up is I was just recently talking to the CIO of a large financial services institution. who said that over the last two years, they were re-buying a lot of GPUs because they assumed that they were going to have lots of AI workloads and who knows if maybe they needed to do some training themselves.

Starting point is 01:15:36 And so those systems are now being installed into their data centers and they're now online. And we're in this world where you don't need to create your own model. And even if you did, you just fine-tune an open-source model. It's not that heavy. And so his view is like, look, if you have AI applications that run on-prem, like it's essentially free. I have all this capacity.

Starting point is 01:15:57 I'm not using it for anything. Inference is light. And so I have at the moment infinite capacity to run AI applications on-prem and it'll cost me zero marginal dollars because all the stuff is up and running and I'm not using it for anything. So I'm ready to buy.

Starting point is 01:16:15 So not only are all these application things that you're talking about, hugely exciting because they unlock RR and all the stuff, but the minute you can run any of this stuff on-prem on our stuff that dramatically decreases the cost for us. And so it's just like win-win-win-win all over the place when you have something like that. And that's the current state of play. Now, how long does this over-capacity last?

Starting point is 01:16:36 Application developers are famous for using all the capacity and pushing the limits. And all of a sudden, what used to be over-capacity ends up becoming under-capacity because all of a sudden we have all this breadbend build-out and we decide to stream video on it. And so, of course, AI applications are going to get more sophisticated and swallow up all this capacity. But that is just a much more predictable world and much more sane world for an investment perspective than scaling to infinity on pre-training. The one thing I am curious to monitor is it's important to remember that the reporting was not that the models weren't getting better. It's that the models weren't getting better relative to expectation for the amount of compute a

Starting point is 01:17:22 apply to that. So I think we do need to be cautious to conclude that the labs are not going to keep trying to figure out the unlock on the pre-training side. I think the question there is, one, what should we be looking for? But then two is, if they continue to push on that vector, do we believe, and this was the question I always wrestled with, was if scaling laws held in pre-training, Would people be willing to spend $100 billion? And I know that everybody says if you're playing for the ultimate prize, you would. But has enough doubt been cats that simply brute forcing pre-training is the path to that ultimate unlock? Or is it now some combination of pre-training, post-training, and test time compute?

Starting point is 01:18:18 in which case, again, I think that the world is just the map is much more sane. And I've seen a lot of notes coming out saying people are declaring the end of AI progress and all that. And hopefully the takeaway from today is none of that is what I think people really in the weeds looking at this are saying. People are saying AI is full speed ahead. I think the question is just what the axis of advancement is. And from my seat, the math seems much more sensible. Everything seems much more rational pursuing this path rather than the upfront cost being spend any amount you can to build this hypothetical God. So I think this is a much better outcome if this is the path that we end up going down.

Starting point is 01:19:10 I'm curious what you think, if anything, is the most under discussed part of this whole story. Are there things that you find yourself thinking a lot more about than you hear discussed from your friends and colleagues? On the public investor side, just reading cell side reports that we haven't seen sell side reports or analysis on what this new paradigm of test time compute means and how things change. And so I'm really looking forward to way more sell side analysis on this new paradigm shift. I think there's also little coverage in the private markets. I think it's known to people that are meeting these entrepreneurs is just how capital efficiently these entrepreneurs are getting to the frontier today. And this is just a shift that's happened very, very recently.

Starting point is 01:19:54 And you're seeing people just show up and having spent under a million dollars to match performance, not broadly, but in specific use cases with the frontier models. And that's just not something that we were seeing two years ago or even a year ago. And so I think that's pretty dramatically undercover. pre-training is a big test of capitalism. If we pursue down this path, I feel much better with a microeconomic background analyzing what's going to happen because you don't have to put in the NPB of God. And I just think that that's much better.

Starting point is 01:20:28 In terms of what I'm looking forward to reading and hearing, yeah, I'd love to see some thoughtful analysis really wrestle with. Right now, I feel like it's a little defensive. people are defending the fact that scaling's not done, it's just moved. So that's great. But let's now work through the second order effects, the third order effects, and how does this really manifest itself? I think it's very good for the overall ecosystem, the overall economy. But I think there's going to be a lot of surplus shifting from pockets that looked like winners

Starting point is 01:21:02 before and pockets that look like losers. What outcome in the next six months would most disorientation? Well, two dramatic examples. On the positive side, if somebody came out with the results that pre-training was back on and there was a huge breakthrough on synthetic data and all of a sudden it's go-go again and $10 billion and $100 billion cluster was back on the table, you would go back, but all of a sudden the paradigm shift would be wild. All of a sudden, we would now be talking about $100 billion super cluster that was going to pre-trained. And then obviously, if my expectation comes out that next year we're going to call AGI,

Starting point is 01:21:43 we're going to have AGI and we're building a $100 billion customer because we had a breakthrough on synthetic data and it all just works. And we can just simulate everything. That would be pretty dramatically disorienting. I think another scenario is it's pretty clear now that while we've exhausted data on text, we are not close to exhausting data on video and audio. And I think that it's still TBD on what these models are capable of on new forms of modes. And so we just don't know because the focus hasn't been there. But now you're starting to see large labs talk more about audio and video.

Starting point is 01:22:23 What these models will be capable of from a human interaction perspective, I think it's going to be pretty amazing. I think you've just seen already how much leaps have gone into image generation and video generation and what does that look like in a year's time and two years time could be pretty dramatically disorienting. Yeah, I think the hard part as a non-technologist is for the last year, year and a half, the question has been, what would GPT5 bring if it adhered to the scaling law? And no one could really articulate because all we know is, okay, training loss would be lower. So you'd say, okay, this thing's more accurate at next token prediction.

Starting point is 01:23:07 But as far as what does that actually mean from a capability stamp, what's the emergent capability we were unaware of before it was released? So I think it's really hard to know ex-ante, what you're looking for, other than the labs coming out and saying, this is so good in its accuracy that it warrants staying on this log-linear trajectory of space. And if someone comes out and says that, I think, irrespective of this entire conversation or what you may believe, you have to say, okay, that's happened. Again, I just think you have to have a super open mind. And if we were out of conversation three months ago, there were whispers, but it wasn't in the open.

Starting point is 01:23:51 I just think you have to be updating your priors constantly. So clearly, like Jason said, I'd be looking for that. Personally, I watch Lama closely. There's clearly a risk at some point that they decide not to keep. open sourcing. And if I were other players in the ecosystem, I would be doing my damnedest to make sure that Lama stays open. And there are certain ways you can go about doing that. But I think that's one thing because their willingness to spend at the frontier and make those models available the way they do, I think has completely changed the strategic dynamic in the model

Starting point is 01:24:32 industry. So that's another one that I would be paying attention to. I have a philosophical question as we near the end of the discussion, which is around ASI. So if AGI is here or coming next year, how the both of you would even think about, I guess it builds on that point about what do we even expect from a GPT5 that stays on the scaling law? What does it mean? Because there are fewer and fewer things, at least in a simple chat interaction, that I could imagine it doing a much, much better job on, even what that would look like. And again, we're probably just in the early innings of application development and fine-tuning and improvement and algorithmic updates and blah, blah, blah. So I'm curious just philosophically what you think the litmus test could or might be for something beyond what we have

Starting point is 01:25:17 naturally as the existing models get tweaked and tuned and better. What does the ASI even mean? Does it mean it solves previously impossible math or physics challenges or something else? What does that idea mean to you both? This is my words. I don't remember who originally said them, but humans are really good at changing the goalposts on expectations. And

Starting point is 01:25:41 AI in the 1970s meant something different than what it meant in 80s, in 90s, 2000s, and in 2024. And so, if a computer can do it, humans have a really good way of describing that as automation. And whatever a computer can't do,

Starting point is 01:25:57 that now becomes the new goalposts for AI. And so I think that these systems are already extraordinarily intelligent and are extraordinary at replicating human intelligence and sometimes exceeding human intelligence. I think if you just look at the path that some model developers like DeepMind and several startups are pursuing with things around math and physics and biology, it's very clear that there's going to be applications and outputs of these models that are going to be things that humans were simply not capable of doing before. We already have seen that in things like protein folding today. We're starting to see a little bit of that as it relates to math proofs. I'm confident we're

Starting point is 01:26:40 going to start seeing that as it relates to physics proofs. And so my optimistic hope for humanity is that, I don't know, we'll be able to open wormholes or something. We're going to be able to study general relativity at a scale that we haven't been able to before, or study backholes or simulate backcoles in a way that we haven't been able to. All of that sounds a little bit ridiculous at the moment, but certainly the way things are progressing and the way things have progressed, we don't know what is possible and what's not possible. And to bring it back to an investor point of view, when you have the unknown future where the possibility is up to your imagination, that's usually a really great time to be an early stage investor. Because that

Starting point is 01:27:19 means that technology has unlocked. And usually when technology unlocks in a grammatic fashion, distribution also then unlocks. And you can now go get customers. that were very expensive to get. And so previously, if you wanted to build a consumer application, you then had to factor in the tax of the app stores, search, ad networks, and all that kind of stuff. And all of a sudden, it just became a very quick exercise in unit economics. And similarly, in SaaS, it was like,

Starting point is 01:27:48 a-productivity, gross margin, and infrastructure costs, and you just tried to do a spreadsheet exercise. And early-stage investing started to become more spreadsheet-like than true technology innovation. I think when you have big breakthroughs like this, everything sort of changes again. Like, distribution is nearly free if you have something unique,

Starting point is 01:28:08 and there's like a word of mouth and virality factor to it. The technology spend is really, again, you go back to just investing in your developers and your research scientists, and the R&D and the R&D and the R&D starts to become remarkable again. That's what's most exciting as an early-stige investor is that we just don't know

Starting point is 01:28:27 what the feature holds, and therefore it's back to human ingenuity and people are able to push these boundaries. When it's exciting for a early stage investor, I think it's terrifying for a somewhat skeptical public market investor. Prices are based on vibes and not mass in the spreadsheet. With ASI, I think we've talked about this before,

Starting point is 01:28:48 this whole concept, the reason people spend so much time on it is because it is so profound ultimately, you have people who are invoking quasi-referral, religion in their view of what we're building. Anytime that comes into play, I think the stakes are just higher. It's kind of unknowable, and it's super complicated. So we all love to debate it. But I think the one thing we haven't touched on here, which is there's a pretty fervent belief amongst a group of people that there will be recursive self-improvement at some point in time.

Starting point is 01:29:21 and I think that that would be a big path to unlock in whatever hypothetically ASI means is when the machines are smart enough to learn themselves and teach themselves. On a less sort of dramatic view, the way I think about this, there's AlphaGo which famously did that move that no one had ever seen and I think it's like the 37. Everybody was super confused about and ended up winning. And another example I love is Noon Brown because I like poker talked about his poker. bot confused. It was playing high stakes no limit. And it continually overbett dramatically larger sizes than pros had ever seen before. And he thought the bot was making a mistake. And ultimately, it destabilized the pros so much. Think about that. A computer destabilized humans in their approach that they have to some extent taken on overbeating now into their game. And so those are two examples where

Starting point is 01:30:20 if we think about free training being bounded by the data set that we've given it, if we don't have synthetic data generation capabilities, here you have two examples where algorithms did something outside of the bounds of human knowledge. And that's what's always been confusing to me about this idea that LLMs on their own could get to superintelligences. Functionally, they're bounded by the amount of data, we give them up front. And so if you have examples like this where algorithms are able to get outside of what they're initially bounded by, that's super interesting. I'm not smart enough to know where that leads us.

Starting point is 01:31:04 But that's the kind of thing that I feel like is the next thing to come, is how do you escape the bounds of what you're given up front? I think what's remarkable from my perspective is how much of this innovation is happening in the United States and how much of it is happening in Silicon Valley. We've had a rough couple of years since the pandemic, and it's really amazing. There was an investor friend of mine who's not based in Silicon Valley, and he was just saying, I can't believe it's happening in Silicon Valley again. And it's just become this beacon where all of the labs were based here. A lot of the people that are working on these applications, these infrastructure companies,

Starting point is 01:31:44 etc. are here or even if they're not here, they're somehow connected to being here and are often visiting here a lot. And I would say that the focus on the innovation here is really extraordinary on AI. The progress made in the United States in Silicon Valley specifically is extraordinary. I do think that there's a level of attention that investors and entrepreneurs now have is how fragile the system is and how much we need to protect it and continue to invest in it. And I think there's a lot of focus now that innovation is something that needs to be protected. And I think a lot of people are now paying a lot of attention to make sure that all of this innovation that's happening in the United States continues to benefit everybody. And I think that's a really optimistic and

Starting point is 01:32:32 cool thing to recognize. The agglomeration of facts are real. If the reporting is right, the way the Transformer paper came to pass is that someone was rollerblading down the hall and heard two guys talking about something and went in and whiteboarded and two more people came by and who knows how much of that is a Bacher followed up. But it is fascinating to see from an economist standpoint that like these human network effects are real and that COVID did not destroy them and that work from home did not destroy them and that there really is something tangible to being together and the synthesis of ideas, multidisciplinary coming together

Starting point is 01:33:12 to build this world-changing architecture. Guys, it's always such a blast talking to you both. I'm lucky to get to do this in private. It's fun to do it in public. Thanks for your time. Of course. Thank you. If you enjoy this episode,

Starting point is 01:33:25 check out join colossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, ideas, quotations, and more, as well as share the best content we find on the internet every week.

Invest Like the Best with Patrick O'Shaughnessy - Chetan Puttagunta and Modest Proposal - Capital, Compute & AI Scaling - [Invest Like the Best, EP.400]

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.