The a16z Show - Chips, Memory, and Power | Pat Gelsinger

Episode Date: October 9, 2026

a16z's Raghu Raghuram and Guido Appenzeller sit down with Playground Global General Partner and former Intel CEO Pat Gelsinger to discuss the next wave of semiconductor innovation and the physical constraints shaping the AI buildout.Drawing on his experience designing Intel's 386 and 486 processors, Pat explains how AI could transform chip design, but also why faster design alone won't solve the industry's biggest problems. They examine the bottlenecks in manufacturing, memory bandwidth, advanced packaging, and power, and why today's explosion of specialized AI chips may eventually consolidate around a smaller number of architectures.They also discuss the potential for new memory technologies, the shift from copper to optical networking, and why energy capacity could become a major constraint on AI growth. Finally, they revisit Pat's VMware years to ask what virtualization might look like when infrastructure is built for agents rather than humans.Resources:Follow Pat Gelsinger on X:  https://x.com/PGelsingerFollow Raghu Raghuram on X: https://x.com/RaghuRaghuramFollow Guido Appenzeller: https://x.com/appenz Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

In an AI digital age, energy capacity equals economic capacity. Why build the new data center and buy a million GPUs if I can't power that? You're going to see more and more defaults happening on many of those data center projects because the energy won't be there. Whenever you have the technology to make something easy, that means the bottleneck moves somewhere out. Nothing's a chip anymore. It's a rack.

It took me three months to design it, but it's nine months until I can actually start to use it. Exactly how many major new memories have you had over the last 30 years? Zero. Memory innovation for the first time in 30 years is nigh upon us. I declared the death of copper about 25 years ago. Eventually, I'll be right for all of us hardware guys. It's like renaissance in front of the AI inference accelerator chips, and I'm sure I don't even know them all.

Why are you guys funding so many of those anyways? You've won't your fair share, too. Historically, there have not been 100 competing processor vendors in any industry. Is this a temporary thing? It'll convert back to a view. I see it as... AI is making it faster to design chips,

but actually building them, powering them, and connecting them may be getting harder. In this episode, A16 Z's Raghu Raghu Ram and Guido Abadzeller sit down with Pat Gelsinger, General Partner at Playground Global, and former CEO of Intel and VMware

to discuss the next bottlenecks in AI infrastructure. Pat takes us back to his early days designing Intel's processors, when engineers were inventing the tools they needed to build the chips themselves. He explains why AI could bring a similar shift today, even as manufacturing timelines, memory bandwidth and power become harder constraints. They also debate whether the explosion of specialized AI chips will last, why Pat believes memory innovation is finally approaching a breakthrough and how optical networking could change the architecture

of AI clusters. In drawing on their shared history at VMware, they close to the technology. with a question, what would it look like to rebuild virtualization for a world where AI agents rather than humans are the primary users? I'm here with our esteemed guest and dear friend and former boss, Mr. Pat Gelsinger. Welcome, Pat. Pat is currently the general partner at Playground Global, but as you're very, very well known in the industry for leading Intel, for being the CEO of Intel, of course, leading VMware and many other things.

So welcome. Hey, thank you, Ragu. Great to be with you and Guido. And to me, this feels just a little bit like old home, right? We've superimposed a year of change, right? But you both went the same. I feel energetic, so let's dive in. There has never been a day when you stop being energetic,

so there's no news there. But yeah, no, let's start actually from your Intel career. Recently and recent Horowitz created this Horowitz and recent Academy, which was to find talented people between 16 and 22 and then put them into a modern educational setting so they'd be well prepared for

either doing things of their own or joining the companies of today. You had a similar experience like you went to a regular trade school and then jumped into Intel when we were, what, 18 or 19? Yeah, it was really a sort of magical period. I'm 16 years old.

I accidentally win a scholarship I go to tech school, right, skip my last year and a half of high school, and Intel comes recruiting when I'm 18 years old. And I never been on an airplane, had already fallen in love with computers at tech school. And the interviewer, right, Ron Smith was his name. He writes on his page. And I was number 12 that he interviewed that day.

And if you've interviewed 12 people in a row, you can't tell male from female by the end of it, right? On number 12, he says, smart, aggressive, arrogant. He'll fit right in. So I got invited to come to Intel and just became really glorious, starting as a technician, moved into the design team at the end of the 286, engineer number four, the 386, architect and design manager for the 486, all of that while doing my bachelor's, my master's, and PhD work. So it was like the best career that you could possibly have.

You're learning by day and you're putting it to practice at night. That's right. That's right. They're creating a completely new era of chips. I bet most of what you learned and what you built, there was a big gap because you're breaking new ground. Yeah, in a lot of ways. And it really was a pretty incredible time in the industry.

And I remember one of the professor classes, he postulated a new carry look-ahead adder design. In one of my classes at Stanford, so I said, well, I'm working on the 386 carry-look-ahead adder, so let me go try it. So I come back two days later to, the professor and I said it doesn't work. Right?

And I started an argument with my professor at Stanford. It was like the next two class periods were just me and the professor arguing over why his design didn't work. He had a book that was being published with that design. It was very annoyed. Oh, it was very time sense. And we started built in self-test, right? And I remember Ed McCluskey, Stanford was my professor. And he was sort of the father of that.

And we had wonderful arguments around why most of his ideas were bad. weren't practical for real chips, but I was putting built-in-self test to work in the 3-6. John Hennessy was my thesis advisor, and obviously he did okay. I helped his career. He became president. He's not okay since then. Later on, we'll talk about the CISC courses. I had the same experience when I was teaching at Stanford.

You come with something that you think is the truth, and then the student comes that works in industries. Well, actually, not quite. So the next lecture you come back with, okay, I look this up. You're right. This is outdated. Yeah, I'll tell you, you know, that interaction, and I think that's part of what makes Silicon Valley so unique, right?

You know, is you have so many companies and entrepreneurship. It's like in the water here, right? Starting things up and new ideas and challenging the professors, and you can't be a professor at a place like Stanford and not expect to get challenged and enjoy it. There's a good foot traffic in and out that ivory tower. Yeah, I think it's good enough. Yeah, no, 486 was your.

a big yeah accomplishment if you will obviously lots of people worked on it and you guys broke some new ground and chip design

there and increased what functions the chip was supposed to do and so on and so forth yeah it was a pretty magic

period as well because it was before what you would think of as the EDA industry oh wow yes

that's right and at the end of the a little bit of the 386 but the 46 was really the first

chip to implement what you would think of as modern EDA techniques. We did a high-level design, right, description and RTL description, but there was no VAROLG. So we invented HDL, right, the Intel Hardware Description Language. I wrote my own language. Well, you had to build a compiler for that language, right?

So we created a compiler for the language. There was no automatic placement route. So we had to invent that, and we worked with Alberto Sanji, Avonji, Vincentelli, Berkeley, and some of his students for the first. the first place where the first routing, the first automated timing management. So they created EDA to some point. Yeah, to a great degree that, you know,

and that was really one of the hallmarks, that not only was the 46, this compatible pipeline microprocessor, but we ushered in many of the foundations of what became the modern EDA industry and was really a pretty magical period of the industry. Yeah, yeah. If we look at today, well, people look back as Blackwell and saying this was of the last generation that was done before AI took over the

design process for Microsoft. I've been a similar transition right now. I think there are certainly aspects of that. And I think when you look at things like Halapino today, right, you sort of say that's sort of a first principle's use of AI in the chip design process, right, where you throw out a lot of those things. Now, there are pieces of design that are still really hard, right, in that sense. And a lot of the analog, right, aspect, surtees is probably the best example of that, you know, are still not A-Iable, right? You just need lots of silicon data, right, to get those. So you either get so conservative in your analog design that you're able to, I'll say, AI it,

or, right, you've got to do those the hardware. But now many of the logic functions can really be done with AI tools and techniques in pretty incredible ways, right? Your transistor budgets are big enough. the design tools are getting smarter. A few experts guiding the tools and how to apply it. And I really think it will be somewhat like the 46 that way. Right.

We just sort of look back and say, yep, that was the beginning of a new era of chip design. Whenever you have the technology to make something easy, that means the bottleneck moves somewhere else. Can we guess at this point where the bottleneck will be in the future? There's some things which seems like AI is incredibly good at, right? Like creating the software layers, you know, writing kernels, a lot of the, I think a lot of the sort of being able to specify the objectives at a higher level, which then gets sort of translated into a very long, right?

And so what is the new frontier? What is the thing that it will be difficult going forward? Well, when you look at today, so let's say Guido and Pat, we're off to do a great new chip together. We're going to build the eye. Right, right. And it's the one that's going to leap ahead because we have understanding of the 8. It's 2036. We have understanding of the AI workloads.

We have understanding. understanding that of how to compose that and then the right, multiply, accumulate, register structure. Perfect visibility in model workloads of tomorrow as well. Yeah, and all of that kind of stuff. And let's say we turn our AI tools loose. We are great. Within three months, we have an awesome design, right? For that, we've been a little bit conservative on all the analog components of it. Now, we still have a couple of really major bottlenecks today.

One is, right, we still have nine months of silicon processing time. Yeah. Right? Show them like that. Right. So I can design the thing in three months, but I can't actually get it into real silicon at scale for nine months. Okay, that sucks.

Right. So we have to really see that bottleneck improve if we're going to have this innovation because we know. What the validation of? Well, you know, first, I can't get anything out of Fab. You know, in less than three months. And that's if I have, you know, like supercharged design flows. Right. So I said, you know, how do I compress that, right, you know, for it?

And then I got to get into advanced packages. 3D packages are getting complex. So that's like another three months, right? You know, and then I have to put it into a rack scale solution because nothing's a chip anymore. It's a rack. Yeah, so we're nine months. You know, it took me three months to design it,

but it's nine months until I can actually start to use it. So how do we start to compress, you know, those aspects of design? And I think we need new forms of lithography, you know, to enable that. We need, you know, design flows that don't require, you know, $50 million of mass cars. until I can get things into prototyping. So, you know, to me, how do you compress that to a month or two, right? Because I can now do the design in three months.

How can I have that done in a month or two? Because if it takes me a year and a half until I actually get scale and software on it, okay, my understanding of the AI workloads is no longer applicable to the chip that I design, right? You know, so, you know, all of those. We've seen that play out live right now, I think with the AIJOT. Right. You know, it's almost, you know, we can look at it historically. It was like, you know, Graphcore wasn't a bad design, but the world moved on, right?

You know, in that. So all of those aspects somewhat obviate the fact that AI has made the design piece easy. All of those manufacturing scale issues to me become a huge bottleneck. Right. Another bottleneck, of course, is memory. Yep. Right. You know, memory bandwidth.

We noticed, yeah. Right. And, you know, for that, as I've described, HBM is a hideous memory. That's just the best one that we got. I was asking about that. It's a terrible memory. You know, I get terrible bit density. I'm limited to shoreline bandwidth, the power, right, issues, thermal A issues,

DRAM doesn't like to be hot. Concentrating everything in one spot. Yeah, you know, it's terrible to give an AI is a memory compute workload. Yes. Right? So, right, you know, for that. So, you know, I need to, you know, compress that development and embodiment into real scale. I need to fix memory, right, associated with us.

So to me, that's another critical bottleneck for us to attack. And power, right? You know, power in, you know, thermal, and most of the power simulation aspects today are bad. Right. You know, you don't have good 3D modeling tools for power, power dissipation, hotspots, etc. You know, my voltage in, right? I'm essentially guard banding almost my full power envelope. Correct.

Right. So essentially I'm losing, you know, 40% of my power and guard banding. So I got to get much better at power modeling, you know, and then my power dissipation techniques are bad, right? So, you know, that'll remodel has to change too. Yeah. So all of these things, you know, to me, those are, you know, the next big bobble necks, right? You know, how do I, you know, improve each one of those?

And now, right, you know, if I've done that, okay, now we're on to an exciting error for what's next. So, you know, thank you for those great AI tools. but I got big challenges, even when those are fully deployed. And so we've seen sort of a rebirth in the number of chips and chip architectures and so on and so forth. Thanks to these workloads. It's like there's 100 AI inference accelerator chips, and I'm sure I don't even know them all.

Yeah, exactly. So... Why are you guys funding so many of those anyway? You've funded your fair share, too. Yeah. So, they're all, to their credit, making different design assumptions, either in the components that you talked about or in the part of the Pareta curve that they're targeting. So do you think there's room for all of these? Or are we in a world where partly because of what we talked about, AI tooling,

we now have the ability to build more different kind of chips? Yeah. The time is enormous, whichever way you want. To address it's even more provocatively. Historically, they have not been a hundred competing process of vendors in any industry ever, right? So is this a more heterogeneous future? Will we see, is this a temporary thing? It'll convert back to a view.

Yeah. And, you know, I see it as a more temporary thing and it will convert, is what I expect. And there's probably three different reasons that I see that to be the case. You know, one of them is, you know, this emerging. heterogeneity that you see in the compute environment, where, okay, now I have specialized pre-fill versus decode. Well, you know, and now people are saying, well, we really need to specialize tier for mid-fill,

not just pre-fill, right? So now I have early pre-fill, and now I have mid-fill, which is, you know, different. Well, maybe speculate and verify if we listen to Open AI here, right? Yeah, you know, and, you know, all of a sudden, my compute fleet becomes more and more granular across the workload. And I think any time that you've seen that in history

is not sustainable. And then all of a sudden, right, when, you know, as you look at that, and now people are saying, well, as I go to reasoning models, I want something that looks more like a CPU again. So, you know, all of a sudden I didn't want that level, you know, that didn't become the predominant portion

of my compute capacity. I needed more of it over here, right, and model and model workflows and, you know, so on. So I really don't like, I'll say, at-scale specialization, because I think the workloads are going to continue to moderate and migrate so significantly. First, you know, second is, I think today's models are about to go through some evolutionary breakthroughs as well, right?

You know, I think the LLN is sort of reaching the limits, and now as people look to how do I really model 3D, right, you know, where a flatish LLM doesn't work well when you go to molecules and chemicals and, you know, right, imaging. So I think we'll see limits of that that will cause different shifts in the algorithmic domain. I also think that some of the most interesting workloads become where you start, you know, I'll say bringing HPC like things back into AI like things, where all of a sudden, you know, 64-bit provision, precision counts again, right?

So I see... What would be an example of that? Well, you know, imagine that my AI model is now inducing the five most... interesting domains for a chemical analysis, right? Okay, those chemical algorithms are going to run with high precision, right, you know, which is going to be the dominant piece of the workload, right? Getting to the five algorithms I need to run or running the five algorithms, right, on different chemical or biological systems. So, you know, I do think that, you know, or as we optimize our AI systems, they will lead us back to some of the things that have been more traditional,

high-performance computing. So I see that aspect of workload as well. So all of that said is this extreme specialization to me in the compute architect. I sort of don't like it because I don't think it's going to be what the workloads will look like two, three, four years from now. So that's one reason I don't see all of these AI chips as they get more and more specialized for portions of the compute workload to necessarily be right. Second is, right, there's like 100 of them now.

Right? You know, there's multiple optical ones. There's probabilistic ones and so on. And you know they're not all going to win, and they're not all going to win because at the end of the day, you have to get scale on these things. And scale requires, you know, you have to win, you have to get capital, you have to get workloads on to it. So I think it sort of defies logic that you're going to have a hundred of these things.

So I see then, you know, coming back as a result of, you know, simple, you know, capital market share, etc. You know, which are the winning teams, you know, winning designs, winning architecture. So I see the workload driving that. I see the natural industry consolidations, you know, that will occur. And then I also expect that the winners will pick some winners, right? You know, where, you know, an OpenAI, an Nvidia, and Anthropic will say, I like this one

because it isn't just hardware. It's how do the hardware and software code evolve. Yeah. Right. You know, for it. And I do expect that there's going to be, hmm, you know, here's this 10 I could pick for this. I like that one. And it does require an investment in the software and the workload evolution to take advantage of those platforms.

So for those three reasons, yeah, we're going to see, you know, a narrowing of the field in that sense. And the deployment complexity as well. Yeah. It makes all sense. And I think I agree in most part, let me still try to destroy a little bit controversy here just to make sense. Do you know what? No. Totally not me, right?

Exactly. You guys have changed it, right? So obviously, 100 companies won't exist or consolidate, right? And I completely agree with your argument that at the end of the day, the large consumers will pick winners to some degree, right? Today, I think to be a first-tier chip startup to some degree, whether you're that or not, is defined by how much traction you have with the really big guys. That's it.

And there's also an argument to say, like, look, in any kind of transition period, you probably have a bit more an explosion of different approaches and all the time that'll shake over. Now, all that said, I think there's an argument you can make that basically says, look, historically,

we standardized on, you know, one or two architectures because, you know, different instructions sets were hard. You had to build multiple compilers, you know, like the, basically the software complexity dictated that you can only have

a certain small number of hardware platforms. That part seems to be changing with AI a little, right? Today, you know, writing an optimized kernel for, I mean, I think Calapenio was a great example, but they're basically that, look, humans can't really program this thing anymore, but an agent swarm overnight can do it.

That's all we need, right? So, where basically you can say, like, well, if you can mill the chip that has a certain capability, it's now much, much easier to build the software layers and talk that fully leverages that. Could a result of that be that we have a little bit more heterogeneity, just because it's easy to program that heterogeneity going on. Yeah.

And I do think it changes some of the, you know, you know, you know, where the bottleneck lands, right, changes in that regard. You know, the first bottleneck is the one I point to that we've already talked about. Still takes a year and a half to build these things. Yeah. Right. You know, and then the second bottleneck is not only does it take a year and a half to get these things into racks and scales and so on like that. Now, to make them useful, okay, you know, could I borrow 50 billion? Right. I mean, you're not talking about small capital expenditures. You're talking about very large capital expenditures and data centers, you know, power, you know, commitments and so all like that. Well, I've got to build

this thing at scale, right, associated with it. So, and that's going to defy a lot of heterogeneity as a result. Now, you know, clearly big players, and we saw this with what, you know, what Vindia did with GROC. Okay, you know, we're going to sort of own some of that heterogeneity, right? And, you know, two years from now, you're not going to tell what GROC is inside of Nvidia, right? It's just going to be like another special, right?

You know, someone like that. So you'll see the architectures, you'll sort of hide the heterogeneity, right, inside of it, which to me is just fine. Right? You know, I mean, that's what layering. That's what abstraction has always done associated with it. Yeah, but I think those forces are powerful ones to, I'll say, expose a lot of the heterogeneity. No.

Right? I think it will get abstracted underneath, you know, some of these. But at the end of the day, it's still hard to build these things at scale. Earlier, you talked about HPM being hideous memory. Mm-hmm. By far, the best one we have today. Exactly. The thing about democracy.

Yeah. We can't recall when there was the last memory innovation. It might have been obtained. I think you probably had something to do with it. Yeah, that one died. That one died. We killed it. Yeah, we killed it.

So do you think memory innovation is around the corner? What needs to happen for? Obviously, we understand the scale problem because it's even more larger there. But if you keep the scale aside, even from a technology point of here. Yeah. let's, you know, look at that history. I've probably personally been associated with at least five different new memory architectures

obtained just being one of them, right? You know, that did not see the light of day, right, as well. I'm probably familiar with close to 100 that have happened in the industry over the last 30 years. And exactly how many major new memories have we had over the last 30 years? Zero. Right? Yeah, which, you know, okay, D-RAMS, RAN, fly.

Okay. What else is there? D-RAMS, flag. You know, so I do think... Flash stacking comes for a little bit. Barely. Yeah.

Right. You know, but anyway, right. So, you know, in that, it, you know, it has been, right? You know, a disappointing that we haven't created, you know, the next physics, you know, that enable, you know, low-cost, high-performance, resilient, manufacturing, manufacturable memory, right, you know, for it. So I do think, And now, over the last 30, 40 years,

you know, memory was a terrible industry. Right? Because you would make money one year and then you'd lose money for the next four. Right. So let's go do R&D in an industry that makes money one out of five years. Right. You know, it's just always, you know, it's just been so terribly difficult because it's been such, you know, driven by such violent commoditization cycles.

That has changed a little bit, though, recently. Over the last five years? Ha, ha, ha. Happy days, man. So all three big member of vendors are in the top 20 most valuable companies on the planet. Yeah. That's a crazy change. It's just so, right, you know, who would have thought, right, associated with it?

And anybody who looks at the AI workloads, it's always, you know, it's much harder to, you know, it's easy to predict the past, but it's a little bit harder to predict the future, right? And, you know, in that, you look at it and say, AI is a memory work. Exactly. Right. you know, inside of it. So I think there is the combination of capital and technology needs that will justify memory innovation. Now, there's two domains that I'm excited about, you know, for it. One is, you know, this idea, you know, I do think that the shoreline bandwidth limitations of HBM is a fundamental issue. Yeah. Right. So with that, the idea, you know,

and obviously, you know, we have a company D-Matrix that's doing this, but I think others will be, whether it's, you know, Cerberus, you know, and their techniques. The future is stacked, yeah. Somehow you've got to bring memory and... In more ways than one. You know, you've got to bring memory and compute together. Yeah. Right.

You know, in fundamental new structures. I do think that given the workload now has such capital being deployed against it, you know, in the AI space, that I do think will actually be able to break through some of those physics challenges as well, as well. I do believe, and, you know, I've just funded a new memory company as well that's still a stealth. But, you know, I'm quite excited about some of the innovations that will occur in new memories. And you're going to see new materials, you know, for memory people, you know, a lot of people looking at things like pharaoh electrics.

You know, finding non-capacitive, you know, high-density memories, finding memories that can, you know, be high-performance, high-density and stackable. Right. But, you know, people are looking at a low-cost flash, you know, for that. I'm not particularly enthusiastic about that idea myself. I think, you know, speed and performance of the underlying cell are somewhat problematic with a flash or an M-RAM structure. You know, but I do think there's memory innovations are coming for the first time in 30 years. And I do think that's going to be an exciting space. And, you know, we'll have a couple of companies there, I'm sure. Yeah, hopefully. We'll do a few of them together. Yeah. Yeah. Maybe, you know, money fixed. Okay, you know, but we'll find some things to do here.

But I think memory innovation for the first time in 30 years is nigh upon us. Yeah. Yeah. It's amazing, what a couple of $10 billion of market cap. Do we go 100 billion? I don't even know what you say? Trillion. Okay.

Sorry. It's like, I think the memory industry has now increased by market cap by two and a half trillion dollars over the last four years. I think that's about right. That's. We could have just invested there and not done any of a venture stuff. Yeah, exactly. Easy money. How long?

What's your view? Sorry, go ahead. Yeah. To expand a little bit on the stacking part, if you have to predict how quickly will our chips get taller? This is going to be like, you know, we're going to go to two and then sit there for a while, or will we have, you know, the 16-layer sandwich, you know, happily mixing logic, memory and S-RAM? H-T-M pulled down this.

HBM is going down at Moffin. But how do you see this at all? Yeah, you know, I think there's some, you know, the problem with stacking, right, is now essentially the value of the stack, right, becomes greater as you go to higher dimensionality of the stack, which means the yield of any individual layer in the stacking process has to keep coming,

you know, essentially it can't go up linearly. Yep. Right? You know, it has to go up exponentially. and improving the manufacturing and defect characteristics in an exponential way, you know, when you get to eight or 16 high stacks, gets to be so, so hard when you run the math on that associated with it. You know, your manufactured flows have to be so close to perfect.

Although you could see designs that take into account a certain model class, like we've seen it for a long time for two-dimensional. Of course. But remember, that was the nature of derram. You always had bad block replacement and, you know, spare lines and so well like that. So I'm not in any way, right, you know, minimizing that. But assuming you're doing that, you still need the yields to be so good. You know, if I have a crack die, I don't care how much resilience you've built into it.

You can't, you know, it's a bad die, right? So, you know, those kind of, you know, so as a result of that, I'm not, you know, a crazy 16, 32 stack, you know, kind of guy. I think there's going to be, you know, nice three, four, five stacks, you know, that just end up sort of being the sweet spot. The mid rises. Yeah. Yeah. Right. You know, for, and I expect for, you know, part of its manufacturing, part of its complexity associated with the, you know, where's those yield sweet spots associated.

So that's sort of where I think most things land. But even there, right, you know, so let's say I now have, you know, I now have a great compute tile, right? You know, how big does that tile need to be? I mean, everything's become reticle limited now, right? You know, can I go to more chiplets, that better yield characteristics, you know, better thorough? normal, you know, and then you're going to say, I want a memory stack on that. You know, is it a two-stack memory, a four-stack memory? You know, I don't see it becoming an eight-stack memory.

You know, so to me two or four sort of right. Yeah, but then power delivery needs to get integrated into that. So that becomes sort of another element. And I want that in the Z dimension, right, associated with it because, you know, I want my power rails coming into this as well, right? And then I need, you know, fairly intelligent RDL layers to distribute signals across that platform. So to some degree, you know, my canonical thing is already about eight-layer stick when you add up all of those pieces associated with it. And I think that's about all you can do, right, at least as we understand manufacturing today.

And then, of course, you know, we expect optics to come in. So you're going to have more three, five material structures, you know, that are going to be closer to this complex because I want, you know, my I.O. getting, you know, essentially, you know, deeply integrated into that construct as well. So my silicon-centric pieces need to get complemented by some of these other 3-5 materials. And how do I get optical connectivity built into this? How do I get power rails built into that? You know, that is a, you know, I'll say that is an engineering feat and a manufacturing nightmare to bring all those pieces. But that's where the physics is taking us over time. So because of that, you know, throwing a 16 high stack and it's like, that's crazy.

a two or four high memory stack. I think that's probably where the sweet spot will be. And didn't talk about the thermal dissipation and new materials needed for that. Yeah, yeah. And obviously there's going to be a lot of, you know, first thing, right, you just can't have some of the, you know, the physics hotspots that you have today.

So you have to be more intelligent about how you use the heat to spread the heat. But, you know, then you're going to be saying, okay, how do I both get power in and dissipate power out of those environments? And by the way, if I can, you know, get much more efficient power in at the voltages I want with much better, you know, power responsiveness of my voltage regulation, you know, then I don't create as much thermal chimneys. Yeah. But I do have to think, okay, how do I cool these things?

And, you know, what role do new materials, diamonds and other things have been postulated as ways to, you know, do a better job of the thermal cooling? How do I then have better, you know, liquid cooling techniques and others, you know, sitting right on it, you know, with good thermal. characteristic. So, yeah. You know, we engineers are becoming plumbers, right? Exactly.

Yeah, you don't worry about MEC, the chip scale. Let me ask you about an operative phenomenon, which is some people have said, hey, let's get the memory off of the package and just go all optical

between the compute and the memory. And that'll give you more degrees of freedom and more ways of building things. What's your view on that as opposed to stacking?

That's other Yeah, you know, I think the more OEO, right, interfaces that you have, you know, optical to electrical to optical, to, you know, the more of those that you demand in your core compute complex, enemy is just hard, right, in that sense. And you're always going to have losses associated with it. You know, people are, you know, working hard to come up with more clever techniques to decrease, you know, both the cost and the power, right? you know, how much of the power am I wasting as I go through my, you know, my Dax and, you know, conversions. So I'm not a fan of those, you know, because I think it gives you access to large pools of memory, right? But I burn a lot of power getting there. Yeah.

Right. And remember my, you know, the essentially, you know, the femtojoules per compute is now actually pretty amazingly good, right? The femtojoules per bit of communication is comparatively a thousand times. worse, right? And if I'm spending that much to go get to large pools, so I don't like those approaches. You know, myself, I think they somewhat defy the underlying physics of the solution. I mean, and then there's sort of the other approach to say, like, look, can we actually integrate compute and memory directly, right? Shift some of the compute into the memory. Some of the PIM approaches. Yeah, exactly. So what are your thoughts about that?

I'm not a fan of those either, right? Because I think workloads, you know, because now I have to take my workload and I have to like push these little pieces of compute into the memory, right? You know, so that, you know, I don't need as much bandwidth of memory coming into my real compute, right? And that puts constraints on workloads. And again, you know, I could be wrong on this, particularly some of the design flexibility and the intelligence that we can now bring, you know, through the design aspects. But most PIM ideas, you know, have been around for 25, 30 years. And I think 25, 30 years from now, they'll still be hanging around, right? in that sense, you know, give me great memory bandwidth, you know,

that I can have, you know, high density, high performance, and get it close to the real compute engines. I expect that's going to be a more generalizable solution across many and evolving workloads over time. So we're still stuck with denser and denser chips, more compute, more logic, more power. Much better I. Now, I am a huge fan, right, of moving to optics for all I.O. Yeah.

Right. You know, I declared the death of copper about 25 years ago. Eventually, I'll be right. You're still right. Yeah. One day soon. I'll be right. But I think is that, you know, to me, all IO should go to optical, right?

You know, but that core compute memory complex, no. Right. So when do you, the first step for going on optical is to scale up, even more so than scale out. Yeah. Yeah. What's your views on that? Yeah, you know, and I think we're now seeing... What would be the application of the workload driver, if you will?

Well, I think the workload driver is the one in front of us. Okay. Right. I mean, you know, today, you know, our scale-up environments, we're putting tons of copper, right? You know, and essentially copper is becoming a wave guide that's getting shorter and shorter, right? And all of a sudden, you know, to make my copper work at, you know, five meters is more expensive than making optical work, right, at 100 meters, right? You know, so, you know, I think the physics is driving us there today, you know, and, you know, so if that's the case, Pat, why haven't we moved there already? Yeah.

It's a complex supply chain that, you know, has never been tested at scale. And I think now, you know, I think everybody is looking at, you know, some form of NPO, CPO, etc., you know, in the 28, 29 timeframe because, you know, I can't scale up to large radix, you know, compute clusters from my scale up environment. without making that move. So, you know, clearly, NVIDIA has indicated that I think everybody else in the industry expects, you know, that's the conversion point. You know, in reality, I should have never built an NVL-72. You know, it's an engineering marvel

and it's the manufacturing nightmare, right? You know, it took the industry, you know, one extra year and a half to digest that beast, right? And it really is amazing that, you know, we're able to get to scale and, you know, so on. But, you know, that really proved that had we had a more mature, optical supply chain, we'd be much more advanced today. We'd be able to build better scale up environments today,

you know, better power, better costs, better energy. But we didn't have the maturity of the optical supply chains to allow that. And those things are not trivial to get resolved. There's a lot of work to happen. But I think 28, 29 is the year. The good is that. We're obviously gaining a lot of experience how impact optics work in networking.

And I think many. of that will translate nicely to the GPUs. Yeah, I do think. But all of a sudden, you know, it's like, okay, these things have to work, you know, at 10 million GPU scale. Wow, right? You know, it has to fit in packages. And, you know, there's just a lot of things there, you know,

and many forms of optical don't like bad thermals. Well, guess what? Bad thermals is the nature of the game for it. You know, so we have thermal problems to deal with, you know, package integration problems. We didn't have enough, you know, laser capacity in the industry. So a bunch of things. But I don't see anything there.

Fundamental. It's just a lot of hard work at this point to pull all of those pieces together. In terms of the networked architecture, builders also mean a switch what's more circuit-switched networks just because, you know, you can't scale these very fast optical networks indefinitely.

Well, you know, I like, you know, Nick McCune, you know, another Stanford professor here, a good friend of mine, you know, as well. You know, I mean, you know him a little bit off. Grita. Was he a thesis advisor? He was not his advisor.

Yeah. Yeah. So, but, you know, to me, he describes it, well, you know, we built, you know, networks to be able to handle any packet going anywhere with no knowledge of where it might go. When you think about AI, you know, it's almost exactly the opposite, right? You know, we're doing flows that are large and predictable. So in that sense, just the very description says, hmm, that looks more like a switch. than like a packet. So I do think this confluence of moving to optical, right, and, you know, having a workload

that is far more predictable, manageable, you know, large, and flow, you know, to me, you know, I do see that, yeah, we will go all the way to bringing optical into the compute complex, and we will go all the way to OCS or some form of optical switching, you know, with more predictable flows. You know, that is the right, you know, architecture, and then you sort of say, is that scale up or scale out? And to some degree, if my scale... Yeah, it sort of dissolves the boundary. Yeah, right. If my scale up is big enough, then, yeah, you know, just saying, okay, what's the radix

in my scale up environment, and when do I want to hook it up to other clusters that are really big? And to me, that ends up being a pretty beautiful answer, you know, to the future. But I'm looking at somebody who's way more expert on this than myself, so... No, I... Look, I always found scale up or there's a scale up to some degree in artificial distinction. You understand where it comes from, right? We're kind of using different protocols.

One is front end, one is backend network and so on. So right now, practically speaking, it is a big difference. If you take a step back and look at it from a system architecture perspective, it achieves the same goal, you know, with slightly different flavors, slightly different protocols that over time I think will converge. Yeah. I mean, especially for the AI training workload. Yeah, although training and inference is actually look fairly similar.

The influence corpus lesson. Yeah, now I'm going to, you know, right, does your brain stop learning? Right. You know, by the way, it's another characteristic of workload evolution. I do think the separation of training and inferencing is going to become, you know, less so going forward, not more so. Right. That's a little bit of the argument against, you know, more and more heterogeneous architectures is, I think, continuous learning. You know, I think there'll be algorithmic domains that will say, yes, continuous learning, you know,

becomes the nature of the algorithm where, yeah, I do want my inference environment to be updating my model weights because, you know, that's giving me more and more refinement, you know, evolution across, you know, workloads, moving into environments that are less formalized training as well. We'll have smaller data sets in those environments. So, you know, I'm not sure that's a... But the number of nodes you need in order to be useful

is a smaller diameter for inference. That's what it means. Today, that's true. Yeah. Yeah. I mean, I still think there might be a role for heterogeneity coming from a different angle, which is that different types of AI workloads require very different underlying. Like, some are more compute limited, some are more memory bandwidth.

We have a small diffusion workload also, right? The optimal card for that looks very different from, you know, top of the line, you know, $15 trillion, $15 trillion weight LLM, right? That needs to operate a massive scale. So I think that might drive heterogeneity. if I think there's some room for that. But I'd also say, okay, now I'm talking about putting a million GPUs to work in my data center.

Right? And you say, well, you know, for that particular diffusion workload, you know, I should really put something optimized this way, right? So I'm now going to start, you know, putting, you know, a thousand of those in, right? And for that workload, I need, you know, 2,000 of these in. And, you know, for some, you know, and all of a sudden, you know, you don't get some of the, you know, the flexibility of scale. right, that you would say, and, you know,

how much does it cost me to manage and upgrade and, you know, connect and, you know, fault domain, you know, all those types of things. The agents will do all the upgrade, so that's no longer a problem. Yeah, yeah, yeah. Thank you. Everything gets easy. So good, I don't know. Yeah, so fleet management is done. So we talked about what?

Computed, memory, networking, power is the next one. Yeah, yeah. And you touched upon it with vertical delivery inside the chip. on the chip. But, I mean, there's 800 world DC, there's power delivery to the data center. There's an entire ecosystem all the way to power generation and the social political dimensions of that

and so on and so forth. So where does innovation come in and where does, like, it's pure execution? Well, you know, and if we start at, you know, first principles here. You know, our nation has done a terrible job with this energy capacity, right? You know, essentially we went through 10 to 15 years where essentially I was taking coal offline at the rate I was adding renewables and essentially the nation was flatlined in terms of

overall energy capacity. Not good. Because in an AI digital age, energy capacity equals economic capacity. So essentially, my economic capacity. economic capacity as a nation flat for 15 years if you buy that thesis, right, which I think is very, you know, approvable. Over the last five years, you know, we've seen this enormous influx and it just give me more capacity.

But, you know, I think, you know, even with that, we've gone to maybe increasing our national energy capacity 4% per year. Something like that. Going from, you know, essentially zero to one, right? Wow. 4x or 4%. So to me, you know, that, you know, it's a bad situation. So we just need more energy capacity for it.

So number one, and obviously we have some companies in that space. You know, we talked about nuclear operating as one of those with ALVA. And, you know, I think nuclear, right, you know, as a baseload is, you know, a fabulous thing that we want to go build more. You know, unfortunately, all of our renewables have, you know, deep dependence. in China, which is very unfortunate for us, you know, we're, you know, gas turbine lead times are only eight years at this point, right? So, you know, I mean, we really have a difficult environment to scale up rapidly, but one, we just need more capacity. And fundamentally, energy capacity somewhat dampens how you fork we can get on AI, right? Because, you know, why build

the new data center and buy, you know, the million GPUs if I can't power them? Right. And I think you're going to see more and more defaults happening on many of those data center projects because the energy won't be there. So you think it'll be a significant headwind? I think it will be a headwind. And I think you're already starting to see some of the first indicators of that, the Oracle. You know, it was just, I think that's the first of what you're going to see many. Now, it's not like we're going to slow down as a result, but I think people will start saying, do I really have the energy, you know, to underwrite the capital commitments that I'm making, you know, for. you know, cement for data center racks and GPU purchases. So I do think it's going to be, you know, an upper bound.

But we need more energy capacity and we need to get more innovation into that. And we need more of those supply chains in the U.S. Nuclear, one of my favorites. When was the last nuclear reactor that came online in the U.S.? Right? You know, it's 20 years ago. Yeah. Right.

You know, we just stopped, you know, a very capable industry. So that's one we need more. you know, we need to get much more efficient at delivering, you know, the power networks. And as you said, you know, moving to 800-volt DC data centers, absolutely, you know, right? You know, and being able to eliminate so many conversion steps along the way because of historical, you know, reasons and, you know, standardization reasons. Okay, we got to fix this. You know, an 800-volt-D-C, the revenge of Edison is upon us. Yes, I was just going to say.

Yeah, I like to say. So, you know, I do think that ends up being, we need better power conversion. You know, we have a couple of companies, one of those vertical, you know, vertical GAN, you know, I think will be, you know, a killer technology, you know, to be able to do 800 to 48 or 800 to 12 or even 800 to 5 and one conversion step, right? Which just gives you a lot more energy. You know, we touched on. Solicitate transformers. Yeah.

You know, all of those are, you know, solid state transformers, switches, et cetera, you know, rebuilding, you know, the power delivery network. And I think that will have a lot of innovation. You know, sort of like, okay, power's cool again, right? You know, for it. And then you're going to need all of those in the cooling systems as well, right? You know, and so there'll be a bunch of innovations in that, you know, so, you know, turbines and refrigerants and, you know, all of a sudden they get to be, you know, exciting new technologies again. So, you know, for all of us hardware guys, this, you know, it's like renaissance and brother.

It's going to be... After 25 years, I had to go back and read about carnals. Cardo efficiencies. Oh, yeah, it's like good stuff. How does that first law of thermodynamics work against? The physics center might have something after all here. So it's going to be an exciting time that way. But we also say, you know, we need to change the physics, right?

Because seam loss hasn't improved, you know, the essential power per terrar, you know, power per tar flop for the last five. generations of Nvidia chips, you know, flat line. Not okay, right? And that's where some of the things, you know,

we have a superconducting company, Snokechap, right, which affords the opportunity that fundamentally change the physics, you know, a thousand times better power performance. So I do think some of those innovations are upon us in the near future as well.

So I've been a terrible time manager here for the past hour. So, so many topics yet to cover, but we have to cover at least one before we leave. Okay. Because now...

You're in charge of, Yes, so I thought. Now VMs are back. How do you think about that? I was a farmer CEO for VMware. Well, you know, I... I feel excluded from this conversation between the former CEO.

I was going to talk about networking as well. But I don't think you have to time. Hey, you know, I think the... Every innovation has always led to the next abstraction, right? And I think many of these ideas, and right, you know, I would like to say, you know, what's new in our world today? You know, data structures, algorithms, you know, an abstraction, you know, we bring it back to those, you know, foundationals. And, hey, you know, I think, you know, the virtual machine abstraction, you know, whether it's done from the infrastructure level or from the application level, you know, I think it's still a foundational abstraction model, you know, that deserves to always have a place in the compute hierarchy. So, you know, so what do you think?

One of us go back and run VMware. Yeah, that's what I tell. I think I told Gudos how it's time to go back and run VMware again. Yeah, yeah. And, you know, and I do think some of these abstractions, you know, but you also think about it and like one of the things we did at VMware, we, you know, essentially, you know, managed every aspect of the computing hierarchy. Yeah.

Right, you know, and you manage the workload. You manage the network. You manage the storage, you know, system, and how do you abstract those? And I think in the AI content, you manage the computer, where we think about these agents' swarms, well, who's going to manage the agents? Who's going to create the security profiles around all of the agents?

Who's going to be managed the performance of the agents? Essentially, every fundamental element of virtualization and management needs to get recreated in this next computing hierarchy. And there's going to be lots of wonderful companies that you'll sort of break through for how those things will be done. Do you want to end it on networking?

think? Actually, I love to stay with the VMs for a second. The most interesting thing for me is that we're now building VMs so that they're usable for agents as opposed to usable for humans. What we've seen in other companies is that the change is a lot, the form factor that changes a lot, how you market them, right? You want the agent to make the pick for your VM offering. It changes a lot, how much complexity you can have.

It changes how quickly the startup should be. Humans are a lot more patient than agents. So it's, if you have to rebuild VMware today, how would it look different? Yeah. And I do think. The VMware for agents, right? Yeah. And I do think there's, you know, does there need to be a VMware for agents for humans, right?

You know, I see what I mean, yes. Right. You know, in the sense that, you know, I need humans. Let's stick to the agents. They're way more interesting that one of the ones right now. Yeah. But, you know, I think it's an important point because I do think that there needs to be, you know, And I'll use, you know, Dario's constitution, right, or guardrails, right, or being able to set policies.

And I think then you really say, okay, every aspect of the virtual machine, as we know, we'll love it today, it is servicing agents, right, as opposed to servicing hardware and humans. Yeah. Right. So I think, you know, that becomes your fundamental design constraint on this side. You know, how do I make agents great, right? How can I make them secure? How can I make them perform it? You know, you know, how do I abstract them?

How can I migrate them? All of those things, you know, have the new embodiments, you know, V-motion for agents, right? It's like every one of these will have the new embodiment here for them. But then I also need to think about how can humans be setting the policies, describing the constitutions, you know, getting the, you know, dashboards, et cetera. And I think that's sort of, you know, where the interesting thing is because you need to do both of those, right? Not just one of those. But the fundamental, you know, I'll say the hard design stuff like we did for, you know, virtual machines was associated abstracting the hardware in a performant way for applications or operating systems.

Here, it's, you know, abstracting the hardware and the operations for agents. It's a fundamental thing they were serving. It's great discussing this. And thank you so much, Pat, for coming in and we'll do it again. Well, two of my favorite people, anytime. All right. That's awesome. Thank you.

Thank you. Thanks for listening to this episode of the A60s podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our substack at A16Z.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax, or investment advice, or be used to a value.

any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.