No Priors: Artificial Intelligence | Technology | Startups - The Computing Platform Underlying AI, with Jensen Huang, Founder & CEO of NVIDIA

Episode Date: April 20, 2023

So much of the AI conversation today revolves around models and new applications. But this AI revolution would not be possible without one thing – GPUs, Nvidia GPUs. The Nvidia A100 is the workhorse of today’s AI ecosystem. This week on No Priors, Sarah Guo and Elad Gil sit down with Jensen Huang, the founder and CEO of NVIDIA, at their Santa Clara headquarters. Jensen co-founded the company in 1993 with a goal to create chips that accelerated graphics. Over the past thirty years, NVIDIA has gone far behind gaming and become a $674B behemoth. Jensen talks about the meaning of this broader platform shift for developers, making very long term bets in areas such as climate and biopharma, their next-gen Hopper chip, why and how NVIDIA chooses problems that are unsolvable today, and the source of his iconic leather jackets. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Jensen Huang | NVIDIA Nvidia's A100 is the $10,000 chip powering the race for A.I. | CNBC Nvidia CEO Jensen Huang: A.I. is at ‘inflection point’ | Fortune Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Nvidia Show Notes:  [1:26] - The early days when Jensen Co-founded NVIDIA [4:58] - Why NVIDIA started to expand its aperture to artificial intelligence use cases  [10:42] - The moment in 2012 Jensen realized AI was going to be huge [13:52] - How we’re in a broader platform shift in computer science [17:48] - His vision for NVIDIA’s future lines of business [18:09] - How NVIDIA has two motions: Shipping reliable chips and solving new use cases  [25:41] - Why no one should assume they’re right for the job of CEO and why not every company needs to be architected as the US military  [31:39] - What’s next for NVIDIA’s Hopper  [32:57] - Durability of Transformers  [35:08] - What Jensen is excited about in the future of AI & his advice for founders

Computer programming has now been completely disrupted. That for the very first time in the history of computing, the language of programming a computer is human. Any human language, it doesn't have to be grammatically correct. And it's fairly incredible that anyone can program a computer now. That's a big deal. This is the No Pryor's podcast. I'm Sarah Goya. We invest in, advise, and help start technology companies. In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

So much of the conversation in AI today is about new models and new applications. But it's all built on the back of Nvidia and specifically Nvidia GPUs. The NVIDIA A-100 has been called the workhorse of today's AI ecosystem. And this week, on NoPriors, we're so excited to have Jensen Huang, the founder, president, and CEO of NVIDIA. So over the last 30 years, he's created a company that began with a goal to accelerate graphics and has transformed the way we compute to become a $674 billion behemoth, as far as I could tell this morning, but that the CEO describes as a company that he's always trying to save.

Thank you for doing this with us, Jensen. I'd like you do it, sir. Why don't we start at the beginning? You worked at LSI and AMD before starting a company. How did that happen? They gave me a job. Let's see. I was at Oregon State University, and it was a campus company day,

and I interviewed at a lot of companies, and two companies really, really connected with me. I love designing chips and designing computers. And at the time, in our lab, in the computer science lab, There was a poster of a 29,000 32-bit CPU from AMD. And, you know, I always thought it'd be kind of cool to build that. On the other hand, there was another company that was a startup at the time built by one of the legends of Silicon Valley, Wolf Corrigan. And they started a company to design chips using software, to design chips not by hand, but by using programmable logic.

And you would describe it in language and it would synthesize. it to chips. And of course, I chose to go to AMD. It turned out I went there to design microprocessors and my lab partner, not my lab, but my office mate, ended up going to LSI. And she insisted I go there after I was there. And after she went there and the LSI team said, hey, we were recruiting this kid from Oregon State. And we really wanted him to come work at LSI Logic. And it turned out to have been her office mate. And so they all reached out to me. And I decided to go there because it was at the beginning of the EDA industry. It was at the beginning of designing chips using computers. And it was probably one of the best things that ever happened to me. And it was in the beginning

of the ability for every company to build their own chips. And it's the reason why I met some really great computer architects. Andy Bechtosheim was the founder of Son. I got to work with a bunch of great architects at Silicon Graphics and John Rubinstein, who was at a company called Dana Computer, who became the vice president of engineering for Apple. And then, of course, the two founders of Nvidia, Chris Malcowski and Curtis, me and myself. And so I got a chance to work with some really amazing computer architects. And I learned a lot about building computers with chips. And so that's my early days. And you were a star at LSI with your co-founders. At what point did you know I have to start a company? It wasn't my idea. It was theirs. Chris and Curtis wanted to leave

Sun. They had their own reasons. And I was doing really well at LSI Logic and I enjoyed my job. and we had two kids, Lori and I, and just like you, they wouldn't stop hounding me. And they said, hey, we want to start this company, and we really need you to come along. And I told them that I really needed to have a job, and they needed to figure out what to do. At the time, the value was the way of designing computers was rather split between general purpose computing versus using accelerators. And about 99% of the value was believed in general purpose computing, and about 1% believed in acceleration. For 25 years, 99% was right. We decided to start a company on accelerated computing. And at the time, the only thing you could really do with accelerated computing is find

applications or find problems that were barely solvable or unsolvable by general purpose computing. And that's kind of what we dedicated our company to do, to solve problems that normal computers can't. And if you follow that mission to its limit, it led us to self-driving cars. that led us to robotics, that led us to climate science problems, digital biology, and, of course, one of the most famous ones is artificial intelligence. You were working on this huge set of applications before the current wave of artificial intelligence. What was the original technical advantage of Nvidia and artificial intelligence, and when did you begin to realize that this was going to be an important use case for you guys? So we had expanded the flexibility of our accelerators to be more general purpose. and we invented a new computing model called CUDA.

And we're doing this podcast like at 4 o'clock or something like that in the afternoon. It was like at the lowest point of energy, isn't that right? Yeah. So this, we need some. That's why we need some nerds. That's why we need some nerds. Thank you, nerds. Now with gummy clusters as well.

So that's very exciting new technology. We need some energy. We need some accelerated computing right now. So we wanted to make our graphics processors more and more. general. And the reason for that in the beginning was because some of the effects that we had to do related to general purpose image processing, post effects. You render the image and you do post image effects. Other applications, of course, we wanted to bring the scene to life. And so we had to do physics processing. And you have to do physics, you have to do particle physics, fluid dynamics,

so and so forth. And so we expanded the aperture of our accelerated computing platform to be more and more and more general purpose. The problem with general purposeness is that the more general purpose you are, the less acceleration you get in any particular domain. And so you've got to find that line really, really carefully. And that's one of the gifts of our company to find that line between, on the one hand, every single generation bringing enormous amounts of acceleration well beyond what CPU could do to the application. And so if you become too general purpose, you're just like a CPU. How can you accelerate a CPU with a CPU?

And so you have to find a way to walk that line. On the other hand, if you don't expand the aperture of the applications that you serve, the R&D dollars that you're able to generate wouldn't be enough to stay ahead of the CPU which had the largest R&D budget of any chip on the planet. So if you think about this problem is actually really nearly impossible

because you have a small application, let's call it a billion dollar market at the time and out of that billion dollar market you're investing $150 million a year out of that $150 million a year how do you keep up with a few hundred billion dollar industry? It's not even sensible. And so you have to find that niche very, very carefully where $150 million would accelerate this particular application abnormally and insanely.

And then over time, you could expand your application space so that it goes from $1 billion to $5 billion to $10 billion, so and so forth, without falling off that cliff. That is the fine line that we walked. And so we kept expanding the general purposeness, and it led us to molecular dynamic simulation. which is what this image seems to look like. And seismic processing was another industry. And slowly by surely we expanded our aperture. But one of the things that we did well was to make sure that irrespective

of whether somebody used our platform for general purpose computing, accelerated computing, we always maintain the architecture compatibility. And the reason for that is because we wanted a platform that would attract developers. If every single Nvidia chip in the world was incompatible, then how would a developer be able to pick one up, even if they learned

that Kuda was going to be incredible for them? How would they pick that up and say, I'm going to develop an application that's going to run on that? Which chip would they have to go figure out? And nobody could figure that out. And so we said, if we believe in an architecture and we want this to be a new computing platform, then let's make sure that every one of our chips perform exactly the same way, just like an X86, just like arms, just like any computing platform. And so for the first five, 10 years, you know, we had very few customers for CUDA, but we made every chip CUDA compatible. And you can go back in history and looked at our gross margins. It started out poor and it got worse. So we were in a really competitive

industry and we were still trying to figure out how to do our job and build cost effective things. So it was already challenging as it is. And then we laid it on top of this, this architecture that was called CUDA that had no applications that nobody paid for. Yeah. It's kind of amazing because now when I talk to people in the AI world in terms of one of the reasons that they really love using Nvidia GPUs is because of Kuda and then because of the ability to scale interconnect. And so you can really like highly parallelize these things as well, which you can't necessarily do with other approaches or architectures that are in the market today. Yeah. And this computing platform, it's strange in a

sense that it performs these miraculous things. And we carried it to the world on the backs of GForce, which is a gaming card. The first GPU that Jeff Hinton got for his lab, ELA would tell you that Jeff came in and said, here's a couple of GPUs. It's called G-Force, and you guys should try to use that for DNA. And so it was a gaming card that he bought. What applications do you have in mind? Because to your point, you started with gaming, or at least you were very popular with gaming,

starting in the 90s when you started the company. And then, you know, I started hearing about Nvidia DPUs more and more, both in the context of cryptocurrencies and sort of mining, and then in the context of AI. And it seemed like those were the two markets where a bunch of people were organically just adopting you. Were you marketing to those communities? Was it just people started realizing that they needed linear algebra?

That's the beauty of a computing platform, right? In the beginning, you have to target the applications. And in the beginning, we did. One of the first applications was NAMD, seismic processing. Both of them are one of those kind of particle physics. The other one is image processing, if you will, and so inverse physics, if you will. And one particular domain, you know, we just went out to hire it to research. We went to scientific computing centers, and we said, what kind of problems are just beyond your reach?

And the list of applications include quantum chemistry and quantum physics. you know, so on and so forth. What was the moment when you said, wow, this AI thing is really important for us? It happened around 2012, I guess, and it was because simultaneously Andrew Ang reached out to Bill Daly, our chief scientist, to work on a way to get the neural network model that they were working on onto GPU so that they could, instead of using thousands of CPU servers, they could use a few GPUs to do training. So that was one. Simultaneously, Jeff Hinton reached out to us, And we started hearing about that. And the same thing was happening with Jan Lacanne in his lab.

And so simultaneously in several different labs, we're starting to feel that there's this neural network emergence, and that attracted our attention. Yeah, I guess 2012 was also the year when AlexNet came out. So I felt like that was a year of transition for deep learning in general in terms of really that was the moment in time, at least that I remember thinking, wow, this really exciting wave of AI coming.

And then I feel like for 10 years, nothing really happened for startups, but a lot of incumbents started adopting this technology. Yeah, we started feeling it. We started hearing about it before that, and then ImageNet kind of, it was the big bang, if you will, got all of our attention. You talk about early AI labs as pulling this from Nvidia using gaming cards because you were solving a problem nobody else could solve and efficiency and scale. Is there a point at which Nvidia begins to like invest in an application because they think it's a growing application? Or is it more it's a platform and the market will take it from us?

No, in every single case, when an application finds use, we ask ourselves, how can we make it even better? And this time with deep learning, the good insight that we made, it was piecing together observations in a lot of different ways, but realizing that this isn't just going to be a new algorithm for computer vision, which is really most of the applications in the beginning, which was going to be very helpful. I mean, if it was just a computer vision, we could use it for all kinds of interesting applications like self-driving cars and robotics, and we did. But we observed that this might be a new way of writing software altogether. And asking ourselves, what's the implication to chip design, system design, interconnect, the algorithm, the system software, to really reason about not just why is this exciting, why was it so effective. which that alone was plenty miraculous. That ImageNet, without specifically any human engineered algorithm,

would reach the level of effectiveness compared to 30 years of computer vision algorithms overnight. It wasn't by a small amount. And so the first question, of course, is why is it so effective? And was this going to be scalable? And if it was going to be scalable, what's the implication to the rest of computer science? What problems can't this universal function approximator, if you will, that can solve problems of dimensionality extraordinarily high? And yet you could learn the function using enough data, which at the time we were starting to believe we can get plenty of, and to systematically train this model into existence because you train them one layer at a time. Can you talk a little bit more about, I've heard you'd be very articulate in terms of how you do this as a broader platform shift.

just even in terms of how pages are served versus generated or other aspects of that. Could you talk a little bit more about what's really happening right now more broadly in computer science with a shift to AI? Yeah, so you fast forward now a decade. The first five years was about reasoning the impact to computer science altogether. At the same time, we're developing new models of all kinds, right? And so CNNs to Resnets to RNNs, the LSTMs, to all kinds of new models. And scaling them larger and larger, making great strides. in perception models, particularly.

And of course, the transformer was a big deal. Burt was a big deal. All of you know that story well. Did you guys see like a step change in volume growth with transformers and Burt and such? Because it feels like having a architecture and an attention mechanism that allowed for scaling of these models really was also a kickstart in the industry.

Well, the ability for you to learn patterns and relationships from spatial as well as sequential data must be an architecture that's very effective. So I think on its first principles, you kind of think Transformers is going to be a big, big deal. Not only that, you could train it in parallel. And you can really scale this model up. And so that's very, very exciting.

I think that when Transformers first came out, we realized that there's a model now that overcame the limitations of RNNs and LSTMs, and we can now learn sequential data in a very large way. So that was very exciting. Bert was very exciting. We trained some of the early language models ourselves, and we saw very good results. But it wasn't until the combination of reinforcement learning human feedback, and of course

some of the breakthrough work that was done with retrieval models, dialogue managers that does the guard railing, it wasn't until some of all of those kind of pieces start to come together, that, of course, that we all enjoy chat GPT. And, Eli, the point that you're trying to make is the observation that computer programming has now been completely disrupted, that for the very first time in the history of computing, the language of programming a computer is human. Any human language, it doesn't even have to be grammatically correct.

And it's fairly incredible that anyone can program a computer now. And so that's a big deal. The fact that you program it differently, it writes different applications, what is the reach of this new computing model, apparently quite large. And it's the reason why Chad GPT is the fastest growing application in history. We had Alex Gravely, who was the chief architect for co-pilot on the show as well, and his favorite, like, obviously, it's, you know, very powerful to have sequential code prediction, but his favorite use cases of co-pilot have been, like, people telling him that they don't code, but now they do. Yeah, right. Which I think is very democratizing, as you said.

It's quite amazing that you could give chat GPTA problem to solve, and it reasons through it step by step, but yet it arrives at the wrong answer. On the one hand, on the other hand, you could tell it to write a program to solve the same problem. and it writes a program that solves the problem perfectly. The fact that there's an application that on the one hand reasons and tries to solve a problem and does a fairly good job at it, it's almost there. On the other hand, it can write a program altogether to solve that same problem. You've got to really wrap your head around the implication of this. So do you it as like the future world is some form of machine sentience?

First of all, I don't even know what that word means in a technical way. Yeah. I'm fairly sure that I'm sentient, less so today. So we have nerves for. Yeah, that's why I need nerds to crank me up here. I'm going to try too. I know. I know.

Today was a tough day. But I don't know. Do I believe that we now have a software that can reason through a problem for many, many types of problems, reason through a problem and solve and provide a solution or a program to systematically provide a solution on an ongoing basis? The answer is, yeah. Yeah. And then as you look forward to that world, how do you think about where you want to take in Vindia's lines of? business, but also you mentioned in the past that NVIDIA has done things like trained models, and you've done some

really interesting things there. Is that going to be an increasing part of what you do in the future, or are you mainly focused on the chip side, or how do you think about that mix of helping to push forward some research as well as being the underlying platform for the industry? Well, we're a computing platform company, and we have to go up the stack as far as we need to so that

developers can use it. And so the question is, what is the developer? And in the beginning, of course, a developer is somebody who controls their own operating system. And so in those days, we might only have to go as far up as device drivers or the layers slightly underneath that somehow to enable developers. But for scientific computing and all these different domains, the developer is actually using maybe a solver. And they need the algorithms of that domain to be somehow expressed in a way that could be accelerated, which is the reason why when we moved into these multi-domain physics problems, we realized that we have to develop the algorithms themselves.

Because the algorithms of solving a problem relates to the computer architecture that's underneath. And if the architecture is CPUs connected through MPI and Ethernet or whatever it is, that algorithm is surely very different than thousands of processors that's connected by a fabric inside one GPU and thousands of GPUs inside a data center. So obviously the algorithm has to be reframed and refactored. And so our company got very good at designing computer algorithms. It could be for particle physics or fluid dynamics. And then, of course, one day, it was related to deep learning and neural networks.

And QDNN is essentially a domain-specific language for accelerated deep learning. And so we've done that for deep neural nets. We've done that for computer graphics, ray tracing, that's called RTX. All of these different domain libraries really is about understanding the domain of science and then redesigning algorithms that make them go incredibly fast. Now, in the future, what's a developer? Well, I think in the future, a developer is likely going to be somebody who engages large language models of foundation models.

Now, if somebody could use ChatGBT or OpenAI's model, I really encourage that. And the reason for that is because they do such an incredible. incredible job. If somebody could use it through Microsoft, I'd really encourage that. If somebody could use it through Google, I would really encourage that. But if somebody needs to build a proprietary model for a domain, maybe create a new foundation model, and let's say the domain was proteins, or let's say the domain was chemicals, or let's say the domain was climate science, multi-physics. That foundation model is pretty niche, and it's not a small market, obviously, because the field of drug discovery is large,

the field of climate science is large, climate tech is large. However, it's not likely to be horizontally useful for every human. And so we might decide to go do something like a foundation model for 3D graphics, virtual worlds, because they're super important to us.

We might decide to build a foundation model for robotics because it's at an intersection of the things that we do very well. And even then, we'll probably take it as far up as necessary, but no further than that. We're not trying to be an AI model company. We're trying to help industries create AI models. Mostly, we're trying to help developers.

Yeah, that makes a lot of sense. You're basically following your customers up to whatever level they need you to. That's right. And then you hand it off to them at that proper point. We're trying to be as lazy as we can. Yeah. You know, do as little as possible as much as necessary. The first principles of computer science, we all know well, right?

To reject work as quickly as you can. to defer whatever work you are left for as long as you can until you could be rejected, and then whatever remains you have to do. We try to do as little as we can as much as possible. That's kind of the principles of the company. Laziness, the principles of the company, that's the takeaway. It's up on the wall outside, I think, too.

Be as lazy you can. I'm trying to square that with some of these very long-term commitments that the company makes, right? Like, Kuda is a very long-term bet. Yeah. And we met a decade ago when Envidio was valued at one one hundredth of its value today and was facing like activist investors and such. And it was probably like, let's say a little harder to make long-term bets. How do you balance the pressures of being a large public company and the opportunities of today with sort of architectural commitments or long-term bets and sort of think about that prioritization? investing in the future and being sustainable now are not in conflict with each other and so the challenge for all startup CEOs and for all CEOs is to find a way to be able to do what you believe in the fundamental core belief of the institution and to be able to afford doing it that is the purpose of the company and it's part conviction it's part skill making money is not a matter of conviction. Making money is a matter of skill. And it's a learnable skill. And it took me a long time to learn it. I'll admit that. I've been at this for 30 years. And for the first,

well, apparently for the first 20 years, since you went back 10 years, for the first 20 years, I was still trying to figure it out. But it's a skill. Learning how to make money, learning how to run a company efficiently. Those are all skills. And the company has to develop the skills. And the way that we ultimately do it is we ask ourselves, do we really believe it or not? And if we really believe in doing something, then it is the purpose of the enterprise. It's the singular purpose of the institution

to go pursue its beliefs. And the rest of it is up to all of the cleverness of the company and try to do our jobs well and build things that people want to buy and try to make it as cost-effective as possible and make the company as efficient. Those are all skills. The hard part, as it turns out,

is not the skill part. It took me a long time, but a lot of companies know how to make money, obviously. So the fact that there are more than one company that makes money suggests it's not that hard. If somebody else can do it, hark hard can be. And so singularly advancing a new computing model we call accelerated computing. And we believe that someday that on the one hand, accelerated computing can help us solve problems and tackle problems that normal computers can't and it exposed us to all of these amazing applications. like digital biology that I'm excited about today,

like climate change that we're excited about, like robotics and self-driving cars. If not for the fact that we're pursuing applications that were impossible with normal computers, why would we have discovered all of those things? Why would we have discovered artificial intelligence? Why would we be the workhorse of large language models? Because large language models are barely possible.

And if you are doing something that's barely possible, you call us. We're the horse you call to solve those problems. And so I love that aspect. I love the fact that we get to discover those future. On the other hand, we deeply believe that someday everything will be accelerated. And the reason for that is very clearly that the CPU will run its course. And there's a limit to how far you could scale general purpose computing.

And you'll always need it. You'll always need CPUs. But the type of applications that we're all going to run, acceleration is really the best way forward. And at our core, we believe that from day one. 30 years ago, that's the reason why we started the company. And so it's the true conviction. You have been enormously vindicated on this 30-year belief. You must have felt that conviction challenged at some point in 30 years of running the company and learning the skills to run the company.

What was the nearest death experience or the most concerned where you're like, maybe I'm not right? Or has that ever happened? I'm not right for the job? No, you're not right about accelerated computing and how important it will be. The second one's yes. First of all, I don't think anybody should assume that they're right for the job, and so you should be gut checking on that almost every day. To be clear, that wasn't the question.

But I'll more than have you answer that question. Did I ever believe that it was wrong? No. I believe that accelerated computing is absolutely the only way to solve problems that are impossible by definition if it's not wrong. Okay, axiomatically, yeah. And on the other hand, if you can solve problems that are impossible today, and someday you need that application to be broad-based,

would Accelerated Computing be the best approach? The answer is yes. Yeah. When do you think the CPU hits its limits? You mentioned that eventually you think everything will move over, or at least big chunks of the future will move over. Is that five years away, 10 years away? For certain applications, it happened 12 years ago, right?

Jeff Hinton and Jan Lecun and Andrew, right? Andrew Ang, they discovered that 12 years ago. It was the only way forward. And computer graphics, it's the only way forward. Yeah. Is the way that you organize and run the company changed as AI has gotten more and more prominent? Like, have you realigned aspects of the business around it, or how do you think about management in general in this environment

where things are changing so rapidly and there's so many exciting things happening in this area? You're asking a really good question, And maybe if I just take a step backwards, the company's architecture should not be generic. Every company in the world should not be built like the U.S. military. And in fact, if you look at every company's org chart in the world, they kind of look like the U.S. military. There's somebody on top, and then it comes down.

And yet the number of direct reports of CEOs are very few, and the direct reports of the people who are just learning how to manage first-level managers are very large. It's exactly the opposite of how it should probably be architected. You would think that the people that report to the CEO requires no management at all. And in fact, it's generally true. My direct reports are sophisticated. They're really talented.

They're incredibly good at their job. They're excellent leaders. They have great business acumen. They have excellent vision. They're incredible. Every single one of them. I guess that means you have more than the management book accepted six or seven or whatever. Yeah, I have 40 somewhat direct reports.

and no one-on-ones, no career coaching, you know. So what would you like to do with your life? Those are conversations you have with new college grads and early career. And we love those conversations, of course, and helping them shape their career and mentor them, give them access to new experiences. But at the executive staff level, we're organized so that we can pursue a whole lot of different things at the same time.

However, one of the most important things about a software company is you have to understand computer architecture. And one of the most important thing about computer architecture is you can only afford one. Just as some of the largest companies in the world only have two operating systems, the single largest company on the planet, only has two. How is it possible that so many companies have so many different computer architectures, and they have seven or eight or nine instruction sets that they're keeping around? We have one instruction set. We have one computer architecture, and we're super disciplined about that. And so where we need to be focused, we are, where we allow for innovation and discovery at the senior level, we allow that. So I think the company is tapered and organized in a way that is consistent with the nature of our work.

So that's the most important thing, and that's probably the takeaway for what I've learned building our company is there is no one generic architecture for every company. It should fit the function of the company, its purpose, and of course, the leadership style of the leaders. Yeah, I think that's a really important note that most people don't really realize is that a company should almost be a bespoke structure supporting the CEO and their staff and what the company is delivering to customers versus it's always the same thing. Exactly. Exactly. an architect something that makes sense for the leader and as well as the function. Yeah. When I was at Google, they had the famous 80-20-10

where it was like 80% is core, 20% is like core adjacent slash new stuff and then 10% was hyper-experimental. Do you have any frameworks or ways to think about that stuff? Or it's just kind of like, let's see what organically is used in terms of this generic platform that we've built with CUDA and other things that are built in to help support a lot of use cases and as they emerge, we say, okay, let's go support that new thing. I don't have any wise framework like that.

couple of things that our company is shaped and structured to do. There's one part, a very large part of our company is designed to build very, very complicated computers perfectly. And so that is one of its missions. That kind of architecture, that kind of organization is a invention and refinement organization. We have a whole bunch of skunk works, if you will. And the reason for that is because we're trying to invent things 10 years out that we're not exactly sure whether it's going to work or not. And there's a lot of adaptation, a lot of pivoting. And so, you know, our company actually has two different ways of working. One of them is rather organic, shape-shifting all the time. If a particular investment is not working out, we give up on it, move the resources

somewhere else. And so that's the agile part of the company. And then there's a part of the company that's not rigid, but it's really refined. And so these two systems have to work side by side. Can you talk a little bit about the H-100 next workhorse and what the most important innovations are and what the design and ship process for that looks like? I would say the big breakthrough for Hopper is recognizing that quantization, the numerical quantization, the numerical formats has a fair amount of innovation and ability to reduce because it's statistical in the first place. And now the question is what kind of models could be created and trained? And we believe that 8-bit floating point rather than, if you look at scientific computing today, 64-bit floating

point. And so just by breaking up 64 into 8, you could increase the performance of an AI supercomputer just by a factor of 8 by not doing 64-bit. So that's almost a factor of, if you will, a factor of 10 almost in just a couple of generation, just by recognizing that 64-bit flowing point wasn't necessary. And so one of the big things is that. The second thing is transformer. The transformer engine is so universal and so useful that it's possible for us to design a pipeline that is shaped for learning and inferencing transformers. And so those are probably the two biggest things. Otherwise, it's the largest chip the world's ever made. It's, you know, the fastest chip the world's ever made and super energy efficient and uses the fast memories

of the world's ever made. And then we connect a whole bunch of these things together. So that is fast and energy efficient. But those are all kind of brute forcey things. But the big architecture idea is FP8 and transformer engine. And when you think about then, so that's the big project refinement part of the company, we think about the more agile piece. What's the impossible application you're working on ticket today? That's 10 years out you think is likely to be important.

I'm sure there are a ton of them. There's a whole bunch of working on that don't work at the moment, but I've got a lot of confidence that will work. Okay, so for example, autonomous driving is still making progress, but I have every confidence that it will work. I have every confidence that a robotic foundation model will be discovered and that through expressing yourself using human language, you could cause a megatronic system of almost different types of limbs and agility to be able to figure out how to bend itself, articulate itself, to do a particular task. What do you think the blockers are to that today?

I have no idea, but I can't tell you. I'm just, I don't know. I don't know. Yeah, because we have to discover our way there. But one of the things that we do know is that we do know how to learn structure from unstructured information, language, images, and, of course, the next big thing is video. And if we could just watch video and learn the structure from the video we're watching, we might be able to learn how we articulate

and we might be able to generalize that and be a the articulation system for robots. And so I think the road signs, if you will, would suggest that the pieces are coming together. But when we get there, I have no idea. But I think it's probably less than, my guess is going to be less than 10 years, probably about five years,

and I think you're going to see some pretty amazing robots. It's so exciting. Yeah, there's some things along. those lines too. I guess like Paul Me from Google came out recently. That's sort of a step in that direction. And I guess that's still in the transformer architecture. And you mentioned the sort of transformer pipeline and sort of baking that into what you all are doing. Are there other new architectures on the AI side that you're watching or especially you think will develop into something especially interesting? Well, there's a whole bunch of derivatives of transformers. And they're all just

kind of generally called transformers. But that basic architecture is being refined and deltied and on the one hand. On the other hand, some of the stuff that we're really excited about that we did a lot of work in, we started with Ian Goodfell's work on GANS, and we did some really great work on a style transfer and high-resolution generation of images, which led to a whole bunch of work in variational auto-encoders, which then became, you know, if you will, a bit of a cousin of the diffusion models that came out. And so that entire path, we played a very large role in, And there's a whole bunch of derivative works that's going to come out of that between the ability to learn structure from a giant amount of data, whether there's video or, you know, multimodality learning is going to be a very big thing, of course. And then the next part of is generating content.

And if you can generate images and you can generate 2D and 3D images, why can't you generate proteins and chemicals and you can generate all kinds of stuff? There are almost no other entrepreneurs that have gone from three founders to CEO 30 years. $700 billion of market cap? What advice do you have for entrepreneurs that listen to the show? It's a really hard job. I don't mean the CEO job. Just building a company is hard. The two of you are associated with a lot of companies being formed from the very very beginning. There's nothing easy about building a startup. And I don't even understand it that anyone would build a startup twice. It is such a ordeal. Yeah, I try to talk people out of it. Like second time founders, because I started two companies. And the second time, like, are you sure you want to do this?

Oh, there's no question. You shouldn't do it. Yeah. There's no question you shouldn't do. It has to be some sort of like forgetting mechanism, like with having kids. You're like, oh, it wasn't that bad the first time. You got it. You got it. It's exactly right. For example, you have to forget how hard it was. And I don't know how I do it, but I just do.

I forget the pain and suffering that goes along with doing something. Once you achieve something, you just move on to the next thing. And once you achieve that, you move on to the next thing. And it's just like life. You put one step in front of the other. What advice, I'm reluctant to give them any advice. And the reason for that is this, almost any advice would probably discourage you from wanting to do it. I think ignorance is one of the superpowers of an entrepreneur, and you'll never get it again.

You'll never have it again. And so the thing I really love about our company is we're reinventing ourselves constantly. We're kind of entrepreneurs inside this company. And all the meetings that I go to are really startup meetings. And they're all painful. They're all painful because you're starting something from the ground up again. you have no momentum, you're basically at zero. It reminds me every single time how painful it is, but it's also so rewarding when you build

something and the people you built it for appreciate it, and somehow it made a difference. And then you combine that skill with some other skills and some other capability, and also you can do something even greater than that. On the one hand, I would tell them that building a company is extraordinarily rewarding and all the people you get to work with. genuinely true. On the other hand, the pain and suffering of doing it is unlike anything you could imagine. And so, you know, you're vulnerable, you're a superhero one day, you're a jerk the next day, you go through these cycles, and somehow you have to look beyond all of that and focus on

what you're trying to do. So I don't know if I gave them any wisdom aside from, if you're determined that you want to do it, don't wait too long, just go do it. Before you lose your ignorance? Yeah, before you lose your ignorance, if therefore there's one attribute, I would say, you have to be determined enough to stay with your conviction on the one hand. On the other hand, you can't be stubborn so that you can have agility so that you can continue to learn. And so somewhere in that balance of I believe in what I'm doing on the one hand, and I simultaneously believe that I could be wrong. On the other hand, that is weird. And you have to believe both equally hard. My firm's name conviction, you can have agility.

Okay. I'll start that as a candy brand. Yeah, yeah. And we've seen startup CEOs that are incredibly talented, and they're almost right. But they were so determined to be right. They forgot to be agile to learn along the way and pivot and adapt. And so I think that's on the one hand. And so that's one thing to remember. And then the other is resilience, which comes along with forgetting. You have to forget the pain, move on. And it's a little bit like coaches saying, don't worry about the last point. You just got your face kicked in.

And you missed a quarter, you know, like when I miss a quarter, when you mentioned crypto, my hand started the sweat. I know. My heart started to beat, you know, faster. Because I remember missing the quarter and when we missed a quarter during crypto, we missed it hard. Crypto was hard to predict, and we went from having no supply to too much. Who misses a quarter by $2 billion? That's a big number. Most of the time you hear CEOs miss it by $15 million.

Yeah. Not $2 billion. I think Sarah had a great point in terms of you built now really one of the marquee companies in the tech world, and you're really pushing forward what is potentially one of the most important ways of all time in technology, which is AI. 10 years from now, 20 years from now looking back, are there actually? any specific things that you want to accomplish, either through the context of the company or more broadly, or other things that you, looking back 20 years from now, you really hope, happen? That's a good question. And in fact, that's a good way to think.

The best way to think about what to do today is to go out into the distance, stand in the future and look back. You guys probably do the same. And so I'll go out 10 years and look back at, what did I wish I had done then? Then do it now. That's the answer. And so there are a couple of industries we really believe we can make a contribution to. One of them is health care and drug discovery. This is a problem that is computationally, numerically, insanely complex. The number of combinations is beyond the number of atoms in the universe. It's a very large problem space.

And we finally have the necessary tools to maybe chip away at that. And at the very minimum, we now have the ability to understand the language of and now potentially the meaning of amino acids and sequences and proteins and chemicals and such. And so if you can understand the structure, you can understand the language, you can understand the meaning of the problem space, you might have a chance of solving it. And so I think one, we're very excited about that. I'm really, really hoping that we go create a foundation model for multi-physics for climate science

And so that we can ask a questions, if these human factors and these human drivers, and we make these impact, what would happen to the Earth 10 years, 20, 30 years from now? It's an insanely complicated problem. Computationally, people have estimated it's anywhere from a billion to 10 billion to 100 billion times more computation than the fastest supercomputer on the planet today. That basically says we'll never get there. On the other hand, with artificial intelligence, we might have a real chance of reducing that computation by a billion times, 10 billion times. So I'm hoping that we have the opportunity in our generation to make a huge contribution to these two areas. So we're doing it. Earth 2 is our climate science system.

And Clara is our medical and healthcare system to understand better how to contribute in that space. I have one last question. That's as important as attacking the most computationally intensive, largest search spaces in the world to save humanity and the earth. From our audience, where did the leather jackets come from? My wife. Okay. So you don't know.

We have to ask your wife? I have no idea. Amazing. It remains a mystery. My wife, my daughter, they're always hunting for jackets for me. Most of the jackets, I have to admit, that I have hanging up are too fashion forward for me to carry out, you know. and so these are more modest ones but some of them are just

you have to actually be cool to wear them and so I just don't want to look out of place so no foundation models are about whoever eats this should not wear those jackets well thank you so much for doing this Jensen it's been an inspiring conversation thank you really enjoy it keep up to good work

thank you for listening to this week's episode of no priors follow no priors for new guests each week and let us know online what you think and who an AI you want to hear from. You can keep in touch with me and conviction by following at Serenormus. You can follow me on Twitter at Alad Gill. Thanks for listening. No Pryors is produced in partnership with Pob People. Special thanks to our team, Synthel Galdaneh and Pranavrudi, and the production team at Pod People, Alex McManus, Matt Saab, Amy Machado, Ashton, Ashton Carter, Danielle Roth, Carter,

Carter Wogan, and Billy Libby. Also, our parents, our children, the Academy, and and tyranny.m.L, just your average-friendly AGIW world government.