The Peterman Pod - Turing Award Winner: TPU vs GPU vs CPU, Computer Architecture, RISC vs CISC | David Patterson

Episode Date: July 13, 2026

David Patterson is a Turing Award winner famous for his contributions to computer architecture. I interviewed him about his past work, thoughts on GPU/TPUs and career advice from half a decade of experience.• My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/• The Kickstarter page for it: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-donePodcast links:• YouTube: https://youtu.be/Pn4ZwlEh5nw• Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835• Transcript: https://www.developing.dev/p/turing-award-winner-tpu-vs-gpu-vs?r=n49kyTimestamps:(00:00) Intro(00:42) RISC vs CISC(12:51) Compilers(17:38) GPUs(23:07) GPU vs TPU vs CPU(32:12) Is Moores law dead?(38:04) GPU benchmarks(41:40) How to have a bad career(49:59) Courage and optimism(55:56) Advice for his younger self(58:15) OutroWhere to find David:• Wikipedia: https://en.wikipedia.org/wiki/David_Patterson_(computer_scientist)Where to find Ryan:• Newsletter: https://www.developing.dev/• X/Twitter: https://x.com/ryanlpeterman• LinkedIn: https://www.linkedin.com/in/ryanlpeterman/• Threads: https://www.threads.com/@ryanlpeterman• Instagram: https://www.instagram.com/ryanlpeterman• TikTok: https://www.tiktok.com/@ryanlpetermanReferenced in this episode:• AlexNet paper: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf• David Patterson's “How to Have a Bad Career” talk: https://www.youtube.com/watch?v=Rn1w4MRHIhc• “Life Lessons from the First Half-Century of My Career”: https://cacm.acm.org/opinion/life-lessons-from-the-first-half-century-of-my-career/• The 7 Habits of Highly Effective People (book): https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People• Working (book): https://en.wikipedia.org/wiki/Working_(Terkel_book)

Well, if Moore's law puts a lot more transitions in a chip, how come they're not getting hotter and harder? This is David Patterson, Turing Award winner famous for his contributions to computer architecture, and we discussed the historic risk versus CISC debate. And he said, in CISC 1, well, that's a pretty myopic. Everything is energy bound now. Register allocation by the compiler was so poor. And his thoughts on modern GPU and TPU architecture. What's the high-level differences between?

a CPU, a GPU, and a TPU. What they realized for machine learning and AI, here's the full episode. Maybe you could explain risk versus CISC. Kind of like what that debate was, and why was it controversial? What happened is in kind of in the microprocessor was invented in the 1970s.

But in the beginning, it was basically a toy. It would be something that would go in microwaves and things like that. Those of us who believed in Moore's Law believed eventually the microprocessor would be the way we did all computing with the doubling of transistors every year or two. Eventually, they'd be enough transistors that on one of these microprocessors would be a serious computer. So the people who are designing microprocessors like Intel and Texas instruments weren't really computer architects. So they just imitated

what the big companies did. So leading companies like IBM, or at the time, digital Equipment Corporation, IBM for mainframes, digital equipment for what we're called mini computers, they kind of drove the architecture, the instruction set design. So what they were doing with Moore's law is building more and more sophisticated instructions. They had, you know, mini computers would be the size of a refrigerator or a couple of refrigerators, and mainframes would be the size of many of those. So that's what they were doing with more hardens sources. and the philosophy at the time was that by having a more sophisticated instruction set, you kind of raised the level of abstraction closer to the software,

so that had some inherent benefits over having something at a lower level. Now, some of us who were in that area, like my friend John Hennessy at Stanford and I, we thought that wasn't necessarily the right thing to do. Compilers would go from programming language, down to the instruction set. So why couldn't compilers hold that? And then the question is,

what's the right instruction set for this emerging microprocessor? And the prevailing wisdom of these more sophisticated instruction sets, complex instruction sets, and if we think of it like a vocabulary, it was like a, having lots of polysyllabic words in your vocabulary. And the alternative was going to be, what we call the reduced instruction set computer,

was having lots of reduced instructions, like monosyllabic words. So you might guess that for a program to execute these instructions, if they're simpler, they'd take more of them. And if they're sophisticated, they take fewer, but the sophisticated ones might take longer to run. So kind of the question came down to what was that ratio. And in the beginning, in the 1980s, when these debates were happening,

there was this kind of very vociferous debates, partly of what those ratios are doing to do. But the big part was kind of philosophical, weren't you doing damage to the software industry by lowering the instruction set level. So the gap between the programming languages and the instruction set was larger. That was kind of the ferocity of the debates. Well, after the dust settled a few years later and we started getting numbers, it turned out you needed like maybe 30%, 40% more simple instructions to execute in a program.

But you can run them four or five times faster. So the net was, you know, or five times faster. say, so three or four X speed up was the potential for risk over CISC. You mentioned that, I guess, philosophical difference, like that gap. Why would that be controversial? I mean, unfortunately, I'd say computer architecture in the 1970s and even 1980s, a lot of it was kind of people designed this by using their intuition or using their gut. their feelings was this is the right strings to do. And even the textbooks of the time were like

catalogs. Here's a computer and just list all its features. And here's another computer listing all its features. So it's pretty dissatisfying what was going on because these debates, it would seem like we ought to be able to sign this scientifically with numbers that are on there. But absence of that, it was more like a philosophical debate, like how many angels on the head of a pin, right? You could you could have arguments qualitatively, but you couldn't settle the arguments. And because there wasn't ways to settle the arguments quantitatively, you know, people would just argue about it. Would you say that, you know, risk versus CISC, did one side win the war? Yeah, I was just reading there's some guy online who revisited it and he said, in CISC 1.

Well, that's a pretty myopic view. is in the PC era, because of the importance of distributing software in binary, so once the X-86 was established and people, the PCs, would ship software in binary, is that was very hard to overcome. That was a huge impediment to changing the instruction set. So PCs are defined by the X-86 architecture largely. But also in the 1980s, there's this company in England. that wanted to do a personal computer.

It was the ACORN personal computer, and they decided they needed their own instruction set of architecture to do that, their own chip. They run the chips that had available at the time they were doing it weren't fast enough. And they were influenced by the papers that we did at Berkeley, and so they built what they called the ACORN risk machine. And then one of the benefits of this kind of reduced instruction set

is that it could be simpler. It would take less resources, and take less energy to execute. And then several years later, when Apple was looking for a microprocessor that could power one of their personal devices that this one was called the early forerunner of the iPhone called the Newton,

they came to this company and said, boy, you really like it. Let's get rid of the acorn name. So they renamed it, the Acorn Risk Machine, arm, and they re-christened it the advanced risk machine to get rid of the acorn. And then Apple used it in the Newton. Now, the Newton wasn't a commercial success,

but it demonstrated the benefits for risk architecture for mobile devices. So the Nokia came along just a few years later with their GDM cellular phone, which is one of the first popular ones, and they embraced Arm. And so ever since, you know, Arm has dominated all the mobile devices. So I think I just checked, there's been 350. billion arm processors, microprocessors with arm technology in it today. So it's like today, 99% of all processors and computers are a risk. Even in personal computers, Apple switched over to arm from the X86 architecture. So even PCs, there's risk architecture is, you know,

is significant. And it's trying to start to get in the cloud. The cloud's largely been defined. by the X-Saddy-6 server architectures, but Amazon, and I think Amazon, Microsoft, and Google all have their own development of RRM processors, so RIM is becoming, or risk processor becoming more popular in the cloud. So right now, I'd say the X-Satty-6 architecture market is shrinking while the risk architecture market is, you know, growing leaps and bounce. Well, how could someone say SISC 1 when you mentioned like 99%? Yeah, well, that's just a, it. If they're doing something in history and they define computers as personal computers and maybe servers, and they go up through around 2000, if the story ended then, you could, well, it looks like a Cisco one. But, you know, once we get into this post-PC era, I don't understand how somebody would reach that conclusion.

You mentioned the energy expenditure. So like, you know, risk makes sense in places or maybe on mobile devices, things like that. Well, even in the cloud, I mean, everybody cares about, we're, everything is energy bound now. So it's, if, you know, these days, of course, you're not dealing with millions of transistors, but billions of transistors. So it matters somewhat less today. There's so many things going on that you can hide that. I mean, even what the x.706 architecture did. is to compete is that it translated the X-86 instructions in hardware into risk instructions.

And so you had to pay that extra overhead of that translation step to get risk instructions, and then you could any good ideas that the risk people had that the X-86 could do. But it was worth it financially for that extra overhead because of the value of the PC software base. So it made a lot of sense for Intel to do that, and they did that in their early, thousands, you know, it was a great commercial idea. Is there any niche use case where CISC makes sense? Is there any, you know, is this an engineering tradeoff or CISC is objectively where? Yeah.

So if we want to go kind of one level deeper into all of this, the what was actually going on in, you know, in designing computers, the hard part is the control. And so what happened is in the beginning, it was kind of. control was kind of ad hoc, you would figure out, you put the gates together to make it to work. One of the computing pioneers, Maurice Welk, figured out a more elegant way to design control. And he said, well, we could just list all the control signals as the output of a memory, and we could have something would keep track of where we were in the memory, and to issue those control signals.

and he called this effort, the instruction, or the control signals you could think of instructions. So he called that a micro-instruction. And he called the programming of this instruction, it's microprogramming. So where technology was in the 1960s, that made a fair amount of sense. And so IBM built these so-called microprogram computers.

So it was basically an interpreter with very simple instructions that would, interpret this much more sophisticated instruction set above it. But you would play this interpretation overhead. And classically, you know, in computer science, interpreting versus compiling is something like a factor of five or ten. But given, you know, the latencies of the memory technologies and the possibility of doing this out of read-only memory, it made sense up until the 60s and 70s. And then, but then the kind of the question came up as came around 1980 is like, is this still a good idea? should we have this microcode interpreter inside there?

And so the alternative is to think of, well, rather than we've got this microcode interpreter in there, why don't we just compile directly into those instructions? And that's pretty close to the risk ideas. The micro-instructions themselves used to be, you know, like 100 bits wide and really complicated. So if you make them not quite so long, you make them kind of more natural,

still we could skip the interpretation set. So with, you know, going this level deeper, kind of the question would be today, would people invent an instruction set that was so sophisticated and needed a microcode interpreter? And probably they wouldn't do that. I mean, you could do it. Nothing prevents you from doing it,

but you wouldn't want to design an instruction set that was forced to use a microcard interpreter. It might make sense in some very tiny applications, possibly where you have, you know, tens of thousands of transistors. Maybe this would a microcode interpreter work, but I think nobody today, I don't think anybody's invented an instruction set in the last 20 years that has anything that would need a microcoded interpreter.

You mentioned the compiler multiple times here, and it seems like that's a critical piece that kind of makes risk work so well. Can you explain the role of the compiler and how it manages the relationship between, software and hardware? Well, you're writing in a programming language like C or C++ or maybe Python, but the quality of the code that gets generated is up to the compiler. If you, in a specific example, is that it's useful when you construct computers to have registers, and those registers are actually kind of visible in the assembly language or the machine language programming. There's, it can be

eight, 16, or 32, these registers for the people, program. without a little level to use. Well, it used to be very difficult for compilers to allocate registers efficiently. Computers just weren't fast enough. We didn't have the algorithms that we could look at a section of code or subroutine or something and say,

how can we most efficiently do registers? In fact, the C programming language, which was invented to do systems programming in C, before that to write an operating system, an operating system, people wrote them an assembly language, believe it or not. But the Unix people showed, Ken Thompson, Dennis Richie showed that if we had a low-level language, pretty low-level language, we get the benefits of writing in something that's much easier for humans to understand and debug. Because register allocation by the compiler was so poor, they had to put in the ability for the programmer

to give hints of what variables should go on registers. It was just easier for them to say, have the programmer step in and say, if the machine had eight registers, I want these six variables to be in these registers. Don't leave them in memory because it runs so much slower. Registers are so much faster. So a big part of the CISC, the RISC argument, was the compiler algorithms are getting better.

They could handle these low-level instructions. They could allocate registers efficiently. And that was another reason in these debates about why it made sense to have a simpler architecture. What we did in the risk architectures, well, if registers are really important, register allocation is really important, one way to make it easier for the compiler was just to have a lot more of them. So typically the CISC architectures of times would have eight or maybe 16 registers, so we put it in 32, and that was one of the arguments, right?

Well, if it's hard to efficiently use a small number, let's give them plenty. So even if it wasn't that good, there'll be enough registers so most of the time. and registers weren't that much more expensive to include in machines because of Morse law. I saw somewhere in one of the talks that he'd given that somehow the compiler, it's more easy for it to optimize the code for risk, but in Cisco there were these complex, bigger ones, and it almost never used them. Is that a shortcoming of the compiler? So what happened, if we go back in the compiler days,

There's this argument is that by having more sophisticated instructions, this would raise the level of extraction. There would be a smaller gap and they'll make it easy to the compiler. But that was a philosophical argument. It wasn't something that necessarily compiler people could work, compiler people weren't making that argument. It was the architects who were making that argument. And as it turned out, when we looked at the programs when we were doing the early research versus risk, is the compilers didn't really use those instructions. Often the compiler riders, the architects would come up with a sophisticated instruction and the compiler writers would say, well, we don't need that. In fact, we found examples

where like for a procedure entry, there'd be a special instruction built in to do all this work that what the architects thought the compiler's run in, and the compiler's design is say, well, we don't need that. It's faster. It's actually faster to use it with separate instructions to use your sophisticated instruction. And so we found a bunch of examples like that. So, you pay the extra overhead of the microcode interpreter to have these sophisticated instructions, and then the compiler doesn't even use them. So this is kind of a nonsensical situation that we're in. And, you know, this is kind of like, you know, why are there startups or why are their scientific, you know, breaking points like that?

As we looked at all the technologies with Moore's Law, you know, ideas like caches, where compilers were in using sophisticated instructions, the ability to allocate register efficiently, it made sense to change the directions away from these microcoded instruction set architectures to these simple architectures. When we talk about instruction sets, I mean, we're kind of assuming that we're talking about these general purpose computer,

are these CPUs? And I know GPUs are kind of talked about a lot, and maybe there's other forms of computing. Are there instruction sets for those types of machines as well? So what happened is around 2000, when GPUs came along, and these were what we called domain-specific architectures. So a GPU is a graphics processing unit. It had one job.

It didn't need to do everything that a general purpose processor needs to. It didn't have to support virtual memory. It didn't have to support a compiler, which is a pretty radical idea for architectures. It was just for graphics. And so around 2000 is when NVIDIA and the other GPUs out there because they were trying to do, you know, that give you the graphics for games and give you the graphics for movies. So it was a niche product. So what happened kind of in the computer industry is it was driven by Moore's Law, which I've

mentioned many times, and also this lesser-known law called the Nard scaling, which is because kind of an interesting question, well, if Moore's Law puts a lot more transitions on a chip, how come they're not getting hotter and hotter as we double the number of transistors every year? And the reason was this observation made by Bob Denard that what, as you added more transistors, people would also lower the threshold voltage, which is a distinction between a zero and one. And that had like a squared effect. So you would end up doubling them transistors, but you'd lower the threshold voltage.

So microprocessors stayed at like 20 or 30 watts. In fact, we did a, John and I eventually did a textbook. and I was just checking that came out in 1990. We didn't even talk about power as an issue for the first three editions. The third one came out in 2000. Power wasn't even a topic because dinard scaling was going on. And so microprocessors just stayed in the tens of watts even as they got faster and faster. What happened is about 2005 or so, Denard scaling stopped working.

And that was a shock. And Intel actually had a microprocessor that failed. one of their generations because they just couldn't get the power down enough. It was too hot to be able to do that. And then so then what that forced us to go to multicore is that before that it was easiest for the programmer for everybody. If there was one very sophisticated processor that did everything, but we couldn't do that anymore. So it went from one sophisticated processor to two and then four and then eight simpler processors.

Then it was up to the programmer to deliver. on the potential of Moore's law by paralyzing their code. So we stayed that way for about another 10 years, and then Moore's law started slowing down. So the general purpose microprocessor was barely improving. It got a little better, but it wasn't getting dramatically better. In the 1980s, 1990s, 2000s, you have a laptop, and your friend's laptop would be like four times as fast

is yours. And, you know, you were jealous. So you would throw away perfectly good hardware because your friend's thing was so much faster because of the rapid change in performance. Well, that all ended in the 2010s where you never, you wouldn't throw away a laptop because the new ones were hardly that much faster than the old ones. You'd throw them away when they break and slow them. Nevertheless, programmers were used to this dramatic improvement in performance every few years because they could add more features to their software. and stuff like that. So what were architects going to do?

They'd already done the multicore trick in 2005 or so. So in about 2015, the idea was we would do domain-specific architectures like the GPUs. So if you tell me I only have to run a narrow class of programs and I don't have to necessarily run all the operating systems, everything else, well, yes, I could shuffle those resources and do something much more efficient for something well. So you could do some things well and as other things either you do poorly or not even at all. So then the question is, okay, what domain? So just coincidentally, where we were technologically, right around 2012 to 2015 is when machine learning AI burst on the scene. And so that is evident

what domain should we do? And the domain was machine learning AI. So just coincidentally where we were technologically. This new domain came along. And then I would say, you know, I work for Google, but I think it's fair to say Google was the one who really saw. It was certainly the first big company who understood the potential of machine learning AI and bet that they or feared that they were going to be swamped with demand and needed to do custom hardware. So the translet tensor processing unit at GPU that Google debuted in 2016 really kind of shocked the world and got people to realize that we should be designing hardware for machine learning. And we could continue to improve performance dramatically for that one domain. What's the high-level differences between

a CPU, a GPU, and a TPU? The CPU has got to be this general purpose thing. And basically, even today, they have these general purpose cores that each core is very similar to what the instruction sets or used to be like 20 years before that. It's not a surprising design and it's just got lots of cores. And modern ones today could have 50 or 100 cores in there. So that's the feature. For graphics, what they decided to do,

which to do the graphics job is they need to get a lot of performance of their memory systems. So they went to multi-threading. So they have hardware threads that you would send a memory request out and the hardware would switch over to do something else while the memory is coming up. So it's this highly threaded architecture and that's what they developed and could run well for graphics. And graphics, it turns out, doesn't need very, doesn't need powerful floating point. It doesn't need wide floating points. So 32 bit floating point is plenty for graphics.

even 16-bit. So they were pushing graphics with 16-32-bit floating point in this multi-threaded kind of architecture. And because it was kind of its own unique thing, it has its own set of terminology all to itself. It wasn't kind of out of the main branch of computer architecture. So when you look at our textbooks, we have kind of like a Rosetta Stone,

which says, here's the terms that Nvidia uses describe GP2, is this is what it means in kind of normal, well, if normal, in the mainstream processor design. So they were this niche product that happened to do pretty fast, single precision floating point and even half precision floating point. And they were pretty cheap. There were like hundreds of dollars. So when people, so kind of, but some people were thinking,

boy, for some applications, if I could turn my program into like, pretend that it was doing creating an image if I could turn my problem into image generation I could use these pretty cheap GPUs which had very good floating point performance per dollar compared to anything else and so people started playing around with that and the founder of CEO Jensen Wang really liked that idea so in 2006 he funded an effort to create a programming language that would kind of handle this multi-threaded hardware architecture that's for graphics to make it easier to program. And so that led to the invention of Kuda, which I can't remember its acronym.

But it's really a proprietary programming language for the multi-threaded GPU architecture to make it kind of easier to program. It was a C-like language, but you can't just compile C-programs and run it. But it was C-like. But people actually liked it. It was certainly much better than trying to turn your program into an image generation. But people like it. But Gens Wying, his vision was try to get these teenagers in the basement who are playing the games

to learn how to program these things. And he had a couple of markets that he was interested in, like some of the Department of Energy labs, like fluid fluid and some of the things like, you know, some. And he would build special purpose libraries that can handle that domain. And then you use his GPUs to do that kind of. special purpose computing, but breaking out of graphics. But that's kind of the heritage there. Now, for the TPU program, and so what's happened, people started, right from the very

beginning, people started using GPUs because they had much better single precision floating point performance than the CPUs. They had a lot more processors on them. So they had the potentially much more performance. So they're kind of one of the breakthrough moments, moments in, it was in 2012. when within the machine learning community, this neural networking piece, which had a few advocates,

but a lot of people didn't believe in it, they went into a competition to see who could do the best image recognition. In 2012, the so-called AlexNet beat all the competitions. This was this historic moment in machine learning and neural networking. And once they did it, and the guy who did that had taken the coup of course, at the University of Toronto learned how to use it.

And so he says, well, as long as I'm doing this, I'll do it on a GPU. And so he was able to explore a lot more space on this cheap, fast GPU. So he entered it in the competition 2012. And he was the only one using neural networks, and he crushed the competition. And within a few years, everybody switched over. And not only they were doing neural networks, but they're using GPUs. So GPUs have this heritage of the graphics engine. but more programmable and started getting used for gene learning.

When Google came along, they decided, you know, it was a clean slate. They didn't care about graphics. So at the heart of neural networking is a matrix multiply. That's the thing. So they designed a processor for at the time, microprocessor at the time, had a giant matrix multiplying unit. That was the main thing. And then they threw a bunch of stuff out that they didn't need.

So a lot of general purpose computing is what's on the chip is maybe three levels of caches to try and have, so you don't spend it all your time going to the relatively slow memory. Well, for machine learning, they knew when their memory access were so they could schedule that. So they didn't. A hardware cache didn't make any sense. They would just have a memory that the software understood would transfer in time. So those are some of the innovations. Also, they innovated on the floating point format. You didn't need hot scientific computing.

Cares a lot about precision. They have, you know, most of it's done in 64-bit floating point where the exponent is, you know, less than 10 bits and most of it is the precision. You know, the fraction can be 50-some bits. Well, what they realized for machine learning and AI, they don't need all that precision. They needed the range.

So Google did the first floating point format where the exponent was bigger than the fraction. That was a radical idea that so-called Brain Float 16, the part of Google that was doing this was the brain research group. So they brought out this architecture. They had this narrow floating point format. They had big matrix multiply in it. It only had one processor in it, unlike the other ones. It was just dedicated between learning, and it just kind of blew the doors off of everybody in the field.

It was, you know, like 30 times better at inference than the, contemporary GPU and 80 times better than CPU by having this dedicated one. So when Google made this announcement at their annual retreat a year or so after they had deployed it inside, it just shook everybody up. Intel started buying companies. You know, Nvidia started modifying the design for to be much more hemp to machine learning. And then a bunch of other competitors started hyperscalers started doing their own effort. So I think that was the watershed, I think the TPU announcement was the watershed moment there. So it's almost like levels of specialization, like the CPU is the most general. Yeah, if we could still, you know, if we still had DeNard scaling,

if, you know, Morris Law and DeNard scaling, where we would be today with general purpose processes, we should have 100 terahertz, you know, microprocessors. If we could build 100 terahertz microprocessors, That's what we do. GPUs would still be a niche. If we could do that, it raises all boats. That would be fantastic if we could do that. But that's long in the history. We haven't been able to do that for 20 years.

And so there were probably two or three gigahertz microprocessors in 2005, and that's kind of where they barely improved today. So we can't do that. So there's still areas where it's important to have CPUs. They're the right, you know, you need operating systems, compilers and things. They're the right solution. But things have been specialized. And then now in terms of industry terms, huge emphasis is being put into the AI accelerators

or just AI in general is where the money is being invested. And so the question for any processor designers was, how does my processor design fit into this AI universe? and what should I do? How should I change it for this important application? So you mentioned that kind of Moore's Law is slowing down. And I watched this other talk from Jim Keller saying that Moore's Law isn't dead, but is it controversial to say that it's not to real engineers. Morse law is very simple.

It says in a chip, the number of transistors will double, originally said every year, and then he amended it to every two years. Just look, do the chips, are the number of transistors doubled? And no. Now, I think what people assume that means if we are no longer on Morris Law, that technology is not improving. That's not the same thing. It's not improving at the rate that more projected. And what was amazing about Morris Law, which lasted 50 years, is that it guided the investment of semiconductor manufacturing. We need to deliver on doubling transitions every year or two. How are we going to do that?

How are we going to build the equipment to do that? So it was a guideline for the whole industry, which was kind of remarkable. So what we are doing now is there's pieces of the technology to get better, and there's pieces that don't improve at all. So one of the big pieces, important pieces on a chip is the state. static RAM, S-RAM. So that's hardly improving at all. Logic, gates still continue to improve. The gates are getting better. So if you're doing adders or multipliers, those are getting there. So pieces, it's not uniform improvement anymore. And we're also going to more exotic

packaging to be able to deliver it. So people not, you know, forever it was, a single ship was the best way to package everything. fits on one chip and that's what we're going to build. And now there's these ideas of chiplets or packaging multiple chips together. The latest GPUs and I think the latest TPUs has actually two what are called full reticle design, dyes, package, put into a package. And so a full reticle design is the maximum you can build on a, well, on a semiconductor. They have a step and repeat motor and there's, you know, in the old days you'd have many chips inside of that. Now the just one. So two maximum radical designs form a node. So they're using packaging. So if you look

from outside, you can say, well, look, look how many transistors. It's, quote, Moore's Law is continuing. But if you see what's inside the chip, you know, that has tapered off. And I think part of it is for a fair amount of the industry, if you were in a semiconductor manufacturer, and people ask you what you do is I make Moore's Law. I sustain Moore's Law, what I do. And if you've been doing that for decades and somebody says Morris Law is over, it's like my career is over. So I think there's an emotional side of it. But, you know, just look at the data. The data doesn't back up with Keller says.

But I also didn't say that the technology is not improving. The technology is continuing to improve. And specifically, the main specific architectures. But if you look at the general purpose architectures or even other memory technologies, you know, it used to be the D-RAMS would improve by a factor of foreign density every three years like clockwork, and now it's maybe 10 years between factors far. So you can see plenty of evidence that Moore's Law no longer applies. Is there some new version or some analog to Moore's Law that is kind of guiding, you know, microprocessor design and architecture these days? I guess a question is both Nvidia,

and Google are continuing to deliver much faster processors for machine learning, tremendously better. What are they doing? Well, part of it is what I said about packaging, you know, to be able to get more transistors and you keep them closer together because distance matters. Part of it is innovating on the flooding point formats. So, you know, unlike the supercomputers of 64-bit, it's not even 64-bit, not even 32-bit. not even 16-bit, but 8-bit and 4-bit floating point is going on.

So narrowing of the data types. The, you know, having kind of what's they're called the matrix multiply instructions are these powerful units that are put in there that can do special purpose applications that are very important, making those bigger and faster. So those are the things that are going on. But we don't have this simplifying guideline kind of underlying all this. You have to be aware of each piece of the technology, gauge how fast it's improving, whether they can deliver on what's going on, and then you assemble that together and make your bets.

There's a paper that we've just got approved that I think we're going to put on an archives, which is talking about the Google TV line, and pretty remarkable. the Google TPU line, particularly for training, has stayed pretty constant in the basic architecture design. Things have gotten bigger and faster, but if you look at the design of it, going back to the first training TPU, that block diagram still works all these, you know, a decade later. So, you know, the people who designed that in whatever, 2015 or two, did a really great job. but the main components of a big matrix multiply unit, the so-called high bandwidth memory,

a vector unit to go with the matrix unit, the basically building blocks are still there. I think in your career with the CPUs, benchmarks were a huge, they played a huge role in measuring which computer architectures were performant and not. In this new space of floating point operations for AI, is there a benchmark?

that people use for GPUs and can you also apply it for TPUs? I was, you know, I and some friends were helped involved in called the ML Perf effort. So it was inspired by the spec CPU effort was the spec benchmarks that they were competing companies and they'd all make claims about theirs was better than the others and they realized that wasn't good for the industry, so they agreed on a set of benchmarks.

So the ML Perf effort is this run now but what's called the MLComberf. is an attempt to do that. If we're going to do this comparisons, let's not argue about what the benchmarks are. So that's a serious effort in that. Kind of interestingly, what's happened, I'd say, is because there's two pieces to design.

There's the machine learning libraries that are to implement a lot of features that you need for these applications. and the libraries will even be rewritten for specific applications rather than just having general libraries. This has turned out to be a big advantage for Nvidia because they have a large corporation with lots of people that are available, lots of engineers who could build these libraries for them.

So it's turned out not quite a, it's not a neutral evaluation, right? It's the architecture plus the, the libraries that go with it and the compiler too. But specifically the libraries that you tailor each time. So, NVIDIA brings out, when they announce the new architecture, they create a new set of, they modify the libraries to run that really well or to run applications really well. So that's a powerful combination.

Kind of in business terms, people refer to NVIDIA's having this Kuda moat. And part of it is Kuda, the programming language, but a big part of it is the libraries that Nvidia makes. So this has made it difficult for startups to be able to compete with Nvidia, partly because, you know, they didn't put enough emphasis in the software, and partly because, you know, Nvidia just has many more engineers that they do to be able to tailor the libraries. So it's the libraries plus the architecture.

This is powerful advantage why most people do things on GPUs. Now, Google has been able to develop their own libraries. They don't have as many engineers. They use compilers more than I think Nvidia does. So they have a set of libraries that they can do. But up until recently, Google has, it's all been internal. It's just for Google to use or you could use it being the cloud. But these startups have to have a difficulty.

As a result, that's why the MLPIRF has. hasn't been as popular. Not everybody runs them because Vivia runs them really well, really better than everybody else. And the startups have a hard time showing off what they can do given they don't have the engineering effort going into the libraries that you need to do well in the ML Perf. You gave this popular talk about how to have a bad career and it's kind of the negation of advice to kind of have a good career. And it's just the, wondering if you could kind of summarize maybe the top three things that you kind of think people should take away. Yeah. So yeah, well, there's been a few things. So the story is, early in my career,

my friend and I said, how can we teach grad students how to give a good talk? And we thought it'd be funny to explain how to give a bad talk. And then if you didn't want to give a bad talk, so this is how to do it badly. And if you don't, here's the things you do not to do it badly. And so then I later did how to have a bad career. That's pretty much focused towards, academia, you know, so for researchers to be able to do that. I later did a how to have a bad, how to build a bad research center, how to build a bad research lab. And then recently I've been giving how to give AI a bad carbon footprint. That's my latest one. But I think the thing that might be more relevant is at the end of my talks, I would kind of reflect in my career

and talk about lessons learned. So I've written a paper called Lesson, I think it's Lessons life lessons from the first half century of my career. I wrote that recently. And so those are divided up into kind of career advice and personal advice there. On the personal side, I'd say if you have a family, you know, make sure your family's first, keep your family first. The technology we've invented makes it really easy for your, you know, to take your work home with you and not pay attention to your family. I had a, when I was first here at Berkeley, I gave a senior faculty member arrived home, dropped him off late at night coming up to Silicon Valley. And he said, well, Dave, if I had to do it all over again, I wish I'd spent more time with

the family. And I never wanted to say that. And nobody in the deathbed says, you know, I wish I'd spent more time in the office. So you got to, you know, whenever, it's easy to kind of, you know, let your career overcome the needs of your family. But you've got to just remember that. I would say in my life, kind of growing up in the 50s, you know, the idea was to be happy had to be wealthy. But those are actually two different goals, wealthy and happiness. So I always made decisions towards happiness versus wealth. And I felt very good about that.

And even now, like, why would I pick something that made me wealthy and unhappy? Why would you do that? And so, and in the field that we're in, it turned out there was a lot of wealth to go with it. So, you know, optimizing happiness didn't mean you had to suffer beyond that. I think it's important to have fun, personally. I think when you're a kid, you don't have to tell kids to play. But as you get so busy, you don't think you have time to have fun. But, you know, you only get to do this once.

So I, right now, I play soccer. I run my bicycle to the interview. I lift weights. I body surf. I, you know, do things with my wife and family and my son. So it's important to have fun. I think on the career side is one of the pieces of advice is by a convent. He wrote this book, The Habits of Very Effective People, I think.

And he has a little quadrant, and he divides it of urgent and not urgent. important and unimportant. And kind of, there's so many things on our technology like email and texting for you to focus on the urgent things, but you really shouldn't be spending a lot of time on the unimportant urgent things. And it takes self-discipline to set aside time for the important non-urgent things. But if you don't block that out, you can just not have time to do anything. I get to see other people's calendars at Google, and there's managers that every half hour from 8,000, to six five days a week are scheduled. And I don't see how you have time to think and reflect on things like that. Other career advice things is I remember I kind of woke up one morning and it was like

God spoke to me. I was like thunderstruck. And it says it's not how many things you start, it's how many things you finish. It seems relatively obvious, but that's not the way I was acting. I had many things going on. But after that, it was like there's one main thing I'm doing at a time. So when Hennessy and I wrote a textbook, that was the main thing. When I was department here, head here, that was the main thing I did. I would do some other little things there. And John Hennessy, he wrote a book to a kind of a career advice book based on his presidency.

And one of the things he said in there, you know, you're only going to be remembered for the five or six things you've done in your life, not for the hundreds of little things. So to give yourself a chance to have some things you're really proud of, it's better to concentrate on a few of them, hoping that some of them will turn out to be a big deal rather than scatter yourself to many things. But if you're interested, you can, yeah, if you look for life lessons, David Patterson, first half century,

you can see the whole list of, I think there's 16 lessons altogether. You mentioned that you didn't want to reflect back on your life and feel like you didn't have enough family time. And yeah, I don't think I've ever heard anyone say the opposite. Why do you think that is? Why is it that everyone looks back on their life and they never regret, you know, a lot of family time? I imagine there's got to be at least one person that says I spent too much time with my family. Like we're suffered? Yeah.

Well, you know, what's, you know, what's, it's pretty philosophical. I mean, what's, what's, what's life all about? What's success, right? You have to, you have to figure out what that means for you. I mean, if you, if you have, I don't know, if you have financial goals of being a, you know, a millionaire or a billionaire or something like that. And that's how you judge your life. Okay. Good luck. You know, it's pretty hard to make it. It's pretty hard to do. But I think it's the, you know, when I was finishing my PhD, I read this book by Studs Terkel called Working, where he interviewed all these people in his careers and they looked back and what they liked and what they didn't like, what they felt about the careers. And what I got out of it, the people who worked with people like ministers or teachers or doctors felt really good about what they do with their careers. And the people who did more ephemeral stuff, you know, like technology things or airplanes or something like that had long gone. didn't feel as good about it. It was the people that they worked with that they really cared about. And I went out with a retired engineering dean here and to a meeting who I knew. And he said, you know, Dave, as I flinked my care, it was, it wasn't the projects. It was the people that I worked at the matter.

And I thought, I knew that from a long time ago. So that's, I mean, this is kind of senior persons offering advice. My, I think you're going to care more about the people you've worked with and people you've helped will be a bigger deal. That's part of, in the, you know, one of the other things about personal happiness is they studied happiness. Psychologists used to just study crazy people, but they started like, why are people happy? And they know what the reasons are, you know, have a job that you like, you know, have friends and family, helping other people, helping other people makes you happy. They know this. Having something, kind of either a religious side of it, it doesn't have to be a formal religion, but like contact with nature,

the grandeur of nature. But the kind of the, the love. list of things you need to do be happy is well understood. And, you know, there are these and our world is filled with unhappy billionaires, right? If money was the thing that made you happy, why are these people, very wealthy people? So I'm mad about things. So yeah, this is me passing on advice. In one of your talks, you had, it said, what worked well for me? And it was kind of some reflections. And one of the things in there I thought was unique and interesting, You mentioned that courage was a big part of your career. And I don't hear that too often.

I was curious why you say courage is so important in a career. Yeah, I think that's, I mean, that might be partially my personal makeup. But, you know, I was kind of the youngest kid in my class and kind of small. It took me a while to grow despite age. So I was always small. But my parents encouraged me to go out for wrestling. And wrestling gives you physical self-confidence because you, you know, spend years doing that. I did it in high school and college.

And so I think partly, you know, technically, you know, having courage to do things, it kind of goes along with the advice as fortune favors the bold. That's this, that goes, that's advice is 2,000 years old. I mean, it's hard to figure this out. Helen Keller wrote, you know, even trying to play it safe, you still get caught. And so it turns out you might as well, you might fail no matter what. And if you take a big chance, that's, you can succeed. If you don't take the chance, you probably won't succeed if you play it safe.

So fortune favors the gold. And I think it takes courage to do that. I think also it just for me intellectually, I feel like if there's something not right, need to stand up and confront it. And I think that kind of ironically comes from the wrestling side of my personality, where if I see something, somebody getting picked on or something like that, I'm going to stand and try and stop it. And I feel that same responsibility intellectually if people are making bad arguments or doing something that we need to stand up and do it. And I feel good about that. The cautionary start about that is because I guess one of my senior faculty members

saw this nature of me. He said, one of the sayings is friends come and go, but enemies accumulate. This is an old saying. So if you think about it, you kind of, people he went to high school with a while, friends, you kind of forget. But somebody who you really does dislike, you never forget that you dislike that person. So standing up when it's important, but be careful when you make enemies because, you know, they're going to stick around for a long time. When you say something's gone bad technically, do you mean someone was incorrect? Yeah, when they're weak, you know, either politically or, you know, or technically when it's, you know, it's a weak argument, right? I think of us, I think I really like there to be a marketplace of ideas and we hone the ideas by arguing them. And so if it's a, if it's a specious argument, even if a person is, you know, a leader of a company and it just doesn't make sense. I feel it's kind of the responsibility.

It's better for the company if somebody stands up and points that out than to just let them get away with it. And then it's a little bit confrontational, but as long as people all agree that, you know, this is for the greater good, we're going to, we need to get the right ideas out there. And so let's argue about the ideas to see, you know, polish them to make them stronger. I think that's important in science and engineering and kind of in life too. There's a lot of stuff going on right now in the country that is worrisome. And I certainly stood up and wrote op-eds about things that I think are wrong and need to be corrected. And, you know, if people are afraid to do that, it's hard to be optimistic about the future. of people are afraid to stand up when there's wrongs and try and stop them.

You also mentioned optimism in the talk, and you had this story I wonder. Oh, yeah. Sure. So I would say in engineering, you know, it's hard to know, right? But I think you need to be kind of optimistic or positive, have a positive outlook, because so many things could go wrong. And then so my story, personal story that illustrates, it's going back to high school when I'm 16. I'm dating this very attractive girl, and I screw up my courage and ask her if we would be exclusive at the time.

The phrase we used was going steady. And she looked at me and said, and she was 16. She had dated other guys and thought we were pretty young. And she's, well, Dave, you're such a nice guy. I don't know how to say no. For me, as a logical person, I don't know, sounded like a yes. And so I hugged her and said, great. And so she, in her mind, she thought, well, I'll let him down gently later.

But we've been married 59 years now. And she hasn't let me down yet. So that was the case where optimism paid off. I think everyone that hears a healthy relationship for that long, they might wonder how you did it. I used to tell people, you know, if you go to weddings, the marriage vows are really great, right? but nobody can remember their wedding about us. I used to say remember your writing about us, but nobody remembered that. So we boiled it down to nine magic words, and it's just three sentences, and they start, I, you I, you have to say all three. And it's, I was wrong, you were right, I love you. Okay,

those are those nine words. And this applies to both partners in a relationship, not just one partner. But yeah, if you can say them all, and no substitutions, I was wrong, you're right, you're a jerk. You know, you can't do that. If you can remember those nine, words that can help you have a long relationship like my wife and I have. And then last question for you, like knowing everything you know now from your career, if you could go back to yourself when you had just entered the industry and give yourself advice, what would you say? I mean, I think when I got here, because, you know, the imposter syndrome, I was a UCLA graduate student and suddenly I'm a Berkeley professor. So that just doesn't seem like that was very intimidating. But after a while, I just thought,

well, I'm probably not going to get tenure, so I should just have a good time. So I think I already had a right attitude about it. I think that first year, I think it was very stressful for while I was trying to handle the imposter syndrome and be a Berkeley professor. But I think after that, I handled it pretty well. I did all the things with the kids and stuff. So there's a version of that question is like anything I would do over again. There's one thing I would have. I was chair of the architecture community, the Sig Archers, it's called, and they have an annual conference this year. And this was in the 1990s, I think, I was the chair. And what I wasn't aware is at these conferences, there were men who were harassing young women at this conference. I just didn't think,

you know, people like, you know, young people, people like me, nobody would do that. Only an idiot would do that. That can't possibly be happening, but it was happening. And I wish I somebody had said something to me about it. And because I would have, I would have straightened out any man And that would have threatened his life if he were to do that day. The only comforting thing is Sarita Abbe, who is a famous computer architect. She said she also was not aware that that was going on. It became clear later, five or ten years later, it became more clear that this was going on. And there were mechanisms.

But that's the one thing I wish, you know, if I could go back on time, I would have figured that out. And I would have straightened men out who were doing that. And they wouldn't that would have stopped. I believe that would have stopped them. Yeah. Thank you so much for your time today. I really appreciate it. All right.

And thanks for the interview. Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskov, Mike Stonebreaker, Mark Brooker, these were all people that I brought on because someone,

left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard. So there's two sides. This is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. We're now working on the long journey of building the tooling now. And so if you still want to pick one up, I've left the late pledges open on Kickstarter. So you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.