Algorithms + Data Structures = Programs - Episode 226: NVIDIA GTC 2017 vs 2025

Episode Date: March 21, 2025

In this episode, Conor and Ben chat about Bryce's experiences at NVIDIA GTC 2017 vs 2025.Link to Episode 226 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)SocialsADSP: The Podcast: TwitterConor Hoekstra: Twitter | BlueSky | MastodonBryce Adelstein LelbachShow NotesDate Generated: 2025-03-20Date Released: 2025-03-21NVIDIA GTC 2025NVIDIA GTC Trip Report⭐ The CUDA C++ Developer’s Toolbox - GTC 2025 - Bryce LelbachIntro Song InfoMiss You by Sarah Jansen https://soundcloud.com/sarahjansenmusicCreative Commons — Attribution 3.0 Unported — CC BY 3.0Free Download / Stream: http://bit.ly/l-miss-youMusic promoted by Audio Library https://youtu.be/iYYxnasvfx8

It used to be back in the day that when a hardware manufacturer, like the day that they announced a new chip, the day that we said, hey, here's this chip with this name, you know, when the day in 2017 when Volta announced, when Jensen announced Volta, he had the chip available to hold it up. And like, you know, that was exciting. It was like you didn't know about this thing before the conference. And like now it's like, you know, we show our roadmap there and it's like we show the road we announce the next chip like a year before it's going to be out and then we say

something about the next next chip and then we even like say what the name of the next next next chip is going to be be which is Feynman. Welcome to ADSP the podcast episode 226 recorded on March 20th 2025. My name is Connor and today with my co-host Bryce, we chat about Bryce's experience at both the NVIDIA GTC 2017 conference and his recent experience at the NVIDIA GTC 2025 conference. How's it been buddy? You are live.

Oh God, Connor, Connor, my butt, my soul has left my body. It is day four of the GPU technology conference and I am an old man. I do not know if I will survive. This is just because it's from morning Tonight's nonstop stuff and you're... It's from Morning Tonight nonstop stuff and me, Jake Hempstead and some of our other co-workers were at a nice team dinner last night but we got home pretty late. And I have had to. Oh, they scheduled my talk at 8 AM yesterday morning, which meant I had to be there at like 6 AM. Oh, man.

This is a crazy, crazy conference. This is like, there's like 25,000 people here for GTC, which is Nvidia's tech conference where we talk about AI things. There's this crazy exhibition floor, and it just takes over San Jose. it's too large for the convention center. So it's spread across the convention center in three or four other locations. And it's a madhouse. It's a ton of fun.

Oh, they have a candy wall, which is a giant wall with, you know those things that they have coffee beans in that you can like open up and then the coffee beans spill out into your bag? What? Oh, like in a grocery store? Yes, like yes, where you'd get like nuts or coffee beans or something. Like in the bulk aisle kind of thing.

Tube, yeah. So they had one of those for candy. Yes, where you'd get like nuts or coffee beans or something like in the bulk aisle kind of thing tube Yeah So they had one of those for candy and I Was doing so good with my diet buddy, but the candy Yes, it is Free stuff in general. I don't know if a candy wall would do it for me a donut wall They had a donut wall at the end of a 10k race I ran once and then I took a bite like right after finishing and almost was sick.

See I can't recommend that. Oh yeah by the way, this is the weekly or monthly fire alarm test that of course they're going to record while we're doing this so you continue telling your story while I mute my mic. No, no, no, it's interactive. My mother said she listened to episode two to one of our of your podcast. And she says, please make sure to tell Connor that the inception slash Juno actress he mentioned is now a dude. Elliott Page, the Alleyne Page. Elliot Page, ne Eileen Page.

I mean, I was actually, one, I knew this, but two, I was actually pretty careful, I think, not to misgender Elliot because I was referring to the character that they played who was a she. I mean, in the Juno movie if yes memory calls correctly that was before they had transitioned to Elliot so I was aware and I was also like cognizant when I was making the point and then when I couldn't come up I couldn't actually remember I can't remember if I was trying to remember

Elliot or the name of the individual but I think I don't know someone can go I'm not gonna go back and check but I think while editing it I was I was careful to make sure that I didn't say the wrong thing Right. Oh, yeah, that's right. Okay. Why don't why am I saying? Um, it's Juno The reason I brought her up was because it was specifically from inception right that the dream folding its self over Or like the the world folding itself over which is it's less important the actress or actor I guess now but more was it important that the character in the movie is capable of folding worlds over on itself but thank you for the feedback I don't know why my mom is now listening to the podcast I

think maybe she's she's stalking me or something um yeah I mean I those two back-to-back episodes, I mean, especially the one after that, there was a lot of, what do I call it, inserts of other media, clips from whatever, movies and whatnot. But yes, thank you, Bryce's mom, for the feedback. Also the lovely note on one of the postcards you sent me.

She had written something, I believe. Oh, that's right, that's right. It's funny because like, postcards have such an unpredictable latency. We sent, we like collect postcards now and so we send a lot of postcards and the worst ones was we sent some postcards from Japan that took like between six to 12 months to get like fully delivered. Damn. And like they also have like a pretty a pretty substantial droplet rate. Yeah. Like I only think like 80 percent of them get through because I know we sent you other postcards

and I think at least one set of the postcards did not arrive. and I think at least one set of the postcards did not arrive. And I think part of it is like, and look, no offense to Canada, okay? But like, we're in the world, like you know, we're in Morocco and we ask someone like, you know, okay, like we want to send a postcard to, you know, to Canada or to the US or somewhere. And we just have to trust that they know what they're talking about. And I think that maybe there's been a couple times where we've tried to send a postcard to Canada

where they've been like, hmm, maybe Canada's in Europe or something. And then they've given us the wrong postage for sending something to Canada. I mean, the topic of Canada may or may not come up later when we talk about Twitter, Tesla, and Elon, because we have to revisit a topic. I'm not sure if you recall what we talked about,

but we're gonna shelf that for maybe part two of this, because we should get back. I mean, and also part of the problem is I moved in between. So I think you sent it, and if they had shown up on time, I think we would have gotten those postcards, but I think just the the delay I suppose that's that's that's a possible. Do you have so in the US? There's like this service where you can like do mail forwarding which um Which is great because I I still haven't updated my address from when we moved last year, so

We still get a lot of mail at our old address and they just route it to us. Who does that? Is it the mail postal service? Yeah. That is something that Canada may or may not have. I have never looked into that. I just assume that like if it's...

Is the Canadian postal service called the Royal something something? No, it's called Canada Post. All right. Anyways, back. We so we should stay on the topic of GTC because one I've never been and I don't think you did. You've never been to JGC. I remember we've told this story a couple of times that I was going to go the very first year.

I joined and I joined in October of 2019, which was for those keeping track, like five months before the pandemic went full swing, GTC got canceled. The next year it was online. I think maybe it was two years in a row it was online. And then it's been two years in person. And just both of these years I haven't been presenting a talk. I actually I had been to GTC in 2017, I had been to GTC in 2017, like four months before I joined Nvidia, but I have not been to in-person GTC since then. Although somebody may be pointing out that may not have been true. I may have been to one other one, but I don't, I really don't think so. I don't think I've been to in-person GTC since 2017, since I joined. Well, you know, we should have had you give,

we gave all these talks about like, Intro to Cuda C++, like how to write GPU code in idiomatic C++, and we should have just had you give the talk that I was gonna give, because I killed myself over the past two weeks, you know, trying to get all my slides ready on top of all the other, all the other GTC related things. So before, before we get to your talk, because it is,

it is March 20th very early in the morning for Bryce 7 AM it's, it's 10 AM local time. Cause I'm still on the East coast. I watched the keynote on Tuesday and I've seen two and a half talks since then. I watched your talk, I believe at like 2 a.m. last night and I've lost many hours of sleep due to having watched your talk because I just I just have I have so many follow up questions. But before we get to my comments and my questions, because there was a couple questions live, and now we'll link it in the show notes, folks.

It is available via replay on the GTC website. Unfortunately, you do have to log in via email. We are working on getting them just posted on YouTube. But by the time I'm done my career at NVIDIA, we will just be posting stuff to YouTube like every other technology company. Nvidia, we're still getting there. The market cap's there, but we're still working on

some, the edges of some stuff. But before we get to your talk, or we can skip straight to it, what are your, having been, at least according to your memory, this is the first time you've been to GTC in person, it's obviously much different. What is it like?

Were you at the keynote in person where there's like a stadium full of people? Like what are the highlights? So I tried to be at the keynote in person. I really did try, but you know what happened, Connor, is a very brice thing, which is I did not read any of the prep material. I read the one email that said don't try to like drive to,

in park at the convention center because there's gonna be no parking. And so I, and that there will be a shuttle from Nvidia's headquarters to the convention center. So I, you know, reasonable person, got up at, keynote was at 10, I left the hotel at 830, got to headquarters at like 845, 850,

and I parked in the building that I used to work in, which is, you know, the new building, Voyager or Endeavor, one of the two, we named them both for space shuttles. The point is those two buildings are across a three-lane highway and I go and ask the receptionist, hey where's the shuttle to GTC? She's like, oh it's in building S. Building S is as far away from where I parked on the Nvidia campus as possible.

And so I'm like, ah, great. Now I got to walk 20 minutes to get to the shuttle. I walk over to the shuttle. It's now like 9.15. And I wait for the shuttle. The shuttle leaves at 9.30. I'm like, all right, great. I got plenty of time.

I get to the convention center. I ask someone at the convention center, hey, where's the keynote? And they're like, oh, the keynote's over at the SAP center at the sports stadium, where they do concert events. And that's a 20 minute walk away. So I'm like, all right, wonderful. So then I had to take a quick Uber over there.

And I get there and the line is like out the door and around the corner and so I get in the line and the line and I start watching the keynote on YouTube because we do we do live stream the keynote on YouTube and I make it like a fourth of the way through the line and then some guy comes out and says, all right, that's it, folks. It's completely packed capacity. You should go to the viewing party in the convention center.

And at that point, I was like, you know what? Maybe it's not my day. But I will tell you the story of the first GTC that I went to, which was before I came to work at Nvidia when I was being seduced by Olivier Gros to come over to the green side. And he convinced me that GTC 2017 was a GTC to come to. And I've been telling part of this story all week that it used to be back in the day

that when a hardware manufacturer, the day that they announced a new chip, the day that we said, hey, here's this chip with this name. The day in 2017 when Jensen announced Volta, he had the chip available to hold it up. And like, you know, that was exciting. It was like you didn't know about this thing before the conference. And like now it's like, you know, we show our roadmap there and it's like we show the

road we announce the next chip like a year before it's gonna be out and then we say something about the next next chip and then we even like say what the name of the next next next Chip is gonna be which is fine. I heard that as well and I was surprised like I'm not sure that was the first time I've ever heard him say like next next but I well he kind of said it as like a throwaway comment almost it was like oh here's Ruben and also the next one will be named after uh Feynman and I was like wait we make that stuff public now yeah well I mean part of it is because it's going to leak part of it is because if you're buying if you're building a 100 000 cluster, you need to do your planning a little bit further in advance than you used to.

And like for consumer products, for consumer products, we still don't announce things three years ahead of time because like with consumer products, you can still have the magic. But I feel like we've lost a little bit of the magic because I think there was something magical about the CEO going up on stage and you're not knowing if there's going to be a new chip or not. And then he says, oh, here's the new chip. And then he does the showman thing and then he holds it up. And it was really magical, GTC 2017, when he launched Volta, which was I'm incredibly sad,

because Volta is now approaching the end of its life. It's an eight-year-old GPU now. And to me, it will always be the new GPU, because it was a transformative architecture. It was the, I believe, the first architecture where we had tensor cores. It was also the first architecture where we could fully and properly support C++ because it was the first architecture where we had independent threading on the GPUs, where each thread had its own independent program counter. And so you could write proper synchronizing code between the GPUs where each thread had its own independent program counter. And so you could write proper synchronizing code between threads.

And there was a bunch of other ways in which Volta was just a huge leap in terms of programmability and performance over the previous GPUs. And really we almost divided the world into pre-vault and post-vault and now it's like it's approaching the end of its lifetime and it makes me sad. That was the GPU that brought me to NVIDIA that I went to the keynote in 2017. I was living in Berkeley at the time. I woke up at like 5 a.m. I drove down.

I found parking at the convention center. This was back when the keynote was at the time I woke up at like 5 a.m. I drove down I found parking at the convention center this was back when the keynote was at the convention center before in video was big thing and I got to the keynote and I was there really early I got into the room I was in the first people in the room and there was a section up front that was labeled press only. Connor do you know where I decided to sit? I'm gonna go out on a I'm gonna go out on a real thin limb here and say that you sat in that section Bryce Press only section and I live tweeted the The keynote so I guess arguably I belonged in the press section because I was live tweeting pictures the keynote

I tried to find the Twitter thread To retweet, you know to be like hey, this is the last time I was at GTC. But I think that built up quite a critter following from live tweeting the 2017 GTC keynote. And it was a magical GTC. And then I went to watch Olivier's talk called Inside Volta, where he and him and our former colleague Luke Durant talked about all the architectural

details and it was just a magical thing. But now, oh man, now the conference is magical in very different ways because it's this giant event and everyone's here, everyone wants to learn about CUDA and about Nvidia GPUs and if you walk around with an Nvidia badge there and you're a rock star and it's really something else. It really is. Yeah and we this year we've been we've launched this whole new curriculum for how we teach modern C++ programming for our platform. Because our platform's been around for 20 years, and so the problem is that a lot of the teaching material

that's out there is older teaching material, and it's teaching people how to write C++ code for our platform in a non-idiomatic way. And like a lot of the teaching material was maybe great, you know, 10 years ago, but it's just become outdated. And so we've completely changed our approach for how we teach GPU programming. And that's part of what my talk covered. We've put out all

this course material that teaches this new way. We're also now like very big on Python and so we're sort of trying to figure out how can we teach C++, how can we teach our APIs in the same way in C++ and Python and in other languages. We're starting to care about other languages too. Alright perfect you brought up your talk. This is a perfect transition. So now if and in other languages. We're starting to care about other languages too. All right, perfect. You brought up your talk. This is a perfect transition.

So now, if you would like listener, feel free to pause this podcast. You're probably not going to. I wouldn't personally, if I were you and I was in your shoes right now, not having watched Bryce's talk. Maybe you have already. But if you want, if you are so inclined,

you can pause this, go watch Bryce's talk. Link will be in the show notes and then you can come back. There was just a couple of questions asked in the talk. If you did go and pause it. And now I shall ask my questions. First of all, it is not a question, but a comment I always, I always forget. And I'm always reminded when I watch one of your talks, Bryce of the shared passion that we have for beautiful slides. You make some of the, I mean, I usually to some extent write a quote unquote trip report

for every GTC ever since I've been at Nvidia when there was a GTC because obviously COVID year they canceled it, the first one. Last year, I think they, because it was the first one back in person they had some recording issues and so my trip report kind of fell apart because I was waiting for j cams ads talk to be released to like complete it but then it never ended up having a recording. Anyways one of my things I like to do is I like to take screenshots of like the highlights you know if I watch a jacks or xla talk and they've got some nice you know slide I take a slide I think I took. Eight or nine screenshots of your deck. What are some of the, I mean, I'm gonna compare, I haven't finished the what's CUDA all about anyway talk, but there's two different slides. One from, I don't actually know him, but his last name's Armstrong. I think Rob Armstrong, he's giving that talk. Rob Armstrong, yeah. He has a GPU versus CPU slide, you have a GPU versus CPU slide.

I don't need to, I don't mean to dunk on Rob's slide, but yours, much, Jeff's kiss, much more beautiful. You also have that host slash device with the gradient from beautiful, beautiful. I saw that and I was like, this is Bryce at his finest. Anyways, so just, you said you spent the last two weeks on your slides.

It definitely shows. Yeah. It's interesting now that a couple of years ago, I made the decision that I was going to speak a lot more frequently, but give the same talk more regularly. I used to make a talk, and I would give a talk once or twice. And what I decided was, I don't want to make three or and I would give a talk once or twice and what I decided was I don't want to make

Three or four or five talks a year I want to make one really really good talk and I want to give it ten or twenty times a year and part of The philosophy is I want to give the talk so many times that it becomes Just like like that. I mastered the. And so I think I make fewer talks, but I put so much more effort into the talk because like I made this talk knowing the reason I was so stressed and pushing so hard for it is I'm gonna like, I'm immediately turning around.

I gave this talk, what's today? It's the 20th. I gave the talk on the 18th. On March 27th, I'm gonna be giving the talk in Paris. On March 31st, I'm to be giving the talk in London. And I'm going to be giving the talk like two or three times a month for the next, you know, eight or ten months. And then the other big decision I made is to script all my talks.

I have a script of what I'm going to say. And it always amazes me where, like, you know, every talk there's a time target of like how long I'm supposed to make the talk for. And what I always do is I go through and I make all the content and then if I have, it's like I very rarely test how long it takes before I give the talk. Because I usually give 60 minute talks and I usually just make the content and it happens to fit in 60 minutes just without any thought about it. And sometimes I can just look through and I can just eyeball it, like maybe this is

too much, maybe this is too little. In this case, I needed to give the talk in 35 minutes and I was a little worried about that because it's not my usual time target. And so I did one dry run through where I went through and ran through my script and it ended with like 36 minutes and 24 seconds. And it was just like, it was perfect. It was just, it somehow kicked the perfect length. But I love, I love scripting the talks because I feel like for the most part after I've given the talk two

or three times I can do the delivery without it sounding too scripted. It'll still sound a little scripted but I can do the delivery without it sounding too scripted and then I don't have to worry about going off on tangents. I don't have to once I've got the timing, I don't have to worry about how long it's gonna take. I can make it more consistent and a better experience for the people I'm giving the talk to. And one of the reasons why I love giving the talk 10 or 20 times is by the time I've gotten to,

you know, the 10th time I've given the talk, I've memorized most of the script. And so now I'm just, I'm just consistent. It just sounds natural and it's just consistent All right food for thought. I mean I do not script my talks But the result of this talk it's it's very good. And like I said these and I actually I went and checked I think I said it was seven or eight. I took one two three four five screenshots five screenshots So not not eight or nine, but definitely the most I've taken of any other talk I've watched is one which leads me to I mean, I'm not gonna share screen share because we don't need to you'll know the slides

I'm talking about and maybe we'll include it in the show notes By far my favorite slide Be sure to check these show notes either in your podcast app or at ad sp the podcast com for links to anything We mentioned in today's episode as well as a link to a get up discussion where you can leave thoughts, comments and questions. Thanks for listening, we hope you enjoyed and have a great day.