Embedded - 151: Captain Stochastic

Episode Date: May 11, 2016

Paul Sidenblad spoke to us about his engineering career, starting off with GE's work on theGambit spy satellite.   Google Protobufs Elecia read Eye in the Sky: The Story of the CORONA Spy Satellites a few years ago and remembers liking it, though this was the first time the information was useful. 

Welcome to Embedded, a show about the many aspects of engineering. I'm Alicia White, alongside Christopher White. This week, our guest is Paul Sidenblatt, an engineer at NVIDIA. Hi, Paul. Thanks for joining us. Hey, you're welcome. We're happy to be here. Usually at this point, I ask you to tell us about yourself. But I think that might take too long, so we're going to skip straight to the lightning round.

Chris, you want to go first? Sure. Favorite programming language? Huh. Yeah, I thought about that. It's not C++, although I can live with C++. I think I'd go back to ModComp Assembler. I think it was a decent register set.

It was pretty simple. It wasn't cluttered, you know, like certain complex assembly languages are that I could name. But yeah, it didn't look thrown together. It was just simple and you could do what you needed to do with it. And I'm sure nobody's ever answered that before yet. Yeah, that's a new one. What programming language do you think should be taught first in college CS courses?

Well, see, I never had a college CS course. Math major. Yeah, math major. But... Don't make fun of us. I know I'm outnumbered. I could see starting with C, but I'd be tempted to go more bare bones. I'd be tempted to teach people about what's going on inside of a computer before we start teaching them about higher level languages.

I might start them out in a similar language. A microprocessor approach. Favorite processor of all time. Oh, wow. You know, I got really inside of RSX-11 on a DEC-1170, and that was good to work with And that was good to work with. That was good to work with. But just for volume,

the first machine that I worked on was a CDC 3800. And that was 64K, 48-bit words, once complement arithmetic. It was just a wonderful piece of work. You can see it in the Smithsonian now. And it wouldn't even begin to fit in this room you know it was uh just just a monster but uh it had like the the star trek control panel flashing lights and it had uh a speaker on one of the data buses

actually an fm receiver on one of the receiver, then wired to a speaker so that you could hear your program running. And you could tell when it was stalled. You could tell when it was stuck if you get these big recursive loops and it starts to just sort of buzz on itself, and you can just tell them when it was stuck. The music of your program. That's right, yeah. And people actually wrote music with it.

But that was, you know, it was just for fun. Yeah, I remember people used to write music for the Apple II floppy disk drives because you could move the steppers with the assembly line, so they'd bang them around. Yeah, yeah. Yeah, and that was an Air Force contract, so of course they had the Air Force fight song with the percussion coming from the line printer.

When you were young, what did you want to be when you grew up? Aeronautical engineer. Oh, I thought that was usually the last question that we extended. No. No, no. Okay. See, I'm totally unprepared. No, the last one is what is your work day like? That isn't for a while yet. All right. Should we bring back the dinosaurs? No, probably not. Probably not. You know, I'm a backpacker and I'm of two minds as it is about the grizzly bear. But, yeah, I could live without the grizzly bear and I'm sure I could live without the dinosaurs. Favorite Girl Scout badge?

Well, you know, I'm a little bit rusty at that. The Girl Scout troop that I led was not that much into the badge thing. But if there's a backpacking badge, that would be the one. Hmm. Where to go next? Go technical. Big or little Indian? Little Indian, always.

Hardware or software? Oh, on the edge. I like to work at the interfaces. Me too. Engineering or managing? Oh, engineering. That's easy. That's easy.

I think we're out. One more for me. Suit and tie or jeans and a t-shirt for me personally suit and tie okay jacket and tie let's say that the suit is you know let's not go crazy i've been watching person of interest too much um what is a day what is a work day like for you uh well okay um starting like with the commute i'm lucky enough to have a i'm lucky enough to drive counter commute so uh it doesn't take me long to get to work but i i go directly to a gym first and uh work out and that puts me in a much better frame of mind when i get into the office and mostly in the mornings i spend um well often there's a meeting go through email

surf the web a little bit and usually i don't get much programming done in the mornings uh there just is i need a i need like a solid block of time and it just doesn't happen in the mornings. So I do the other stuff. I do the communication with the other engineers. I answer other people's questions and that sort of thing. And then after lunch, I try to find a couple hours where I can really focus. And if I fall into it, then I can just keep going. Uh, I, uh, haven't done that for a while, but, but, uh, uh, I really enjoy the part

where you get into solving the problem and you forget about the clock. You know, that's the best part of the job. The flow state where it all just, you understand what's supposed to happen and you can get from here to there and there's no barriers. Right, except the compiler. The compiler is not always my friend. Well, and sometimes yesterday me put in bugs that now I have to find. Those are always amusing. But yeah, that solid block of time.

We don't get enough of those. No, that's the part that I, if I could have more of those, that would be an improvement. But aside from that, I love my job. So you work at NVIDIA now? Yes. And I know I can't ask you about what you're working on now because you can't tell me. What's the last thing you can tell me from your work at NVIDIA? Okay, well, so I was working with the libraries group and working with these really, really, young engineers from Poland. Uh, as a matter of fact, we had one guy

come in from Krakow and he was, he was like the outlier because everybody else was from Warsaw. But, uh, anyway, and so they did all the real heavy mathematics and I did a lot of plumbing. Um, and that was, that was just fine with me. I was close enough to the math. I never really got down to the GPU level except once or twice. But I was working on FFTs. And getting really big multi-GPU FFTs working. So FFTs are fast Fourier transforms. We got a comment recently that we don't always define our acronyms,

so I'll do that, but it's signal processing. And it's the sort of thing that I used to do with audio, identifying gunshots and whatnot. It's breaking down signals into their component frequencies. But audio, you don't need GPUs, let alone multiple GPUs. What kind of signal processing were you working on? Well, so we just provide the library. And so our customers vary.

They use FFTs and seismic. They use, well, there's radio frequency signal processing that some agencies would be interested in. And so, yeah, but as I say, we didn't actually write the applications. We provided a library of, just a really flexible library of transforms that you could do, well, basically all different kinds of strides, multi-dimension FFTs.

Yeah, we used that at one of the medical device book companies I was at. We put large-scale FFTs on the GPU, probably used your library. I'm not sure what the timing was, but I think we used OpenCL, and then there was a library that we dropped in. But we had high data rate imaging stuff that was coming in as spectroscopy data. So it was already the FFT, so we had to do the inverse FFT of a huge amount of imaging data.

Which looks pretty much just like the FFT. With a couple of scalars. Right. Yeah, so a lot of imaging, a lot of compression in signal processing. Yeah, and people do, you can do a correlation with FFTs too. FFT in and transform and then FFT back out. Okay, so... And then before that it was random numbers. Random numbers. Generating random numbers is much harder than it seems. Yeah, there's a whole science to it. We didn't have our own

we didn't develop our own generators. What we did is adapted existing algorithms to the GPU. That was fun. For a while I was calling myself Captain Stochastic. Just to bug my children. I imagine that was a great Halloween costume. But when you

got to NVIDIA, that was a while ago, 2001. You started there working on networking. That's right. When I went to NVIDIA, they were making a chipset for Intel weatherboards. And among the components, it had an Ethernet controller built in. And so I was doing drivers. I had come from 3Com.

So that was the background that I had that they were interested in. And so I was doing drivers. I had come from 3Com. So, uh, that, that was the, the background that I had that they, they were interested in. Um, so they hired a really good team of, of engineers, brought us in from several of us from 3Com and some others. And, uh, we did just a whole lot of, uh, right, right at the hardware software interface, uh, right where I like to be. You know, that was a great experience. When we talk about hardware and software, I usually envision that point of writing software for microcontrollers and running on the hardware. But NVIDIA does a lot of chipset stuff.

Are you involved with that? we did um well we did the chipset for for intel cpus that was that was it was like for for an ibm pc clone and um then yes i did work on the tegra uh line uh but in that case um i was doing more of an architecture job i was working on the design for debug aspect of the part where you're putting in hardware to help you debug the hardware, basically. How is it different to work on a board versus working on support for a chipset? Not really all that different. I mean, it really all that different. I mean,

it's all just gates. You know, I mean, and so the thing that I like about it is trying to make them do what it is

you want to do. And it doesn't really matter what level you're working at, to me, whether you're doing parts for an embedded controller or parts for something higher up the food chain. Software, software.

Okay, so you were at 3Com doing networking and standards? Oh, I took part in I2O and InfiniBand discussion groups and stuff, but the guy that was really involved in standards was another guy that they had that was doing that. So I wasn't really on the 802.3 committee, for example, or the 802.11. I assume you understood all that? Yeah, 802.3. Oh, yeah, if you're decoding, then 802.3 is Ethernet and're if you're if you're decoding that 802.3 is ethernet and 802.11 is is wi-fi wifi nobody pronounces it that way except you i don't even pronounce it that

okay um and you worked supporting novelle which is not a name we hear a lot anymore, and Microsoft, and OS2. Oh, yes. When I first got out of college in 96, one of my jobs was supporting a wide number of operating systems on server hardware, and writing drivers for OS2, Novell, three different flavors of Windows, and something else?

HP Ox, maybe? Probably. Because I was at HP, so that sort of makes sense. Oh, that makes sense, yeah. And it wasn't easy, because they were all very different. I mean, Novell and OS2 had some weird commonality. But even the different flavors of windows weren't the same which one did you like best oh of those i'll have to choose yeah that's right

um well which one did you like least yeah that's easier yeah well novellas call always kind of an outlier you know they they they were a little bit a little bit, and they were kind of on the way out. I'm motivated by the volume of customers. I really like the idea that people are using the stuff that I've built. That's important to me. And so, you know, by that measure, then it had to be the Microsoft OS's of various flavors. I guess it was NT4 I think was kind of where I came in at 3Com. That was one of the later ones.

Well, later by what definition? I mean, OS2 I was at Olivetti. Oh, okay. I mean, Windows 98 is I was at Olivetti. Oh, okay. Yeah, yeah. I mean, Windows 98 is still the best Windows. What? No, Windows NT is still the best Windows.

So, Olivetti was before 3Com. Let's see, 3Com was 94 to 2001, and Olivetti was 87 to 94. Thank you. It's nice to have your resume in front of me. It's really much easier. I couldn't have brought that back up. And that was, what is Olivetti? Well, so they started off as a typewriter company, but by the time that I joined them,

they were making PCs. They made the AT&T PC that was in existence at that time. And it had a very unique display. It was 640 by 400, I think. Memory fades. And monochrome. But, yeah, anyway. So I worked on their display driver and I worked on their keyboard driver. It was fun to put, uh, to make, to make a single key, uh, insert three zeros instead of just one because Olivetti was a, like they're, they were into the banking business, PCs for banking and at the time it was pre-euro and so there were 1200 lira to a dollar so if you're

in a bank you need a lot of zeros right and so they had a key on their keyboard that made three zeros instead of one zero so i hacked the os2 driver to driver to take this keystroke and generate three keystrokes. There's so much you could, I mean, that is really a hack too. What was your response when somebody came into you and say, hey, Paul, we need this feature? It's okay. You know, I get it. You know, it wasn't that crazy. No, it makes sense.

It just, yeah, okay. Well, I guess your fingers get tired typing zero too many times. And it's depressing. Yeah, I guess so. It reminds you of inflation. Why grind it in? And I once had a variable named poo name, and it was horrible. So it is an odd thing developers sometimes have to

do uh okay so before that uh bell northern research yes right and that was uh pbx oriented so they had their own language that they had developed and their own compilers and stuff to to run inside the pb. What's a PBX? Oh, uh, it's a telephone system, office telephone system. And, um, yeah. So the, the best part about that was that the, the customer in, at least in my case, they, they sold worldwide, but the, the customer that I was working for was a tale of Eric at the Swedish telephone company. And so, uh, interacted with a lot of Swedish people, and then we transferred the software technology over to a team in Sweden.

And that resulted in a six-month stay in Sweden for my family and me, and that was fantastic. That's where my family is from, or my ancestry is from. Was that your favorite job? Oh, well, that aspect of it yes and my favorite job is probably the one i'm doing now you know i mean that's that's probably what i would tell you anywhere along my career but uh but since this is my my 15th anniversary this month at nvidia uh i got a sort of congratulatory note from my VP and I said, yeah, you know, looking back at 40 years, I think this has been the

best 15. That's pretty cool. Okay. Since you've given away the 40 years, let's go through the other ones too. 1977 to 82, System Development Center. Development Corporation, yeah. Development, okay. Navy, Moffat, ModCom. Yeah, ModCom, that's right, yeah. That's where I worked on the ModCom.

So they had, I worked at a place called the Acoustic Research Center on Moffat Field. And you have to think about what the original mission of Moffat Field was in World War II. Those big blimp fingers were there for blimps that would cruise out over the ocean scanning for submarines. So I was working in the Acoustic Research Center listening to the ocean and I can't tell you what the mission was but uh because as far as i know it hasn't been declassified but uh but nonetheless we were listening at the ocean and uh for whales yeah for whales whale farts yeah magma disturbances yeah so anyway and so um so that's where i learned first about ffts, actually, is because we were collecting the data out at remote sites.

And then the part that I was working on was the data collection and also communicating it back to Moffat Field, where we processed it. Did whatever classified things we can't talk about. In as much as you can say, now that you've worked on FFT processing systems in the 2000s and in the 70s, there's a vast gap in what you were able

to accomplish, right? Oh, yes. Yeah, that's absolutely true. I mean, they had the specialized signal processing engine that they were doing the transforms with. But, yeah, it was different.

The frequencies that you could deal with were different. The size of FFT that you could do were different. The sampling rates and the storage. Oh, yeah, yeah, that's right. Yeah, yeah, yeah. I mean, we had a couple of the sites had 56 kilobit lines. And then one of them, the real powerhouse, was like 1.3 megabits. That's huge for then.

Yes, it was. It was, for then, it was a lot. And so, you know, we had a custom deck. Well, that's actually, you know, since you asked me what my favorite machine is i had more fun with that it was called a kmc 11 um and it was a it was a plug-in card with a microprocessor on it and it was a it was bit slice micro programmed and it was it was just really fun to work with because you were now now you're right you're you're not you're not writing an assembly code even. What you're doing is controlling the gates that get you on and off of the arithmetic processor and in and out of registers and in and out of memory by setting little bits in the control words.

So it was good. More microcode. Yeah, it was it was it was good more microcode yeah it was it was microcode okay and before that was general electric satellites which has recently been declassified so you can tell us about that's that was fun uh yeah that we were at uh kathy and i were at a a luncheon for uh one of her friends birthdays and this guy sits down next to me and starts talking about this project. And I had buried the code word for that project and just put it out of my mind.

I hadn't used the word gambit in like 35 years. And here's this guy talking gambit this and gambit that. And I'm going, excuse me, can you talk about that? And it turns out, yes, it had been declassified. You can look it up in Wikipedia. You can go see the satellite itself and the CDC 3800 that controlled it at the Smithsonian in Washington. So, yeah.

So what was Gambit? Oh, Gambit was a spy satellite it was a it was basically a big camera but the way it worked was that it had uh like a 10 inch wide uh roll of film that as the satellite flew over the earth much gesticulation here and it's not bad it's audio only. But anyway, as a satellite would fly over the Earth, the film would go past a slit. And then there was a big mirror that directed the view of the Earth onto that slit. And the mirror could be rotated

so that you could look at an angle. You didn't have to look straight down. You could look look at an angle. You didn't have to look straight down. You could look off at an angle. Oh, that was a big improvement over the previous satellites. Well, I guess I don't know what the previous satellites looked like, but yeah, it was a big deal. And it could only rotate this mirror so fast. And so the problem that everybody worked on when they were waiting for their clearance which took

about six months because it was a code named project and they actually did a background check i mean they went out and interviewed my old landlady in san diego and so on and what kind of a person was i did i have any dark secrets anyway so the um the problem everybody worked on was the motorcycle and diamonds problem so you have a a road in front of you, and it's sprinkled with diamonds of various values. And you're on a motorcycle. You can only steer it so fast. And your job is to pick up the best value of diamonds as you go down the road at a fixed speed. And so this is exactly like the satellite cruising over the earth with its mirror turning back and forth.

And that was the that was the the main optimization problem as far as we know as far as i ever knew it was an mp complete problem non-polynomial complete problem so uh basically tough is you don't get an optimal answer for that problem after you're past about a dozen points so uh so it was fun that's how i learned that's what i learned to to code with as jovial and the motorcycle and diamonds problem so it was film yes what happened to the film and came back yeah it came back so yeah so that so the film was in a canister and there was a way to release the canister and it was in its own little re-entry vehicle with its parachute and it would come down through the atmosphere just like any other you

know current satellite i mean not something but any manned mission uh parachute would open and uh there would be a c-130 a big cargo plane with the the rear doors open and a big net and they had this you can imagine just like these the spiders that hold the the web out behind them to catch flies right this c-130 would come along and snatch this uh film canister with its parachute out of the sky and haul it in i I think I saw that on the Avengers movie. You might have. It probably was after the thing was declassified. I think they did that to a person in Batman.

Yeah, maybe. It just sounds so fantastical. Well, it sounds so, how are we going to do this? Well, we're just going to do this. That's right. Yeah. Yeah. I mean, one of the pilots worked in the control center, one of the guys that had been a pilot. And he said, yeah, it's really, it's right. Yeah. Yeah. I mean, one of the, one of the pilots worked in the, in the control center.

One of the guys that had, had been a pilot and he said, yeah, it's really, it's not that hard. You just. Well,

you do have control over when it launches its payload. Oh yeah. So you're not going to do it when it's in a thunderstorm or whatever.

Yeah. But I don't think GPS was deployed at this point. No, no, it was not. Definitely not.

I had to do a little more navigational triggering probably. Well, and, and, well, I mean, they didn't really know where it was not. Definitely not. I had to do a little more navigational trickery, probably. Well, I mean, they didn't really know where it was going to come. I mean, exactly where it was going to come down. I mean, it was, you know, within a, you know, some hundred mile kind of range, but. Well, you didn't really want it to touch the ground because you didn't want the chance of intercept. Yeah, well, it was over water, but yeah, yeah. I can just see how interception would be bad

because the motorcycle and diamonds problem doesn't actually specify the part where you don't want other people to know what's valuable to you. Right, exactly. Or what resolution you can gather information at. That was probably the most important secret of the secrets. I mean, everybody knew everybody was taking pictures of everybody else.

As fast as you can, as much as you can. But the quality that they were getting was really pretty good. Huh. Okay. And before that was college, right? Yes. Okay. And you did math and then you also did a teaching credential.

That's right. Yeah. Yeah. I spent a year learning that I wasn't cut out did a teaching credential. That's right. Yeah, yeah. I spent a year learning that I wasn't cut out to be a teacher. Oh, yeah. It's different. I mean, teaching versus engineering. Well, really what it was is that teaching somebody,

I had done some tutoring and stuff in high school and college and, you know, the sort of the imparting of information, that sharing and the sort of the aha moment from coming from the other side, you know, that's really fun. But it's way different teaching somebody that wants to learn than teaching a classroom full of kids that really don't want to be there, you know. And I didn't have the way, I didn't have the tools to get through to that level. So I wound up not teaching. Okay, so how old are you? 65.

Any plans on retirement soon? Not soon. But I have a new granddaughter. Congratulations on that. Yeah, she's six months old now. And it has occurred to me that by the time I'm 70, she'll be five. And that's a fun age.

And that's about the right time to teach her how to paddle a canoe. So, you know, I have a canoe, I know how to use it. And that's kind of my rough plan is I'll probably keep going another five years. Because you're still having fun. Yes. I love it. I love what I do. I go into the office and I look forward to it. I work out at the gym.

That's work. And then I go have fun. What do you like best about it? The problem solving. The sort of throwing your brain at a at a a situation you know it doesn't it doesn't almost doesn't matter what the problem is i mean you've talked about my career i've worked on a lot of different things but uh having a an end result in mind that you want to make something happen whatever whatever that is

it doesn't matter if it's a physical thing if it's a visual thing uh and or whatever and then making that happen making it work i think that's what draws most people into engineering. Yeah, it's the practical aspect of it, you know. I mean, and I really enjoy mathematics. I'm not really that good of a mathematician, but I enjoy it. I loved working on, I actually got to do a little bit of work on the square root one time and that was that was fun because it was it was an optimization and it was but it the thing that that i do in interviews is i is i i have a bit manipulation quiz problem that that i ask people to solve and see because it's not

something that a lot of people really have to do and see it happens that i'm doing that now again but but that's that's that's i really enjoy that that level of of kind of fiddling around with things and the the optimization was basically fiddling around with the exponents of the transform, rather of the... Function. Function. Well, no, actually the exponents of the floating point numbers

and range checking and sort of the just really, really basic stuff, but just masking and shifting and, you know, whatever and making that faster. And finding the pattern so that you can do this optimization. Oh, yeah.

Yeah. I mean, if it was a random number, you couldn't optimize out the steps, but square root isn't, but it isn't linear and it isn't anything else. It's special. Yeah, that's right. And so, you know, I got to relearn a little bit of calculus and I got to, so, you know, that was a lot of fun. I sometimes feel that as I get older, I get more forgetful or it becomes more difficult to learn. Do you find that? Yeah, no. I mean, the forgetful part, sure. I mean, you know, the Dave Barry quote, the nouns go first.

Yeah, they do. But he was an engineer, right? He isn't an engineer. I have to tell you that the three-letter acronyms go first. At least that's what I find true for me. You know, I'll be struggling to spell FFT one moment. Luckily, most of them go out of date pretty fast. Yes, that's right. Well, and people appreciate from, you know, luckily most of them go out of date pretty fast. Yes,

that's right. Well, the people appreciate it around you if you don't use them so much. Yeah, I know. I actually had my, my current manager, uh,

asked me not to, to acronymize my thing. So I spell it out every time I write it down, I spell everybody else's, you know, is, is got the initials for it, but,

but I always, I always write it out longhand. So how do you stay current? By solving problems. I didn't know C++ until I joined the math group. But I got a real bath in the FFTs because they're highly, highly templatized in our implementation. And so, yeah, the compiler earns its keep when we build our FFTs. So do you like C++? We've had some discussions about C versus C++ and kicking and screaming

engineers going into the new world and finding out it's both much worse than ever promised and much better. The new world. Yeah.

Yeah, I mean, you know, the language has some benefits. You know, I mean, the thing that's really good about it is, again, not having to write all that stuff out. That you can have a consistent set of operations on different data types.

That's really useful. I like that part. It's also possible to completely bamboozle yourself with with template spaghetti you know so and and so that you know there's trade-offs it's you know it's like any other tool you know you can you can abuse it you can use a a monkey wrench to pound in nails but it's really not what you should do so in your career it didn't look like you had any startups no that's true I always worked for big companies the best experience I had

was working for companies that I was working at headquarters I didn't know that that was the case until I worked for 3Com which was kind of late into my career but it's much better

working at HQ than not, at least in my experience. We work remotely a lot, so I'm sort of out of the habit of working at HQ. What makes it better? You know, I think you're just closer to where the decisions are really getting made. And I think that's better. The communication is better. Usually the food is better. Yeah, I wouldn't be too surprised. They really have a pretty decent cafeteria at NVIDIA. Every once in a while they fall down.

I was on the Apple campus recently and a friend had moved from one of the buildings that was just across the street to the central part and she was like, this is totally different. It's like a different company.

Yeah. Yeah. When I was at, I guess, HP turned Agilent, all of my friends were going to startups and it just it was like uh 1999 and it was just it was so the bubble you weren't drawn to any of these wackadoodle places in a way you know i mean when i when i joined nvidia they were just really kind of

coming on online in terms of of so that they were pretty close to a startup i mean they're they still had pretty much startup mentality when i joined them uh and that was my big leap of faith you know i mean as far as joining a quote-unquote small company. My badge number is 1111, which has all the Asian guys really jealous. But one's a good number. But yeah, anyway. But a thousand people? That's so many.

Yeah, I understand. I mean, because of course... Well, they got there pretty fast too, because I mean, it was the mid, let's say 96, 97 when they first released their first product right right yeah that's right yeah yeah so they were really growing fast when i when i joined and you didn't look like you had any big breaks in your resume no stints for consulting for a couple years no going back to school no just no i don't have with all this i don't have the

discipline to go back to school that would be i don't think i could do that yeah no i i haven't uh haven't felt like i needed a break i've had a couple good sabbaticals but uh do you do anything technical on your sabbaticals or did you use it as a time to really get away from tech uh well so 3Com was actually the only place I got a sabbatical. And I went backpacking for

three weeks. And I did a week on my own. I did a week with my Girl Scout troop. And then I

did another week on my own. And that was a good break. That was a good break.

And I came back and the first meeting I went to was this organizational thing, a project tracking meeting. And I swore at them and I said, whatever I said, and I've been away in the wilderness for weeks and now I'm back at work. You brought me back that fast. You know, it's too bad.

Because 3Com was not always a fun place to work. I liked the job. The work, the technical work was great. But they were kind of on the way downhill. You know, I rode them down quite a ways down their slide. Is it more important to have a good team or to have interesting technical problems? Hmm. Yes.

Ideally both. Yeah, yeah. I mean, a team where there's not a lot of mutual respect can really ruin a good technical problem. But I haven't had that experience. So that's, I have to say, I've been lucky. With many senior engineers, there's a lot of bitterness and cynicism. Hey, I'm right here. And it does seem to correlate with experience, possibly with startup experience, which might

be how you seem to be avoiding that. But how do you avoid it? How do you help others prevent becoming cynical with teams? I mean, teams don't always respect engineers. Teams don't respect each other. Yeah, that's true. I mean, I think it goes both ways, right? I mean, you get reflected back to you a lot of what you carry into a conversation and uh the one thing that i that i will say does happen

it's happened everywhere i've worked is there there are individuals that don't have a very wide input channel they have a wide output channel and that's just not my style you know i i i try to try to have some balance between hearing what the other person has to say and i'm not always good at it but hearing what the other person has to say first and then you know putting in my own two cents and sometimes it's hard to get that the the two-way conversation going yeah i guess the other thing is there's some style things you know uh and i had a boss one time who was uh like a shotgun he would do 20 things 20 approaches at once to solve a problem and and keep them all going in parallel and that's not my style uh i'm more of a you know pick a direction auger in that way and if it doesn't

work then back out and try another direction but um this guy and he was effective at it you know so fine you know i just got out of his way you know it was okay um so yeah so it's a matter i think of of a certain amount of tolerance yeah and the reflectivity is a good point the people i know who have a lot of bitterness and bring it to the team tend to make the team more bitter and that makes them more bitter and it it really is a downward spiral um but what about boredom you're you've been doing ffts at least twice now um don't you get bored doing the same thing yeah if you i mean if you look at it every everything i've done what i'm doing what i'm doing now is is wildly different than working on FFTs, which was wildly different than doing random numbers, which was, it was going from 3Com to NVIDIA and continuing doing network drivers. You know, that was kind of probably the one thing I did for longest.

But there's always some new challenge. I mean, in the days when I was working on Ethernet drivers, it wasn't enough to fill the wire. It was to fill the wire with the fewest CPU cycles expended. And so there's always a way to improve, to invent another widget in the hardware that helps you not have to spend as many CPU cycles. And so it's not the same problem. It's a different problem.

And you don't get lost in the tediousness. No, no, I don't. I mean, I'll sometimes get a little frustrated when I'm trying to get some code to work and I've just been kind of sloppy with the way I put it together. And so slogging through the 200 compiler errors, you know,

is, is tedious, but you know, that's, that's just part of the job. When I, when I started in, in,

on the CDC 3,800, it was key punch days. And so you write everything up on a coding sheet and you hand it to the key punch operator who punches it up for you you get your card deck back and check it and then you wrap it in two layers of brown bags because it's top secret classified and you hop on the back of a motorcycle with this other guy that i was working with and go over to the to where the computer was because it was in the satellite test center uh not in the office we didn't work in the same building as the as the 3800s um anyway so you know go down the street and and put in your card deck and run it or if you have somebody else do it and get it back overnight the first time i had a compiler that

i could actually run iteratively from a console that was just heaven that was like oh my gosh you know i don't i don't actually have to wait overnight thinking i can i can make a mistake and and it doesn't take me forever the fftFTs, on the other hand, because they're so heavily templated, if you change one of the inner files, you're in for an hour wait to build them. And that was just painful. Don't you work at a company with massive supercomputers? Yeah, yeah, but they don't...

We don't compile on them, unfortunately. That's the next job. So what's been like that more recently? What tool or feature or change has had that level of, oh, wow, this is better? Oh. Well, so when I started working on this new project, uh, I'm, I'm sort of integrating what I'm doing into another set of infrastructure. And as I start to talk to them about that, they say, well, could you use protobuffs? I don't know. What's a protobuff? You know, and this is, this is the way my career has been, right? Can you code in C++'m sure let me let me go learn i can't not code yeah right exactly i mean yeah so it's it you know and and that actually is another thing if you want to know about the the boredom factor is i don't i

don't go out and and take courses and stuff i learn what i need for the job when I'm doing the job. So anyway, so Google has this construct called protocol buffers that basically are a scripting language that you write, and they generate all the C++ objects for you. And they do all the tedious bits about, well, they do compression and you can add elements. They have repeated elements and you can just sort of add them at will. And they take care of all the allocation stuff behind the scenes. They generate all the C++ methods for you. And so a lot of the sort of the drudge work of writing itty bitty C++ objects is taken care of for you by this protobuf thing.

It's a useful tool. I don't know if it's efficient. I'm almost afraid to look at the code it generates. Well, sometimes you need efficiency in getting things done fast for yourself. And sometimes you need the processor to do it fast. And those are different coding styles. Yeah, they are. And I'm not too

worried actually at the moment about the sort of the run time efficiency of what I'm working on. It needs to be within reason, but it doesn't have to be optimal. Do you think your experience over your career has made you a better engineer?

Oh, yeah. In what way? I think I know where the bodies are buried. I think I know what to look for. I think I know when something's going to be a train wreck. I can pretty much always tell when some other engineer is telling me what he wants to think is true when in fact he hasn't done his homework, you know. We do get good at faking it. Yeah. And it's good that people can call us on it sometimes, but we get away with it much too often.

Yes, I know. I know. So yeah, so i'm in the process of one of those little exercises now and it's it's it's interesting does anything surprise you or do you uh find the kind like you said the kinds of similar mistakes cropping up and it's oh it's this again oh yeah well yeah well the thing that always surprises me and it doesn't happen all the time but every once in a while um i'll generate a piece of code and it'll just work the first time, but every once in a while, I'll generate a piece of code and it'll just work the first time. Isn't that a shock? That terrifies me and worries me. I just assume that I must not have compiled it

or it's just broken and I don't know how yet. Why does this work? I'm not observing correctly. It looks like it's working, but it's really not. Exactly. Are there ways that your experience works against you oh no i not exactly i i would put it a different way maybe i i tend to um as i had said you know i tend to take a direction and kind of run it to ground and a lot of times i'll have a

preconceived notion about in case of a you know looking for a bug or something'll have a preconceived notion about, in the case of looking for a bug or something, I'll have a preconceived notion about what must be going on under the covers. And I tend to hang on to that a little bit too long sometimes, I think, before I start getting my head back up out of that particular morass and think about what all the inputs are. Am I using all the data? Back to my sort of problem-solving roots. So if anything, but again, that's a case where I know that about myself. And so as I get older, I think I tend to kind of self-examine. Maybe I'm sticking with this one too long. I'm better as an engineer than I was at 20, for sure.

I'm more patient and more persistent and I'm much better with people. What can I expect for the next 20 years? I think it's just a matter of evolution. I don't think it's, I wouldn't say that I'd expect anything revolutionary. At least I haven't experienced anything revolutionary. But you just get broader. You learn new things. You adapt to the way the environment, the programming environment uh the tool sets whatever you just you just sort of adapt to that and it it gets easier it gets easier to learn if you if you're used to learning you know that's the the thing that um yeah that's that's the thing i think that's that's

most important is just assume walking in that you can learn whatever you need to do to do this job. I was going to ask if you had any advice for new college graduates and I think you just gave it. Yes, in fact, and that's a true statement. You know, you go to college, you should be learning how to learn. Yeah. You should, and you know, it almost doesn't matter the subject i mean it's it's helpful i mean if i if i had had formal courses in c++ uh i would be more familiar with the standard libraries and i would know more about the standard you know all the the usual suspects and i would know about

uh the patterns and those things you know but i can learn that stuff you know it's if i didn't if i didn't know how to learn i'd be stuck i'd be stuck with whatever i had studied i know some people have taken a break voluntarily or not and now they're afraid sort of or or trepidatious, is that a word? That's the same as afraid. All right, well, they're worried that they can't get back into the tech world because there's too much to learn and it's just a giant wall. Do you have any advice? Yeah, don't do that.

Don't be afraid or don't take a break? No, I mean, it's fair to be afraid. I mean, you know, when I took the leap from the math library stuff to what I'm doing now, I was afraid. But that's not a reason to stop. You know, the thing that I think would be difficult having been away would be kind of being out of the habit of learning, for one thing. And it's true, you know, the technology goes pretty fast. And I could imagine that might be a time when you might want to look around and say, okay, here's a, here's a technology that I really should learn and go through the

act of learning again, even if it's not going to be the thing that gets you your next job. But, but just the, the idea that you're, you're keeping your hand in, you're, you're, you're making progress in your own development. I mean, it's easy for me to say i haven't ever had the experience but but i think that practice is very important it's one of the reasons i really like the online courses because you can make that measurable progress and remember that learning not only isn't always painful, it can actually be fun sometimes. Oh, yeah.

And once you remember that joy, it all becomes a lot easier. Yeah, that's right. I mean, it's like backpacking. You know, there's the getting there and the being there, you know, and both are worthwhile. Yeah, but sometimes you're hiking up that hill and you're like, why? Yeah. Why am I doing this? This is my free time? Yeah. Well, I like what you said about it's okay to be afraid because that is, I mean, you don't want to be panicking, but a little bit of fear is a signal that you're

challenging yourself, right? Oh, yeah, absolutely. Yeah, you should be a little bit. I think the thing that I would say to employers, though, is that I think that a lot of employers don't understand the value of somebody who assumes they can learn whatever they need to know, as opposed to who already has on their resume exactly the thing that the employer thinks is needed. And so I think a lot of resumes don't get looked at twice that maybe should, because they don't have the right buzzwords. It's hard to put enjoys learning on a resume

and have anybody take you seriously. Yeah, I know. I know. Exactly. And I think that's a problem. It's like my greatest weakness is my willingness to work too hard. Come on. Yeah, yeah. Have you, well, I know some people.

We had a guest not too long ago a talented engineer who said uh well i'm 40 and i'd better work on important things because i'll only be an engineer for a few more years and that it just kind of made my brain fizz like really why what um do you see people like that? Um, not so much. Um, I see people that, that are in engineering that really aren't enjoying it. Uh, and that's too bad because it's, it's too much fun to, to, to not enjoy it. Um, and I could, I could imagine, you know, somebody with that kind of mindset or that just wasn't really having fun anymore, thinking that it's getting old and that they're going to burn out on it. But that's not me, obviously. I've been at it for 40 years now, so 42, I think.

And do you ever feel like you've topped out? No, I mean, I... What does that mean? Topped out like you can't learn anymore or topped out like you can't advance any further? Not advance any further. Yeah, I mean, you know, they ask you, where do you want to be in five years?

You know, that's always the... Yeah, how do you answer that now? Because I haven't had a good answer for that in about 10 years. I like that question. I mean, now I have a pretty good answer. I don't want to be on a canoe with my granddaughter. But, yeah. And my answer for probably 20 years has been, you know, I'm enjoying working at the level I'm working at.

I'm enjoying being an engineer. I want to learn new stuff. I want to work on different projects. That's enough for me. I don't want to be a manager for sure. Have you done the manager thing? Oh yeah, I did for a couple of years. Everyone should do it for a couple of years. That was not the picture of, oh, that was fun. No, that wasn't fun. That wasn't fun. And it wasn't because of the people under me, it was because of the people over me. But there's a story there, but I don't need to go into it.

I just, I got crossed by one of my upper managers and it had to do with the salary of one of my reports. And it was really unpleasant. Yeah, I think that's experience that a lot of people have when they move from engineering to management. It's not, oh, you have this whole team and now you can accomplish this. Oh, no, you have to talk to these people and your job is to actually protect these people from these people.

And it's not what you think. The upper people don't think your job is to protect. They think your job is to exploit the lower people. And you think your job is to protect and it goes very badly sometimes. It doesn't have to be that way. But I had a bad experience and I just said never again. So that was.

Did it make you a better engineer? That's an interesting question. For me, the answer is definitely yes. I gained a much greater understanding of why my managers do what they do and why they want schedules and what's probably going on in their lives when they tell me stupid things. Yeah. of why my managers do what they do and why they want schedules and what's probably going on in their lives when they tell me stupid things. Yeah, and I have a manager now

who's really good for me. He's got a way of helping me be a little more disciplined about my organizational skills and setting sort of short-term goals and hitting them one after the other. And that's really healthy for me, you know. And so that's fantastic. I mean, I haven't, you know, there are all different managerial styles and I just happen to have fallen into another good one.

And you're still getting advice and you're still taking it. Oh, yeah. Yeah. Yeah, I think that's important. Do you have any advice for people feeling like they've reached the end point of their career and they can never be more than senior engineer? Is that a problem? That's the advice. That's it right there.

That is actually the advice. Yeah, I think so. Why is that a problem? Sure, you went from intern to junior engineer to engineer to senior engineer in five years. That was great. You were fantastic.

You were a genius. But what about brigadier general engineer? And there it stops. Admiral engineer. Why is that a problem? Yeah. Yeah.

You know, it's not for me. You know, go find another problem to solve. If you're working in a company that everything is the same every day and you're tired of it, go find another problem to solve. If you're working in a company that everything is the same every day and you're tired of it, go find another problem to solve. It happens to be a fairly good time in the industry now to be looking. Oh, yeah.

If you're tired of what you're doing, go do something else. There are so many jobs. Was there a time when you went from being one of the youngest people in the office to realizing you were one of the oldest no no although i guess um it there was a point in a meeting where i realized it happened to be my birthday where i realized that the digits of my age in hex were the reverse of the digits of my age in decimal. And so I mentioned this to the- I feel like we should make that a puzzler or something.

I mentioned this to my group and anyway, and so I won't give it away. Everybody else can figure it out on their own scram match scribbling on the paper here but uh but anyway yes uh and it turns out really at nvidia you know it's it is a pretty young company uh they have a lot of young guys there but they have a lot of graybeards too you know there there's there's some some some seniority in the upper ranks and the higher engineers and architects. And so I don't feel stranded at NVIDIA in the least. Silicon Valley has a bit of a reputation for ageism.

Have you ever noticed it? No, although I'm pretty sure that, although I don't expect it to happen, but, but that if I were to be laid off, I would experience it right now, you know, at, at my age, would I expect to, you know, just drop into another great tech job? Probably not. Probably not. That's tough. And you have been in NVID nvidia for long enough you haven't had to look for a while that's right that's right and and you know the the job i'm on now kind of just came to me uh which is great you know but that's a result of being you know established at a place and for for a number of years i have built up sort of a reputation, you know, for being competent. And, you know, people trust that if I take something on, I'll get it done. And that's, that would be a hard thing to sell, I think, at my age, if I were just looking off the street cold, if I didn't have a reference, if I didn't have somebody on the inside that,

that had worked with me before and knew what I could do. But you've worked all over the industry, probably do have people on the inside for a lot of places i do that's right yeah so you know it might it might go both ways uh but yeah when i when i joined nvidia there were a couple of questions they weren't i mean they hired me obviously so it wasn't it wasn't like that that they were concerned, but they're kind of along the lines of, uh, you know, you're, you're, you're getting along in your career, you know, and I just said flat out, I'm not tired yet. I'm still having fun. As a manager, when I asked people, what did you want to do in five years? If they didn't have an answer,

it was hard because it meant that I didn't have a way to motivate them. I didn't have a way to encourage them. And I found that difficult. You know, everybody has buttons and levers and you just took away one of my buttons. Now I have no way to A, tell you you're doing a good job or b convince you that this job will continue in a direction you want to go have you had managers try to push you in directions you didn't want to go no not really not really man you just had a charmed career it's annoying i it what i've been lucky i will freely admit i've i've been lucky i've i've enjoyed it you know if you were 20 something and unencumbered by grandchildren or children or girl scout troops what what would you go into oh you mean would i do it over again

would you do this would you do biotech would you would you do embedded systems would you do it over again? Would you do this? Would you do biotech? Would you do embedded systems? Would you do space? What is the most exciting thing for you right now if you could choose anything to start? Yeah, I don't know. I guess I would be hard-pressed to pick something that is better suited for my, sort of what I see as my skills, my abilities.

Than to be working at the interface between hardware and software. And designing on both sides. That's contributing to the problem solving on both sides of that interface. That's what I, I understand how that works. And it'd be hard to pick something else. It'd be hard for me too. So, I totally agree with that answer. And I don't think

the whole embedded software is going away anytime real soon. I know. I think we've got a while yet for the robots to figure out how to replace us. A friend of mine, her husband moved up to Seattle some years ago. This is back from Olivetti days. What is he going to do up there? Work on windows for pay toilets.

I just thought of that as a really odd answer. I believe it was embedded Windows. Yeah. No. Things we get to do as engineers. What? Do you have a final question?

Oh, is that what the nod was for? Well, it was actually because I typed you a suggestion for your final question in case you didn't have one. Paying attention to the interview here. I'm not just... I know, I know, I know. And then you tell me my typing's too loud. This is, yeah. 151 episodes.

Insert Bacardi rum joke right here. Okay, one more question. Was there any point you made a decision that you look back on and think, well, if I hadn't made this, things would have gone much differently? Oh, yeah. Well, yes. That would be the very beginning of my career. Because it was kind of a tough time. It was after the first big oil crisis.

And first time people had experienced, you know, lines at gas stations and it was a little hard looking for work when I was just out of, uh, out of my, my, uh, teacher credential program with just a math major. And, uh, I had two offers. I had an offer in San Francisco in a high rise to be an actuary for an insurance company and i had an offer from general electric space division to go do uh satellite control basically a glorified operator computer operator uh and i took the GE job. And I didn't really know that much about computing. But I got there and I had a six-month wait before I got my clearance. And that's when I learned how to code in Fortran and Jovial.

And I haven't looked back since. So there's another universe out there where you're an actuarial person? Yeah, that would have been a possibility. I'm pretty sure i wouldn't have enjoyed it as much i might might be wrong but uh but that was a that was a that was a fork in the road you probably would have ended up at the nsa forensics accounting it's pretty cool i could see that all right well i think we have kept you long enough um do you have any last thoughts you'd like to leave us with? Yeah, I think that we already touched on it, but it's this. It's just assume that when you're looking at a project that scares the heck out of you, that you can learn whatever you need to learn

to make it happen. That is excellent advice. My guest has been Paul Sidenblatt, architect and software engineer at NVIDIA. Thank you so much for being here. Oh, you're welcome. It's been a great conversation. Thank you also to Christopher

for producing and co-hosting. And of course, thank you for listening. Please also check out our blog and newsletter. If you are so inclined to send newsletters, you can find it all on the Embedded FM website, along with a contact link if you'd like to say hello, which you're welcome to do, or to tell me what the answer is to that hex problem, which I will have figured out by the time you said, nevermind. We'll be here next week. In the meantime, here's a final thought.

This one is from Neil Gaiman, The Ocean at the End of the Lane, which was a fantastic book. Grownups don't look like grownups on the inside either. Outside, they're big and thoughtless and they always know what they're doing. Inside, they look just like they always have, like they did when they were your age. Truth is, there aren't any grown-ups, not one

in the whole wide world. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, Thank you.