Microsoft Research Podcast - What’s Your Story: Desney Tan

Episode Date: November 16, 2023

In this new Microsoft Research Podcast series What’s Your Story, Johannes Gehrke explores the who behind the technical and scientific advancements helping to reshape the world. A systems expert whose 10 years with Microsoft spans research and product, Gehrke talks to members of the company’s research community about what motivates their work and how they got where they are today.Across his time at Microsoft, Desney Tan, Managing Director of Microsoft Research Redmond, has had the experience of shepherding research ideas into products multiple times, and much like the trajectory of research, his life journey has been far from linear. In this episode, Tan shares how he moved to the United States from Singapore as a teenager, how his self-described “brashness” as a Microsoft intern helped shift the course of his career, and how human impact has been a guiding force in his work.

. Early in the career, I always looked at successful people, and it always felt like they had a goal and it was a very nice straight line to get there and they did all the right things. I don't know anyone today that I deemed to be successful that had a straight line path and did all the right things. . Microsoft Research works at the cutting edge. But how much do we know about the people behind the science and technology that we create?

This is What's Your Story and I'm Johannes Gerke. In my 10 years with Microsoft, cross-product and research, I've been continuously excited and inspired by the people I work with, and I'm curious about how they became the talented and passionate people they are today. So I sat down with some of them. Now I'm sharing their stories with you. In this podcast series, you'll hear from them about how they grew up, the critical choices that shaped their lives, and their advice to others looking to carve a similar path. In this episode, I'm talking with Desney Tan, a long-time Microsoft executive whose experience with the company spans computational neuroscience, human-computer interaction,

and health and the life sciences. His research contributions have impacted a wide range of Microsoft products. Desney was previously vice president and managing director of Microsoft Health Futures and is now managing director of Microsoft Research Redmond. Much like the trajectory of research, Disney's life journey has been far from linear. He left Singapore to attend school in the United States as a teenager, then worked in autonomous navigation for NASA and in VR for Disney before landing here at Microsoft. Here's my conversation with Disney, beginning with his childhood.

Born and raised in Singapore. Dad was an architect. Mom did everything to run the family. When I turned 13, mom and dad came to me and they said, hey, would you like to try something new? I said, sure. Had no idea what they were thinking. Two weeks later, they sent me to the US to study. Looking back, sometimes I flippantly claim

I was just eating too much at home and so they had to send me away. But actually, I think it was prescient on their part. They sort of looked at my path. They looked at the education system. They looked at the way I learned and the way I created and the way I acted. And they somewhat realized, I think, very early on that the U.S. would be a great place for me to sort of flourish and sort of experiment and explore and grow. And so how did it work? You just went by yourself?

I had an aunt and an uncle in Louisiana. Spent a couple of years in high school there. Sort of fun side story. The high school looked at my math curriculum in Singapore, and they said, oh, he's at least a year ahead. So they skipped me a year ahead. And then through some weird miscalculations on their part, they actually ended up skipping me nearly two years ahead.

Oh, wow. And by the time we realized I had already integrated into school, the courses were just fine. And so I ended up skipping a lot of years. So you graduated from high school? Pretty early. I was 15. I graduated from high school.

Got to college, had no idea what I wanted to do, what 15-year-old does. Ended up in liberal arts college, so University of Notre Dame. So I don't know how mom let me do this, but I got all my acceptance letters together. I said, I don't know anything about college. I don't know where I want to go. I don't know what I want to do. I'm going to toss all the letters up in the air, and the one that lands on top is the school I'm going to. And that's what you did? Yeah, that's exactly what I did. The divine intervention, let's call it. Neridame landed on top.

You know, I switched majors a bunch of times. I started off in aerospace. I did chemical engineering, civil engineering. I was on the steps of becoming a priest until they sent me away. They said, hey, if it's not a mission and a calling, go away and come back later. And I ended up with a computer engineering degree. You know, had great mentors, you know, who looked out for me. I had a couple of guardian angels out there, you know, guided me along.

And that was just a wonderful breadth of education. Went back to the military for a couple of years, served there for a couple of years, did a bunch of growing up. That's quite a change, right? From being like in college and then going back to the military for a couple of years. Served there for a couple of years. Did a bunch of growing up. That's quite a change, right? From being in college and then going back to the military. Yeah. It was a mandatory service in Singapore. So I went back.

Had a ton of fun. Learned a bunch of stuff about the world, about myself. I claim the military is one of the few organizations in the world that takes an 18-year-old and teaches them leadership. teaches them about themselves and teaches them about how to push themselves and where the boundaries are. And so fairly accidentally, I got to benefit from all of that. At the end of that, I realized my computer engineering degree was, you know, I realized two things. One, my computer engineering degree was a little outdated by the time I got out of the military. And two, that I didn't love being told what to do. Okay. So I came back, did grad school. I was at Carnegie Mellon. Ended up getting hooked up with

wonderful professor Randy Pausch. The last lecturer, right? Who gave the last lecture in his last days. You know, learned a ton from him, not only about academics and scholarship, but also about life and leadership. And he was at the intersection of graphics and HCI, right? That's correct. So what was your PhD in? My PhD was actually looking at distributed displays in virtual reality. So how the human brain absorbs information and uses the world around us to be able to interact

with digital data and analog data. Early on in a really important field already. Yeah, no, it was great. Spent a couple of years with NASA in the Jet Propulsion Lab doing autonomous navigation. This was the early days of AI and planning.

So those aerospace engineering classes, were they actually useful? All the classes I took ended up coming back to be useful in a number of ways. And actually, you know, the diversity of viewpoints and the diversity of perspectives is something that sat very deeply in me. So anyways, you know, spent some time at NASA,

spent some time at Disney with the Imagineers building virtual reality theme parks. This was the late 90s, early 2000s. So Disney at the time had all the destination theme parks, Disneyland, Disney World, places you would fly for a week and spend a week at. Their goal was really to build a theme park in a box that they could drop down into the urban centers. and spend a week at. Their goal was really to build a theme park in a box

that they could drop down into the urban centers. And the only way to get a theme park into a building was digital experiences. And so this was the very early days of VR. We were using million dollar military grade headsets. They were 18, 19 pounds. Wow. Disney was one of the companies and has sat with me for a very

long time that designs experiences for every single person on earth. So these headsets had to work on your two-year-old. They had to work on your 102-year-old. They had to work on a person who spoke English, who read, who didn't read, who didn't speak English, you know, tall, short, large, small, all of it. And they did a wonderful job finding the core of what it means to be human and designing compelling experiences for all of us. And that was a ton of fun. We ended up deploying these facilities called Disney Quest. There was one in Chicago, one in Orlando. They just closed them down a couple of years ago

because actually all the VR rights have now migrated into the theme parks themselves and it was actually a VR experience you would go it was a VR experience they dropped them down they had basically buildings there were floors full of classic

and new age arcade games and then there were VR experiences that you could run around in and interact with. Interesting. I've never, I mean, I lived in Madison for four years, but I've never heard of that quest experience. Because it seems to be a fun way to experience Disney. It was super fun. Yeah, I personally got to work on a couple of rides. There was Pirates of the Caribbean.

Oh, wow. So you put on, you know, family would put on headsets and kind of run around shooting pirates and what have you. And then the Aladdin ride was, I thought, one of the better rides. Oh, wow, yeah. Sitting on a magic carpet, as you can imagine. Oh, yeah, that sounds fun. Perfectly scripted for it.

Anyways, I ended up at Microsoft largely because entertainment technology, while a lot of fun and while I learned a ton, was strangely unsatisfying. And there was something in me that was seeking human impact at scale in a much deeper and much more direct way. And so I thought I'd be here for three or four years, largely to learn about the tech industry and how large pieces of software were deployed before going off and doing the impact work. I've now been here for nearly 20 years. And where did you start?

Did you start right away at Microsoft Research or were you first in a product group? My career here has been a cycle of starting in Microsoft Research, incubating, failing, trying again, failing again, at some point screaming Eureka, and then doing my tours of duty through the product groups, commercializing, productizing, commercializing, seeing it to at least robustness and sustainability if not impact, and then coming back and doing it again. And the thing that's kept me here for so long is every time I've completed one of those cycles

and thought I was done here, the company or the world in some cases would throw a bigger, thornier, juicier thing in front of me. And Microsoft has always been extremely encouraging and supportive of taking on those challenges and really innovating and opening up all new, whole new opportunities. I mean, this whole cycle that you're talking about, right, of sort of starting out small

at MSR, you know, having sort of the seed of an idea and then growing it to a bigger project. At some point in time, transitioning it into the product group and actually really making it a business. So tell me about, you said you have done this a few times, and once you were even highly successful. I'd love to learn more about this because I think it's so inspiring for everybody to learn more about this. It's been magical. I have to say before going into any of these stories that none of these paths were architected. As you all know, they never are. So actually, my first experience was as an intern here. And I was a sort of brash, perhaps rash intern. I was working on virtual

reality. And in the evenings, I would meet with folks around the company to learn more. And I met with a team that was building out multi-monitor functionality in Windows NT. Prior to Windows NT, Windows computers had one and only one monitor, and they started to build the functionality, build multiple. As the Brass grad student, I had different thoughts about how this should be implemented and couldn't convince anyone of it. And so in the evenings, I ended up starting just to build it. At the end of the internship, in addition to all the stuff I was doing, I said, hey, by the evenings, I ended up starting just to build it. At the end of the internship, in addition to all the stuff I was doing, I said, hey, by the way, I've built this thing, you know, take it or leave it, here you go. And it ended up being the thing that was implemented

in NT for a variety of reasons. That really got me hooked. Prior to that, I had imagined myself an academic going back and, you know and being a professor somewhere in academia. As soon as I saw the thing I did and that Microsoft actually polished up and made good... And shipping out millions and millions of desktops. There was no getting away from that. When I first got here, MSR had actually hired me thinking I'd work on virtual reality. And I got here and I said, hey, VR, I've just done a ton of VR. VR is probably 15 or 20 years out from being democratized and consumerized. I'm going to do something for a couple of years and then I'll come back to this.

So I got into computational neuroscience, looking at sensors that scanned or sensed the brain and trying to figure out mental state of people. And I had the imagination that this would be useful both for direct interaction, but also for understanding human behavior and human actions a little bit better. We won't go into that work,

but what happened with the productization of that was I went, this was at the time when Bill Gates was actually pushing very hard on tablet PCs and the stylus and the pen as an interesting input modality. The realization we had was, hey, we've got spatial temporal signal coming off the brain we're trying to make sense of.

The tablet guys had spatial temporal signal coming off a pen they were trying to make sense of in handwriting recognition. And so we went over and we said, hey, what interesting technological assets do you have that we can steal and use on the brain? Turns out they were more convincing than us. And so they said, hey, actually, you're right. The problems do look similar. What do you have that you could bring over? And so if you look at the handwriting recognition system, even that stands today, it's a big mess of a neural network, largely because that came out of interpreting neural signal that got transferred into the handwriting recognizer.

And so I ended up spending maybe two and a half years working not only on the core recognition engine itself, but also the entire interface that ran around the tablet PC and tablet input panel. But that's sort of an interesting realization, right? You came because you thought you would land technology X for application Y, but actually you landed for a very different application.

That's right. And each cycle has had a little bit of that surprise and that serendipity, which we've now built into the way we do research. And you sort of head down a path because it moves you forward as quickly as possible, but you keep your eyes peeled for the serendipitous detours and the discovery that comes out of that. And I think that's what makes Microsoft Research as an organization so compelling and so productive, right? Is we do run very fast, but we have the freedoms

and the flexibility really to take these windy paths and to take these detours and to go flip over rocks, some of which end up being dead ends, others of which end up being extremely productive. Right. And so if you think about, let's say, a junior person in the lab, right, they're sort of looking at you and your career and say, wow, what steps should I take to become as successful as Disney?

What advice would you give them? Right. Because it seems like you have always had sort of MSR as sort of your rock, right. But then you jumped over the river a few times, but then came back, jumped over again, came back. First off, I don't know that Disney has been so successful so much as, you know, the people around Disney have been extremely successful and Disney has gotten to ride the wave. But yeah, no, I mean, everyone's, you know, as I look around the table and the board, everyone has a slightly different journey

and everyone has slightly different work styles and mindsets and personalities and risk tolerance and what have you. So the first thing really is not to try to fully emulate anyone else. I always claim we're kind of like machine learning models, right? We should be taking input data, positive and negative, and building our models of ourselves and our models of the world

and really operating within that context. I think having a North Star, whether it's implicit or explicit, has been extremely useful for myself and the people around me. By North Star, you mean like a philosophical North Star or a technical North Star or a North Star in what you want to be? Yes, yes, yes. All of it. So tell me more about your personal North Star.

For us, for myself, it's really been about human impact. Okay. Everything we do is centered on human impact. We do research because it's one of the steps towards achieving human impact. We productize because it's one of the steps towards human impact. Our jobs are not ever done until we hit the point of human impact.

And then they're not quite done because there's always more to be had. So I think having that, perhaps a value system, at least sort of grounds you really nicely and creates, I think, or can create a courage and a bravery to pursue, which I think is important. You know, different people do this differently, but I have been very lucky in my career to be surrounded by people that have been way, way, way better than myself. And extremely generous of their passions and their skills and their expertise and their time. You know, ask it just about any successful person

by whatever definition, and I think they'll tell you the same thing, that it's the people around. And then being tolerant, maybe even seeking of this windy path. You know, when I was early in the career, I always looked at successful people, people I deemed to be successful. And it always felt like they had a goal and it was a very nice straight line to get there. And they did all the right things and took all the right steps.

And I don't know anyone today that I deemed to be successful that had a straight line path and did all the right things. Yeah. And it's often these setbacks in, you know, one's career that actually give you often some of the best learnings because either some things that you've sort of done structurally wrong or some things that, you know, you really need more experience. And, you know, that setback gave you that experience. Yeah. So, so one of other question around this is also just around change, right? Because especially right now, we're living in this time where maybe the rate of change,

especially in AI, is kind of unprecedented. I mean, benchmarks are falling in like a quarter of the time than they were thought to be lasting. You know, we all have played with chat GPT, just extrapolate that out a few more months, if not years, right?

Yeah. OpenAI is here talking about AGI. So how do you think about change for yourself and evolution and learning? And do you have any routines? How do you keep up with everything that's going on? Yeah, it's a good question. I guess the overarching philosophy, the approach that I've taken with my career, is that everything's constantly in change.

The rate of change may vary and the type of change and the mode of change might vary, but everything's constantly changing. And so our jobs at any given point are to understand the context in the world, in the organization, with the people around you, and really be doing the best that you can at any given moment. And as that context changes, you kind of have to dynamically morph with it. I subscribe pretty fully to the lean startup model. So, you know, formulate hypotheses. And this is the research process really, right? Formulate hypotheses, test them as quickly as you can, learn from that, and then do it again and rinse and repeat. And then, you know, you could sort of plot your path and steer your path through based on that. And so we operate very much on that. As the world changes, we change.

As the org changes, we change. And there's a certain robustness that comes along with that. It's not all roses and obviously changes and uncertainty is a difficult context to operate in. It's super interesting because it also speaks to some of the things that one should look out for when doing research. If you're saying, well, I have these hypotheses and I want to quickly test them.

If I'm in a field or if I work with data that I cannot really use, where the testing of any hypothesis will take months, if not years, to bring out, this might not be the best research direction. So how should I think about the choice of research problems with this change in mind? Yeah, yeah. I don't know.

I guess, again, I'm brash on this. There are very few problems and spaces that can't be navigated. And so things that seem impossible at first glance are often navigable and with a little bit or maybe sometimes a lot of creativity. If our jobs are to take Microsoft and the rest of the world to places that Microsoft and the rest of the world might not get itself to, hopefully positive places, then we're going to have to do things in a way that is probably unnatural for Microsoft and the rest of the world to get there. And the company and the organization MSR has been extremely supportive of that level of creativity. Can you give an example of that?

We had Cortana, which is our speech recognition and conversational engine. We didn't really have a platform to deploy that on. At the same time, we saw a bunch of physicians, clinicians struggling with burnout because they were seeing patients for less than half the time. They were spending more of their time sitting in front of the computer documenting stuff than they were seeing patients and treating patients. We said, hey, what if you put the two together? What if you sat in the room, listened to the doctor and the patient, and started to automatically generate the documentation? And in fact, if you did that, you could structure the data, which leads for better downstream analytics.

And if you did that, you could start to put machine learning and AI and smarts into the system as well. That project, which was called EmpowerMD, led eventually after a bunch of missteps and a bunch of learnings and a bunch of creativity to a very deep partnership with Nuance and creation of Dragon Ambient Experience and the eventual acquisition thereof of that company. And it's just a wonderful product line. It's kind of a neat way to think about data and intelligence and human augmentation and integration into otherwise messy, noisy human processes.

But yeah, I think with enough creativity, we've bumped into very, very few brick walls. What I love about the story is that it's not about a specific technology choice, but it's more about a really important problem. That's right. If your problem is right and if your conviction is right about the value of the solution,

you build teams around it, you build processes around it, you're creative in the way you execute, and I'd say more times than not we end up getting there. Yeah. Well, I love that insight because it's often

much more valuable to solve an important problem than to learn to land some deep technology on a problem that very few people care about. And it seems like that's what you've done here. Yeah. Well, it was really great and inspiring to hear from you, Desney. Thanks so much for the conversation. Yeah. Thanks for having me, Johannes.

To learn more about Desney's work or to see photos of Disney during his winding journey to Microsoft, visit aka.ms slash researcher stories.