Microsoft Research Podcast - What’s Your Story: Emre Kiciman

Episode Date: August 1, 2024

Emre Kiciman shares how some keen observations and a desire to have front-end impact led him to make the jump from systems and networking to computational social science and now causal analysis and large-scale AI—and how systems thinking still impacts his work.Learn more:AI Controller Interface: Generative AI with a lightweight, LLM-integrated VM (blog)AICI: Prompts as (Wasm) Programs (GitHub)

I think it's really important for people to find passion and joy in the work that they do. At some point, do the work for the work's sake. I think this will drive you through the challenges that you'll inevitably face with any sort of project and give you the persistence that you need to really have the impact that you want to have. Microsoft Research works at the cutting edge. But how much do we know about the people behind the science and technology that we create? This is What's Your Story? And I'm Johannes Gerke.

In my 10 years with Microsoft, across product and research, I've been continuously excited and inspired by the people I work with. And I'm curious about how they became the talented and passionate people they are today. So I sat down with some of them. Now I'm sharing their stories with you. In this podcast series, you'll hear from them about how they grew up, the critical choices that shaped their lives, and their advice to others looking to carve a similar path. In this episode, I'm talking with Emre Kicermann, the Senior Principal Research Manager leading the AI for Industry research team at Microsoft Research Redmond. After completing a PhD in systems and networking in 2005, Emre began his career with Microsoft

Research in the same area, studying reliability in large-scale internet services. Exposure to social data inspired him to refocus his research pursuits. His recent work in causal analysis, including Do-Why, a Python library for causal inference, is helping to connect the what's and why's in the abundance of data that exists. Meanwhile, his work with large language models is geared toward making AI systems more secure and maximizing their benefit to society. Here's my conversation with Emery, beginning with some of his work at Microsoft Research and how he landed in computer science. Welcome to What's Your Story? So, can you just tell us a little bit about what you do at MSR?

Sure. I work primarily on two areas at the moment, I guess. One is causal analysis, where we work on trying to answer cause and effect questions from data in a wide variety of domains, building that horizontal platform. And I work a lot recently, especially with this large language model focus, on the security of AI-driven systems. How do we make sure that these AI systems that we're building are not opening up new vulnerabilities to attackers? Super interesting. And maybe we can start out, even before we go more in-depth into that, by how did you actually end up in computer science? I learned that you grew up in Berkeley? Yeah, on average, I'd like to say.

On average? On average. So I moved to the US with my parents when I was two years old, and we lived in El Cerrito, a small town just north of Berkeley. And then around middle school age, we moved to Piedmont, just south of Berkeley. So on average yes I grew up in in Berkeley and I did end up going there for college and you asked about how I got into computer science when I was probably around third or

fourth grade my dad who was a civil engineer decided that he wanted to start a business on the side and he loved software engineering and wanted to build software to help automate a lot of the more cumbersome design tasks in the design of steel connections. And so he bought a PC and brought it home and started working on his work, but then that was also my opportunity to Learn what a computer was. So there was your first computer. It was a x86. Yes. It was an IBM PC The first x86 the one before the 286 and It wasn't the very original PC did have a CGA color graphics adapter.

So we could have four colors at once. Nice. And yeah, it came with, luckily for me, I guess it came with a basic manual. So reading that manual is how I learned how to program. And there's the typical IBM white box with a monitor on top of it and a floppy drive? Exactly. Two floppy drives. Two floppy drives. Two floppy drives, yeah. So you could copy from one to the other. Five and a quarter of it and a floppy drive? Or how should I picture it? Exactly, yeah. Two floppy drives. Two floppy drives.

Two floppy drives, yeah. So you could copy from one to the other. Five and a quarter or three and a half? Five and a quarter. Five and a quarter, okay. Yeah, the loud clickety-clack keyboard. And yeah, a nice monitor. So not the green and black,

the one that could display the colors. And yeah, had a lot of fun with programming. So what were some of the first things that you wrote? A lot of the first ones were just the examples from the book, the four loops, for example. But then after that, I started getting into some of the building like little mini painting tools. You could move a cursor around the screen, click a button and paint to fill in a region, and then save the commands that

you did to make graphics. Eventually that actually turned into, a friend and I really enjoyed playing computer games, so we had in our mind, we're going to build a computer game. Who doesn't think that? Of course. Of course. Right? And so we had a choose your own adventure style program. I think we had maybe even four or five screens. You could step through, right? And he was able to get some boxes and we printed some manuals

even. We had big plans but then we didn't know what to do, how to finish the game, how to get it out there. But we had a lot of fun. Wow, that sounds amazing. Really fond memories, yeah. That sounds amazing. And then you went to Berkeley afterwards. Is that how you realized your passion or how you decided to study computer science? Yeah, so from that age I was set on computing. I think my parents were a bit of a devil's advocate.

They wanted me to consider my options. So I did consider mechanical engineering or industrial engineering in maybe junior year of high school, but it never felt right. I went into computing, had a very smooth transition into Berkeley. They have a local program where students from the local high school can start to take college classes early. So I'd even started taking some computer classes and then just went right in in my freshman year.

Sounds like a very smooth transition. Anything bumpy on the ride up there? Nothing really bumpy. I had one general engineering class that somehow got on my schedule at 8 a.m. freshman year. That's a tough one. That's a tough one, yeah. And so there were a few weeks I didn't attend class. And I knew there was a midterm coming up. So I show up. Because next week,

there's a midterm. I better figure out what they're learning. And I come in a couple minutes late, because even though I'm intending to go, it's still an 8 a.m. class. I show up a few minutes late, and everyone is heads down, writing on pieces of paper, the whole room is quiet. And the TA gives me a packet and says, you might as well start now. Oh no. And I'm like freaking out, like this is a bad dream.

And I'm flipping through, not only do I not know how to answer the questions, I don't understand the questions. Like the vocabulary, it's only been three weeks, how do they learn so much? And then I notice that it's an open book exam and I don't have my book on top of it. But what I didn't notice and what became apparent in about 20 minutes, the TA clapped his hands and said, all right, everyone put it down. We'll go over the answers now. It was a practice. Oh, lucky you.

Oh my God, yes. So I did nothing but study for that exam for the next week and did fine on it. So you didn't have to drop the class or anything like that? No, no, no. I studied enough that I did reasonably well. At what point in time was it clear to you that you wanted to do a PhD or that you wanted to continue your studies? I tried to explore a lot during my undergrad. So I did go off to industry for a summer internship. Super fun. Where did you work?

It was Netscape. Oh, Netscape. And it was a joint project with IBM. Which year was that? This would have been around 93. 93, okay. So the very early days of Netscape, actually. Yeah, they were building Netscape Navigator 4. The project I was on was Netscape Navigator for OS 2. IBM's OS 2 had come out and was doing poorly against NT,

and they wanted to raise its profile. And this team of 20 people were really just focused on getting this out there. And so I always thought of, and I was an OS2 user already, which is how I got onto that project. And how was the culture there? The culture, it's what you would think of as a start-up culture. They gave out all their meals, there was lots of fun events, dentists came into the parking lot like once a month or something like that. The dentist? There was like a, yeah, it was, yeah, everyone's working too much at the office, so the company wanted to make things easy.

But the next summer then, I did a research internship, a research assistantship at Berkeley. I worked with Randy Katz and Eric Brewer and got into trying to understand cell phone networks and what they were thinking about cloud infrastructure for new cellular technologies. And Eric Brewer was here at that point in time already running into me? He was already running into me, yeah. He'd already started it. I don't think it was public yet at the time, but maybe getting there. Well, this was right at the beginning when all the cloud infrastructure was defined and a lot of the basics were set.

So you did this internship then after your junior year, the second one? Yeah, after my junior year. It was then senior year and it's time to apply for what's going to come after college. And I knew after that internship at Berkeley, I knew I was going to go to a PhD. So what was the thing about the internship that made you want to stay in research? Oh, it gave a vision of the future. We were playing with, there were people in the lab playing with video over the internet and teleconferencing and just seeing that, it felt like you were seeing into the future

and diving deep technically across the stack in a way that the industry internship hadn't done. And so that part of it, and obviously lots of particulars, lots of internships do go very deep in industry as well, but that's what struck me and it's that kind of it, and obviously lots of particulars, lots of internships do go very deep in industry as well, but that's what struck me and is that kind of wanting to learn was the big driver. And what excites you about systems as compared to something that's more applications-oriented or more touching the user? I feel like systems, you always have to have this kind of drive for infrastructure and

for scale and for building the foundation as compared to directly impacting the user. I think the way I think about systems today, and I can't remember what it was about systems then, I'd always done operating systems was one of my first upper division courses at Berkeley and everything. So I certainly enjoyed it a lot. But the way I think about systems now, and I think I do bring systems thinking to a lot of the work I do, even in AI and responsible AI, is the way you structure software. It feels like you should be making a statement about what the underlying problem is, what is the component you should be building from an elegance or first principles perspective. But really, it's about the people who are going to be using and building and maintaining

that system. You want to componentize it so that the teams who are going to be building the bigger thing can work independently, revise and update their software without having to coordinate every little thing. I think that's where that systems thinking comes in for me, is what's the right abstraction that's going to decouple folks from each other. That's a really great analogy because the way it was once told to me was that systems is really about discovering the beauty in large software. Because once you touch the user, you sort of have to do whatever is necessary to make the user happy. But in the foundations, you should have simplicity, you should have ease, you should have elegance.

Is that how you think about it? I do think about those aspects, but it's for a purpose. You want the elegance and the simplicity so that you can have one team working on layer one of the stack, another team working on layer two of the stack. And you don't want them to have to talk to each other every ten minutes when they're making any change to any line of code, right? And so thinking about what is the more fundamental layer of abstraction

that lets these people work on separate problems, that's what's important to me. And of course, that then interplays with people's interests and expertise. And as people's expertise evolves, that might mean that that has implications for the design of your system. And so you're an undergrad, you have done this research experience, you now apply. So now you go to grad school. Did you do anything fun between your undergrad and grad school? No, I went straight in. Right straight in?

Right straight in. I did my PhD at Stanford, so I went a little ways away. To a rival school, isn't it? To a rival school. Right. Well, the undergrad school wins. I think that's the general rule of thumb. But I did continue working with folks at Berkeley.

So my advisor was also from Berkeley. And so… Who was your advisor? My advisor was Armando Fox. And we had… Recovery-oriented computing? Yes, exactly. Recovery-oriented computing.

And the other person on the recovery-oriented computing project… Dave Patterson. …was Dave Patterson. Yeah. So it was really a true sort of Stanford-Berkeley joint project in a way. Yes. Yeah. And that was my PhD. The work I did then was the first work to apply machine learning to the problem of fault

detection and diagnosis in large-scale systems. I worked with two large companies. One of them was Amazon. One of them was anonymous to test out these ideas in more realistic settings. And then I did a lot of open-source work with J2EE to demonstrate how you can trace the behavior of your system and build up models of its behavior and detect anomalies.

Funnily enough, and this is going to sound a little alien to us now, maybe in today's world, Dave and Armando would not let me use the phrase artificial intelligence anywhere in my thesis because they were worried I would not be able to get a job. I see. Because that was sort of one of, I mean, AI goes through these hype cycles and then the winters again. And so this was one of one of, I mean, AI goes through these hype cycles and then, you know, the winters again. And so this was one of the winter times. This was definitely a winter time.

I was able to use the phrase machine learning in the body of the thesis, but I had to make up something about statistical monitoring for the title. So what is the actual final title of your thesis, if you remember it? Statistical monitoring for fault detection and diagnosis in and Diagnosis in Large-Scale Internet Services or something like that. So you replaced AI with statistical modeling and then everything? Yes. Yeah. Everything.

Then it didn't sound too hypey. And then after your PhD, you went straight to MSR, is that right? Yeah. I mean, so here I'm coming out of my PhD with a focus on academic style research for large scale systems. Kind of boxed myself in a little bit. No university has a large scale internet service

and most large scale internet service companies don't have research arms. So Microsoft Research was actually the perfect fit for this work. And when I got here, I started diving in and actually expanding a little bit and thinking about what are the end-to-end reliability issues with our services. So assuming that the back-end is running well, what else could go wrong that's going to get in the way of the user?

So I had one project going on wide-area network reliability with David Maltz. And one project... Who is now CDP in Azure Networking. Who's now, yeah, leading the head of Azure Networking. And one project on how we can monitor the behavior of our JavaScript applications that were just starting to become big, like around then is when the first 10,000 line, 100,000 line of code, JavaScript applications appearing.

And we had no idea whether they were actually running correctly. They're running on someone else's browser and someone else's operating system, and we didn't know. A big one at that point in time, I think, was Gmail, right? This was sort of a really big one, but did we have any big ones in Microsoft? Gmail was the first big one in the industry.

Hotmail, was it also based on JavaScript? Hotmail was not initially JavaScript based. The biggest one at that time was our Maps, not Bing Maps, but whatever we called it. MSN Maps? MSN Maps, yeah. And so you applied your techniques to that code base and tried to find a lot of bugs? Yeah, this project was, and this was about data gathering, right? So I'm still thinking about it from the perspective

of how do I analyze data to tell me what's going on. We had data for the wide area network, but these web applications, we didn't have any. So I'm like, I'm going to build this infrastructure to collect the data so that in a couple years, I can analyze it. And so what I wrote was a proxy that sat on the side of the IIS server and just dynamically

instrumented all the JavaScript that got shipped out. And the idea was that no one user was going to pay the cost of the instrumentation, but everyone would pay a little small percentage, and then you could collect it in the back end to get the full complete picture. Right. It's so interesting because, I mean, in those days, right, you still thought maybe in terms of years and so on, right? I mean, you said, well, I instrumented that maybe in a year to have some data, and today

it happens, and I instrument it tomorrow to have enough data to make a decision on an A-B test and so on, right? It was a very different time, right? And also, it was probably a defining time for Microsoft because we moved into online services, and we moved into large-scale internet services. So it must have been exciting to be in the middle of all of this. It really was. I mean, there was a lot of change happening both inside Microsoft and outside Microsoft.

That's when, soon after this, is when social networking started to become big, right? You started seeing Facebook and Twitter show up and search became a bigger deal from Microsoft and we started investing in Windows Live and then Bing. And that's actually my manager, Yimin Wang, actually joined up with Harry Shum to create the Internet Services Research Center with the specific focus of helping Bing. And so that also shifted my focus a little bit. So it had me looking more at some of the social data that would kind of take my trajectory on a little bit further. Right. I mean, so you're unique in that people very often, they come in here and they're specialists

in systems. And they branch out within systems a little bit and, of course, move at the time. Maybe now they do AI infrastructure. But you have really moved quite a bit, right? I mean, you did your PhD on systems and AI, really, the way I understand it. Then you worked here a little bit more on systems and wide area and large scale systems. But then you really became also an expert in causality and looked at sort of the social side. And now you, of course, have started to move very deeply into LLMs.

So rather than talking about the topics itself, how do you decide? How do you make these decisions? How do you...you're a world expert on X X and how do you, in some sense, throw it all away and go to Y? Do you decide one day, I'm interested in Y? Do you sort of shift over time a little bit? How do you do it? I've done it, I think, two or maybe three times depending on the account now.

And some of the transitions have gone better than others. I think my transition from systems to social data and computational social science, it was driven by a project that we did for Search at the time. Shuo Chen, another researcher here at Microsoft Research, built a web application that let you give very concrete feedback back to Windows

Live. You could drag and drop the results around and say this is what I wanted it to look like. And this made feedback much more actionable and helped really understand DSATs and where they were coming from, DSAT dissatisfactions. And I looked at that and I was like, I want to be able to move search results around and share with my friends. And I kind of poked a trowel and asked him if he would build this, and he said no. He said he's busy. So eventually I, because I knew something about JavaScript

applications, decided to just drop things and spend six months building out this application. So I built out this social search application where you could drag and drop search results around, share it with your friends, and we put it out. Actually, we got it deployed as an external service. We had maybe 10,000 people kick the tires. Within Microsoft or...? No, externally.

Okay. Yeah. There was a great headline that Google then fast followed with a similar feature and the headline was Google fast follows basically on Microsoft. Our PR folks were very excited about that. I say this all, it's all history now. But certainly it was fun at the time. But now I'm giving this demo, this talk about this prototype that we built and what we're learning about what's in people's way, what's friction, what do they like and not like, etc. And I'm standing up and giving this presentation, this demo, and someone says, hey, could you go back in the browser? On the bottom right corner, it says Mike did something on this search page. He edited some search results.

Could you click on that? I want to know what he did. I'm like, okay, yeah, sure. I click on it. And it's like, okay, that's great. That's really interesting. And this happened multiple times. Like in a formal presentation, for someone to

interrupt you and ask a personal question just out of their own curiosity, that's what got me really thinking deeply about the value of this social data. And why is it locked up in a very specific interface? What else could you do with this data if it's so engaging, so fascinating that people are willing to interrupt a speaker for some totally irrelevant, basically, question? And that's when I switched to really trying to figure out what to do with social data. I see. So it was this kind of really personal experience of people being so excited about that social interaction on the demos that you're giving. Exactly. They cared about their friends and what their friends did.

And that was super clear. So coming back, let's go there in a second. But coming back to the story that you told, you said you had 10,000 external users. Yeah. So I'm still, you know, also always trying to learn what we can do better because we sometimes have prototypes that are incredibly valuable. They're prototypes that have fans. They're prototypes that the fans even want to contribute. But then somehow we get stuck in the middle and they don't scale. They don't become a business. Or what happened with that? Also in retrospective. Should we have done something

different or did it live up to its potential? I think we learned something. I think that there were a couple of things we learned. One was that every extra click that people wanted to do took the number of interactions down by an order of magnitude. So starring something and bringing it to the top, that was very popular. Dragging and dropping, a little bit less so. Dragging and dropping from one search to a different search, so maybe I'll search for Johannes, find your home page, and then drag and drop it to people's publications list to

keep an eye on or something. That, almost never. And people were very wary about editing the page. Like what if I make a mistake? What if it's just me who wants this and I'm messing up search for the rest of the world? And it's like, no, no, it's just your friends. Just you and your friends who are going to see this. And so we learned a lot about people's mental models and like what stood in the way of interactions on the web. There were lots of challenges to doing this at scale. I mean, we needed, for example, a way of tracking users. We needed a way of very quickly, within 100 milliseconds,

getting information about a user's past edits to search pages into memory if we were going to do this for real on Windows Live. And we just didn't have infrastructure. I see. Those problems were hard in those days. Yeah. A prototype is fine. People will handle a little bit of latency if it's a research prototype.

But for everyday use, you need something more. And there was no push to try to land it somehow? There were big pushes, but the infrastructure was really... I see, it was really an infrastructure problem. Yeah. Yeah. Essentially because it sounds to me like, wow, there's an exciting research problem there.

Now you have the infrastructure to try to make all of these things really, really fast. It's always fascinating to see where things get stuck and how they proceed. Yeah, I think it'd be a lot easier to build that from the infrastructure point of view today. But of course, then there's lots of other questions. Like, is this really what, you know, the best thing to do? Like, I mentioned Google had this fast follow feature.

They also removed that afterwards as well. Yeah, hindsight is always 20-20. So you're now starting to move into social computing, right? And trying to understand more about social interaction between users. How did you end up in causality and then how did you make the switch to LLMs? Maybe even more about this, I mean I understand here this was sort of this personal story that you really saw that the audience was really asking you about what's happening here and that sort of motivated you. Was it always this personal drive or was it always others who pulled you and how did you make

these switches? I think the switch from systems into social, it was about trying to get closer to problems that really mattered to people. I really enjoy working on systems problems, but oftentimes they feel like they're in the back end. And so I wanted something where, even if I'm not the domain expert working on something, I can feel like I'm making a contribution to that problem. The transition with social data then into causality and LLMs, that was a bit smoother. So working with social data, trying to understand what it meant and what it said about the world in aggregate was a super fascinating problem.

So much information is embedded in the digital traces that people leave behind. But it was really difficult for people to come to solid conclusions. So there was one conference I went to where almost every presentation that day gave some fascinating insight. This is how people make friendships. This is how we're seeing signs of disease spread through real world interactions as they're in social data. Here's how people spend their time. And then people would close their conclusion slide every time was, and of course, correlation is not causation, so anything could actually be happening.

That is such a bummer. Like beautiful theory, great understanding, you spent so much time, I feel like I got some insight, and then you pull the rug out and say, but maybe not. And I'd heard about this work on, that there was work on causal analysis and that there were certain conditions and ways to get actual learned causal relationships from data. So that's the day I decided I'm going to go figure out what that is and how to apply it to social data for these types of questions. And I went out and the first work there was a collaboration with Munmun, the Chururi faculty at Georgia Tech looking at online traces

related to mental health and suicidal ideation and trying to understand what some of the factors were in a more solid and causal fashion. And so this really became, like this was interesting computational social science really ended up branching out into two areas. One, obviously I'm caring about what can we learn about the world. Part of this is, of course, thinking deeply about the implications of AI on society. What is it going to mean that we have

this data for all of these societal challenges? And then causality. So the AI and its implications on society is what led towards the work on the security of AI systems and now security of AI as it relates to large language models. And then causality was the other branch that split off from there. Both of them really stemming from this desire to see that we have a positive impact with AI. So you mentioned that you were sitting in these talks and people were talking about correlation and now you finally have this new tool which is causation. So what are some of the examples where with correlation you came out with answer A but

now causation gave you some better, some real deep insights? I haven't gone looking to refute studies. But there are many well-known studies in the past where people have made mistakes because they didn't account for the right confounding variables. Ronnie Kohavi has a great list of these on one of his websites. But a fun one is a study that came out in the late 90s on the influence of nightlights on myopia in children. So this was a big splash. I think it made it to Newsweek or 60 Minutes and stuff

that if you have night lights in the house, your kids are more likely to need glasses. And this was wrong. My parents told me all the time, don't read in bed with your flashlight because your eyes are gonna get bad. That's the story, basically, right? This was, yeah, the night lights are plugging in the wall.

But that's the idea. It's the same thing. Yeah, right. And so these people analyzed a bunch of data, and they found that there was a correlation. They said that this is a cause. And the problem was that they didn't account for the parents' myopia. Apparently, parents who had myopia were more likely to install nightlights. And then you have the genetic factor then actually causing the myopia.

Very simple. But people had to replicate this study to realize it was a mistake. Others were things like correlations, I think, around vitamin C have been reported repeatedly and then refuted in randomized control trials. But there's many of these. Medicine in particular has a long history of false correlations leading people astray. Do you have a story where here at Microsoft your work in causation had a really big impact? You know, the one, it's still ongoing, but one of the ones that I'm really excited about now, and thinking also from the broader societal impact lens, is a collaboration with

Ranveer Chandra and his group. So with a close collaborator at MSR India, Amit Sharma, we've developed a connection between representation learning and underlying causal representation of the data generating process that's driving something. So if you imagine like we want to learn a classifier on an object, on an image, and we want that classifier to generalize to other settings. There's lots of reasons why this can go wrong. You know, like a classic example is the question of is this picture showing you a camel or is it showing you a cow?

The classifier is much more likely to look at the background. And if it's green grass, it's probably a cow if it's sandy desert It's probably a camel, but then you fail if you look at a camel in the zoo or a cow on a beach, right? So how do you make sure that you're looking at the real features people have developed algorithms for these but no Algorithm actually is robust across all the different kinds of distribution shifts that people see in the real world. Some algorithms work on these kinds of distribution shifts, some algorithms work on those kinds of distribution shifts. And it was a bit of an interesting, I think, puzzle as to why. And so we realized that these distribution shifts,

if you look at them from a causal perspective, you can see that the algorithms are actually imposing different statistical independence constraints. And you can read those statistical independence constraints off of a causal graph. And the reason that some algorithms work well in some settings was that the underlying causal graph implied a different set of statistical independence constraints in that setting. And so that algorithm was the right one for that setting. If you have a different causal graph with different statistical independence constraints, the other algorithm was better. And so now you can see that no one algorithm is going to work well across all of them. So we built an adaptive algorithm that looks like the causal graph,

picks the right statistical dependencies, and applies them. And now what we're doing with this algorithm is we're applying it to satellite imagery to help us build a more generalizable, more robust model of carbon in farm fields. So we can remotely sense and predict what the carbon level is in a field. And so, the early results… And that's important for what? And so, this is important because soil is seen as a very promising method for sequestering carbon from a climate change perspective. And it's also, the more carbon there is, the higher your soil carbon,

usually the healthier the soil is as well. It's able to absorb more water, so less flooding. Your crops are more productive because of the microbial growth that's happening. And so people want to adopt policies and methods that increase the soil carbon in the fields for all of these reasons. But measuring soil carbon is really intensive.

You have to go sample it, take it off to a lab, and it's too expensive for people to do regularly. And so if we can develop remote sensing methods that are able to take a satellite image and really robustly predict what the real soil carbon measurement would be, that's really game-changing. That's something that will help us evaluate policies and whether they're working, help us evaluate what the right practices should be for a particular field. So I'm really excited about that. That's really exciting. You'd mentioned when we talked before that you'd benefited in your career from several good mentors.

How do you think about mentoring and what are the ways that you benefited from it and how do you live that now in your daily life as you're a mentor now to the next generation? Yeah, the way I look at all the people, and there's so many who have given me a hand and advice along the way, I often find I pick up on some attribute of my mentors, of a particular mentor, and find it something that I want to emulate. So recognizing everyone is complicated and no one is perfect, but there's so many ways that individuals get things right, and trying to understand what it is that they're doing right and how I can try and repeat that for

like you said the next Generation, I think it's really really important. It's like one story for example around 2008 while I was still working on large-scale internet services I Was going around the company to kind of get a sense of you know What's the current state of the reliability of our services and how we architect them and run them. And so I was talking to developers and architects and ops folks around the company.

And James Hamilton was a great mentor at that moment, helping me to connect, helping suggest questions that I might ask. And- So he was working on SQL Server reliability, right? At that point in time? Or on Windows reliability in general? He was already starting to move over into data center reliability.

I think right before he moved over to the research side of things, I think he was one of the heads of our enterprise email businesses. And then he came over to research to focus on, I think, data centers in general. And yeah, and he just donated so much of his time to... He was so generous with reviewing this large report that I was writing and just helping me out with insights. That struck me as like, he's a very busy person. He's doing all this stuff and he's spending,

I sent him an email with 15 pages and he responds with feedback within a couple of hours every morning. That was astonishing to me, especially in hindsight. And so, but that kind of generosity of time and trying to help direct people's work in a way that's going to be most impactful for what they want to achieve, that's something I try and emulate today.

So you've benefited from a lot of great mentors and you said you are also a mentor to others. Do you have any last piece of advice for any of our listeners? I think it's really important for people to find passion and joy in the work that they do. And at some point, do the work for the work's sake. I think this will drive you through the challenges that you'll inevitably face with any sort of project and give you the persistence that you need to really have the impact that you want to have.

Well thanks for that advice and thanks for being in What's Your Story Emre. Thanks very much, Jan. It's great to be here. To learn more about Emre or to see photos of Emre as a child in California, visit aka.ms.com.