Microsoft Research Podcast - 116 - Harvesting randomness, HAIbrid algorithms and safe AI with Dr. Siddhartha Sen

Episode Date: May 27, 2020

Dr. Siddhartha Sen is a Principal Researcher in MSR’s New York City lab, and his research interests are, if not impossible, at least impossible sounding: optimal decision making, universal data structures, and verifiably safe AI. Today, he tells us how he’s using reinforcement learning and HAIbrid algorithms to tap the best of both human and machine intelligence and develop AI that’s minimally disruptive, synergistic with human solutions, and safe.

I feel like we're a little too quick to apply AI, especially when it comes to deep neural networks, just because of how effective they are when we throw a lot of data and computation at them. So much so that we might even be overlooking what the best human baseline is. We might not even be comparing against the best human baseline. So a lot of what this agenda is trying to do is trying to push the human limit as far as it can, and then kind of integrate AI where it makes sense to do that. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen

Huizinga. Dr. Siddhartha Sen is a principal researcher in MSR's New York City lab, and his research interests are, if not impossible, at least impossible sounding. Optimal decision making, universal data structures, and verifiably safe AI. Today he tells us how he's using reinforcement learning and hybrid algorithms to tap the best of both human and machine intelligence and develop AI that's minimally disruptive, synergistic with human solutions, and safe. That and much more on this episode of the Microsoft Research Podcast. Sid Sen, welcome to the podcast.

Thank you. So fun to have you here today. You're a principal researcher at the MSR Lab in New York City, which is a lab with a really interesting history and mission. But since research is constantly moving and evolving, why don't you tell us what's new in the Big Apple from your perspective? What big questions are you asking these days? And what big problems are you trying to solve as a unit there? Our lab is really unique. We have four main disciplines represented in addition to what I represent, which is systems and machine learning and systems. We have a machine learning group, we have a computational social science group, we have an economics and computation

group, and we have a faith group, which is this a computational social science group. We have an economics and computation group. And we have a FATE group, which is this new group on fairness, accountability, transparency, and ethics. And I would say that if I had to sum up the kind of work we're doing, and a lot of us work on decision making, decision making with opportunities and under certain kinds of constraints. And I think a lot of the work we're talking about today is trying to understand where computation and, in particular, artificial intelligence can help us, and where it's not so strong, and where, you know, the more human approach is more appropriate, or is even better, and then trying to figure out a way for these things to work together. So that's

going to be a very core kind of theme in what we're going to talk about. The word hybrid is going to come up a lot in our conversation today, but in different ways with different spellings. So let's start with your own hybrid nature, Sid, and talk about what gets you up in the morning. Tell us about your personal research passions and how you bring what I would call hybrid vigor to the research gene pool. So I think I've always been a bit interdisciplinary in everything I've done. And ever since I've kind of grown up, I've always been able to do like a lot of different things reasonably well. And so you can imagine I always questioned myself in terms of how much depth I had in

anything, right? I even questioned myself throughout my PhD. When I went to do my PhD, I was co-advised in more systems-oriented work as well as theoretical work. I had two different advisors, and I essentially ended up doing two separate PhDs because I couldn't quite get those two disciplines to gel well together. It's very hard to do that, and a lot of students have asked me, oh, I want to do what you did. And I said, yes, that's a nice idea in principle, but it's not so easy to synergize two different areas. Since then, I think I've kind of figured out how to do

that. So how to kind of leverage and appreciate the breadth I have and the appreciation I have for different disciplines and make these things work together. I figured out how to synergize ideas from different disciplines to solve bigger problems. Problems like, you know, how do we use AI responsibly in our systems? How do we leverage what humans are good at? Or how do we keep things safe? I mean, I think these kinds of problems are not problems you can solve with expertise in one area in one field. And so a lot of what I figured out how to do is how to bring these different ideas together and work with different colleagues to bring these solutions out. Well, your current research mission is to, and I quote, optimize cloud infrastructure decisions with AI in a way that's, framework for your work, which we might call reinforcement learning in real life or RLIRL. Give us a rationale for this framework, and then we'll get technical on how you're going

about it. I think a lot of AI is going to be this kind of gradual integration into the background of our lives, into the systems that we use. So I think it's important to first see how much we can learn and how much we can do without disrupting existing systems, just by kind of observing what they're doing now, without having to change them so much. What can we do to learn as much as we can from what they're doing? And to kind of even reason about what would happen if we tried different things or if we used AI in different ways

without actually changing what they're doing. So that's why I want to take this minimally disruptive approach. Then at the same time, I think it's important to understand where AI should fit in. Because if you just come in barging in with an AI solution, that is not something that's so easily digestible by existing systems and processes that we have in place. And so understanding where the AI is good, where it should fit in, where the human solutions are good, and finding a way that they can complement each other is something that, to me, is critical to ensuring that you can gradually make this change. And then finally, when you do make the change, how do you maintain people's trust? And how do you keep things safe?

You know, I don't think AI and humans are ever going to be held to the same standards. Even if an AI system statistically affects or harms fewer people than a human operator would, they're still going to be held to a different kind of standard. And so I think it's ultra, ultra important, once we make this kind of change change to try to keep things safe. Well, let's talk in turn now about the three research agendas or roadmaps, we might call them, that you're following on the road to optimal decisions online. So the first one is something you call harvesting randomness. And this one takes on the minimally disruptive challenge. So why don't you start by giving us a level set on the problem, and then tell us all

about harvesting randomness and how the idea of counterfactual evaluation is gaining traction in reasoning about systems. This project kind of came out of a simple observation. One thing I observed was that whenever we want to try something new or figure out how something new is going to work, we often have to deploy it and try it out. This is what we call an A-B test. And people use it all over the place. They use it in medical trials. They use it online. At any given time, when you're using Bing or Google or any service, you're seeing something different than what I'm seeing because they're continuously trying out different things by deploying these live A-B tests. And the reason that works is because they randomize what people see.

And because they randomize what people see, they can evaluate how good different alternative solutions are and avoid any kind of biases or confounding issues that might come into place. And I realized that randomness is all over the place. We randomize stuff all the time. When we load balance requests, when we decide where to replicate or put data, when we decide what to do if too many people are trying to do the same thing at the same time, we make these randomized decisions all the time. And I thought, what if I could, you know, harvest all of that randomness to do all of these experiments without actually having to do the experiments? So that's

in essence what counterfactual evaluation or counterfactual reasoning is, what if I had done this? Can I answer that question without actually running a big A-B test and exposing the world to it? Because that's risky and that's costly. And so that was the theory behind this project is that I'm going to go around to all these systems that are already doing all this randomized stuff and using a slightly different set of statistical techniques, I can maybe answer questions about what would have happened if I tried this other policy. And that way, I could, you know, without disrupting or changing the system, reason about what would have happened if I had changed things. And I think it turns out that it's not as easy to do that as we thought it would be.

And the reason is, all these systems that are randomized are very complicated and stateful systems. We don't really have good enough practical techniques to understand them. But what we have found is that in the systems that surround us, there's a lot of information that's similar to the kind of information you would get from randomization that's already just sitting there. And I call that implicit feedback. And this is something that we're trying to leverage.

It's kind of a very simple concept in systems. But when you wait for something to happen, and we do that all the time in systems, you know, we're waiting for some event to happen. If you wait for five minutes, it turns out you find out what would have happened if you waited for four minutes or three minutes or two minutes.

Or if you provide a resource to someone, you give them 10 cores or 10 virtual machines, you might find out what would have happened if you gave them 9 or 8 or 7, because you gave them more than those numbers. And so because of this kind of implicit structure, we're able to look at a lot of the decisions that existing systems are making now, and we're able to say, oh, look, here are situations where they're actually making decisions where we could answer the counterfactual question of what would have happened if they made smaller decisions. Can we leverage that feedback to learn more from

that existing data and then, using that information, come up with better policies for making those decisions? So it's not the same as like the natural randomness that we started out initially envisioning we could harvest, but it's similar because it gives us the same as like the natural randomness that we started out initially envisioning we could harvest. But it's similar because it gives us the same kind of information. And now that we have a technique for harvesting this extra feedback, we can basically take this feedback, you know, use probabilities to weight things in the right way so that we get what I call an unbiased view of the world. And, you know, this way we can kind can find a better policy to put in there without actually changing the system that much at all. What results are you getting with this?

Are you using it or deploying it in places where it would matter? Yeah. One of the teams that we've worked with for a while is the team that deals with unhealthy machines in Azure. What happens when you can contact a machine? It's unresponsive. What do you do? How long do you wait for it? And how long should I wait for that machine to come back alive? Or should I just go ahead and reboot it, recycle it, and put it back into the pipeline of the process of what we call a fabric controller that takes machines and recycles them and gets them ready for production use? So that's a decision where we are able to kind of

harvest existing data that they have to find a better policy for choosing how long to wait. Another example is whenever you ask Azure for a certain number of virtual machines, you say, hey, give me 100 machines. They'll actually allocate more than 100 because they're not sure if the 100 will succeed and how long it will take for them to deliver those 100 to you. So they might say allocate 120 and they'll just give you the first 100 that get started up the fastest. And so here's an opportunity where we can harvest that information because of this over allocation, that implicit information that's in there to do something that's much more optimal and save on the amount of extra unnecessary work

that they have to do when they're spinning up these large clusters of virtual machines. Let's say you wait for 10 minutes and something happens, but you also know what happened at nine minutes, at eight minutes, at seven minutes, right? So this is how you're explaining it. Is it always going to happen then at 10 minutes? I know that I'll wait for nine. If I just wait one more minute, I'll be fine instead of having to go to the expense of rebooting and so on. Yeah, I think that's a great point. We might not know. And one thing that's unique about the feedback we're getting is it kind of depends on what happens in the real world. I might wait for five minutes and, you know, nothing happened. In that situation, I only

learned what would have happened if I waited less than that amount of time. But I don't learn anything about what would have happened if I waited longer than that amount of time. Whereas sometimes if I wait for five minutes, the machine comes up in two minutes. And now I know everything. I know all the information that would have happened if I had waited for one minute or two minutes or 10 minutes or nine minutes or whatever it is. So there's kind of an asymmetry in the kind of feedback you get back. And this is a place where existing counterfactual techniques fall short. They're not able to deal with this kind of the different amount of feedback you get, which depends on the outcome of what actually plays out in the real world. And so this is where we had to do most of our innovation on the, you know,

the statistical and reinforcement learning side to develop a kind of threshold to think that worked for this scenario. And, you know, I should say that this work is something that we see come up in many different system decisions across our cloud infrastructure. Anytime we're allocating any resource, whether that resource is time, machines, memory, cores, anytime we're doing that, there's this implicit information that we should be harvesting. It's like a crop that we're not harvesting, and there's an opportunity cost to not doing that. And so part of the goal of this work is to show people that there is that opportunity there, but also to kind of show them how we can leverage this in a principled way so that our system designers know how they can design their systems

in a way that allows them to continuously evolve them and optimize them. And a lot of what this harvesting randomness work is showing you how to do is, how do I, in a statistically correct way, collect data and improve my system, collect more data and improve my system and keep doing this? Because the way we're doing it now is not actually, you know, fully correct. No one can ever see the grin that I have on my face. Also on a podcast, people can't see how things are spelled. And in the case of your second research agenda, it actually matters. You're developing what you call hybrid algorithms.

Here we come on the hybrid. And that's spelled H-A-I-B-R-I-D. So what are hybrid algorithms, Sid, and how do they move the ball forward in the world of data structure research? So I always get a snicker or two when people see the spelling of this. And I don't know if that's because it's lame or if it's because they like it, but it's spelled H-A-I because the H stands for human and the AI stands for AI, for artificial intelligence. And so the idea of these algorithms is to basically find the right synergy between human and AI solutions, to find the right balance, the right way to use AI. And I do strongly believe in that because I feel like we're a little too quick to apply AI, especially when it comes to deep neural networks,

just because of how effective they are when we throw a lot of data and computation at them. So much so that we might even be overlooking what the best human baseline is. We might not even be comparing against the best human baseline. So a lot of what this agenda is trying to do is trying to push the human limit as far as it can, and then kind of integrate AI where it makes sense to do that. And we're starting very classical. We're starting as simple as it gets. We're looking at data structures. You know, there was a lot of buzz a couple years ago when folks at Google and MIT were able to replace classical data structures with very simple neural networks. A lot of what data structures are used for is to organize and store data. And so oftentimes you use them by

giving them some kind of key and telling them, hey, can you look up this piece of information, find it for me and give it back to me. And they kind of observed that the key and its position in the data look like training data for a machine learning model. So why don't we just shove it into a machine learning model and make it answer the question for us. And, you know, they show that if the data set is fixed, it doesn't change, then you can train this kind of layer of neural network to answer that question for us. And you know they show that if the data set is fixed, it doesn't change, then you can train this kind of layer of neural network to answer that question for you and they can do it way faster using way less memory than let's say a classical structure like a B-tree. The problem with a lot of that was the human

baselines they were comparing against were not nearly the best baselines you would use. You wouldn't use this big fat B-tree to find something in a sorted data set that doesn't change. You would use something very simple, like maybe binary search or some multi-way search. That's something that is way more efficient and doesn't require much space at all or much time. And so when I saw this work, I thought about every human baseline they were comparing against, and I felt bad for those human baselines. I said, oh, that's not fair. These human baselines weren't designed for that. They were designed for something much more. They were designed for these dynamic changing workloads

and data sets. And that's why they're all bloated and inefficient because they're keeping space so that they can accommodate future stuff, right? They're not designed for this particular thing you're looking at. And so we set out to kind of design what we thought was the best human data structure for this problem. We call it a little bit presumptuously, we call it a universal index structure. And the idea of this universal

index structure is that it's going to basically give us the best performance all the time. More concretely, what that means is that I'm going to look at what the workload looks like now, and then I'm going to try to metamorphose myself into the data structure that's perfect for that workload. And then I'm going to give that to you. So it sounds like an impossible thing, but you know, Gretchen, I think a lot of the work I do is impossible sounding at a high level. And I kind of like that because one of the reasons I've been able to appreciate kind of my hybrid view of things is that I come up with a crazier idea or approach than most people would. So something like a universal data structure to

me sounds like an impossibility. But then what you can do is you can say, okay, suppose I had access to some kind of oracle, some supreme being that gave me for free the things that I wanted, then maybe I could solve this problem. So that's kind of how I plan my research. I say, okay, if someone told me what the current workload was, and if someone told me for this workload, what is the best structure to use? And if someone told me, how do I go from what I have now to this best structure, And I got all these answers for free. Well, then I would solve my problem and I would just use it. I'd have a universal data structure right there. So now we go about trying to solve these oracles, each of which is hard. But you

know, when it's a hard oracle, you can break that down into sub oracles and make a roadmap for solving that sub oracle, right? And then the problems slowly get easier and easier until there are things that are attractable enough for you to actually solve. And then you can put the pieces back together, and you have a solution to the higher level problem. All right. So how are you doing that? Right. How are we doing that? So what we're doing now is we're taking a black box approach. We're taking existing data structures without changing any of them.

All the classical stuff that people use, hash tables, trees, skip lists, Redix trees, and we're saying, how can I move efficiently from one of those structures to another? So what we've developed is an efficient transition mechanism that allows us to take data

from one of these structures and gradually move it to another structure if we feel like that other structure is going to be better at handling the current workload. So we use some ideas from machine learning to profile these data structures to try to understand what regimes are good at. Okay so we are using ML a little bit here and there when we think it makes sense. So we're trying to understand the space of when these data structures are good. Once we understand that space, then we're trying to come up with an efficient way to move from one of these data structures to the other.

And so the big innovation of this work is in coming up with that transition mechanism. It's a way of sorting the data that you're working with and gradually moving the data from one structure to another in a way that you can do piecemeal where you don't lose your work, where you don't affect the correctness of the queries that you're trying to respond to. People are going to be continuously asking your system to answer questions while you're doing all of this and in a way that it makes it worthwhile. We try to make sure that this transition mechanism is efficient enough that we're actually getting a benefit from it. And if we find that we're just flopping around all over the place, that we keep moving from data structure A to B, back to A, back to C, well, we check for that.

And if that's happening, we kind of back off and say, no, let's not do that. This is not worth it right now. And so this transition mechanism is kind of where most of the innovation has happened. It's a simple transition mechanism because it just intelligently uses the existing APIs provided by these data structures to do this kind of transitioning between them. And later on in the future, I hope that we can open up the black box a little bit. And by understanding these data structures from the inside, maybe we can actually do something even more intelligent and more efficient. Let's talk about a specific example of that while we're on the topic of hybrid algorithms. One example of some very current work you're doing involves using the game of chess as a model system for bridging the gap between what you call superhuman AI and

human behavior. So this is really interesting to me on several levels. Tell us about it. Yeah. so whereas the universal index structure was trying to say, let's take a pause and not use ML just yet. Let's see how far I can push the human design and find a way to just, you know, switch between them and use all of them in some fluid way so that I can do as good a job as I can. And then the idea would be, how can we then take that strong human baseline and improve on that with AI? And I think the verdict is still out there.

It's still unclear to me which one is a stronger force there and how they'll work together. In chess, it's a little different because we actually know today that chess engines and AI are far stronger than any kind of human player can be. So chess is an example of a task where AI has surpassed human play, and yet we still continue to play it because it's fun, right? A lot of times you'll find that when AI does better at humans, it'll just take over.

It just takes over that job from who was doing it before, whether it was a human or a human-designed algorithm or heuristic. But chess is a game where that hasn't really happened because we still play the game. We enjoy it so much. And so that's why I think it's an interesting case study. The problem with chess is that the way humans play and the way engine play are very, very different. So they've kind of diverged. And it's not really fun to play the engine, you just lose. You just lose all the time. So now interestingly, people are using the engines now to train themselves, which is kind of an interesting situation because it's almost like this subtle overtaking of AI. It's like our neural networks, our brains are now being exposed to answers that the engine is giving us without explanation. The engine just tells you, this is the best move here, right? It's figured it

out. It's explored the game tree. It knows AI to tell you this is the best move. And you're like, oh, okay. And then you try to reason, why is that the best move? Oh, I think I kind of get it. Okay. So a lot of the top chess players today, they spend a lot of time using the engine to train their own neural nets inside their brains.

So I actually think that our brains, our neural nets, they're a product of everything we put in there. And right now, one of the things that they're a product of everything we put in there. And right now, one of the things that they're getting in as input are answers from the engine. So that's the subtle way that AI is seeping into your brain and into your life. But going back to what we did, I mean, we basically realized that there's this gap between how the AI engines play and how humans play. And when there's this gap, maybe the AI engine should be helping us get better, right? But right now, we don't have that

kind of engine. These engines just tell us what the best move is. They don't have what I would call a teaching ability. And so what we're trying to do in this project is trying to bridge that gap. We're trying to come up with an engine that can teach humans at different levels, that can understand how a weak player plays, understand how a strong player plays, and try to suggest things that they can do at different levels. So we looked at all the existing chess engines out there, ones that are not based on AI, and ones that are based on AI. And we found that these engines, if you try to use them to predict what humans will do at different levels, they either predict kind of uniformly well across all of the humans, or their predictions get better as the humans get better, which means

that none of them really are understanding what different humans at different levels play like. So what we did was we kind of took this neural network framework that's underneath it, and we repurposed it to try to predict what humans would do. And we were able to develop engines that are good at predicting human moves at every different level of play. So this is, I think, a small building block in getting at this idea of a real AI teacher, someone who can sit with you, understand where you're at, and then from there, maybe suggest things that you might try that are appropriate for your level.

Let's talk about your third research agenda, and it's all about safeguards. The metaphors abound. Training wheels, bumpers on a bowling alley, apparent. Explain the problem this research addresses in light of the reinforcement learning literature to date. And then tell us what you're doing in this space to ensure that the RL policies that MyAI proposes are actually safe and reliable. Yeah, so safety is something that has come up in all of the things we've talked about. It comes up repeatedly in everything we've done in the harvesting randomness agenda. You know, after harvesting information and coming up with a better policy, every time we try to deploy something with a product team or group, they always have some safeguard in place, some guardrail in place.

If this happens, page me. Or if this happens, shut this off or things like that. It comes up even in the universal data structure we've talked about. What happens when I keep flip-flopping around these different data structures and I'm wasting all this time and my performance is going down? Well, maybe I need to put a brake on things and the chess work as well it happens because one of the things that the chess teacher is trying to do is prevent the human from walking into a trap or walking down

a path of moves that they can't handle because they're not strong enough yet as a player to handle it and so you can reason about safe moves versus unsafe moves and so because I just saw this cropping up all over the place, I realized it's time to take a step back and reason about this and formalize this. So this is something that systems people do, which I think they're good at doing, is they see people doing all kinds of ad hoc stuff and they say, oh, this is an opportunity here to come up with a new abstraction. And so safeguards are that new abstraction. I think the sign of a good abstraction is it's something that like everyone is already

using it, but they don't know it. They don't give it a name. And so if everyone's already doing something like this in their own ad hoc ways, what we're trying to do is extract this component. We're calling it a safeguard. And we're trying to say, what is its role? So to me, a safeguard is something that kind of sits alongside any artificially intelligent system that you have,

treats it like a black box, and it protects it from making bad decisions by occasionally overriding its decisions when it thinks that they're about to violate some kind of safety specification. So what is a safety specification? Well, it can be anything. It can be a performance guarantee that you want. It can be a fairness guarantee. Maybe it can even be a privacy guarantee. And we haven't explored all of the realms. Right now, we've been focusing on performance guarantees. And I think the point is that if you have this component sitting outside the AI system, it doesn't necessarily need to understand what that AI system is doing. It just needs to observe what its decisions are. And as long as it has an idea, like the safety specification, it can say, oh, I think what you're going to do now is violate the

safety specs, so I'm going to override it. But if I see in the future that you're doing a better job or you're within the safety realm, then I'm going to back off and I'm not going to change your decisions. So the safeguard is like this living, breathing component that itself might use artificial intelligence. And it can use AI to adapt. It can say, oh, I see that this system is doing a good job. Okay, I'm going to back off. I don't need to provide that much of a safety buffer. But the moment I see that the system is not doing a good job, then maybe I need to put the clamp down and the safeguard will say, okay, I need a big safety buffer around you. And so that's why I like to think of this analogy of parenting,

because I think that that's what we do with kids a lot. In the beginning, you childproof your house, you put training wheels on bikes, and you constrain and restrict a lot of what they can do. And then slowly you observe that now they're getting bigger, they're falling down less, they're able to handle corners and edges more, and you start removing the bumpers. You might start raising the training wheels and things like that. But maybe you go to a new house or a new environment or you're on a vacation, and then maybe you need to clamp down a little bit more, right? Because you realize that, oh, they're in a new environment now and they might start hurting themselves again. And so that's kind of the inspiration behind this idea of a safeguard that adapts to the actual system. Okay. So I want to drill in again on the technical side of this because I get the metaphor and how are you doing that technically? So what we're doing now is we're taking some example systems from the previous work we talked about and we're hand coding safeguards based on the safety specs of those systems.

Someone might say, well, how do you even know what a safety spec should be? What we found is that people usually know what the safety spec is. Most of the teams we work with, they usually know what things they check for, what things they monitor that trigger actions to happen, like guardrails that they have in place. So most of the teams we've talked to, they check for like six different things. And if those six different things exceed these thresholds, they do some big action. So we can usually derive the safety spec for those systems by talking to the product team and looking at what they do right now. Once we get that, we're kind of hand coding and designing a safeguard. And we're using tools from program verification to prove that the safeguard that we code up

satisfies what we call an inductive safety invariant, which means that it satisfies some safety property. And every action that that safeguard takes will always continue to be safe. So as long as you're in the safe zone, no matter what you do, as long as the safeguard is active and in place, you will stay in the safe zone. And so we can use tools from program verification to write these safeguards, prove that they satisfy the safety invariant. Then what we do is we add a buffer to that safeguard. Think of the analogy of like adding more padding to the bumpers on the corners of the tables or putting the training wheels a little lower.

So I can use the same kind of verification approach to verify the safety of the safeguard plus some kind of buffer. And then what I'll do is I'll use AI to adapt that buffer depending on how good the system is doing. So now I've guaranteed to you that everything is going to be safe, even if you have this kind of buffer. Okay, great. And now I can shrink and grow this buffer. You're doing well, the buffer will be really small, which is good for a lot of systems. Because if you keep that small buffer, it allows systems to be aggressive. And when you're aggressive, you can optimize better, you can use more resources, you can push the systems to their limits a little more,

which is good for you. But if you're in a situation where you're not doing well, or there's some uncertainty, or the environment has changed, then we kind of increase that safety buffer. And the whole time, you're still guaranteed that everything is going to be correct. So that's kind of what we have now. What I really want to do, again, because no project is worth it, if it's not impossible sounding, is to automatically synthesize these safeguards. Like we're coding them by hand now. I want to use fancy program synthesis magic. I was just going to ask you if that was next. Yeah, yeah. So that's, that would be ideal. Like I don't want to have to sit and do these things by hand. I want to get a safety spec and some structured information about the application. And I want to automatically

synthesize the safeguard and then show you that it is and prove that it's correct. So we're working with an amazing colleague in the program synthesis and program verification world, who's actually the director of our MSR India lab, Sriram Rajamani. I had him on the podcast. He's amazing. Not only is he amazing, he's maybe the nicest person i know like in the world he's working with us on this and it's a lot of fun i i love working on projects where i love working with the collaborators in fact i think i find that these days i i tend to pick projects more based on the collaborators and the topic sometimes but this is one where it's both the topic and the

collaborators are are just a complete thrill and pleasure to work with. And so we're hoping that combining our techniques, I do systems and AI stuff and reinforcement learning. He understands program synthesis and verification. We have someone who understands causal inference and statistics. And so with these three disciplines, we're hoping that we can come up with a way to automatically synthesize these safeguards so that any system that is using AI and that has an idea of what safety means for them can leverage the safeguard and come up with one. Well, speaking of collaboration, that's a big deal at Microsoft Research. And you're doing some really interesting work with a couple of universities.

And I don't want to let you go before we talk at least briefly about two projects, one with NYU related to what systems folks call provenance, and another with Harvard trying to figure out how to delete private data from storage systems in order to comply with GDPR. Both fall under this idea of responsible AI, which is a huge deal at Microsoft right now. So without getting super granular, though, because we have a couple more questions I want to cover with you, give us a Snapchat story version of these two projects as well as how they came about real quick. Okay. Yeah. So with NYU, we're working with two professors there, Professor Jin-Young Lee and Professor Orijit Panda, and two amazing students on what we're calling ML provenance. And so the idea is, can we find and explain the reasons behind a particular decision made by an ML model? And so there's been a lot of work on trying to do this. And I think what we've done that's

different is we formulated the problem in a different way. We've said, what if there are different sources of data that go into training this model? So if that was the case, and you have all these different sources, you can look at a particular decision made by the machine learning model and say, okay, what were the sources that contributed to that decision? So it turns out you can formulate a new machine learning problem that uses those sources as the features and as its label, the decision. And we can use statistical techniques to try to narrow down and zoom in on which of the sources actually caused that decision. So this is going to be useful for doing things like detecting poisoning attacks, which is a very common security problem people

look at in machine learning models. With our technique, you can kind of find the data source that caused you to do that. So the idea there is we can find the data source that caused that decision, or maybe even find the actual data point, in some cases, that caused that decision. And we can do this without training too many models. And the coolest thing about it, I think, is, again, this idea of harvesting existing work, is that I think we found a way that we can do all of this by just leveraging the existing training work that's already being done to train these models. So that's the NYU work. The work that we're doing with Harvard is with Professor James Mickens, and he has a student we're working with. And there, what we're trying

to understand is this question of, can we really delete a user's private data from a system? Let's take a simple classical storage system as an example. Suppose all the person does, it goes in and says, insert this data item into your storage system. Like, what happens? And what we found, it's kind of fascinating to see that just one thing that you insert into a storage system, all the different things it touches. Like, you have all this transient state created, you affect these global parameters and state, and then you put the actual data into a data structure.

That's the easy part, right? But there's all these other kind of side effects that happen. Maybe some of those statistics that are used are used to train some kind of machine learning model. I mean, there's all kinds of things that can happen. So we're trying to basically track all that. And in systems, we have this notion of taint tracking, which is like, imagine you put a color on that data and you see where the color spreads. So we have some techniques for doing that in systems already. But what we're trying to understand is how do I measure like how

much I care about all the things I've touched, right? Like these taint tracking things, if you touch something, they're like, oh, it's tainted. But, you know, if I add one number of mine, that's a private number to like a thousand other numbers, do I really care about what I've done there? What we're asking here in this work is also like, how do I reason about how much of my privacy is being leaked, Which has connections to differential privacy, actually. How do I reason about how sensitive my input is to whatever state it's affected? And once I figure that out and I decide, okay, I don't care about this stuff because it really hasn't affected things enough, but I care about this stuff because I've affected it

in a meaningful way, then how do I go about deleting that data? It's not so clear how we can, in a generic way, allow the entire system to fully and completely delete what it needs to delete when it comes to that user's private data. So another work in progress. Well, it's about now that I always ask what could possibly go wrong, Sid. And I do this because I want to know that you have an eye to the risk as well as the rewards inherent in research. So while you frame a lot of your work in terms of AI playing nicely with humans

or collaborating with humans or helping humans, is there anything you can see down the line or maybe are afraid you can't see down the line, so that's even worse, that gives you cause for concern or keeps you up at night? And if worse, that gives you cause for concern or keeps you up at night? And if so, what are you doing about it? I think that AI solutions will always be held to a different standard than human operators or human solutions. I do worry about what happens

when even a system that we deploy that has, let's say, this kind of safeguard in place, if something goes wrong, how do you react to that? What kind of policies do you put in place to determine what to do in those situations? What happens when one of these AI systems actually hurts a person? And this has happened in the past, right? But when it does happen, it's interpreted very differently than if the cause for that accident or incident was driven by a human. In a lot of the applications we've looked at, if the safeguard gets violated a little bit here and there, it's actually okay. And we actually can leverage that. And in the work we do, we do indeed leverage that. But what if it's not okay, right? What if it's never okay for even the slightest violation to happen? How do we in

those situations still learn, still do what we need to do without the risk. So that's something that does worry me because it makes me realize that there are some things where you just don't want to replace a human. But I do believe that this kind of hybrid approach we talked about, I think it will ensure that both sides have an important role to play. I think what's happening is that AI is showing us that we're not playing the right role. And so what we need to do is just kind of adjust the role. I'm not worried about AI replacing creativity and elegance. I mean, a lot of people are worried about that.

I think there's just too much elegance and beauty in the kinds of solutions humans come up with. And it's one of the reasons why I spend a lot of time learning from and using human solutions in all of the AI work that I do. I happen to know that you didn't begin your career doing hybrid systems reinforcement learning research in New York City for MSR. So what's your story, Sid? How did you get where you are today doing cutting-edge research in the city that never sleeps? That's a good question. I grew up in the Philippines, but I had a lot of exposure to the U.S. and other countries. I grew up in an international environment, and I went to college at MIT in Cambridge in Massachusetts. And I remember seeing a student solve a problem that we had been assigned in a way that he wasn't told how to solve it. And this kind of blew my mind, which is a little sad if you think about it. I was like, why did you do it that way? I mean, they told us to do this way, and we would have figured it out if you did it that way. Why did you do it that way? But that resulted in a new idea that led to a paper that he submitted. And it was, I think, at that point that I realized my professors are not just giving

me problem sets, that the stuff that they're teaching me is stuff they invented, and that we have this ability to kind of innovate new ways of doing things, even new ways of doing existing things. And I had not really realized that or appreciated that until that point. All this time I'd been like, you know, thinking of my professors as just people who gave me problem sets. But now when I go back and look at them, I realized that I was taught by these amazing superstars that I kind of took for granted.

And so after I left college, I went to Microsoft to work in Windows Server actually for three years. I worked in a product group as a developer. But I always ended up going back to thinking about the algorithms and the ideas behind what we were doing. And I was fortunate enough to have a good boss that let me work on researchy type things. So I would actually visit MSR and talk to MSR people as an engineer from a product team and ask them questions.

And I always had this respect for them. I always put them on a pedestal in my mind. I think this kind of inspired a more kind of creative pursuit. And so that's why I went back to grad school. And I think the reason why I ended up at MSR after doing my PhD is because MSR was the one place where they really appreciated my interdisciplinary and slightly weird background, right? The fact that I did some system stuff, the fact that I did some theory stuff, the fact that I worked in the product group for three years, like they actually appreciated a lot of those things. And I felt like this is a place where all of that stuff will be viewed as a strength rather than, oh, you're

kind of good at these things, but are you really good at one of these things more than the others? You know, that kind of thing, which is something that does come up as an issue in academia. And even in our labs, we struggle with it, at hiring people who are at the boundary of disciplines, who are cross-cutting, because it's not so easy to evaluate them. How do you compare such a person to another candidate who is an expert in one area, right? How do we put value on this kind of interdisciplinary style versus this deeper, more siloed style of research? So it's not an easy question. It's been a recurring theme in my life, I would say. Sid, what's one interesting thing that we don't know about you that has maybe impacted your life

or career? And even if it didn't impact anything in your life or career, maybe it's just something interesting about you that people might not suspect. So I was not a very typical Indian kid growing up. I grew up in the Philippines and partly in India, and I spent a lot of time doing like hip hop and break dancing and things that drove my parents a little crazy, I would say. They would let me do it as long as my grades didn't suffer. When I came to MIT, I saw these people doing a style of dance where they didn't have to prepare anything. It was just like, do you know how square dancing works? Yeah. Right. You just follow a color, it calls all the moves. And, you know, I joined that dance group and I eventually became the president of that group. And that's actually where I met my wife, who was a PhD

student at the time. In a square dancing group? No, it was in a salsa version of that kind of work. So it's called Casino Rueda. It's like square dancing, but it's a collar-based dance where the collar calls all the moves. So I would call these moves with hand signals and all that, and everyone does it and makes these beautiful formations and patterns, but it's all salsa. So that was a very important kind of part of my time at MIT. And I do appreciate the university a lot because it gave me the ability to have a balanced life there. So that's something kind of non-work related that people don't usually expect. I guess something work

related that people may not know so much about me is that I don't really like technology. I'm not a very good user or adopter of technology. I'm always kind of behind. And in fact, I think I use it when it becomes embarrassing that I don't know about it. And this has happened repeatedly in the past. So I do a lot of things because I need to for my work. But I think this actually works to my advantage. I don't like technology. I don't like writing thousands and thousands of lines of code. I like thinking more about how to do things minimally. Finding the simplest and most elegant route is really important to me. One thing that I have a big passion for is teaching. If I'm honest with myself, I probably should be in a university because I love working with

students and mentoring them. I actually feel that might be my greatest strength compared to the other things that I do. And one of the things you need to be able to do well if you want to teach a student is explain something in the kind of simplest way. And of course, it's a lot easier to explain something in a simple way if that thing is simple to begin with, right? Not only is it easier to explain, it's easier to code up. It's usually easier to program. And that means that it's less likely to have bugs and other issues in it. So there's a lot of value to simplicity. And it's something that I think bugs and other issues in it. So there's a lot of value to simplicity, and it's something that I think about all the time.

It really has permeated everything that I do. Well, it's time to predict the future, or at least dream about it. So if you're wildly successful, what will you be known for at the end of your career? In other words, how do you hope your research will have made an impact, and what will we be able to do at that point that we wouldn't have been able to do before? Wow, that's a good one. That's a tough one to answer.

Let me tell you two things that I think would make me happy if at the end of my career these things happened. If our world is run by AI systems that operate in a harmonious way with humans, whose safety we are so assured of that we take it for granted, if a part of that can be attributed to the work that I've done, that would make me happy. So that is one thing. The other thing, though, which may even be more important to me is that if I'm remembered as someone who was, you know, a good teacher or a good mentor. So if, for example, if my ideas are being taught to undergraduates or even better, if they're being taught to high school students, I think the further down you go,

the more fundamental and the more basic, the more simple the stuff is. To me, that's, you know, a sign of greater impact on education. So I've had a bit of a taste of that. When I was in my PhD, I did some work on very classical data structures, and we came up with some new ones. And those are being taught to undergraduates now, and they're in textbooks now. And that kind of thing makes me really happy. You know, I hope that by the end of my career, that big chunks of my work, or even parts of it, will be taught to undergraduates, or or even better taught to high school students. To me, that would make me super happy.

Sid Sen, thank you for joining us today on the podcast. It's been terrific. Thank you, Gretchen. I really appreciate it. To learn more about Dr. Siddhartha Sen and how researchers are working to optimize decision making in real world settings, visit visit Microsoft.com slash research.