Microsoft Research Podcast - 022 - Machine Learning and the Incredible Flying Robot with Dr. Ashish Kapoor

Episode Date: May 2, 2018

Autonomous flying agents – or flying robots – may seem like the stuff of sci-fi to the average person, but to Dr. Ashish Kapoor, Principal Researcher and Research Manager of the Aerial Informatics and Robotics Group at Microsoft Research, they’re much closer to science than to fiction. And, having built – and flight tested – his own airplane, complete with state-of-the-art avionics designed to run AI and ML algorithms, he has the street cred – or should we say flight cred – to prove it. Today, Dr. Kapoor talks about how cutting-edge machine learning techniques are empowering a new generation of autonomous vehicles, and tells us all about AirSim, an innovative platform that’s helping bridge the simulator-to-reality gap, paving the way for safer, more robust real-world AI systems of all kinds

So the way I see my airplane, the technologies are already there. If I could just solve landing and takeoff, that would be a flying car that I would bet my life on. You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huizenga. Autonomous flying agents, or flying robots, may seem like the stuff of sci-fi to the average person, but to Dr. Rishish Kapoor, principal researcher and research manager of the Aerial Informatics and Robotics Group at Microsoft Research, they're much closer to science than to fiction. And having built and flight

tested his own airplane, complete with state-of-the-art avionics designed to run AI and ML algorithms, he has the street cred, or should we say flight cred, to prove it. Today, Dr. Kapoor talks about how cutting-edge machine learning techniques are empowering a new generation of autonomous vehicles, and tells us all about AirSim, an innovative platform that's helping bridge the simulator-to-reality gap, paving the way for safer, more robust, real-world AI systems of all kinds.

That and much more on this episode of the Microsoft Research Podcast. Ashish Kapoor, welcome to the podcast. Yeah, thank you. I'm excited to be here. Great to have you. You're a principal researcher and the research manager of the Aerial Informatics and Robotics group at MSR. Give us a picture in broad brushstrokes, generally, of what your group does. Our group focuses on robotic agents, and more specifically, flying robotic agents that can positively influence the society. So we are focusing on how the current advances in robotics, machine learning, and AI can help us enable that.

So talk about this phrase, aerial informatics and robotics, also known by the acronym AIR, which is delightful. It's a group and is also a platform. Yes, definitely. Tell us about that. So when we started a few years ago trying to build these agents, this is an incredibly difficult area to work on. And one of the reasons is the amount of effort that you need to make sure that the robotic systems work well in the environment is huge. You are not just dealing with software,

you're dealing with hardware, you're dealing with hardware and software integration. You're dealing with a field study that you need to make sure that all your robotic components, all your software components, all your hardware components, all the sensor components work well. You need to make sure that in the presence of external environments, such as exogenous variables, such as rain, wind, you know, maybe other human users nearby, animals, birds. So the amount of progress was very tiny compared to the amount of time that we were spending on it. And we quickly realized that what we need to do is we need to come up with a simulation platform that would let us recreate these elements in the lab setting so that our end-to-end cycle between development and testing gets reduced rapidly. So consequently, we built this platform, a simulation platform, and we released it under the name of Aerial Informatics and Robotics Platform.

So that's the platform. That's the platform. And what it is, it's a simulation for AI to build autonomous agents and train machine learning systems. It's an AI simulation. That's how I think about it. So group is aerial informatics and robotics platform. And it is really about flying robots, you know, and they are powered by machine learning. They are powered by AI techniques. They are powered by intelligent human in the loop methods. So we constantly think about how easy would it be for a pilot to actually fly these drones and carry out missions, which might be risky, which might be ambitious.

But at the core of this group lies this philosophy behind simulation. We think that simulation is one of the foundational pillars that would enable us to build these kinds of technologies. A simulation like this allows us to rapidly build prototype and test out our technologies, but it also allows us to make sure that we make the mistakes in the safety of our labs. It allows us to make sure that our softwares, the bugs have been ironed out before we go and fly the real thing out in the world. When I think about flying robots, it usually has somebody behind the controls, right? And so you talk about humans in the loop and that they're always going to be there, but yet it seems like you're moving towards a more and more autonomous flying robot.

How is that different from autopilot today? I would claim that, you know, there have already been flying robots for a while. And your commercial airliners, it's a giant flying robot. In fact, most of the flight is actually carried by an autopilot these days. And we are comfortable with the technology. And of course, as human in the loop, they are pilots manning the cockpit, and they are making sure that things are correctly set up, for instance. So what has actually changed is in the recent past, all these technologies are now available for drones as well. So it's exactly the same technology. It's pretty much the path following. But when we think in the context of drones and gliders and

other objects that everyday people can just go and fly, you know, the stakes becomes much higher because now you're closer to ground. There are other people in the vicinity. There are buildings, there are trees. And the potential for anyone to just pick it up off the shelf and go and fly, I mean, the risks are huge. So consequently, we need to start thinking a little bit more about safety, about how these AI technologies can positively

influence our world, but also be safe while doing so. Tell us about the flight of your AI-controlled autonomous infinite soaring machine that you did last summer. I watched the video. It was really cool. What was that about and why is it important that we have these autonomous, infinite soaring machines, gliders? That project, this was more of an AI aspirational project. And if you think about some of the recent breakthroughs in AI, they have things like reinforcement learning, deep reinforcement learning, or deep machine learning algorithms. But much of this breakthrough has happened in the software space.

When an agent needs to make a next move, they don't have to worry about the exogenous variables, etc. And the second thing is because of the inherent software nature of these problems, it's possible to potentially generate near infinite amount of data. And that's what many of these techniques are banking on. So we started thinking about what would it take us to actually think about similar problems, but in the domain of real-world AI. And this autonomous glider is basically, if you think about it, it's kind of a game,

but you are playing this game in a real world. So the whole goal in the autonomous glider project was to design an autonomous flying agent that can travel large distances or that can stay in the air for a long amount of time. And the way it does it, it has some information about the terrain, some information about the weather. It has some statistical model of how the rising air bubble work. And it exploits all this information in order to make a decision whether it should hang out at a place and wait for a rising air to carry it up and gain altitude, or it should go to another place. And while in doing so, it needs to trade off between how high it's flying and how far and

what's the probability of finding such a bubble of rising air. And by doing such kind of trade-offs, it's possible to keep the glider aloft for a long amount of times. However, this is now a real-world problem. We don't have the luxury of obtaining near-infinite data. And in addition, we have to deal with exogenous variables, possibly variables that we have never accounted for in our algorithms. So if we could solve this problem, this will definitely be a big step in the development of AI and ML techniques.

So the glider doesn't have an engine. You send it off, and then what it's doing and learning is how to stay aloft for a long period of time without any external force other than nature. The nature, exactly. So the glider doesn't have any external propulsion system. All it relies on is these bubbles of rising air that it hopes to piggyback upon to get an altitude. And then really the decision that the glider has to make is where to go in order to attain the maximum lift, but also such that it doesn't veer too far away from its core mission. Yeah. So it's a trade-off between exploration and exploitation that we talk about in machine learning. Well, let's talk about the role of simulation in AI for a bit.

You refer to this as high-fidelity simulation to enable real-world AI. And you've even said it represents the next generation of machine learning models. So how and why is simulation important to your work into the larger field of AI? You know, as you mentioned, high-fidelity simulation, I think, is going to be one of the most critical component of delivering real world AI systems. Let me tell you the top two reasons. Right. So the first reason is that if you look at the recent advances in machine learning and AI, these advances are very data heavy. We have spent billions of dollars in thinking about compute architecture. We have

looked into advances in algorithms in order to make machine learning systems fast. But a critical component to any of the efforts is the existence of data. If there is no data, none of this is going to matter. Right now, data is really the bottleneck you have many startups you know many labs trying to just annotate and label the data for you so that you can build richer and richer models so the scarcity of the data is one of the core reasons that simulations are going to become important and important so that's the first reason the second reason is that if you look at you know recent advances is reinforcement learning algorithms. So any of the games, any of the arcade games, those are perfect simulation to the

situations your agent will be playing in and they bank upon it to get more and more data, to gather more and more experiences and build richer and richer and better models. However, in order to utilize similar techniques for building real-world AI systems, we would need similar kind of simulation, and there doesn't exist a simulator that would mimic the real world. Let's talk about that creating reality in software, which is basically what simulation is, right?

You're using software to bring a real world into play. And you said that machine learning simulation models are like video games on steroids. Yes. What do you mean by that? And what role does reinforcement learning play in that? Yes. So, you know, when I think about simulation, especially simulation to train AI and ML agents, I think about, it's a video game. And as I said, it's a video game on steroids. And video game on steroids,

that's played not by a person, but by a software, a machine learning and AI agent that expects to explore and learn from those experience and build better machine learning models. And in fact, if you look at the recent work in reinforcement learning, in deep reinforcement learning, for instance, they are actually playing games.

They are playing Atari games. They are playing Pac-Man. And they are going to play it on and on for millions of times and get a lot of data and get better at it. So imagine if we could create a video game, which is mimicking reality in many different aspects that

we wish to deploy our AI and ML systems in, and we let these software agents play on and on and gather valuable experience, see all the corner cases, we can expect that these agents will start to get better, build better machine learning models, which then can be deployed in real world. This is not your average video game. The simulation is a game where we are not only simulating the virtual world and the character, but we are also simulating each and every sensor that your autonomous agent is going to see. All of them are being simulated to a detail such that the kind of data that your video game is providing is very similar to the real world. So consequently, if you train your system end-to-end in simulation, you can expect that a lot of that knowledge will be transferred to the real world.

So, yeah, reinforcement learning is a very interesting technology. And, you know, if you think about it, how would you take a reinforcement learning algorithm and maybe train a car or a drone to fly with it? You know, naive application to that technology in real world is going to be disastrous. Why? Because if your car makes an error, very likely it's going to be a costly mistake. Just to give you a statistic. So for instance, if you look at any of these, you know, reinforcement learning games that have been played and that have been successful, the amount of data that has been pushed through the system is incredible. So for instance, for, you know, something like Pacman, you are looking at hundreds of millions of frames before your agent starts to

do something that's reasonable. And, you know, like 100 million frames at 30 frames per second is more than a month of compute that's needed. That's going to be expensive for real-world tasks. So consequently, if you had an access to simulation where these mistakes didn't cost you anything, except for the compute cycle, maybe, your learning would be very efficient. So all of these advances in deep reinforcement learning would be, you know, very tangible for the real world applications.

If there exists a simulator that could mimic the real-world behaviors of the vehicles, as well as the consequence of all the actions those vehicles would take, you know, when they are out there. Tell me a little bit more about AirSim and what it's doing to bridge the simulation to reality gap. And why do researchers need to stress test their algorithms? So as I mentioned, right, the AirSim was really a byproduct of our efforts in building real world robotic systems. And, you know, we quickly realized that in order to be efficient, we need to come up with a simulation platform that mimics reality as closely as possible.

So we embarked on a journey to build a simulation platform that's near realistic, but also caters to the problems in the domain of AI. But the thing is, there exists no other simulator that can mimic behaviors of AI systems. And that's what AirSim aspires to be. It says that if your goal is to train a real-world AI system, let's build components. So things like actuators, the worlds, the trees, the pedestrians, other exogenous variables like wind, all the sensors you might employ, magnometer, barometer, accelerometer, gyroscopes, cameras.

Let's simulate everything in detail as much as you can. And then you can simulate the entire AI system end-to-end from collection of training data to actual deployment, validation, and testing. We will try to make the simulation as close to reality as possible so that when you deploy in the real world, a lot of the learnings in the simulation world gets transferred. This is what AirSim is. It aspires to be a simulator for solving hard AI and ML tasks for real world. That's so important for multiple reasons. Because when I hear the word AirSim, I think of like an airline simulator, right? Just you get in a cockpit, you're flying. That's not what it's about. You know, it started as a simulator for aerial robots.

This is an open source project. And as soon as we open sourced, we started to get a lot of requests to extend it to other domains, such as automobiles and other scenarios such as farming, warehouses, etc. So given that the foundation of AirSim was really about thinking about AI and machine learning system, it didn't take much effort for us to extend it and open it to other possible AI applications. So right now, AirSim is not just limited to aerial robots. In fact, it has a very rich model of ground automobiles. It has ability for you

to create vehicles and robots of your choice. You know, we have constructs, we have extensibility built into AirSim so that you can design your own robots, equip them with the sensors that you prefer, so things like multiple cameras, multiple gyroscopes. You can place them on the chassis as you wish and then let them go in the AirSim. You can even create different worlds. With AirSim,

we also release a bunch of pre-compiled environment binaries, which include cities, neighborhoods, suburbans, African savanna, mountainous environments, and many other such examples where you can deploy a robot and see how they interact. How about snow on the freeway? Yes, we can do that as well. We have at least one environment where your car, you can basically drive off-road, and it's mountainous and has all those problems with unpaved roads, etc. And that's one of the advantages of simulation.

So for instance, what we could do is we could not only simulate cities and behaviors of your vehicle, we can actually simulate time of day. We can simulate whether it's raining, how much is it raining, whether it's snowing. All of this can be simulated. And that leads to much more richer experience for a vehicle in a simulation than possibly in the real world. So, you know, at the lower level, these problems have been looked for a long, long time. You know, your brake assist, your park assist, these are the things that are already in a pretty, you know, mature stage of production. However, what's lacking is the

decision-making ability. So things like, you know, how do I expect the other car to drive? If I'm at a stop sign, when should I go? If I see a pedestrian, you know, at a certain distance, how should I act? So all of these high-level decision-making tasks are the hard ones that humans right now are doing. And that's where most of the car technology lacks. So what AirSem aspires to be, it aspires to be a platform where you can try and train and test and validate those high-level decision-making tasks. So let's talk about safety in AI systems. What are you doing to tackle this uncertainty problem in the open world? You know, as I mentioned earlier, any of our robots, autonomous cars, or flying systems,

or any IoT device that's AI-enabled will have a certain kind of a perception system, and it's never going to be foolproof. There will always be exogenous variables that we have not modeled. So there will be cases in which it's going to make a mistake. So consequently, the challenge is, how do you then come up with strategies or algorithms that would work safely, despite knowing that their perception system might sometimes fail. One of the key insights that we are banking on is that if we can build a machine learning system that has an understanding of when it's working correctly versus when it's going to make a mistake, we can then come up with a strategy that's safe. So it's akin to, say, a flying robot

that decides to fly fast when it's confident about its surrounding, but when it can't see things, when it's raining, when it's hazy, it decides to slow down in order to think more. So we are investigating strategies such as that, where we embed machine learning systems that are aware of when they are working well versus when they are going to make mistakes. And again, you know, at the core of all this is actually simulation. If you'd like to think about a brain of an AI agent, we simulate all different possibilities that we think are plausible, given what the agent has seen in the world. And after running many of these simulations, we then make a choice between taking a decision that might be aggressive

versus something that will keep us safe. And we see that in our own lives. Humans do that every day, right? Yes, yes, yes. Caution when I'm afraid. Yes. And of course, you imagine things on how things will be. If you think about our imagination as a simulation portal where we can reason about how things might

be if we took certain actions. And you want our machine learning and AI agents to have access to such imaginative portal as well, where they can introspect, where they can analyze the consequences of actions that they might want to take. Yeah. And that leads me back to this open source nature of AirSim. Why is open source so important for this particular platform? And in research in general, I'm seeing it more and more. We released AirSim as an open source platform in February 2017. And one of the reasons for making it open source is the fact that what we're trying to accomplish with AirSim is of AI complete nature. And let me explain what do I mean by AI complete nature. What we're trying to do is we're trying to create reality. And you know, to create reality is to understand reality. So it's a seemingly impossible

task that a small group of people cannot accomplish. So we hope that by open sourcing it, we would excite, you know, many researchers, students, developers, you know, who want to participate and contribute back to this project with realistic model of vehicles, sensors, agents, environments, so that we can bootstrap and build something that we individually ourselves couldn't build. And since open sourcing, we have had an active engagement with developers and researchers all around the world, with many of them contributing in very significant ways. Tell me how collaboration happens in your world. As you can imagine, our work is of very interdisciplinary nature.

The team comprises of roboticists, machine learning researchers, developers, engineers, pilots. And the thing is, when you think about such an interdisciplinary project, you need to realize that many of these collaborations have happened very spontaneously, like over lunch or a coffee or tea. It is clear to all of us that given the ambitious nature of some of these projects, we need to work together and we need to actively go and seek out folks who are accomplishing some things that can add and complement to our

research. One of the great things about this place is that there are hardly any boundaries. And many of our research projects have been a result of an informal hallway conversation or a conversation over lunch or dinner, where pretty much we talk about what we are doing, and then an idea clicks, and we go ahead and build it. Your background is really interesting. You're an AI researcher, but you're also a pilot and a flight instructor and you've actually built an airplane. How did aviation and machine learning research come together in your life? I was always enamored by anything that flies from a very young age. And, you know, computer science was an important thing as well growing up. You study computer science because growing up in India, there weren't many opportunities for aviation, but, you know, computer science was an upcoming field. And I was always in love with algorithms and AI specifically,

ended up coming to US and did my PhD in machine learning. Once I started working at Microsoft, I realized that it's time for me to go back and see if there's something I could do in aviation. And, you know, Seattle is the birthplace of aviation. That's how I think about it. I got my private pilot's license in 2009 and quickly realized the potential of technology and specifically the role of AI and ML that can possibly change the field of aviation. In fact, you know, beyond autonomous cars, there's so much discussion on autonomous flight. So autonomous personalized vehicles that can possibly take you from here to, say, Seattle downtown in less than 15 minutes. Right.

And I mean, to be honest, the technology is maturing very rapidly. And the way I think is that AI and ML are going to be one of the critical components to enable that. So, you know, if you think about self-driving car or self-flying car is even a harder thing to do. And consequently, we need to make sure that those vehicles are perfectly safe and they would be pretty much piloted via an AI and ML systems. It should be as simple for anyone to go in a car, put an address, and just relax and sit. And that vehicle should go and drop you there. And in order to enable that scenario, AI and ML would be the core engine that would make this thing happen.

This is what I had been thinking about. And through our informatics and robotics research program, we have been trying to answer some of these questions. I think we are at the cusp of technological breakthrough in that space. There's already, you know, many efforts in the space that are trying to accomplish it. So you mentioned the airplane that I built. Yeah. Right. So, you know, I ended up designing the avionics system as well.

And it is designed to run AI and ML algorithms on it. It has sensing for other vehicles in the vicinity. It has onboard weather. It has an autopilot system that can maneuver it not just horizontally, but also vertically. But more importantly, we have the ability to interface that airplane with a planning or a robotic decision-making system

that can guide that airplane to fly planning or a robotic decision-making system that can guide that airplane to fly, you know, autonomously, at least in the cruise phase of flight. So the way I see my airplane, the technologies are already there. If I could just solve landing and takeoff, that would be a flying car that I would bet my life on. That's the first and last mile problem of trekking or cars. It's just that last bit where the tricky parts come in. Yes, exactly. And I think those are the hardest parts because now we are very close to the ground in the vicinity of population and other moving vehicles. And we need to make sure we do a better job in those spaces. Have you flown the plane you

built? Yeah, yeah, yeah. So I flight tested it. I did the first flight and probably it was the scariest thing that I ever did. But it also taught me, you know, you pretty much build your own airplane and then push it through the certification as well as the flight testing regime. And it gives you a sense of what it takes to build robust systems. I'd like to contrast the state of AI systems to aviation systems, say, 100 years back. It's pretty dangerous to fly. But as a race, we not only enable that technology, but we came up with protocols. We came up with processes that took that technology and matured the whole technology. And now, you know, we can do things that possibly 100 years ago, we never would have imagined. And that's what the state

of AI is. It might seem very scary. There are a lot of unknowns, a lot of uncertainties, but there are a lot of positive benefits of the technology. And I think together we should strive to think about processes and regulations so that we can start using these technologies in beneficial ways. So that kind of leads into the question I ask every researcher that comes into the podcast booth. Is there anything about your work that keeps you up at night in terms of, you know, possible downsides or unintended consequences? So when you think about AI or machine learning, machine intelligence technologies, there is a lot of skepticism about, you know, potential negative effects. And a lot of this

fear and skepticism, I personally believe, is coming because of lack of proper understanding. So as we move forward, we need to start debunking some of the myths and education and knowledge dissemination is one of the ways to do that. And this knowledge dissemination needs to go all the way to the lawmakers, the regulators, so that they can make better decisions. You know, as I mentioned, AI is a pretty powerful technology. You can utilize it to do social good, but you can also use it to do harm. You know, it's the prospect of doing harm that raises a lot of fears and skepticism in everyone's mind, which might further affect how laws and regulations are being made. So right now we are seeing that the advances in AI are happening at a breakneck speed.

Yeah. However, many of these advances in order to be realized in real world, in order to start affecting our lives, you know, we have to start thinking about regulatory and the law aspects of it. And there is a chance that the regulation and the lawmaking might lag the amount of innovation that's happening in the AI space. So consequently, I worry about that possibly, you know, regulations, which might be a knee-jerk reaction to certain situations, might lead to stifling of innovation in our space. And so consequently, thinking about mechanisms in which we can keep innovating as well as think

about consequences in society so that we can come up with laws and regulations at a pace that can match the innovation sphere. As we finish up here, what do you see on the horizon in your field? And what are the big challenges that we want the next generation of best and brightest minds to be working on? What advice would you give to young aspiring researchers who might want to help tackle these problems? One thing that I'd encourage everyone who aspires to work in this field is to actually think about technology that would influence a large amount of people in the world, right? Often, you know, the technological innovations, by the time they trickle down to the masses, it takes ages. And given the power of AI,

we want these technologies to be effective as soon as possible. Like you probably know, for instance, from the farming perspective, we need to double the yield of our crops by 2050 without increasing arable land. So if we invest in AI technologies that can help solve that goal, we can assure that we will reach masses. So I would strongly encourage folks

to think about these problem areas where most of the population of the world gets positively influenced. Ashish Kapoor, it's been absolutely fascinating talking to you today, and I'm so grateful that you came in and sat down in the podcast with us. Thank you. It's been wonderful talking to you. To learn more about Dr. Ashish Kapoor and the machine learning approach to building autonomous and robotic systems, visit Microsoft.com slash research.