Microsoft Research Podcast - Abstracts: May 20, 2024

Episode Date: May 20, 2024

Andrey Kolobov discusses WindSeer, a small CNN capable of estimating the wind field around an sUAV in flight more finely and with less compute and data than traditional models. The advancement can help support longer and safer autonomous flights.Learn more:WindSeer: Real-time volumetric wind prediction over complex terrain aboard a small uncrewed aerial vehicle

Welcome to Abstracts, a Microsoft Research podcast that puts the spotlight on world-class research in brief. I'm Dr. Gretchen Huizenga. In this series, members of the research community at Microsoft give us a quick snapshot or a podcast abstract of their new and noteworthy papers.

I'm here today with Dr. Andrei Kolobov, a principal research manager at Microsoft Research. Dr. Kolobov is co-author of a paper called Windseer, real-time volumetric wind prediction over complex terrain aboard a small, uncrewed aerial vehicle, otherwise known as an SUAV. Andrei Kolobov, great to have you on Abstracts. Thank you for having me. So let's start with a sort of abstract of your abstract in just a few sentences. Tell us about the problem your research addresses, and more importantly, why we should care about it.

Right. So the overarching goal of this work, and I have to thank my collaborators from ETH Zurich, without whom this work would have been impossible. So the overarching goal of our work was to give drones the ability to stay aloft longer, safer, and cover larger distances. The reason why this is important is because drones' potential for, for instance, quick delivery of small goods has long been understood, but in practice, their usefulness has been limited by the time they can spend in the air, by how quickly they drain their battery. And lifting these limitations brings the reality of getting the stuff that you order on the internet delivered to you quickly by drones closer. Is that the core problem, is drone delivery? Of course, when we were starting this project, we were not interested in any one application. We were interested in implications

of AI for drone flight. The limitations of drones time aloft ultimately come from drone flight technology, which is very well established, very well understood, and ultimately relies on drones actively fighting forces of nature, such as gravity and wind, and because of this, draining their batteries quickly. So within the framework of that technology, it's difficult to get around these limitations. So what we're aiming to show is that using AI, drones can reason about their environment in ways that allow them to embrace these forces of nature rather than actively fight them and thereby save a lot on energy and increase their time in the air. Right. So are we conflating drones with SUAVs, as it were, small uncrewed aerial vehicle? Yes, this work, we are somewhat conflating them, but this work focuses specifically on small UAVs, small drones, because these drones' ability to fight forces of nature

is quite limited. Their battery life is way more limited than that of larger drones, and for them, this work is especially important. Okay, and I'm assuming it's not a new problem, and also assuming that you're not entering a field with no previous research. So, what's been done in this area before, and what gap in the literature or the practice does your research fill? Yeah, of course. So, certainly, many other very, very smart people have thought about this area. What we have tried doing and what we have accomplished differs from previous efforts in how much compute, how little data at inference time our method requires and also the fine scale at which it makes its predictions. Obviously, there are weather models that model various aspects of the atmosphere. And they can predict wind. But they can do this at the scales of hours,

at spatial scales of tens of miles, which is way too crude to be useful for drone flight at low altitudes. And also, these models do this at much higher altitudes, not where drones fly, close to the ground, where it's very important for them to know about wind, to avoid collision with terrain potentially, but very high up in the air. The tool that could solve the same problem that we were trying to solve conceptually are computational fluid dynamics simulations, so-called CFD simulations. However, they're very expensive. They cannot run

on the drone. And so if you want the drone to be fully autonomous, they're not really a feasible solution. So how would you describe then how you attacked this problem? What methodology did you use for this work? And how did you go about conducting the research? So one thing that people reading about this work might find funny is this déjà vu feeling of seeing the overarching technical insight that we had in a completely different context, in the context of trading models such as Microsoft's PHY. The reason why it's funny is because we were trying to solve an entirely different problem in a project that started in a different era, research era, in the pre-large model era, and yet we came up with something quite similar. And this overarching technical insight is this.

If you want to build a small but powerful model, one way of doing this is to find a powerful but potentially computationally expensive or expensive in some other way, generative data source, generate data from that source in a very carefully controlled manner and use this carefully constructed data set to train your model.

This is exactly what we did. In our case, this powerful but expensive generative data source were the computational fluid dynamic simulations, which we used in combination with 3D terrain maps that are publicly available on the internet to generate a lot of high-quality data, throw in a few more tricks, and get the model that we wanted. Can you talk about the few more tricks? Well, so we needed to train this model to make predictions based on very little data. Computational fluid dynamic simulations

typically need a lot of data at prediction time. And so the so-called boundary conditions essentially need to know the wind at many locations in order to be able to predict it at the location that you're interested in. And so we had to structure the data generation in a way that allowed us to avoid this limitation. Talk to me a little bit more about the data sets that you used. Yes. So all the data was synthetically generated. All of it? All of it. All of it was generated from computational fluid dynamic simulations. And was this methodology unique and new, or is it kind of building on other ways of doing things? So the idea of using high

quality data sources under various guises had been known in the community, to various research communities in any case. Some would refer to it as distillation, some would refer to it as data assimilation, so in the context of these predictive weather models it would be known as data assimilation. But none of them were doing what we're trying to do again, which is getting a model that would make predictions on a very limited compute with a very limited amount of data at inference time. Well, let's move from research methods to research findings. Give us a quick overview of how things worked out for you and what you found.

So in a nutshell, as trivial as it sounds, the surprising finding was that that it works the again the the reason why it's surprising is again we used only synthetic data to predict something very very real and something that people have put a lot of uh thinking into modeling as part of weather models for instance and it turned out that using just synthetic data, you can get a small model that as the drone is flying through the air, and as it's measuring wind at its current location, this model allows you to predict that there is a downdraft 300 feet away from the drone on the other side of the hill. It's just amazing that something so small can do something so complex and powerful. Right. Well, let's drill in there and kind of talk about real-world impact here, because this is really important for a lot of wind prediction scenarios. How does this impact real-world scenarios? Who benefits most from the kinds of applications that you might get from this?

Yeah, so there is a number of scenarios where it's valuable to have a drone, usually a fixed-wing drone, that due to its inherent characteristics can stay in there longer than a copter drone. Where it's beneficial to have such a drone stay in the air for long periods of time, silently observing something. So the applications range from agriculture to environment conservation, where you want to track the movements, migrations of animals to security. And of course, the technology that we developed does not have to be applied to fixed-wing drones. It can also be applied to copter drones, which is the drone model that is usually considered for use in drone delivery. And those drones can also benefit from it, especially in city conditions where presumably

they will have to fly around skyscrapers and take into account the effects that the skyscrapers and other buildings and structures have on the wind near terrain. So, one more question on the real-world impact. In your paper, you talked a little bit about wind farming and other places where understanding how wind works and being able to predict it matters. Is that one? Are there others? It for sure is one area. Again, in this work we focused mostly on applications of wind prediction that have to do with drones. Besides time aloft, one application is safety. In many places around rough terrain, you know, in the mountains, predicting wind, predicting downdrafts and updrafts has safety implications because drones fly so close to terrain and the winds, the airflow can be so strong in some places over such terrain

that it can basically drag the drone into the ground, no matter what drone does. It can do it very, very quickly. So again, predicting such phenomena there becomes a matter of drone safety. The same applies or will apply in city conditions where drones will be flying among buildings, and wind can be so strong that it can carry a drone into a building or into another obstacle. Well, I assume you didn't solve everything with this paper, and that there might still be some open questions remaining in the field. So what are some of the big outstanding challenges people still face here? And what's next on your research agenda to overcome them?

Of course, this work is in some sense just the beginning. This work is about helping drones make sense of the environment around them. But this ability to make sense is not by itself useful without drones being able to use the results of this estimation in order to plan how to fly in a safer and more energy way, and to adapt their plans as the environment around them changes. So this is a natural next step. Have drones take their predictions into account when planning their actions. Well, Andrey Kolobov, thanks for joining us today. And to our listeners, thanks for tuning in. If you want to read this paper, you can find a link at aka.ms forward slash abstracts

or you can find one on Archive. You can also read it on Nature Communications in Volume 15, April 25th. See you next time on Abstracts. Thank you.