The a16z Show - a16z Podcast: The Self-Flying Camera

Starting point is 00:00:00 The content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. For more details, please see A16Z.com slash disclosures. Hi and welcome to the A16Z podcast. I'm Hannah, and we're talking today with Adam Brie, co-founder and CEO of Skydeo and A16Z general partner Chris Dixon about what happens now that we're reaching truly autonomous drones, in this case, cell flying cameras, what it means when you take the pilot out of the loop and what becomes possible that wasn't possible before. We take a deep dive under the hood into the tech that makes this work from propellers to perception, then talk

Starting point is 00:00:43 about what autonomy will enable and what we can build on top of it, to finally how our relationship and interactions with computers of all kinds will change as they become increasingly powered by AI. Let's start by maybe talking a little bit about the evolution of drone technology. Where are we now? did it take for us to get here? In a lot of ways, the products that we have today essentially grew out of RC airplanes and RC helicopters. This is stuff that's been around for a long time, 30 or 40 years. Electric power was a huge transition point. And then the combination of cell phone sensors with an imaging device just turned out to be very powerful. And that's what's caused kind of the recent spike in attention in the space. So the paradigm with existing drones that

Starting point is 00:01:23 are available today is essentially they're manually flown devices. But still, it required a pilot. Above all. There needed to be a human learning how to use it and directing it. Yeah, exactly. You buy the thing. You hold joysticks. You're essentially the pilot, and you're responsible for flying and controlling it. And if you're an expert, if you're proficient at that, you can do some pretty cool things, but it's a very difficult thing to master. And I would say a typical experience for a lot of people is they take it out of the box and they crash it into a tree. So it feels like an industry that is still very, very early compared to what's physically possible. There's all these great notions of what you might want to do with a drone. You know, on the consumer side, having this camera that makes it easy to capture amazing video

Starting point is 00:02:02 and for commercial applications like inspection, mapping, monitoring, security. But most of them, we think at their core, in order to really work and in order to really scale, need an autonomous foundation. And so our focus is on giving the drone the ability to fly itself intelligently, which is a very simple idea, but it's a very technically challenging thing to deliver on. Can you explain why the autonomy is. so important, what the human failing is that makes that change everything. I mean, I could use an analogy here. Imagine if on your iPhone, if you wanted to take a picture with it and you push the wrong button at the wrong time, it would irreparably destroy itself. I think we'd all agree that

Starting point is 00:02:42 that would be like a pretty big barrier. Not really workable. To using iPhones, and there probably wouldn't be nearly as many of them out there, and people probably wouldn't like them as much as they do. And that's basically where we are in the drone industry now. We think autonomy is a huge enabler. just at that basic level of being able to trust these devices to do reasonable things in every situation, it removes the need for the operator to be paying attention to it or flying it at all times, which depending on what you're doing, can be pretty huge. I also think, going back to your iPhone analogy, one thing the iPhone enabled, because it abstracted away things like GPS and accelerometers and made it really easy for software developers to access those things.

Starting point is 00:03:21 It allowed, for example, Uber and Lyft and Instagram and all sorts of things to focus just on the kind of Apple application development side of things, so allowed app developers to be app developers and provided the phone operating system provided all the functionality of the box. And so longer term, in my view, that, you know, sort of autonomy, knowing the drone will avoid obstacles, be able to do kind of high-level functionality like identify people and objects, will eventually allow software developers to build applications in the same way they did on the phone. Yeah, exactly. The iPhone, in a lot of ways, it's basically a big touchscreen that you can put whatever you want on and it abstracts away the hardware and makes it possible to just use that screen to interface with the user for a bunch of

Starting point is 00:04:00 different applications. With drones, autonomy is likely to be sort of that key layer that separates sort of the underlying physics, worrying about collisions, worrying about a lot of the nitty-gritty robotics navigation problems with the application so that if you wanted to do roof inspection, you could be an expert in how roof inspection works and the workflows there and write for that rather than having to worry about what the drone's doing. For commercial applications, you don't actually want a pilot flying every drone. Like, you just want to push a button, have the thing do its mission, or have many of them do their missions, and then get the data back that you care about. And that's what autonomy will enable, sort of taking that core layer and making it possible to do a lot of different

Starting point is 00:04:40 kinds of things on top of it. Well, let's talk about the autonomy. Is it the same kind of autonomy that goes into, say, self-driving cars? How is it different from the different modes of autonomous robotics across the board? So in order to make a fully autonomous system, we've really designed it for autonomy from the ground up. So it's not sort of like an afterthought where we strapped on some extra sensors. There are a lot of similarities to self-driving cars. And we call our first product a self-flying camera for that reason because it's taking control itself. But the way that the algorithms are designed are specifically tailored for flight.

Starting point is 00:05:12 So whereas a car relies on the structure of the road and staying in its lane. Right. There's certain givens that you can't. Yeah. There's a lot of givens on the road that you can. can't take for granted in the air. With a car, your goal is basically just to follow the rules. The perfect self-driving car is just going to follow the rules and be predictable. Whereas with drones, we do a lot of just sort of general 3D reasoning, understanding the scene in every direction in any time.

Starting point is 00:05:36 And it's kind of a different set of challenges. In some ways, it's harder, in some ways it's easier, but it's also a regime that sort of allows for a lot more inherent freedom. And there's just sort of more of a creative aspect to it. It's interesting when you talk about the givens on the road, you know, that you think about like a static road that you're, I mean, do you guys have to factor in things like wind and breezes, or is that just a much older, obvious technology? Because it's basically like a helicopter. You know, how do you think about moving through space like that? Yeah, you're right. I mean, there are a lot of variables in play. And there's a lot of things that are sort of taken for granted in manually flown drones that, oh, if that comes up,

Starting point is 00:06:11 the pilot's just going to be able to deal with it. Like what? Like a giant gust of wind, for example. Oh, you feel it. You can see it. Yeah. If you're flying it manually, a big gust of wind comes up, you'll see it and you'll respond appropriately. And if you're next to a building, you'll probably fly away from that building to make sure you don't get blown into it. And if you couple that with you have a low battery or something else, like there's just a lot of things happening. And one of the challenges has been building a system that can reason intelligently in all these different situations. And are you using cameras to do that too? Or what are the other ways you collect instant data about physical movement?

Starting point is 00:06:45 So actually when we register through a sensor called an IMU, which is an inertial measurement unit, So this is an example of where phone technology has kind of transferred over into robotics. Your phone has an iMU in it, so it can tell which way is north, and it can tell what its orientation is. And so we use basically exactly the same component, and you can actually use that to sense the force that's being applied to the vehicle externally. And so it's constantly seeing the force that it's experiencing from the world, and then using that to estimate what's causing that and what it should do about that. And the drones are constrained on the sensors in a different way than with cars. Yeah, there's much stricter limits around the same. sensing and compute that you can carry.

Starting point is 00:07:22 And that's because of the size of it. Because of the size and the weight. Everything that you're carrying, you have to have propulsion to lift off of the ground. So, for example, LIDAR is very popular with self-driving cars and is not feasible for drones. Well, it's feasible for drones, but it's not going to work very well. We did a lot of work with LIDAR, but the vehicles were super heavy, super expensive. So when we started Skydeo, we made a big bet on vision because we felt like the progress that's happening in computer vision now, especially with deep learning, but even in sort of

Starting point is 00:07:49 traditional geometric vision is just super fast. And the amount of information that's in images is incredibly rich. Extracting that information is challenging, but the tools to do that keep getting better, and we're really riding that way. We have 13 cameras on the device. They see in every direction. And then at its core, it has a super powerful computer. We're using the NVIDIA TX1, which is basically like a deep learning supercomputer

Starting point is 00:08:13 that uses the same GPU design, the same architecture that's found in their cloud compute systems, and all of that is necessary to run the software and flight algorithms that give it the autonomous behavior. So break it down for what does that actually mean in Skydeo's case? You're using cameras plus intelligence instead of LiDAR, right? What are the layers that make that work? We've developed what we call the Skydeo autonomy engine, and that's basically the complete system that does all the perception, decision making, and control of the vehicle. So it's a technically very complex thing. But it's actually fairly intuitive. to understand what it's doing because it's similar in a lot of ways to what people do.

Starting point is 00:08:51 It basically processes visual information. It uses the visual information to figure out the 3D structure of the scene, so where everything is. And then it builds up sort of a deeper understanding of what the different kinds of objects are. So in particular, we care about people, we care about objects that we might run into. And then all of that information goes into a planning system that balances a bunch of different objectives about what the drone's trying to accomplish. It's trying to capture smooth video. You definitely don't want to run into things. need to obey the maneuvering limits of the device. So all of these things are constantly being traded

Starting point is 00:09:22 off for it to decide what it should do. Thinking about how this tool is navigating when it's following you around and mapping its way through space, a different kind of space, right, because it's aired. How does it actually do that? How does that work? How does it choose where it's going? So one of the keys to getting intelligent behavior is you actually have to predict what's going to happen. Because if you don't predict what's going to happen, you end up being purely myopic and you just react to whatever just happened. The path planning algorithm is actually predicting about four seconds into the future. So it's looking at what you're doing.

Starting point is 00:09:52 It's looking at the environment around it. It's reasoning. And then it's using all of these sort of future possibilities to figure out what it should do. And even though it's predicting four seconds into the future, it's not waiting to the end of those four seconds to then decide what to do. It's constantly doing that. So many times a second, it's making these predictions with the latest information about what you're doing and what the environment around it is doing

Starting point is 00:10:15 and then updating its notion of what it should do based on that. How about some of the design choices you're making as you're combining these different types of technology, right? The propeller aspect and the weight and the cameras and how they're all fitting together, the whole stack, what were your considerations? So I think one of the overall things here, this general idea of like the RC helicopter

Starting point is 00:10:33 that's gotten better and better, but then there's this other sort of paradigm of like, what if this thing's a flying computer? And we've kind of come at it more from that direction, And that's reflected in a lot of the design choices we made. So we really do think about it more like a flying computer. And we've tried to make aesthetic design choices that reflect that. So one example of this is that we built in this perimeter blade guard structure as a first-class citizen of device.

Starting point is 00:10:57 So it's made out of carbon fiber to be super light and stiff. And it means that you don't have exposed propellers around the perimeter. Right. You don't chop off your finger-discuit. Yeah. So it gives it a significant safety benefit for flying around people. but it also, I think, is just closer to the kinds of things that we're used to sliding into and out of backpacks because it's just sort of this single compact thing that you can, like, hold anywhere you want and is easy to manage yourself as opposed to having propeller blades sticking off and landing gear hanging down and things like this. So we've talked a bunch about the under the hood technology.

Starting point is 00:11:29 How about on a more basic level what the personal experience is like of how the technology actually works when you first begin using it? So, like, the goal is to make it really easy to capture amazing videos. from this dynamic perspective that you wouldn't otherwise be able to get of you doing your favorite activities. So there's a lot of complex technology under the hood to make it all possible, but the end user experience is pretty simple. The way this works is you take it out of your backpack, you turn it on, and you're controlling it from your smartphone. So you can hold it in your hand and swipe up to take off, and it'll take off from your hand, it'll fly away from you, and turn around and look at you. We have a deep neural network that's been trading against a bunch of different

Starting point is 00:12:07 datasets, some of it are own, some of it open source data sets, to recognize people robustly. So when it looks at you, it'll know that you're a person, it builds up a unique visual identifier of what you look like based on your appearance, your clothing, things like this, and it'll use that to tell you apart from other people in the scene. And after that, you can put your phone away and you're done, and it'll just follow you, it'll avoid obstacles, it sees in every direction, and you can go for a run, you can go hiking, biking, skiing, things like this, and it will move itself in a nice, smooth way, and give you video that literally looks like it was filmed by a professional film crew.

Starting point is 00:12:41 No longer have to be staring up at it. Yeah. I mean, it looks like you have a Hollywood crew there with like booms and dollies and these things, like moving a camera around, but it's just doing all of it itself through software. It's like a two-step magic trick. The first step, people see the drone fly around and follow you,

Starting point is 00:12:56 and when the drone does that, it's moving a lot, right, to navigate. With all the stabilization, both hardware and software stabilization, people are shocked at how the video looks afterwards. It's not just the tracking, it's the actual quality of the image that you're getting. How do you think storytelling starts to change when we're now enabled to be at the star of your own movie like that? You know, when the point of view just becomes sort of you in your life. And it's technologically possible to just let that unfold. How do you think that starts to change?

Starting point is 00:13:22 I think it's really exciting. I mean, I think we're still very early days on all this stuff. But if you just think about how fundamental cameras are to our daily lives now, like the major use case for small. smartphones is taking pictures and videos. This is the feature that gets the most attention there. We have huge social networks that are built up primarily on top of sharing picture and video. And the ability of a camera to understand what it's looking at and move itself and autonomously capture footage, I think is going to be like a really powerful ingredient in the world in a lot of different ways. It's a totally new tool.

Starting point is 00:13:57 Yeah. And, you know, the most interesting pictures in video are generally of people. Over the last six months of development, we've had a bunch of prototypes around the company. We've done a lot of internal testing. We've also had external beta testers. And one of the exciting things to us is seeing the footage that comes back. There's these wacky, creative ones where... Yeah, what stuff do people do? So we had an employee who basically made a music video in his driveway

Starting point is 00:14:19 where, like, he had the camera flying, he was dancing around, and there was this interaction between him and the device that resulted in this video that's kind of captivating to watch that you really wouldn't get any other way. It's not like he was going to hire a film crew to come and film this thing in his driveway, but with the device, with almost no pre-planning, he could just put it in the air and get this kind of amazing thing. I want to talk about how this actually feels, you know, how it's first recognizes you and the relationship you kind of develop with your drone, right? Because it's really now a relationship thing, kind of. How does that play out? What does it feel like?

Starting point is 00:14:53 I think it's impossible not to personify it, really. And we like that. I mean, it responds to you. It acts intelligently based on what you do and what's around it. And so there is this kind of like fun aspect of it's like it's sort of your companion. The real element of the product experience. Well, it also makes it part of your life in a different way, right? You're not just a bug under a microscope.

Starting point is 00:15:13 It's also interesting when you talk about the point of view changing, right? There's a perceptual distance that you're getting. There's a perspective. There's a perspective shift. It seems to me like that's an interesting kind of like mental consciousness shift. That happens when you start seeing yourself from that. other perspective. The broader trend is kind of a new thing in computing where basically for the entire

Starting point is 00:15:33 history of computers, they've been these static objects that you like type on or tap on. And it, you know, at the end of the day, almost all computing results in just showing you an image on a screen that then you react to. And so the ability of a computer to sense its environment and act in the physical world, I mean, that's basically what a robot is, but we're entering this new sort of world where these kinds of things are possible. It's a much more dynamic relationship. Just from the technology point of view that people started to see it for like Alexa and products like this.

Starting point is 00:16:01 And generally, there's sort of, you know, there's been all of this incredible progress in AI in the last couple of years. But I think still a lot of it hasn't reached, you know, production products yet. But we're going to see more and more over the next couple of years computing devices that do feel like active agents. Active agents. That's a really good way to put it. And most public is thinking now of AI as this like dialogue. But the relationship with you is silent. You're not talking to it.

Starting point is 00:16:25 you're not telling it to follow you. It's all like implicit and silent. Yeah, I mean, it's kind of like physical. Like, it's in your space and it's responding to what you do. I mean, in a lot of ways, you control it through your motion, like not through gestures, but through how you're actually physically moving. So right now the drone is designed primarily for consumer use cases. Imagining down the road, what do you think other possible use cases for autonomous drones are?

Starting point is 00:16:49 There are a lot of exciting things on the commercial side. And in a lot of ways, I would say it's kind of less mature and, wide open than consumer even. General possibilities of using drones to automate collecting data that either isn't collected today but should be or is kind of difficult and manual and slow to get. So we mentioned roof inspection.

Starting point is 00:17:07 There's this fairly manual, slow, painful process where people actually have to get on a ladder, climb onto a roof, look for physical damage to inventory it for insurance purposes. You know, it's not a particularly efficient thing. It's pretty dangerous to the people that do it. This could be done super efficiently by a drone,

Starting point is 00:17:23 but flying it manually to do that would be hard. People's roofs have trees and power lines and things like this around. And so you don't want the world's best drone pilot to have to go out to your house to safely fly this thing around. That doesn't scale. To make it work. There's a lot of other things like that where autonomy could have a really big impact. Construction sites have become actually a popular case for drones. The drone flies over and looks at the state of the construction, make sure things are put away safely, you know, sees how the progress is going, check the inventory levels.

Starting point is 00:17:52 Yeah, I mean, I think, you know, infrastructure inspection, mining, bridges. make sure it's not going to fall. These things are very hard to inspect with humans. A lot of these things are basically like efficiently digitizing the physical world in some way so that that data can be tracked over time and issues and errors can be found. There's something in the order of like 10 to 20 million jobs in the U.S. where people have to climb things. Like for example, the most dangerous job in America is climbing cell towers to inspect them.

Starting point is 00:18:19 They have to be inspected every like six months or something to see if the equipment's working and if it's like for safety issues, things like this. There's 100,000 oil rigs in the world and the salt water can corrode the steel and then you have gas and fire and you can imagine the bad things that can happen if you don't inspect it properly. And today it's a very manual intense process. I met a company recently using drones to do that. In all these cases, they're all using kind of manually piloted drones. And so the big constraint is they have to have these expert pilots do this, right? Because imagine if, obviously, if you hit something, it could be catastrophic.

Starting point is 00:18:50 It's precisely for all the things that the human scale is not enough. but we've still, until this point, been held back by the human scale. Now suddenly that's out of the picture. So I'm just going to ask, when those humans are out of the picture, in this new autonomous drone ecosystem, where are the roles for the humans in that? I think there's still humans. It's just the human doesn't have to do the dangerous job

Starting point is 00:19:09 and you can do it more efficiently, but you're still going to need all sorts of things around, like an insurance case, around kind of analyzing the claim and paying it out and doing all the kind of work and bringing the drone out there. I think it's all leverage, so it means that, like, the people who are involved are working at a moment, much higher level of abstraction and commanding a lot of resources to get the data in an

Starting point is 00:19:28 efficient way. It's about strategy of data kind of, an analysis. And then making decisions afterwards about what do we need to do based on whatever we've learned. The history of technology is, you know, there's a sort of fundamentally asymmetry where it's easy to imagine the jobs that go away and hard to imagine the ones that are created. But the history of it shows that, you know, with every new technology wave, yeah, there are things that were done before that become obviated, but there's a whole new set of things that come along. Dron pilot wasn't even a job description five years ago, and fundamentally, I think drones are likely to enable a lot more new areas of creativity, new kinds of value to be created.

Starting point is 00:20:04 If one of the big limits to the drone space, which we're now achieving is autonomy, what's the next state of the art? What are we pushing up against next? I mean, I think you can look at birds as an example of what's possible. Like, how far is a drone that you buy today from what a bird is capable of? I would say pretty far. These things are kind of like bulky and difficult to manage. And autonomy and intelligence seems like one key element of that. But a lot of the hardware design aspects also, I think, are like relatively early and immature. And it hasn't been possible until now to build a small lightweight device with a bunch of onboard intelligence, a bunch of sensing, an electric power

Starting point is 00:20:42 system all packed into one. There are some really powerful new combinations of those things that we're going to start to see emerge over the next few years. This is part of a broader wave, which is if you just go back and look at the history of computing, we had the main frames, where we had sort of a one computer for every 10,000 people or 100,000 people,

Starting point is 00:21:00 then you had PCs, sort of one for every 10 people, and then you had smartphones. We now have 3 billion smartphones, so sort of a computer per person, or we're going to get to that point. And now we're going to start to see kind of 10 plus computers per person, and that means, you know,

Starting point is 00:21:14 computers embedded around your house, at the office, in the air, in drones, in your car, you know, VR, AR headsets. I think we're on the cusp of this kind of Cambrian explosion of computing devices. A little galaxy around each of us. Yeah, a little galaxy around, all powered by AI. AI is the critical ingredient, which lets these devices understand and interact with the real world. And new interfaces like speech and gestures and the ability just to walk around and have it follow you.

Starting point is 00:21:42 Gestures you might not even be aware of, right? That's right. And then just the ability to understand the environment. And the second order implications of this, I think, are really profound, which is, you know, we don't even know yet what people are going to do with these devices and all the applications. You know, if you said to somebody in 2005, there's going to be these amazing smartphones, I don't think people would have predicted some of the applications that people developed. So, for example, ride sharing, it wasn't a widespread prediction, right? And so I think there will be things like that. Once developers and creative people have drones in their hands, drones that are fully programmable sort of flying computers, what are all the new things they come up with,

Starting point is 00:22:16 It's going to be, I think, a really exciting time of the next three to five years as we discover that. Well, if you had to take a wild guess and say, okay, this is the human behavior that I think is going to change. What would you say one of the first ways we're going to start seeing human behavior change because of this self-flying camera? People call me an optimist, but I think a lot of what computing does is democratizes things that were only available previously to, you know, super wealthy people. Today, someone with an iPhone and Google has more information than the President of the United States did 20 years ago, right? similarly like you could film things like you could do with skydeo but you had to have a Hollywood film crew right now you're democratizing that in the same way that you know i movie sort of democratized film editing what happens then right what happens sort of second order to that

Starting point is 00:22:56 well i guess we're on the way to soon finding out thank you both so much for joining us on the a16s podcast

The a16z Show - a16z Podcast: The Self-Flying Camera

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.