Embedded - 246: Sacrifice That Computer

Episode Date: May 18, 2018

We spoke with Dr. Bennie Lewis (@_benjoe02) about machine learning and robotics. Bennie is a Senior Research Scientist at Lockheed Martin, content creator, and Twitch streamer (benjoe02) NVIDIA Jetson platformand Cuda for deep learning LEGO Mindstorms SAMS C++ in One Hour a Dayby Siddhartha Rao

Welcome to Embedded. I am Elysia White here with Christopher White. Our guest this week is Dr. Benny Lewis. I think we're going to talk about machine learning and robots, but Benny's worked on so many interesting things. We'll see where the conversation takes us. Hi, Benny. Thanks for joining us today. Hey, thank you. Thanks for having me. Could you tell us about yourself and your background? Yes. My name is Dr. Benny Lewis. I am a senior research scientist at Lockheed Martin Space. I work in the Advanced Technology Center where we do a lot of R&D from deep learning, machine learning, user interfaces, robotics for not you know, not only space payloads

and aircrafts and vehicles and also ground stations as well. Before that, I was a software engineer at also at Lockheed Martin Missiles and Fire Control in Orlando, Florida, where I also got my bachelor's, master's and PhD at the University of Central Florida in computer engineering with a focus on AI and intelligent system and robotics. And my thesis was human-robot manipulation, collaboration for multi-robot systems and user interfaces. And that's just a small cut of my research. You know, I've been, got a couple of publications out there. Of course course you need those for getting a phd and um i also was able i just had the pleasure to you know talk about my

research in multiple parts of the world from japan to toronto to spain and you know numerous of states United States. And so, yeah. And you Twitch. Oh, yes. That isn't right. I also am an avid gamer, streamer, and content creator. And I do STEM Sundays where I post a lot of experiments that I do with my kids as far as miners from Lego robotics to science experiments from building volcanoes to looking at the solar system and things like that. So I do a couple of those. I stream a lot of, like I said, video games. I usually do Tuesdays, Thursdays, and Saturdays just because of my hectic work schedule and travel. But then I also like to create little videos about video games and also about

technology that's out. Cool. Well, we have a lot of questions and I'm hoping we talk more about your channels, but I have so many questions about robots and AI that we'll see where it gets. Okay. Let's go. But before we do that, we want to do lightning round. I think you've heard of this where we ask you short questions and want short answers. And if we're behaving ourselves, we won't ask you for lots of detail.

Okay. Do you like to complete one project or start a dozen? Yeah, I work on a lot and I complete a lot. I can't just sit and focus on one thing just because that's not how my mind works. So if you get to know me, you'll know that I'm not only am I doing one thing, I'm doing a couple of things at the same time. What one thing do you want people to remember about you? Science, you know, technology, you know, AI, you know, it's the future. I want people to know that, you know, it's here to stay and it's a good place to be right now living. It's where, you know, a lot of AI from autonomous vehicles to, you know, refrigerators to appliances are all getting AI and machine learning involved in them. So I would like for everybody just to prepare themselves that,

you know, the whole adventure that we see in movies and that future that we see is actually becoming a reality and is a good place in life to be. What's a tip you think everyone should know? Don't, don't never be afraid to learn anything, you know, no matter your age,

you know, gender, sex or anything, don't never be afraid to pick up a book about from from ai to being a great chef you know just don't ever be afraid to keep learning when do you think the singularity will happen as you i think you know give it truthfully about 10 10 more years you're ready start becoming a lot, at least from the things that I work on, you'll start to see it blend. You'll start to see things go a lot public a lot faster.

You'll see that click happen. Okay. What's your favorite evil AI? Skynet. What about hero? AI hero? There aren't as many. If I had to pick an AI hero, I would probably say Jarvis. Yeah, that's the one.

Or Katana. You know, you got to love that Halo. You know, I grew up on that. Okay, so deep learning and machine learning and artificial intelligence. Could you summarize everything I need to know about that? Jeez. Okay. Yes.

Machine learning, you know, is more traditional types of way that machine learning. Deep learning is more using neural networks to learn. AI is a realm of all of those things packaged that allows a agent, you know, not necessarily how to be a physical robot, but an artificial agent or entity to be able to learn and adapt itself based upon input either from other software-based applications or the human input. So in a the summary it's basically artificial learning of some type of behavior or need to complete some type of tasks autonomously for a long time we've been hearing about artificial intelligence and then machine learning and deep learning and mostly neural nets um that's part of deep learning, right? Yes.

They kept getting hyped for like, I don't know, since the 90s. I've run into them and then it doesn't work and they're overhyped. But now, now they actually are doing things. What changed? Processing power. Hardware finally caught up with, you know, with the things we can do. You know, it's just like with our mind is one of the greatest things you know ever invented so we're able to process things a lot quickly and a lot faster that we don't think we can but we can it works very good

the problem was a lot of neural networks needed the processing power and back then we didn't have it because if you look at how a resnet a you convolutional neural net, and how much of GPUs, of tensor TPUs, or if you go on Intel, those processes, how much is needed just to do something like object detection. Even though we take it for granted how quick we can identify a motorcycle, a toy, or something. It takes a machine, a lot of processing power to do something like that. And so we just get into that niche where we're able to utilize the hardware, utilize the processes, and it be semi-affordable for people to get it to academia, get it to the, you know, the entrepreneur, and they'll be able to leverage this hardware now because it's not, you know, something that is only at a big, you know, data center mainframe kind of thing. You're able to right now go buy, you know, maybe under $1,000 or about $2,000 to get you a decent system that you can

actually train a you know a neural network on and there's training and then there's the actual doing the inference yes and those have significantly different processing requirements. Correct. Do you ever think about, you know, training takes a hundred times inference or is there some metric I should start looking at? It depends on your data. You know, deep learning is data driven. You know, data science, it's all based upon your data. It's how much data you have depends on how much process and you have, how good is your model can train. Like when you're dealing with what everybody's

dealing with now is the whole image processing. That has to have a lot of data. It has to have a lot of differences data so it can find it, so the network

can find the differences and be able to identify the different things that it needs. The problem with inference, all I'm doing is taking that model and I'm applying it in an application. So I don't need that much processing power. Where it gets tricky, at least in my work, or where it gets a little interesting is being able to retrain on

that edge or on that device that only can support the model pre-trained. So now you have to be able to try to build a better system that can not only input the trained model, but also how it can leverage and kind of retrain it and update that model in real time on the fly. That's when it gets a little, you know, hardware intensive, kind of, so to speak. I was fascinated to find out that there's this thing called transfer learning where you can have part of a model and this retraining on finer grained information. Can you give an overview of how that works?

Well, I can explain in a scenario that we kind of use or I kind of use on my time and I can kind of explain it that way. For instance, let's say you want to identify all your sports team jerseys from hockey, basketball, football, soccer. You want to identify the jerseys. So what happens is your first model will be able to identify, let's say I want to identify the difference

between a hockey jersey and a football jersey. Similar in size, it's almost, may have the same type of manufacturer and kind of brand. A little different in places, but similar at the same time. So you're creating that work that's able to be trained on a lot of different jerseys from professional to high school to peewee.

And it'll find a difference between the jerseys. Now where it gets difficult and where you need that refreshing, that training, is when you start to want to identify specific teams per jersey. So let's say we identify a football, a football team. So we know it's the same. We're going to say the Seattle Seahawks. You're just going to throw it out there. Not my team, but it just came in my head for some reason.

You'll make somebody happy. Exactly. So let's say that. So, now we need to not only identify that it's a football team jersey, but we need to identify that it's the Seattle Seahawks jersey. And not only that, they have home away alternate. So, when it comes to those type of things, that learning comes into play to update the model based on specific techniques and how to identify those things. Some people use a subnet approach where you go down one network. Boom, let's identify that it's a football jersey. Let's go down another network. Boom, let's identify that it's a professional jersey or collegiate or whatever.

Then we go down to identify what team what home away alternate so by updating the model with new information and one thing that you got to make sure that is the new information that's coming in is correct information because you don't want to false identify things in your model that's going to give you false positives or you're going you don't want to false identify things in your model that's going to give you false positives. Or you don't want to identify something as being a collegiate jersey as a professional jersey. So there's a lot of different steps you have to go to to get that from a list, a pile of jerseys. OK, this is the Seattle Seahawks home jersey. I say that because it takes even though we can look at it and know off the bat, hey, this is a Seattle Seahawks home jersey. I say that because it takes, even though we can look at it

and know off the bat, hey, this is a professional team jersey. It's just based off of identifications on it. What brand? We know that Nike sponsors all professional NFL jerseys, so that's a cheat almost for now. At one point, we know where the logos would sit at. You know, that's also a cheat, you know, and that can be trained as well in the model. We know where the numbers sit.

We know where a lot of design paths and things that are unique to the Seattle Seahawks would reside on that jersey. So these are things that the model will learn. And so as we throw in another variant of that jersey that it didn't know, but it has, let's say your model sees this same jersey, but you know, see how colors are like blue, which is green. Let's say it's red. Just somebody just made a random red Seattle Seahawks jersey. It's going to give it most, if your model model is good it's going to be able to understand that it's a seattle seattle jersey it's not the correct colors but it's going to give you like some type of it may think it's 60 sure that that's what it is but maybe but maybe that's their new throwback jersey or something like that you want to incorporate that into the

model that's where you're able to start updating the model with newer things coming in. Another area I heard about this updating the model on the hardware is with cars. Correct. You do the self-driving car. You train it to drive according to the rules of the road. And then you give it a little break-in period where the owner drives and you learn how the owner prefers to drive the behavioral cloning style transfer of information so you're trying to to drive like them i mean like i've driven in cars where people like to be a little

heavy on the pedal and then they it's a binary thing their pid loop is bad on their foot but uh and they like to to go and fast and then slow and then fast and slow but maybe the original uh neural net was trained to be more smooth and so they learn that last little bit of how this human wants me to drive um and i'm always really impressed by that. Okay, so that's some of the things we can about neural nets and hardware and inference. But other than self-driving cars, how do I use a neural net to do real things? I mean, object identification, okay, I get that. But how does it really work in the real world in my opinion deep learning is just a piece of the entire ai it's just one part of it there's so many variants of machine learning and artificial intelligence that now because deep learning is the thing deep learning overshadows them but you got to remember there's a lot of vision techniques

you know that are in my opinion that work just as good as a deep learning technique that what makes deep learning very you know cool or very you know is that it's able to do a lot of things quickly that took longer with other different types of algorithms and approaches. If you mean where I feel like deep learning itself will not solve the

problems. It's an aid to help the AI in general. For instance, if you want to say, hey, how would deep learning get me a self-proficient maid in the house or a Rosie, per se? Rosie the robot, yeah. Yes. Let's say, okay, there's a lot of social, emotional types of AI that will have to be applied to that robot as well. Deep learning will aid in certain algorithms and certain techniques as far as, like you say, image detection. Probably have sensors that will be able to detect smoke, detect carbon dioxide, you know, or something like that. So she can warn you that, hey, something's wrong or something's

going on. And, you know, that can be trained. So deep learning memory is very, very data driven. It's very, very historical and new data coming in driven. So that's all it's really, in my opinion, you know, doing is just basically learning from data and adjusting itself and making that decision based on the data it's given. So there's a, in my opinion, it's only an aid to the bigger scope. It's not the entire solution. So this adjusting to data as it comes in, that was something I heard about with Kalman Filters like 20 years ago. Exactly.

Is that, I mean, is that considered machine learning and AI or is that just, I don't know, navigation and controls? It's a little bit of both. You know, you're able to add the data to the database that you already have. You're able to properly identify if this data is good or bad or is it something that can be leveraged? A lot of things with incoming data coming in is that your AI, your algorithm is going

to fail on the unknown. If there's data coming in that's unknown, it's not going to know what it is. So the idea is to be able to take some of that data and incorporate it into your network. But beforehand, that data coming in has to be justified, has to be verified that it's good data coming in. Usually that happens from the human analyst or some sort like that. In my opinion, AI would never be or some sort like that. From, in my opinion, you, and this is,

AI would never be standalone. It would never be something that's per autonomous. You're always going to have some type of, you know, interaction with the human just because, you know, the fear of Skynet,

and also the blame. Everyone's going to have to take the blame when something bad happens in AI, you know. And so it's easier to design your algorithm that incorporates the human input. So if something do happen, you have a physical human to blame. You know, it sounds bad saying it like that. But as we get further in this detail let's we talked about autonomous driving let's say that that car's driving by itself it doesn't see the dog the child and it veers off and hit them randomly who do you blame at that point

do you blame the driver because the ai was mimicking him. So is the driver to blame or is the person who actually created the algorithm that, that models the human to blame? Or is it the car company or is it the AI company? It's the neighborhood designer. The neighborhood designer. And I mean, it's not just one person who did the software. Exactly. With all this training and all this data, is it the person that collects the data?

Is it the person who retrains? It's an impossible question, and it makes nobody to blame. And that's a scary thing. If you have to think about it, Bosch is usually to blame. The censoring company? Yeah, sure. And that's another thing, too. You know, you have to have AI to detect when your sensor's going bad.

You know, you have to, you know, is this car stable with only two of the three operating? You know, those are things before you take off from your garage, from your carport, from your parking spot? Do you have to your car have to do that? You know, a 10 minute, 30 minute check before you leave the house? I hope not. so let me ask you a kind of a variant of the question that alicia asked which is there's a lot of hype around machine learning deep learning right now and it's very exciting nvidia is selling lots of gpus and things do you see it being applied in places where the traditional methods are actually superior and people are just kind of oh it's deep learning

let's just apply it to everything. That's happening now. Yeah. So where is it the best place to apply it? Yeah, it is. It's becoming, like you said, it's becoming, in my opinion, it's becoming sometimes a little overkill. Okay.

Where it's a good selling point. It's a good thing to kind of boost up, you know, like I was stating before, like you said, there's so much research being done in computer vision, so much research being done in a lot of these areas where, you know, a different approach. What do you really benefit from using a deep learning approach? What's the gain other than the fact that it's the new the new, you know, the fad right now. I gain points on my resume.

That's what it feels like. It's pretty much like everybody like every resume I see coming in, someone is using it. Someone says it's on there, you know. But then it's also, you got to understand that the really

thing that's cool about it is that you can prototype something really quickly which is great which is which is awesome but then that comes with the even bigger problem that i don't know if we're going to get time to talk about what we can is applying these things to a real life system a real life architecture something that's not using ubuntu something that's using like a real-life system, a real-life architecture, something that's not using Ubuntu, something that's using like a real-time operating system, something that's a life-or-death matter, something that's time-sensitive and critical. You know, we can run the greatest algorithms, you know, on these machines, but if we can't put them on an actual system, then it's like, why are we even wasting time?

Okay, that's a great segue. We can put them on the systems. I've seen inference models go on Cortex-M4Fs. And they were, I mean, they took three days to train and then they said, okay, use these features and these weights, and off we go to the races. At the time, I was pretty annoyed because those features didn't make sense. I mean, you don't do FFTs that way. If you want to have a 512-point FFT, you don't then just use the first 256. Back on topic. Back on topic. Back on topic. Okay. So we do, and processing power increases alarmingly every year.

Less alarmingly than it used to, but. That's true. And yet I do feel like we aren't actually putting these things in devices because they aren't that reliable. Do you agree? I agree, totally. I, from my understanding, as far as what's a real platform, for instance, based upon what you get from NVIDIA or what you get from, you know, I don't think there's a true path to application.

Just for the simple fact that the requirements needed to run this neural net is something, in my opinion, is not defense ready. It's not space shuttle ready. It's not spacecraft ready. It's not aircraft ready. It's something that's still in the research area. Yes, don't get me wrong wrong you can throw it on some on some cots you know commercial off-the-shelf products or something like that you can create something but when it comes to validation and verification of this newer network that's not

that's not you know that's not officially been there's no standard yet for these type of things. And that's the problem when it comes to putting it on a real application. You know, what, and I'm not even going to talk about, you know, classified or those type of applications, because the OS alone has got so many flaws in it that you can't even get that on the system. So when it comes to, in my opinion, when it comes to real application, it's just not there yet. Just because of the development tool they give you and then that path to actual, it's just, like I said, people throwing Jetsons on payloads, people doing things, you know, but it's still all, you know, experimental. It's not nothing that I can package and start selling you and it's going to work every time guaranteed because you're right. There's flaws in the network. There's flaws in the incoming data. If you're trying to upgrade,

update that network, there's a lot of things that still need to be tested. I'm, wow, I'm happy to hear you say this and slightly disappointed. And you mentioned throwing Jetsons on payloads. One of the things as you started to talk about, it's not there yet. I was like, well, what about Jetson? Isn't Jetson cool? Even as I use Jetson a lot and sometimes like it and sometimes find it infuriating. Thank you. I know you have a fair amount of experience using them. Why, without totally bagging on NVIDIA, because they don't deserve it, they've made a lot of progress.

Why aren't they ready yet? Is it just the testing and the observability and the figuring out how to test things that aren't deterministic? I feel like hardware wise, Jetson is a great platform. Like I have a couple of them. It's very good. Software wise, it is horrible. Thank you. thank you i mean like i i can tell you right now like the whole using the jetpack to install i understand why you have to do it but it's just not it's not practical so you're telling me not only do i have to buy your your machine but i have to have a ubuntu based host with an nvidia gpu to leverage all your tools to even work on the Jetson. And then randomly you get the Ubuntu errors where your desktop start working,

you know, your Ethernet cards are not working. And so and that's what I mean by being able to leverage that hardware just because I can't live demo something. Oh, I can't SSH into the Jetson because it's having a problem. And so now I have to reflash it because that ARM-based Ubuntu architecture is not working. And that's what I mean right there. That's not ready. Jetson works 80% of the time, 80% of the time. So that 20%, that Murphy's law happens every time. When? When you're demoing something, when you're trying to show something to someone.

And so then it just makes it more frustrating to use it. Don't get me wrong. Because it's so close. Yes, it is. Exactly. And you use it. It's like missing all your putts in golf. And then you finally get that one hole in one that keeps you going.

That's what it does it it you know it it gives a little error here error there error there and then boom it gives you that green light that it magically works for that one time and then it keeps you going back in it's it's so nice to hear you say this because i thought it was just me no no like i say this is a i went to you know gts my jetson that i was in training crashed on me in the with them so it's so it crashed to the point where it took three of them to figure out what was going on and that's what i mean like as when you look at it from the outside in or you look at it on paper it is is a great piece of hardware. It is. And people make little carrier boards for it. It is an awesome tool. It is not the hardware that's the problem.

I feel like it's the software capability is the problem with the Jetson. It's that I need something that's more of a Red Hat variant. I need something that's more of a VX or a Red River variant. I need something that's a of a VX or a Ren River variant. I need something that's a little more stable. I understand why they went Ubuntu. I understand. I do. You know, it's free.

You know, free enough, at least. You know, people are used to it. We can do things. The problem is it's just not stable. And that's just the God honest truth. My Jets at home right now, the desktop's not even loaded up on it for this one apparent reason. I probably have to reflash it. Just random. Things like that is frustrating to me at least because I use multiple Jetsons,

multiple drones, multiple ground and air vehicles showing off this stuff, but then you have to always secretly hope that everything works right. And like I said, no one doing experiments with Murphy's law, you're going to have at least one out of your five Jetsons is not going to be cooperating correctly. And then that's one you have to fix. All right. All right.

We should stop talking about Jetson because it is a little tempting to just keep going on them. Yeah, like I said, I love it. But yeah, it's really useful. I am a big fanboy of that product. I'm just, like I said, with anything, you get frustration. And like I said, I don't blame the hardware. What it's doing, a lot of companies are not doing.

So kudos to that. It's just frustrations. That's all. And they do try to help beginners to some extent. They do. Don't get me wrong. I got them on speed dials. So when it comes to getting the latest, you know, alpha version of a jetpack, kudos to them for helping me out. No issues there i'm just saying sometimes it do get a little frustrating when i'm in mid desert you know area doing a flight test and something's not working but with the latest you know tx2 you know

yeah tx2i that shouldn't that's a little bit more you know outside ready kind of a Jetson. So I'm kind of looking forward to utilizing that one a lot more. Okay, switching topics a bit. I've done a little bit of AI, mostly taking classes at Udacity and reading books. And I found it all very interesting. I did everything from computer vision to Kalman filters and particle filters to neural nets and vision, mostly with the vision, but neural nets in general and behavioral cloning, all that. But then when I am faced with the problem on my robot or a problem outside the classroom setting,

I can't figure out which of these tools will work on this nail. I get so lost in there's not enough data for a deep learning solution. Maybe I'll just go back to filtering the noise really hard in a PID control because I know that really well. How do I figure out how to start applying these more intensive problems to my own work, even if it's not ready to go outside, even if it's just I want to start learning how to use them, not just use them in class? That is a good question. And this is how I, I guess, in my opinion, how I take a stab at that approach. The first thing is a lot of people, fair to realize or don't fair to understand, is that tools like MATLAB, Python are sometimes a handicap more than a benefit to you because they're able to, you properly can understand how the algorithm works.

But sometimes it's going to be a little difficult to convert that to a platform that's able to utilize all of those things. For instance, most people use TensorFlow or Cafe and most people use Keras to run on top of that and TensorFlow is the back end. But most people use that in Python, correct? Yes. So you create your model in Python. Most

embedded is C++ driven heavily. 99 99 of it from yeah and most of them use a real-time operating system screen hills vx words you know remember you know some you got got red hawk now which is a you know a soft real-time version of of, you know, something like that. So a lot of people just want to apply the algorithm. I think you also have to look at the integration part, because that's the biggest part of the whole system. How can I even leverage this technology? Now, NVIDIA does a great job with

TensorFlow.js using CUDA, because that's still C++, and you get to incorporate your TensorFlow model into it, your CAFE model or your MXNet model that's coming into C++, which helps great. So that's how you're able to do that. You can also recreate a convolutional network in CUDA if you want to. I wouldn't recommend doing it. It's just easier just to use Tensor IT. For instance, particle filters. You talk about particle filters, common filters, and things like that.

Most mechanicals or most people I have worked with, they do a lot of MATLAB modeling first. It has a lot of different functions, a lot of features. You can create your particle filter. You can see everything. We use that now. We do a lot of those things now with a lot of robotics that we have. The problem is converting that to C++. You know, MATLAB has a little export tube, which is not good at all, which is the worst thing known to MATLAB history.

I don't believe it exists. That shouldn't even be there. They'll get more credit taking it out than keeping it in in my opinion because it just does so much extra stuff that's unnecessary so you end up having someone who knows the math then you have someone who knows c++ recreate that algorithm for you which i think in that sense that's what you need to look at first what platform what code baseline you want to do this in. And that's how you're able to leverage which type of AI you want to use. Because if I can create an autoencoder,

then I can get a better outcome than trying to write some type of C++, C Sharp, the Python wrapper, and try to bring in something else like that. I was just going to use what I already have. And I think what you talked about is that whole instance where people feel like you have to force deep learning into the application. And I don't think you should look at it like that. I think because we already proven

that some algorithms that we already have worked a lot better than the deep learning approach and so if it ain't broke why try to fix it kind of thing but there's a lot of applications where deep learning takes an advantage of a really good advantage so and i and I think if I was started from, I want to build this cool robot, I'm going to go what language first? Because I need something, and I need to know what hardware and what language it supports

because I want my entire programming language to be that. So if that's C sharp, you know, and that's.NET, it is what it is. I'm going to find everything and every type of way to create those algorithms that I can utilize and leverage in.NET. I will make sure that all of my sensor information and all the comports and all the things I need to read in is in.NET. That's what I'm going to do. If that's if that's the case, if I'm doing more C++ variant, I'm going to make sure that the hardware supports that. If I'm doing more PyScript-y kind of basic, I'm going to see that the hardware supports that. Some things do, some things don't. I feel like you have to look at it that way first, or you're going to be pulling your hair out and screaming at yourself at the end. Okay.

I understand that advice. I am more on the point of if I work with the algorithms PhD guy who wants to write in C++, I can do that translation. I've done that in a couple of jobs. I find that work pretty fun, actually, because I get to learn about the algorithms. But as I'm learning about the algorithms on their own, what you're talking about, that part doesn't phase me. It's more, I have all these algorithms and I don't know what to do. So let me give you some very specific examples that will sound slightly familiar to long-time listeners. I have a cheap, and we're not even going to say inexpensive, we're going to say cheap robotic arm with some position problems in that it only goes where I tell it to within a little bit. And I have an off-the-shelf webcam and I have a Jetson TX2,

and I want the robot arm to type. And I know it can do this because I can successfully make it type using very specific go here, go there, push down, go up, go down. Sort of distinct kinematic moves. But I want to do something smarter where I can ideally conquer the cheap mechanics with superior smart software. And this is a learning project. It's not for sending to space. It's not for doing... So far. So far. But it's really about understanding how to alleviate crummy hardware with fabulous software.

Cause I think that's an area that AI machine learning, deep learning can all help us with. Yes. I totally agree. But now I have, okay. So I have the arm. I know where it's located because I'm doing some neat computer vision.

I know where the keyboard is located in the real world because I've hand done a picture of the keyboard and I can use the computer vision homography to find exactly where that is on the camera. So I know where each key is and I know where my robot is, but I have them in different frames and I have them with different amounts of uncertainty for each thing. Like when I move the robot, there's some uncertainty. When I measure the robot, there's some uncertainty. And I kind of know how to use, I mean, a Kalman filter, a particle filter, that's what you use them for. But I don't, there's a gap.

There's just a gap in how do I go from here to there? Help. Now, from my experience, if you need help with your uncertainty and you can't get more hardware, what I would do, how bad is the uncertainty in certain areas? Um, a human can fix it. A human can, can fix it without too much problems. So, um, centimeter, a centimeter is kind of what I like to think about, but it's part of the problem is that in the beginning, on boot, you can make it all work with fixed parameters. But then as soon as it gets warm, the fixed parameters go out of spec, and so that laborious calibration does not stick. And so the calibration needs to shift. And I'm not sure how it needs to shift other than it works for a few minutes and then it stops working because the keys, well, I mean, it's not like the keys move.

The arm doesn't move the same way. Loosens up, yeah. And, you know, this is a very specific example, but this is sort of the thing, like, okay, I have all of these things and I just don't i mean can i can i apply a neural net to this yes you can um i actually was thinking of one when you were talking um what what you could do is and this is you can do a train set set point you know you know a lot of things you know how long it takes before your actuators and your motors get loosened up so what's going to cause slippage which is going to cause you to be away from a certain key and you could what what i would do is um based upon i guess say 30 minutes of of of time and you know okay the arm is pretty warm i don't know how many i guess finger types of pointuses your arm has is that five four three two or one just one okay which which

is even even easier yeah what what i would do is um i would get it get it get it to a a a good temperature you know a good warm day okay everything's loosened up greased up and ready to go what i would do was um do a study on how correct can I press the letter B. I just say B because that's what my name starts with. And I'm looking at a keyboard. So I say B. OK. And, you know, from the start position. And, you know, you got all your sense information. So you're reading all the information you have and it falls down. Press B. And let's see how many times it correctly presses B.

And, you know, is it pressing B? Is it pressing B? Or is it pressing N? Or even a space bar? Space bar. Yeah. Gs and space bars in that case, usually. Okay.

So with that data, we now know that when we do our normal command, we're variant off by, we're going to hit G, this certain consent. Now we can apply a adjust. When we know this, we know it's going to, let's say it hits G a lot. Let's say it hits G in the 80%.

Like we start from, we try to hit B. We're hitting G 80% of the time. So G is now where B should be in the mind of the AI. So we know that. We know at this temperature of this machine, we know at these radius calculations, at this point of each motor, it's hitting G. So we need to adjust it, like you said, by a centimeter or such to hit B. Why not let the AI do that for you? We already know it's going to hit G based upon the training. So we have trained the neural net to take in that information and press the button, and we know 80% of the time it's going to hit G. We know the temperature it's hitting it at.

You know how you say everything's been on for about 30 minutes, so it's been warmed up, ready to go, loosed up. We know each point. We got all the sensor information coming in. So we can use that to our advantage now. We can apply that to the net to figure out when it's hitting G. And then when we know we apply the AI to do that, we can adjust that centimeter or such automatically so it hits B. Based upon we know when all that information come in, it's likely to hit G instead of B. This is where simulations become so important. I mean, because what you're saying, I would have

to run my arm for many hours in order to gather this data. And by the end, it would be in a completely different configuration because it self-destructs on a regular basis. Did I mention the really cheap hardware? Yes, and that's another thing. But sometimes, and that's another thing when it comes to these neural nets and it comes to people who work on actual hardware, you know, seeing it in the sim, it works great every time. So, you know, your algorithm works great, you know, but actually when you got slip rich, like you said, you have actuator, you know, gear slip rich, you got things that's happening that you don't account for in the algorithm. Yeah. But in a simulation, if you, you can do it for one letter and then you can start to read, look at the, how it changes and, and do your learning there and then see what the difference is.

And it goes back to. You have to be good at modeling your sensor. You have to be good at modeling the arm, got that modeling the arm which i can't do you have to be good at that because you want it to you want it to have the same flaws as the actual hardware does so you will have to be able to know that okay this arm start to offset when it's 30 minutes out and it offsets by this many centimeters you will have to apply that to your sensor model that you're training this algorithm on. And that comes another issue with deep learning is comparing actual data to synthetic data.

And that's another thing that is another topic. But for your issue, you can either A, you can, since you can't train it, but you physically know and you probably measured how far it gets off at a certain time, you could just apply that to the network without training. That could just be another layer that you apply this offset to. Because it's something that you know. It's a known. It's a sort of known. It's a known for one key, but not all of them. Or it's a feeling that, oh, on that half of the keyboard that doesn't seem to go as far early in the test.

I feel like if I knew all of these things to the specs that I could put them in a simulator, or even what you're saying, I wouldn't need any of the intelligence. I could just put them in and not, I mean, hard code in tables. When do I decide to use a fancier algorithm? If I need to have all this information up front, I can just use the information instead of fancy algorithms. You're kind of asking, when do I apply which algorithms, right?

Absolutely. When do I apply which algorithms is really part of it. And that's the piece you're kind of missing in your education, you feel like? Oh, very much. Okay. I feel like your issue is more of a vision problem because... Yeah, me too. You physically can see when the arm is not on B. So if you can just map that to your vision, then you can just adjust the arm accordingly. Okay. If I know I already made this filter to hit B, but it's a little off, I have to adjust the arm. That's the key. You have

to adjust the arm before it presses. But since you do use vision, you could see, oh, it's on G. Let me move it down. But then I guess that applies to the biggest issue now is how much slip rich do you get in your gears to when you can, if you go down a little bit, is it supposed to be just one centimeter down and it kind of slides to two? Yeah. That's, you know, that's where you have have to look at you know how much power you giving your actuator but that's that's even then that's not really an ai question that's just more of you knowing the hardware itself and it can it can because what's going to happen is it's going to keep applying trying to hit trying to hit b and it's going to slip away because it's probably going to apply too much now you can teach it that, okay, you can make it lower how much power is given to the motor

so it doesn't slip as far. So then it can eventually press B. But even then, that's not a deep learning question. That's more of just vision and seeing where the actual arm is and knowing which way to adjust it to. I was hoping the deep learning part could come in where I can do one key and I want it to smartly decide similar things for other keys. It could with knowing that your arm is slipping at this much based upon one key input. It could.

So let's say say and that's another but that's another thing where you need training because deep learning is big on training data it's big on how much data you need it would it would need to know okay we're at the point now where our arm is slipping at this many at this many centimeters left right up and down this is keep it easy left right up and down so we know that to go one centimeter we have to give it this much percent of power because if we put the power that we usually give it it's slipping now it's going to go up so we so we have to drop the power so you would take you identify when that happens with your vision. And then you can apply that, you know, that slip ridge function to all your movements now.

But detecting that function, you can use deep learning, but you will need to identify when that feature, when it happens, when that slip ridge happens. But it's a lot easier just to do a vision because you can physically see it. Yeah, my vision has errors too well i think the answer is the book says you can have noisy measurements and you can have noisy movements and it all works out it all works out asterisk if you apply tons of work

which is i think the thing you're trying to avoid. Yes. So there are times when it is better to just go with classical control theory and not try to solve everything with AI? Yeah, I believe so. I believe also it's easier to sell the classical approach than it is the AI approach, because it's easier to validate and verify the classical approach than it is the AI approach. You know, people want AI. Most are afraid of it. How do we get to the point where we can test it? I mean, that's one of the reasons I'm afraid of it is the testing teams. Trust. It's all about trust.

It's all about putting a little bit in until you get to learn it. How many people hated Paperclip? Hated it. You know, I don't know a person who loved Paperclip because they threw it at you.

They threw it all in at you at one time you said go away he came back but let's say they would have gave you a little bit of it at a time until you needed it all and they learned the personality of the user first hey this person's more of a dominant type of you know personality they don't they're not gonna let's let's make paperclip be more of a dominant type of personality. They're not going to... Let's make Paperclip be more of a robin to this person's Batman, to his personality. So let's let it aid it more versus just all in.

But let's say someone who doesn't really know tech want to help. Then understand that and then bring it in now. At the same time, Paperclip came a point in time where that technology wasn't there. At least it wasn't consumer-ready or plausible

to be in someone's home. So, you just got all paperclip at one time. And then you kind of felt some type of way. And then most people disabled paperclip, like with the

quickness. But at the same time, you trust, you trust autopilot with your life. Literally, everybody does. We do. Yeah, you trust it with your life and all the people around you too. Exactly. But at the same time, it's because we know there's a pilot there. There's two, just in case. I think what I'm getting at is that we have to get baby steps. Yeah. You have to be able to trust a little bit of it first, then to get immersed in a lot of it. And also, you know, it's crazy

because to say this, that I can get in a car accident with a taxi cab or with a Uber driver, I'm going to still drive a Uber. But it's only going to take one time for that AI car to hit a pole that's going to just drop everybody from using it. Yeah. And yet, there is goodness. It's hard to go back and forth. It is.

And to do this testing rigorously. I mean, we used to be able to test things rigorously. And now you kind of hope and pray? I mean, it's not the right way to go. Are there pathways forward? Yes, there's a lot of steps into validating and verifying AI, not only from older approaches, but also the newer network approaches. And that's what I mean by breaking down the requirements and breaking down the decision that the AI is actually making.

I think sometimes people look at deep learning as a catch-all, as if we just throw all the data in, it's going to pop out an answer. To me, I feel like it shouldn't be leveraged like that because that brings the unknown factor like times 10 i think we need to you know what kind of you have to take that watson approach almost it's not really it's an aid more than it is you know control so you know and that's why a lot of a lot of deep learning is always used for identification you know detection something because i can give it a percent that it thinks is positive or that it thinks is it and then I can back it up with actual data. There's a lot of things that you do

that we start to look into where you get an analyst to also check the data as well. Let's say this decision made okay i think it's a blue car i look at it it's a blue car and it's a ford it's a it's a blue ford okay and i it's a 90 that i think is this based on you know the imagery data based on you know you know lighting and stuff like that you know it's a blue ford car all right. So let me look at that. You take a chunk of your of the runtime and you run it through an actual expert of vision, which is anybody who can see and know what type of card is. They check off. Yep. So not only is your your algorithm running at 90 percent,

you also have this human factor that that verified that the algorithm is working accordingly on a certain amount of data. That's one way you can look at it, even though you may feel like, okay, that's maybe random, but then you also have to check certain, you know, and this is in your instance, when the hardware starts to fail, like with vision, dead pixels, you know, clarification, fall camera, bad camera, failing camera. Do you even let the algorithm go at that point, or do you just say system failure?

But sometimes, let's say from an aircraft or from a fighter jet that has got damage from a camera now can't see, what do you do then? Do you stop the AI from working, even though it's working at 70-80%? Because it still can identify something, but do you feel

at that point in time comfortable with a damaged hardware and knowing that your AI may give you the wrong answer? Or would you rather have no answer at all well at this point i just want to hide under my bed and not interact with any ais because clearly they're all out to get us but i'm just saying as we look at it we you know ai zone is good as the information and as the data that is. When we start to have faulty hardware or bad imagery or something like that, there's AIs that's able to adapt to those bad imageries. But when we start to get

false data coming in, you're going to get a false answer coming out. But that's another step of AI. And that's what i mean by you have to not let ai do it all just do a little piece and keep the human in the loop until we can trust and understand and be able to come up with better approaches and ways to make something fully autonomous yes i can make a robot drive around my living room and avoid objects, but would I put my baby on there and let it drive around and avoid objects at like 30 miles per hour? No, because I don't even trust the AI enough to allow that to happen. I trust it in the house when it's just going like

whatever, you know, speed, you know, turtle speed and just avoiding objects. But I'm just saying as we go forward, we can learn to trust and learn to, you know, I think you hit the nail on the coffin when you said deep learning is just not mature enough like all the other approaches are. Once they get to that maturity level, it'll be more trustworthy. I think it's nice to be in a world where, you know, the next, you know, high school to, you know, tinkerer to retiree can go pick up some stuff and create a little deep learning system and may come up with the greatest algorithm known to man. and we live in a world

where it's affordable to do so i just think time has to catch up with all the different ideas that people are doing because deep learning went from you know properly creating um 3d printing something to now aiding you know robotics in in surgeries to obviously trying to be a self-driving car to a lot of things. I just think that the actual approaches has to be mature because we went from a full five-layer network to now you got ResNet, you got all these different types of approaches, but they're emerging because you know the technology is there it's available and i think we just have to wait

for things to mature first before we can get a proper understanding on which way is the best it's interesting to hear you say that and i so agree that we are throwing this processing power that we barely understand at problems we don't thoroughly understand and can't test around. So, yeah, it's moving a little fast. Yeah. And not just because I want to do it and I don't thoroughly understand it, but also because I truly worry about testing and verification. And yet my next question is if someone wants to experiment and learn how to do this, do you have any advice for beginners?

Yeah, yeah, yeah, yeah. I always tell people, you know, and this is the truth, get you, because most, most, most, if you get a couple of books, I know because my second language, you know, I wish I could count as a foreign language. You know, when I was in college was on C++. So I always recommend using that one first, because if you can understand that, you can go up or down you know you can go to a more less memory managed language you know you know c sharp java that's more you know on on that with garbage collectors and dot net framework or you know the runtime or you can go more even more down you know to c base you know if you if you want to dabble in VHDL or the assembly language, you can do that stuff as well. But I would say learn, learn program language, you know, get familiar with how that type of stuff works.

There's a lot of cheap, a lot of inexpensive commercial off the shelf products. I personally love what Lego is doing because you're not only are you able to use a LabVIEW approach, you also can use .NET and you can also use C++ or C Sharp in.NET to program those as you get. And I say if they break, they're Lego pieces. They're very

easy to replace. They're not that much. They're very inexpensive. You want to always if they're teens, you can go Raspberry Pi. You can go more of that. It's just that for me personally, a Raspberry Pi is not going to work well with someone that's 10 and down or someone that's not 15, just because it's going to end up getting juice or something spilt on it,

something weird happening, and then now it doesn't work anymore just because it's just an open board. That's why I like Lego, Rick. Very doable. It's Lego, so people like it, and they actually get to learn real programming language without even knowing. Even if they just do the lab view flow code approach, they're still learning how functions and things work in the world of robots. Now, as far as deep learning, you know, most, you can even get a, if you go in the world of NVIDIA, you get a

low, about 2010 and up laptop with a decent NVIDIA card. You can learn. You can run some stuff.'s gonna i would you want anything that you're doing at home you're not going to be having pounds of data so training if it takes 20 minutes to hour it is what it is it's not nothing extreme it'll probably take 10 minutes maybe two depending on how much data and most people use online little data little data sets it'll probably take literally like five minutes so you can learn it that way i say go older because you have to sacrifice that computer

and put ubuntu on it just you know you know i say it like that because it is in my opinion it's a sacrifice i feel like i shouldn't have to you know go to that because because i you know maybe maybe because i'm a red hat kid you know grew up red hat centos maybe that's just my flavor of it and maybe i feel some sort of way but i don't i don't know i just feel like that's what you have to do it's a very it's it's a very open source i guess. So and then, you know, just read, get books, you know, learn. Don't be afraid. When I got into this, my sister was actually in college and I learned C++ when I took one of her books. And I got hooked ever since.

You know, I was, you know, I was fortunate enough to go to a technical high school where I was able to always work on computers, get parts and update things. And so, yeah, I was able to have so many of those, you know, kind of things around to leverage from. But like I said, I didn't, everything, you know, a lot of stuff was old, but I was able to get it working. And that's all you really need to do. Like whenever you, if you have an older laptop that you know you're going to get like whenever you if you have an older older laptop that you know you're gonna get a new one don't throw it away throw you know a unit space operating system on it and just learn how to do some um some some robotic types of ai types of work on there and just just learn cool yeah i mean sometimes it isn't about having a list of tutorials. It's about getting what you need and once you get past that hump and you actually get something moving something working that's a feeling that no one could take away from you because i just remember all the time when i get things working i remember when i was doing my my phd

research when the robots finally started working together and picked up objects and i saw it you know and people saw it happening the goosebumps you get get knowing that, you know, the all nighter it took to figure out that one issue that was causing everything to fail kind of thing. And you finally figured it out and got it all working is, you know, that's that's a good feeling. And that's and that's what I will want for all the, you know, commercial AI developers and stuff to be able when when we actually get to the point where cars are avoiding objects really good, you know, no accidents and things are really happening, you know, we can just step out and, you know, into this, you know, AI world with trust, you know, with security and with understanding that, you know, everything's working well. And I think we just, with time, you know, with all the fails, you know, there's diamonds that's going to come out the rough and create these awesome AI things that would change the world.

That seems like such a good point to end on. But I think Chris has one more question. Well, you mentioned books, and I wondered if you had any books you recommend alongside the... Sam's. Sam's Teach Yourself, C++, or any language, 24 hours, 27 days. You know how they go. 24 hours, 7 days. I believe those books give you...

They leave off on a day, and they leave off on an hour. So for someone who's busy like me who wants to learn something really new, it's easier. I recommend those. If you want to be more into the deep learning, CUDA. CUDA is a must if you want to apply, if you want to take that next step from a Python base to a CUDA base, Python, but you can, Python tutorials are a dime a dozen online. So,

you know, I wouldn't, but I like the SAMs because it's like, if you are a heavy reader, it's just an hour or it's, you know, it's a day. If you've got a weekend thing, like, okay, I got a Saturday and I can work, work on something about four hours. SAMs to get you through and you make you get your understanding of the things that also sam's comes with a lot of um free software or links to two things that you can utilize like um if you video studio community i love i am such a video studio fan boy it's a shame i get everything working in it regardless if it's going to run on a different platform it all works in Visual Studio the tele sense and everything like that to me is a plus to help when it comes to integration and when it comes to understanding software in general so that's that

you can get you know Visual Studio Community you know you can download that you can get Visual Studio Code which is something that you're able to read and understand multiple languages. That runs on Unix and Mac. You can get that as well. And it reads Python too, which is also cool. All right. We have kept you for long enough, although I still have many questions. Yeah, we can always do it again. The door no, no. The door is always open, you know. I can talk for days about this. So just let me know. Do you have any thoughts you'd like to leave us with this time?

Yeah, yeah, yeah. Since it was more about, you know, embedded, you know, deep learning, like I said, if you're interested in this world, the easiest ways to start researching, you know, commercial off-the-shelf products that are in this world, the easiest ways to start researching commercial off-the-shelf products that are in your budget, find one that feels

like it has everything. Do a little bit more research about it on what type of sensors, cameras, things you can add to it. I always recommend the Lego one because it comes with sensors and they're really expensive and there's a lot of third-party companies that make even more sensors for it so it's really inexpensive and it's legos so it's very resale value is also great and also the hand-me-down or the pass it

along to the next generation is also great because the parts intertwine with each other so i always that's the one I would recommend from starting off, especially if you're in a younger age or if you're in an old age because it's an arm process. You can reflash it and upgrade it. You can also do that.

Just never stop learning. Always apply yourself. You never know what greatness you can come up with from. That's why I tell people, you just don't never be afraid to reach for the stars. That is some good advice. Our guest has been Dr. Benny Lewis, senior research scientist at Lockheed Martin. He's also a content creator and Twitch streamer.

Look for him on Twitch and your favorite platforms as Benjoe02, B-E-N-J-O-E-0-2. Of course, there'll be links in the show notes. And, you know, he's been a great guest here just chatting with us, but I suspect watching his videos is pretty educational. Thank you for being with us, Benny. Thank you for having me. Thank you also to Christopher for producing and co-hosting. And of course, thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm.

One of the things I was going to ask Benny about was this idea of a book about applied robotics that tells you how to use these things on your robot. Because I'm looking for that book and I haven't found it. I wrote an outline of what I want on the blog. So go check out the blog and then tell me what this book title is because I really want it. Or go write it. Or if somebody is currently in the process of writing it and you need a reader. Please. A quote to leave you with this week from Nelson Mandela.

Education is the most powerful weapon which you can use to change the world. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.