Embedded - 208: What If You Had a Machine Do It

Episode Date: July 27, 2017

Elecia gave a talk about machine learning and robotics at the Hackaday July Meetup at SupplyFrame DesignLab (video!) and LA CrashSpace. She gives it again in the podcast while Chris narrates the demos.  Embedded Patreon Embedded show #187: Self Driving Arm is the interview with Professor Patrick Pilarski about machine learning and robotics applied to prosthetic limbs. I have also written more about my machine learning + robot arm on this blog. My code is in github (TyPEpyt). My machine learning board is Nvidia’s Jetson TX2. The Two Days to a Demo is a good starting point. However, if you are new to machine learning, a better and more thorough introduction is the Andrew Ng’s Machine Learning course on Coursera. To try out machine learning, look at Weka Data Mining Software in Java for getting to know your data and OpenIA Gym for understanding reinforcement learning algorithms I use the MeArm for my robot arm. For July 2017, the MeArm kit is on sale at the Hackaday store with the 30% off coupon given at the meetup (or in Embedded #207). Inverse kinematics is a common robotics problem, it took both Wiki and this blog post to give me some understanding. I wasn't sure about the Law of Cosines before starting to play with this so I made a drawing to imprint it into my brain. Robot Operating System (ROS) is the publisher-subscriber architecture and simulation system. (I wrote about ROS on this blog.) To learn about ROS, I read O’Reilly’s Programming Robots with ROS and spent a fair about of time looking at the robots on the ROS wiki page. I am using OpenCV in Python to track the laser. Their official tutorials are an excellent starting point. I recommend Adafruit’s PCA9685 I2C PWM/Servo controller for interfacing the Jetson (or RPi) to the MeArm. Finally, my talk notes and the Hackaday Poster!  

Welcome back to Embedded. I am Alicia White. My co-host is Christopher White, and we are back from our work vacation. Work-cation. Did you have a good time? Yes. We went to various places.

Not only did I talk to many people in San Diego, just lunches and coffees. We also, I went to two hack centers, design centers. Crash, no. So I went to the Supply Frame Pasadena Design Lab. Hacker Spaces, that's what they're called. Gave a talk. And Chris and I both went to the LA crash space

and I gave a talk and he hung up. Yes. Okay. That is correct. But I don't want you all to miss that talk because I know that we, I met some wonderful listeners while we were there. Hello to all of you who came.

I'm really glad you did. And I want the people on the show to be able to hear the talk, even though they don't get to see the demos. I think that'll work. I was hoping you'd kind of describe what the demos were since you've seen them a couple times. Because otherwise it's just me babbling for like 30 minutes. So I'm going to be the assistive description track?

Yes, yes, yes. I see. Otherwise we can just talk through my talk, and I won't, like, give it. Well, that's probably what'll end up happening anyway, because I'll get antsy. Bored and heckle? Well, I wasn't going to say that. I know how this goes. If you don't want to hear this, or you want to see the demos, Hackaday, or, sorry, Supply Frame, the Pasadena Design Lab,

whatever that company is. Nothing to do with Hackaday at all. No, no, I mean they're Hackaday too. It was a Hackaday meetup. They took a video so you can see me waving my hands furiously as the demo succeeded, mostly. You can try to sync this podcast up with that. Oh no, no, no, mostly. You can try to sync this podcast up with that. Oh, no, no, no, no. All right, so my talk was called On Cats and Typing, an Intersection of Machine Learning and Robotics.

And it started out with, you know, an introduction of myself and that I host this podcast called Embedded FM, which I guess all of you probably know, and how about six months ago, we interviewed Patrick Polarski. He was the professor from the University of Alberta at Edmonton, working on robotics and machine learning, but he was in the Department of Physical Medicine and Rehabilitation. So I remember being kind of gobsmacked during that show

where we learned that he worked on intelligent robotic prosthetics and limbs that you could, intelligent prosthetic limbs. And these would be things that were way beyond the state of the art for current prosthetics. If you went and got a current prosthetic now, if you were unfortunate enough to lose your arm, you could get a prosthetic that was human-like. It would be very artistic. It might have hair and moles, but it wouldn't be very functional. The hand would be in a single position.

Moles are important. Hey, they are. Someone with freckles, I find it important. My other hand have freckles. They have to match. But it wouldn't do much. It would just stay in a static position. On the other end of the prosthetic spectrum, you could get one that didn't look much like a human hand, but had dials and levers that could be set so that you could make it turn into something that could hold your coffee cup or that could help you brush your teeth.

So that's sort of what happens now. But Patrick Polarski is working on the next generation, the research on how to make it so that this lever and dial thing happens inside the arm. And we as engineers can imagine that there's a way that we could develop something that maybe an arm that had an accelerometer in it and a clock, and it would be able to tell you, it would be able to go into different motions, different grips based on the time. So you wake up at seven o'clock and the accelerometer says

you're upright. You probably either want a coffee cup grip or a toothbrush grip. We could add a camera to tell where you were and be able to figure it out that way. But that's what most people do. Coffee cup, toothbrush. These are very fixed heuristic based things. Polarski was looking more at non-fixed things, things that you do as a person, not that most people do. He was looking at, I guess the example he gave was this soup, where he was saying that maybe you personally often have soup on Thursdays. And so you go in and you set your arm up so that it can chop for you. And you do that one Thursday at 1130. And then the next week around 1120, you go and you set the arm up again. And it's kind of laborious and annoying. But then the third Thursday, or maybe the fifth one or whatever, it finally learns enough through this reinforcement learning

to do it automatically, to do it for itself. And that's where the machine intelligence comes into the robotics. And at the end of that show, I just wanted to build what he was making. I wanted it to go from idea and research to, okay, let's ship this puppy, because that's where my expertise is, is the ability to put things in production. That's why I'm more of an engineer than a scientist. But I don't have the background to do this machine learning. I don't have the background to do this robotics. I've touched on these things. I mean, I have implemented features and I have implemented machine learning things on various devices, but it's always implement these features and put them together with these weights. I've never

done that part from scratch. And the robotics was sort of the same, where I have done modules, I have done sign commutation for motors, I have done PID loops, I know the same, where I have done modules, I have done sign commutation for motors, I have done PID loops, I know the modules, I know the pieces, but I've never designed a robot system. Not a real one, not an actually useful one. And some of you write to the show, some of you listeners write to the show, and you ask, how do I get into X? And I have to say, I should just cut and paste that email now. Every time you ask, I say, well, what you have to do is learn X, and you have to do X in order to learn X. And if I want to do machine learning and robotics,

that means that I have to do machine learning and robotics. If I want to learn these things, I have to do them. And it's not easy and not being easy. Learning hard things is good. It means that I will be ahead because then other people have to do hard things in order to keep up. So I wanted to do a little bit of machine learning, a little bit of robotics, and had a little bit of time. So I'm waiting for contracts to finish. And that led to the typing robot, which I have talked about a little bit on the show before. Its name is Ty, and I want it to type. You could look up in a makebook, there are a couple of different robots you can make with popsicle sticks and servo motors. And it tells you how to build them and the controls for how to make it type on a

keyboard. But it's very fixed. You have to have this particular keyboard. And if you want it to work on your keyboard, you have to redo all the numbers. It's all hard-coded. But with my typing robot, I wanted it to use a camera, and I wanted it to learn some things, and I didn't want to have to teach it about every keyboard. I didn't want to have to worry about split-key ergonomic keyboards, and I wanted it to work on my clicky-clacky keyboard or your laptop that I desperately don't want to break. I just want it to work on everything. I want to take it to Maker Affairs and Meetups and just set it up and it will work with whatever it has.

So that means I need to identify that there is a keyboard. And that's a pretty standard computer vision problem. And I need to detect the keys. I know that as soon as it starts to work, somebody's going to put a Dvorak keyboard in front of it. So I might as well just plan on using that camera to get beyond Cordy and into the more esoteric things. And then I want to use the reinforcement learning that Patrick Polarski talked about so that I don't have to teach it everything myself. I want it to be able to figure some things out on its own. Now, maybe that's not the most efficient path, but it's what I want to learn. So I definitely want to put that in my typing robot box of things. And of course,

I want the robotics part. We've all seen janky robots that twitch and fidget in ways that just don't look good. And I don't want that. I don't want it to be bad like that. I want it to be beautiful and smooth when it should be smooth and cute when it should be cute. I want it to be good. I want it to be well-controlled. So that's my plan.

And I don't know that I will succeed, but that is my goal. It's always good to have goals. Why are you building a typing robot because it's funny because it's hilariously funny because it's so dumb and yet it has so many practical use uses i mean once you have something that can see a keyboard and interact with it there are a lot of things out there. I mean, you think about pick and place machines. It's not that different of a problem. Or you think about agricultural

systems where you have to pick out good peaches versus bad. The difference between that and typing, it's not as big as it might seem because it's just replacing my target with a different target. Okay. So during all of that, I had a slide up behind me that was just my name and a couple of pictures, picture of my book, a picture of the embedded logo. And at this point, I guess that continues. I don't know why I brought it up at this point. But at this point, I start gesturing. Retroactively imagine a background slide

starting at the beginning. At this point, I point to my Jetson TX2 board, which it's pretty big. It's bigger than a piece of paper. 8 by 10. No, it isn't. The whole board is. It is? Yeah, it's pretty big. It's not half as big as an 8 by... No, it's pretty... No, it's a square.

I've got one right over there. You don't, because you gave it to me. So now you can't prove it. But it's not tiny. And I gesture to it, and I explain. This is the Jetson TX2. It's NVIDIA's hotness for embedded machine learning.

And the TX2 part is just this credit card-sized area where you can see the heat sinks. The rest of it is just pulling out all of its features. And NVIDIA makes GPUs, graphical processing units, and all of the things that surround it. Graphics processing is a lot of matrix operations. When you rotate something on a screen or you apply textures to a wall in your Doom game,

those are all matrix operations where you have vectors multiplying by vectors and matrices and all of that, blah, blah, blah. But I like all my talk stuff, blah, blah, blah, and it just helps. But machine learning is really similar have a whole bunch of features, and then you have a whole bunch of weights, and you have to multiply them together. The faster you can do that, the better your algorithm generally works. So NVIDIA has this GPU system, and they are targeting now the machine learning market.

The TX2 runs Ubuntu, so it's much bigger than my normal processors. If on one hand, way stretched out, you have the Cortex-M0, and then a little bit closer to you, you have the M4F, and then you have the BeagleBone Black and Raspberry Pi sort of in front of you. And the Raspberry Pi 3 is sort of moving off into the other hand. And Jetson's, you know, four times the size, four times the processing power, four times the oomph of a Raspberry Pi 3, which is more than a Raspberry Pi. And then my laptop, which I got six months ago and was top of the line then, is sort of at the other end of the scale.

But Jetson's a lot closer to my laptop than just about anything else in here. Well, that 4X is just the CPU. That doesn't include the GPU part, right? That is true. Which is much faster. And the GPU is where all the goodies are. It's got 256 core Pascal. I don't really know what that means other than people seem very impressed by it.

I'm sort of more impressed by the eight gigs of RAM. What do you do with eight gigs of RAM? Not only does it have the eight gigs of RAM, it's got a memory bandwidth of almost 60 gigs per second. So you have this RAM and you can just push stuff in and out and you want to do images, you just can push them in and deal with them and get them out of here and it's amazing and the whole thing takes about 7.5 watts which is a lot less than my laptop takes and it has a high performance mode that takes more but 7.5 watts pretty amazing more than my wearable cortex m4f or m0 pretty amazing. More than my wearable Cortex-M4F or M0s, but much less than my laptop.

NVIDIA has targeted this towards the autonomous vehicle market, towards autonomous drones, self-driving cars. And some of the features that they demo make that apparent. Were you to get one of these systems, a TX1 like I have, that was gifted to me by somebody who works at NVIDIA, and I should say thank you repeatedly. But if you got this board, or you got the predecessor, the TX1, you would immediately start their demo, or I would suggest you immediately start their demo system, which is two days to a demo. And the idea is that it takes two days to get this to work along the lines of what you want it to do.

Step one in the two days is learn machine learning. Yeah. It takes me more than two days to do that, certainly. What if you had a machine do it? Yeah, exactly. There were many tutorials they pointed to, and they talked a lot about computer vision and neural nets. And even though I took Andrew Ng's machine learning class on Coursera like eight years ago,

most of it didn't make sense. I returned a little bit to OpenAI, Jim, and Weka, which were the things that Polarski suggested, because those were more generalized, how does this all work? But yeah, it didn't take me less than two days to learn machine learning. And even then, I'm nowhere near having learned it. It was two days to a demo started with like a whole week of trying to figure out what I should do. The second step in the two days to demo took slightly more time than learn machine learning. And that was install a lot of the stuff. It installs Th Thanos and TensorFlow.

Oh, sorry, no, TensorRT and digits. Now, tensors, tensors are sort of like matrices, matrices, vectors. This is not something you want to say to somebody who's doing machine learning professionally. But if you think in your head, it's a lot like these matrix operations. That's what a tensor is. Then this whole tensor RT, which is tensor real time or tensor flow, which is Google's version of similar stuff. You stop hitting that word and failing, which I did a lot. I didn't like the word tensor. I kept thinking there was something more to it. Thinking about it as a matrix operation or the tensor itself is the vector matrix thing. That was very helpful.

I stopped stumbling on the word and went on. It's nice that NVIDIA has a standalone package because one of the difficult things with doing machine learning is that there are many dependencies. When you try to install all of Google stuff, you might hit it at a time where it all just installs, or you might end up with dependencies that crush you and you can never figure out how to get it even working, let alone understand the things you want to understand. One of the things that the NVIDIA two days to a demo installation package installs is the digits library or digits system, which is, which uses the

images from mailing addresses. And so it has all of these handwritten digits, 1, 2, 3, 4, 5, whatever, all 10 of them. And that is used to train different forms of neural nets. And it helps you understand what the neural nets mean and what it means to have three hidden layers versus four or this architecture versus that. It's really useful and interesting. And yet it's still, I mean, mean again this has not been two days at this point but we're still on two days to a demo finally once you're all installed and you start just typing randomly exactly as a monkey does exactly what they say because if you mistype you end up with warnings and errors and it's just typing and typing, make, and try this and that, and verify everything works.

Finally, you get to this demo that is about identifying objects. And so now I get to start the demo about identifying objects. Now imagine there's a camera and a screen behind her, and people are holding up random things. Oh, there's a frog to start with. And for the most part, it says, so there's a frog and people hold up the frog puppet and it says tree frog. But if you turn it around, it says broccoli.

And now I'm holding up a flashlight and it's saying wrench. And now I'm holding up some keys, and it's saying hammer. So this demo, which I do start with the tree frog because I have a tree frog puppet, and it gets it right. And it usually says 80 to 90 percent probability. Right, and then a little confidence value next to the thing. And then when I turn it around and it says brain coral or broccoli, it's pretty funny. And when there was nothing on the screen, it says shower curtain. That was because of the screens we had behind me.

So this identification system, as Chris pointed out, it's not always right. But it's actually much more impressive than I'm making it out to be. It really is. It's very good with fruit and it's kind of bad with vegetables. It's very good telling the difference between my beagle and my Maltese poodle cross, but it cannot identify scissors to save its life, which I found strange. Maybe it's not in the database? Well, that's the thing. It has a thousand items it can identify and it tries to choose which of those thousand items you are showing it.

And if it's in its database, it does okay. But if it's not, then it does terribly, of course. Well, it always tries to present an answer. The way this works is it doesn't say, I don't know what that is. This is the closest I've got. It will say, you know, this is a shower curtain with 30% probability, which is a lot different than a reasonably accurate tree frog puppet at 85% accuracy. You can use this.

And you can imagine finding the objects you want to find. If you want to find good peaches versus bad peaches, all right, we can do this. If you want to find resistors that are color-coded, yeah, this starts to make sense. This identification problem, it has a lot of applications in many different industries. So I like that demo a lot. The next thing that we would do in the two days to a demo, which at this point, I should go back to the talk and let you know that during my formal talks, there was somebody standing behind me showing different things to the identification talk. Some of which I really should watch the video because I don't

know what they were laughing about, but it must've been pretty funny. Anyway, so then the next demo that NVIDIA has for us is the detect faces demo. And it will show faces as they go across the screen. And if you have multiple faces, it will track multiple faces. And if you have bodies or people carrying luggage, you start to see what the purpose of this demo is. It's clearly Big Brother-esque sort of thing. And most of the images they show are from airports. So, you know, it makes a nice demo, but not one that I showed during my talk.

The final NVIDIA demo was segmentation. And this is where you can see that the whole system is directed towards autonomous vehicles. There's one where it shows the segments, the screen, into sky and ground, which for a drone would be super important. But it does this reasonably well, even if it's blue on blue, like you're flying over the ocean or it's a dark gray mountain and puffy gray clouds. It does the segmentation well. And with cars, there was a road, not road segmentation. So that was the two days to a demo. And the next thing I need to do is modify the demos. I tried to do this, but retraining was really hard, computationally intensive, way beyond what my laptop could do, way beyond what our virtual reality computer could do with its NVIDIA card.

And I think my next option is to try AWS, which I'm sort of excited about. That's a new thing. It'd be really interesting to be able to say, yeah, I've trained a model on AWS. So that's my next plan for that. That's what I've done with Jetson is mostly run the out-of-the-box demos. It's not like I wrote a lot of code for that. So why is the retraining so hard? It's because it's a lot of data, right? It is. It has over 100 gigs of data to do the identification demo.

And that's because it is trying to build this enormous library of features and to choose which features are most useful. And then as it runs on the Jetson itself, so the training doesn't run on the Jetson. The actual machine learning part algorithm, the neural net runs on the Jetson after it's been trained. So it's a very two-part problem. And machine learning and embedded systems is coming because they can make these things so much smaller than the training problem.

As long as you can do the supervised learning training off-board, then it makes a lot of sense. Okay. So after I have done the Jetson part, I say, this board, it costs about $500 just for the little card, $600 for the big breakout board with the little card on it. And that's really the state of the art in embedded machine learning right now. And so I want to show you the state of the art in robotics as well. And then I pull out my little Mi arm, which is the $50 robot. And there's still a coupon for the month of July for Hackaday. It's awesome embedded and 30% off of a

$50 robot to make arms was just great. I'm getting two more. And of course, this is not the state of the art in robotics. It's not a bad robot arm. It's really fun and I have had a good time with it. But if you're going to get a $500 board, it probably should be controlling a $10,000 arm. When I talked to Sean Meehan on the show, he said you should either get a large arm that is comically ridiculous for typing on a keyboard or you should get the smallest possible arm that is also comically ridiculous. And given the space and power constraints, I went for smallest. Anyway, so the Mi Arm comes in a flat pack, which means that you have to build it. And I am, you know, I'm an embedded software engineer. Emphasis on software. I come from the software world. I have some nice math.

I have enough electronics that I've learned over the years, but I am not mechanically inclined, like really not mechanically inclined. Building the Miata was nice. I ended up having used two different instruction sets and had to make my brain see things from different angles. And I got to play with gears and I just had a really good time building it. I know people have found it frustrating but I went in thinking it would be frustrating and found it to be quite fun. I did have to do some Arduino code to do a configuration of the motors because if you put them in non-optimally you can't move the arm as much as you want. In fact I had to rebuild part of my system when I realized that my arm could go badly. It wouldn't move it. It wouldn't open all the way.

So a little bit of Arduino code. It was all PWM and servo. So I didn't have to write any of that library. I could just take it. And then I wrote a little command console to say, go here and go there. Wasn't bad. It was kind kind of fun but as part of this i found i found out this thing called inverse kinematics in which you have to translate between xyz some point in front of you to the angles that the arm makes and it's this gnarly math and so i actually now would switch the slide to the gnarly math that says inverse kinematics and has some doodles that describes how it works. Which you went through rather quickly, clicking through. I mean, my intention is not to show how inverse kinematics

works. That's not going to happen in a 30, 45 minute demo. But it is typical of the robotics things that I've found, that every time I have a problem that sounds so simple, I want to go from here to there, it leads to something that looks gross and awful, and how am I ever going to understand that? And then the inverse kinematics problem led me to the law of cosines, which is the generalized form of the Pythagorean theorem. Pythagorean theorem, you know, c squared equals a squared plus b squared, that only works for right triangles. But the law of cosine works for any triangle. Why did we ever learn Pythagorean theorem? We should always have been using this law of cosines. It's so much better. But with robotics, I keep running into this, that I go from this problem to impossibility to, oh my God, I wish I had known that this was true all along. It would have made so many other things more interesting. Working with the joint angles would have made trigonometry so much more interesting than it ever was. I mean, it was always just theorems and memorization before, and now it's how I make my robot go.

And the difference is amazing. If I were teaching trigonometry, it would be with a robotic arm, because finally this stuff is not theoretical anymore. It's physical, and it makes sense. So yeah, and so after I paced up the inverse kinematics, I paced up the law of cosines and a little bit of its, not derivation, but its usefulness, different ways of looking at it. And again, these are doodles that I did in my free time. So robots not only need this, how do I get from here to there? It's how do I get from here to there safely? Motion control. You don't just want to jerk the motors on and off. I'm running hobby servo motors, so I can do some quick moves and not

worry too much about acceleration. But in general, I want it to be a good, well controlled robot. So I want to think about these things. How do I make a good step size? How do I get from here to there? How do I deal with obstacles? I mean, if I have two of these arms, which is my goal, how do I make sure they don't collide with each other? That's a motion planning problem. And again, we get into this, everything in robotics goes from an easy thing to understand to an impossible thing, and then back to some math I wish I had had a long time ago. I started out with Arduino with the Mi Arm, and then I did hook it up to the Jetson. On Arduino, I could just use the onboard PWMs, but moving to the Jetson, I had to use a

little board from Adafruit, the PCA9685, which would take I squared C in and it would do the PWM out. And then for some reason, I know I really program in C and C++ like all the time, but as soon as you put me in Ubuntu, I'm like, oh, I must use Python. So I did write it in Python, the control in Python, and I did it using Raspberry Pi code from Adafruit. They had a PWM library, they had a servo library, they had an I2C library. I had to make minor modifications to make the Jets and work because robotics in Raspberry Pi is really common. So I didn't have to do a lot of work there, which was good. Left more time for worrying about the inverse kinematic problems and other code that I could write.

So all that's kind of working. I can run the system and I can run the machine learning demos. All I have to do now is add some computer vision, some motion planning, some obstacle avoidance, my own machine learning. And that led me to Robot Operating System, ROS, which we've kind of mentioned on the show a few times. It seems like the sort of thing that you're like,

oh my God, how am I going to do all these things? And then you look at robot operating system, you're like, other people have done these. I don't have to be an expert in everything. I can use what other people are using. And that will get me a little further. I can learn their APIs instead of learning all the gnarly math on the bottom.

And it was really neat. It's a great architecture. It's a publisher subscriber model. So like my arm would publish its joint angles and my camera would publish its images. My machine learning part could use those images and it could publish identification. This is a keyboard. My other machine learning part could publish keys. This is the letter A. This is the letter B. This is the letter C.

And then my motion planner could take the ideal goal. I want to hit the letter A. And it could try to find it in the detection. And then it could plan the motion from where the joint angles are to where the joint angles should be. It could just publish that information and the arm will listen and it will take that as a command and it will move. So this publisher-subscriber model is really common in a lot of things. It ends up looking like spaghetti, but it keeps everything separated. And one of the big robot operating system tenants, one of their important founding goals, is that everything should be testable in simulation on its own.

And when you think about designing a system of this complexity, that is important. They have created physics engines and world models that will let you have little fake cameras inside the world model to help you make your robot testable and testable in a way that doesn't require you to chase it down the hallway and doesn't mean that you have to have your power supply on and it means that if you crash it into a wall it's hilariously funny but if you did that with the real thing and it was very expensive that would not be funny. So robot operating system had these things with the simulation and the testing, and it's loosely coupled with the architecture, and it's got a huge community. So lots of things already done, this obstacle avoidance, motion planning, somebody's already done it. But there's a lot of documentation, and there are a lot of pieces that were half-finished PhD theses, and you don't know what's what when you go in.

And it's huge. I mean, it's truly huge, which is great and daunting. So wanting to finish a demo before I needed to give my talks and hitting a point where I had to recompile everything in order to get it to work. There was a dependency deep in the Python OpenCV or something and on Rebel Operating System. I would have to recompile everything, and that broke one of my goals, which is that other people should be able to do the same things I'm doing. I don't want to do any magic. And if I have to recompile huge swaths of code, that's hard. I mean, I don't want to do it because I don't want to do it, but I also don't want to do it because I don't want to ask

other people to do it. So I gave up on robot operating system. That was just too overwhelming. There were too many magic boxes, too many weird dependencies, and I hope I get back to it. But giving up was a little bit of a relief. It's not an operating system either. It isn't really an operating system. It runs in Ubuntu. It's a program, really.

But it's a program that gives you this publisher's subscriber model in a formalized way. And these modules that do things. And a number of the modules that already exist. So having given up on Robot Arboring System and wanting to have a demo for these talks, I got back to my goal. And I started thinking about,

I had the machine learning demo and that was fine. But what do I do to show that I'm on the path to making a typing robot? How do I show that I control the robot arm? And that's where the cats finally came in this whole talk. Everybody's like, where are the cats? So I built this little robot arm, the me arm, and I connected it to the Jetson and I turn on the power supply at this point. And it really was like that, too. And it shows a webcam, a different webcam than we saw before, because I have two cameras here. And in fact, I'm using the identification camera to show the whole robot arm.

And the robot is twitching. It's like its elbow is twitching up and down, but its fingers, its gripper isn't moving very much. And then I pull out a laser and I shine it at the wall. And the little robot arm tries to be a cat. It tries to follow the laser. And if the laser stops and it thinks it's in position, it starts to do this little gripper thing where it would open its gripper very slowly and then snap it's in position. It starts to do this little gripper thing where it would open its gripper very slowly and then snap it closed and open it very slowly and snap it closed. And it has all the code for the laser finding in OpenCV in Python. And that makes a goal position.

And then I use the inverse kinematics and I move the mi-arm and there's a whole bunch of silly, this whole butt wiggle thing where it's wiggly and its elbow has to do with the inverse kinematics. And there's a line along which there are multiple solutions to the joint angle to XYZ conversion. And so I can stay in one position and move my joints even though I'm mostly in the same position. And I found that sort of by accident. But as soon as I saw it, I was like, that looks just like my cat when she's thinking about going after a laser. And then there were the other things with the inverse kinematics. And the whole time I'm playing with the little robot and still talking about inverse kinematics. Nobody followed this

part of the talk because the cat is so cute. Anyway, there's also this inverse kinematics problem where you have to go from one position to another, where you just have to turn through one angle. If you think about you as a person, it would be just turning your shoulders or your hip around. So you're just angling one across the other. But the robot doesn't do that. When it's in inverse kinematic mode, it tries to create a straight line, which makes it look really funky. This is where visuals would really be helpful in this, wouldn't it? Well, just imagine you're holding your arm out and you're trying to move it to a different position. If you just rotate, you trace out an arc. Right. But if you do it the way the inverse kinematics solves it,

at each step it's trying to get to the intermediate. It draws a line between the two points. Instead of an arc. And then it follows each of the points in between that. So if you made your arm go in a straight line between two points, you would no longer trace out an arc, but you'd have to move your elbow inward as the position came closer to you and then back outward as it was away from you. And so instead of an arc with a single joint

moving, you end up with all of the joints moving as it creates a straight line. And it looks odd, but it is what you end up with, with robotics. It's one of the neat things that I discovered. And of course, all of this is not the end of what I want to do. This isn't a typing robot, clearly. I have many things to do. Some of them are in my normal set of skills. I have an ADC. I want to do current feedback so that I don't have to carry around the loud power supply. I needed it because sometimes the robot gets jammed and I wanted the immediate interaction of being able to see the current numbers spike and be able to put my hand over there and help it.

It just can't do gravity sometimes. And so I have the ADC. I know how to do that. I'll put that in. And I want to do some motion smoothing because right now my robot is based on the camera update frequency which is kind of slow I should do a timer in python and have it update much faster it'll be

smoother and look better although I was pleased with how it looked partially because it was supposed to look like a cat which was a little animatronic looking but my goal is beautiful robotics not not the cat I mean it was cute. And then I want to add the second arm, which means retrying the robot operating system so that I can get better modeling and obstacle avoidance. And then there's the Jetson code and modifying that. And then, of course, there's all the rest of the software I want to add. You know, this all seems like a very doable list, right? So at that point, that is the end of my talk. And I've put up slides that I forgot to tell you about,

but only like one of them, so it doesn't really matter. And I invite people to come up and get stickers and to ask questions, but you can't ask questions. They can ask questions. They can ask questions. They're not really shy about asking questions, are they? So yeah, that was my talk. All the links are at embedded.fm slash blog slash Ty. But of course, they'll also be in the show notes.

So you can see them there. So that was it. Was that pretty much what I said? You went to the crash space one. Yeah. I think you might have talked about inverse kinematics a little longer. Here or there? There.

I don't know. It was about the right length, about the same length. Well, they don't have the equation in front of them, so they can't really see the inverse kinematics problems. Yeah. Yeah. Okay. So when are you going to finish the typing part?

I don't know. Having looked at the two days to a demo again, and they are updating it, so I understand how it's out of sync with what I think it is. It's mostly in sync, but they do talk about how to update some of the models, which I didn't understand how to update them. And now I do. So I'm going to try some of that. But to answer your question, I have no idea. I mean, this is hard. I've been reading about reinforcement learning and I re-listened to the Polarski episode and I got the book that he suggested and I'm like 30 pages in and totally over my head. I need to know optimal control theory. I need to know game theory. I need to

know machine learning. I need to know more math. I'm just so over my head and I think I'm happy with that. I'm learning stuff, but sometimes I get a little frustrated. So we'll see. And I think I'm happy with that. I'm learning stuff, but sometimes I get a little frustrated. So we'll see. And I got some really great suggestions. I had some problems with the robot operating system with modeling my arm. And Dan Hinch in the design lab suggested putting it in Fusion 360. There may already be a model for this in Fusion 360, but if I do that, I can start looking at the physics engine for that, and that might help me build the physics engine for a robot operating system. So there's a lot to do, and that's okay. I'm learning stuff. I am going to have to find a paying job soon, because I've been doing this for most of the last month in preparation

for our trip, but also because my clients said, hold off for a little while. And I was like, okay, but you're good clients. I'm only going to do this for a little while. And now they're saying hold off for longer. Maybe we'll see. We'll see what next week brings. So I'm going to have to eventually go back to work and stop my summer vacation but this has been a great summer vacation project if i show the next piece i think it's going to be the reinforcement learning where i tell it you have to push a button and i put the button somewhere and it has to just find it so that's that's my next plan um or maybe the adcs or maybe i'll just read more. Gosh, I don't know. It's not going to be done next weekend. I can tell you that much.

All right. I do have a couple more announcements. First of all, yes, Patreon. We still have Patreon. We really appreciate those of you who support us with um with the monies it lets us do things like send guests microphones as well as stickers to people uh patreon people do get first priority for stickers but the rest of you can ask i'm very slow about it. Ask the person who's getting them two years late. But I try to be a little better than that. Oh, and Patreon supporters, since we had a little bit of money left over,

we did become an open source hardware supporter. And that came out of the Patreon money. Well, it mostly came out of Patreon money. Some of it came out of the Lodgwell against money. But it's all sort of the Patreon money. Well, it mostly came out of Patreon money. Some of it came out of the Logical Elegance money. But it's all sort of fungible. Okay, the other announcement. What if they really don't like open source hardware? And now that their money has been given to a cause they don't support?

Sorry? Sorry, but probably next time we'll do a cause you do support? I mean, it wasn't really. Sorry. Next. In a couple of weeks, we are going to start a new series on the podcast. So about every five or six weeks, we're going to have an episode that is Christopher and myself and a couple people that have been past guests. I think Andrea Tichak and

Christopher Svek will be the first ones, but we'll have people like Dennis Jackson and Elizabeth Brenner and Alvaro Prieto and maybe even Chris Gamble will join us. But people who are experts will have a panel style series where we talk about basics. And it's not just about basics. It's how to explain basics. So I don't think the rest of you will be bored. It's not basic. No, no, no.

It'll be things like interrupts and different serial protocols and writing a driver and other topics like that. And the goal will be to explain these things, but also to talk about how to explain these things, what metaphors work, and when you need to let somebody just struggle through because you can't you can't teach everything they have to learn on their part too and the first one's going to be sort of general getting into embedded moving from arduino when to start going after embedded jobs so that one's going to be a little general but after that we'll get to more technical and topic based uh i noticed topic-based. I noticed in the, what was it, survey we had, that a lot of people wanted to know, you know, specific technical information, not just interviews. So that's the plan. We'll try it. And if you like it, great. And if you don't, well, I believe it's, I should have a funny email here, but you know, you can always contact the show

even if I give you a funny email, you're still just going to contact the show yeah, anyway, that's all, do you have anything, Christopher? any things? anything you want to announce before we go? no, because I have an appointment in approximately five minutes well, that's about how long it's going to take me to read the next section in Winnie the Pooh. Okay, when we last left Pooh, he was looking for a balloon so that he could get honey. And Christopher Robin was trying to explain about balloons. Well, it just happened that you had been to a party the day before at the house of your friend Piglet,

and you had balloons at the party. You had had a big green balloon, and one of Rabbit's relations had had a big blue one and left it behind, being really too young to go to a party at all. And so you had brought the green one and the blue one with you. Which one would you like, you asked Pooh. He put his head between his paws and thought very carefully. It's like this, he said, when you go after honey with a balloon, the great thing is not to let the bees know you're coming. Now, if you have a green balloon, they might think you were only part of the tree and not notice you. And if you had a blue balloon, they might think you were only part of the tree and not notice you. And if you had a blue balloon,

they might think you were only part of the sky and not notice you. The question is, which is more likely? Wouldn't they notice you under the balloon, you asked? They might or they might not, said Winnie the Pooh. You can never tell with bees, he thought for a moment and said. I shall try to look like a small black cloud. That will deceive them. Embedded is an independently produced radio show that focuses on the many aspects of engineering.

It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.