Embedded - 283: Flippendo Is Kind of a Swirly

Episode Date: March 28, 2019

Jennifer Wang (@jenbuilds) spoke with us about machine learning, magic wands, and getting into hardware. For more detail about her magic wand build, you can see Jen’s Hackaday SuperCon talk or her !!ConWest talk. The github repo is well documented with pointers to slides from her SuperCon talk and an HTML version of her Jupyter notebook. Check out this good introduction to machine learning from scikit-learn. It was their choosing the right estimator infographic we were looking at. (Elecia has bookmarked this list of machine learning cheat sheets.) Jennifer’s personal sites are jenbuilds.com and jewang.net. She recommends the Recurse Center and wrote a blog post on her experience there.

Welcome to Embedded. I am Alicia White. I'm here with Christopher White. Our guest this week comes to us from Hogwarts, Gryffindor to be specific. Jennifer Wong is going to speak with us about IMU's Magic Wands and Embedded Machine Learning. Hi, Jennifer. Thanks for being on the show with us. Hi, it's a pleasure to be here.

Could you tell us about yourself? Yeah, happy to. I'm Jennifer Wong. I'm a software engineer, hardware prototyper, and maker. What this means is that I'm good at creating concepts for new product ideas and hardware. When it comes to scale and productionization, I'm definitely much more experienced on the software front. And I've built sensor algorithms that have launched to over 1

billion devices and people worldwide. So definitely really happy to have that feather in my cap. In general, I'm like super excited about basically everything, but especially sensors, documentation, technology for empathy, human-computer interaction, and technology for well-being and health. It sounds like we have a lot to talk about. But before we actually get to the magic, we have some questions for lightning round. Short questions. We want short answers. And if we're behaving ourselves, we won't ask how and why and all of the other questions. Chris, you want to get started?

Sure. Potions or spells? Spells. Are you responsible enough for a time turner? I feel like I should say yes. What is the core of your wand? Unicorn hair. What Quidditch position would you play? I feel like Seeker would be fun, but a little bit isolating.

What animal is your Patronus? Well, I would love for it to be like a chicken or a fox. You get to choose, don't you? Oh, I get to choose. We're not responsible for this. Okay. Must not ask why. Must not ask why.

Okay. Probably a fennec fox. Ooh, nice. Who would you send a howler to? Oh my gosh. There's, uh, I don't know, probably my sister as a prank. It'd be pretty great. Do you like to complete one project or start a dozen?

Um, I would like to complete one project, but that doesn't stop me from starting a dozen. What's your favorite flavor of Bernie Bot's Every Flavor Bean? That is a really hard question. I really like watermelon or popcorn. What's a tip you think everyone should know? I think that patience is very, very important, and you can always have more of it. Where do you get it? Is there someplace you can go to buy it? Because I always am running out. Well, I hear there's probably a potion for it, but I haven't found the recipe yet. Well, as some people may have determined, there is a theme this week. And it started with you wanting to be a wizard for Halloween and making a magic wand.

Could you tell us a little bit more about that story? Yeah. So I really like making costumes and dressing up for Halloween in general. And this year I kind of, I wanted to do something special because I actually borrowed a robe from my friend who like sews all her cosplays by hand. They're like pretty amazing. And I'm like, well, like I can build things and I should like make this costume my own and like make it special. So I was I came up with the idea of building kind of this magic wand that did kind of gesture recognition. And

I came up with this idea not too far from Halloween. So it was like a mad scramble to kind of put it together. But this project was made a lot easier because I scoped it very small. It only had to work for one night and it only had to work for me. And when you have these kind of constraints, it's so much easier to build something than something that'll work for like a bajillion people. Yeah, but now you've been talking about it at Supercon and at Bang Bang Con and it's been more than a night. Has the effort ballooned beyond what you expected? Oh, I think it's pretty incredible. And I think the most fun part of the project is like, depending on the background of the person I talk to, they have so many different questions.

So it's definitely expanded in an unexpected way. Like, I'll go up to, like, my software friend and I'll be like, let me talk about, like, the ML. And he's like, no, no, no. Like, how is the battery attached to the wand? Or I'll, like, go up to a hardware person. They'll be like, I want to learn about the ML. And I'm like, but sensors, let's talk about sensors. So definitely kind of pulling my knowledge in several different directions. Okay, so sensors, let's talk about sensors.

You have an inertial measurement unit on there. And if I understand correctly, you use the inertial measurement unit to detect gestures and then that leads to magical things yep how would how would you have described it yeah um so i i the magic wand is pretty simple um i have an initial inertial measurement unit and i mostly use the accelerometer and the gyroscope on the IMU. And I just sample that data over time. And when it detects that there's this user doing some sort of magical gesture, I'll play a sound effect. There were different stages of this project where I was like, well, if I have more time, I'll like make it turn on the lights or whatever. And, you know, I scoped it down to just play a sound effect to get things in time for Halloween.

But you have a Raspberry Pi W. So you could, you know, change something on the internet if you wanted. You could go pretty far with whatever you wanted to do with. Yeah, exactly. Exactly. I thought, you know, one thing that could be really fun. Oh, I saw this out, you know, Bang Bang Con, there was that talk about kind of, let's make Postgres play Pokemon. I was like, well, it'd be so cool if I could like, you know, very inefficient, I could wave my wand, and it would like cause you to do things running in like a pokemon emulator um that would be really fun um but yeah that was definitely the intention behind choosing this flavor of a raspi that um the wireless component meant that i could eventually make it an internet of things device

and so you bought the magic wand that seems like cheating but it where do you buy a very specialized construction process so you kind of have to buy them unless you're the guy in the store with the yeah and then he has to audition you and everything yeah yeah yeah it was definitely a really complex process i needed to kind of work on my like you know one waving technique i needed to work on it, you know, an intake interview, and then I had to like fly to London. It was a very involved process. But my friend like suggested that I could just also order things off Amazon. So I kind of just like typed in like Harry Potter cosplay wand and I bought the first thing that came up.

Amazon takes the magic out of everything. Add the magic. And so this was just a static piece of wood plastic. And then you added batteries and the Raspberry Pi and must be a speaker, the IMU. What else did you add? I think you covered most of it. Let's see. You have the Raspberry Pi. You have the battery, the speaker. A most of it. Let's see. You have the Raspberry Pi. You have the battery, the speaker, a bunch of wires, a bunch of glue, the IMU. Yeah, it's a pretty compact project.

And then the IMU, this is one of the Adafruit ones? Yes. So that must be the Bosch BNO55, which is a very cute little IMU. Yeah, it is. Had you used anything like this before? How did you choose that? What was the hardest thing about learning IMUs? That's three questions in a row.

I was just going to continue to ask questions and hope she could keep up. I think IMUs have come a long way. I was just going to continue to ask questions and hope she could keep up. reality headset for 3D audio, because I guess that's what you like experiment with when you're a student. And I think IMUs were a lot less mature at the time. There definitely weren't really nice libraries for interfacing with them. So I think that this time around when I wanted to work with IMUs, it was really nice to just, you know, just Google like, you know, what are some very common IMUs and like, more importantly, which ones have like Python libraries that work really well. So that's basically my own only criteria. What can I like buy easily? And what has a really nice Python library to interface with? Okay, so Python, Raspberry Pi, that's I mean, you can use Python, it makes life so easy, especially if you're going to add some machine learning aspect to it. What else are you using in Python? What packages did you have to install to get this to work? I do a lot of Python kind of both as a hobby and professionally. So I can rattle off a bunch of like Python libraries that,

you know, everyone should just like import into their kind of Python projects. Even if you don't like use them offhand. So there's like matplotlib, there's numpy, scipy. And I really like using this thing called pandas. Super fun library allows you to kind of handle like

massive amounts of data, but you end up with these like really fun Google search queries. Like one time I typed in like rolling pandas and I got like all these pictures of pandas rolling over and I was like, what is going on? Who doesn't like pandas? Who hasn't typed in panda pickle? That's a Python joke. Sorry, Chris is looking at me like, you must be kidding. No, I'm not. I think that's hilarious.

Yeah, so I think for the machine learning portion of this project, I use Scikit-learn. So very, very common machine learning library for Python. And I did a lot of my explorations using Jupyter Notebooks. So Jupyter Notebooks is kind of like an IDE that runs in a web browser and is very nice for being able to write things and run Python code and then be able to send all of that to someone else. What made you choose that? I mean, this seems like the sort of project that I would run command line because I don't know, it's just me. Why notebooks? I think for me, let's see, a long time ago, I was doing, I was using Mathematica. And Mathematica was like super exciting. And I really liked that you could kind

of type code and then like get the plots kind of in line. And then you could like fiddle around with all the graphics and everything. And so naturally, when I like started using Python, when I discovered that there was this way to kind of get Mathematica-like things in Python, I totally jumped on that. And it's pretty incredible how I think in Python, if you like generate plots and the image plops up, you have to like, you know, close the windows over time. And you would say that, you know, it's probably only like a few hundred milliseconds of like delay. But when you have like a Python notebook and you have everything in front of you, it's just so nice to be able to kind of just open all of that. Another thing that I really like about using Jupyter notebooks is that especially when you're kind of hacking on these hardware projects or doing ML, you're not going to finish it in a day. It's going to take you several days.

So you're going to have to be taking notes anywhere. You're going to have to come back to your graphs and be like, oh my God, what was I doing? What version of the code was I running here? And if you have everything in a Jupyter notebook, you can type your own notes. You can have your graph there. You can have the exact kind of code you ran. So everything's really nicely packaged together. Just generally with ML, you're going to try like 10 different things. And like, you know, one or two of them are going to have only one or two of them are going to work. And you have to remember which ones worked. And so you can have your like paper notebook, or you can have like a Word doc or Google Doc in front of you, or you can just have it all in one place. I love Jupyter notebooks, and you have made a great commercial for them.

I should mention, you did mention Mathematica and if people are interested in checking it out, well, they probably shouldn't, they should probably do Python and Jupyter. But if you're interested in Mathematica, it's actually free for the Raspberry Pi. So you can get a free version for the Raspberry Pi. It's usually very expensive, but I don't know why they did that. Maybe because it doesn't run super fast on a Raspberry Pi, but if people want to check it out, it's free. Okay. Mathematica.

That was just such, I haven't used Mathematica in years. Neither have I. And you mentioned Scikit and Scikit-learn. These are pretty big libraries. And I usually think of them in line with TensorFlow. Do you know the difference between scikit-learn and TensorFlow? Yeah, I can speak to that a bit. So TensorFlow is generally used for kind of deep learning and neural networks and, you know, all the shiny stuff that you see in the news nowadays.

Scikit-learn is more used for kind of very kind of traditional machine learning. What I like to call kind of applied machine learning, it's kind of, you know, your traditional set of machine learning tools that you're going to learn in kind of an intro machine learning class. It's not kind of deep learning, but it's just a toolkit of a bunch of machine learning models and algorithms that work and have been proven to work over a long time. So these are like the classification, the more linear regression kind of classification things versus neural nets? Yeah, yeah, exactly.

And actually, Scikit-le learn has this great algorithm cheat sheet that walks you through what kind of algorithms most likely to work for you. Have you seen that, Jennifer? Yeah, yeah, it's, I definitely referenced it when I was working on my Harry Potter magic wand project. It's really phenomenal, because I like to say that when you enter a lot of computer science disciplines, you have topics that kind of build on to each other. But when it comes to machine learning, especially when you're not playing with the deep learning stuff, you kind of just handed a whole bucket of tools and you're like, I need to know when to use these. And generally, like you'll have this framework in your class, in like whatever class you're taking or when you're

learning where you're like, okay, do I have labeled data? So is this like a supervised machine learning problem? Do I have like unlabeled data? So that's like unsupervised. Do I have a mixed? And then you have to think about how much data you have. And then you have to think about, you know, am I trying to decide between apples and oranges? Or am I trying to classify, you know, the ripeness of an apple? And so, you know, the ripeness of an apple would be kind of a linear regression or some sort of regression problem where you're kind of taking data and mapping it to a number, whereas classification, you're kind of taking data and kind of bucketing it and you're trying to classify it. So there's kind of like a lot that kind of goes into it.

But in one kind of algorithmic cheat sheet, you really only have to learn like linear regression. And then you're like, okay, this is like, you know, linear regression for like classification problems with not much data. Yeah, it just makes everything super simple. It's definitely still useful to kind of understand how each of these models work. But it takes a lot of time to like learn all 10 of them. So at least this way, you can figure out, okay, like, maybe I need to spend some time learning more about, you know, K neighbors clustering. It seems to me like everybody these days reaches for deep learning without fully understanding it. Neural nets will solve anything. And that's like a little corner of machine learning where all this other stuff that people probably don't think about at all, if they haven't been exposed to it, don't use it. It might be more appropriate for all kinds of problems. Anyway, that was just a little editorial comment to people. Well, this cheat sheet is a flow chart.

And so you start with it, and if you have more than 50 samples, you can continue. Otherwise, you have to get more data. And then there are these, I mean, flowchart pieces, the decision trees. And if you're producing a category, you go one way. And if you're producing a quantity, you go a different way. So can you walk us through this tree for the magic wand? Yeah, certainly. So thank you so much for describing that.

The cheat sheet is definitely a flowchart. So when you start, the first question is, do you have greater than 50 samples? And you're like, you have to have greater than 50 samples. So yes. And then you ask yourself, are you predicting a category? And for us, we are kind of taking sensor data and we're figuring out, is there kind of a Wingardium leviosa signature in this sample data? Is there a flippendo signature? So this is very much producing a category. So definitely we proceed to the next item, which is, do you have labeled data? So I'm

collecting all the data myself. So I will be labeling all my data. So yes, I do have the labeled data. Next, the question is 100K samples. Do I have less than that? Do I have more than that? Well, as I said earlier, I don't have a lot of time before Halloween. So I'm not going to have the time to even consider collecting more than 100K samples. So definitely less than that.

And with that, just a few questions I've landed on using a linear SVC. Okay, that's a support vector. I always thought it was SVM. What is an SVC? It's a support vector classifier. Oh, okay. Yeah, yeah. Not machine classifier. Okay, so support vector classifier. What is that? So in general, the way that this SVC or SVM works is the mental model to have is that you have a bunch of data kind of plotted on a 2D graph, and you're trying to draw a line between this data. And whatever is on the left side of the line is one class, and whatever is on the right side of the line is another class. So one example of that might be, let's say you're trying to build a fruit classifier, and you have a bunch of random

things on this graph. And you're trying to classify between apples and blueberries. So maybe your x-axis is color or or color frequency because it's a number. And your y-axis is like roundness, right? And so you have a bunch of different samples. And you're going to tell yourself, well, apples kind of come in red, yellow, green. And they're kind of round. But a lot of them like, you know, red delicious apples are less round versus blueberries. So like, you know, very ripe blueberries are like, you know, very dark blue, less ripe blueberries are

green. So you can't just classify based upon color. You need like this other aspect, which is roundness. So you kind of, this is a little bit long winded. No, no, no. Actually I was going to do apples and oranges. So you're totally, and then at the very bottom where you have they're small and they're round, well, I guess you didn't have size. So they're round and they're green. You can't really tell line and you say this one or this one. Now, if you add another axis for size, then you can say small versus large, and that probably separates things. And SVCs, support vector thingies, are all about having lines in different dimensions. And so every problem becomes linear, but you have a whole bunch of linear problems. And at the end, you end up with a classification. It may be green, it may be round, but it's big. So it's definitely not a blueberry. It may be

blue. And so therefore it is definitely a blueberry because you can just skip all the rest of them. I mean, you can't really, but you do. And so support vector machines are all about data that is linear in multiple dimensions. Do you agree with that? Yeah, exactly. That's a very good summary. Okay. So you use this chart, which doesn't even include neural networks. I mean, deep learning is a separate thing. And did you consider neural networks? I mean, deep learning is a separate thing. And did you consider neural networks? Yeah, I definitely did. When I did this project, I kind of wanted it to be kind of an intro to like machine learning and whatever sense, like project for people to use. And I was like, oh, wouldn't it be cool if I used a neural network? The issue with neural networks is you need a lot of data. And kind of you need to think about the gestures

you're going to build. And like, in general, like, you don't know how much data you're going to need. Like, you just know that you're going to need a lot of it. Okay, so what does your data look like? You said accelerometer and gyro. What frequency are you sampling? I'm sampling it at about 60 hertz. And my traces are generally about 1.5 seconds long. I'm collecting data at 60 hertz. So that's about kind of 60 hertz times 1.5 90 yeah yeah so about 90 and then because i get data in xyz that's 90 times three and then because i'm getting two different sensors uh gyros i don't remember if i just use kind of the synthesized data or the raw i use the raw data. Yeah. So because I have two sensors, that's like 90 times three times two.

So you end up using a fair amount of data. Yeah. But only a little of it is useful. Let's see. I saw your two gestures. Describe the two that you used. Flipendo, is that the one that's mostly a circle? Yes. So Flipendo is kind of a swirly and Wingardium is kind of making this big air W.

Okay. So those are pretty different gestures. But when you do gesture recognition, traditionally, you would look for the start of a gesture, which usually would be a single motion. A lot of times it's a turn, and then you can indicate that that's how you're starting a motion and that it should pay attention. And then you look for the next step. It's almost a state machine as you do gesture recognition. You're looking for this and then that and then that. And that leads you to a gesture that is recognized. And anything that falls out halfway through is not a gesture. But with support vector machines, that's not how it works. Yeah, the way that this one works is it's actually just running kind of on the data all the time. So I

just basically have a sliding window and I have a lot of kind of data coming inside. And whenever I kind of run my SVC and I get that there is a gesture there, I'm like, okay, there is a gesture there. What you describe kind of having this more kind of state machine approach. I think that that's honestly, it's a lot more practical. And the reason that I think that is that you can imagine that for some gestures, it might be harder to detect. So you might want to turn on more sensors, but sensors are very power hungry. Like if you want to build things for real, it's always about kind of optimizing power. So knowing when you want to turn on and off a lot of different kinds of sensors to use to kind of refine the gesture recognition itself. And that's kind of why you have this kind of state machine so that you can figure out, so you can kind of optimize for the amount of power you're using. But for this project, because I only had to make it work in an evening, and because I had a fairly big battery compared to, you know, using a Raspberry Pi Zero W and like a tiny sensor, I knew that I could just run things continuously, and it would

be fine. There are also like, I think in general, when you're doing gesture recognition, like there are a lot of different techniques that you can use to kind of optimize for battery life. Battery life is always the biggest question when it comes to, you know, doing a lot of kind of embedded projects, as I'm sure you know. I want to get back to the support vector machine and your 500 and some samples per gesture. So your window is about 500 samples in your three axes and two sensors. So you have these sensors, you have this data, and you have some labeled data so you know when you're doing this gesture versus that gesture. And you probably have some labeled data that says no gesture.

Given this data, given these clipped data snippets that I've labeled, how do I get to a support vector machine running in a pie? What are the steps in between? How do I train this thing? That's a really good question. And so I think first, one thing that can be really helpful is kind of being very precise on kind of the language that's used. So the language that I like to use is for each kind of, you know, for each piece of data that's like coming out of the wand, I call that kind of a data point. So, you know, in like over 60 Hertz in one second for one sensor and one axis, you'll get like one data point. And then when I say like 1.5 seconds of data, so like that's when the gesture is going to happen, I call that a trace. So you have data points and then you have data traces. And for a long time, I guess I like to use the term data segment, but it gets really

confusing because people don't know what I'm kind of cutting. So what I basically did was after I kind of hooked up the sensor to the Raspberry Pi and I was able to kind of start getting data on it. I wrote some basic code to just take the data, take the data that was streaming through and then kind of just write it to CSV. So like comma separated values. So I was able to kind of have the script that would be like record a bunch of like

Wingardium gestures. And then I would hit start, do a gesture, hit start, do a gesture, hit start, do a gesture, hit start, do a gesture, do this over and over again, and create like a bunch of files for like Wingardium gestures, for Flipendo gestures, and then for no gestures at all. So I had these CSV files. Then I basically, I used the Scikit-learn kind of library. And all you have to do is you kind of make these lists of, you know, here are some traces with a gesture. Here are some traces without a gesture.

Please classify for me. And I'm making it sound really simple. And I think that's the thing with machine learning. It's always like, it sounds very complex, but most of it is kind of collecting data, cleaning data, getting your data in the right format. And then when all that preparation has been done, you kind of just load up this library and you're like, here is my clean data, please produce a model. And with that, it just gives you a model. I should be clear that this whole process,

when you do machine learning, it's split into two stages. So there's the training stage, and then there's the inference stage. So right now I'm doing the training stage because all I have is a bunch of raw data, and I'm going to kind of create a model. So this is training the model. It all happens on kind of my laptop. In this case, I'm using a MacBook Pro, which is a lot more powerful than a Raspberry Pi. Once I have this model, then I can take this model and I can like pickle it in Python. This is basically like serializing it, like making it a binary. And I take like this binary or the serialized model or this pickle, and then I put it on the Raspberry Pi.

And on the Raspberry Pi, I basically do inference. Wait, wait. I want to go back for a second. The pickle is not your data. The pickle is the generated support vector machine. Yes. Okay. Where do the pandas come in? Just because we talked about them earlier. So pandas is a really good way if you have a lot of metadata attached to your trained data to just filter through things very quickly. I think one way that I think about pandas is it's like a very powerful library that allows you to do kind of SQL queries in Python. It's like, give me all my like Wingardium traces and it'll just kind of load them all. And it also has kind of built in like plotting tools. It just makes your life really easy. But you only really see it like when you're in the training

phase and you're just trying to clean your data. So you were training and you created a pickle file from your pandas. Sorry, I'm just going to amuse myself so much with this. add salt to a microphone. Like, probably salt in electronics usually is not a good thing. Okay. But yeah, I choose to... So in general, after I run my training, it gives me kind of a Python object or like a Python... It gives me basically a Python object that I can call. And it has a method that's called something like classify. And then I like dump the data in and it'll spit me like a true or false or a classification. And so I have this like magic object that you know, that scikit learn has handed me and I can just shove data at it, and it'll tell me the class. And so this is this is like my object that is the model that'll do the classification. So I take that object and then I pickle it. And then I can like use that object anywhere to classify. okay in the last second and a half have i had one of the classifier which of the classifications is

closest and then you just keep asking that over and over again do you do it every sample every second um how often do you run the svc um let me check this is actually something that I tuned over time. So I think in the ideal case, because I'm running it at 60 hertz, I would rerun it every 60th of a second. However, the Raspberry Pi isn't very powerful. So if I'm doing that, all my sensor data is going to kind of be, it's going to start kind of slipping and all like my timings are going to go off. So I don't actually remember the exact frequency at which I classify, but I basically kept modifying it until like I didn't end up with clocking issues. Yeah. I mean, you really don't have to go at 60 Hertz. You probably could easily do 10 Hertz or even five Hertz and you'd still keep up because the whole point with the machine learning aspect of it is that it doesn't have to be precisely the same every time.

Yeah. Did you look at your trained results in any way? Did you find a way to look inside the black box that is the support vector machine or the pickle and see what it was queuing on? That's a really good question. So I did not go through the process of like visualizing the exact SVC. However, when you do these kind of traditional machine learning approaches, so in deep learning or in a neural network used for deep learning, you might just take a bunch of raw data and you'll like just dump it into the model and the model will do magic. And it'll be really, really difficult to start to kind of understand how your model works. In fact, I think this is an open area of research, like creating visualization tools just to kind of understand deep neural networks. When it comes to kind of linear SVCs, it's much, much more simple.

One of the reasons it's more simple is that you're not putting in raw data. You're actually putting in features. So, for example, like if we go back to the fruit analogy, if you're doing deep learning, you might like just dump in pictures of fruit. And so your neural network is just getting image files. And it has to figure out that color is an important thing. In comparison, when you're doing kind of this SVM approach, you have to think about like, what features are going to be important? What, like, if I had like a small child,

and I needed like my small child to like figure out, you know, is there a gesture or not? Like, how would I kind of describe this gesture to a child? You can't say something like, look for W's, but you can do something like, you know, W means that you're going to be doing like two or three peaks. So like count the number of peaks. And then if this like, if the number of peaks is like this much, or like roughly this much, like classify it this way. So going back to the fruit analogy, this is kind of you're describing you're like blue or green or red. I didn't actually look at kind of what came out of it. But because these models are so simple, like, it's very easy to kind of just create your own understanding of

how the model is probably going to run. It's a very good point that I think in the debugging phase, being able to kind of visualize and really kind of look and understand your model can be very, very helpful. So had it been kind of, had this problem been more of a challenge to kind of detect gestures, I definitely would have gone into kind of exactly like, what is it training? Like, am I overfitting? Am I underfitting? Does the model being trained kind of match my expectations of how I think it should run? So it all depends on the features. And feature selection is very important.

But because scikit-learn is sort of a magical thing, you didn't have to choose features. You just passed it data, right? I actually chose features. Oh, okay. What features did you choose? So I looked at very simple kind of the range of the accelerometer, the standard deviation, the range of the gyroscope signal, and the standard deviation of this gyroscope signal. And most importantly, when I talk about range means standard deviation, these are super easy. So I can just throw them in because it's one function called to calculate them. Just using these kind of base features didn't work. So then I had to think about what things are actually useful. And so as I alluded to earlier, I was

like, well, maybe the number of peaks is useful. So I actually went and I like wrote a snippet of code that would like count the number of peaks, and then also throw that into the model. Because W is a down, up, down, did you look at the sign of the peaks um i i love feature selection i mean i just we could we could dive down talk about all kinds of features chris is so bored already um so i think that um when you're doing classification like there's so many ways that you can be like oh these are the special parts of the feet like these are the special parts of the feature. Like, these are the special parts of the gesture. These are the features that are going to matter.

For us, though, we had, my focus was figuring out, you know, what are ways that Wingardium is different from Flipendo? Like, how can I figure out the differences? So, in this case, like, the sign of kind of the way that I was doing the gesture didn't actually matter. What mattered was that I noticed that in Wingardium, when you look at the accelerometer signal, there are kind of two peaks. And when you do flimpendo, there are actually three peaks. And so all I had to do was count, are there two peaks in this one and a half second snippet,

or are there three? And this ended up being one of the core things that distinguished these two gestures. Does that mean that you can do Wingardium and it can be an M or a W or a sideways or E or... Gosh, now I want to try them all. Yeah, yeah. I think... I haven't tried it, but it's possible that an M could work. And it's possible that an E could work. The data that I collected was very specific to me that one time. So I have no idea how well it would work, you know, if someone else used the wand, if I were left-handed, if I did things

differently on a different day. I think when you're doing kind of machine learning for a lot of people, you need to collect data from a lot of people. And you have to worry about, you know, is this person making M's or W's? Because not everyone's going to understand exactly the gesture that you intend for them to do. So definitely, I'm also very curious what would happen if you tried to make those symbols. How hard do you think it would be for someone else to replicate your wand? I think that my hope is that it would be pretty straightforward, because I've put up some code on GitHub and I've also kind of made this Amazon list. So you can just go to my Amazon list and buy everything there. And I've provided graphics so that you can hopefully, it's obvious like what you glue to what,

what you kind of wire together. And then once you have that, you can just kind of pull my GitHub code and be able to run the whole model. Because this was still a relatively lightweight project, I can't say that the documentation is perfect for it. But the hope is that anyone can kind of take it and in one or two weekends just have something running. And it does have a wire to something else. So there's the wand, which has the IMU on it. And does it have the Raspberry Pi on it? Yes. So the wand has the Raspberry Pi and IMU both on board. I think when you're talking about why you're referring to the internet of things aspect of it? No, no, the battery, that it's not on the wand.

Oh, the battery is glued to the wand. Oh, I thought you had some sort of widget up your sleeve. I thought it was a speaker. Oh, that makes sense. Yeah, yeah. So I very carefully chose the Pi and the battery and the sensor to all kind of be very small and be on the wand unfortunately it was just very hard um to find a speaker that could like gracefully be attached to the wand so that's what went up to my sleeve but everything else was on board to the one

so you can imagine that with this wand if i did not want it to kind of make sounds if i wanted it to do something else entirely there might be like yeah yeah exactly um there might be like no wires up my sleeve at all okay sorry i i went away for a little while to design my own tricolor led magic wand because if you do this gesture it should go red and if you do that one it should blink and okay um yeah i'm i'm all over it. Yeah. You could totally use the code I wrote to do that. Yeah. It's just like at the end, after the classification, it's like, if do this, like hook into kind of the LED controller. You should do it. Yeah. You're my copious free time. Okay. What do you wish people would ask you about the project um i think things that i wish people would ask me are kind of um how can i get into this like how can i like learn more

about the space i think the embedded ml space it's really exciting um it's definitely like a growing space right now. And there's so much more that can be done. I think it's very easy just because this, just because kind of sensors embedded and machine learning kind of combine disparate kind of fields, people are often very intimidated by it. So I'm hoping that people can just like,

ask me like, how do you do this? Or, you know, if they get stuck, they can like ask me the appropriate question. I can be like, it's not that complicated. Like, here's how you get unstuck. Cool. And I believe you have questions for us. Is that right? Yeah, yeah. Although, let's see. I think one of my questions is, what is it like to kind of be doing kind of embedded work from a consulting standpoint? How does it differ from kind of owning a project end to end? You're the one doing consulting right now. I do own projects end to end. Sometimes I own them more than I did as a full-time engineer. Yeah, it depends on smaller companies.

Definitely, that's true. But I'm sometimes the only firmware person, and they don't really know firmware. And so a large part of my job is making sure they understand enough that when they're done talking to me, they can manufacture whatever it is effectively. A lot of mine were prototypes for startups too. So that was kind of end to end if you define that as, okay, here's a prototype and if you guys want to continue this, then here's how to productize it. But, um, I literally had somebody bring me a napkin sketch and worked with them through

production. Um, but there's big company consulting too, where you, you know, you kind of parachute in for a particular problem or a subset of a system. Oh, yeah. Sometimes I get to be a fixer where somebody will be having crash problems or some, oh my God, we can't ship because over the air updates aren't working. And I come in at the very end and like wander around and break things and then they pay me wait no that isn't right i fix things and then they pay me um so yeah it's it's not some things are very different in that i'm quite independent they don't say write this uh serial module say, make this thing. So yeah, it is end to end ownership. Got it. That's really exciting. And another thing that I've been wondering, because you get to see

so many projects, is I think, I think in a lot of traditional or like you can imagine very small team environments, you end up having like an electrical engineer kind of write a lot of the firmware. And I think in today's world, as kind of embedded development becomes more accessible for people who have more of kind of like a cloud-based kind of programming background, like do you go into code bases and you're like,

okay, I know exactly the background of the person who wrote this code. Yes. Absolutely. Especially if it's a pick. Oh, yes. Especially if it's a pick. But yes, people definitely have accents based on their development and their code shows their accent. That's a polite, polite euphemism.

Dialect? Yes, absolutely. Um, and sometimes, you know, it's, it's good. And sometimes it's makes me want to tear my hair out, but I mean, the traditional thing is that a non-firmware software engineer who writes code doesn't do things. Doesn't understand pointers? No, no, that's very specific. Just large-scale program structure and data structures, and I don't want to say design patterns, but sort of patterny things, like this is how you organize a program in a way that makes it extensible or maintainable.

Usually it's just a pile of code that solves the problem. That's what I tend to see from that kind of stuff. We call it resource constrained. Thank you. I'm not talking about your code. Yeah, it sounds like that some code may be more or less procedural as well. Yeah, yeah, yeah. Yes. Or very brittle, like it's hard to extend or hard to make changes to it, to do anything beyond, you know, the five things that it's designed to do. Well, sometimes there's a flexibility versus reliability versus complexity thing.

Yeah. And people who come from bigger software generally are more into the flexibility and complexity. And people who come from medical devices or, or vehicles tend to be very, it's going to do one thing and one thing only, and don't mess with it. Do not make it more complex.

We don't need it more flexible. We need it to be 100% reliable. Um, yeah. So there's, I think it's a false dichotomy in a lot of ways. I think that, I think, well, sometimes you get better flexibility with less complexity. Right. Um, and so you, you just have to be aware of, it's very easy to write complex code that does very little and is totally unreliable and inflexible. That's called obfuscation. You know, one 500 line function, that's just a cascade of if then else. It's very complicated, but not the way you should do things.

Yeah. And I think to that end, like one of the things I've been wondering is, I think that as kind of an embedded engineer, you end think to that end, like one of the things I've been wondering is, I think that as kind of an embedded engineer, you end up having to be you talked about accents and dialogues, Lex, you end up having to be this translator between kind of different disciplines. What has that been like? For the most part, it's fun. I mean, I like communicating.

I like helping people communicate. And I had to learn a lot of hardware. Coming out of school, I had a lot of signal processing, so the math part and the CS part, but I didn't have the, what the hell do capacitors do, piece of the education and so that was a lot of learning on the job and learning how to learning what i needed to do my job and also how to communicate effectively with hardware engineers uh and you know how to explain that there are good cs ways to do things that may be different than the obvious way to do it. So, yeah, it's... Yeah, and I had to work with a lot of optical engineering and optical physics stuff in most of my contracts. So I learned a ton of stuff that I would never have learned had I been in larger companies doing traditional software. It's why I like Embed like embedded because you really learn a lot about the applications too. I mean, I know more about race car dynamics than any software engineer

ever should. It's just not, not something I need to know, but it sure was fun learning it. That's amazing. That's amazing. And one more question, if that's all right. Sure. So one thing that I've been thinking a lot about recently is kind of testing. And I'm curious how you divide responsibility for testing. Let's say you have like an entire project that's kind of, you know, shipping a piece of hardware, because you need to test the reliability of the hardware. But you also have to test the reliability of the code, and you're going to have to write code that kind of monitors both of them. Because testing is as horizontal, is there a clean line to draw between the folks? Do you just have a separate testing team? What setups have you seen, and what setups do you think make the most sense? Well, one of the most common testing methodologies is to build a wall between the hardware team and the software team. When the hardware team is done, you just throw it over the wall and the software

team says this isn't working and they throw it over the wall back. That is the least effective method. I have had the privilege of working with some extraordinary electrical engineers who will produce hardware, test it to the best of their ability, and then work with me to figure out how we can test all of the hardware pieces. And they recognize that it is a dual responsibility. The goal is to ship something. It's a team sport. And so you have to think about it that way. And when something is broken, when I know that the hardware is broken, when I can see that it is not working the way it's supposed to, and it is clearly a hardware problem, the thing I have to do is make it so the hardware engineer can reproduce it

so he can see it or she can see it and to make it obvious and reproducible because that's what I need when I have software bugs. I need someone to do a bug report that includes how to reproduce it. So, yeah, there is no line. There is no separation. There is between testing and development, though, a lot of times. Oh, well, and you have manufacturing tests,

which are often hardware tests to verify the system is built correctly. And you do have system tests, which may go to a different team. At LeapFrog, where we made toys, we had a totally separate QA team because the software would do something. But once you added all the sounds and you added all the system stuff, it wasn't just the software. You had to do an integration test of the whole system, which was often lengthy. My software just needed to deal with, say, a letter. But the QA people needed to make sure all of the letters were said correctly. So there are lots of different

ways to do it. I tend to think that if you aren't considering how the hardware is going to be tested, how your software isn't going to be tested while you are thinking about how to build it, you may be digging yourself a hole, a hole that's really hard to get out of. And so, I mean, I'm not going to, I am not the biggest proponent of test-driven design, test-driven development, that you have to write a test for every line of code.

But I definitely think you need to be thinking about testing just as much as you're thinking about the code. Got it. Got it. Cool. Thank you so much. Sure. think you need to be thinking about testing just as much as you're thinking about the code. Got it. Got it. Cool. Thank you so much. Sure. I thought you were going to ask us questions about your wand, but that is cool. I am excited. One of the things you mentioned in your Supercon talk was that the hardware aspect was new to you, that you have a software background. How is it different? What was the oddest part to you? Yeah. So first off, I should preface with, I feel like a new is always a relative term. So for me, I've been doing software for probably about 15 years now, and I've been doing hardware for, in comparison, more like four or five.

And I think that the biggest things that I had to learn in order to kind of make the jump is that, you know, hardware takes kind of time and budget. Sometimes you can trade one for the other. But there are a lot of skills that you need to develop in order to kind of work with time and budget. I think the typical software story is, I think a lot of software engineers really enjoy that kind of instant gratification. Like you type of your code, it compiles, you run it. That was definitely kind of what got me into programming.

And as I've moved like, you know, closer and closer to hardware, you know, my compile times have gotten longer, like things like that. But nothing like, you know, ordering something. And even if you use like Amazon Prime, it'll take like two, two days to arrive. And I think when I was originally kind of trying to do prototyping in my free time, I would try to figure out like the exact list of things I needed. And that would take a lot of time

because you don't like, yeah, that would just take a lot of time. And it wasn't until like someone made an offhand comment that was like, well, you need to order like three of everything you're going to use. And you're just going to assume that you're going to like throw away like half of the like things that you try to buy. And maybe these half can go into like future projects. So that's fine. But just assume that you're going to have like error margins in both like your budget and then also error budget, but also error margins in time. And in order to kind of, I've already talked a little bit about kind of error margins in budget, which is you just assume that you're going to like spend a lot more money than you plan to

within reason. But when it comes to like error a lot more money than you plan to within reason. Always a good assumption. When it comes to like error margins in time, that means you have to like sit down and like do a project plan. Like I think in software, you might do a project plan if you have a large team, but in hardware, you need to have a project plan, even if you're going to do a team of one. For this project, I definitely sat down and I was like, for this weekend, I'm going to try to hit this milestone. For this weekend, I'm going to try to hit milestone.

And in order to hit this milestone, I have to have like made these decisions and I have to have like these parts need to have arrived so that, you know, if I chose the wrong part, I'll have time for the next session. So it's just like sitting down and doing some project planning and knowing that it's going to kind of take time. I think once you kind of have that plan laid out, you kind of have sizable chunks. So you can like sit down and kind of enjoy your weekend building things as opposed to like sitting down and being like, I have like half the things I don't need, like half the things I have in front of me. And the other half, like I don't have yet because I didn't think about it. So it's just kind of planning on a larger time horizon. See, that's funny. You live in Silicon Valley, the only place that I can

think of that you might be able to find a store to go pick up most of this stuff. Yeah, yeah, definitely. I'm definitely familiar with a lot of kind of the Silicon Valley area stores. And I think that I'm extremely fortunate to have access to them. At the same time, they don't have everything. And you might realize that you need some random cable that you can only order online. I remember years ago, I did kind of hardware. It was called a make-a-thon, or it was a hardware hack-a-thon. And we just had to drive between hardware stores probably five different times because we were all people with software background

who didn't really think things through. What do you think you'll do next with the wand? Are you moving on to a different project or are you thinking about more features, more things? I think for me, it could go in either direction. Right now, I don't have very like concrete plans to kind of extend the wand other than like ways that I can help other people if they want to kind of extend the wand other than like, um, ways that

I can help other people if they want to kind of replicate or extend the project. Um, I'm definitely really excited if other people kind of approach me and are like, please give me advice or like, um, how, how do you think I can extend this? Like what tools would you make? Uh, what tools would you use? What frameworks would you use? But I think that one thing that I would love to do, if I had more free time is start to build like a database of like pre-baked sensor data, so that you don't have to build like, you don't have to collect all of your own data. And eventually you can like reach a volume that you can use for deep learning. So like the hard part about sensors is like the data that I collected is very specific to

like my magic wand and the way that you know, I attach things to the magic wand. But maybe that there are a series of cosplay props that you can do something similar to like, maybe I can use like maybe the same sensor configuration on the wand. And like on a Kingdom Hearts keyboard is similar enough that you could actually reuse the sensor data. I don't know, but I generally like having lots of data available so that you can start to play with more sophisticated machine learning algorithms. Yeah, no one ever said, you know what, we have just too much data. We just know what's going on too well. Yeah, I mean, I think that's where privacy kicks in, right? There's always this fine line between like, collecting data and being able

to kind of create what you want versus, you know, for me, I could just attach a sensor to my arm and just record my data for 24 seven all the time, but I'm not sure I would want that either. So yeah, it's always a trade off. So I saw your talk initially at Supercon on the videos, but we actually met at Bang Bang Con West a few weekends ago, which I think we talked about on the show. Yeah, we had Lindsay Cooper on. What made you go and what did you think? Yeah, so the way I found out about Bang Bang Con West was I was actually at the Recur Center earlier in 2018. And Recur Center is amazing. Everyone should go to it. I had so much fun. But so many people at RC or the Recur Center had gone to Bang Bang Con. So there's also like a Bang Bang Con on the

East Coast. And they were like, oh, it's incredibly awesome. It's really exciting because people are just excited about things. It's kind of this alternative. I feel like very often you kind of go to cons on a more kind of professional basis or to kind of learn about a technology. But BangBangCon is just about being excited about things and like loving computing. And I think that it's very easy to kind of lose sight of that in our day to day. So I really saw going to BangBangCon as an opportunity to kind of celebrate, get to know other people who are really excited about computing and learning things. And overall, it's just such a, it's such a positive crowd. Like I was just so incredibly happy to be there. You know, you kind of go and

you're like, I love everyone here. Why can't these people just, why can't there be more of these people everywhere all the time? So yes, this is a huge plug for BangBangCon, BangBangCon West, and the Recur Center, but I definitely feel this way about these events. Let's see. I think in terms of favorite talks, I already talked about Postgres Plays Pokemon. I thought that was really, really cool and really, really funny. I also really like this guy, this other guy kind of made this way so that you would take a 3D model and then you would convert that 3D model into instructions for building the model using a bunch of Lego pieces

I thought that was really really cool yeah it was awesome he came in with like a whole hat that he had made so I really enjoyed that I feel really bad because there's so many good talks. And if I don't mention the talk, it's not because it's not good. It's like, I just feel like all the talks were so high quality. I think another talk that I really enjoyed was there was a blackout poetry talk.

So I wasn't familiar with the concept of like blackout poetry, but I'm always interested in ways that you can combine technology and art. So this was kind of a way I think like, you know, machine learning was a part of it, but a way to kind of use like natural language understanding and some amount of kind of language processing to help people generate poetry. So I think the field of creative tools or machine learning to assist composers and artists is definitely evolving. And I love to see what people come up in this space. Yeah, you hit many of my favorite ones too. And, and the fact that it is a conference made of exclamation points is a good point. Uh, really describes it well. Um, yeah, it was a neat conference. And what did you think of Supercon? I mean, those are pretty different conferences and yet they're both

sort of celebrations of having a good time with computers. Yeah, definitely. Supercon was also a lot of fun. And I love it because it's also a lot of people who are just like really, really passionate about doing this stuff, even in like, yeah, like, people who who aren't doing it for like money or job or career, like they're just doing it because it's exciting and fun and inspiring. And I remember this year's Supercon, I was so excited about the person who did like an IC fab in their garage. It's like, oh my God, is this like possible? This is like the most amazing thing ever. Just everything. It's so incredibly inspiring. I think the key differences between the talks is Supercon is very much tied

to like the Hackaday community, the Hackaday crowd. And what that comes like, most people have more of a hardware background. Still very, very fun, even for someone who has no hardware background and just wants to get into things. For BangBangCon, most of the people have more of like a software background. So you're going to be talking to excited, enthusiastic, great people at both conferences, but their backgrounds are going to be pretty different. All right. I think those are good points about both conferences. But I also think it's about time to close up this podcast. Jennifer, do you have any thoughts you'd like to leave us

with? Yeah. I think it's very important to have discipline and patience. I think I talked a lot about kind of time. And I think people always kind of talk about time management. But I think it goes into project management and doing hardware and doing software and having the patience to kind of both plan out a project, but also have the patience with yourself when you're kind of learning and growing in these disciplines. Patience, I could use more of it. Our guest has been Jennifer Wong, a software engineer at Google. She is also a hardware prototyper and a maker. You can find her magic wand information at GitHub. Check out the show notes for the link and many others. Thank you for being with us, Jennifer. Thank you so much. It's been a pleasure.

Thank you to our Patreon supporters for Jennifer's mic stand and for the excellent lightning round of questions. Thank you also to Christopher for producing and co-hosting and thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm and now a quote to leave you with. I'll take breath before i start it i'm sure you'll recognize this mr and mrs dursley of number four privet drive were proud to say that they were perfectly normal thank you very much they were the last people you'd expect to be involved in anything strange or mysterious because they just didn't hold with such nonsense. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting

company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.