Embedded - 327: A Little Bit of Human Knowledge

Episode Date: April 9, 2020

Daniel Situnayake (@dansitu) spoke with us about machine learning on microcontrollers. Dan is the author of TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. You can read the first several chapters at tinymlbook.com.  TinyML is a part of TensorFlow Lite. See the microcontroller getting started guide. Dan works for Edge Impulse (@EdgeImpulse) which is making tools for easier machine learning integration at the edge. Their tools are free and they also have a getting started guide. Dan recently posted on the Edge Impulse blog about training a TinyML model to capture lion roars. For TinyML meetups and a forum, check out tinyml.org Lacuna Space: low cost sensors transmitting to space

Welcome to Embedded. I am Alessia White. I am here with Christopher White. We have another show about machine learning for you, but this time, definitely Embedded. It's on microcontrollers. Our guest is Daniel Sitaniaka, and we are talking about TinyML.

Hi, Daniel. Thanks for joining us. Hi, good morning. Could you tell us about yourself? Absolutely. So my name is Dan. I'm a software engineer, and I am from the UK originally, but right now I live in Sunnyvale, California. I've been working in a lot of different areas across Silicon Valley really for the last decade or so. But what I'm doing right now is sort of like what I've always wanted to do my whole life, which is play with artificial intelligence and machine learning and hardware. And I'm super excited about it. it and yeah i'm looking forward to talking

more about this topic today before we do that we want to do lightning round where we ask you short questions and we want short answers are you ready yeah i'm ready do you like to complete one project or start a dozen? Hmm. I guess this is where like reality and my goals collide a bit. I think with, I'm pretty good at like in each area of my life, I generally have one project. So I usually have one cool technology project I'm working on or one other random hobby i've picked up um but i tend to have a lot of different areas that i have projects ongoing in um so right now i have i just bought a sewing machine and a bunch of um hepa filter material to sew some

um face masks i have uh this morning woke up at 6 a.m and started playing around with this awesome um satellite-based laura um system called lacuna space um and that's my my kind of uh embedded project that i'm working on right now um and i have a huge list of other things that I'm kind of chipping away at. So I guess I'm in the camp I wish I wasn't in. They're both good camps. They are. They have to be. Do you have a favorite AI or ML algorithm?

Favorite algorithm? I mean, honestly, I think just the fully connected deep learning network is such a crazy, amazing thing. very simple bits of state and some fairly simple rules and it's able to learn to represent much much more complicated things um through through what is basically a fairly straightforward set of calculations is is kind of mind-blowing um i think when when i started to learn about this stuff and i i think i have a thing generally in life for um these sorts of like complex emergent properties where you have a lot of a lot of little simple things that add up to something very complicated um and that is able to represent all kinds of subtle and amazing um phenomena and and fully connected deep learning networks are basically an amazing example of that. So I just find that whole idea really cool.

What is your favorite microcontroller? My favorite microcontroller? So actually, we've been doing a bunch of work with microcontrollers based around ARM's m um platform at the moment and uh for for what we're doing with embedded ml those really hit the the sweet spot that sort of m4 type microcontrollers are generally like they're they're shockingly powerful considering how small and energy efficient they are um are able to run these types of models that are doing really quite amazing things um but still get incredibly good um potentially get incredibly good energy consumption um so i'm really enjoying that there's also a sort of lot of platforms or a lot of dev kits

around these microcontrollers so um it makes it easier for what we're doing to be able to um create code that will be runnable by people who are building all sorts of different cool stuff if all of the computers in the world suddenly disappeared what would you do hmm i mean i guess it depends is it is it that they never existed in the first place and i i just would have had a different path in life or would it be that like okay cool so in that case i don't know you know i i always think about what would i do if like i make a bajillion dollars and can just have a whole restart in life. And I think I'd like to sail around on a boat and like investigate marine mammals.

I think that would be pretty cool. I'd like to be a marine biologist. I don't know why. I've always been fascinated by the ocean and the mysteries contained within. But I think that would be pretty hard without computers so um i'd have to get some some sailing skills i think yeah okay uh let's talk about your book you you co-wrote a book tell me what it is yeah so um we wrote this book, TinyML, over the last year. And it's basically a book that is designed for people who want to get started with machine learning on embedded devices.

But it doesn't make the assumption that you know anything about embedded devices or machine learning. So we had this kind of challenging remit of trying to figure out how do we explain embedded stuff to ML developers and ML to embedded developers and both of those things to people who are just interested in the topic. It does assume some knowledge around programming, but not anything too intense.

And it was a collaboration between um pete warden who leads um embedded and mobile uh machine learning stuff at google and he's the originator of the tensorflow light for microcontrollers project um and myself how did And myself. How did you split up the work? So we came at it with the sort of like goal of splitting between tutorial and introductory content, which is something I focused on. I've done a bunch of stuff in education in my career, just sort of, I don't know, my mum was a teacher and I kind of like feel pulled in that direction a lot um and uh so so i love writing tutorials and explaining how things work and doing introductions to stuff and pete was able to get really deep on like the the philosophy behind um the tensorflow life and microcontrollers project and the types of things you need to think about when you're building a project and really bring in his deep experience on building these types of mobile and embedded applications and learning where all the points of friction are.

It's a very project, hands-on sort of book. It's less theory and more here's a recipe, go try it, and here's how to tweak it. Which is your favorite project in the book? So I think my favorite project really has to be the vision one. And the reason is just, it's absurd that this stuff is possible. We were writing the book while we were still working on the vision model and while we were still working on the tooling that would enable it to run and so we we're kind of doing something that no one's really done before very few people have really done before while writing a book about

how anyone can do it and it's just sort of incredible that like we're living in this time and place where you can have a microcontroller that's consuming like a couple microwatts of power while attached to an image sensor that is capturing i think it's a 96 by 96 grayscale image we're able to run inference on those images and tell you whether a person is in the frame or not which is like such a high level crazy question to be able to answer like does this picture contain a human being um and it's being answered by this tiny device that can literally run for for weeks or months on a tiny battery. And it costs very little money.

And I think, like, just because it's so easy to understand that, like, vision is hard. Like, this is something that's really challenging. someone asks you to like whiteboard out a solution to a problem you wouldn't want someone to ask you to whiteboard out like detecting a person in a an image it's not something that like you're going to be able to sit down and hack out a algorithm to approximate very easily there's just so much so much variety and like what does a person look like um what do people look like from any possible angle what do people look like in different types of surroundings and with different lighting and what do people look like wearing different clothing and all this type of stuff and somehow we're able to to fit all of that knowledge all of that understanding of what is a person visually into a model which is like a couple

hundred kilobytes and runs on this tiny little device and i just think that's incredible so um it's really exciting to me did you start with the notion knowing you could do that or did you just say i bet we can and work toward it. I mean, presumably you did simulated small models on full computers. Yeah, absolutely. So there's not any real difference between running a model on a microcontroller versus running on a super high powered server. It's still the same type of architecture and the same kind of code. It just needs to be able to fit on that platform and run within a reasonable timeframe. So we know that you can create a small enough vision model that gets decent performance. You have to think about scoping out your task a little bit. So if you think about cutting edge

vision, vision models are generally doing some kind of task that is across multiple classes so maybe you're you're classifying the content of an image and you're wanting to decide like which of a hundred different objects could be in that image and that's going to require a much bigger network than being able to decide whether an image has one object in it or not gotcha um so scoping the problem like that is really important um and we kind of knew that you'd be able to do that and we obviously at google working with like incredible people who um like really have an amazing deep understanding from a research point of view about what's possible what's feasible to aim for um and then the tensorflow light for microcontrollers framework essentially it's built in this modular way where a deep learning model is composed out of lots of different operations

and these are just like mathematical transformations generally where you take an input tensor basically an array multi-dimensional array transform it in some way and spit it out the other end and we knew that we have implementations of the right operations within the framework to run these types of models so we had all the ingredients but it's just a matter of getting it all together and it's still a matter of getting it all together and it's still this kind of magical moment where you first push the thing to the device and it it works um and that stuff isn't possible without a lot of hard work from a lot of people doing everything from the the like integration work get it bringing up the board getting the camera sensors working it's like people across multiple companies working in multiple fields all coming together to like have this artifact that's able to do something

amazing um but once you've done that once you can reuse all of those pieces that you've developed and it makes it easy for people who don't need to have all of that that knowledge to pick it up and start building cool stuff. You were at Google when you did this work. And you mentioned there are lots of people contributing to it. TensorFlow and TensorFlow Lite are both open source projects that Google has sponsored. How does that work inside Google? I mean, you're not making any money. Yeah. So that's, that's a really common question that we get, um, uh, while I was working at Google and it's, it's sort of like Google, Google basically is this big, crazy company that does

loads of amazing stuff. And it's so huge and has so many different teams that there's the potential for a lot of duplication of effort. So imagine you've got a team that's working on creating machine learning models to forecast the weather. And then there's another team that's working on machine learning models to put on embedded devices. And there's another team that's working on machine learning for powering digital assistants or checking for spam in your email. It would be an absolute disaster if each of these teams has to implement its own deep learning libraries from scratch. And that's something that is like pretty much inevitable to happen when you have this giant company that's spread across pretty much every continent except

Antarctica and like loads of smart people who love reinventing the wheel. And so I think TensorFlow, the main reason TensorFlow exists is because Google needs a awesome, like world-class deep learning framework that everyone in the company who needs to use deep learning can use. And so because TensorFlow exists, if you work at Google and you need to do some deep learning work, you have this amazing tool set available to you and you don't have to reinvent the wheel. You don't have to build it yourself. And so just because Google wants to do so many awesome, cool things that are based around AI, it makes sense for Google to create TensorFlow. And if Google's going to create TensorFlow, it makes sense for Google to give it away because there are just so many big advantages to open

source. So, you know, it's like you get get community contribution that's a part of it but you also get to expose what you're building to the world and get feedback and understand what people's needs are in situations other than your than your own um it allows google to like build services around tensorflow so you can like use some google cloud service to to run your tensorflow model um and and google is able to make some money from that. And it just is like a good way of giving back to the community, engaging with the researchers who are doing the amazing work that a lot of these products are built on. So with TensorFlow Lite, did you work on the machine learning side or the embedded side?

So it's interesting because it's kind of like a blurry distinction. So with TensorFlow Lite Micro, so TensorFlow, main TensorFlow, it's sort of like this big set of tools that's originated for running machine learning, training machine learning models and running them on servers. Andflow light was created as a um sort of subset of tensorflow that allows you to do stuff efficiently on mobile devices um so like android and iphone and that type of thing and then tensorflow light for microcontrollers was spun off as like a sub component of that and so um it is separate i was confused so it's the team it's sort of like a russian doll like tensorflow light for microcontrollers team is inside of the tensorflow

light team and the tensorflow light team is inside of the sensorflow team and then the tensorflow team is inside of the google research team um who make use of all these tools. But so what we're doing on the micro side, we're such a small team that it's sort of like everyone's like knee deep in everything and working really closely together. So we had some people who are focused on ML and training models, and they're the folks who would create, for example, the vision model,

which was something where it's like, hey, this isn't something that's been done a lot before, making a vision model that's so small. And so we had dedicated ML folks working on that, but then they work really close with the engineering team to make sure that that stuff can actually run on the microcontrollers and that we have good um operation support and that type of thing and then we'd also be working on the embedded side to make sure that we've got platforms that this stuff works really nicely with

so we work really hard to make sure that this stuff all runs on on a few different widely available developer platforms so like a lot of the examples in the book work on arduino they work on spark fun devices um uh we we actually work with spark fun to develop a device that um specifically was well suited to running these kind of low low power ml tasks um and uh there's there's now a bunch of devices that uh the examples that the tensorflow light team has created are compatible with and so i kind of sat in the middle of all that um i actually was um my role on the team was developer relations so i'm kind of the interface with the open source community and with the the wider developer community and my job is to sort of build content and samples and examples and communicate to the community

the kind of cool stuff that they could do and that's both on the embedded side and on the ml side and also take the community's feedback and integrate that into what we're doing engineering-wise. So that might be fixing bugs. It might be adding features. And it might also be just like providing guidance to the wider team on what we should be working on. So that's the kind of crazy developer relations role where you're an engineer, but you're also deeply immersed in what's going on in the community um and so i ended up covering stuff from um the the ml side all the way through to the the darkest um corners of poking around trying to get uh audio encoding to work correctly

for feeding into a model when you say Arduino, you don't mean the Arduino Uno with its tiny, tiny processor. No, unfortunately not. But the devices that we're targeting on the Arduino, so the board that we sort of recommended people pick up to have a nice experience getting up and running is the the um arduino nano ble 33 sense or i think i said some of those words out of order um but it's basically um uh cortex m4 i think um core and uh it has enough memory and um a bunch of onboard sensors that it makes it a nice dev board to just quickly get up and running with this stuff. It has a built-in accelerometer, so you can just pick it up and start.

You can upload the magic wand example that we walk through in the book and immediately get some results. So it's nice, especially when there are people from an ML background who don't necessarily want to do any soldering and don't know what a breadboard is. Having these types of platforms where there are some integrated sensors makes it really easy to get started and get a sense for what's possible. Because it's not always intuitive. No, it isn't. But let me state the info on that chip. It's the Arduino Nano 33 BLE Sense, and it has the Nordic Semiconductor NRF2840, which is a Cortex-M4.

It doesn't say M4F. And it's got Bluetooth. The other board that is often mentioned in your book is the SparkFun Edge, which has the Ambeek Apollo 3 Blue microcontroller, which is another Cortex-M4F. And it's specced to run TensorFlow Lite using only 6 microamps per megahertz. And it normally runs 48 megahertzhertz 96 megahertz in burst mode and so the claim is that you can run it on a coin cell uh cr 2032 for 10 days uh which doesn't sound like much except there are much bigger batteries out there um it doesn't sound like much, except there are much bigger batteries out there. Mm-hmm. It does sound like this would run on a Teensy, too. I think anything Cortex-M4F.

Yeah. Okay. There's a list, which I'll put in the show notes, of boards that are recommended. It's interesting that you have the book, which is nice to have it as a book. But much of the information is also available online. I mean, the TensorFlow.org light microcontrollers page lists the processors and does have links to the projects and tutorials. And they seem to be the same projects and tutorials. Are they different? Why should we buy your book instead of just reading it? Well, I mean, so we didn't write the book to get rich and make tons of money. That's a good thing to say. I'm sure the movie rights are the option soon

um yeah unfortunately uh we we actually just wanted to make sure that this information and like these types of tutorials are out there in a form that's accessible to a lot of people and we've actually got the first six chapters of the book available on tinymlbook.com for free. So you can get a sense for like what's going on in this book, what's possible and actually do some of the tutorials without having to pay anything. You can just grab the PDF. And our goal is to have all the content in the book be available online in some form or other in addition to in the book be available online in some form or other um in addition to in the book because we want people to start using this technology we want more people to understand what's possible on embedded devices which is definitely something i'd like to kind of delve into a little bit later

in the the um conversation and i i just really think that it's important that we get these tools into the hands of people. And so we've got like a lot of this info is duplicated on the TensorFlow docs. We've got, Pete's actually done some amazing screencasts based around the projects in the book that are on YouTube. And we're trying to engage as much as we can with the community to share this type of stuff. And you have much of the model generation and information in Colab documents, which I thought I hadn't seen before your book, but I had a bunch of them when I went. I mean, I'd saved a bunch of them that I never looked at, apparently. But it's like Google Docs, but Jupyter Notebooks. It's a way of looking at code and even running it in a webpage. Could you explain Colabs?

Yeah, absolutely. Maybe better than I did. Yeah, I mean, I think that was a really good explanation. So imagine, I mean, if you're familiar with a Jupyter Note notebook, it's basically a kind of interactive document where you can mix just regular plain text and you can drop in Python code as well. And it's in the form of blocks. So you can write a paragraph of explanation, you can add a block of code, and then people who are reading through the document can run the code and see the output and so it's this amazing tool for all sorts of things but where we're using it is basically a way to write a tutorial where you go through read the tutorial and then you can actually run the code that it's teaching you about in place and see what it does and experiment with it and play with it um and so colab is a platform that runs jupyter notebooks you can also just download the files and run them locally um but colab provides an environment with a nice beefy gpu that you can

train models on really fast and um it takes care of all the ugly stuff like dependency management one of my pet hates is python dependency management yes it is the worst thing in the engineering world i think um and the nice thing about colab is that all of that stuff is just abstracted away and you can um you can run terminal commands through colab if you need to sort of install specific packages and stuff like that. But everything's encapsulated within this document sort of setting. So you don't have to worry about setting up

all kinds of wacky Python tooling on your machine, which is a big barrier to entry for a lot of folks who don't come from the Python world and want to play with this stuff. So it really makes stuff a lot of folks who don't come from the Python world and want to play with this stuff. So it really makes stuff a lot easier. And so the other cool thing is you're able to display graphics so you can output graphs that display model metrics during training. You can visualize the results of what you're doing

and all of that gets shown visually. And then you can download files from the Colab. So we use it as a way that you can have a document that's like a tutorial that shows you how to train a model but then you can actually download the model from it at the end and you can make your own changes and download your own model and then just um put that on on device so it's it's like a kind of step towards making this stuff more accessible for more people i think it's a successful step um i i definitely in machine learning suffer from the blank page problem where i just don't know how to get started i know all the words i can tell you what i want to do but faced with data and code i don't know where to do that but i can easily modify other people's

code which is why these collab things are pretty cool yeah absolutely and that's always the best way to learn i think is when you've got something that works and you can see sort of what the different pieces of it are doing and you can just start to fiddle with them and try to get it to do what you want it to do. And I think that's a really successful model for getting people to understand this type of thing. So that's all the good news. Now I have some questions because as much as I did enjoy your book, I didn't actually put it on a microcontroller for a few reasons. And thinking about projects in the future, I'm not sure that this is a path that I want to go down. So the SparkFun board, the SparkFun Edge edge it gets some really bad reviews about how the yes no doesn't work and how the machine learning doesn't work does it does it really work i mean

is it only work if you if on on full moon nights okay so this is where we come into the the sticky questions about ml because everyone's heard so much about machine learning and ai and how it's gonna take all of our jobs and how it's gonna make the world a completely different place overnight and then you you come to use ml products i'm sure everybody's had the experience where they've like got out their phone, tried to use whatever digital assistant is in there. And it completely misunderstands them and calls someone they have not spoken to for years instead of setting a timer.

That's a very specific. And yet I think it's happened to me. Yeah. I mean, every time we do a talk at Google and we talk about the google assistant we have to avoid saying that the wake word for the google assistant because everyone's phone will suddenly wake up in the audience um and and start talking so that like ml ml is a a specific tool and it is it has limitations and it it can be applied to certain areas very successfully and other areas not so successfully um and the big thing that is underlying everything

with ml is that it's completely dependent on data you cannot have good machine learning models if you do not have good data. And what good data means is actually quite hard to pin down. So you mentioned the speech model in the example is trained to recognize yes, no, or decide whether it heard an unknown word. And so this little model is basically taking as its input some transformed audio. We actually run a MFCC, get a spectrogram that is representing the audio, the raw audio that's captured by the board's microphone, feed the spectrogram into the model, and the model spits out whether it who've contributed to an open data set, which has, I think, about 10,000 total samples. out there and they all have different voices and they all say words in slightly different ways and they all have slightly different ambient noise going on around them and um they are all using devices that um maybe the the microphone placement is slightly different between um one one dev kit or another maybe um there's some kind of interference from from stuff going on around them so the the conditions that your device is in when it's trying to run inference which means when it's trying to make

a prediction based on the model could be very different from the conditions that the um the data was captured under and so the the issue that um exists with that model that's flashed onto the board as sort of the like hello world is basically that it wasn't trained on a diverse enough set of data to represent the way everyone talks and all of the types of background noise that are happening while they are trying to get this thing to work. So we've actually, so i've recently been like playing around with audio models and trying to get this same size model to work with better performance and the way that i've been able to do it is just throw in more data so i've got a load of samples

of background noise and like um i i added another category that the the model's trying to classify which is just background noise um so rather than trying to shoehorn things into yes no or an unknown word we're also looking at whether something is background noise um this allows the model to to understand how to segment those things a little better. But essentially, without pouring in more data, without getting more data, more recordings of people saying yes or no, there's a limit for how accurate the model can be. And so that's something that's really important to understand with machine learning. It's something that makes it quite hard to do these types of example applications, because you're publishing an

example with some open source code. You don't have thousands of dollars to spend paying people to record audio. You don't have loads of money to license a data set from a company that builds them. You just need to somehow figure out a way to pull this data from the community and then make use of that. And so it's challenging to show what's possible because the types of data sets that are available to people who are building examples and building toy projects are not the same as what might be available if you're building a consumer quality app and so you you can't necessarily expect the same results but the nice thing is over time these types of data sets do become available and there are like really big data sets for example for vision that are available for free and there's a bunch of them and they have their own problems um there are a lot of problems with like non-representational data so maybe for the voice example maybe you have a lot of um voices with american accents but not many with british accents that's a problem i struggle

with every time i use any kind of speech recognition thing um so that means that your model is going to work better for people who have that sort of modal characteristic that your data set represents. Um, and, and this is always a challenge when you're, you're building projects with ML, but it's important to know that that doesn't mean that ML isn't useful. It just means that you have to set your expectations and you need to collect as much data as you can. And your expectations can actually get pretty high if you have a good data set. It's possible to be really accurate

with speech recognition or wake word detection. And you can tell that by the fact that when I'm giving a talk and I talk about wake word detection and I say the dreaded phrase, okay, Google, everyone's phone is lighting up in the audience because they have got an embedded wake word detection model working that works extremely well. If you have lots of data, you can can do amazing things and for easier tasks like speech speech detection is and speech recognition is actually quite challenging because of the diversity

of things but there are a ton of areas where um you're using less diverse data um less kind of um variation between different types of input and you'll be able to get get much stronger much more dependable results and with a much smaller data set so for example um the company that i i'm now working at which we'll talk about in a moment uh edge impulse is basically building easy tools for developers to use to train deep learning models without having to have any experience with doing so and we've got a couple of tutorials and examples that show how you can collect like a couple of minutes worth of data for like gestures using an accelerometer or of audio for classifying different types of things. Like we have one

example with classifying whether a faucet is currently running or not based on the noise. And you're able to get good results with a really small data set, like 10 minutes of data, something like that. So it all depends on the scope and the type of your problem, what kind of results you can expect and how big a data set you're going to need. How do I know if it's the data set or the tininess of the model? How do I know if I have a big data, if I have a lot of data, how small can I make my model? Or if I have a very, yeah, how do I tell if it's the data or the smallness of the model? Yeah, that's a really good question. And so the way you would do that basically is train a big fat model using all the resources at your

disposal, run this thing on your laptop, train it, train it, and see if you can get something that performs well on a more powerful device. And then you know that it's possible with that data set to accomplish the task that you're trying to do. And if you find that even with a big model using state-of-the-art approaches on a powerful machine, you're not able to do this, then you're not going to be able to do it on a tiny device. And if you do find out that it works um then you can start reducing the size of the model so you can start chopping bits out removing layers reducing the size of layers and see if you can maintain that performance as the model gets smaller and um you it's really an iterative process of sort of making little adjustments seeing the impact they have and repeating until

you have something that works. And that's generally what the ML workflow looks like. So there's a really big difference between writing an algorithm and training a machine learning model. And when you're writing an algorithm, generally you sort of figure out what you need to do. You figure out what your inputs are. You identify the processes that need to happen in order to get from your input to your output. And then you figure out how to implement that on the device that you have access to with the APIs you have access to. And with machine learning, the process is very different. It's very iterative and you don't

necessarily know if it's going to be possible to do the thing that you want to begin with. So really your whole development process goes from planning and implementing to exploring and testing and trial and error and tinkering and trying to figure out if something is going to work or not. And that's definitely not the normal way we do the type of engineering we're talking about. If we're trying to develop a heuristic for something, generally there's some domain knowledge that we know that can be like expressed in the form of code and we can use that type of code to figure out um a way of of knowing what's going on in our application but with with a deep learning approaches or machine learning approach it's more

like um you have to go through this whole discovery phase of of trying different types of models um trying random changes to stuff literally random changes to stuff to see what kind of impact it makes like there's a an idea called hyper parameter optimization which is essentially the idea that um that your model has a bunch of properties, high-level properties about it that you can change. And certain combinations of high-level properties will enable it to perform well, and certain combinations will mean that it doesn't perform well at all. And the only way really to discover those, you can use some intuition, but the only way to really discover them is to kind of test a bunch of them out. And one effective and widely used way of doing that is just randomly picking parameters and training a bunch of different models and then evaluating them all and looking at which one works the best and using that one. So that's so different from what we usually do as engineers.

It can be a little bit unpleasant to think about and counterintuitive. It makes it a lot less easy to plan how long something's going to take to do, which is obviously not easy in the first place. I find this fiddle with it until it works method, extremely objectionable. I distrust it enormously. And there is an element of randomness in building the model, as well as choosing all of the parameters and choosing the data and choosing the size and the structure of the model. And in your book, you have some different options for figuring out RAM size and deep learning network size. And it was just try it until it works. And if it works for me, it may not work for you. How do we formalize these processes to, to stop hand waving our way around this? It it doesn't it doesn't feel like engineering to me yeah so so it's interesting because there's there's a few different ways you can think about this so like on the one hand machine learning it's it's innately sort of probabilistic

it's a statistical tool and so inherently there is some degree of of uncertainty involved with this type of thing um you also can't know from looking at data whether it's going to be able to result in a a good model or not there are clues and there are ways you can explore the data that let you know hey is this going to converge well or not um For example, in Edge Impulse, our tooling can give you a visual sense of the way that the data points in your data set relate to one another, and you can see if they separate on a graph. And if they separate in this three-dimensional visualization, then you can see that the neural network can start to learn

to recognize the distinct clusters of data points and relationships that you might have. But there's always going to be this aspect of exploration and a bit of unknown. Exploring the data makes sense to me, about visualizing it to make sure it's separable, at least in some axes. But, I mean, the data exploration, I totally get.

The algorithm exploration, I just... Well, we've experienced that ourselves. I mean, you have a spreadsheet of the parametric search that I can go find on our server right now for trying to figure out which of the networks for work we've been working through. I know. Your objections are not directed. Oh, no, they're not.

You're directing it for the ML community. Yes, very much so. But it is worse with tiny ML because there's another variable. Another axis. Yeah. And I just, I know that Dan's book doesn't try to formalize it. I know that Garon's hands-on TensorFlow, et cetera, does try to do a little bit of formalization with building intuition as well as making predictions based on some of the statistics parts um but i'm still it's it for me it's the worst part about ml is is fiddle with it until it works yeah so so so parts of that are to do with the kind of maturity of the tools like the the

important thing to think about right now is that um tensorflow light for microcontrollers for example it launched around this time last year um it hasn't been around six seconds in the ml community exactly yeah um there must have been um oh sorry there are definitely a lot of rough edges still there are bits of tooling that can exist but don't so a good example there is like um part of the the process of getting a model up and running on a given device is to figure out how much scratch space it needs for calculations basically we call it the the tensor arena um and it's it's basically the memory that needs to be allocated um for these the the storage of tensors that are produced during intermediate calculations throughout the model. And that's something that we can just figure out.

You know, that's something you could look at the model and look at the op implementations and know how big it should be. But that's just something that the tooling hasn't been built for yet. So some of these problems are going to go away over time. And the work that we're doing at Edge Impulse around tooling, that's one of the things we're looking at. It's like, how do we make it so that you can just drop this thing in as a library and not worry about that type of stuff? And that's definitely doable.

Some of this just comes out of immaturity. But there are other things like when you're designing your model and experimenting to try and get it working, that's always going to involve some degree of trial and error. But the thing that you can do is rather than doing the trial and error yourself as a human being, you can get the computer to do it. And so what we're trying to do with our tooling at Edge Impulse is basically have you upload data and then we make a bunch of smart decisions on which model to use and how big the model should be. And we can test a bunch out and figure out what works the best and just give you a recommendation and then if you want to deviate from that if maybe the model we recommended

is a bit too big for the amount of memory you have to play with you can adjust the levers and get something different but as an engineer you don't want to have to think about that kind of rubbish like you just want to basically feed in some data know whether the thing you're trying to do is feasible or not and what kind of accuracy you're going to get and assuming it works nicely you want a library to come out of the other end that you can integrate and that's what it should look like um all of this uncertainty should be taken away abstracted away so that people who are just trying to solve a problem don't need to think about it.

I'm also sort of bothered, and this is again more to the machine learning aspect, I'm bothered by using a neural net to do things on a microcontroller that maybe domain knowledge and heuristics would do better. I mean... Yeah, how do you decide? PID controllers has been the thing for me.

Could I replace a PID controller with a neural network? You've been trying that. Well, I stopped once things happened in the world. Right. I just couldn't... That's not ML the world. Right. And just couldn't. That's not ML's fault. No. Well, we don't know that, do we? Well, that's true.

Anyway. There's probably some conspiracy theories. As somebody who really enjoys gaining the deep domain knowledge and working on really efficient algorithms, I'm totally kicking and screaming with machine learning. Is it... Should we inform our client of this? Are there tough but understood heuristics that compare to developing and training a neural network?

So I think my guidance would be really, if there's an algorithm to do something use the algorithm you know this stuff is hard it's messy um and it's complicated and you can abstract a lot of that away and make it easy for developers to do but it's still going to be hard for your poor little microcontroller that's like trying to instead of just directly calculate something it's basically across multiple layers of a bunch of complex matrix math trying to figure out an approximate version of the answer you know like if you if you can help the program out by telling it exactly what to do, you should absolutely do that. And so I would say that if there's an understood heuristic for something, if there's an existing algorithm for something,

or if you have access to the domain knowledge that it takes to build that efficiently for your subject area, you should probably err on that side before thinking about machine learning. But where you would want to use machine learning is when maybe there isn't an understood heuristic. If you know the problem, but it's very difficult to describe in a an algorithmic way or a simple way how to solve it maybe ml can figure out how to solve it for you and so speech is a really good example of that so like if i say the word yes there are certain things about it that you could probably write an algorithm very quickly to recognize. Like when I specifically say yes, there'll be like a certain pitch that my hissing sound at the end of it makes. There'll be like a certain length it might have if I was talking normally.

And you can figure out how to do that. But what if you then want to identify when someone else is saying yes it might sound completely different those things you've identified just go out of the window and once you try to do that across an entire population of people across everyone on earth um imagine it took a month just to say that a random amount of time imagine it took a month just to say that a random amount of time, imagine it took a month to figure out the heuristics for recognizing the word yes, um, for most people. Um, and, and you wrote a nice hand-tuned algorithm to do that. Uh, that's great. But then when you want to match the word, no, you're going to have to spend that time again. And then if you add another word, it gets even more difficult.

And then the more words you layer on top, the more collision you're going to have between your rules. And so it gets very, very difficult very fast. And so what you can do if you're using an ML approach is just provide this data and try and train a model and if you can train a model that works well that might be something you can do in like a day or two like or maybe even a couple of hours is get a sense for whether you can get a reasonable way towards solving the problem with ml and then maybe you might take a month after that of fine-tuning your model and getting it to to work at its absolute optimum and collecting more data and feeding it in. But then that model isn't going to care so much whether it's recognizing yes or no or up or down or other commands that you want to feed in because it's potentially able to generalize um and and be flexible in that way

and so you have to be really careful to identify these types of problems where machine learning is is going to help whereas uh in some cases it it is just going to be a way of doing the job approximately when you could have done it specifically and precisely. And it might take a lot more computation to do something that could be expressed much more simply. So there are some kinds of things that it's really well suited to and others that it's not. And so when you think about what are those things, like recognizing messy stuff, it's really good for, recognizing spoken words, recognizing what things in the real world look like, recognizing the dynamic state of motion of an item, whether it's someone going for a jog versus swimming in a smartwatch, or whether a machine is operating

correctly based on vibration. Those are the types of things that it's whether a machine is operating correctly based on vibration those are the types of things that it's really ugly and difficult to try to describe in code there are so many edge cases and like bits of weirdness and noise and variation in the types of signals you're getting and ml can be a tool that can help sift out the signal from that noise, which is essentially what it's always doing. It's sifting signal from noise. And if you can do that already, you don't need ML. But if it's too much of a challenge with handwritten code, then ML is a really cool tool. So as Chris alluded to, I am using a lot of TensorFlow and ML in work, but I'm not doing the tiny part. We have a pretty big processor. I don't have to worry too hard yet. I want you to do this. Is it ready to go? Or is it still in things are changing so fast, I may not be able to be able to rebuild my code in the future?

So it is absolutely ready to go. And I think the biggest example of that I can think of is in the fact that it's deployed like this exact use case, wake word detection that we've been talking about so much is deployed across millions of Android devices, maybe billions. I'm not even not sure. But like a huge number of mobile handsets have a chip that is using some kind of machine learning algorithm to wake up the phone if it hears the right words. And this is something that works today. And there are tons of companies working on products that are incorporating ML. We're still early on. So like a bunch of those haven't come to market yet, but there is a lot happening and a lot of excitement in this space from people who are creating hardware products, people who are building solutions to problems that are really difficult to address without this type of on-device understanding. And the key areas where it makes the most sense to deploy ML on device are where you have limited connectivity,

where you want to limit the amount of communication you do because of energy constraints, and where you want to guarantee people's privacy. And those are things that are becoming more and more important over time. And embedded ML has arrived basically at the exact right moment where people are starting to care about their privacy people don't want all of the things they say being streamed into the cloud for some mysterious server to digest if you can leave data on device then everyone is happier and you get better latency and better performance and better energy use and there are just so many so many applications for this and the the ml side the technology is there the models work if you have the right data set then you can do amazing things and on the embedded side the tooling is getting

there too and um projects like tensorflow light for microcontrollers are like this amazing low-level fabric that enables this type of stuff. And then there are collaborations going on between companies like Google and ARM, where ARM has worked really hard on getting all of the SimCIS and optimizations into the TensorFlow Lite for microcontrollers library so that on ARM devices that support these APIs, you get super fast inference. So if you're a developer, you don't have to worry about doing these really efficient low-level implementations of ops.

You can just make use of these amazing tools. And then companies like Edge Impulse, where I work, we're building the high-level layer that allows you to really quickly go from a data set to a model that you can deploy as a library to a device. And the types of use cases and types of amazing things that people are doing, like we don't even know what is going to be around like a year or two from now because domain experts people who have that that domain experience are able to now get their hands on these tools and start building

things and experimenting and come up with like incredible applications of this technology that that none of us who have been building the tools would have even thought of but that was my next question where do you see tiny ml going in a year or five years do you have any expectation i know you just a second ago said you don't know but do you have any fun things you've seen or interesting projects that we should be looking for yeah so i mean i would say that like where we're going in in the next five years or even in the next year is things are going to get a lot easier to use and so you're going to see so many people who have incredible domain expertise about the problem they're trying to solve suddenly having access to these tools

that can make a really profound difference in how quickly you can build something and get something working and be able to do something really impressive on device. And so in basically every area, we're going to see ML start to touch things right at the edge. So one of the things I've been most interested in is in conservation. There are a bunch of organizations and companies looking at how can you use machine learning to classify animal behavior or monitor an environment and figure out what's going on in the natural environment on a low power device that can occasionally report data back um so imagine you've got a, we actually wrote a tutorial about this recently. Imagine you're

looking at animal behavior and you have an animal that makes a certain sound, like a lion roar was our example. And you want to count how many times a lion roars during the day. And maybe that's a good proxy for some aspects of lion behavior or health or something like that um really simplified example but with ml you can deploy a sensor anywhere um on a small battery that is able to understand when a lion has roared and no matter what other crazy background noise is going on you probably can get some decent accuracy and just keep a tally and then once a day connect or send a message maybe over like a low power radio and you're able to keep track of what's going on in all sorts of different places so that opens up like the whole

world to being understood better by scientists and conservationists who want to know like the health of ecosystems and like maybe like detecting like the the level of biodiversity in a forest and whether that's being impacted by human behavior or even doing things like we've looked we've seen people who are doing um classification of the noise of chainsaws in the forest to detect whether people are logging illegally. And so that's something I never would have thought of by myself, but there are people out there doing that right now. And we're just going to see that across every single field, every part of industry, from consumer devices through to the most obscure tools that are used in very specific industrial applications, you are able to essentially encapsulate a bit of knowledge.

So what you're doing when you're collecting a data set and labeling it is you're taking some knowledge from a human and conveying it onto this data set. You're taking expertise from a domain expert and you're applying it to this data set. And then you're training a model which essentially encapsulates a little bit of human knowledge and understanding and is able to put it somewhere on a tiny device that uses very little power to answer a simple question and give you a simple answer. And that's something that can just be applied anywhere. It's about pushing our understanding of the world down to where it can make a difference without requiring direct human intervention. And that's very exciting, I think. And I think in the longer term over the next five to ten years this is going to lead to like

a transformation of the world that no one's quite expecting like i think this is the technology that where like mobile phones kind of came out of nowhere and changed the way we do everything almost overnight i think embedded ml has has the potential to do the same thing. And to me, what it means is that maybe we can go from a world where your day-to-day life is a set of frustrating interactions with things that don't quite work the way you want them to, to a world where everything around us is responsive to us and understands what we're trying to do with it and understands what's going on around us, but not in a creepy way where it's reporting back to big brother, but in a helpful way where it has some simple limited on-device intelligence and

is able to more easily help you get your job done or make your life simpler or look after people who need to be cared for in a certain way or be able to be better stewards of our planet. And I think that things are going to look very different once you start to be able to push this intelligence down to where it needs to be used. Wait a minute. This was the part where I didn't like ML. Remember? But we had the part where I liked ML and then we had the part where I distrusted it. And you can't come back to the beginning again. Talk me into this being super cool. there's also the potential to create a lot of frustrating and exclusionary experiences too. And we've seen from the ML community recently has started to get more of an understanding of the fact that like when you use limited data sets,

if you don't have good knowledge about a problem or you, you don't collect enough data from enough diverse types of situations, you can get really bad results that lead to increased discrimination or make people's lives more difficult and like the most horrible example of this i've seen is these um pieces of software you can use to interview someone by having them submit a video and it rates how good an employee they would be based on a couple minutes of them talking like that kind of stuff is horrifying and there are there are plenty of them submit a video and it rates how good an employee they would be based on a couple minutes of them talking like that kind of stuff is horrifying and there are there are plenty of

people out there who are sort of trying to move our thinking away from ideas like that and make sure that as we start to build applications with ml we we make sure they work well. We make sure they work well for everybody. And we think before we apply machine learning in places where really you can't replace that human experience. So it's all really important to think about. Okay. Now I have a question here. Crickets?

There's no escape. So, so yeah from um my my ancient past um there's there's this uh story that i always have to tell whenever i talk to anyone which is is basically um back i guess it was about seven or eight years ago at this point, I was working as an engineer at a software company around here. And I had a couple of close friends. And we've been talking about starting a company, doing something with technology and sustainability. And we were looking at different ways.

But how did you get the models onto the crickets? Wait, what are you talking about? So basically, we ended up figuring out that we wanted to build a company around food security. And an interesting angle on improving food security is can you use insects to create protein that can be used in animal feed and in human food? And I hope the answer is no. No, the answer is totally yes. Absolutely. Yes. And this is something that's like already happening all around the planet and like billions of people depend on insects for their nutrition already. But it's something that's a bit weird to us in the West.

But we started thinking about like, hey, could we apply some of our engineering and design skills to making this work at an industrial scale in the US to really make a dent in the sustainability of the food system? And so we tried our best to do that. And we raised some money from investors and we built an automated farm platform. And we had this big warehouse in the East Bay of the Bay Area that we were raising crickets at a frightening scale. And that was a little bit of my introduction to embedded ML because one of the problems that we had was there's a lot that can go wrong on a cricket farm. There's like water and food being piped around if you're automating things. There are all sorts of different states that you can get into with humidity and temperature and with the state of the crickets that you want to avoid and you want to know about as soon as they happen. And so we were looking at ways to deploy intelligence to

the edge in sensors to be able to understand what was going on in our farm. So for example, we want to be able to detect if there's a water leak when there's a humidity spike. That's fairly easy. Another question, and you don't need to necessarily use ML for that. Another question we wanted to answer is, given a set of substrate that has baby crickets on, how many baby crickets are on there? And that's something that's a lot more difficult to answer without some kind of complex, like either computer vision or machine learning algorithms and that was the

the time i got to start playing with this and thinking about like can we run something on a small device that's able to encapsulate some of this um knowledge that we have about this very strange field that we've we've managed to get involved with and uh give us some answers on what's happening so we don't have to wake up at 4 a.m and come into the farm to look in a box and see if the crickets are doing okay um and uh that's sort of a an interesting little um snapshot of like that's the most niche kind of specific thing that you could possibly be trying to build something around but this type of tooling can even have a role there so uh it's it's quite cool but we uh we got to a point where we we were able to produce a bunch of crickets um efficiently and process them into a product that you can use in food.

The market was not big enough in the US to support what we wanted to do. And so the company's actually released all of the stuff that we developed or is in the process of releasing everything we developed as open source. So if you're interested in starting a cricket farm, you can grab all of our documentation and build your own. What is the biggest hurdle of the cultural resistance that people just could bring themselves to it's the legs so it's really interesting so i mean there's obviously a big angle of cultural

resistance but um i think the bigger thing is is really just the issue of having to meet a certain scale of production before you can hit the economy of scale that you need. You need to have a really low price point to compete on the food markets. And where we were looking at is, can we supplement animal feed with crickets that have been raised on agricultural byproducts so if you can feed a cricket with a waste material from some other agricultural process and then feed that cricket to farmed fish that saves wild fish from being scooped up and ground into powder and fed to farmed fish which is mind-blowingly stupid um the problem is to compete on price with the fish that has been scooped up and ground into powder, you have to have massive, massive scale. And there's a big gap. It's easy to get some

funding to be able to build an R&D scale facility. And once you've got a business that is producing revenue, it's easy to scale up and get funding to grow your business. But in order to get to that point from the R&D facility, there's this huge gap you have to cross where your crickets are too expensive to sell into the market. And that's the big difficulty, I think, that hasn't been crossed yet. how come one of the projects in your book isn't how to control crickets with machine learning because that would have been a good project i mean i i'll have to start thinking about that one but um i actually think after after uh many years working directly with crickets i think it's time for a break so i'm gonna be imagine yeah yeah actually my current projects um i i'm playing with this uh lacuna space um like satellite

satellite laura transmitter thing which is is absolutely amazing and i want to train a model that can recognize different bird calls and keep track of which birds are calling and how much and how often and send me the results back over this like worldwide satellite low energy radio network. And that's something that's, I don't know, I just think it's a fun project to build. And I have a bunch of birds that come to my balcony and eat seeds. So I think it will be fun to see if I can do something with a different animal that's a little bit more enjoyable to be around. You hate crickets, don't you? I don't hate them. They're lovely creatures. So that's your past. You are working at Edge Impulse now,

and you've mentioned them um do you want to tell us a little bit more about them yeah absolutely so so one of the big things that i kind of observe working with the community through the work that um i did at google and with the process of writing the tiny ml book is that even though we can make this stuff easy to sort of get started with um you you can like take the the tutorials in the book and get something up and running you can um pull down the example code and deploy it and make some changes and do some cool stuff it's still really challenging to train your own models. Even though the embedded side isn't that complicated, and you can basically like plug in a library and plug in a model and have stuff work,

training a model and even like identifying a data set and making sure that the data set is reasonable is extremely hard. And that's something that you can anyone can learn you know you don't have to go and do a PhD you can just start developing these skills and there's amazing resources out there but most people aren't going to have time to do that and if you just want to get something done if you have a an embedded project you're working on and you need to add some type of intelligence to it and you want to figure out if that's possible and implement it quickly you don't really have any good options apart from doing all that learning and so i saw that these guys that edge impulse had started a company trying to solve that problem um and when i when i like saw a presentation by the

ceo uh at one of our tiny ml meetups i was just like wow okay this is is fixing everything that is currently difficult right now with with what's going on so i immediately just thought like i need to be a part of this so edge impulse is basically a set of tools that like starts with firmware that you can put on a supported device. It will help you grab data from that device. So whatever sensors you have present, you can grab that data, encode it in the right form and squirt it up to the Edge Impulse servers. And once it's on the server, you can use the sort

of like easy to use UI to chain together some signal processing and some machine learning models. And it doesn't have to be deep learning. You can also go with classical ML models and basically create this pipeline for understanding the sensor data and classifying it and making decisions based on it so edge impulse basically walks you through the process of doing that you can just click okay through everything and go with all the defaults and hopefully come out with a model that like is a good starting point and then to deploy that you just tap a button and you can either download a binary for one of the boards that we directly support. So you can try it out immediately, or you can grab a library, which just plugs into any C++11

projects. And that wraps a bunch of our own really efficient DSP code. And it also wraps TensorFlow Lite for microcontrollers. And so as a developer, you just get a function that you can pass some data into and you will get out of that a classification result. So I think this is going to be a massive deal for people because it means that you don't need to have all of this experience. You don't even need to be a super, super crazy embedded developer because our API is really simple. All you need to

do is collect enough data, which we will help you understand and make sure you achieve that, and train a model, evaluate it, and deploy it. And I just think it's so cool. We've seen so many people in the community who have picked it up and started working on amazing projects. There's someone who's training a classifier that can recognize a cough for tracking people who are coughing as a result of the pandemic. And their goal is to eventually see if you can distinguish between a regular cough and a coronavirus cough. I don't know if they've made any progress along that, but like... The price if they can get enough samples to do that.

Yeah, exactly. That's the difficult thing is the data set. But we're currently supporting arbitrary time series sensor data and also audio. We're going to be adding vision very soon. And we're working with hardware vendors to make sure that we support all the awesome like hardware acceleration that is available on some platforms, that we can output the right types of libraries and the right formats to use with certain platforms. So it's really exciting. We're building this sort of hub that you can use to convert data into awesome ML models that you can evaluate and understand work really well before you deploy them to whatever device you're targeting.

How much does this cost? So it's actually totally free if you're a regular developer. So we have kind of two sections of Edge Impulse. So the most part of it where you you ingest data you train a model and evaluate it and deploy it that's totally free and a lot of it is open source as well so like all of our firmware is open source um a bunch of the the code that makes up the platform is just available on our GitHub. And then we work with enterprise customers who have different needs to developers,

regular developers, and provide like big enhanced features for dealing with massive data sets. So if you have like terabytes of data that you've collected from the space you're working in, and you want to be able to segment parts of that out and train models based on subsets of it and manage that data. We have a set of tooling that you can use to do that type of thing as well. And we charge for that. So that's

the business model, basically. So we have enterprise customers that we work with who need to do things with like big heavy lifting with the huge volumes of data and we make it easy for them and if you're a regular developer you can just try it for free and the cool thing is actually if you we've just released in the last couple of days a mobile client so if you just want to get started and capture some data and train a model, you can take your mobile phone, do a quick dance with some QR codes and open up a little client that will collect data from your instrumentation on your phone. So your accelerometer on your phone and it can collect audio from your phone and you can use your phone to collect some data around and classify the data around a certain problem import that into edge impulse and then train a model and you can deploy it down to your phone and test it out and you can also deploy the

same model to an embedded device and if you have enough sort of data data and you've you've done a good job of um managing your uh data collection like you can potentially have something that you've trained on a phone uh based on data collected on a phone and on some other devices work as your production model this sounds so good to be true maybe i need to go look at your website more yeah totally and we're super excited to get feedback from people um like the the company's pretty young and the product's pretty new and we're really like keeping our ears to the ground to hear what like the community needs and what is difficult right now and what types of models that people like to be able to train so we'd love to hear your feedback and build what is going to unlock this technology for everybody. Dan, it's been wonderful to talk to you. And I imagine I will

have more questions for you in a few months. Do you have any thoughts you'd like to leave us with? So, well, thank you so much for having me on this call. It's been a privilege to talk with the author of one of my all-time favorite technical books. And yeah, I hope people have enjoyed hearing about Embedded ML. And I just say like, yeah, this is a new thing. It works a little bit differently to the type of engineering that we're all familiar with. But the best way to learn about it and understand what it's capable of is to just give it a try and head to edge impulse

and see if you can train a model that does something cool and show your friends and feel excited about it. Or grab the first six chapters of the TinyML book from tinymlbook.com and look through some of the tutorials and have a go and get your hands dirty and um i'm just so excited for more people to be able to try this technology and see what's possible because to me it's mind-blowing our guest has been daniel sitanayaka tiny ml at Edge Impulse and author of TinyML, Machine Learning with TensorFlow Lite on Arduino and Ultra Low Power Microcontrollers from O'Reilly.

Thanks, Dan. It was fun. Thank you so much. Thank you to Christopher for producing and co-hosting. And thank you for listening. Thank you to our Patreon supporters for all that they do, including the Slack channel. You can always contact us at showatembedded.fm or hit the contact link on embedded.fm. And now I have a quote from Robert Frost to leave you with.

Have you ever noticed that his poetry is like sort of sarcastic? I didn't even know. Anyway, in three words, I can sum up everything I've learned about life. It goes on. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.