Embedded - 437: Chirping With the Experts

Episode Date: December 8, 2022

Daniel Situnayake joined us to talk about AI, embedded systems, his new book on the previously mentioned topics, and writing technical books.  Daniel’s book is AI at the Edge: Solving Real-World Problems with Embedded Machine Learning from O’Reilly Media. He is also the Head of Machine Learning at Edge Impulse, which makes machine learning on embedded devices simpler. They have a Responsible AI License which aims to keep our robot overlords from being too evil. We mentioned AI Dungeon as an amusing D&D style adventure with an AI. We also talked about ChatGPT. Daniel was previously on the show, Episode 327: A Little Bit of Human Knowledge, shortly after his first book came out: TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers Transcript

Welcome to Embedded. I am Alicia White, alongside Christopher White. We're going to talk about making things smarter. And by things, I mean not us. I mean AI at the edge, making devices smarter. Our guest is Daniel Sitanyaka, author of the new O'Reilly book, AI at the Edge. Hi, Daniel. Welcome back. Hey, thank you. It's so good to be back. And yeah, really, really excited to be here today. You've been on the show before, but it was long ago. Could you tell us about yourself as if we met at an Embedded Systems Conference? Yeah. So I'm the head of machine learning at a company called Edge Impulse, and we build tools for basically making it easy for non-machine learning engineers to build stuff using machine learning and AI.

So I've been working at Edge Impulse since we were like three people, and we're now, I think, 90 at this point, between 80 or 90, I kind of lost track. Before that, I was working at Google on a team that was building TensorFlow Lite for microcontrollers, which is basically the Google open source library for running deep learning models on embedded devices. So really excited about running deep learning and classical machine learning on tiny devices. So really excited about running deep learning and classical machine

learning on tiny devices, but also about making that easier for people. Is it actually easier for people really now? Oh, absolutely. I mean, so when we last spoke, I think I was like, just off the the kind of treadmill of writing this previous book um tiny ml with pete warden and we'd sort of gone out of the door trying to write this book that was like a 101 introduction to the idea of um embedded machine learning and we ended up with this book that's like a couple of inches thick and uh this is sort of less than what you need to know to get started really it's like kind of a uh a overview and a deep dive into certain sections but um really if you wanted to build a legit project using embedded machine

learning there's a lot more to learn beyond that so over the last like three years or so since since we wrote that book things have become so much easier that really if you're a domain expert in uh you know a particular thing whether it's understanding the health of machines in a an industrial environment that you might want to monitor or if you're building a human computer interaction device that needs to understand some keywords people are saying or all sorts of different things it's got to the point where you can do all of this without writing any code you can pretty much point and click at things and you still need a bit of understanding of what you're doing but there's so much content out there now that you can use to help learn that we're really off

to the races there's obviously huge amounts of work that needs to be done to continue to make this stuff easier and more accessible and and help it make it harder to make some of the common mistakes that people can run into but it's really night and day compared to where we were before, which is very exciting. So your previous book was Tiny ML. What was the cover animal on that? It was a little hummingbird. I forgot the name, the scientific name, but yeah, it's a little red hummingbird. Okay, we want to do lightning round where we ask you short questions. We want short answers. And if we're behaving ourselves, we won't ask more detailed things.

Sounds good. Christopher, are you ready? Do we need to warn him where some of these came from? No. Okay. I mean, if you want to. What are some common applications of machine learning in the real world? And how might they relate to the interests of crickets and other animals?

All right. So machine learning, I mean, in the real world, machine learning has been used mostly for analysis of data until recently. So you imagine you've got like a business and you're selling things and you want to understand what kind of factors can predict whether you're going to sell more or less of the thing you care about you could train a model on all of your data about sales and come up with some insights or come up with a way to predict whether a new product is going to sell well or something like that but um what we've been able to do in the past sort of five years or so is start doing machine learning on sensor data on edge devices.

So we're actually out there in the real world on things, what we can do now is potentially like warning signs of maybe being worn out and needing replacing. One thing you could do is send out model using insight from that engineer and then deploy that model out to devices out in the field and have the devices tell you when they get into that state, which allows you to scale much better because instead of just having your one engineer who's working eight hours a day, you can have a device on each of these machines and you can catch the problems immediately. So you can scale that or move that around to loads of different applications so one thing that's been really popular is with conservation researchers putting models onto cameras that get installed out nailed to a tree in the middle of the rainforest and whenever a particular type of

interesting creature walks past, it takes a picture of that creature and then reports back via a low bandwidth wireless connection that it saw that creature. And so instead of having to have people fly out and collect memory cards from these devices to see which animals walked past, you can get the information instantly. That's an excellent answer. And you have failed lightning round. Well, I mean, I think that's because it's supposed to be short answers. And I think he really missed the part about how...

He brought it back to animals. To animals. But I think he missed one particular. Oh, I missed the crickets. So let me ask you a different question. Still in lightning round, what are the biggest challenges facing the use of machine learning by crickets? By crickets. So I think really they are quite small and don't have good access to technology. But if we can figure out that part. How are your crickets?

Well, luckily for me, I haven't seen a cricket for years at this point. That's good. Luckily for the crickets as well, I think. Which is more important, data set or algorithm? Oh, that's a, actually, no, it's definitely data set. Which is more important, the training set or algorithm? Oh, that's a... Actually, no. It's definitely data set. Which is more important, the training set or the test set?

The test set. Do you have a favorite fictional robot? Oh, there's too many, but either the robots from Batteries Not Included. Have you guys seen that? I saw it as a young person. Yeah, that was one of my favorite things of these cute little self-replicating robot creatures. Or maybe Johnny Five.

Siberian Ibex or Hummingbird? Ooh, that's a good question. I think I'll have to pass on that one and do you have a tip everyone should know oh um if you're writing a book and you're trying to figure out how long it's going to take add 50 to your estimate that's very conservative yeah i mean that that's wow that's actually really good most people it's two or four oh yeah i guess it's if you're writing your second book how was writing your second book different from writing your first i mean it's very um very different in that you know what the process looks like and you know what's going to happen at each stage.

But then that opens you up to a whole bunch of new risks where you think you're going to be done at a certain point by a certain time, but you massively underestimated it and end up having to work a lot of late nights. So I think it doesn't really help that much, but it's definitely less stressful to know up front. I agree. I mean, it would be less stressful to write another book understanding how the art works because I was so stressed about that in the first version. Oh, yeah. Oh, my God. Well, this time, last time, I spent loads of effort making the diagrams

perfect. And then I found out that the publisher could have just done them for me based on sketches. And this time, I did the same thing again, for some reason, because I, well, I liked the way they looked in the previous book. And then they ended up redoing them for me anyway. So I guess I just don't learn. So the book that's coming out in mid-December is AI at the Edge, not AI on the Edge, no matter how many times I say it wrong. Could you give us the,,000-foot view? Yeah, absolutely. So I think as this technology matures and becomes more available for use as a tool in people's toolbox,

there are all sorts of questions that come up around how do you actually use it? What are the best practices for designing products that make use of edge AI? And edge AI itself is this kind of newish term and needs to be defined. And then within that, no matter what your field, whether you're a machine learning expert or an embedded expert, there's going to be huge chunks of this field that you don't necessarily have direct personal experience with. And so what we try to do with this book is create a book that basically gives you an understanding of everything that you need to know if you're going to be building a product in this field or working adjacent to this field. So it doesn't necessarily teach you everything you need to know, because that would be like an encyclopedia size book, but it at least gives you an understanding of what all

the pieces are and maybe even who you need to hire or who you need to work with to make these kinds of projects happen. Where the tiny ML book was pretty tactical, pretty hands-on, code was on most pages. The first third of this book, there's no code. Yeah, it's really designed to be workflow oriented. So the goal is to kind of give you first the background you need to know about all the moving parts, what they are, you know, how to think about them, and then to go through the workflow end to end of working through an idea, understanding whether that idea makes sense for this type of technology.

And if it does, what particular pieces of this technology are useful for it? And then going through this iterative workflow of designing something, training models, testing them out, understanding how they work in realistic scenarios until you've got something approaching a product. And the other really important thing there is looking at responsible AI as you go through

all of this. Obviously, when you're designing any product, you need to think about ethics and responsibility but with ai and machine learning you really need to because it's a core part of the workflow so the book tries to weave that in through every stage also one of the solutions that seemed to be presented in the book was basically hire an ethicist who can give you better insight into how ethics and AI go together. Is that the only option? Well, so I don't think it's the only option, but realistically, if we're talking about building products that are supposed to scale, right?

So I guess, you know, to take a step back, I'd say the whole point of AI is to take human insight and multiply it by factors that we couldn't do with people. And so what we're also doing in that process, like we're making it cheaper to apply human insight in different situations, but we're also decoupling the complete human insight from this little piece that we're going to go and scale up. So any situation where we're doing that, it's really important to make sure that it's done

in a way which doesn't cause harm. Because by taking the person out of the equation, you're introducing a risk that the system's going to do something that a person wouldn't do in that situation and so you know a good example of this that we use in the book is the idea of a driverless car like it's incredibly important to get that right if you're going to try and do it because the risk is that your system is going to potentially put human lives at risk. So in that light with the seriousness of the situation I would hope that any organization that is working with machine learning or AI in a context where it could put people's lives at risk potentially, would see it in their

budget to hire somebody who can really understand the implications of that from a harm reduction and ethics perspective. So obviously smaller, more trivial things that have less of an impact on the world maybe you're making like a a toy that's designed for bored engineers to play with on their desk maybe you don't need to worry so much about this type of thing but then in that same example imagine you're building a children's toy and you haven't fully thought through the ethical implications of what this AI-powered feature on this toy is doing, that really needs to be part of your work. And a good way to think about this is that you really need to know what you're doing, right? And so part of knowing what you're doing is making sure that your team contains domain experts

who are guiding the development of the product. And typically, one of these domain experts should be somebody who understands the ethical and responsible elements of this work that you're doing in that chapter which i think was like chapter five where you talk about building a team for ai at the edge you you had a team that forget the ai part i want that whole team anyway i want somebody thinking about the ethics i want the product manager thinking about the customer's view as well as when are we going to ship? I want the tester. I want the engineers that can review things. I want CEOs who actually understand how the market's going to work. I wanted the team and it wasn't related to AI. Why did you feel that that was an important part of the book instead of just saying,

make a reasonable engineering team and stop trying to do everything yourself? Yeah, that's a good question. I mean, I think that we all come from different perspectives as engineers or product people or business people or whatever role that we might have in organizations. And depending on our own history, we might not be aware of all of these different pieces. I certainly wasn't aware of all the different parts that come into building, for example, an embedded project until I started getting deeper into the world of working with embedded engineers. So I think it's important to have a kind of baseline of like, what does a good team look like? And one of the real things there to think about is like,

obviously the tools that exist to create a company and create a product are getting more and more accessible over time. So it's easier and easier to make things and put them out in the world and have them interacting with people. And it's easy to do that, but the knowledge about how to construct a team to do that well and do that safely

doesn't come for free with the tools. That comes through experience and insight and all of that good career stuff. And what we don't want really is that because the tools are accessible, it means people who have an interesting idea, but don't necessarily think through all of the implications are going to go out there and try to build it and then either fail because they missed some important stuff that they needed to be aware of or cause harm in the world by doing something that was ill-advised. So just, you know, even just from a kind of selfish point of view, if you're

trying to launch a product and you think you've got a great idea, but you don't know anything about one particular portion of the process of developing a product, you've got to figure that out somehow. And so hopefully by covering that in the book, we at least point you in the right direction. Like I was saying, it's not an exhaustive resource for how to hire a good program manager or something like that. But at least if you didn't know that program managers exist, you now know there's a name for them and you could go out and hire for one if they do the kind of thing that you need on your team. When you say tools, you mean the AI tools. And when you say tools, I hear compilers, because all of that is true with just someone with a compiler.

It doesn't have to be AI. Yeah, absolutely. You do have so much definitional, is that a word? Sure. information in the book about sensors, which is all the embedded parts, as well as algorithms to deal with sensors. And then there's the AI, different methods, neural nets, and even heuristics you were calling AI. What is AI? I mean, is it always i i guess these days it's mostly machine learning for me but what do you define it as yeah this is it's an interesting question and you know there's so much that it's almost like a philosophical question well it's not almost it kind of is

you can go down to like okay what is intelligence and when you read about what people who are at the cutting edge of answering this kind of question say, there's not like a clear answer. Everybody has a lot of differences. But what I wanted to kind of define for this book as like a really basic way of putting this in terms of what is artificial intelligence, I try to answer like, what is intelligence? And I think intelligence is knowing the right thing to do at the right time. You know, like that's a very trivial definition. It's not going to stand up to someone who's a philosopher,

but in terms of when we're talking about intelligence in engineering products that do things, if we want to add intelligence to a product it usually means we want the product to be better at doing knowing the right thing to do at the right time and so artificial intelligence means knowing the right thing to do at the right time without a human being involved and i quite like that little definition because it's kind of tidy. It sidesteps all of the stuff about computers being sentient and things like that. And just lets us focus on the basics, which is that if I'm creating like a thermostat for my house that like learns the best time to turn the temperature up and down, that's not necessarily going to involve deep learning. It's not necessarily going to involve some really fancy code, but I'd argue that it is AI because we're using a computer to do something which we as humans would think is kind of intelligent. So the sort of goal with

the book, right, is that we want people to build good products and we don't necessarily want to direct people and funnel them to like, hey, you've got this idea, you should immediately train a really fancy deep learning model to try and implement it. Because quite often, in fact, the majority of the time we find as practitioners in this field, you can get some pretty good results using heuristics and using rule-based logic. And if you're a machine learning engineer and you've just had an amazing idea for a gadget you want to build,

maybe your first instinct's going to be to go and train this fancy model. But what we want you to do is take a step back and have a look at what is the most basic thing that you could call intelligent and that someone would pay money for because it makes their life easier? Because maybe that's the place to start. Did you have a particular reader in mind? For me, it helped me to have actual people I was thinking about so I could think about

what their skills and their background were and how much math they needed. Did you have somebody like that? Yeah. So we kind of wanted to be as broad as possible because Jenny and I, Jenny Plunkett's my co-author, we've done a lot of work through Edge Impulse and through other jobs working with people who want to apply AI to solve problems with embedded systems. And what we found is there isn't really like a typical person who is the person directing this kind of work. We've talked to everyone from hardcore embedded engineers

through to machine learning engineers, people who have written books about either of those things but don't know anything about the other one, all the way through to people who are totally business focused or product focused. So what we saw is that it doesn't matter what someone's background is, there are certain things that they need to know in order to think rationally about this type of space and know how to solve problems with it. So we try to make something that would be a useful reference and provide some good insight to people who are building in this space, whether or not they are highly technical or business focused or so on. How much math do people need to use AI techniques?

Is it enough to be able just to use them even if you don't understand the you know gradient descent and all of the mathy bits? Yeah so I'd be the first to admit that I'm not a good mathematician, but I think it's okay. So, there is huge value to knowing and understanding some basic maths concepts when you're working with machine learning. For example, to understand how models are performing, there are a few really important evaluation metrics, which basically give us numbers that describe whether a model is doing its job well at maybe, I don't know, telling you whether something is a lion or a rat that's walking past in the jungle. So having some insight into mathematics is really helpful with interpreting those types of numbers and understanding what they mean and how they

apply to your situation. Also, it's helpful to understand a bit about things like probabilities, because that's something that's kind of inherent to a lot of these types of technologies. And if we're doing signal processing as well, that can be really valuable. So if you're a signal processing expert, I think people who are DSP experts are so well suited to working with embedded machine learning. Really, they're kind of the natural users of this type of technology. So that can be a real advantage, but there are really cool tools out there like Edge Impulse that make that stuff easier so that you can just kind of plug in pre-created signal processing algorithms and use those to interpret your signals and feed into models. So it's useful. There are big advantages to learning some mathematical concepts and getting better at understanding that type of stuff, but it's definitely not a barrier to entry. And I'd say, you know, a typical embedded engineer is absolutely mathematical enough to be able to work with this technology. You mentioned Edge Impulse, which is the company you work for.

Did you found it? No, unfortunately not. I was beat to it by the actual founders. So Zach Shelby and Jan Jongboom, who were working at Arm at the time, and realized that there was this need for much easier tooling to help people build solutions with this type of technology. And I was coming to that same realization at Google, you know, writing this book with Pete that ended up being like really, really long. And so... The tiny ML book, not the current book. Yeah, the tiny ML book, exactly. So I think we were all in the field at the time realizing that there are different ways to go about this, but we need to make it easier for people to use this technology.

I think you've kind of touched on this, but there's a tension between making things easier and the ethical angle. And you said, you know, if you make things easier, then people are set up to fail. But I also feel like people could be set up to use it irresponsibly without knowing what they're doing. Is there a place for, at the risk of solving social problems with technology, is there a place for making the tools

in some way easier to evaluate ethical questions i guess or something absolutely um so i i think that one of the things that's motivated a lot of people who have entered this field is it's almost a chance to start again and reset and do things better this time around when it comes to AI and responsible AI and ethics. Because we're solving a whole bunch of different, a whole different bunch of problems with different types of data, we've got the opportunity to build tools that have some of this stuff baked in. So for example, one thing that's really important in building AI that works responsibly is making sure that your data set contains data that reflects the type of diversity that is there in the real world. And it's not just the kind of diversity

we're talking about with human beings and races and ethnicities and genders and things like that, but just diversity in terms of like there is a lot of difference and heterogeneity in the real world. And any data set used to train a system that's going to be working in the real world needs to reflect that. And so we can think about things like what are the tools that we need to build that make it more obvious to people when their data sets do not reflect that type of heterogeneity. If we can look at your data set and we see, oh, there are these types of things and these types of things, and there seem to be these types of things as well. But for those ones, you don't have very many samples. Have you thought about going out and collecting more? Giving people that type of input and guidance can be really valuable in helping people avoid making these sorts of mistakes. But what it can't do is stop people doing bad things on purpose, building applications that are just harmful by

nature. However, we can do things like our Edge Impulse software user license is something called a responsible AI license. If you search for responsible AI license online, you can find that. And we expressly prohibit people from using our platform or our technologies for building certain types of applications, for example, in weapons and surveillance. So there are things that you can do even on that level, license-wise, to limit what people are going to do with your products. And I think it's really important to think about that from the beginning. In the first third of your book, there was very little mention of edge impulse. And in the last third, there was quite a lot as it went through specific examples showing the software on the page on the screen. Was it hard keeping Edge Impulse out of every page? And we can help you with that sort of, I have a solution to many of these problems. Stop reading here and go play with

it. Yeah, it was a really conscious decision we made actually right at the beginning to avoid making the book tied to a specific tool. So that's why we chose to separate the vast majority of the content is just sort of general and universal. And then we have these specific practical chapters at the end, which are supposed to demonstrate the principles we went through in the rest of the book, but using a particular tool. That happens to be the tool that Jenny and I have helped build. And so the reason we did that was because it's horrible when tech books come out and then six months later, they're out of date. It's kind of inevitable to some degree, but a lot of the stuff to do with the machine learning workflow and how to work with

AI responsibly and how to build good products is kind of general. And it's going to change a lot more slowly than the stuff that relates to any individual tool. So, you know, if you pick up this book a year from now or two years from now, that majority of that first part of the book is going to still apply. And maybe the latter parts might need updating, but that's okay because it's only, you know, for kind of demonstration purposes, really. How much of the book was part of your job? So, I wanted to make sure that it was all doable within sort of spare bits and pieces of time while I was working. So the original goal was to spend, you know, an hour or a few hours on it a week over an extended period of time. It really ended up being a lot more work than expected. So I spent vastly more time of my spare time on the book

and not that much work time in the end. And same with Jenny. But I think it's really, really fortunate to have an employer that's willing to support you on a project like this. I'm super grateful to the company for letting us use any of our work time on building it. to support you on a project like this. I'm super grateful to the company for, you know, letting us use any of our work time

on building it. It's been really cool. We haven't really talked about the edge part of this because the show is about embedded systems, or at least that's what they tell me. And we understand we're making smarter systems all the time.

Our processors get more powerful. But there are reasons to put more intelligence at the edge and there are reasons not to. You have an acronym that talks about why you would want to put things at the edge. Could you go through it? Yeah, absolutely. And I love if I could take credit for this acronym. But actually, it was an invention of Jeff Beer from the Edge AI and Vision Alliance. And it's absolutely brilliant. Because it's, it's a stupid sounding word, blurp. And the letters of the acronym are B, L, E, R, and P. And so each of those gives a sort of

argument for situations where edge AI makes sense. So I can go through those if you want. Well, the first is bandwidth. And that totally makes sense to me i mean that you were talking about animal pictures you don't want to send every picture up you want to say oh i saw 23 rabbits and one cheetah exactly and then maybe you get the is the second one right and that's when you don't want to have to talk to the great internet you want to give a response quicker so when i'm i want to have a pedometer or a smart watch i don't want it to have to send all of my data up to the internet and then get a response. If I took a step, I want it to say I took a step.

Exactly. The next one is economics. I didn't get that one as well. Could you explain it? Because to me, it's more expensive to put intelligence on the edges. Yeah, it's really interesting. So I think economics, one thing to remember is not all of these need to apply for a given application to make sense. So you probably need, you know, at least one of them if it's a really strong one, maybe a couple. Some of them come together quite nicely, like bandwidth and privacy, for example, which we'll go into later. But yeah, you don't have to have all of these in every project. And so there'll be some places where economics could go one way or

the other. But potentially, if you're sending data from the device that has a 3G connectivity on it to send up video and you've got to pay per kilobyte that you're sending. Or maybe if you're doing the sneaker net thing and you have a device out in the middle of the rainforest taking pictures and you have to fly someone out there every six months to pick up the memory card all of that stuff costs money so by reducing the amount of data that needs to be sent around by making some of these decisions on the edge you can save costs also it potentially can reduce costs on your server side processing so for, if you're trying to spot keywords in a bit of audio, you can either do that on device or you can do it on the server. If you're doing it on the server, you've got to then provide a server and keep it alive for the entire time that your product's being used. And if your company

goes away, then maybe your product will stop working. Whereas if you're doing that stuff on the edge, you no longer need a server. You no longer need the people required to keep a server running. You don't need to keep it up to date. You don't need to have this sort of security liability. So that's kind of where it can come in as well. That also comes into the next one, which is reliability. It can be more reliable to have smart things on the edge if you don't have a reliable connection or a reliable company. Are there other ways that reliability plays into it? Yeah, absolutely. So if you've got any kind of system that's dependent on a connection somewhere

else, that's just another point of failure. And it involves a whole chain of complexity that goes from the device all the way back to the far side. And it's just always good to have less complexity. And there's complexity involved with machine learning and AI, but it's a different type of complexity. And sometimes it's worth trading one for the other. So potentially maybe it's better to have this model that's running on a device versus depending on there being a reliable wireless connection. A good example of that would be a self-driving car. Imagine you're driving through an intersection and suddenly your self-driving cuts out and you're sitting in the middle of the intersection with cars streaming around you. You know, that's kind of the least bad thing that could happen. Whereas if all of the AI stuff is happening in the car, that's much less likely to

happen. And then the last one was privacy, which we kind of touched on with bandwidth, but mostly I don't want you sending all of my data across the internet, because I don't trust that it will be private forever. Exactly. And this is my favorite one, I think, because I think beyond all of the others, privacy unlocks things that are just not possible before. You can kind of brute force all of the others, like bandwidth, for example. you can always throw money at something and give it more bandwidth uh latency same kind of thing if you can move your server close to the edge even if it's not at the edge exactly you can reduce latency um all of these things are things

that you can kind of throw money at an engineer around. But privacy, if something is a privacy concern, it just is a privacy concern. If there's privacy concerns with having certain types of sensors capturing data in certain places and sending it elsewhere, then that problem is not going to go away by throwing money at it. For example, having security cameras in your house. Like I personally wouldn't feel comfortable having a camera in my home that's streaming video and audio to some remote server. I don't care how cheap it is. I don't care how supposedly reliable this company is. It's just a line I don't really want to cross.

Whereas if you've got a little model that's running fully on device, supposedly reliable this company is it's just a line i don't really want to cross whereas if you've got a little model that's running fully on device doing whatever it is that's supposed to be done by that device um and you have a guarantee that no data will ever leave the device i don't care whether the camera is there um so that opens up things like maybe I can have the light shut off automatically when I leave the room. I don't really want that to happen because I was broadcasting video of my bedroom to the internet, but I'm quite happy to have a little, you know, addition to my light switch, which has a camera built in, but has no way to have data leave the sensor going back to reliability and we touched on this before ai doesn't always work safety critical systems like cars are definitely

part of it and that that's a big thing um but one of the worst parts about AI on the edge locally for me is that it works until it doesn't. So it feels unreliable. It ends up being more frustrating than if it doesn't work. Like if I, we have a garage door opener that 90% of the time opens when I tell it to. And either 10% of the time i have to hop around and do some sort of dance in the driveway in order to get it to work it's the 95 that's worse than if it just didn't work at all yeah exactly it didn't work at all i would you know carry the button but it's it's it's so i mean 95 sounds good our first yes but that's one out of every 20 times it does not work.

It's unbelievably frustrating, yeah. And I'm sure everybody's had this sort of effect with voice assistants, which I've done a lot of work in in the past, where you're able to get it to work. I'd say 95% would be a bit generous. But most of the time, it will understand what you're doing. But when it doesn't, it's embarrassing because you're literally speaking and then something doesn't understand you and does something else. And as human beings, that's inherently a horrible feeling.

So yeah, totally see that. As an engineer, I totally understand that the words on and off are essentially the same word. But as a human, there are some important differences. And as we work on the edge, we have constrained models. I mean, this goes back to the economics of small devices and making sure they're low power. So we have to be constrained. That's a normal embedded systems thing.

But balancing that trust and reliability against Privacy? Well, I was thinking more against the economics of constrained models on devices. Sure. I understand that there's a trade-off there as an engineer, but as a user, that trade-off has always been wrong. I mean, it's just 95 is not good enough. where all of this sort of extra stuff around who is on your team really comes into play because you need to have a team building the product that you're building that has insight into this type

of stuff what would be acceptable to have 95 accuracy for what would be frustrating to users. There's no need that these things even need to be particularly accurate sometimes. So, for example, if I have a situation where I've got to identify whether a machine's about to break down, because if it breaks down, it costs me $100,000 to fix it it if i can improve my ability to understand when it's about to break down even by 10 that's going to save me so much money um that it becomes a no-brainer whereas if i'm in a different situation where i'm trying to let someone into their house using face recognition and 10 of the time it's letting the wrong person in who shouldn't be allowed in that's a totally different kettle of fish so having some good understanding of the situation that you're trying to work within and really from every angle and and how your solution

is going to impact that is absolutely critical and you need to be building your projects in a way that you build in this sort of go and no-go decision at various points within and try as early as possible to get to it. So you need to find out very early on whether your product is going to work effectively. You need to get that figured out at the model stage before you've spent loads of time building hardware for it, for example, or putting stuff in packages and getting it onto store shelves. I think it is a big challenge. On the other hand, a lot of the places where we're looking at deploying AI to the edge, first of all, it gives us another signal that we can use in conjunction with existing signals

to potentially make better decisions. So, for example, you mentioned your garage door opener maybe we could um create a system that sort of used a camera to look at whether your car was present and gate whether the door opens based on whether it looks like your car is present and whether the radio transmission is happening and that might allow us to reduce the requirements for the radio signal to be as strong because we also have this other signal. And so maybe it would start feeling more reliable for you. So using these, that's kind of a dumb example, but using these things together, multiplying the number of signals you have can lead to more reliable products. A good example of that for a case that I mentioned with the speech recognition stuff is a big challenge with that is like,

how do you know whether somebody's talking to the digital assistant versus just talking to someone else? So one idea that's been thrown around is putting a camera on the smart speaker and looking at whether somebody is looking at the camera or not when they're saying a command. And so by doing that, you're able to be a bit more sure that the person's talking to the device. So you can adjust all of your thresholds and calibration to be more likely to pick things up and less likely to miss things. And with Edge AI, you can do that on the device, whereas previously you wouldn't want to have streaming video

up into the cloud. That example makes sense, but my brain keeps saying, that's just sensor fusion. We've been doing that for 20 years. It's nothing new. Oh, totally. Well, so that's what i i really would like to kind of get across to people actually is that this technology isn't it is something new in

some ways but more it's just another tool in your toolbox that you can deploy alongside the tools that you're already using in order to get better results in some situations. So there's some cool, fancy new stuff we can only do with this type of technology. But a lot of the time, I think it just fits in as another tool that an embedded engineer might deploy. And that's the way I want people to see it. Not something fancy that requires all this extra, but actually just something useful that you can rely on in some situations.

I want to shift gears just a little bit because I've got you here and you're a machine learning expert. Alicia and I have both worked on machine learning projects for consulting. So we have a passing familiarity with most of the mathematical concepts and how models work and test

and they're developed and stuff, but they've been mostly classification kinds of problem solving so very straightforward typical vision things as i've done more work with machine learning as i've played with some of the things that come out of open AI, I am slowly turning into a Luddite. And last night, I think I said I was going to burn all the computers in the house. As witches. Because I got to play with... Chat GBT. Chat GBT.

I was just asking about that. Which is extremely fun to play with. Extremely fun. Totally wrong about 90% of the things it talks about. Dangerous, but also extremely fun to play with. So we're in this place where the public-facing view of AI is... Like the data assistance. Look at these fun tools that are heading towards C3PO or can make art for us.

And how are you feeling about all this stuff? I forgot to ask him when he thought the singularity was going to happen. Oh, yeah. I mean, this is a whole thing. And I honestly, like my whole life, I've been absolutely fascinated by artificial intelligence, but also the big questions around consciousness and sentience and experience and and and that whole sort of stack there but i just don't think about that at work because what i mean there's there's there's a lot of excitement in these kinds of um innovations especially around large language

models and image generation and stuff and it And it's really fun to play with. It's cool. It will have important applications. But what really needs to happen for it to actually make any kind of difference to anybody is for there to be tooling that can be used to make it actually useful. And I think it's going to take a while before we really get there most of the time with ai like having worked on on this type of stuff for

my entire career really um one of the the companies i worked at about um 12 years ago we were doing uh artificially intelligent call center agents which use um nlp it was all doing done using kind of rule-based systems back then but um we had the problem that our system was too smart and it would sort of ad lib things and if you've got a call center agent that's supposed to be like helping you book a theater ticket or get your delivery delivered in the right time slot and then it starts just interjecting with unrelated topics that it thought that you were talking about that's absolutely unacceptable from the point of view of the business that's trying to use this as a tool um so i think providing these kind of constraints is going to be the the biggest challenge in making this type of technology useful and until then i'm not really that worried about it right i think that in order

for someone to use something for good or for bad it needs to be effective and these types of things we haven't got to the point yet that they're actually effective for anything beyond playing and there's nothing wrong with that there's loads of cool things that exist only for for entertainment and for diverting humans from our own sort of existential um fear for a brief time so if you can have a chat with the the the open ai models there's actually an amazing thing called AI Dungeon, which is so crazy. It's like a role-playing text-based adventure game

where it will generate the story as you go and you can have conversations with the characters. It's just like, it's mind-blowing. You couldn't put that in a corporate product interacting with the public because it's going to say all kinds of weird, unacceptable stuff.

But if it's in a video game for adults to play with, that's fine. I think I disagree with you. mom talking about computers and them getting into homes and how so many people thought that it was going to come in through data analysis, like anybody does that at home. Or there was that giant computer that Neiman Marcus sold for recipes. Oh, right. But the way that computers came into our homes was through video games. It was through play. And with the image generators, I used it for playing for a while, but now I use it to create thumbnails for blog posts or for shows. These are not things I would pay an artist to do.

Before, I would just use random pictures. But now, they're slightly more relevant and they're kind of fun. Maybe I have to go through a dozen versions before I find one I really like. But that process is fun. And the chat GPT that, I mean, Chris and I've had a lot of fun with anything having to do with screenplays for Star Wars and yogurt was just genius. But I had... But you can also ask it to write a book report, and I could see a kid turning that in and getting a passing grade sometimes. Well, more ethically. No. Okay. Sorry. Yes. Last week I had a colleague who was very frustrated with the client and we talked and it was, it was useful in the end to him because

he was too close to it. And it was easy for me to come at it from the outside and say something calm that was exactly what he needed. He could have just told the GPT chat or the chat GPT. And it would, I mean, I asked it for a form letter for someone. I don't remember. Someone who was a non-paying client. Which is a letter I hate to write. And you'd think I would just save it.

But now I could actually put in a few extra details and it would write me a letter that I didn't... I am checking it. It's not I'm sending it. I'm using that as a template, a starting place for something more useful. And I am just... Like I said, we've been having a lot of fun with it. And that's actually going to be today's quote at the end of the show is, is what I got from GPT chat.

Awesome. But I don't, I agree. I like, I had the problem earlier where if it's not reliable, it's not useful, but I think there may be another path for some of these other things that aren't business focused. They're more human focused, human solving a problem I didn't realize I had, and now you can solve it more easily in a fun way. So that's where I totally agree with you. And i think the key thing for me in all of those examples is there's a human in the loop um and these are being used as recreational experiences um or creative tools or a helpful sounding board for something um and that's amazing and i i think

you know there's there's huge potential there. But the thing that makes it useful in all those cases is that you're gating what it does. So you're creating images to go on a blog post. You get to pick which image goes on there. Or you're having, you know, helping yourself write a letter and you get to look at the letter and edit it before you send it and so it's like it scales in one way because like lots of people can use that

creative tool at the same time with very little costs versus manufacturing like a widget that does it but it doesn't scale in the sort of scary multiplicative way that worries people about the takeover of robots, for example, because it's not automatically sending emails with no oversight. For every email that gets written with GPT-3, there is somebody reading that email and editing it first and right now if you decide not to go down that route and not edit the email first you're going to end up causing a lot of problems for yourself exactly um so i think as we i think it's inevitable we're going to come up with tools for better controlling and mediating the output of these things until they become um useful in a sort of uh scaling sort of way but we're not there yet and i've been working on on this stuff for basically my entire career and i haven't seen we've made any progress whatsoever. So I'm not holding my breath. That said, if I buy a, um,

an R2D2 to live in my house and his job is just to poke around in my house and be my friend, I don't need him to be nice all the time, um, and say the exact right thing and be on brand. Um, I just need him to be my friend. And that's the same as my cat, which, you know, regularly attacks and bites me. So as long as we're not putting harmful content out there by building irresponsible products as organizations, you know, you don't want the r2d2 to be like attacking your child or whispering you know horrible things into your ear while you're sleeping but as long as we can can be in control of technology ourselves I think it's it's really exciting I'd love to have a little robot friend that I can chat with but maybe I'd like to have a little robot friend that I can chat with.

But maybe I'd like to have one that I created rather than one that somebody else built and is going to like secretly market things to me in the night. That is worrisome. I, as somebody who likes to try out conversations before I have them, likes to get scripts for conversations. I imagine this will be very useful to me. to try out conversations before I have them, likes to get scripts for conversations, I imagine this will be very useful to me. I don't do well when things are super awkward.

And so I can tell it, play the role of someone who is angry and then I can, you know, go back and forth. And I can really see that being useful. I don't want to say for developing my communication skills, but for decreasing my worry with respect to some communication difficulties. I love that idea. Like imagine you've got to give a talk and you can give your talk to a system which then asks pointed questions at the end so that you can add the stuff that you missed.

Yes. And it was, I mean, I made it talk about curved crease origami, which is something I love to talk about and nobody really will talk to me about it. So it was really fun to have it. I mean, we talked about topology and differential geometry. It gave me all kinds of ideas. But some of them were probably garbage. Oh, and the code it gave me was total garbage. See, that's the thing.

That's reassuring. All right. I am in agreement with both of you, but also disagreement. Because this stuff worries me. I know I'm not worried about, my worry, to be sure, is not AI becoming sentient, taking over the world, robots. I'm just worried about the interaction of humans in this stuff and where that leads.

I'm just not sure. It just makes me weirdly uncomfortable while also enjoying it, which is a strange place to be. And maybe that's the place all of us should be with technology. But, I mean... Yeah, I think people have felt the same way, really, at least for my entire life about technology. And I'm talking about these, what are my favorite robot, fictional robots? They're all from sort of warning signs about technology movies that were made so long ago that they look quaint now.

And none of the fears have come to pass. So the problem is much worse fears that we didn't see have come to pass. Yeah, and that's what bugs me is where is this going to go? And right now, ChatGPT, I mean, it's fun, but it's also a really good BS artist. Oh, my goodness. It's a finely tuned BS machine. I see things when that thing talks that I said when I was trying to

explain to a teacher about a book I hadn't read. And so that's, that's very interesting to me, but also kind of, you know, gives me some trepidation. So it's an interesting time. Yeah.

If you think about it, these things are trained on a corpus of data from the internet, which is basically a giant group of people talking about things they're not qualified to discuss. Well, Daniel, it's been really great to have you. Do you have any thoughts you'd like to leave us with yeah sure so i i think if you're listening to this podcast um and you're interested in this kind of thing at all i really recommend just getting your hands dirty and trying some stuff out and with tools like edge impulse uh tm or whatever whatever kind of things you you have at your disposal to play with, you'd probably be just surprised with how much lower the barrier to entries have gotten.

And what I'm really keen on is as many people as possible who actually know about things in the world can use this type of technology to help solve problems and the kind of problems that they were looking to solve already. So yeah, like if you take anything away from what I've said here, it would be like, go and give this stuff a try. It's really fun and exciting and interesting. And there's some good resources out there at this point,

which can help you build real stuff. And Edge Impulse does have free tutorials. People can try this stuff out on your website yeah absolutely and also the the book um if you want to check that out you can head over to o'reilly they have this like online learning platform thing and if you sign up i think you get 30 days free trial so you can read read the book for free, basically. And then cancel. And then buy a copy. Our guest has been Daniel Sitanyaka,

author of the new O'Reilly book, AI at the Edge, solving real world problems with embedded machine learning. Daniel's also the head of machine learning at Edge Impulse. Thanks, Daniel. Fascinating, as always. Thank you. Such a pleasure talking with you both. Thank you to Christopher for producing and co-hosting, and thank you for listening. You can always contact us at show at embedded.fm or at the contact link on Embedded FM. And now I have a little thing to read you. I asked ChatGPT to write a podcast

transcript where the podcast is hosted by a cricket and asks lightning round questions to a machine learning expert. Cricket, welcome to Chirping with the Experts. I'm your host, a cricket. And today we have a special guest joining us, a machine learning expert. Let's jump right in with our lightning round questions. What excites you most about the field of machine learning and how does it relate to the interest of other crickets? I'm not going to fill in Daniel's part because he did it already and it was very good. Let me ask a couple more of these lightning round questions. Can you give us a brief explanation of what machine learning is and how it works in a way that other crickets might understand?

And finally, the one that we did in fact ask Daniel. What are some of the common applications of machine learning in the real world and how might they relate to the interest of crickets and other animals?