Embedded - 428: Sprinkling a Little IoT

Episode Date: September 22, 2022

Jonathan Beri spoke with us about the different IoT development tools and how to categorize them.  Jonathan (@beriberikix) is the CEO of Golioth (@golioth_iot). He wrote a blog post called An Introduction to The Five Clouds of IoT, breaking the clouds into individual clouds: device, connectivity, data, application, and development. Jonathan was previously on Embedded 222: Virtual Bunnie when he worked for Particle.io. A partial list of the IoT tools we mentioned: ThingsBoard Freeboard Grafana Ubidots  Renode Memfault Golioth Particle.io  Node-RED Soracom Hologram.io   See also A list of IoT platforms – Systev post mentioned in the show (also Building The Infinite Matrix Of Tamagotchis | Hackaday).   Transcript

Welcome to Embedded. I am Elysia White alongside Christopher White. We've talked about Memfault. We've mentioned Reno, Dio, and maybe UbiDots and Goliath and Particle. And I have just discovered these are all Pokemon and that we need to collect them all. And Jonathan Berry is going to help us unravel which one will win at which Pokemon towers. Is that right? Maybe if the Pokemon have antennas in their tails?

More seriously, we're going to talk about the Internet of Things and the tools that we can get around the internet of things. They are not, in fact, Pokemon. How are you doing, Jonathan? Welcome back. Thanks. Great to be here. Could you tell us about yourself as if you weren't on a show 200 episodes ago? Yeah, quickly about me. My name is Jonathan Berry. I'm the founder and CEO of Goliath, an IoT platform company. But I've been in embedded IoT for the last decade or so at big companies like Google Nest, at tiny companies like Particle,

because they're not so tiny anymore. And yeah, started Goliath in a similar space. All right, and we're going to ask about Goliath, but we are also going to ask about a lot of other things. But before that, we want to do lightning round. Are you ready? Yes. Complete one project or start a dozen?

Definitely start two dozen. What do the S in IoT stand for? Well, generally not security. Sometimes I like to say same defaults, but that doesn't quite have the same ring. What is your favorite video game? Favorite video game? Pokemon. Pokemon?

Should we bring back the dinosaurs? Five movies tells us no. If you had your own podcast, who would you want to be as a guest, and what would you ask them? Oh, I wasn't prepared for any of these lightning round questions. I would have a fictional guest. Jean-Luc Picard. And grooming tips. I don't know.

We may have asked this the last time you were on, but we can go check to see if you give a different answer this time. Favorite fictional robot? I probably have a different answer every time that question is asked but probably Lieutenant Commander Data going on the Star Trek theme and one last one

do you have a tip everyone should know I remember my tip from last time which is still very valid which was don't cross the red and black wires I think this time it's more which was don't cross the red and black wires. That's still valid, yes. I think this time it's more, once you get the bootloader working, don't change it.

And, might I add, don't put too much in the bootloader. Yes. And don't look at my PR that I just filed yesterday, which has too much in the bootloader when we last talked to you you worked at Particle what does Particle do? Particle is an IoT platform they are a full stack solution

so you can buy a component whether a dev board or module from them that includes the operating system the software update capabilities, connectivity. So if you are building a cellular product, they even give you a SIM. And then the tools to manage the devices. And now you have founded a company called Goliath.

What do they do? Similar space. We help hardware companies who are designing internet-connected devices by providing those capabilities like software updates, secure communication, but in a different approach where we say, hey, you've got your chip, you've got your sort of vendor of choice, your product is working. Now, how can we help connect that device online, but in a way that's working with the components you've chosen or the operating systems you want to use in a little bit more bring-your-own-device kind of way? And our approach to that problem space is, let's look at an IoT device, and we'll definitely go into IoT, I'm sure. And the cloud parts of that are very complicated, very specific, and hard, especially for hardware companies.

So can we be that team who's figured that all out? What do you need from the cloud, and let's provide that. So kind of a longer answer that's slightly different. So I've been doing embedded systems for what seems like forever, but isn't. I definitely started after networks. But I went to several companies who were building various IoT devices, as we would call them now,

who would start with the idea that they should build their own radios or their own GPS receivers. And when I would go, I'd be like, no, maybe we should buy those. Even though they seem super expensive, we should maybe do whatever it is you wanted to do.

But we would still have to run our own servers, build our own bootloaders, update our own firmware, monitor devices for system health and why they crashed and whether or not the bug that they have actually has to do with something crawling inside the box because Boston has salt fogs. All of this was from scratch.

Sometimes it was an assembly even. But now there are tools for all that. There are so many tools. Okay, we have Memfault. do you remember what memfault does christopher yes okay tell me tell me tell me i'll record it and insert it here memfault does a lot of diagnostic stuff and and capture a lot of telemetry and things from IoT devices, if I remember correctly. So they provide infrastructure so that you can have your device and have it send in a compact way, statistics and things, as well as firmware update things.

But they didn't actually, they didn't care about the data. It was all about the operations aspect. Yes, from my memory. And Jonathan Goliath provides... You just said, and I already am like, I don't know where that fits in. It's not AWS. AWS provides...

I think it is on top of AWS. If you want to do IoT, and I'm going to stop talking, let Jonathan talk in a second. If you want to do IoT, and I'm going to stop talking and let John talk in a second. If you want to do IoT, if you do it from scratch, then yes, you put a bunch of stuff in front of AWS. But that's a lot of hard work. You don't just connect AWS to your device. I think maybe Goliath is the thing that makes it a lot easier.

Do I have that right? Yeah, yeah. It's going back to what does an IoT device need from the cloud? And you could totally build everything that Memfault and Goliath and Renode does yourself. I feel like I have. Yeah, like many people have. I mean, in Nest, we had everything from scratch. Even the way we use databases predates a lot of the capabilities

that Amazon has today. But when you're talking about a product, for the most part, those things are actually not part of your business. It's not core to the value you create. Because if you're building a smoke detector, safety, reliability,

user notifications, that's your product. Great hardware, reliable sensors, that's your product. Great hardware, reliable sensors, that's your product. The things in the middle, the things that go into developing your product, the things to monitoring your product, the things to making sure that you can scale your product, again, is not part of the thing your business

wants to build and hopefully will be successful with. To your original point, over time, more and more people have realized that, whether it's coming from the open source world or commercial offerings like Goliath and Memfault, that should be a commodity. We really wish that you wouldn't have to start with, okay, let's write our own assembly bootloader, then bring in our own IP stack, and then write a custom driver with our modem vendor. And then, okay, now we have to figure out security and communication and encoding. And then, okay, well, now we have a database.

I haven't even built the mobile app yet. So how do we go from having a chip on our desk to having that end product? And that's where a lot of the solutions in space have started to be developed and show up over the last five, 10 years. It's sort of parallel to IT where five or ten, well, probably more like ten or fifteen years ago, if you were at a company, you had a server room. It was

packed with servers and you had an IT department that was tasked with managing those servers and everybody's files went on those for the company and now nobody does that. It's all elsewhere and managed by some other company because if I'm running a company that makes X, running a data center shouldn't be necessarily part of X. Yeah, exactly. Why does most companies use Gmail for their email server? It doesn't help your business by running your own SMTP server anymore.

What? This is so true, but it's hard to understand the landscape. And to some extent, it's easier for me to explain to managers and clients what they have to do to make it work instead of trying to figure out which company I should be recommending and whether or not I believe it will exist in a year. Which company should I be recommending and which ones won't exist in a year? Yeah, I think there's a third question there. What do you need from the cloud? Not everything is an IoT device and not all IoT devices are created equally and

don't have the same requirements. As an example, I met with a large consumer device manufacturer last year, and they have a lighting solution. And they're very much privacy focused, and their customers don't really want to use a cloud service. And so by design, they made sure that their lights and light switches just work. And they get paired in the factory, and there's a mobile app for some minor tweaking, but it doesn't talk to the cloud, mostly. I say mostly because they have a compliance requirement to have the ability to software update those devices, in case there's a vulnerability or in case there is a major feature that they want to push out, but they

plan never to use it. And so going from, hey, I have an offline wireless link between two devices to now I have to be able to push a firmware update to millions of devices sometimes is this huge lift. And so I call it sprinkling a little IoT. It's just as hard as building a full-fledged IoT system. Well, that company has to figure out OTA and fleet management and being able to push signed firmware binaries down to a subset of devices,

to 100% of devices, et cetera. And for them, they just need to go figure out the OTA problem and have a solution versus, let's say, an agriculture product which needs to sense a lot of different sensors all in near real-time, combine that with weather data and location data and other telemetry from third-party vendors. They have a whole other set of problems. And so it's hard, it's messy, it's complicated.

But first, it's useful to figure out or to determine, or even just to ask, to figure out what questions to ask. What do I need from the cloud as part of my IoT solution and what don't? Because that will help you figure out which vendors in the space actually serve your needs. Okay, so OTA, over-the-air, firmware update. That's one giant feature that many people don't expect the complexity of.

Even though we talk about security and how important it is not to create bricks, it's still something that gets tacked on at the very end of projects. You forget to have enough flash on your device for AB updates and didn't choose the right component with hardware-accelerated cryptography, so you can't actually do the OTA fast. Yeah, so usually it happens too late in the process, but...

Another thing that happens often very late in the project is realizing that you need to manage the device. You need heartbeats to figure out if the devices are dying or there's more crashing of this kind. This whole, what I'm saying, the operations side. And then you need some, maybe you need to command something, you know, maybe the agriculture device needs to command something that says, tell the farmer to turn on the water. Or maybe the farmer says, okay, now I want you to turn on the water and it commands down to the device. So there's this command and control and it can go either way. And that most people get. Most people say, okay, I understand what that's for and I understand why my product would need it. But the first thing most people think about are the data streams.

The sensors, where are they going? Which way do they go? And how do I collect that data? Are there other things I'm missing here? Well, so you're definitely right. The data is usually the first thing and sometimes the only thing companies think about as it relates to the IoT device because that's where the value is. Maybe there's an aspect of command and control

but how they get the data, how much data they're going to have, how they're going to store it. These other elements are really part of the early phases of the product development lifecycle, as well as the operations of the product development lifecycle. But those are kind of the main things. I would say the different phases, you might actually be using different tools. We talked about observability, monitoring the device, the things that Memfault does really well.

That's great for when you're initially developing your solution, but also as you operate the device. And some things are actually even more important at the development phase versus the operation phase. So being able to test at scale. There are companies that help with, in the cloud,

let's say scaling out your virtualized tests, like Reno is an example of a tool that helps you do development in the cloud or assist you developing with the cloud capabilities. But those kind of four things you hit on, which you can probably drill in if we want to go a little bit deeper, but device management,

that sort of command and control, the data, and really just, you know, interfacing with other systems. Yeah, that's probably the last one that we didn't touch on, because sometimes you have to talk to other systems. Maybe it's another team in your cloud application team or your mobile team, and we can call that

data, but it's actually quite a different problem, because if you get a raw sensor reading, let's say you have an accelerometer, get its raw values, you might send that to the cloud, denormalize it into actual coordinates. Well, the application team was trying to show

the tilt of a trash can on a mobile app has to then process that in something else. And that goes on to the cloud, the general cloud world. And so there's some tie-in between the world of device and device management and device data, and the world of application and application data. So there's that bridge.

Renode, R-E-N-O-D-E. I think it's Renode. Dot I-O. They do simulations. They don't solve any of these problems, actually. Is that right? Well, they actually solve a good any of these problems, actually. Is that right? Well, they actually solve a good chunk of these problems. So I was talking about the role of the cloud,

and I've started this mini-series that I call the Clouds of IoT. It's probably a good framework for us to talk about. You're giving me flack before we started that I haven't finished the blog series. It's coming, it's coming. It was a great start. And then I was like, okay, what about the other four?

But go ahead, talk about it, please. And it's trying to debunk part of these challenges you're asking about, that there's not just a, air quotes, cloud. It's not just someone else's machine, right? But the capabilities of a solution that is enabled through the cloud for IoT is actually specialized. And I talked about five different types of clouds. And to be very clear,

there might be one company that does more than one of these clouds. But at least I try to point to one specific example or two specific examples. And so Goliath and Particle and a few others in the space are what I would call a device cloud. That's the thing that helps you connect the cloud to the internet, security, software updates,

the thing that you as a firmware engineer might be using on a regular basis while you're in the prototyping phase, when you're in the pilot phase, even operating a device at scale, pushing new firmware, that's the device stuff. And so that's what a lot of conversations end up talking about when we talk about OTA, et cetera.

But there are other types of clouds that are part of a solution, and I mentioned the connectivity cloud is another kind of cloud, and we can we can double click on that I'll list the rest before before we go on in more detail what I would call the data and data cloud I'll kind of rattle these off a little bit faster so it's not taking too long application cloud and the development cloud and as an example a cloud. So if you're building a cellular-based device, you might need a SIM provider, like an MNO, an MVNO.

Think Bryzen or Hologram. If you're building an application, so a mobile app or a dashboard, you might use tools like Node-RED or Blink or Ubidots. And a development cloud is a tool that helps you in the initial phases of development and through the development lifecycle as you continue to iterate on your product.

I would categorize Memfault in the development cloud. It's a key part of writing your firmware, testing your firmware, doing new releases that is actually made better because the cloud exists versus having a similar tool on your desktop that you would batch download log files and then try to process those locally. It's just a complete

different kind of user experience than having a real-time tool that can aggregate all that information and make use of it. Those are the five clouds, device, connectivity, data app, and development in no particular order. UbiDots and Red somethings.

Node-Red. Node-Red. Where do they fit in? So what they help you do is, well, they kind of overlap. Data and apps tend to kind of share because first you need to be able to collect all the IoT data and then you need to be able to use, create dashboards, mobile apps, et cetera.

So Node-Red is a great tool for taking in sensor data or other sources of data with a low-code experience, a lot of time IoT, and then process it so it can be averaged or summarized or reformatted so it's more compatible with your dashboarding tool like UbiDots.

So Grafana, where I can just graph things, would fit in there? native Android and iOS tool. And Grafana is definitely an app builder for a lot of companies, especially if you're not building a consumer product where there's an installer or a product manager or a building manager, as an example. They just need that dashboard. Grafana would fall in that category. And the data cloud, is that just always AWS and Azure

and whatever Google canceled last week? Yes and no. And this is where we're starting to get into that. Well, some companies do multiple things. AWS IoT has some device cloud capabilities, device management, software updates, but they of course have a huge swath of stuff around data processing and data storage. But there's a lot of IoT-specific databases. So in general, IoT sensors tend to be time-stamped. So we call them time-series databases. And there's companies like InfluxDB or TimescaleDB. They build specialized databases that are extremely efficient,

easy to store the sensor-based data, but also easy to query it for visualization or later use. And so a bunch of companies make these optimized databases for things like time series data. It seems like Amazon has, to a certain extent, owned a large chunk of this market with AWS, at least the data storage market. Is there a way to spread the risk if you're an IoT company and say, well, I don't want

to put all my eggs in an AWS basket in case the Cascadia fault breaks? Or do you kind of have to choose, okay, this is where my stuff's going and that's just the way life is if that company screws up and loses my data? Yes and no. There's a standards problem in IoT, as we're all familiar. But it's not as the same when we talk about standards, we talk about data standards and how your Google device for Google Home can't talk to your Alexa.

What I mean by that more is the way that you build a device for AWS IoT is fundamentally different than you'd build it for Azure IoT or for Goliath or for anything you built yourself from homegrown. So the portability across different vendors makes that challenge. Some solutions like Goliath can work on multiple cloud providers. So if you build for that one thing, at least then it can run in multiple places. But all in all, each IoT project, for the most part, is kind of a snowflake, even if it's doing a very similar thing

to the nearest competitor. And so if you take on a whole all-in solution, let's say you built vertically on AWS, well, you can't really run that against Azure IoT. So either you pick solutions that are designed to have sort of multi-environments, which there aren't that many,

or you pick some of the pieces. So a lot of companies that I've come across who, let's say, use AWS for their IoT product, they might just use AWS's database. And then they wrote a lot of, they hired a big team, they wrote a lot of their own IoT device management on top of AWS, not depending on their solution because of that key factor.

They want to use a database that's high availability and cost effective. That same database might be running also on Azure because it's a standardized database. But that's the only real way, except for, again, solutions like Goliath that is designed so that it can be portable. But yeah, there's that baked-in challenge for most solutions. Is it baked into operating systems as well? I mean, I know AWS has a free RTOS branch, and I think Goliath likes Zephyr, which I'm liking Zephyr more every time I see it.

And Azure has ThreadX. I mean, do I have to, I'm starting an IoT project. If I choose one thing, does that mean I've chosen everything? So there's actually a couple good questions in there. And I'll start with my recommendation for anyone who's starting an IoT project. Start with your component. What chip do you want to use?

What radio technology do you want to use? Get that vendor down. And each vendor has made their own bet at the OS level. And generally speaking, those vendors have FAEs that will support you on the firmware side and the hardware architecture design side, and they've partnered with the cloud parts. So they have middleware components

that you can drag and drop and pull in. So, for example, if you're using Infideon with Modus Toolbox or MCS Expresso from NXP, and you're starting a project, you're going to have the ideal software SDK starting experience. And they have multiple RTOSs in there project, you're going to have the ideal software SDK starting experience.

And they have multiple RTOSs in there too, by the way, for the most part. And the middleware is the thing you can swap in and out. So before you get too far along into your development, you can build upon well-tested and supported middleware. And for the most part, you actually get some choice. Now, how vertically integrated that OS and middleware is will kind of make it a little bit harder. So, for example, if you're trying to use AWS,

if you started with, let's say, the cloud side, if you were to pick AWS IoT and try to design your hardware from there, well, if you're really proficient in embed, you're kind of stuck. I have a friend who has built an IoT product, and they chose AWS IoT, and their dev shop chose embedOS, and the two shall never meet. Nine months later, they didn't ship their product,

and they had to scratch the entire V1 of the solution without another platform. So if you start with the OS and the hardware, find which one you're going to get the best support, then go from the middleware up, that makes it a little easier for you to design your solution. And I mentioned

Infineon and MoSoolbox, they support FreeRTOS and Zephyr today. So you can actually go with the software that you're most comfortable with or maybe meet the timing requirements or whatever and then pick from the growing number of cloud solutions from there. We actually, of course, you mentioned Zephyr.

We're big fans of Zephyr. But the real driver for us, the reason why we like it so much, it's a good RTOS and has a good kernel and drivers, etc. But the key, and it may sound a little repetitive, is that vendor support for

Zephyr. And so we know that if a customer is using an NXP part with Zephyr, we know the FAE who maintains that SDK and integration and that we can find bugs, whether it's our bugs or their bugs, and have that resolution.

So IoT is such a complicated thing. You want to reduce as many variables of complexity and find the most amount of support you can. Picking that hardware OS is usually going to give you the happiest path, and then adding the cloud on top of it will just continue to reduce those areas where things can go awry. I have a napkin sketch for an internet-enabled fishbowl.

And I don't write code, and I don't want to know about firmware updates. Is there a completely vertically integrated solution so that my fish can talk to me while I'm at work? Yeah. So there are projects and there are products. It's something very important to keep in mind. I'm a maker. I build hobby projects on the weekends. I've also worked on products that have supply chain

and design for manufacturing and customer support and all the other things. The category of things you could build for projects, there's actually a huge ecosystem in the maker community. We're seeing a little bit more of that makery low

technical effort making its way into finished products or tools for creating finished products. But by and large, almost every product I've ever worked on or customers I've ever worked with, they have a C, C++ developer, they have a hardware and loop testing lab, they are doing manufacturing with a CM,

where things that are low code and low friction for, let's say, non-engineers to design and develop really fall in the category of makers, educators, etc. And there are solutions for sure that are great. There's tools like Blockly, which have been

integrated in projects like the BBC Microbit, but we're starting to see more of that kind of, let's say, drag and drop, or don't require you to write C code, and things like Arduino, the Arduino Cloud IoT platform.

They have the ability to design a solution and actually generate the code for you. And under the covers, it's Arduino and C++, which is pretty neat stuff. But if you're thinking about making 10,000 of those fishbowls, you probably want to start there, Arduino and C++, which is pretty neat stuff. But if you're thinking about making 10,000 of those fishbowls, you probably want to start there,

get the solution with a vertically integrated product, and then find someone who can design a PCB or a breakout board, write some firmware, think about the OTA, because one, it may not be secure, it may not be scalable, and it may not be cost-effective. So it's a good place to start. But certainly, I haven't seen a lot in the partizable realm yet.

You made a good point that trying to, and this is something that is philosophically where I am as an engineer, is try to make things as easy as possible for yourself. Don't reinvent the wheel. Don't go off re-implementing things that other people have implemented. Try to find solutions that are, quote, you know, not your job, not part of your product elsewhere. Are there pitfalls and shiny objects that people need to watch out for that can drag you away from the easy path because they look like a great idea?

Have you seen a lot of that kind of thing where somebody's like, oh, I saw this new technology and I'm going to go for it? How do you advise people to be cautious about what they're implementing? Well, I think there are shiny objects all the time in software. I think programming languages is one of those examples. There is amazing stuff happening in MicroPython and Rust.

And there are people actually starting to ship product with those. But if you're trying to build a consumer-grade IoT product, are you going to be able to find 30 Rust developers who know the Tokyo real-time stack and be able to interface with the networking? That tends to be a challenge. And so I go back to, what is the prescribed path? What is the happy path with the most amount of support? Whether that's from the chip you bought

and the vendor who makes it and the support team they offer you as part of your being a customer to the modems and the other forms of connectivity. Are there stacks that you can just grab as opposed to finding other challenges? Because IoT is probably the culmination of most of software, right?

It's hardware, it's embedded firmware, it's networking, it's modems, it's connectivity, it's security, it's infrastructure, it's servers, it's databases. Applications. Yes, mobile apps. Everything. Yeah.

And so if you can just circle around the path of least resistance, you're just going to be more successful because there's so many places where you can either get lost or confused or fail, really. And you'd choose based on the device first. Yeah. I would actually say choose on the product first.

So that fishbowl example, is there going to be a consumer product? Is there going to be an aquarium? Is there going to be a Vegas hotel? Is there going to be an aquarium? Is there going to be a Vegas hotel? Who's going to operate it? Does it need to have one controller? Multiple people controlling it. Is it part of a larger system? A consumer electronics company might do what's called a MRD, a market research document, understanding the landscape of how people

use products today, existing solutions, and how people interact with them. And from there, you can start to pan out or build out what kind of connectivity do we need? What kind of on-device computing do we need? Do we have to interface with other devices locally, over the internet, or with some other transport? And from this pyramid of figuring out what you're trying to build as the product,

you can start to develop that sort of roadmap of the hardware design, the firmware design, what kind of connectivity in cloud. And of those five clouds, what do we actually care about? What do we actually need in our solution? And not just pull on something because it's shiny and new or someone told you to use it. I mean, I totally agree with you.

I like the way you've broken this up because the device cloud is an area where I spend a lot of time and then usually I have a UART or something that goes to the connectivity cloud, which might be BLE, it might be 2.4 gigahertz radios, and it might be cell modems. And for the most part, I don't care. Those are very separate things. And what I talk to after that, yeah, I mean, it's fine. And then whoever, and then as the clouds go up, you know, the application cloud, sure, as long as the data gets to the data cloud, I don't care what you do with the

application cloud, as long as I can understand that my device is working correctly. And really, that goes back to the development cloud, where somewhere between the device cloud and the development cloud, I know my devices are being updated, I know my devices are alive, and that they are performing as their self-test indicates they should. But who wants to plan anything? I mean, nobody wants to plan anything. Everybody just seems to want solutions or to write it themselves. Yeah, yeah. It's interesting.

I did a Twitter survey a few months back about sensor drivers. And FreeRTOS is a kernel, and there are distributions of that kernel. I found out that most of the distributions don't include sensor drivers. So Espressif has the ESP-IDF, which is based on the FreeRTOS kernel, and they've brought in a bunch of stuff

to make it really easy to use FreeRTOS. But every single person who uses the ESP-IDF writes their own sensor drivers. You know, it's a little ITC here and a little accelerometer there. Which, you know, when I asked why people do it,

I've always done that. Versus grabbing a library of well-tested sensor interface from maybe the sensor manufacturer. And so I think there's... Wait, those exist? Where? Some do. Some do.

Or different operating systems include those. So like Zephyr has a whole sensor library with a bunch of sensors provided by some of the sensor makers. But for example, Bosch has a bunch of sensor drivers and some of them are actually optimized and lo and behold, actually performant. But people still end up writing their own Bosch,

you know, BME 680 sensor driver from scratch. And maybe that's just the way some people are wired and there's not much you can do about it. But if you would have looked around and maybe surveyed the ecosystem, those early decisions, that planning, maybe you picked a solution that had some of that already built in, that you didn't have to focus on those things

that you were doing over and over again with every project. I think that's one of the things embedded systems is going to be changing in the next five to ten years, is that we're going to start figuring out that we can't keep writing the same. Please don't make me write another accelerometer driver. I mean, I haven't written a UART driver in years and before that. UARTs at least are well covered.

I mean, I used to write, you know, one or two a year. And now it's usually, I don't have to do that, but I'm tired of writing display systems and. Well, even display. I mean, yeah, it's, it's usually, I don't have to do that, but I'm tired of writing display systems. And even the display, I mean, yeah, it's a mixed landscape. And like, I think you're right, Jonathan, when you're planning out your hardware in order to walk that path of making it easy on yourself, picking hardware that you don't have to write a bunch of software that is not part of your product's job. Nobody

bought your internet-connected fishbowl because it had a custom spy driver for the light. And there's things that do matter, especially in the world of IoT. A lot of applications are, say, power-sensitive or memory sensitive.

A lot of the companies we work with, they spend a lot of time on power management and actually taking the device driver they got, it was working, and they were optimizing the driver. That's, one, is more interesting and relevant to your product, but two, sometimes more fun, depending on the DNA of the engineer. So it's better to spend time on those kind of things as opposed to doing the defaults, you know, pull off the ITC buffer, read it, dump it on the UART, and then, oh yeah, cool, that works.

Okay. So cloud layers, I think I have them. And data that we talked about initially goes pretty well into these cloud layers. Device management goes to the device cloud. It might go also into the data cloud, but it definitely comes from the device cloud. Is that right? Mm-hmm. Okay. And then we have the data stream, which comes from the device cloud through the connectivity cloud into the data cloud, and then probably the application cloud does something with it. Yeah.

Yeah. put in some place things instead of just general names. Let's say I want to talk about the data stream, and I'm going to choose Goliath because I'm talking to you, and it seems rude to choose particle at this point. What happens next? Which connectivity cloud should I use? What does going to Goliath mean? Does that mean that I also have to use some other piece?

This is where I get to say it depends. Of course it does. I get that one card, right? And it really does depend on the product and the company. Because a lot of products, there might be already engineers in that company. There might be already building mobile apps. So you probably want your mobile team to have access to that data

coming off the sensor and just let them run with it. But if you're a consultant and you're working with an entrepreneur who has a great business idea, you're designing the PCB and some of the firmware, you kind of don't have that resource. So you want something that's super well

integrated with wherever the data's coming off of, and you can just go and drag and drop and do what you need on the visualization piece. And so we see a lot of mix. And if you're not building a cellular device or a LoRa device or anything with a managed connectivity, you don't even need to worry about connectivity cloud, right?

You're building a Bluetooth app or you're doing something on Ethernet or Wi-Fi, and you can just cross that part of the stack all together. And so if we go back to your fishbowl example, your fishbowl monitoring example, it's probably, I don't know, some pH sensor in the fishbowl monitoring example, it's probably some pH sensor in the fishbowl, maybe some

turbidity to measure the activity of the fish. I don't know. You want to stream that data and show it on a mobile app. It's a good example to use as a framework. You have picked your Wi-Fi components and the sensors and you've got some firmware and you have your UART-Fi components and the sensors,

and you've got some firmware, and you have your UART plugged aside, and you're seeing the data there. And so probably the most commonality across all these types of products, whether it depends, is some form of device management. So being able to connect that device to the internet, secure communication. Instead of having it on your, maybe you have it on our log output in the cloud. And you're like, cool,

this is connected to the internet. Now I want to test out OTA, so I build a new image, push it through the device management solution, and you see it, instead of blinking red, it blinks the green LED. Great, OTA works. And now I need to show to my investor,

or my, I don't know, my business partner, the values coming off that device. Log output's interesting, but let me just grab Grafana dashboard, wire up the data from the sensor, the streaming information, and then show the pH levels rising and falling or the turbidity as you kind of twirl water around there.

And so that's your prototype. And that kind of checks off some of the boxes. And as you go from that prototype to, let's say, a pilot, a lot of other things have to happen. But fundamentally, you're just building on where you started. Maybe you replace Grafana with a web app that the subcontractor built out

and that's taking the data from your device management like Goliath and then dumping it into a web app instead of a Grafana dashboard. And then maybe you decide that you want to have an outdoor fishbowl so you have a swap out of components and you now have a cellular device instead of a Wi-Fi device.

Now I have to go find my connectivity provider, bring that in, and these two skews, one is the Wi-Fi, one is the cellular, one has connectivity, one doesn't, and you can continue to refine the product. At some point, usually in that pilot phase, you have all the cloud parts figured out. Who's doing what if you're using a third-party vendor,

let's say Grafana, or if the web developer built your own dashboard, whether you need connectivity or you don't. Your device manager is probably plugged in from early on. And the rest is product development. New features in firmware, testing, changing from a Adafruit Feather board to a custom PCB. But I think one of the key things is there's a certain point where you don't have to make any decisions anymore

because you've figured it all out. And now it's just matched to whatever those initial product requirements were i'm always afraid of that i mean what if i chose wrong what if i chose sigfox so what now and what if i had a laura device i wanted it to be super low power and because you know everybody else was using the things network, I was sure it was going to succeed. But now I have a device and I can't talk to it anymore because I chose the wrong connectivity or I chose a data cloud that doesn't exist anymore. That's why you put eight different radios on every product so you can adapt as necessary. Yeah, and I think one of the things that has changed,

we talked a little bit about the changes in embedded and how reuse is becoming, moving towards commonplace. Just the ability to prototype as a professional engineer has also changed significantly. Oh, yes. You can actually build a working prototype with dev boards and breadboards, and it will be a lookalike. RF might be different in power, but at that phase, unless you have a very specific application, you're not optimizing for those things.

You want to know, does this work? Does the idea pan out? Can I build it with these basic components? And, you know, choosing flexibility around the software you're using. So let's say, we talked about operating system earlier, having one that will allow you to, let's say, swap out a cellular AT modem for a Wi-Fi AT modem. Those kind of things can help you at least reduce the risk at that phase. And in some ways, even further down the road, as you get to, we deployed in our first 100 sensors in the field, it's like, shoot, I need to do OTA,

and LoRaWAN doesn't support OTA. What am I going to do? That low-cost, almost low-cost prototyping phase, you can go back one step without a significant, crazy amount of cost versus you have to manufacture the full thing with enclosures, with IP ratings, with FCC, and then find out when you have your first thousand in the field, oh, shoot, I can't get through a metal box.

I should have thought about that earlier. I should have gone with plastic. So I think those tools, whether it's the cloud-enabled versions or counterparts, we've gone with debuggers and developers and IDEs and just cheap commodity sensors that you can easily wire in

makes it a little bit easier, a little bit less scary, but still risky. I was teaching my Making Embedded Systems class today, and their homework was to make Blinky work. And when they heard that last week, I'm pretty sure I could hear their eyes rolling. It's like, make Blinky. Everybody's made Blinky. That's dumb. And then today when they had to answer

the questions of, okay, did you step through the code? What compiler did you use? Did you use one of the hardware abstraction layers? Are you using an RTOS? Did you use the class board or something else? And suddenly the tooling becomes so much harder than the development, than the engineering. I mean, the tooling is engineering. I shouldn't say it that way because that makes it sound like it isn't. It totally is. But it's the part of the engineering problem that we don't always remember. And I feel like we're hitting that with the different clouds that, okay, it's all there and I might be able to put it all together once, but I don't know that I could put it together in a way that I could be confident it would still go together that way in a month,

or put it together in a way that I'm confident somebody else could put it together with my notes. It's a tooling problem right now. Do you see that as well, or is that just me not knowing the landscape? No, I think the tooling problem is one of the things holding embedded back. I really enjoyed Tyler from MFAL when he was talking about the analogy that a lot of large software companies have engineers dedicated just to make tooling better,

to make those engineers highly productive. There was a humongous group at Google just building internal tools for code completion, for code sharing, for continuous integration and deployment. And you can see that. I remember a lifetime ago, there was a public API we wanted to develop.

And a single engineer could build that public API. Draw the code, configured it, and deployed it on a global scale. And that's only possible because of the tens of thousands of hours from hundreds of engineers who built the tooling to make that possible. And we don't really have that. And I think we can pontificate a why in the embedded world that's the case but there's at least one component that I feel pretty strongly about

is because even though there were teams of embedded engineers the embedded work was mostly individual you were running your driver, you're integrating into some code repository but the tools that you interact with are the thing at your desk

the code on your screen the IDE that's configured in your environment. And as modern software practices have made things more distributed, but also it's a complex system of even things that you don't even know exist yet or haven't been built yet, that doesn't work anymore. And things like modern development tools that we are seeing in the classic web server and mobile application are slowly trickling

in to embedded. And by tools here, I mean more of the standard definition, like an IDE, a compiler, a debugger. But also, the things around those tools, like observability for

what's happening on your device at your desk or in the field or how to get a packet off of a UART into a database and all the things in between. We're still getting those more classic tools to be improved like Visual Studio Code.

When I was working at Particle, we launched a plugin for Visual Studio Code and no one was doing your terminal output from a physical device. So figuring out how to plug into slash dev slash gt0 on the Linux machine and then show the log output from a Particle device was actually really novel at the time

and hard to do and hard to repeat and wasn't reusable because it was tied to that specific platform. But now Microsoft has a team building embedded tools for Visual Studio Code. Things like step debugging for real-time operating systems, things like terminal output, and yeah, we're slowly getting there. And I think the tools around the IoT parts

for that part of the embedded world is also catching up, like Memfault, like WeNote, like Alive and others. So many people are used to rolling their own because they control all of the pieces, which means they don't have to worry about a company going out of business, which I think is a common fear. How do you, as Goliath's CEO, say,

no, no, we'll be here next month and get people to believe that? As an engineer, on the other hand, how do you figure out when to trust a cloud service? Those are three good questions. And I think part of it is just some of the things that are happening in the IoT space. So specifically IoT, not talking about Goliath yet. Open source is a thing now. And leveraging open source has actually been a big part

of our company and our product, which I would say 10 years ago, we couldn't have built some of the things that we wanted to build that we could build today because of leveraging open source. That's both on the device side, but also a lot of stuff we do in the cloud. And our approach to interfacing with device-side firmware

through the RTOS is we just have a little library. So you can look at our little library and see all the codes there and all the interaction points. And if you need to design Goliath out, you can just take that library, figure out what we're doing, and swap us out. It doesn't prevent us from going out of business, but it certainly prevents the risk of you

being stuck with a service that no longer works. And on the other side of the UART is the cloud communication and the security and other things that we talk about. Our approach to the device management, device connectivity,

security, all those other pieces, is using open standards. Either open source standards or open standards that are defined by big groups of PhDs. And so we document all of our APIs on the cloud side. And you can go on our website and see our reference information and you could literally reproduce what we're doing from the device to cloud communication.

And so the device side is open source, the cloud communication is open and documented. If we went out of business, we could show people today how to design us out so that you can limp along until you design your own solution. And so that's like worst case scenario which people want to hear.

From a business perspective and when we have these conversations, especially with larger companies, they don't, they have those concerns, not just an engineer from a business perspective, and they don't want to build a device management platform, again, because it's not part of their core product and it doesn't create value.

But they need to be able to control it, they need to be able to understand it, and they need to be able to communicate that to their end customers. And so part of our product has an enterprise version. So we can run a version dedicated just for your company on Azure, on Google, on an oil refinery if you had your own data server there.

And that allows them to have that control without having to basically build a glide-sized company. And those, between the openness of our solution and the ability to actually run it themselves without having to build it themselves, is how we mitigate some of that. It doesn't make

everyone feel good, or answer all those questions, but at least it's different than the, you know, lots of IoT companies today, especially consumer ones, they just run out of funding because they built something proprietary and did it from scratch. Literally, their gateway devices, their lighting controllers

just stopped working. At least that's our answer. So as an engineering side, how do I avoid vendor lock-in for my IoT tooling? Is it even worth bothering to worry about that? Or do I just figure out what my company needs and not worry about... Yeah, three abstraction layers.

Yeah, how many abstraction layers do I need? There's a fine line, of course, because the more abstractions you have, the harder it is to maintain. Probably performance gets impacted. But what I usually recommend, or the analogy I usually give

when I talk to engineers as opposed to the business person is when you're designing a PCB, you have a keep-out zone to deal with issues like, oh, what if there's a component shortage and I need to redesign my PCB? Do I have to redesign the whole thing or just swap out the modem? Or if you're

building, you know, writing some firmware, you don't have to create your own Halibstraction layer on top of a Halibstraction layer, but maybe modulize your code a little bit. And that's probably good practice. And I would say best practice. And because at the end of the day, the more effort you put into

buffering from, let's say, the chance that there's a solution that no longer works or you have to redesign somewhere in the future, well you really just, you can't abstract, you can't spend all your time writing that abstraction. So if you can prevent the inevitable redesign,

if you're afraid of that, without having to spend as much time preventing that through abstraction layers than you are writing firmware code, you're doing it wrong and you really have to strike that balance. And the secret, and I'll tell this to your listeners,

about IoT products is they're actually pretty simple. All this complexity we talk about, the thing that's happening on the device or the thing that's related to the IoT parts is actually usually pretty simple. Other companies take care of it, like your module lender doing the modem

is probably handling security and certificates and network packets. But the actual thing you're doing is probably just taking the sensor data and then sending it to a UR. And that piece can probably be redesigned if you really need to without too much pain. It's the actual application, the firmware, the logic,

the algorithms, the power management, all the things that's your product, your code, your innovation, that's always going to be complicated and that's where the time should be spent. But that IoT part actually tends to be pretty simple and the least amount of places you have to worry about. It's where scaling and large numbers of units come in that it comes back to being a device problem.

That getting a bug report that says unit number 1,000,012 crashed. That's not actionable. So that comes back to device management. Does Goliath do any of that? We do a bit. We've decided to focus on our swim lane, which is connecting device, securing device,

making sure they're online and healthy. But things like understanding the firmware and what's going on on that particular device at the stack or heap, we don't do. That's where folks like Memfault do really well. And kind of where we stop and they start, we see a lot of folks using those solutions together, actually. I kind of feel like I need a map of, you know,

those cartoon maps that are for tourist cities. And over here is the application cloud and these are the stores and restaurants over there. And here's the device cloud and here are the stores and restaurants over here, and here's the device cloud, and here are the stores and restaurants over here, and look, there's a little road that goes between them. I think you need to do a whole book of understanding embedded systems through maps.

Okay. I think you do a great design of... The memory maps. Yes, yes, exactly. Yeah, I had that visual in my head of IT products and solutions. Yes. If somebody could make my memory map land,

but instead for IoT solutions, that would be awesome. And then we can have the ocean of AWS, and it can have little ports along little areas that it needs to touch. FreeR TOS can be like the planes. Anyway. FreeRTOS can be like the planes. Anyway, so you were a product manager and did some code for Particle, but now you are CEO. How's that gig going?

How is it different? You both run your own companies, and I would say I spend a lot more time running a company than I am product managing or writing code. Things like payroll and insurance and weird laws between states because we're a remote-first company. But that part aside, it's just been kind of fun to scratch the itch and build a team

and work with some great people. It's not for everyone because, you know, it's Saturday and I'm probably going to get off this podcast and do some emails, but I think it's worth it. And you've built a pretty fun team, it looked like. Indeed. And let's see, one more question. What are the best and worst uses of IoT that you have seen so far?

Is that a listener question? No, but I think maybe they asked what was the worst, but I added what was the best, because you can't just have the Internet of fish bowls. It's a fishy example. Well, it's worth calling out. There's a Twitter account, the Internet of where they literally have

an internet connected toilet with health monitoring capabilities. I'll leave it for the listeners to look up what that means. My rule of thumb for connecting a device to the internet is it should be better than if it wasn't. Yes.

Don't take away the features because you've added the internet and want me to pay for them every month instead of just the once. Or actually make it perform worse because now you have to have a 2x size battery to deal with the Wi-Fi radio you put on there. And it literally does the exact same thing that it was doing without internet. You just created a dashboard. Except now you have to update the firmware every six months, randomly.

Hopefully there's a firmware update every six months. You're not stuck with a terrible device. I would say my general rule of thumb is something that senses something in the world that you wouldn't have knowledge otherwise. Something that provides automation or a meaningful enhancement of convenience. I'll try to describe those.

Or let's say does something really cool. That farming example we used earlier, you have no idea how much UV, how much rainfall or moisture is happening to a crop at any form of scale without having some sort of sensor there. And with the cost and sort of deployability of sensors, you now have insight into your crops that you never had before, except for sending someone else to put their thumb into the

ground. You still wouldn't know about those. So that actually improves crop yield and crop health and reduces environmental waste on water and other rolling up trucks to go, or tractors to go check on those things. So getting information which

you couldn't get otherwise, I think, is my favorite use case of IoT. And this idea of convenience, not just, hey, I'm sitting on my couch, I want to turn on my lights. I like consumer products altogether. But

being able to monitor things that you couldn't easily send a truck out to. Talk about waste management, elevator monitoring, street lighting poles, windmills, or wind generators. It's really expensive to send someone

up there, like thousands of dollars. And instead, is the generator working? Thumbs up, great. We don't have to send someone out there, dangerous, whatever. So those are my two general use cases of when I think about,

okay, this is IoT making devices better, products better, our environment better. I just think of cool things like, I don't know, particle accelerators and being able to beam form crazy stuff and the convenience of don't know, particle accelerators and being able to beam form crazy stuff in the convenience of the upstairs lab

where, yeah, you could have a PhD, sit there with a dial and move things around, but being able to have that remote control experience, there's fun examples. There's an example that Zach, the

CEO of Particle, used to always bring up, which I also agree is a terrible use case. There's a basketball from a basketball company. It was a Bluetooth monitor with an accelerometer so you can monitor your shots.

They had to put the Bluetooth radio and the accelerometer somewhere. So they put it at the bottom of the ball. Well, if you dribble the ball and it hits on the sensor, it just, one, breaks the sensor, and two, doesn't bounce back. So you've literally made a basketball not a basketball anymore.

So it's just figuring out where it can enhance that versus... Yes, make things better. Jonathan, it's been really good to talk to you. Do you have any thoughts you'd like to leave us with? Yeah, I think IoT is such a

fascinating space. It's obviously what I spend a lot of my time and my career on, but it's complicated. And if your listeners are thinking, wow, there's so much stuff I've got to learn and figure out, yes, that's what makes it both enjoyable but also complex. And the

advice I give is find resources, whether it's if you're working with a chip set vendor, get some help because they have engineers who are helpful. There's online communities. There's very active Slack like Embedded and Interrupt as well as Discord. They're popping up all the time. And local enthusiasts who are bridging the divide between Embedded and

hardware and connectivity because we can figure this out. We see people shipping products all the time. Just know that if you feel overwhelmed, it's not just you. It's a vastly complicated space, but there are people here to help you figure it out. And so if you want to collect all of the Pokemon that are the IoT tools companies, you should search the internet for IoT Pokemon.

I wonder what's going to happen there, if anybody actually tries that. Any product idea? Our guest has been Jonathan Berry, founder and CEO of Goliath. Thanks, Jonathan. Thank you both. Thank you to Christopher for producing and hosting, and thank you to our Patreon listeners Slack group for their support and their questions. Of course, thank you for listening. You can always contact us at

show at embedded.fm or hit the contact link on embedded.fm. And now I do not have a quote to leave you with. I think Jonathan has said it all.