Embedded - 447: All Sorts of Weird Problems

Episode Date: April 13, 2023

We spoke with Chris Gammell about IoT, podcasting, relaxing, and learning. Chris works at Golioth.io. They have a neat blog that talks about reference designs, Zephyr RTOS, and making products. We talked about ESP chips which are made by Espressif. The ESP32 line is RISC-V. Constrained Application Protocol (CoAP) Some YouTube channels we discussed: Wendover Productions: explaining stuff CGP Grey, especially the recent one about vexillogy and US state flags Blacktail Studio: Soothing woodworking Adam Neely: music theory Shawn Hymel on Digikey’s channel explaining continuous integration and delivery: Intro to CI/CD The H note in music Want to know more about self-paced Making Embedded Systems? Sign up for the waitlist at Classpert. Want to learn electronics? Check out Chris Gammell’s Contextual Electronics. Transcript

Welcome to Embedded. I am Elysia White, alongside Christopher White. But I really want to confuse you today, so we're having Chris Gamble on as well. Hi, everybody. I'm back. Thanks for having me back. How are you doing? It's been a long time. It has been a long time. I will be honest, most of the time I'm hearing both of you at like 2x speed. And so this is like down-tempo. This is a down-tempo embedded FM for me. That's the live thing. I'll try to talk a lot faster. We can talk a lot of time in the internet

as well in places like the amp hour and contextual electronics and i work for a company called goliath i run a consulting forum and i enjoy talking about electronics on the internet with folks like yourself you don't introduce yourself as an electrical engineer. Oh, interesting. Yeah, I guess I haven't done that in a while. You know, usually when I'm introducing myself in public, I usually just say I design electronics because that's pretty straight to the point what I want to talk about anyways, you know, and because some people are electrical engineers, they're working on power lines, whatever. And I want them to know, I want people to ask me about electronics. Is that selfish? It's selfish.

And the lines have gotten very blurred in recent years. Yeah. Yeah. I have been doing more firmware. That's, um, I was excited to be asked back on here. I've been voraciously listening and, uh and all your great guests and doing more studying and learning alongside my coworkers. And so I don't consider myself a firmware person yet, but I aspire to be. Are you ready for lightning round? Sure. What is your favorite lightning round question? Robot.

What is your favorite fictional robot? WALL-E. Okay, I just figured I had to ask it since it was his favorite. Yeah. You go to an industry conference evening party where you don't know anyone. What do you do? Bring out the electronics that I brought with me and start showing it to people. I feel like we're starting a role-playing game.

Somebody reacts negatively to your electronics. You turn left and there's a door. You forgot your power supply. What do you do next? What is your favorite mic to use yourself for podcasting or anything? I'd say Zoom H1 is the most versatile because I bring it mobile with me.

If you didn't have a podcast, would you start one now? I'd probably start a video channel now. Favorite IoT device? Chipset or end device? End device. Pass. Least favorite IoTot device most of them

do you have a favorite kid's toy um i it's not technically a kid's toy but i i own a melodica and my daughter plays it a lot and i think that's my favorite kid's toy what are your personal feelings about firmware ranked from one to ten one being it should be banned ten you'd like to run firmware in your own head five what would you have answered five years ago five oh all right you're non-committal but those were big extremes if you could teach a college class what would you want to teach oh uh take your time this is kind of like 2x yeah how to like like how to build anything like the mit style class but probably i wouldn't be that good yeah you didn't say what you're qualified

to teach it's what you want to teach oh yeah what yeah. What I think, how about this? Can I rephrase the question? What is the class that I wish I would have had, maybe? Sure. And it's like a mix between how to build anything at MIT and the analog electronics class that Larry Sears taught at my alma mater out of art of electronics. maybe also Jay Carlson's class too can I like smoosh a bunch of classes together sure I have no problem with that okay great we'll make it a whole semester yeah whole year whole college career how about that so I hear you work at a company called Goliath. Nice one. Yeah. I work at a company called Goliath and I've heard it pronounced many different ways.

I've seen, usually it's easy to tell when people are writing from that, like without much context, because then they, they spell it the traditional way with an a G O L I H. I corrected all my notes this morning spell check entire notes this morning because i forgot that it was right and if people haven't seen the actual spelling it's there's iot in the middle and uh and i have razzed john barry the um the founder

and ceo many times about it um there are many companies out there where if you like look at the spelling like wait why is that spelled like that and there's there's usually an iot in there somewhere um so you know what are you gonna do what do you do there i am the uh developer relations lead and for hardware and firmware people i often transpose that to be it's like being an applications engineer um so i do a lot of reference design hardware. So basically, I'm the kind of the first user of a lot of different things. And I work with my good friend, Mike Stish, former editor of Hackaday and firmware par excellence. He's my, my right hand man around firmware and teacher do you have like people users come to you and say here's what i'm trying to do can you help me or i'm having a problem or are you

more on the push stuff to developer side yeah it's the second one um and actually we're trying to hire the first one right now because so the way we split that up is like um if someone who came us and said, hey, help me build this thing, it would be like a field applications engineer. Right. Because they're often kind of re-implementing a lot of the stuff that an applications team at the factory might do. And so Mike and I are making this stuff kind of at the, you know, just trying to like retarget all the Goliath stuff to different platforms, different application spaces, different hardware, different, you know, weird visualization things that we can dream up, that sort of thing. So it's really like a playground in the IoT space, which is great. Actually, maybe we should do a little bit better definition. I mean, we've made jokes about IoT being in the name Goliath and that's why it's possibly actually pronounced Goliath. But, okay, so there's

something to do with IoT. There's something to do with Zephyr. It's got hardware. It's got software. What is it? What is it? Yeah, that's a good question. So the way I talk about it is there's two pieces. The product, the actual thing that people would and hopefully do buy is the server stuff, right? So it's a co-op gateway, and there's a bunch of databasing stuff, and there's management consoles, like a website you can log into and see all the devices you have, device management console there. And then there's a bunch of services, managed services built on top of it. And then there's the internet and, uh, and then kind of make it suitable to talk to things that are very processing heavy and out in the world, out in the cloud world, basically. So you think about like, uh, someone's running a massive database on AWS. It's not a great idea to do an HTTP call from a little cellular device out in the field that's battery powered. It's better to do it over co-op, which is a constrained application protocol.

And so it puts some compression on it and to talk to this gateway and then use a bunch of the APIs that are set up to handle all of the data that you want to be sending back and forth to that constrained device. And so Goliath is that cloud stuff. That is the actual product. Then there's a second device side aspect, which is the enablement. And that's basically now that I have this co-op gateway and all this stuff that sounds big and scary, which it can be, how do I actually talk to it from a piece of constrained hardware, like a cell, you know, cellular IoT device. And that is the device side and the enablement piece that is basically Goliath has an open source SDK or multiple open source SDKs built around NRF Connect Studio,

which is based on Zephyr. There's an NXP port, also in Zephyr. There's a Modus Toolbox SDK, which is really free RTOS. And we can talk about the free RTOS versus Zephyr thing. And then there's an ESP IDF SDK as well. So in all of these different vendor ecosystems, all of the things that it takes, so like a simplification of all the stuff under the hood, not the connectivity, but like basically you don't have to open any sockets. You don't have to manage any of the connection stuff. It's basically a single line call. And all you say is, I want to send this sensor reading to the cloud and I want it to be

timestamped. And that then is handled all the way up the chain, all the way through the cloud. And then that data is available for a certain amount of time on the Goliath platform. And then it can also be exported out to these large hyperscaler things and your cloud team that might want to have it, you know, pass through in a HTTP, you know, a JSON packet, basically over HTTP. So kind of just that, you know, I think about it like Goliath helps with that really tight pipe down to your constrained device in the field, and then helps that communication between the cloud and your device. And then it turns around and says to cloud teams and says, hey, now that this data is here, let's make it easier for cloud teams to process it and put it on an app and control the device that's out in the field, that sort of thing. So there's a lot there. Yeah. So I'm going to make an example because I need concrete things.

Let's say I have an array of a thousand seismic sensors. Okay, that's a great example. And they're scattered around, I don't know, let's say California. And I can get GPS readings. I can get good time. I can get the current activity level of tectonics. And so I bundle these up and I send them to a Goliath server through some magic, through some Zephyr magic that's a single call and super cool. And then Goliath acts as an intermediary so that I don't have to talk directly to an AWS server or some other server that may change.

Yeah, that's yes.'s that is all correct stuff yes and do you handle logging errors and firmware update that's right yes yeah so um so just let's just stick in the zephyr world because that's just a that's where I'm mostly operate with, with our device side stuff. Um, so Zephyr is a real time operating system and ecosystem. Uh, and, uh, what we do is inside the Zephyr Goliath Zephyr SDK is there's basically hooks into a lot of the stuff that's under the hood. So under the hood in a lot of Zephyr systems, well, I think actually all the Zephyr systems, the bootloader is MCU boot. And there's a logging subsystem as well. And so Goliath then, the Goliath SDK hooks into the logging subsystem. So the four levels of logging that you might have, which are usually debug, info, warning, and errors that might pop up on a serial console, you basically flip a switch and you say,

I want this to hook into the goliath sdk and then all of those logging messages also get pushed up to the goliath cloud and they're visible on the console you can push them out externally if you wanted to same thing for the the over the air updates you decide you want to use the OTA service. I'll say OTA normally for over the air. And you turn on a couple of switches, you configure some stuff, you follow some of our example code,

and then it downloads. Basically it listens for new firmware updates. It downloads the package to a secondary memory location. And then it uses the, it uses the MCU boot APIs to say, hey, there's a new image here. It's been validated. Go switch from zone A to zone B. And I'm sure both of you can relate to it. All of this stuff has always been possible, but it's often been custom. But there have been other companies that have been trying to do this.

I mean, Electric Imp and Particle and Memfault. Memfault's a little, a corner of this though, right? They don't do so much the telemetry. So Memfault does really great OTA stuff, really great OTA stuff. But yeah, they don't do the, they kind of say the data stuff. That's, you know, you do that however you want to. And so we know the Memfault folks, they're great. They do a really good job. Some of the other examples you gave there are tied to hardware and that's really the big, so there's two, two big things there. One is if you want to do custom hardware,

you're kind of out on your own there. So some of the other platforms that are vertically integrated, they really depend on having kind of like a daemon sitting on the custom hardware where there's some part of the code that you don't control. So you're not compiling in the OTA part. It's already been pre-compiled or it's built into the hardware or you're doing like a two-chip solution there. So that's one thing. And then another thing is, even some of the platforms where maybe it's optional custom hardware, well, the, the reason that I joined Goliath was actually john sold me on the thing when you when you go to production, which is what happens when you don't want to use Goliath specific managed servers anymore? What if you want to like,oliath-specific managed servers anymore? What if you have 10 million devices?

And the answer is the Goliath cloud piece, the product piece, that can be packaged up and then licensed to go on anyone's infrastructure. So it can go on your AWS instance. It can go on your server closet in your facility, right? Behind a firewall. You can basically take the Goliath product, the software, and deploy that anywhere. And that is not completely rare.

There are one or two other companies that do that. But in this specific space with this hardware focus, I had never seen that before. So you said licensing. Are you making money by licensing the whole thing so other people can implement their own servers

or by not the hardware because you're allowing people to design their own? And the device side is a free api so where does the money come from where's the money come from good question uh yeah i think that's always a good question when you're talking to like a startup and you're considering using power harvesting yeah yeah tiny tiny bitcoin miners uh no um please don't make that the title of the show that would be really false advertising right i know how you do uh uh yeah so uh at the beginning so like first 50

devices are free if you want to just use the so like the goliath cloud solution right now like if if the stuff i use every day i talk specifically to the Goliath cloud version. And that's basically the Goliath software, like run the Goliath, effectively the Goliath enterprise software running on our own Google cloud instance. But it could be AWS, it could be Azure, whatever. And that's basically you pay past 50 devices, you start paying per device per year sort of thing. And that then, you know, much like DigiKey, the price drops as you add more units to it. But at a certain point, there's a crossover where it makes sense to basically just have your own, your own cloud and your own, sorry, your own version of the cloud. And then that's when licensing comes in. And normally that is, um, uh, administered, still administered by Goliath, but it's your infrastructure. So you're paying for the Amazon EC2 instances or whatever the thing it's installed onto. I guess it would actually probably be like Kubernetes stuff. I am not in that, that world. Um, and, um, so I probably sound stupid there, but, uh, that would

basically be like, you'd be paying your cloud bill and you likely already have your if you know your company that's big enough to do this sort of thing you probably already have a cloud team paying a cloud bill but it doesn't actually matter that uh what that underlying infrastructure is because it's all abstracted by kubernetes i have a slight side question before we go back to discussing that since you get to see all this stuff and presumably you get to see what customers are doing, has there been any move to, like you mentioned, you could conceivably deploy all this to your own infrastructure, like your really own infrastructure behind your own firewall.

Has there been any movement away from, quote, the big cloud, to like, oh, I'm going to host my own stuff? Or is it just pretty much still all people trusting AWS? And Azure. Don't forget Azure. Excuse me. And Azure. Azure's huge. I mean, it's getting so big.

They're eating the market. Yeah, I think there is that move, but it's usually teams that are so big that they have their own internal cloud teams basically making something that looks like Goliath. And really that's the, when you get big enough, right? So like John, the founder used to be at Nest and they had this, but it was like a, you know, many,

many hundreds of people doing that sort of thing and building, building out a lot of the infrastructure. It was early on the same kind of thing. Like you could build some of this stuff, but it's not accessible to small teams and so again i the reason i'm excited was excited about it because it was like me as a hardware consultant i could deliver stuff that i'd just never been able to do before like i had projects where i would hire a firmware person hire a cloud person that sort of thing and even still that was kind of risky um just having you know three people doing that sort of thing.

Now it's like you actually have these capabilities without needing to have your first couple cloud people. Okay. So this sounds like software. I mean, it all sounds like a lot of software. Yep. Yes. And I know you, but you say electronics. Well, yeah.

Have we delivered you to the dark side yeah it is a lot of it is a lot of software um um another reason i was excited is because i was excited to get better at firmware like i i you know i was consulting for three years in the hardware space and i need i one thing i talk about is like it was like super choppy work right like you do hardware and like you know i go hard on a project and design a board and it would be like maybe a month of like all the way through and maybe I'd have a couple of people doing that. But then I wasn't doing the firmware. So it was like, all right, now I wait for the firmware person to go and tell me what I messed up and turn the crank in three months from now, that sort of thing. And it was just like super choppy work.

And I started wanting to do more of it. And then I kind of had an opportunity to do more of it at Goliath. So that's why that's another reason I was excited about the opportunity, you know, learn more about RTOSs, learn more about firmware, learn more about deployment, that sort of thing. So for RTOSs, Zephyr or FreeRTOS? Por que no los dos? I think it's important actually to make distinctions about ecosystem versus RTOS. That's one thing I've been doing a lot more lately. Because Zephyr is an RTOS, but it's really an ecosystem. It's both.

Whereas FreeRTOS is an RTOS first, and there's no ecosystem unless you put one around it. So ESP-IDF is an ecosystem built on top of FreeRTOS. And Modus Toolbox is the Infineon ecosystem built on top of FreeRTOS. That sort of thing. And Zephyr is both. And Zephyr is both, that's right. And we've talked about Zephyr. It takes its cues more from the Linux world than the traditional RTOS world.

Yes, but it's still an RTOS. It's still an RTOS. It's still small. It's not Linux, but many of its design choices and things. Have you tried Zephyr or no? Just a little bit. Yeah. Mostly I had some students when I was teaching at USIT and that made me go investigate more.

And it was, I mean, it actually seemed a little nicer than FreeRTOS. Definitely more integrated into the NRF. And so it was a, it depended on which chip you were using. Like MODIS, you definitely would use FreeRTOS. And last time I looked, ST seemed easier with FreeRTOS. I'm being so good right now. Why? Because you guys keep saying Modus.

And you just want to barf on it, don't you? Well, that's a bit extreme. Make gagging sounds? Maybe. They're trying it's better than the previous IDE that they had oh okay so

where were we? Zephyr we were talking about Zephyr you asked him why he became a firmware engineer no it was it seems like the RTOS you choose is in large part dependent on the chip you choose. Right. I think there's happy paths for each chip vendor.

And I'm doing some of the Silicon Partnership stuff for Goliath in early days as well. So I know a lot of the people in those companies I mentioned. And it's actually kind of cool like meeting them and stuff but it is um you know i'm i'd be curious to hear if you can relate to this but like once you're in a chip ecosystem once you decide on your micro i feel like that kind of defines so many other things about the project yes and it's distressing, especially, well, some micros are more about that than others. Like the Infineon stuff is very opinionated, I feel like.

And so is Nordic. Nordic is super opinionated. ST is probably the least opinionated, where people kind of can do what they want, even though, you know, Cube is what they recommend, but you can always do more. But I felt like doing projects,

we did a recent project with Infineon. Sorry, I keep trying to say Cypress. Same, same. They are merged for people that don't know like that. They bought Cypress. Yeah. If you wanted to do something different than they wanted, it was like you spent a great amount of time switching us to GCC.

And that was really, really hard compared to doing that for ST, I think. Would you say so? Oh, yeah. Yeah. All sorts of weird problems. So, yeah, definitely. And clients, forgive me for going on about this, but clients tend to like that, I think. It's like, oh, we got this chip and we got all this stuff and we don't have to think about it.

And so there's been a number of clients I've gone into. I was like, oh, we got this chip and we got all this stuff and we don't have to think about it. And so, there's been a number of clients I've gone into. I was like, well, we chose this and here's what it is. And we hate it, but also we love it. Yeah, I think one thing I think about is I look upon my past choices with a bit of guilt. And, like, so much of it is, like, I am just specsmanship and just, like, like looking at data sheets and like, oh, this one has this peripheral. And I really, you know, I sell myself. I'm like, oh, I really need six USARTs and I really need this, you know, this DMA thing that's like super fancy, whatever. And I had like no, I gave no thought to the firmware.

But also you have to program it in Fortran. Right, exactly. Exactly. And now I am that firmware person. I'm like, oh oh yeah, like that. And I think increasingly over time that companies are still going to do this, especially ones that are super siloed, right?

Where like there's a hardware team and there's a firmware team and they talk like during integration and that's about it. But that, you know, talk to your hardware folks, firmware folks and ask them to be nicer about this sort of thing. I wish I would have been a little bit more empathetic on this side of things.

Because like now that I, you know, I love Nordic, a lot of the stuff I listed, they all have ups and downs, but the Nordic stuff is what I'm using most often right now. And their software focus, they are becoming a software company that makes hardware which is interesting and i think i think the chip companies that win over the long term do that and and to their credit expressive is doing that too which is interesting like they have a very strong um development team so yeah expressive came out and they had a super cheap, interesting chip that was a Wi-Fi chip and people just adored it. And I never really got into it because it didn't seem professional level. It seemed like a great hobby thing, but didn't have all of the features I wanted for clients.

Firmware or hardware features? Or the polish, maybe? The firmware features, the software features. Yeah, okay. It was a little buggy. You had to play in their playground. The firmware update was meh. The Wi-Fi API was weird, too.

Wi-Fi API. At least initially, yeah. To set the user's Wi-Fi was totally dorky. Which is important. That's such a user-focused thing, though. If you don't get that right, that can be really impactful. And it's the first thing the user does. So if it looks bad, it just all looks bad.

And there wasn't an easy way to change it. But that's, I mean, that's, I don't even want to say a dozen years ago, because I don't know how long ago it was. But that was their first foray. And it made a pretty big splash, but I never got into it. And I know ESPF Espressif has changed a lot since then. Yeah, they keep investing quite a bit. So they have the ESP IDF team. I feel like there's, you know, most people come to Espressif in kind of the Arduino context

and increasingly the platform IO context, which is usually using Arduino core and making that switch over to, I remember Bart Dring, who's a Chicago friend. He, he was using ESP IDF and I just remember looking at him and be like, I have no idea what you're doing uh but it was you know he was getting really into like the as as that was getting built up and um and uh i i think they've they've invested a lot in that ecosystem and it's increasingly like like i said it's ecosystem level they're introducing like a kind of like marketplace for sensor drivers and things like that um so all of these things are making it a better experience, I feel like, for pro-level developers. But I think most people are still coming to it from Arduino,

so there's different expectations. And then they also have a Zephyr team. So they're also porting to Zephyr. They've continued to port to Zephyr as well. So kind of a dual solution there. It's an interesting device because I feel like it lives in its own architectural corner compared to the other ones, which have traditional... They're Cortex-Ms now. Oh.

They're RISC-V now. Oh. They're RISC-V. So yes. The C3 is a RISC-V, but it's the smaller part and lower power. They still have extensa on some of the newer stuff. Increasingly, the H2 is one of their new 802.15.4

chips, so that can run Thread and Zigbee for the Matter. They're getting big into Matter. That is, I believe, RISC-V. I think all of them have a low-power RISC-V part And I think all of them have like a kind of a low power RISC-V part as well. It's funny because I keep seeing ESP parts being taped onto low power parts as the Wi-Fi.

So it's like this thing with, you know, I don't know, 200 megahertz, 32-bit tons of power attached to like, you know, the Raspberry Pi 2040. Right. It's just so funny. Why are you doing this? Right. And they still use it and they just, they slap the ESP-AT firmware image on there.

And then they just talked it over AT command, which is like, I mean, that's another way, you know, to go back to the IoT thing, that's how a lot of stuff still operates, right? So if you want to get away from, like you want to use AWS IoT, but you want to still use your PIC-18 while you grab a USB 32, or you get like a quick tell modem. And there's images that you can flash onto these things that you again, you're sending at commands effectively

to it. But then there's added costs, there's added power, like so all of these architectural decisions that get made very, very early on can really impact your product performance when you really get to the optimization stage. I have an RV towing a Mini. It's worse when the Mini tows the RV. Yeah, it's true. Okay, so... I had a couple more questions. Oh, go ahead.

So something I was curious about, you talked about how the API is super simple and you just, you know, whatever, send to the cloud. But having done some projects that do IoT stuff, there's a lot of security and certificate stuff that needs to be managed. How do you make that easy if you did? So at the beginning, so like if you're booting Zeph,

or sorry, if you're booting a Goliath program right now, I'd say don't use certificates for Protosite. Oh, okay. So we have pre-shared key IDs and pre-shared keys, PSK ID, PSK, which is basically username password. Uh, and, uh, you, uh, so Zephyr has a bunch of shells in it, which is awesome. Um, so basically you can turn on, so the way that we normally do it, um, you know, I say we, I mean, Mike does it and then I go and type in stuff, um, after, after he's done it. Uh, but basically there's a settings service and a setting shell inside of

Zephyr, right? So you turn that on in the prj.conf files, which are basically like the kconfig, you know, basically how you turn things on and off. And you so you turn on the shell there. And then you say, in one of the other configuration files, hey, I'm going to set the Goliath credentials in over the Uart um and so then when your device boots up it uh you go onto the serial terminal you type in your psk psk id and uh then you are validated onto the glad cloud using embed tls it's actually a DTLS, um, encryption, not encryption, uh, authentication is DTLS connection, um, because it is a UDP instead of TCP. So it has to be DTLS. I've learned a lot of these things. I still don't quite get it, but I know, so you basically, you can't, you can't

connect to, um, uh, Goliath unencrypted, but that's kind of the simple way to do it um then like you said certificates are like this big hassle um but it's um pretty easy so we have a a new certificate uh generation feature as well so that you can generate and then you'd have to have your own private key store or what is it called a vault i don't know private key infrastructure um yeah basically like a vault where you would have your private keys stored somewhere secure and then you would have the public keys you would never um upload the private keys to goliath you would only upload the private no you put private private keys yeah right right uh put them on Twitter. Why not? Nobody goes there anymore. And so Goliath wouldn't have access to my data then? Goliath would have access to your data because it would be decrypted once it hits the co-op endpoint.

What if I didn't want it to be? Then you probably wouldn't send packets to goliath because we visualize the data that comes through goliath um so where would you want it to go in that case i mean like uh so let's go back to the um the size seismology example that you gave so you'd want to have like encrypted accelerometer data that doesn't get displayed golioliath. It just forwards it along. Yeah. I mean, I don't want you to be able to see my data because nobody will ever know what my seismologists, what my seismic sensors are actually doing. Where is it getting decrypted?

On my servers. Those servers that are supposed to pick up the data from Goliath. I think that, that yeah i don't have a good answer to that one because i've never seen anyone do that uh it kind of defeats the purpose too right so like you might have kind of the constrained pipe at that point but at that point you would just kind of set up your own co-op endpoint uh on your servers and then you would just talk directly to that right so you like you're assuming all these little sensors are cellular based right they uh have a connection to goliath over co-op right and they're uh they have a certificate or a psk psk id talking to the goliath servers uh and you're just getting uh

maybe maybe just want ota updates from the goliath server that that's fine. You could just do that. But then if you wanted to only send your data somewhere else because you didn't want Goliath to have access to it, then I'd say just bypass Goliath and go around and have your own co-op server. But then you're paying for infrastructure twice at that point. And you'd have to manage your own co-op endpoint, which is non-trivial. There's some good reasons to do that, but I have a couple of questions before I get to them. Sure. What about health checking? Do you check heartbeats? Do you plot sensors that are rebooting constantly?

Or is that all mem fault? So there is a heartbeat that gets checked on the console. Basically, you can see kind of like last connection time, that sort of thing. In terms of like trace data, like off the actual processor, that's where Memfault is just fantastic. So like that kind of thing,

you probably want to do Memfault for that, you know, the kind of lowest level. But if you're okay with just like logging messages, connection status, that sort of thing, that's kind of where we don't have any of that stuff today. So, sorry, we have logging and connection status. We don't have like the trace kind of stuff because Memfault's a really great solution for that sort of thing.

But if I had a thousand sensors and I needed to know which one was offline, I could probably do that. That is accessible from Goliath. Yeah. Yep. And it's accessible from, from the rest API too. Right. So you think about like, the other thing that I always think about is like, uh, so a common context in a lot of like IOT things, well, I can set up my own MQTT broker. I'm like, yes, yes, you can. And that is great. But like an MQTT broker kind of sitting out in the world, you say you just had like a Raspberry Pi in your home office running an MQTT broker, like, yes, you could use that as IoT kind of concentrator of sensor data and you can connect to it uh but there's a couple things missing from there one is uh a databasing aspect right so like there is actual like a caching timeout on mqtt

broker so you send a you say the temperature is 23 and you're sending that from afar to this broker and maybe something's listening and saying oh i, I just saw that the sensor, you know, sensor 55 just updated and says its temperature is 23. But now, sensor 55 now sends the temperature is went up by a degree, that last reading is gone, right? So unless you're doing some kind of databasing on that MQTT broker, then you that data doesn't it doesn't persist. And at least as far as I understand it, maybe I'm, again, maybe I'm wrong about that sort of thing, but I'm pretty sure that's, that's correct. So you need to basically be doing that databasing yourself. You need to like, listen for data. You know, often I hear people say like, oh, I write Python scripts and I put it into a cockroach DB or whatever. And you know, there's just a lot of, it's totally possible to do it, but like the, this kind of like writing scripts to do that sort of thing is it's very fragile in my perspective from my perspective uh especially if i'm writing them and uh and then the other thing is like then how do i access that data so you think about like

i can maybe my device out in the world is speaking over mqtt which is its own protocol or sorry network layer uh and no it's protocol it's a protocol uh and um so that sensor sensor 55 is out in the world and it's giving me temperatures over mqtt well if i want to be listening to that i also need to be speaking mqtt to listen to it and uh it's possible to do that you can have interfaces but at some point uh it's usually better to have something like a rest API. So like, basically you think about an app probably doesn't want to be, you know, an app on your phone probably doesn't want to be speaking MQTT to some broker somewhere. Instead, it wants to do a, a call to a, to an API, uh, that is kind of fixed in time and then they can go and fetch the data.

And so that's kind of the rest API layer. So again, things that are external to the service, whatever it is, if not Goliath, you have an app, you have a website, whatever, you want some kind of generalized interface to access that data and hopefully control the devices out in the field because you want it to be bi-directional. So back to the private data. Let's say it's someone's heartbeat or someone's location. Any personal information you don't want Goliath to have access to. You're just not handling that case right now? Yeah. All data sent through the Goliath SDK to the Goliath console is decrypted and displayed for your convenience.

So, yeah, that is. I mean, I think about companies like Fitbit, where that was a huge deal for them. Sure. It was encrypted end-to-end. Or ShotSpotter, where they absolutely would not let you have any data, but they would have loved the monitoring service. Which actually reminds me, can I burst data to Goliath? Like if I get a seismic event, can I then send you high quality data that covers 60 seconds or is it really just one piece of information at some constant frequency um that is determined only by the uh the the tier of service that you have so there's different tiers of service based on like

how often you can send data but there are burst limits as well so yeah it is possible but there's not like it's defined so basically if you think about it like uh you know the co-op endpoint if you're if you're validated to send data to that co-op endpoint um you can kind of send it as often as you want if you send way too much it's going to start rejecting calls because um we had an early user who who said yeah something's not working with this device and we went and looked at it and um they were writing that writing this really tight loop on an ESP 32. And it was like, they were saying like 20,000 data points a second. And we're like, yeah, that's too much. That's too much. You would never ever do that. And yeah, that's why it didn't work for them. You said co-op endpoint a bunch. And I should have asked what what exactly is that so co-op is constrained application protocol and uh basically it is a lower overhead

networking layer i think i'm saying this right uh but basically it's kind of an alternative to mqtt that has lower overhead it's's UDP-based, and it runs things like a lot of things. It's optimized for cellular. That's kind of where Goliath started was in the cellular space. And it also works on things like Thread. So I keep saying CoF endpoint, but it's really that part is invisible to the user. You won't have to be thinking about it.

Yeah, we use it in Fitbit in the later days. But yeah, it's just another networking protocol that tries to take some of the overhead out of what people were doing with it you know like you said to http yeah another uh thing that's built into the sdk as well as uh is encryption encryption sorry compression as well so there's uh c c-bore compression so again like you don't have like the uh you really you're just like using the api but then you can basically say if you want it to be json or if you want it to be uh c-, which is a serialization, um, compression, and that saves another like 20% overhead as well. So again, if like you're doing cellular stuff that really starts to add up on your cell bill and your power and all that other stuff. So do most of your clients, most of your users use cell modems or are they heading towards wifi-fi um i think uh because it's a lot of industrial users there's there's a lot of cellular but it's it's a it's a healthy mix

um so the four things that we support are cellular wi-fi thread and ethernet not zigbee zigbee doesn't have an IP address. And so, yeah, no. I mean, they have. So the only reason. Yeah. Sorry, I'm not going to argue with you about that. Zigbee does have addresses, but not IP addresses.

Why would I say anything different? Okay, go ahead. Yeah, the only reason that the thread works is because the border router in a thread context kind of makes that stuff invisible, and then all of the thread devices all have IPv6 addresses, but the border router has to have a NAT64 translation layer, or else it won't work, because you need to go from IPv6 to IPv4. Somebody could totally make a ZigBee router to do that.

Yeah, well, Thread's going to win. Everyone's implementing Thread. That's good. Yeah. Well, I have more questions about Goliath, but I also want to ask you questions about other things. So, let's see. Are you still doing that podcast you used to do?

Which one? Oh, well, that wasn't... The Empire? Yes was the empire yes i still do the empire still do the empire and i have not done the context i did the contextual electronics podcast for a while but that uh that fell away uh once my daughter was born and uh uh and uh you know maybe maybe we'll go back to it someday but yes um still have the amp hour and have um interesting guests on and uh yeah good times for folks who haven't heard the amp hour all three of them who listen to this show um tell me about it sure uh it is a mostly weekly podcast about uh designing electronics and people that are in the industry doing novel things um increasingly we talk about ai stuff which bugs me but um um we're not alone in that i hear uh i'm not gonna be talking about it much more yeah and um yeah you know trying to find

uh you know things that are interesting in the space because there's just so many different areas you know you think about the differentiation between someone talking about like a super super low overhead like risk 5 processor versus like uh someone who's doing you know power you know 250 kilovolt power line installation type thing it's just like they're just so disparate and yet it all still kind of falls into the same general area and so there is a lot of interesting stuff out there and we're just trying to still find some commonality amongst it. Mostly it's just whatever I like, to be honest. Yeah, I understand.

I was going to say, well, how does AI fit in there? But really it's whatever you like. Yeah. Hard not to talk about AI these days. Because the AI tells me to talk about it. Yeah. Are you still having shows with Dave very often very often yeah we do about every other week um yeah uh dave's busy i'm busy so sometimes we miss uh but yeah how are you feeling about

podcasting in general ah that is a good question um Um, I, I will start with, uh, I am incredibly grateful and amazed that people still listen. And for everybody who is listening, thank you. It's like, it really, it warms my heart. Um, it's, you know, we've never, I don't think I'm alone here. We've never been doing it for the money. I still think it's a great way to convince people to talk to me for an hour. Like, whereas they might not, if it was just a random, like LinkedIn ping. So from all those things is, is still great. Whether it is a medium that's going to keep up or like, you know, whether anyone seeks out new shows anymore i don't um i don't really have much visibility into that but i i think much like both of you i don't um i don't spend much time on the marketing aspect and i i've i'm a lot better off um psychologically if i'm not like worried about whether or not people are listening. I think it's just focusing on the people that do and, and enjoying that.

Yeah. I think, uh, I agree with everything you've said. That's pretty much how I feel. And, you know, I feel like we've built a little community around the show and people give us feedback and talk about things and that that's very gratifying. And, you know, there, there were probably times a few years ago when I was thinking, well, what, what can we build this into? And yeah, I'm not so much thinking about that. It's like, okay,

this is what it is. And it's very enjoyable. And, you know, we've backed off a little bit because it was getting challenging to keep up with the pace we were going. But yeah, but yeah,

I don't know what's happening. I mean, I feel like the podcasting as a medium has kind of plateaued a bit. There was the whole explosion around all these very famous people doing things and the whole Spotify thing. And that seems to have been a miss, I think in, you know,

in broad strokes. And so I, I think it'll, it'll probably continue as, as a successful medium, but it's not, I think it's past the hype stage. Yeah.

Yeah. I think that's right. Yeah. Yeah. I think that's right. Yeah. I have been listening with a bit of concern with both of you and your burnout and I'm hoping you're hoping you're doing a little better. No. Well, Chris just started his, his rest and I, I have, I am doing better

but it's easy for me to backslide still I was very burned out I was reaching levels of high burnout two or three weeks ago and then I had to push really hard to finish something for three weeks

solid and now I am totally fried so I am hoping that some rest. But you're taking a month or two. Yes, but I'm not good at rest. My, my brain is full of well,

now that I'm not working, look at all these things I should do. Look at how much music I can do. I have to start one project or finish one project or start a dozen, right? Yeah. Work on the ukulele. I have two bass for that guy.

I have two woodworking projects to do. I have a session Do bass for that guy. I have two woodworking projects to do. I have a session work to do for bass. I have another album to start writing. You know, all that stuff. But I'm like, but also, I could watch YouTube for six hours straight on the couch. Yeah. I've never watched on the couch before. That's a bad.

That's not good. I wouldn't do well there. I mean, I would melt on the couch at that point. That's exactly right. Yes. YouTube is mostly lunchtime for me. That's my main time. What do you watch on YouTube?

What do I watch? A lot of Colbert and late night shows, just clips from that. I like their monologues. That's kind of how I get my news as well, for better or worse. If I'm not laughing, i'm crying kind of thing uh and then i really like uh like wendover productions do you watch that one i don't think i've heard of that one oh it's so good it's like it's about like a lot of like logistics so like oh oh yeah yeah okay i think yeah a lot of like airlines right just weird stuff uh real engineering cgp gray did you watch the cgp you watch cgp gray i have watched cgp gray uh there is a video that just came out

about state flags obviously us-centric but it's like a critiquing of state flags like he's grading the the states in a class it is 18 minutes and it's it's fantastic uh highly highly highly recommended it i can send the link after this if you want to add that in um it was just so snarky and fun off to watch that one yeah yeah i i do like that guy um i don't i mean there are some i like for particular reasons like if i have a math problem that I don't understand, three brown, one blue is my go-to for that if they have an answer. Oh, uh-huh. But if I am trying to rot my brain, there's this woodworking guy, I think it's Black Table, Black something. Oh, uh-huh um and he does woodworking

and it's just so soothing yeah yeah like that and then like uh john the glyphonder got me onto a uh vintage tool repair you've watched those watch some of those i'm not sure the same channel there's a couple of them but yeah yeah there's a couple of them, but yeah. Yeah. There's a couple of them. Yeah. Yeah. Yeah. And then, you know, like the applied sciences of the world and the Mark Rovers and like all of the, you know, the science-y kind of stuff. That's great too. Yeah. Well, and I mean, we didn't mention Z Frank because that's just a given, right?

Yep. Yep. Jeff Geerling. Yeah. I'm just going through my channels now so yeah so many good ones uh really a lot of good stuff out there and i could just keep going i need to get my recommendations to stop just giving me music theory youtubes because they can be relaxing and fun because some of them go through like oh here's eight songs that use this chord progression and here's how it works and stuff and those are fun to watch well there was there was the one about there being an h oh the h note the h note yeah there's in fact

an h note it's not just really a b c d e f g it made me so mad it was so awesome it's so much fun adam neely i'm guessing you watch adam yeah i watched some adam neely uh and his he's got like a ecosystem other of other YouTubers. Oh, yeah. That's right. That, you know, he interacts with. Yeah. But yeah, for the garbage stuff, I tend to, like you, I watch clips of late night shows, especially old ones. Like stuff from the 80s is like, oh, it's David Byrne in 1982 on Letterman or something.

It's like, oh my God, this is so weird to watch. I could watch Grace Hopper on Letterman like on repeat. She was so funny. So good. Yes. That's great. Okay. So, but when you aren't podcasting and you aren't working and you aren't hanging out with the kid, you also, which I mean, how is there a time for this?

There's contextual electronics. do like i mentioned here like i'm doing more like partnership type stuff and you know just edging away from technical stuff always kind of like gets my hackles up a little bit because i'm i uh want to stay sharp and engaged and i'm just happiest when i'm doing technical stuff and so i've been kind of putting together like a curriculum for myself and then i started thinking like oh you know some of this would be good if i made this curriculum for the empire or for contextual electronics rather and it's like oh bad chris nope what is the curriculum you're looking at um so i have a couple of goals for myself around like uh cicd type stuff um so like sean himel has a really good series on digikey and i want to be putting my, uh, like some tool chains on there just to like get better at like, uh, having standardized firmware builds and things like that.

Um, but then also just firmware stuff, uh, hardware stuff, you know, like I have all these things that like, um, I want to be learning and getting better at. So, uh, hang on to that. It's not really curriculum. Yeah. It's not really curriculum. Yeah. It's not really curriculum. Hang on to that. Yeah.

Don't ever give up. I did, but don't ever give it up. No, it's, I don't know. I feel like you're, you're going to get back there.

I hope so. I hope so. I was just, it's, it's, yeah, it's, it's hard. Cause,

uh, I want, I want to be excited about technical things. Yeah. I was thinking about Alicia's class actually. And I, I want to ask about it technical things yeah i was thinking about alicia's class actually and i i want to ask about it actually the um so like when you prompt the students when you're you're not doing the class anymore right i'm not but there might be more and so i'm trying to figure out how much to say so go ahead i'll figure out okay oh sure so how about the past classes let's just talk about that in past classes when you prompted them to like what is what was the prompt of like things they were

supposed to build because i remember it wasn't like a specific hardware platform right right um it was something that had a cortex m that had at least one button at least two different peripherals like spy or pwm had a uart debug console i think that oh and had an interesting algorithmic component okay um and so that's good yeah it was a it's very open-ended but with you must at least do these things yes yeah okay yeah that's great i think you know it is always kind of like that balance of like how prescriptive are you so like compared to like so like jay's college jay carlson's college class that he taught was with the efm8 specifically right and they had like set curriculum and and like the nice thing about that is that it's like well there is kind of fewer ways to do things uh so maybe taing that kind of thing is easy

slightly easier yeah right exactly and then but but then the opposite is like you don't get as much variety and people are more constrained into like what they uh what the output could be so i was kind of thinking about like well what if contextual electronics was like prompting people like go and build this thing and then and then kind of uh showing how i did it um and uh seeing how people change stuff it's it's tough especially with like people coming in with different contexts of like well i've never built any hardware before it's like oh okay well that's gonna be different than like i built some hardware but i want to get better at the microcontroller piece of it it's like you know so like kind of just all the balancing there yeah i mean it would have been easier to grade something that was more consistent i think the ecosystems were the thing

that was the hardest for the tas because i didn't limit them to any particular ecosystem and yeah some people you say no arduino yeah i said no Arduino. I said it has to be something you can step through. Yeah, that's a good... I can step through Arduino if you just go in and you take the clock off the chip. No, no, they have it now. Yeah, I know. Arduino has that built in. Yeah.

And so not having the consistent overall thing made it more difficult. But the variety of what the students came up with and how they got to talk i mean i didn't have to tell them very much about the different processors because they told each other about the different processors based on what they were using and that part worked out really well and we also did um micro madness where everybody had... It's like a Final Four kind of thing? Exactly. And everybody had a processor and a dev board that they were assigned and they were just randomly assigned and told to look up the data sheets

because we were talking about how to read data sheets and how to compare processors. And then I asked them totally arbitrary questions. I mean, sure, there was price and size and power, but then there was also how many vowels are in it. How many vowels are in the name?

Yeah, yeah. It matters. Sometimes they're arbitrary. And yeah, we did this basketball-style elimination. It was a lot of fun. And when you lost, you went on the team of the person who

you lost to. And so then everybody got a little bit more information as they went on about the different processors and the comparison. And they really enjoyed that exercise. And it was a lot of fun for me too, because I mean, the hardest part for me was every year trying to come up with a different list or at least augment the list so I didn't have the same list as, you know, time to time. Yeah. And so things like that, I wish I could do more. The group teaches itself activities. But those are, of course, the hardest ones to do.

And you had to teach a live class to kind of have that group thing. Yes. Right, exactly. Yeah, so like synchronous versus asynchronous type of stuff, too. That's another challenge. I feel like the synchronous is very motivating because there's kind of that social pressure and just the wanting to interact and whatever. But then the downside, of course, is time zones. And yeah, just making sure people are there every week, that sort of thing.

And getting yourself there every week. Yeah. I mean, it's a lot of work. So I guess I will mention that I'm not doing another class anytime soon. But we are probably going to have a cohort that does it without me in that they can watch the video it isn't won't be as expensive but they can watch the videos and the exercises and get the quizzes um and work through things themselves i think the

part we're still working on is how much will they talk to each other? Will there be a Discord? Will there be a forum? Will there be a TA? How much of it will be solo? So I'm working on that. I'm hoping it comes out in May. But as it's April and I haven't been working very hard on it, we'll see. It is April, yeah. It's a week into April, Chris. Sorry.

Sorry. Week later, April fools. Oh, no. Yeah, and I think that's kind of a similar, you know, kind of where contextual electronics is, too. Like, all of the recorded content's on there. And people can, there can there's like forum support but that's it so um yeah nothing nothing new recently and and that's hard because i i want to keep adding but it's also there's only so many hours in the day and i found teaching live to just be exhausting yeah even very i mean it was supposed it was kind of energizing because the people were neat and it was fun, but I

would get off of class. I would then just fall apart for the day. They were long classes too, right? It wasn't an hour. It was an hour to an hour and a half. Oh, okay. And usually the TAs would take over. Well, I mean, it was an hour to two hours and usually the TAs would take over after

an hour and we'd break into groups and they'd discuss time. But sometimes it was a couple hours. Yes. And I mean, I really liked the people. I liked meeting them. I liked talking to them, but I'm just not a people person. Yeah. Yeah, that's like a classic introvert. Like, you can talk to people, but then you gotta recharge. And I can can talk to one person at a time and it doesn't bug me too much, but talking to people sometimes even to, you know, well,

Chris doesn't care. He's inside the bubble. Yeah. I see. I see. Okay, well, thank you for having me to the bubble.

This is a fine bubble you have here. It's comfy. Yeah. Yeah. Um, well, I, I, is there anything else you want to say about now i can't even say it right goliath um here's here's another interesting thing about uh pronouncing names uh when i went to so i was just in this germ germany for the better world conference and i'm not sure what it is about the spelling but um zephyr many people in europe they say like sapphire sapphire i'm not sure if it's like just how the vowels are pronounced but

like the first couple times that it happened at embedded world last year i kind of did i'm like what what are you saying what is sapphire sapphire no zephyr zephyr so yeah yeah that's i think that's the right pronunciation for the ph yr fire yeah huh like fire festival f y r e um but i mean we say zephyr at least we don't say zed for well well that doesn't make any sense uh but goliath um you try it out uh it's um mike and i make a lot of videos and uh blog posts and uh not specific to goliath but zephyr just got a new devrel person as well benjamin kabe and um he has a great newsletter um that I've been really enjoying where he does like weekly updates

about what's new in the project. So recommend that as well. He might make a good guest on the show. And yeah, making lots of content. So check it out, blog.goliath.io. And Zephyr and Goliath both have very good Discord servers. Yes, yes.

Not a Discord person myself. both have very good Discord servers. Yes. Yes. Not a Discord person myself. I don't like Discord servers. We do have one, but I don't know. Not for me. I don't like that. That's like synchronous again, right? It's just like I like asynchronous. So there's a forum too.

What? You mean you don't just drop a question on there and leave until the next day? That's usually what I do. I use Slack. I'm the person who has to answer a lot of those questions. Well, yeah, there is that.

Yeah. Yeah. Do you have any thoughts you'd like to leave us with? I am, uh, interested in, in learning more stuff about firmware. So,

um, please point me to resources either, um either in the comments of this episode or send me a note on Mastodon or email or anything like that. I love learning new things. So hit me up. Our guest has been Chris Gamble, developer relations at Goliath and host of the amp hour podcast. He is not our sworn enemy any longer.

We like him very much. Thanks, Chris. I'm keeping the handle, though. I'm going to keep calling myself that. Thanks for having me. Always great to talk to you. Thank you to Christopher for producing and co-hosting. Thank you for listening.

And if you'd like to contact us, show at embedded.fm or hit the contact link on Embedded FM. And we have a newsletter. And we have a newsletter and we have a Patreon. If you want to hear more of us at any time, there's more to be had. So now a quote to leave you with from Dr. Seuss. Sometimes the questions are complicated and the answers are simple.