Embedded - 528: Goldfish Chunks

Episode Date: June 25, 2026

Tyler Hoffman returns to the show to discuss diagnostics and observability data in embedded systems. We catch up on his life after startup acquisition, explore the hows and whys of keeping product data separate from operational data, and consider the realities of fleet management at scale.  Tyler is the co-founder of Memfault. Memfault was acquired by Nordic Semiconductor about a year ago. While Nordic has nRF Cloud as a smaller scale solution for Nordic devices (~100 devices), Memfault will continue to maintain support for non-Nordic platforms as well. During the discussion, Tyler advocates for a "device-in-control" philosophy, emphasizing that edge devices should retain the intelligence to manage their own firmware updates and telemetry. We also discuss the practical constraints of remote fleet debugging, outlining why tools built for high-bandwidth web infrastructure will quickly bankrupt an IoT company, and identifying exactly when a project is too low-bandwidth, or too small, to justify an external observability platform. Christopher shares his recent experiences with Memfault which leads to a discussion of chunks, flash memory buffers and MDS. The Memfault Diagnostic Service (MDS) is a standardized way for BLE devices to send the chunk payloads to a gateway device (mobile phone) which can then forward the data to the Memfault cloud. If you want a deep dive into the reasoning around starting Memfault, Tyler was on Embedded.fm episodes 390: Irresponsible At the Time and 395: I Can No Longer Play Ping Pong. Reaching back into the archives, Elecia, Tyler, and Phillip Johnston were on the Memfault Coredump Sessions podcast, a special crosspost with Embedded.fm, episode 451: From Concept to Launch  You can also find technical deep dives on Memfault's Interrupt blog. "What we do makes a difference and you have to decide what kind of difference you want to make."  – Dr. Jane Goodall, Reason for Hope: A Spiritual Journey. Transcript

Hello, and welcome to Embedded. I am Elysio White, here with Christopher White. Our guest this week is Tyler Hoffman, who has graciously returned to answer our many, many questions. About Memfault and other things. Hi, Tyler, welcome. I'm both of you. Yeah, I'm very excited. This is, I guess, the third recording we've done together.

Is it? Okay. We've been doing this a long time. Yeah, we did a part one and part two last time. Okay. Could you tell us about yourself as if we met at an embedded systems conference? For sure, the intro that I always give, basically, no matter who I meet. Yeah, so I'm Tyler for the people who know the space, co-founder of Memfault, if you've ever heard of it, if you haven't, probably heard of Interrupt, which is one of our blogs that we write a bunch of stuff to.

But I got my start at Pebble, a smartwatch company back in 2014. They ran a Cortex M3, and I just wrote a bunch of firmware there and found my way getting tired of rebuilding a bunch of developer tools made for firmer engineers that solved a lot of issues that they had. And Memfault was kind of the final attempt at building a set of tools that I didn't want to keep rebuilding at other companies. And that got acquired by Nordic, and now I'm working. a Nordic semiconductor. It was acquired like a year ago.

Have things settled down? If things changed? I think we're pretty much in a, well, it can always be better, I think, with acquisitions, especially into a large company. It's pretty much hit a steady state where every product, we can go down a rabbit hole here. Most products they release, and that we release, we do it in lockstep, which I think is a big win because it means anything they release has Memfault baked in or has it kind of a turnkey

solution and then anything we do on our end has the Nordic chips kind of on the top of mind or at least built in as well. We are going to do lightning round. You know the world's to this, so let's just get started. Great. I don't know why I'm asking this. Favorite backyard theater play? It's hilarious

The Princess Bride It was the one my wife wrote But this is my wife's hobby Okay cool Favorite dinosaur T-rex Because they're ridiculous They're huge

Go ahead Favorite bug either software or entomological Oh man Pass I don't have one Do T-rexes have feathers Not to my knowledge do you like to complete one project or start a dozen?

I love to complete a project. When someone finds out what you do, like on an airplane, what questions do they always ask you? It's always something like, can you fix my son's computer, or that's what my son does, or can you fix my phone just immediately. If you could teach a college course, what would you want to teach? I thought about this one, because I cheated.

I, this was one of the ones you gave me. I would like to teach a course on cooking or sustaining yourself through, you know, non-processed foods. Nice. Do giraffes make sense? No. No idea how they work. I don't know if we've asked you this one in the previous shows. Probably we have, but we're going to ask it again, and then we'll refer back to the other shows to see if your answer has changed.

Favorite fictional robot? Wally, and I think that one still stands. All right. Yeah, that sounds right. From previous ones. Have you ever looked a 9-volt battery? No, I have not. What?

I also, like, well, I was going to say my daughter definitely has because she puts everything in her mouth. But I don't think we really interface with batteries quite yet. Well, I mean, 9 volts are special. They're spicy. That's how you test them. Yeah. Did you see they came out with a coding for coin-sill battery? Yeah, yeah, yeah.

It's special. Bad tasting. Yeah, batteries taste. bad now. So I don't know if they do this 9 volts, so we might have to test that, see if it's shocks you and taste bad. It's just a little shock.

I mean, a fully charged one is pretty good. Depends on if you've had a lot of, you know, salty crackers. And how old it is, you know, the first ones are the best. Favorite acronym. In case you missed it, because I think it took me so long. I see why, you know, I can't even

it, but I just didn't know it for the longest time. But now I use it. I thought, if you know you know as an acronym, now in all the Ks and the Ys, I was just toast for. I've never seen that one in real life, so I'm looking forward to it the day that I come across to it. Do you have a tip everyone should know? Many. I think one of the ones that I have been realizing recently is you can blend any sort of green into pasto. and put it in pasta. Hmm. Speaking of food.

So, like, tail and linguine. Let's do it. Yeah. I mean, you should probably put some garlic in it, but yes, I do it all the time. Salon. And, like, I thought I thought I got you. Any sort of green, huh? All right.

So when I buy the big box of Costco greens, and I'm like, there's no way I'm going to use all of this before it goes bad. It goes into pesto, and we consume it as a family meal in pesto form. Yeah. kids and all. Has, is cooking a new passion for you? Or, so we talked in like 2021. Four years ago. Four or five years.

Yeah. 20, 21 was not five years ago. Come on. I just keep saying that. Is it a new passion? I would say it, for sure, growing. I also see it as a necessity. And I have two kids now that are both eating.

solid, so one's almost a year and one's two years, and it's born out of, you know, we all need to eat, and I don't want to eat out every single day or buy frozen meals. So I am the, I buy all the stuff, I procure it all, I cook it all, I know exactly the inventory and everything like that. And so it's just become my role in the family. That does bring up the other question of, is there anything new, going on in the last five years that's changed? I mean, you're still at Memphal. What could possibly have changed?

Two kids. So many things. Basically everything. In a lot of ways. I mean, yeah, I think the standard family items have all kind of happened. I probably, yeah, I got married, had two kids. We moved probably into our lifelong home in San Francisco, which I'm very excited about. The company sold.

and I think those are the big ones. Those are the big ones, yes. Seems like a big list. It all happened, yeah. It all happened like last year, basically. A lot of them happened last year. But yeah. It was a surprise to me that Memfault sold to Nordic.

But kind of cool, for the reasons you said, that Nordic is one of the chip vendors that you would really want to use this on because they're internet enabled. But I want to say, was it a surprise to you? I mean, I hope not in the end. No, it was not. It's also the, I mean, I guess I'm biased.

It's one of the coolest semiconductor companies. They're so focused on software. They contribute to Zephyr. I mean, they've literally baked it into their SDK, and they're the largest updream contributors. And they have such a sweet, of products that, I mean, are working better and better together every year and every release that they produce.

But they, I guess, twofold. One realized one of the blockers to being successful with their products was the end users actually successfully writing software and making a bug-free and not having to spend all of Nordic's F-A-E time to help the customers kind of write bug-free code. there was always like back and forth. Is it your bug? Is it our bug? Who knows?

And then secondarily, I think the semiconductor companies are realizing that it's far better to make annual revenue on licenses or something you can sell their customer rather than selling them a chip and then hoping they come back in a few years and buy more chips from you. So that was another reason. I think Nordic was excited by the SaaS business. model. It evens out the revenue. Yeah. Otherwise, it's very lumpy. Yeah. And then for us, it wasn't necessarily a surprise. And I think we got this advice over the many years of having the company is your acquire or potential acquires will always be the ones that you partner with and have been

working with for a while. And so I think we had been working with Nordic for five or so years. We had an integration and the integration kept getting better and better. So like Memfault was kind of an automatic K-config flag in their SDK that you could enable. And we had a bunch of built-in metrics that you could capture LTE metrics and performance and Bluetooth as well. And they wanted us to spend even more time on the integration. They wanted obviously the revenue. And from our side, like, we're like, finally, great.

Like, we finally have access to your engineers and the teams that, we have been trying to get a hold of and to help improve our integration of the Nordic chip sets because it can improve the way we sell and how easy it is to sell into Nordic chips. Has the acquisition changed your job significantly? Yes. I think probably the primary way that it has changed my job is we are now trying to merge. So Nordic had an existing cloud product called NRF cloud. that it was very much on the kind of leading edge,

which makes sense. Like you buy a chip or you're thinking about buying a chip, and you need a few things. You need security, provisioning a way to contact the device just in general, and you need OTA. And that's what NRF Cloud was. I think Memfault actually took more of a ladder role. It was when you realize you have problems

or when you're scaling, you're going to run into issues around debugging, gathering, gathering analytics, doing large scale analysis on all of the devices in the fleet. And so the goal now is to merge both NRF Cloud and Memfault into one product that you may sell at different times at different companies, but it feels the same. Of course, in the cloud back in, it will be, you know, 10 more years until it feels like the same sort of software project, but that's my goal now, is helping and working on that transition and slash merging. That's why I have to log into NRF Cloud. Yes. I hope many of those experiences will continue to improve over the coming months, and they will. Like, that's the goal.

But yes, there's still a little bit of login to NRF Cloud, you know, redirect to Memfault, log into Memfault, redirect to NRF Cloud. Yeah. Both Christopher and I are putting Memfault in client devices. Have put. I'm finished. It was an average. I'm in Will put. Okay.

What do we need to know? And what do we need to tell our clients about what Memfault is? That is such a huge question. Yeah, because it's sunny outside. I wanted to go hang out with the dog outside while you just record this part. I'll be back in about 10 minutes. Yeah, exactly. What do you need to know?

I'll start with that. Try and take the easy, well, it's easiest on, I guess, chip sets that have more of an easy mode. Zephyr, ESP 32, Nordic chipsets, they're quite easy to integrate on if you are just integrating into a bare cortex M sort of firmware. There's probably more that you'll need to do. to help out the Memfold SDK integration because it is firmware after all. You have to modify linker scripts manually, and Windows is hard.

But all of it's in the documentation, and honestly, like, I'm sure you've talked about it a lot in the other podcast, but the AI tools are really good. We now have a little AI helper chat bot on our docs page, which I think actually helps quite a bit as well. It continues to improve as well with the new models.

And then what do your customers need to know? What do our clients need to know? Clients. Not the customers for the devices, but the device vendors. The people we write software for. And I guess in what sense, is this like how they should use Memfault or why they need to use Memfault or? I think a little bit of why and a little bit of what does it do for? Does it do?

We know this, but there is sometimes some confusing. about, oh, does it do OTA for us? Oh, is it going to consume our customer data or store our customer data for us? Or will this find, I keep getting questions like, any time is it bug, will Memfault help with this? And sometimes yes, but many times, no, because it's like, that's not the kind of bug work. Totally. And it's not a magic wand. As much as I wish for the thing to be magical, I continue to believe.

So, yeah, I get tons of questions like. Oh, we're having Bluetooth performance issues where sometimes it disconnects from the app and we can't tell why and blah, blah, blah, blah. And I don't think that's really going to help. I think we need to look at our detailed logs and catch it. This is more for fleet performance and things. But it's a difficult question to answer as an engineer sometimes because it is a fine line. It will find specific bugs if there crashes and asserts of things too. And to your point about Bluetooth performance, we're trying to do both, I would say.

Like, as we've gotten pretty good at the LTE side of it, because that was honestly the first project we were working on with Nordic for the last two or three years, is they were trying to optimize and fix a lot of issues with the NRF-91 series. So now you can actually capture a lot of the metrics from the modem, which tower is it connected to, which country is it in, all these sorts of stats from the actual cell modem, which is great. Most people don't actually know how to capture those. And then Nordic has a feature where you can actually capture an LTE modem trace. And then you can download that and throw that into Wireshark and actually see exactly what's happening. And that's really how Nordic debugs. They're like, you know, if you have any issues, please capture a modem trace.

Most people don't know how to do that in general. And then definitely don't have remote diagnostic or remote capabilities to grab that. But Memfault has that baked in. Like you can just say grab a modem trace, Memfolt, pull it over. and then you can download it from the Memphold interface. So that's very cool. And hopefully we do the same for Bluetooth soon. But what do your in-clients need to know?

It's not a magic wand. I think it's the effort that you put into it is what you'll get out of it. And I think this is something that I've always believed in is you can write software in a way that is trying to surface as many bugs as early as possible. You can also write software in a way that you're trying to hide or, you know, sink as many bugs as possible into the system and not raise them. But if you, the more you raise issues, the more you not necessarily log. I mean, I like asserts if it's still in development mode or it's, you know, on a small subset of devices. The more you assert, the more you'll catch the momentum fault because then you'll get the exact stack trace and memory stack of what's actually going on. It's a good point. I don't think I don't think I have articulated it as well as you have, which is there are times when folks, when I have written software to expose as many bugs as possible, as early as possible. And there have been a few times where I want to hide any remaining bugs. I want the failures to fail very quietly and just sort themselves out.

and reboot or whatever. I don't need to know about them. I just need them to go away. Because also they may not be essential, right? Like it may be like if it's every minute you're trying to capture a temperature or sensor, you'll just like, we'll capture it the next minute. It's totally fine. I was actually thinking about times when I had big demos and I didn't want to see the bugs in the next week.

I just need things to work as well as they're going to work. and we will pass, we'll go back to bugfighting mode after. I did, so on the, what is it that Memfault does? I looked at the Memfault.com website, and I've done it before, and I've sent other people there before. And when I sent a colleague there for my current project, he came back and he was just like, I don't know what that does, but we need to do for more. update, we need to do security, we need to do a bit of monitoring on the devices, and we need to handle hard faults. And I don't think they do any of those things.

What? So your website's a little perplexing. Like maybe it's written in CEO speak. Well, now I have to go look. And I think it's honestly, I think it's very, honestly, I think it's very. difficult to write to everyone as well. Like, if you do need to write to CEOs and you need to write to firmer engineers on the same website, I would say that's just challenging in general. We do not A, B, C, D, E, test depending on your, you know, persona logging in based on

tracking of sorts. We don't actually do that for our marketing website. The ultimate goal is to get somebody to write in and say, hi. I'd love to have a call. And so ultimately it doesn't need to explain it as well enough to turn that person away to be like, I actually don't need all those five things or some of these five things or something like that. And then they'll make the decision themselves. We love for them to just say hi.

Are you sure they spelled it correctly? Because this website says a lot. Resolve quality escapes to improve customer satisfaction. Well, that's just the first line. You've got to scroll down. It also sounds like that person may not have read all that much on the website. And I totally agree that the first two lines may not exactly spell it out. But I think some of the subpages actually are very explicit now.

Oh, I think so too. And you have a section that's for embedded engineers. And they didn't really get to that part. I'm pretty sure what they wanted to do is say, no, that doesn't work. We have to build our own. I just went back to. They were just trying to prove it for themselves. Talk over what we're doing here.

Never build it again. What we actually came up with was we can do it ourselves, but let's use the same interface and framework that Memfault uses in case we ever decide to change our minds. Sure. And then, you know, like, three days later, we should just use Memfault with the response. Like, yes. Yes. Thank you. It is, we had to have talked about this before.

It's just never worth anybody's time, especially early on, to build out a whole suite of tools. What product are you making? Exactly. Like what does the thing actually have to do? Oh, it has to like, you know, the Yodo. It has to play music for kids. Great. Let's build debugging tools. I don't think that should ever really happen. But it also depends on scale. Just how many devices they're trying to ship. If they only want to ship 10, then like don't use Memfault. That's actually a good question. I mean, have we really defined? Okay, so let me define what Memfault. does. And then you can tell me how wrong I am, which is great. Amazing. Love this exercise. With Memphal, if you have like a device that has a connection to the internet. Even by proxy.

BLE, a smartphone or a Raspberry Pi with Wi-Fi. Or somebody watching the serial trace and cut and pasting and then pasting them to the website. That works too. It does work. All right. That wasn't what I planned to do. It's the worst way, but it is possible. It's the way we prototype.

Yes. It's the first gate. Step one. And, okay, you have something that connects to the internet. Yep. And you want to build one of them. That's easy. You just do that yourself.

It becomes your pet. It lives in your house. You check on it whenever you need to check on it. Now, your friends and family want the same device. And so you build it for them and you realize you are constantly calling Aunt Agatha and telling her that her goldfish is about to die unless his bowl is cleaned. Whatever thing that this device does, you have to monitor it constantly. But hopefully you don't also have to monitor that Uncle Freddy's bowl hasn't checked in in three weeks. So you don't even know if his fish is dead.

clearly we're building the internet enabled goldfish again goldfish bowl not goldfish um and so memfault separates the idea of the data that is related to a product whether or not the fish is alive to the data that is related to any internet enabled product which is does it check in does it have a heartbeat has it crashed is its battery doing well how do you do firmware up update, how do you connect the user to the device provisioning? And it takes those common IOT device problems and puts them in their own little box called Memfault. And then this data that is important to you about the goldfish status and the water health. That goes to your own servers.

Memfault doesn't care about your data. Memfault wants to talk about the operation of the device, not what the device is doing. And the way you accomplish that is you put some pieces of hooks in your code and those send up special messages to Memfault that provide all the data and Memfault will host your firmware update bytes and through secure magic means it will send the data down and you will have to firm or update locally, which is as it should be, of course. Okay. What did I get wrong?

Wow. I mean, pretty good, actually. And I think the one thing that I really enjoyed, and maybe it's a feature gap or not, but I do think I really appreciated your distinction between the data mempholt collects and the data that your product and backend and the thing, the data pipeline that your company owns should collect, meaning we do believe there should be a separation between the diagnostics and observability, data and systems, and separately the thing that makes your product work. Because ultimately, like, you may cancel Memfault or you may realize you're not going to make

any more money on this device, but you want to keep these devices operational for the most minimal costs. You may not want to have active monitoring on a bunch of devices anymore, and so you may want to cancel the subscription. you don't want the main data pipe to kind of go away either. But yeah, that's the goal, is build reusable pieces and components that people can plug into their hardware or firmware, I should say. And that is, we're talking about probably MCU-specific stuff,

since that's what this podcast is about. But we also build things for Linux and Android. And having all that data plumb into one single system, regardless of what hardware you're building, is pretty cool, I would say. And we, I think the other just high-level sort of aha moment, was we don't want you to have to hire cloud and data engineers just to get the observability data into a system.

Like, you may have to have it back an engineer to build your actual product, but hopefully they're not working on helping the firmware team debug. That should just be a firmware-only solution and Memfault can help firmer engineers just do that alone. And that becomes really important if your internet-enabled goldfish bowl goes from 10 friends and family to 1,000 or 10,000, where you could spend all of your time just trying to figure out why one of them crashed,

when what you really need to do is figure out this other crash that half of them experience. Correct. But unless you can see which is which, you can only respond to the one that cries the most. And I think our, the customers that I think are great matches for Memfault, they have already started experiencing or they are already assume that, like, there will always be devices in the field that are failing. Like, if you have 10,000, one per day will have some sort of hardware failure, whether that is a permanent one or a temporary one. or a chip out of spec. And they're just wanting to track these things down. And maybe, you know, they have a queue of bugs and they'll write a workaround.

Or they'll ship that customer a new device. They'll email them proactively and be like, hey, you have a really bad battery. We'll go ahead and replace your device. No, no. I did that. I did that. I emailed early days in Fitbit before Memfall. I said, your device seems to have had.

some sort of catastrophic error that I don't understand at 3 p.m. on Sunday. And the person who I emailed was an internal beta user was also really creeped out when they said, I don't know how you're finding these things out, but that was when I dried my Fitbit. In the closed dryer? It took a swim, and then it went for dry, and that was a laundry day, and please don't monitor my activities. I don't know why you need to know that I'm laundering. We just want to improve your experience. I just want to improve this. Yeah.

Okay. So what do you call the data that goes to Memphal? You said diagnostics and observability. Chris said telemetry earlier. That's because I'm old. Also a good word, too. I think people use that all the time. I would have said operations data.

It also works. None of these things are wrong, I would say. Well, now I feel like that's a challenge. We should come up with wrong names. The other Chunks is the official name, so The other core principle that we believe in in memphal

and it gets maybe just a conversation point is we we generally believe the device is always in control rather than the cloud is in control and so I think for devices where the brain is on the device they decide when to firmware update

the devices decide when they're in a bad state and you put as much logic into the device I think that is a great match from MFault. If the device is truly dumb, I think this is like AWS greengrass, where all of the logic is in the cloud and the device just basically gets told when to make operations, that's not a great fit because, you know, there's no bugs happening on the device itself. The bugs are probably all on the cloud or, yeah, anyways.

So, like, with firmer update, the device. shouldn't be commanded, hey, you need to go for a more update. The device should instead be smart enough to say, you know, it's about midnight my time. I haven't seen any action lately. I'm plugged in. Let me go see if there's firmware. And so it's responsible for checking. I think it is ultimately responsible, yes. Like, I think there are optimizations where the cloud may, you know, I think a few of our products are deployed into buildings. Like, it's a lock on every unit in a building or access control. And so they want to update all the devices on Sunday at 3 in the morning.

Which building is this? Exactly, right? Go break into it. There is obviously some cloud control from that. But, yes, the device is ultimately going to know, like, is it being used? Like you said, Alicia, is it plugged in? is it connected to Wi-Fi? What is its battery life state?

Does it have enough charge to a firmer update? All these things, I think the device is in more control and has more knowledge to be able to make the right decisions. We're talking mostly about Wi-Fi devices, but Bluetooth-only devices, there's a little bit more work to do because they obviously can't go hit a Web Rest API endpoint and say, is my firmware. They have to talk to the phone and the phone has to do some of this for it. And as well, the memphal chunks, which I'm going to keep saying because I love it, love it.

Great name. Have to get from the device to the proxy that does have connection to the internet. So there's more infrastructure there. And that's usually more work for the firmware developer, not. That's the stuff that's kind of like, you got to figure this out because we don't have internet access, right? Generally, yeah. So the thing that, that we call it, or the connectivity path that we call it as eventual connectivity is what these devices generally need to have because the devices, all of the memphal data is almost exported. The only thing that's kind of like input into a device is the OTA payload in a little bit of

configuration. And that is all generally commanded by a Wi-Fi device or honestly BLE via like Nordic has a couple of standard protocols, whether that we call it Menfault Diagnostic Service, but we call it MDS, but then there are a few other protocols that the NRF apps and NRF Util and their BLE libraries will use, which now speak a little bit more Memfault-like data

and can ask for OTAs. But ultimately the Gateway device is going to be responsible for kind of queering for which OTA, for which device, given which software version, and then pull that down and then kind of pass that.

device in whatever way it transferred as data. That made sense. Yeah. Yeah, and we ended up, I ended up using the Memfault RAM buffer and then flushing that to Flash. So I have a bigger buffer on Flash. And any time the phone connects, it says, is there any Memfault data for me? And the device says, yeah, here, here's, you know, 10K of stuff. Does the app talk to the company servers alone or to the Memfault servers as well?

I mean, like, my goldfish chunks, do they, sorry, I just want that to be the title. Do the goldfish, do the Memfault goldfish chunks go to my company's server and then to Memfault, or does the app send them straight to Memfault? You can do either. The easiest is directly to Memfault at any point in time in the process. So whether that is a Wi-Fi device or an L-V-E device, it can go directly to Memfault. That's, I would say, the easiest. It requires less engineers within the company to kind of allocate to us.

However, and I think it's a good thing ultimately. Many IoT devices are kind of single secure channel to the company's backend. And so it has to go to their back end. It's the only way the device can communicate to anything on the internet. And then usually it's AWS, like 90% of the time. And then so for every single MQTT packet or co-op packet, that the AWS receives, it'll just trigger a little function or a post request to the Memphal Cloud saying, I got a new chunk, here's a chunk.

And then we will reassemble all that data on our backend. It's more expensive because AWS makes all of its money on data transfer. So that's why we honestly advocate for a lot of companies to send it to us directly. And that's why many companies ultimately switch and bypass AWS. But they can do it. But if your plan is to replace Memfault when you have enough users that makes sense, then you plan for the, I'm sending all my data to my home and forwarding what I need for now. It's one way to think about it.

Yeah. The data that Memfault's SDK exports is not, I guess, generic enough, or rather it's not a standard protocol. We don't use open telemetry, for example. I think it's another thing people are kind of coming across. It's not really built yet or small enough for embedded devices. So we built a lot of protocols and data transports ourselves. But yes, that is a thing people are always thinking about. It's like, how do I, and it's not something we're preventing it.

How do I eventually plan for the time or the emergency in which I would have to redirect data from Memfault elsewhere? Most of the time for us, it's like create a domain name that is essentially an alias for a Memfault domain. So if you're going to send data to you know, device data.govol's.com, have that be an alias for a memphalt domain, which is the ingress URL, so that you can ultimately redirect it somewhere else.

Oh, okay. I was thinking I would have to do a firmware update, or an app update at least. I mean, multiple things, but yes, firmware update is the way the firmware engineers think about it. If you're a cloud engineer, you think about everything in DNS routes and domains. And the data.

data is really compact. Like I was, when I was testing, I was like, I'll leave this device. It's like, oh, you've accumulated 800 bytes in the last eight hours. Like, okay. It's super cool. That's pretty amazing. The fact that a quorum can be a few kilobytes of data to get the stack traces and the metric payloads are like, it's, I mean, as little as five bytes per metric value because it's C-Bore. It's pretty cool.

So the analytic tracking and the error tracking. is data that comes to you. Firmware update, which everyone is scared of firmware update. And I find that so hilarious because it's so much easier than it was. You should have been scared 15 years ago. Oh my God. It's true. Yes.

You should have been terrified 15 years ago. Why are they scared? What makes them scared? The things that they should have been scared all along, the security, bricking devices. I think the breaking devices is the big one. And it's still, everybody still is afraid of that. And it's like, no, we don't have to do that anymore. We have plenty of Flash.

We'll just put a lifeboat on there. Security reading it out. All that stuff is the next thing out. Somebody's going to get our precious software. Someone's going to botnet our devices as though it was worth it. Do you believe that it's an issue if the OTA is not encrypted on Flash or that somebody's going to read the firmware out? People will always come to us with that question. I've worked for companies where that's, I mean, FIPP it was a little bit more, more cautious about it,

but PEPL was like, read our firmware. Like, you're going to get to it anyways at some point. I personally don't believe that is an issue unless you're doing something you shouldn't be doing, like putting, you know, really important credentials in there that shouldn't be there. Which you shouldn't be putting in there. Right. Yeah. But my general feeling about devices that you sell to people is once you don't have physical control over it anymore, and even if you do, somebody can get that stuff if they're motivated enough.

So you have to assume that that's going to happen, even if you encrypt it. You know, there's all these power attacks and things now. So, yeah, exactly. Power attacks, sending off the ROM. Yeah, looking at it in our microscope. It's really hard to actually hide your bits these days. You just have to make it not worth it. Yeah, just don't be successful.

Don't make, you know, security reaches one, want to look at your payloads. And I think the thing, the fallback is we make it very hard to put arbitrary firmware on the device. We don't make it. We don't make it impossible to read out the company's firmware. And I think that's the more important thing is putting arbitrary firmware on our device. Yeah.

And that's where my recommendations end for the most part is like sign the firmware, make sure no one else can install the firmware and like put your keys somewhere design where they're designed to be held. Yeah, GitHub. not GitHub. Do not put your keys in GitHub. What feedback do you have for Memboldt SDK or the experience? Well, so mine is going to be, all of your answers to any feedback I have are going to be update your Nordic SDK from 2.7.

Oh, man. Yeah. Yeah, we're on a really old, and we're about to ship, so we don't do anything. Well, I hope you'll least updated the Menfault SDK, because we still support old SDKs. You just have to not use the build. one? I did not. So I'm using the old one for now.

We have it on the list to update everything as soon as things calm down. But it's working fine. There are some limitations. Like it was not easy with the old old, old SDK to put arbitrary, I wanted to put arbitrate. I wanted to do something arbitrarily when the heartbeats happened. And there wasn't an easy hook to do that without rewriting a function. I figured it out.

but it wasn't like, here is the hook to do something every heartbeat. Yeah, I think it would be helpful, and this is a general thing, helpful to provide more examples to people of how to get stuff off the device when they don't have Wi-Fi. So I did have to scratch my head a little bit. And we have our own methods for communicating for our data. And I just eventually did it build on top of that, which is fine. It was probably the appropriate thing to do.

But for a couple of days, I was like, how do I do this? and how often. And I think for me, there's a little bit of confusion still about how sampling works. Like, okay, we have, we don't. But say we have 150,000 devices. Not all 150,000 devices are checking in every day and sending stuff to Memfault because that would be insane. So how do those get partitioned? How do we control that?

How does that, you know, feedback into how much it costs and that kind of stuff was a little bit confusing. Do you control that? Does the client company control that? Or does Moinfeldon fault control? we have control over that. Like, we'll report, but Memphal will toss them. Correct me. But like...

Yeah, it depends. I mean, with SaaS, our... Our costs come from processing data. Right, right. So yes. So the increasing device count will cost us more and will cost the customer more. For MCU devices, yes, we do exactly that. We basically toss the data if they send it to us for...

for higher level software, Android or Linux, they actually download the config data or not, and then it's controlled on the device. But MCU, it's honestly just so difficult to get data down to the device. And OTA is a special one that people will build out, but getting that config down for the device for it to know whether it should be in data or not,

usually gets built by the customer. And nobody wants to do rate limiting things on their own software. Time is hard. But they usually build a flag, that turns on and off our SDK itself. Yeah, okay. What was the other issue? Oh, time, but that's not their issue.

It's just you have to have the clock set for things to work correctly. I know. I was hoping that would be a solved problem these days, and it's still not a solved problem. Many devices are still being built today that don't have time. I'm just like, oh, no. Why don't the devices get the time from BLE? Well, they do, but you have to write that.

Oh. Then you have to keep it and persisted across the reboot. and that's not dependable and yeah. Yeah, we're having a lot of issues with that. It sounds like a deferral feature that should exist. I'm sure it does. It's just config underscore Bluetooth clock, NTP, something, something, something,

and it's only available in, I don't know, sorry. We could talk about Zephyr in a few minutes. But yeah, that's how this can scale, right? Is eventually you end up with some statistical sampling of the devices that are in the field, not every single device, dumping telemetry all day long. Yeah, my philosophy is

you want to capture the bugs that happen one and of every 10,000, 100,000 hours, and you need to know the number of devices and the number of hours. Those devices are actually running real software. And you need that many devices

running at any given time. So, like, there's an equation to run And then it also depends on whether those devices are absolutely critical or not. I would argue that we have a customer, wearable company, and every user pays them per month. And if a device doesn't work for a customer, that customer gets kind of mad and wants to stop paying them. And so they actually, like, really care how all their devices are working. So they actually track most of the devices and their data. some companies sell a device for $100

and never make any more money on those customers. And so they're like, we can sample. It's totally. What about FTA devices? Do you deal with those? I mean, are those a, we don't sample, we have to track everyone? All those customers are generally tracking every single device. Yeah.

But also like the medical device space, as you know, is just such a hairy place. We only work with a few of those companies. And usually it's during like internal tax. or very close relations with their end customer. That's like, can we track the data? Is the firewall open? Can it send data to Memfault? All these sorts of things.

And then also is it actually like a class one, class two, class three device. Okay. But as we've discussed, it's just observability. It's telemetry data. It's not hippo data. It's not. I mean, you could do something stupid because you can add, You can add custom metrics.

Some moronic. That's one thing I wanted to mention. When you say the company data and the Memphalt data is separate, it's under your power to keep those separate. Like, we do have some things that are specific to our device that we log in Memfault. Like, this device did not manage to collect any data, any customer data for the last hour that we log because we want to know. But that doesn't say which customer, it doesn't say what kind of data. It just says data collects failed or data collection.

succeeded or something timed out. But you could log whatever you want. It's just a metric. So you could put, you know, customer ID in there or social security number. Don't do that. Just with any sort of observability tool. You can log whatever you want to it. And so we usually push that on the customer.

We have a lot of hooks and recommendations on how to capture things for core dumps. It's basically don't grab those regions of RAM and, like, put all your private and sensitive stuff in a certain section and don't capture it. For logging, don't send it. For metrics, usually it's just numbers. And so it's honestly not that bad. One of the questions, a listener posed, Mark posed, was when shouldn't people use your product? Super low bandwidth. I mean, there's a few things, right? As we've grown, we continue to find places where we shouldn't use it. Super low bandwidth, Laura, is really difficult because also Laura's really hard to firmware update, but it's also just like hard to get data out in some sort of

reasonable capacity. So that's really tough. If the device requires 100% uptime, it's just another like nuance thing where it's like, once you scale to 10 to 100,000 devices and you're kind of at those numbers expecting that many devices fail and you're just kind of trying to mitigate those, we have customers that come to us and say we have 10,000 machines on the, the factory floor and if anyone goes down like we're host can you help us and like that kind of actually is their bread and butter is an observability system that allows them to do that like that

would actually be a differentiator for them we can help them we still do sign them but it's more of a supplementary thing yeah because that's a timeliness issue right memfault they need to be alerted within five seconds if something's going down or wrong and like memfault is built or just has been built over time for our sampling. And like we said, like, you want to sample some of the devices, not all the devices. So we've made some compromises along the way or optimizations to where we're not trying to track second resolution of data and build that a learning product. Right. I mean, when I think about systems like this that I've worked on, there are some systems where you want to know right away. But for large parts of...

wearables or other industrial things. It's pretty much just, oh, I just walked into work, did anything catastrophic happen last night that I needed to add to my list today, not wake me up at 3 a.m. when things go wrong. Can you imagine me woken up at 3 in the morning when a FIPP and went on dryer? First of all, who's doing laundry at 3 in the morning? There were days when it felt like that. But yeah, but that's where like that's where that observability, that alarm is, part of the product. Like, right? That's, it's part of the product for you. Like,

exactly. And they, and they will, you know, it'll either require tons of money to observe it or they will charge their, you know, downstream customers, tons of money. And honestly, like, those customers should probably use something more similar to Datadog because Datadog charges, you know, $10 per device per month. But they will give you, they, you know, they will process and store and that much data and and have built a product that is more along those lines. But that will bankrupt any company that is trying to build something. Yeah, our $99 wearable $10 a month per device. That's for like faster data where you need alarms.

Yeah. And very timely response. Also like data dog is not built. They're sending data in JSON and binary blobs. And, you know, it's going to be megs of data per minute. is kind of their threshold. And we're built for a different system. Oh, Christopher's telling me I can't type and talk at the same time.

I don't know what he's talking about. I do have a support question for you while you're here. Great. I'm available, apparently. I don't understand sessions and where I would use them or what they do. Yep, that's a good question. You've been validated. Yeah.

Okay, well, it's been a good show. I agree. You don't understand. There are devices that are operating all the time for those devices. We, and it's built-in, capture heartbeat metrics. Heartbeats are typically captured every hour or two hours, which is just like a summary of what has happened over the last hour. How long was your Bluetooth connected?

How many packets didn't you send? How many bytes were written to flash? All these sorts of things. And then there are devices that are used sporadic. like a camera or like a personal camera you want to take a session or a playback session for a media player. All those are kind of you're trying to track a summary of what has happened during that session, which may be a minute or an hour or a few days. That's when we guide customers towards sessions.

There's nothing by default built in because we don't know what your product does. I think one session we honestly should implement, but we've not yet because it can lead to abuse is like a Wi-Fi session or a Bluetooth session. Like when that connects and then disconnects, like give us a summary of what happened over that session. Oh, okay, okay. But sometimes devices just get into loops or they're meant to disconnect and connect all the time. So that would kind of just lead to a bunch of noise. It seems like you could get in trouble with too much data with, with, with, that paradigm. Too much data. And for some companies, like, you know, a kilobite of data here and there

is actually a lot. Like, we're talking to companies that are, that are deploying. Cell modems. Or even worse, deep-sea modems. Deep-sea modems. Cell-moder are so bad anymore. They're so slow. I'm glad you're working on those, or at least have understanding. Yes. Deep-sea modems are just acoustic couplers separated by a kilometer, right? Let's go with yes, because wow, they were loud. I listened to you testing it in your office. That's what it sounded like. But all my nostalgia for how the modems used to sound.

Now I could just have it be louder. I wasn't quite done with that when we shouldn't use your product. I get the idea that very small systems like my 10-person family goldfish bowl. wouldn't work because I wouldn't want to pay for it. At what point does Memfault recommend we start thinking about integrating? Is it 15 or 1,500? It's a good question. I would say it's more along the lines of around 1,000.

It's like, I guess for NRF Cloud, like the provisioning security, you're buying NRF91's, and you need them to work over LTE, like that sort of division of the product, a hundred or so. Like, you probably should be using NRF cloud rather than trying to roll your own solution.

And then for the diagnostics piece, I would say a thousand. I would say we don't talk to too many companies or it's not worth the effort. And I'll expand about that. unless the companies are trying to shoot for a thousand and then hopefully more. Like we rarely talk to customers that are like, we only own a bill a thousand and that's it forever because it's kind of a tough sell in that sense.

And by the effort is like, we put forth a lot of effort to help our companies instrument their firmware. There's only so much memful can provide out of the box because every product is ultimately different. every the way people think about things and how you want to track your either as product analytics or firmware or what signals a broken device like we will try to help them with that we'll run sessions every week or so about how to do that sometimes they come to us with a new MCU or a new version that we've never of an artas that we've never worked with like we'll expand the SDK support and kind of help them along the way. And so some companies are not worth talking to because they don't have the appetite. And yeah, I could go on.

At what point is somebody too big to go to Mimphalt? We have a few companies in the millions of devices threshold. Anything beyond that, it's a really tough sell because they do start getting into the into the cost. It's cost prohibitive, basically. But those companies are starting to look into the sampling. They're like, we don't actually need. Also, if they're selling a million devices,

they don't need all million to work every single day, basically, or they're not as critical. They also assume that many are going to fail. And it's just how quickly can we drive these customers through customer support and replace these products, which I guess Memphis does help with to an extent. Any company that has many more than those devices, they probably also have many more engineers, and they can allocate a few engineers and a cloud engineer and a data engineer to build something

that's more tailored to exactly their use case. We've done our best to make a generalizable solution that works for most customers. But if you're trying to, for instance, look at your back-end and mobile. app and firmware logs and how it pertains to a couple other devices in an ecosystem, and you want to all view that in one single pane of glass. Like, Memfault is not built for all of that. And if companies are kind of like needing that and requiring that, then we can hook into that and we can help you with like API integrations, but maybe they should just go off

and build their own. there is a point at which it is worth going off to build your own for sure it's all about money it's all about money and like do they have the resources and firmer engineers like sitting on their hands enough to honestly like go do the thing we talk to a lot of companies that just are like out of resources they're like we have no time to debug our products and we're completely underwater and support requests can you help yeah and there's a lot of companies that just wouldn't do that period. Like my, the company I'm consulting for, there are two, two firmware engineers, me and the other guy. And we would not be doing observability for the two of us because we can barely, you know, keep up with the work that we have. So doing this was. But you still had to do firmware update? Permware update already existed. Oh, okay. Well, I mean, we just did Nordic's thing and yeah, I mean, we had to do firmware update, but there was a lot of stuff already there. Yeah.

But thankfully, the hard stuff on firmware. update is, like you said, it's much better than it once was. The bootloaders are more or less figured out. You have AB partitioning to kind of pull down new firmware and flip over to the new one. And the MAM fault solution is a lot of, like, help me build the back end to firmware update so that I can roll it out to certain devices based on certain rules, which again requires, like, a back-end engineer for this diagnostic piece, which we hope no one has to hire themselves because they're very expensive and kind of just very single use case.

And this client had already done the OTA piece on the back end, so we'll be replacing that. We'll just be switching over to get it from you instead of S3 plus a lot of features. So it's nice. The other question that Mark had, which was a very good question, is what do you wish customers would ask before they start a serious engagement? That's a good question. It's not necessarily, I mean, I guess it is a little bit of an ask, but it's like ask themselves, like, what is this worth? The reason is because I think there are many, our equivalent in the cloud space, our century, it's data dog, bug snag, crashlytics, a lot of these tools are very good at like starting from zero.

one engineer can instrument their back-end application, their Python application, and get value out of it, and they can generally scale to an enterprise scale if they're working for a company. firmware is less plug-in-play than a PIP install library. I would love for even the Nordic stuff to be even simpler, but ultimately, like, people rearrange their memory maps differently, and they need to, like, store a bunch of data

and a lot of that stuff and transports itself just make it much more difficult. So we spend my time. I spend hours with customers trying to help them and my hours is not, you know,

my fees and my hourly rate is not $5 an hour. So it's like, I wish the customer would come to us and be like, is this problem worth an eighth of an engineer's salary a year? And if so, like let's use Mimful. If my budget is $1,000 a year, than like I can't I can barely hop on a call with you to solve your problem or like talk about your problem basically.

I think it's just always a surprise. It's always a sticker shock. And I know the pricing, I think we actually are pretty explicit at the price on our website, or at least like ballpark wise. Yeah, it was pretty quick to find when I looked for it. And like I wish it was cheaper. I wish it was $100 a month. But the number of companies that would be successful without having. a lot of our time spent with them, kind of pair programming with them. It just wouldn't work out ultimately.

So you do have to do a fair amount of handholding. You haven't gotten to a crank solution where you say, here's our website, and you never really talk to them. I didn't talk to them. There's a spectrum because I've never talked to either of you about non-vault except for this session maybe. And there are engineers we've worked with in the past. who've kind of used us through multiple different companies, they are pretty much self-serve. Like, they actually will come to us. They'll be like, I need an account.

I'm scaling the $10,000. What's the price? Great, I have budgeted. Like, truly, like, this is maybe like tens of companies that have done that, but it's because engineers have kind of traversed through different ones. And there are very smart engineers who never want to talk to a salesperson in their life, who are committed, who will do it themselves. And we'll ultimately just, like, give us a call or a ring when they,

and they have a hundred devices, and they're like, cool, I'm like blocked now. I would say the average engineer almost always needs a good amount of time from us. And especially to, like, convert the problem into the solution. I would say some can do the integration, some can capture a core dump and can see it. But, like, now how do they write code in a way that once they have this new ability, how do they now put use to it or make use of it?

And that is where I think you require, like, my eight years of thinking about it. Yeah. Or a lot of our teams. The problem space is hard if you've never seen it. I mean, I did toys and there was no internet connectivity for most of my toys. And even with that, I still had to think about scalability and bug fixes. And then with ShotSpotter and we had hundreds to thousands of units in the field,

and I needed to know which ones were healthy, which ones weren't, and how that would affect my entire distributed system. And so we built Excel sheets that did what involved us. And we spent a lot of time on firmware update. And so when I got to Fitbit and we were looking at problems of scaling and how to know that systems were healthy and all of the telemetry and diagnostics. And I'm like, oh, well, I'm definitely not using Excel again. Let's just go to Python. And I thought I was so cool for that. Of course, my login for that lasted way too long,

which you don't really want your engineers to have access to the customer data or even the diagnostics. Well. Unless that's what they're doing. Yeah, yeah. But since I was the only one who really understood the Python script, I had access a lot longer than I should. but the whole process of understanding how these things work together, it doesn't come up when you're building the internet-enabled goldfish bowl. It's not, it doesn't feel like a core product feature.

It isn't part of the MVP. The minimum viable product is, did the fish die? Correct. And so it's, you're right. you need someone to spend a lot of time thinking about this because most people won't get the, we'll only be able to see firmware update because that's something we all talk about.

They won't be able to see the observability or the horrific things that happen when you have a one in a million bug, but you also have a million pieces of million devices out there. In which case, you're one and a million bug happens all the time. And we started the company when, of course, we weren't, it wasn't at the very beginning, but many companies had never thought about any of these problems before.

They still don't. Truly, truly, they still don't. Many people have still never searched for OTA yet in Google to find us, or I've never heard of MMVolt. And so it was a lot of education. We spent a lot of time educating customers on why they should care. And I think that was still something that we are continuing to do all of the time, which is why we do these podcasts and why we have our whole podcast series as well at Memfault is sheerly education. And what is the name of your podcast series, which I should know because we did an episode with you?

Coredump. The Memfault CoreDump series. It's usually with Francois and Chris. But yeah, the people are never going to, and I understand it, the people are never going to understand it. the people are never going to understand the pain until they felt it, which is why we have so many repeat customers that have moved from one company to another. So that is a very common, honestly, path for inbound. And sometimes it's not the right time or the right moment, which is why we try to sell them the security provisioning beforehand or I guess earlier in the life cycle of their product, because those are things that they absolutely. need, whether that's for, you know, legislation and requirements, and then later we can sell

them the observability piece when that feels more on fire to them. Yes, because that's when they'll need it, is when the fires break out. Yeah. I mean, I wish companies would put it on 10 devices because I think at Pebble, like, one of the, like, our critical piece to get working on, we had a dev board, basically, and it was like, cool, dev boards get OTA working and get CoreDumps working, because otherwise we're not going to move fast. I wish more companies would think that way,

but I understand it when they don't prioritize that work. No, because that work is not the work that is the features. Correct, exactly. Show me on this spreadsheet, the dollar is coming from debugging. Yeah. Nowhere. Well, I, it sounds like, Nordic is a good

thing for Memphal, but I have to say I'm quite disappointed selfishly. So here's the thing. What? One, we were a startup after all. We had to continue raising money. We,

at our, I guess, the growth expectation, we were never going to last forever. Or it would probably turn into a little bit of a different company in terms of what we could support.

Maybe we'd have to cut a platform. Maybe we'd cut Android or something like that. With Nordic support, though, we can do a lot more. And they want us to do a lot more. One thing that they love and that we will not stop doing is supporting every chip and every hardware platform out there. Because if Nordic, God forbid, doesn't make the sale on a Bluetooth chip or a Wi-Fi chip, they can still get some revenue from other company's chips by having them on MIMFOL. So that is something that we will continue to do forever. I'm sorry, Tyler.

I was totally teeing up a joke and you ruined it. I did. Sorry. But you, you, you, why are you disappointed at this? I was triggered because every single company that talked to us for six months after the acquisition came to that as the first question. Be like, are you going to support NXP chips? Like, should we just stop talking right now? My personal beef with this is that Memphal was a sponsor of the show.

Nordic was a sponsor of the show. And then you guys got together and now neither one of you is sponsoring the show. Oh, I didn't know that, actually. So not a big deal. I just wanted to cancel you. I really didn't mean to trigger you. Well, it's good. because I was probably not going to say my little spiel had to not come up.

Tyler, it's been really good to talk to you, but we should go out and enjoy that sunshine. Yeah. Do you have any thoughts you'd like to leave us with? I do. I thought about this one. One of my favorite things in the last few years since we last chatted was we met a lot of our neighbors. And it has turned into a wonderful community, both of families and friends and kind of a lot of different people and a lot of different times of their life, which is amazing. And I hope everyone else kind of prioritizes it because it's really changed their lives,

my wife and I. Meet your neighbors. Meet your neighbors. It's one of the pieces of advice you would get if you joined your community emergency response team. The folks who withstand emergencies in the community fires, hurricanes, earthquakes, whatnot. are the communities who actually can say more than hello to their neighbors, who know just a little bit about their neighbors. It really is effective for disaster situations,

but it's also just nice. Our neighbor's moving, and we are bombed. Oh, man. Hopefully we'll get another one. Maybe a cute family. Our guest has been Tyler Hoffman, co-founder of Memfaults. Thanks, Tyler. Thank you both for having me.

It was a pleasure. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack group for questions and coffee supporters for coffee. And thank you for listening. You can always contact us at show atembedded.fm. Or hit the contact link on Embedded FM the website. Now a quote to leave you with from Jane Goodall. What we do makes a difference.

and you have to decide what kind of difference you want to make.