Embedded - 463: Layers of Band-Aids

Episode Date: November 2, 2023

Kevin Lannen is an embedded systems engineer making powered wheelchairs safer. This sounded interesting to us. Kevin works at LUCI Mobility (luci.com). Check out their tear jerker introduction video as well as technical description of over-the-air update concerns on smart wheelchairs. We also talked about the app that goes with the system: LUCI View. You can find Kevin on Twitter (@kevlan) and LinkedIn. Go Baby Go - The Adaptive Sports Connection Transcript Memfault is making software the most reliable part of the IoT with its device reliability platform that enables teams to be more proactive with remote debugging, monitoring and OTA update capabilities. Try Memfault's new sandbox demo at demo.memfault.com. Embedded.fm listeners receive 25% off their first-year contract with Memfault by booking a demo here: https://go.memfault.com/demo-request-embedded

Welcome to Embedded. I am Alicia White, alongside Christopher White. Our guest this week is Kevin Lannan, Embedded Systems Engineer at Lucy Mobility. We're going to talk about wheelchairs, control, and people. Hello, Kevin. Good afternoon, morning, evening. Hey. I don't know where you are, so I'm just going to cover all the places. Yeah, thanks for having me on the show. Could you tell us about yourself as if we met at, I don't know, Supercon, Hackaday Supercon?

Yeah. I'm Kevin Lannan, and I'm an embedded systems engineer at Lucy, where I work on really everything from designing the electronics, hardware, embedded Linux. We do some bare metal C programming, all the way up to application software that controls our whole system. I kind of always try to push my understanding of the systems I work with to really include as much of the stack as possible. And I really love doing system optimization. So squeezing as much performance out of the hardware and power budget that we have. Outside of my engineering work, I spend my time skiing and hiking a lot here in Colorado.

And what exactly is Lucy? Yeah, so Lucy is a safety system that installs onto existing power wheelchairs. So it provides sensors that provide a 360-degree view of obstacles around the chair, as well as the ground. So we actually can look and detect steps and drop-offs. And then we interface with the control system of the wheelchair to prevent the user from being able to run into things, drive off curbs. And we also provide a little bit of cloud connectivity

that provides some alerts to the user's caregiver team. So you can kind of keep tabs on your user. So if somebody tips over their wheelchair, we can send an alert and say, hey, you tipped over your wheelchair at this location and they might need help. All right. So I have questions about so many of those things, including controls. But first, lightning round, are you ready?

Yes. Complete one project or start two dozen? Oh, three dozen, is that an option? Black diamond, double black diamond, or green triangle? Oh, double black diamond. Wow. I have been ignoring rust. Is it ready yet?

It depends on what you're doing, but for a lot of things, yes. Favorite type of motor? Ooh. Brushless DC motors. Have you started any motor fires? I have not caught motors on fire. I have gotten

some FETs awfully hot to where I think one of them fell off of a board once because it unsoldered itself. Least favorite analog component? The, ooh. Analog in general is kind of ugly and nasty. Favorite fictional robot? R2-D2. Good one. Favorite microcontroller?

Not necessarily my favorite. The one I know most about, I really like the MSP430 family because they are super simple to use. And finally, I do have a tip everyone should know. Yes, it is debug to the root cause. So when you're debugging a problem, try and get all the way to what the actual problem is and not just try and cover it up with layers and layers of Band-Aids on top of it. You really learn something when you actually dive in and solve a problem all the way to the bottom.

We are happy to be sponsored this week by Memfault. Memfault provides a device reliability platform for IoT monitoring, debugging, and updates. Device operation no longer needs to be a scramble as issues with fielded units pile up. Instead, Memfault gives developers a more scalable and sustainable process to accelerate time to market, de-risk product launches, cut development costs, and deliver higher quality products. So if you're wondering how you're going to monitor your units once they're shipped, or whether your firmware update plan is secure enough, it's time to take a look at Memfault. Or you can read their Interrupt blog

for all of its fantastic goodies on how to debug hard faults, monitor units, or generally write good embedded code. Embedded FM listeners will get 25% off their first year with Memfault if you request a demo through go.memfault.com slash demo dash request dash embedded FM. It's a link you can find in our show notes. Thank you to Memfault for sponsoring this week's show. Okay, so Lucy. A few years ago, we were at the beach, and a woman in a wheelchair was being pushed over the bridge. The bridge had a little bit of unevenness to it, and the pusher didn't see the lip.

And the user was dumped out of the chair. Do you remember this? No, but that's probably for the best. The chair landed on the person who was sitting in it and it was awful and the person seemed okay. The pusher was very distraught and it just, the feeling of powerlessness. Lucy's supposed to prevent that sort of thing uh yeah to an extent so we we help prevent that with with power wheelchairs where we can detect um places where it's unsafe for the wheelchair to go like uh yeah dropping off the edge of a bridge or something like that it's a little harder for a manual chair where somebody's pushing um but there's definitely a lot of uh off-road wheelchairs

and there's other things out there, but it's hard for people to get access to them, get funding for them, that sort of thing. So there's a lot of things that are both kind of technology that can help prevent that, as well as there's some questions about how folks in wheelchairs get access to technology and to the equipment they need to really live a fulfilling life. I mean, it's always a matter of cost, right? At the end of the day, yeah, it all comes down to cost and some inertia. I mean, there's a lot of things that I think we at Lucy, we think we could reduce costs in a lot of areas for people in wheelchairs as far as if you make something safer, fewer injuries that insurance might have to pay out, that sort of stuff. But there's a lot of inertia behind it.

The insurance companies don't like to go after new billing codes and new technology as much as I think the folks that are relying on this technology would like for sure. And so is Lucy a certain type of wheelchair? Lucy is not a wheelchair itself. So we work with about four or five wheelchair manufacturers right now. And we integrate kind of as an aftermarket accessory. We have a frame that installs on the wheelchair that has all the sensors. And you kind of wire it in, plug it in, and works with these existing wheelchairs on the market. Existing powered wheelchairs, though. Yes, existing power wheelchairs. And specifically, it's what are called kind of group three wheelchairs.

In the power wheelchair space, there's kind of three main categories of wheelchairs. You've got the mobility scooters, which folks see around a lot for helping people get around. And then there's the group two chairs, which are a more advanced power base with just a pretty simple seating system on top of it. And then the chairs that most of our users are in are kind of in this group three, where they're very complex wheelchairs. But the main thing that gets more complex with them is the seating system. So these seating systems can go from tilt and recline to being able to elevate. So you're up at eye level with folks

to even kind of putting you in a standing position in the wheelchair. So that adds a lot of cost and complexity under the wheelchairs. And a lot of cost and complexity under the wheelchairs. And a lot of the innovation recently from the manufacturers has really been on the seating side of these complex seating assemblies. But there hasn't been as much attention paid to the drive side, which is kind of where we're focused on with our safety system. And so group one wheelchairs are probably, you said mobility scooters. So I'm thinking sometimes they even have those at the grocery store use the scooters to get around so they're not having to walk as far. So those are for somebody who like normally can walk but can't walk a super long distance. So the scooter helps them get around kind of to complete their everyday activities. And group two sounds like it's similar, that it's for people who

maybe can't walk any distance but aren't in the chair 100% of the time. Yes, they're not in the chair 100% of the time, so they don't have the complex seating and positioning needs that somebody who's kind of full-time in a wheelchair would need. And that's where group three comes in. Yep, that's where the more complicated and much more expensive wheelchairs come in. We can talk about Lucy, people say, oh, Lucy's kind of expensive. We're just under $10,000.

But some of these wheelchairs, the sticker price of them can be easily as much as a luxury car in a lot of cases. So it's a really expensive product with all the complex needs that these users have. I mean, you're spending your life in it. This isn't your desk chair. This isn't your couch. And even people spend a lot of desk chairs too, but not that much.

But yeah, but you spend a lot of time with it. It makes sense to invest in it. And it affects your health. It affects your ability to interact with other people. It's a home. It's a home and it's your legs. So being able to get out in the community, get outside, go to the mall with friends, like these are all things people in your wheelchair want to do and

so it's all about making it easier for them to get out and enjoy life i mean we've seen with some people we've talked to and interviewed huge kind of uh like mental health benefits of being able to to get out and be a teenager for instance for some of these kids so it's that's a really empowering thing of working on this is seeing how we've really changed some people's lives of letting them get out and do stuff that they were not able to do before. One of the things I saw in your videos that I hadn't considered at all was crowds. I mean, I don't like crowds because I don't like crowds, but I know, you know, going to a soccer game or a concert or wandering the mall.

You're with three or five people and they're shifting around and moving and walking. I'm used to, you know, going wherever I want to go and standing next to whoever I want to stand next to. that as a wheelchair user, a powered wheelchair user, keeping up with the crowd is a constant thing you have to control. And it would be really easy to hurt somebody in front of you or to stop suddenly and be hurt because somebody walked into you. Yeah, that's a huge source of anxiety for a lot of wheelchair users. If they really don't want to run over somebody's toe or bump into somebody, these wheelchairs are 300, 400 pounds. And so you can do a lot of damage pretty quickly. Our CEO loves talking about Disney after the fireworks

because they brought their daughter Catherine to Disney. And you run into a dilemma as someone in a wheelchair. You can either leave before the fireworks so that you're not in this big crowd, or you have to sit there and wait for everybody to leave so you've got a clear path to drive out. It really limits and changes how you think about these experiences out in public. And so that one's a big thing that we've done a lot of work on. Our system kind

of works like the adaptive cruise control in a car when you're in a crowd. So you can just hold forward on the joystick and it'll just kind of keep up with the person in front of you and kind of move you along. And that's a huge deal for a lot of people, being able to have confidence in these crowds that they're not going to hurt somebody. How does it work? Yeah, so we use a variety of sensors. We have a saying, this came from White Cork a long time ago on Twitter, that all sensors are wrong and some are useful. And so we use a variety of sensors to kind of make up for the inherent things that they're good at and not good at. We use some stereo vision cameras from Intel RealSense. We have some millimeter wave radar that we've built on top of some of Texas Instruments radar chips. And we use some

ultrasonics and we've got an IMU on board. And so we're constantly looking out and detecting objects. And then we scale down the joystick of the user proportionally to how close they are to objects. So that as you approach something, if you're in a tight area, we kind of slow the speed of the chair down until you get right up against it, and then we stop. And if you're in a wide open area, we let you drive around as you would normally, as if the system wasn't there. We're starting to add a little bit of some kind of semi-autonomous features to help people in common scenarios, like getting up ramps on wheelchair vans, for instance.

So in these tight areas where you want to do the same thing every time and not think about it, we're working on how can we help automate those. But we're not really an autonomous system in general. We don't want to take away the the input of the user that's really important of you you want to tell your you don't want to have to like have your legs take you somewhere without you telling them where to go sort of thing so we're we're focused on the um how do we engage the human keep them in the loop of control, but provide them kind of safety and confidence in what they're doing. You mentioned your CEO, Dean? Yes, Barry Dean. He's a country music star?

Yes, he has written several country music hits. He's a singer-songwriter out in Nashville. And so he's written for like Little Big Town, Pontoon, along with quite a few other songs that people have probably heard. So yeah, kind of an interesting story if we go from him being a singer-songwriter to the CEO of a tech company working on power wheelchairs. And it was because of his daughter. Yes. His daughter, Catherine, has been in a power wheelchair for pretty much her whole life. She has cerebral palsy. The very first time she got in the wheelchair at the hospital, it wasn't configured right. And she drove it down the hallway and into a wall. So it's kind of from the very first time, the first experience they had with a wheelchair,

they've known that these things are dangerous. And she also has some vision limitations and reaction time limitations. And it made kind of everybody nervous. They wanted her to be safe as she drive around. And then as she got older, they also wanted her to be able to be more independent, to go do things on her own. She's super social, loves to go talk to people and hang out and stuff. So she wanted to be able to get around and do stuff on her own. And so Lucy was the answer we created for that. Does having a prototypical or archetypical user help you define the system? A hundred percent.

We really focus on all sorts of use cases. We talk about users as far as we have the safe drivers, that they're the people that drive slow naturally, even without Lucy, want to be safe. They're really nervous up to, we call our race car drivers. So the people that want to go fast everywhere all the time. And thinking about how those different people would use the system really helps us kind of tune the behavior

and how we've developed the product for them. The other thing we think about in terms of users that's interesting with a medical product like this is that the person in the wheelchair is not our only customer. We really talked that we have multiple customers. So we have the person in the wheelchair that's actually using it. We also have the clinician that's prescribing the system. We have the care team of the parents or the spouse or the paid caregiver that's looking after the person

that's with them every day. They have to interact with it. And then there's also who's paying for the system. So you may have the insurance or the VA in some cases, or in a lot of cases, it's the family of somebody. They want this now and don't care about getting insurance to pay for it. And we've had a lot of people just buy one kind of on their own. And so we're thinking through all those use cases too.

If it's not just about the person in the wheelchair, for these folks, life really is a team sport with this whole care team. So we really have to focus on meeting the needs of every single person in that chain. When I spoke with somebody else who has worked on powered wheelchairs in the past, he said that one of the main questions that came up often was, where is the cup holder? Do you have to deal with that sort of thing? Yeah, I mean, people want all sorts of different customizations. We've actually 3D printed some cup holders that we had on some chairs at the office so we could have our cup of coffee while we were driving around the office in the wheelchairs. So these sorts of customizations and the stuff that people come up with for their chairs are really cool to see.

Both the folks themselves as well as some of the ATPs that are the wrench turners that help set up these wheelchairs and get them and maintain them for the users. People come up with some really creative solutions to all sorts of different customizations and needs of the users. What's an ATP? They're the assistive technology professional. So it's a certification that you can get and it allows you to kind of work on these wheelchairs. So if you get like a wheelchair delivered to you, the ATP comes along and does the install and the setup, makes all the adjustments. They might help tune the control system on the wheelchair for how you want to drive it. So they're kind of the, I guess a wheelchair mechanic is not a bad way to look at, but then they also do a whole lot more as far as kind of helping address the medical needs of the...

How open are these wheelchairs? Like you said, you integrate with other manufacturers' wheelchairs. That's your thing, you're an add-on. How much time do you have to spend reverse engineering multiple manufacturers or are there only a few out there? I know nothing about this industry. Yeah, it's an interesting industry in that regard. The answer to how open they are is basically they're not. There are only a couple control systems on the market, or two or three

as far as the electronics. And so we work with one of those control systems. We've reverse engineered pretty much everything about that. We don't really work with the manufacturer of that a whole lot. So most of the communications and stuff we had reverse engineered. And then we've worked with some of the wheelchair manufacturers themselves that buy that control system for the mechanical integration. We have these kind of sensor pods that we've designed that we can place all over the wheelchair. And then for each model of wheelchair, we do a

unique sheet metal frame that allows it to mount to whatever points are available to mount on the wheelchair as kind of our point of customization there. So that's where we've been trying to collaborate and get stuff a little more open. But it's a tough space. It's pretty closed. And even like modern software development practices are not so much of a thing in the wheelchair industry right now. And do those companies not see the need for this kind of enhancement? They don't know how to do it.

Why haven't they done this? It's a bit of both. Some of them, I think a lot of it is they don't know how. The wheelchair manufacturers are actually really good on some of the mechanical engineering. Some of the suspension systems on these

and the seating assemblies and systems that they've come up with are pretty incredible. The software integration point and how they can use software to help out the users is pretty new to them. We were definitely the first in the industry to do over-the-air updates. We've seen a couple manufacturers that seem to kind of be following us in that regard. And we've also done some stuff that some folks are copying as far as setup.

So like previously to configure a wheelchair, you had to plug in a little programmer and pull up on your Windows laptop with the specific special software than the ATP could go tune settings on the wheelchair.

And now we're seeing a little more of that being driven through kind of mobile apps and web interfaces to these. And we're seeing kind of the software updates start to become a thing in that area. So folks are definitely kind of looking at what we're doing and taking note. But it's an industry that hasn't historically spent a ton of money on R&D.

And getting that ship turned to where they're booking at the world in a modern way has not been a very fast process. It reminds me of LeapFrog, where one of the best things they did before they went defunct was they forced Mattel and other big-name toy manufacturers to put the whole alphabet on toys instead of stopping at the letter F because that's as big as the toy was. Do you think Lucy is putting pressure on other wheelchair manufacturers through customers or through regulations? It's a little bit of both. So we're definitely we're seeing customers kind of asking for more that having the users be able to actually customize their chair a little bit without needing the ATP. And so some of that's some new chairs coming out, we're seeing some of those apps side. So we're involved with a couple standards committees to try and work on defining actual standards

for this smart technology of how do you test a wheelchair that has these sorts of safety systems. There was no existing standard for that. So we made up a process internally that we're pretty happy with right now. And we're working with the standards organizations to try and figure out how we can actually turn that into a standard so people can compare safety systems. There's some other options out on the market that don't as much talk to the wheelchair, but have like beepers and stuff. So kind of setting up that. How would you look at,

or how would a company go about testing and comparing the capabilities of these safety systems? Some of the Lucy documentation compares it to a car, including talking about the level two auto drive. And I can see how you'd get to standards from there because we're doing better with talking about cars. But cars don't go into houses. They don't go on sidewalks. They don't go in the grass. Is it a reasonable comparison? It's a great place to start. And it was where we started. It definitely breaks down once you get more into the details. So we looked at the cars and said, okay, they're doing these lane keep assist features. They're doing the automatic emergency braking. There's the adaptive cruise control that you can get.

And you kind of start putting those together. And that was kind of some of our initial inspiration for really looking at this technology was the wheelchairs had a seatbelt was the only safety feature. So we looked at, okay, well, what does the auto industry have? And how do we bring that technology to the wheelchairs had a seatbelt was the only safety feature. So we looked at, okay, well, what does the auto industry have? And how do we bring that technology to the wheelchairs? But then as you get deeper in, it is a very different set of testing.

One of the big challenges with a wheelchair is they can zero point turn 360 degrees. You can spin around. Cars, most cars on the road can't do that. And so it requires a different level of testing, level of putting sensors around it. The sensor coverage has to be a lot more complete around the wheelchair because it can move in really different ways. And it is operating in tight environments.

One of the things we struggled with early on when we had to develop our own radar and ultrasonics was we tried looking at automotive sensors, and most of them can't detect something that's only four centimeters away from the sensor or in really close and so we had to kind of go take a step back and say okay we need to actually detect stuff in really close because as a wheelchair user you have to be able to get right up next to a toilet to transfer or get right through tight doorways. Some of the apartments people have for these that they live in with a wheelchair are incredibly small, really tight quarters.

And so being able to navigate those, it really is a different kind of control problem and way of looking at the world from a wheelchair compared to a car. But the high level of how we talk about the autonomy and some of the testing that needs to be done does end up being really similar to the automotive industry. And your sensors, you mentioned vision and radar and ultrasonics. And do you, I'll tell you what I think you do with them and you can tell me how wrong I am. I would want to create like a cloud bubble of everything that you could identify that was near you. So that you could say, okay, there is someone walking in front of me from the left. Or there's a curb coming up and it's three feet away, it's two feet away, it's one foot away. Sort of a point cloud of everything out there, which is

kind of what radar does on its own, but it could be augmented with other things through Kalman filters or other heuristic models. How close am I? That's actually really close. So yeah, we combine all of the feeds from the different sensors kind of into this one, yeah, point cloud view of the world. If folks go look at, we have a video of a, there's a visualizer that's available in our app

that actually shows all the points around the chair. We stream this over Bluetooth from the system onto your phone. So you can use it kind of as a backup camera view or almost looks like some cars have that like top down view of the zone goes from green to yellow to red as you get closer backing up to something. And so we use that view of the world and then we have kind of zones around the chair. So we say, okay, if something's in the zone that's around your right front caster, then maybe you can't go turn to the right. Or if something's right in front of the footplate, you can't go turn to the right. Or if something's

right in front of the footplate, you can't drive forward, but you can go every other direction. And so we combine all these sensors and create that view of the world and then kind of scale the joystick in different directions depending on where the user's trying to go. Can you talk about the algorithm of combination? Yeah, most of it right now, we just kind of add all the sensors up. We have some algorithms that take out specific data we know to be false with the radars. If you have certain stuff, it'll show like a false point right in front of it. But for the most part, we just combine them all. The main separation we do is we treat the world pretty differently

between what we call collision, which is things you can run into versus our ground detection. So the driving off steps. Steps are a little more tricky because we only have really one sensor that can detect them, which is the camera. And so we have kind of a couple different approaches and processing that we do on that data to make sure we don't miss steps around. The other thing we have to use for some of the ways we combine sensors to make up for shortfalls is like the cameras is not very good at seeing glass. So like where we have a camera on the system, we have an ultrasonic right next to it that's looking the same direction

so you wouldn't back through a glass window, for instance. So that's kind of where we have that all All sensors are wrong, but some are useful. Some of them are going to give wrong data, mostly missing objects, but some of the other sensors might be able to pick up and kind of fill in that gap. Like ultrasonics have a hard time seeing a soft, fluffy thing sometimes, for instance.

So then the radars that are looking in the same direction of those can pick up the object that the ultrasonic would have missed. Do you have trouble driving over black rugs? I know our vacuum does. It's very afraid of black rugs. We don't have too much problem with dark floors. Shiny floors can be a problem for stereo vision because it'll actually see the reflection.

Yeah, I was going to ask about reflections, yeah. Yeah, reflections are nasty. We're constantly working to kind of improve how we handle those. We've gotten a lot better than we used to be, but they're still pretty hard. The cameras we use have an infrared projector on them that sends out basically noise. It's a little infrared laser laser and that helps it resolve areas where there's not good lighting or if it's a really flat surface because stereo vision doesn't work

really well you need you need something that contrasts so it kind of adds uh contrasting noise into the scene and so that helps in a lot of situations um but the the reflections are really the difficult one to deal with. Can't just fix that with polarized filters? We actually do use polarized filters. So that's one of our tricks and it does work pretty good, but... As long as you're aligned right. Yes, we do. And we do align them right. So yeah, we have different ones. The front cameras are vertical and the back cameras horizontal. So we actually have different, the flipped polarizers between the two. Going back to sensor fusion, I'm afraid right now to say the word machine learning here

because there's been some crankiness about chat GPT lately. I'm in favor of machine learning. I'm just not in favor of large linear models. Do you, have you tried out machine learning for these problems? Because in favor of large learning models. Have you tried out machine learning for these problems? Because it is a lot of data. It's a lot of data, and machine learning is something we've kind of, we've played with it a

little bit, and I think there are some applications for it in our product, but right now we don't have any machine learning on our product. Everything is deterministic, which is really nice for... Testing.

Kind of documenting how it works. And how the... We can make a case of like, okay, this is exactly how the algorithm works and can explain it. And so that really helps with some of our safety case. We're looking into it a little more for some of the individual sensor processing.

So like looking at the data we get back from an ultrasonic and determining where an object is, like that could potentially be a really good application of it. We've kind of explored it in a couple of those places. But as far as a whole world look at machine learning of all of our data, it's not really a good fit. We're also pretty compute limited is the other aspect. So we don't have a GPU that we can throw all of our data through

and run a big ML model on because we're limited by the power. We use the battery in the wheelchair, so we don't want to suck up too much of the range of the wheelchair through the battery. There are a lot of different things you need to balance. I mean, there's ease of use. There's the chair itself that you need to balance. Yes.

But cost and safety and agility and ease of use and power budget. Are there other criteria that go into designing a powered wheelchair like this or a control system like this? Yeah, there's a lot that goes into where we place the sensors is a big balancing act of keeping the sensors where they can see everything, but keeping them out of the way where they're not going to get damaged or where they're not in the way

of the user transferring out of the wheelchair. One thing that was interesting when we kind of first started looking at this is there's been a lot of university research projects and stuff that have, oh, we built an autonomous wheelchair that can drive around, which is really cool until you look at the picture

and realize that they put a several thousand dollar 3D LiDAR in the seat. So yes, the wheelchair drives around, but where's the user going to sit? And how are they going to afford it? Because you used a really expensive 3D LiDAR. So like, there's a huge balancing act between the, yes, everybody's like, oh, an autonomous wheelchair would be really cool. But when you actually look at what that would be and what that would take, it's not actually what the users end up needing at the end of the day. There are people for whom an autonomous wheelchair would be good. But taking away the autonomy probably is one of the things you have to balance is you want it to react safely and well, but you also don't want it to do... I guess I don't know what I'm saying. Maybe the question is, how do you And so there's a button on Lucy that lets you override it. And so if you need to go drive off a curb that we think is a little too high,

you can hit that button and drive right off that curb. Or if you need to get... Anytime you disagree with Lucy and say, nope, I need to do this thing. A great example of that is pushing open a door. So if the door doesn't have an auto door opener, you might need to actually run into the door to get it open. So you can hit the override button and do that. And so there's several different modes

and a lot of our users kind of customize that for how they use override. Some people drive with Lucy and override a majority of the time, but then just turn it on when they're in kind of tight situations. So that's another thing of keeping the human in control. We want them to be able to do what they need to do and not be limiting of what they can do. We don't want to tell them, no, you can't do that. It's just, hey, we don't think that's a good idea right now. But if you need to go do that to get across the street,

then go for it. How much of the system is open source? Not a ton. We have some open source work that we're doing with a few research partners. So there's been some grants we've gotten with Northwestern has been a big one where we're working on some open interfaces to use Lucy to drive wheelchairs, as well as

provide an interface for some of their researchers that are working more on the autonomy piece. We just saw a demo the other day where they've got a system where you hold forward on the joystick and it tries to center you up to get through doorways and stuff like that. So we've open sourced kind of that side for them to help do research and look into techniques. So that's actually a set of ROS packages that expose our data into ROS so that they can use that for control.

But the system itself, the code, is currently not open source, but we're kind of opening up pieces as we've seen opportunities to collaborate and give some of our... We've got some users that are very tech savvy that are working on different ways of driving. So we've given them some hooks

to plug their systems in to help control how they drive the wheelchair. So we're trying to, it doesn't make a lot of sense to open source all of it because with the hardware we have, it really is very specific to the hardware and you wouldn't be able to use it standalone. But where we've found good opportunities for working and collaborating with other folks, we're definitely open to open sourcing it. So if other people have things they'd like to do with it, be sure to let us know and have a conversation about that. How would they reach you?

They can reach out probably the best way is my email for lucystuff is kevin at lucy.com. And I can direct that to our group that is working on that interface. We'll leave that in the audio bits, but we're going to take that out of the transcript or you will get lots of spam. Sounds good. Anybody misses that and wants to contact Kevin, just email the show or hit the contact link on the show web page. So how did you, this sounds like a fantastic

job. How did you, how did you land here? Yeah. So the very first prototype of this system was built by my boss, who's Barry's brother, Jared. And it was built while Jared was a professor at the Colorado School of Mines while I was there, and he was in charge of the senior design program. And he was kind of working on this on the side just for Catherine. So we had what we called our SparkFun special prototype that was a bunch of Raspberry Pis and Arduinos

and off-the-shelf Ultrasonics hooked together. Jared had asked me to help on that when he was first working on it. I did some work while I was at Mines. Their original plan was to go talk to the wheelchair manufacturers

and say, hey, we've got this proof of concept. You guys should build this thing. They went and talked to the wheelchair manufacturers and they all said, no, not interested. And so I had by that point graduated and was working at Keysight for a year and then kind of got a call from Jared and Barry and they said, hey, we're starting a company. We're going to build this thing. And so I moved back to Colorado and joined on. So I've kind of been with it since it was a very early prototype

and then all the way through turned into a full product. And it has been a lot of fun. It's a great job. We've been able to work on a whole bunch of different hardware, see the impact of what we've done for a lot of people, and honestly, driving wheelchairs around the office a bunch is actually quite a bit of fun. So it's been a great job. Do you have anything you do specifically to test that's particularly fun? I'm imagining

wheelchair basketball. We don't do wheelchair basketball. No, our main testing is we do kind of a button seat for every release and every sprint where we all hop in the chairs and we have kind of a course around the office. We drive them through and we take full speed runs at our target we call Gandalf, so thou shall not pass or run him over. It's a mobile target, so when we hit it, it moves out of the way and we don't smash stuff. In our previous office, we had put numerous holes in drywall walls all over the place. So we've since learned that testing on actual walls is a pretty bad idea. We had a sign up that said, we crash wheelchairs so you don't have to above one of the holes. And then we go drive at the curb outside and we've got that. We also have

done stuff where we took a whole bunch of chairs, the whole team, everybody grabbed a wheelchair and we rode a mile up the street to go get lunch one day. So we go try and take them out in the world and actually do stuff with the wheelchairs. One of my funniest testing stories, unfortunately, I was not a part of this, but my brother also worked for us and Jared were out testing at a library and they went into the elevator with Ross in the chair, went upstairs, did some driving around, swapped who was in the chair, came back down and the person that was sitting by the elevator did the funniest double take as they came out with who was in the chair came back down and the person that was sitting by the elevator did the funniest double take as they came out with who was in the chair swapped.

So we've had all sorts of funny interactions with the public of having a bunch of folks that are swapping around who's in the wheelchair and some general goofing around. But it's actually, it's quite a bit of fun to go take these out. And it really helps us learn what our users are up against,

navigating the world from bumpy sidewalks to getting across the street in time. The world is not built for wheelchairs, and it really changes how you view the world when you try and drive one around for a bit. Do people treat you differently? Somewhat. A lot of people give you a really wide berth, which is interesting, or kind of let you go. We go test in the mall a lot because it's a big open kind of public space

that we can drive around and have a lot of different flooring and different other stuff. Just as people kind of give you a really wide berth around. So you definitely get treated a little differently in the wheelchair than you would just walking around normally. I mean, sure, you're in something that weighs several hundred pounds that could run over toes.

Yes. Yeah. Do people realize that? I don't think able-bodied people realize how heavy and powerful those chairs are. The powered ones especially, yeah. Yeah, the large chairs, one of our tests we have to do is tip them over to make sure that our tip detection works.

And actually, it usually takes like two people really leaning on a chair to even get the thing tipped over. Which is probably good. They weigh a lot. Yes, it's a feature that they weigh 300 pounds but then it's also makes certain things harder with them moving them into vehicles i imagine yes getting them into vehicles you need a ramp like you're not going to be able to even with multiple people you really can't pick up a chair and set it in a car what is the longest time you've been

in a chair without getting out? Probably just for a few hours. I mean, we'll drive around for maybe half a day. We've definitely done some testing where we've spent a good amount of time in the chair. So half a day driving around the office, kind of trying to live in it. Usually we end up doing that when there's some particular bug that we're trying to suss out that only happens once every few hours or whatever. And so we'll be in a chair for a good chunk of the day. I was just thinking how hard it would be if you didn't get to get up and stretch after that. But that's, let's see. So how much training does it usually take to get wheelchair skills? I mean, I've sat in one before and I think I might have run over my own feet.

Yeah, it really varies. So for folks that have, for users that have kind of normal motor control and normal cognitive skills, it can be pretty quick to figure out how to drive. But then the subtleties of how to approach and get through a door, what like the right line to take so that you don't hit it is something that like a clinician may work with someone training them on. All the way up to it can be a really hard learning process. There's folks that drive. So if you don't have motor control, you might be driving with a head array where you're moving your head to hit switches. Or you might be driving with a sip and puff where you control the wheelchair literally by sipping and blowing through a little straw that's hooked up to a pressure sensor.

And there's some solutions out there as well that let you drive with your eyes that do eye tracking. And learning to use those systems is, that's not a day or two, that's weeks or months of being in the chair and getting good at driving it. And so that's really, we've been to, there's a clinic here in Colorado called Craig Hospital that does, is like a spinal cord injury rehab center. And so they'll get folks that went from being fully functional, maybe they had a sporting accident and ended up paralyzed. And a big part of their visit is there is learning how to live their life. And part of that is in the wheelchair. And so they're learning, yeah, how to drive, how to get around, how to live their life. And part of that is in the wheelchair. And so

they're learning how to drive, how to get around, how to transfer in and out of the wheelchair, all that. So it's a huge variance kind of depending on the needs of the user. But with some of these drive methods, it can really be a long process. And that's one of the things that Lucy can really help with is that training process because you can put somebody in it, they can drive, but you don't have to worry about them running into something if they get it wrong. So it's actually a really helpful tool just for the driver training,

especially with kids and these alternative drives where you don't want, the cost of an accident during that training can be really high. So does Lucy work with the alternative drives? Yes, we do. So those kind of plug in the same as a joystick. They show up as just like you hit the button and it goes forward or goes backwards or turns. And so we plug in and work with most of those.

There's a mode of driving that we don't quite support right now that's called latch driving where, and this is pretty typical for a Sip and Puff user, where you actually say, okay, go forward, and then you don't have to continue providing that forward input. It just goes forward until you give it a stop command. So that one's a little bit more difficult to work with, but all of the switch, like the head array and stuff, we work with and we test with some of those pretty regularly. A lot of our chairs have these variants of these alternative drives on them. So we can test with those and experience driving. For a while, I had

a chair that had a joystick that you had a little thing that mounted it so you could drive with your chin. So you would use your chin on a joystick. So I think you've picked up like your Xbox controller and stick your chin on the joystick and try and navigate around your video game. It takes a little bit of getting used to for sure. Do you see many of those that are fully custom? I mean, there's more of a movement to use maker technology, adaptive design. There's a group called AT Makers. I think you sent me one about adaptive sports connection that helps people on a one-on-one basis to develop adaptive technologies. How much of that do you see as you are working on Lucy? We definitely see some of it.

So yeah, those adaptive sports people have a program called Go Baby Go. And it's a fantastic program where they customize those big wheels, little power toys for young kids to use to get around that are disabled. So they may customize the steering wheel so they can use it more easily or put some bumpers on it so they don't get their fingers pinched, that sort of stuff, so these kids can get around that can't otherwise walk. There's quite a bit of research that shows that for these kids that can't walk, giving them some method of getting around really can help improve how they learn about the world and everything.

The other side of that that we're working on is other ways to drive the wheelchair. So one of the things that is on our GitHub and is kind of open source is we're working on a digital interface for the wheelchairs that if you have Lucy, you can use off-the-shelf gaming controllers. Like you can use an Xbox controller

or PlayStation controller to drive your wheelchair. The one we're really excited about with that is Xbox has an adaptive controller that you can plug in all sorts of custom buttons and stuff into. It's got just a bunch of headphone jacks on it that you can hook up switches to to help people game. But so we're working on how you can hook that up to Lucy and let people drive with that. So instead of having to wait and go through the whole whatever's available from the wheelchair manufacturer for switches and stuff, you can use some of this tech that has been

developed for the gaming world to help control your wheelchair. You mentioned earlier documentation and safety. Is this a medical device? Is the FDA the people you have to get approval from, or is this not regulated? This is a medical device. So we are a FDA class one device. So that's where we have to kind of control the manufacturing. So the wheelchairs themselves are class two, so they have to go through the more rigorous approval process. And then we are an accessory to the wheelchair, so we're a class one device. So we do operate as though we're designing a medical device. We have a lot of quality control and design processes that have to be followed to check all the boxes

that the FDA needs checked for that. As well as a lot of those processes, like having requirements and doing testing are just a good practice after having done it a bunch for a medical device. I think I would have a hard time not doing that on consumer devices. It's the stuff you should be doing anyways.

Yeah, I was going to ask how it worked since you're an accessory to a medical device. I'm trying to think of other examples where it's a little odd. Does the FDA have a process for that or do they kind of just, this is a new medical device and you're in a different class because you're attaching to something that's in a different class?

There's a bunch of guidance and there was a little bit of prior. So the alt drives, for instance, I was talking about, the head arrays and stuff, those are usually not. In some cases, they're made by the wheelchair manufacturers, but they're often made by third parties. And so those are an example that there was kind of that prior art of it's an accessory to a wheelchair, and those are all class one devices as well. So we fit under that umbrella right now. How much real estate does this take up? I imagine part of the design challenge is adapting a suite of sensors to something that's already large and has its own sensors and its own.

I mean, they don't have like accessory bays for you to just put an electronic system in and rails to attach new sensors, right? Yeah, it's a big effort to get it to fit right. So that's where we've kind of had this pod level approach and we do the custom sheet metal for each model. But even with that, depending on how somebody's ordered a chair, the seating assembly may move differently. So like we have to keep it out of the way of the seat that can recline all the way back to, I don't know, it feels like you're upside down and you recline some of these seats all the way back to, I don't know, it feels like you're upside down and you recline some of these seats all the way back that folks use for pressure injury mitigation. So you actually kind of roll back and take the load off your skin to get blood flow back in. So there's a whole bunch of, there's a lot of spaces on the wheelchair that we have to be very

careful about what we put there and making everything small and compact so it fits in, keeping cables out of the way so they don't get pinched by the mechanisms and everything. So there's a lot of effort for every chair that we've designed for to get all the sensors in a place where they're out of the way and not getting crunched or run into. And that we can still see the world from them. You can put a sensor out of the way, but it might not be able to see what we need it to see.

Yeah, yeah, right. How often do you have to talk about cost-reducing the system? It's not an everyday conversation, but it's definitely something, it's less of a cost reduction, more of a complexity reduction. It's trying to take complexity out of the system so that it's easier to service or it's easier to install or maybe allows for less complex software. Like you get rid of a micro

controller, that's a great win. I now don't have to write code for that micro controller. So I think we think about more of kind of system optimization is cost is one axes on that, but there's a lot of other pieces that go into that. How long does it take for somebody to install one? These get installed often in the field by somebody that has... Our goal is to keep the number of tools down

that you need to install it with, that sort of stuff. So there's a lot of that optimization that goes in, but it's not necessarily just along the cost vector. That makes sense. Kevin, thank you so much for speaking with us. Do you have any thoughts you'd like to leave us with? Yeah, I have a thought, and this kind of came from the podcast you guys had recently

where they were talking about learning stuff from different programming languages. And I get some advice to everybody is to go learn something in a field adjacent to what you're working on. So if you typically write embedded code, go try and design a PCB, or maybe go learn a web framework, do some web programming. Or go learn Rust even, I know that's a sore topic here. But every programming language I've learned has taught me a new way to approach problem solving, even when I'm not programming in that language. And doing hardware and embedded software design, there's a lot of things I do different on hardware because of that. Because I've been the software developer that the hardware

was thrown over to and realized I didn't have the test points I needed or we didn't match the reference design. So then all the code I wrote on the dev board, now I have to make a whole bunch of changes to get it to work on the actual hardware. So I think there's a lot of, even if that's not something you're going to do regularly, is go learn something adjacent to what you're working on.

Go talk to the hardware engineer. Go talk to the web programmers. There's a whole bunch we can learn outside your immediate skill set, and it'll apply in ways that you wouldn't have thought possible when you started it. Our guest has been Kevin Lannan, Embedded Software Engineer at Lucy Mobility. Thanks, Kevin. Thank you.

Thank you to Christopher for producing and co-hosting. Thank you to our Patreon group for their support. Thank you to Memfault for sponsoring this show. And of course, thank you for listening. You can always contact us at show at embedded.fm or at the contact link on embedded.fm. And now I actually have two quotes to leave you with, one for you and one for me. The first one, Helen Keller, it has been said that life has treated me harshly. And sometimes I have complained in my heart because many pleasures of human experience have been withheld from me. If much has been denied me, much, very

much has been given me. And the second from Charles Schultz, happiness is a warm puppy.