Embedded - 415: Rolling Computers

Episode Date: June 2, 2022

Lead Solution Architect at Cymotive, Benny Meisels spoke with us about implementing embedded software security in cars. The discussion touches ECUs, IoT vehicles, threat and risk analysis, and how reverse engineering plays a role in security testing. Benny works at Cymotive (https://www.cymotive.com/). You can find him on LinkedIn benny-meisels or on Twitter @benny_meisels. Resources for automotive security: Automotive Security Research Group (ASRG) Upstream Security Hacking a VW Golf Power Steering ECU - Part 1 – Willem Melshing's Blog  Instrument Cluster (ICM) Simulator: ICSim on github Program | escar USA conference | Embedded Security in Cars Car in a box, also on github and Arduino based: A lower cost approximation of the Toyota PASTA:Portable Automotive Testbed with Adaptability  Ghost Peak: Practical Distance Reduction Attacks Against HRP UWB Ranging Framework Laptop  Transcript

Welcome to Embedded. I am Eliseo White, alongside Christopher White. We're talking to Benny Meisels this week about automotive security. And we don't mean car alarms. Hey, Benny. Thanks for joining us today. Thank you for having me on the show. Could you tell us about yourself as if we met at a large embedded systems conference? So I'm the lead solution architect at Cymotive.

We offer various cybersecurity products and services for securing vehicles in the automotive ecosystem. Previously, I was a security researcher and a manager for the security research team for onboard in-vehicle penetration testing at the company. And before that, I served for five years in the military as a security researcher. Are you more into the security side or the cars side? Security. That's my background. The car stuff was new when I joined the company. I believe you're familiar with the show, so you know Lightning Round is where we ask you short questions and want short answers. And if we're behaving ourselves, we won't ask why and how and everything like that. Are you ready? I am.

Which is better, canned bus or canned beans? Canned bus, definitely. What book would you like to write? I would like to write a C++ book in Hebrew. There aren't any good ones currently. When do you think cars will be really self-driving? I think within the next 10 years, we'll see commercial cars, as in trucks and taxis. But personal cars, I think, are much further out for mostly business reasons.

Do you have a favorite car? Not really. I actually don't own one either. Favorite fictional robot? Baymax from Big Hero 6. Cute and cuddly. Do you have a tip everyone should know? I think everyone who's doing some protocol research or working with embedded should learn MicroPython and CircuitPython to test protocols

and use Compiler Explorer to test assumptions on what their code will actually end up being like. Those are, setting aside lighting around, those are really good tips. The Python, I hadn't really thought about using MicroPython as a protocol tester, but that would be perfect.

I've been using a lot of Python recently with Uarts, and yeah, that would be really cool. I find that CircuitPython and MicroPython are really good for quick, rapid prototyping. And especially with protocols, especially if you have a one-off thing, it's a lot of times much easier than getting bored and bringing up whatever you need to work on it. Yeah. And it would be trivial to program a spy flash or something you have to do in manufacturing later, but you just put it together pretty quick into MicroPython or CircuitPython. Cool. So security. I didn't ask you what your favorite security hack is, but is there something that people do often that just makes you wonder how we get anything done ever?

You mean in security specifically? Yeah, or generally. I mean, generally is good too. I think the problem with security is that it's a never-ending process. And I actually had a discussion with some people in the office today about general IT security. And it's a question of threat models and risk management and how far do you really want to go? Because you could end up spending years on optimizing some specific security part in any system. But the question is, is it really worth it? The joke is, I guess, on vehicles, if we go back to classic vehicles, they are more secure than today's vehicles. But would you want to own any one of those?

Right. I mean, is that really true? Well, it's pretty hard to hack a carburetor over Bluetooth. Although it might get stolen for other reasons. Exactly. It might get stolen for other reasons, right? Exactly. It might get stolen for other reasons. So what difference does it make? I think there's a big expectation today that the car is more than just a transportation device.

I think many people assume that the moment you enter your car, it extends your phone and you're able to do anything you would be able to do while walking, just with your car, while you're driving. It's kind of like how our phones no longer are really just phones. They're not even just communication devices. I don't use mine as a phone very often. Yes, there's so much more. And we do treat our cars... As rolling computers. As rolling computers and an extension of our homes. And I think that will be the case even more so for the years to come.

So what's the worst thing I could do as a hacker on a car that is not secure? So I think many people would think about crashing it into a wall, remotely controlling it. Although that is possible, I don't think most hackers are that interested. I don't think there's that much money involved in crashing people into walls. I mean, unless you're 13 and have been playing far too much of that driving game that's just mean. Crashing into walls, I mean, I suppose assassination pays well, but I wouldn't think that that would be a goal for most people. So yeah, how do you make money by hacking cars?

So I think there's multiple ways. I think what's more common today is hacking to open features on the car, which you didn't pay for, or do some modification. And there's markets for that. Wait a minute, I want that. As a consumer, I remember we had a car that we couldn't do something in, and Chris got a different module that made it go faster or something. What? I don't remember that. The TT. I didn't do anything. Are you denying it for legal reasons? Oh, okay.

I think there's a line between modifications which make sense and the ones which can be dangerous. And I also understand the fact that the manufacturers are interested to sell you a service, for example, and they want you to pay for it. So hacking your way to get that service for free is counterproductive to what they're trying to do. So it makes sense that they'd like to try and protect from that. Well, and it's the trend of everything becoming a subscription

that if you have a rolling computer and you can do general things with it then you can enable and disable certain features and suddenly it's it's not that it's not capable of doing anything it's that the company would like you to pay them extra to to enable that thing and so yes a huge incentive to to bypass that right so that's true but i also think that once they figure out the way to sell you the services that make sense to sell that way, we won't feel about it the same way. In a sense, people don't feel Spotify is a way to take more money on things. It's useful. So the first, I guess, services will be weird.

And why are we paying for heated seats when there's a heater in the car? But eventually they will find services which make sense and maybe even follow you when you rent another car. So you could just have that service forever and wherever you show up, if you get a car from a specific brand, you'll have that service. Have a Ford subscription to having power door locks. And it would have your favorite stations and the volume that you prefer. Oh, that makes perfect sense. And how you want it. Your seat adjustments.

Yeah, that makes perfect sense. So I think they're mostly trying to figure that out. I would say, though, that what I'm afraid of and I think will end up happening is as we have a consolidation of the vehicle space and we're slowly going to see a consolidation of the main components in the vehicle space and we're slowly going to see a consolidation of the main components in the vehicle space, you're going to see very similar effects that we saw in the mobile

space. When we had our old dumb phones, they were so different one from another and they also didn't have so many features. So it wasn't so common to hear about some advanced attack or ransomware. But once we have hundreds of thousands or millions of cars running the same exact code, if you get the right bug in the right place, you can exploit this at scale and probably cause a mass ransomware attack. Where I think also, you're not going to ask for money from the individual user. You're going to ask for money from the corporation which built the car, which actually is able to pay. Aren't we close to that already?

I mean, I feel like Delphi supplies a huge portion of all components and Bosch and Magnet does a lot of manufacturing. How close are we to just the brand exists, but most of the components are from other suppliers, a few small number of suppliers. So there are tier ones which are very large, but I guess you'll also see more consolidation in the sense that the OEMs, the manufacturers of the vehicles, want to do more stuff in-house. Most of them want to try and compete with Tesla, and they understand that to be able to deploy features at such a rapid pace, you got to do more stuff in-house. You can't just be an integrator anymore.

I was going to say monoculture is usually bad, but now we have the OEMs like Ford and Tesla and we have the people who create the parts, which like Bosch and Tesla. And so it isn't entirely a monoculture, except that there are pieces that most of them use the same part underneath. And so attacking that part has a wider benefit than just being able to unlock one person's car.

And I think what you're saying, Benny, is that the major corporate, the major car manufacturers are moving to design those parts themselves instead of pay Bosch. So they might do that, or they might do a long-term agreement with one of these tier ones. So they secure, you know, the supply chain going forward and don't start new projects one at a time they might you know say we want this component for the next 10 years and secure the development of that in a more coordinated fashion that they used to is that dangerous too because now you've lined up a part that might have a vulnerability discovered in two years that you have 10 years worth this is an issue with cars in general.

And the way this is being solved is by deploying over-the-air updates. Most of the manufacturers are going into this year with new models, which support most of the components would probably support over-the-air updates at this point. I want to get back to over-the-air updates because, oh, there are so many ways to do that wrong. But you said tier one, you've said that twice now. And what does that mean? So the automotive ecosystem is a bit different maybe than some consumer electronics.

When you do consumer electronics, you might have a company in China producing for you, an ODM or something. But the car industry is much larger and the vehicle is much more complicated. So you have different levels of suppliers. You start with the OEM, the manufacturer. They buy complete components from a tier one. Tier one might develop one or more components. Many of them are known for specific components that they are the top of the market at developing, but they will buy specific modules or software from tier twos and tier threes. And you have this whole supply chain, which is very complex. When we find a bug in a system,

we don't actually necessarily know outright that it was the supplier who introduced that bug. You might find out that it was actually an SDK, which was bought from a third-party company, which was incorporated into a module, which the module was after incorporated into the larger ECU, the component. We spoke recently to Laura Abbott from Oxide about the trust zone and Cortex-M or Cortex as a whole and how important it is to be able to be sure that you are running the code you intend to run. As soon as you have over-the-air updates, that root of trust becomes kind of more of a tree of trusts and goes all over the place. As a developer, how do I deal with that sort of thing? So before you even write one line of code, you really need to have a good concept

of how your software is being verified, as in secure boot, and also how your updates are being done. I wouldn't necessarily say that it becomes a tree of trust because there are standard ways of doing this with cryptography, which don't require creating very complex structures. It doesn't make it foolproof, but if you follow best practices, you can set up an over-there update in a good way, which from my point of view is actually better than saying I won't update

and therefore I don't have the attack surface of over-their updates. That's interesting because I do usually think of over-their updates as being more vulnerable, but that does allow for patches of vulnerabilities as time goes on. So you mentioned before, cars I think these days in the US are on the road for about 12

years on average. In that kind of timeframe, the chances of not having a major vulnerability, when you have such a diverse type of code, you have Linux, you have embedded software, you probably even have at this point a web browser, which bugs are found in those on a regular basis. If you don't have over-the-air updates, you don't have a chance to fix any of that. You have to wait for the person to show up at a dealership. And even then, it really depends on the business model. You might not have an update for your system if you haven't paid or depending on where you live. So over-the-air updates are the only way that I see to long-term secure these kind of devices.

I was going to ask about if over-the-air truly meant over-the-air like Tesla does, or if it meant going into the dealer and having your ECM plugged into their computer. So you're saying truly, and is this Wi-Fi networking or is this cell modem? So in general, I think most manufacturers are going for cell modem. But I have seen discussions of using the Wi-Fi with the consent of the owner on their personal Wi-Fi as they might have a much faster internet connection or they might not have good coverage in their area. So I assume you're going to see both models, both cell and Wi-Fi. Okay, so back to earlier question of what can I do if I hack my car? I can get features. Okay, that's great. I can see how the car manufacturers don't want that.

I might be able to remote control drive into a wall, which, all right, there will be some people out there who enjoy that, but most people are more sensible that that's still wrong. What are people really doing? Are there more features that are being attacked beyond the monthly subscription models? So I think the most visible attacks, which we should care about today, are car theft. There's a lot of ways to steal cars using hacks. It used to be people broke the window, took out the ignition wires and ignited the vehicle. Hot wiring. Hot wiring! Hot wiring, yes.

But these days, many of the ways to steal cars are by figuring out how the keys work, by relay attacks, if you've heard of those, where you have a keyless entry system and you just use radio transmitters in order to extend the range of that device. And therefore, you could be further away from your vehicle and it shouldn't open. But because they use these devices in order to extend the range of the key, your car will open and they can steal. And that's what the normal key fob kind of mechanisms, right? Not necessarily the newer stuff that Apple and Tesla are doing with using your phone as a key, which is a whole different kind of...

Beans. Yeah. So this would be the mid term. You have classic car keys, which basically work when you press the button. Yeah. But these keyless systems, the idea is you show up next to your car, they open. Oh, okay. And the next generation is what is being developed by Apple and Google

using ultra-wideband, which is only available on some smartphones. And this is supposed to be much more secure. But there already is research showing that this can be circumvented with the right tooling. So I'm not an expert on this topic one way or another, but it doesn't seem that this has been solved at this point. Walking up to the car and having it open is very handy. You know, I don't necessarily need the keys in my hand. I walk up, I just open the door, and it's all good. It's really bad when I switch cars and forget to lock the door. Setting that aside, this seems like a battle between security and convenience, which is a constant battle,

where can the consumer or the developer make a difference in having that be truly secure? I think it's a bit much to ask the consumer to do the security here, considering he doesn't have the knowledge. But I think this is going to be a long-term effort from the industry to try and find out ways to stop this distance-shortening attacks. Because I don't think we're going to go back to not using keyless entry systems. It's kind of what people have gotten used to. So the only way we can go forward is by figuring out better ways of when these kind of attacks are being performed and preventing them. Do you think in

five years somebody's going to go back to having a physical key that you put into a slot and call that more secure because you can't spoof it? You'll always see people who are trying to do their own small thing, but this won't be at scale. You might decide that you want to mod your car to do that, but I they, as it stands now, is it a worse situation than older cars where you could walk up to them with a coat hanger or a flat thing and stick it down the window and pop the lock? I don't have the numbers, but I guess, you know, you deal with what you have now. And right now police is dealing with these kinds of attacks, mostly. Okay. When we have cell modems in the car, it's much easier to track them and locate them. And so doesn't that decrease theft already without worrying about the keyless entry parts?

So I guess this is a really good question of when you figure out that the car was stolen. And does the car have the feature of reporting, you know, either from your app or from something to report to the police that this has happened? Yeah, it doesn't take too long if it's a sophisticated thief for them to get it somewhere where they can do what they want to, yeah. Chris brought up the ways you can open a car if you lock yourself out, as well as if you're stealing it. And there have been incidences where you do lose your keys or somehow lock them in the car or have other instances where we truly do want to defeat the security. And so we have tow trucks and locksmiths who can get into our vehicles and houses on purpose. With the new keyless entries, if my phone dies, am I not going to be able to use my car? Or if I somehow lock my phone in the car and my kid, is it going to be catastrophic? So I assume we're not going to a point where you only have one set of keys in your phone and you don't have any keys anymore. But I guess just like today, if you could stick all your keys in the car by mistake, it might happen.

But I'm guessing you're always going to want to have that backup key or maybe share your digital key with your partner. And that way you'll be able to get into the car one way or another. I also, I recall that there is a way, even if the phone is off, possibly, to open, but I'm not sure about that. Yeah, I think some of them have NFC too. And you can always call the manufacturer. This has happened.

You call the manufacturer and they unlock it for you? Yes. See, that's actually reassuring. And then you social engineer them to unlock somebody else's code. Exactly. Security comes in many forms. I guess also some of that could be done through an online portal. So as long as you have a way to log in on a PC, you might be able

to do some unlocking from there. So what are the surface of these attacks? I mean, what are you looking at when you're looking at the security of vehicles? So one thing that I've learned from the last couple of years is that looking just at the single component in the car is probably not the right way to be looking at the problem. We do work on individual ECUs, but when we look at the functional level, when we want to have a look at a specific function, for example, we said remote locking, unlocking. If we only look at the software running on the ECU,

which actually unlocks the door, we're going to miss a lot of what's going on. And I mean also in the vehicle, but also on the servers. So what we like to do, if we can, is actually research this function end-to-end from the click

that you have on your cell phone, the app itself, through the servers in the backend, which are the OEM servers, or maybe a third-party server, look at what's going on there, look at what is going back out to the car, and look at the software in the car. And when you understand this at a high level, then you can understand where it's going to go wrong.

Sure, because it's always the cracks. It's always the places where things meet that things go wrong, because people don't understand the other piece of it. The embedded software engineers don't understand the security on the server and so they leave in a back door and there's or they just don't check the right thing or they allow you to somehow bypass it. And even the mobile developers going up to the server, there's a lack of communication, and those cracks are where the bugs will be found. I agree, but it's even more than that. It's the mental model, which is different between developers,

and it's also the mental model of specifically, even if you're sitting in the same room discussing, because they come from different backgrounds, and they see the function through a different perspective, they might not think about the same issues. You know, everyone's thinking, did I do my job well compared to what I'm expected to do on my side? And they're not asking, is the overall project

in the right direction from a security perspective? Again, there are cases which are different, but you get the point. And what's the way to overcome that? That sounds like an organizational thing rather than a technology thing. So I think this requires looking at security through different perspectives.

You have to look at the base software running on the ECU, whatever it is, if it's an operating system or if you have a hypervisor or Linux, but you also have to look at things on software running on the ECU, whatever it is, if it's an operating system, or if you have a hypervisor or Linux, but you also have to look at things on the functional level, you have to do cross examination of these functions, wherever they're going. And you got to figure out how you secure them both on the individual component you're on right now, but also end to end. So for example, if we're talking about cryptography, you have to have that cryptography work from the first point where the command for this remote unlock enters the system up until the vehicle.

And you can't have it split up in a way where each node is in charge only of itself. You need a systems perspective. Exactly. I mean, somebody also needs to be focused on getting the pieces done, but somebody also, that person probably cannot see the whole system. They can see their part of it, which is great because they need to do their part of it. But the systems perspective as a security, as a, I mean, there are lots of system problems, cost, integration, schedules, and security. Agreed. I think this is also something where we spoke about over the updates before, but this changes everything.

Because what you said now, it has been traditionally focused on hitting the start of production date for a vehicle. That's the day where everything has to be aligned, everything has to be perfect. And if you miss that date, you won't enter the vehicle at all. But when you have these updates, we can actually release content features, but also security upgrades later on. We don't have to take the decision that we're not going to invest in something. We can deprioritize it and get to it later over the life cycle of the vehicle. Oh, yes. This is why anytime you open something electronic, it has to do a firmware update first. Looking forward to unboxing my car and having three days worth of updates.

There's a risk, too, that people won't update, right? worth of updates. There's a risk, too, that people won't update, right? Oh, yeah. I often click don't update. I don't want you to change everything. Yeah, I think by law, they're required in many places to ask you if you want to upgrade.

But if they entice you with new features, then you're going to do the update. People who are, let's call them hackers, people who are intentionally trying to break the security for their own benefits, they have infinite time. They have 12 years to break the security, or however long you mentioned, or however long it is. But as a developer, I have to hit the ship date.

And as a company, investing more resources in a car that is 12 years old is just not feasible. How do you balance all of that, that the hackers have infinite time, but the developers don't. So I think this is part of the understanding that the engineering of the vehicle doesn't end when it hits the street. And in a sense, you need to have a long-term plan of if you really expect the vehicle to be supported for the next 12 years, you need to understand what that means. You say you get five years of feature updates and then you get another seven years of security updates. This is what we're going to commit to. This is what we can do. You got to think about, for example,

updating a Linux kernel at some point because at some point it won't be maintained anymore. But I don't think it's necessarily such a big deal if you in advance commit to producing the same vehicle or the same platform for many years. Because by the time you're going to end producing that specific vehicle, it's going to be a couple of years into its existence. So it's not just that you make one batch and you're done. You're going to be producing a very similar vehicle for multiple years. So this is a worthwhile investment. You just made the problem a million times worse by

having multiple versions and being able to have a code base that can support tweaks in the hardware and firmware update for the past version of the car. I mean, this is a versioning nightmare. So I don't think necessarily all components are going to be treated the same. But if you think about it, the same thing is true for our phones. And we're seeing gradually that our phones are getting more updates. The latest Google phones get more updates than the previous ones. And they're doing that by figuring out which components need to be updated in which way and how they can make it easier. So I think the same thing is going to happen in vehicles.

Much of the software will become standard. And you'll see that you don't really care what is running under the hood if it's not a user-facing feature. So as long as that code is not business-wise worth doing yourself, you're going to see much more standardized code. It's going to be maintained, whether by open source or by companies. And it's going to be worthwhile, the investment of keeping it up to date, both on a functional level, but also on a security level. What's changing in the architecture of vehicle, for lack of a better word, networks or electronics, in the old days, say, before all of Bluetooth and all this stuff? You know, it was lots of very not sophisticated processors, maybe connected by, I think, CAN bus and stuff in large, complicated wiring harnesses.

Is that persisting and new technologies being built on top of that? Or is the entire infrastructure being replaced by things that are more modern? So there's different takes on this and different OEMs are doing this differently. CAN bus is not really going anywhere. There's actually a new standard in process now called CAN Excel, which is going to be even faster.

And I think you can even put full IP packets on top of it. So that will be very different. But you also have Ethernet now in cars. The physical medium is a bit different, but the actual Ethernet stack is very similar. And you have also Ethernet in the sense that it's going to be a multi-drop Ethernet,

which is similar to CAN bus. It's going to be a bus version of Ethernet. Some of the OEMs are going for that. That's more of the internal network. When we look outside, we have 5G, we have vehicle communication between vehicles, vehicle to vehicle, or vehicle to everything.

You also have the charging port on the electric vehicle, which is also using power line communications and vehicle to grid communication, which should enable plug and charge in the sense you can come up to the charging station, plug in the charging gun, and it will just start charging automatically since you have already some agreement with a provider, similar to your phone roaming when you go between states or countries. Okay. So yes, that's advancing.

What is the virus in my car going to look like? That's a great question. I personally feel that the first generation is just going to be mostly bash scripts on Linux, because much of the way things are set up today, you don't really need to create a sophisticated C-based binary. Maybe if you want to go to the more embedded side, you do that. But I think many of the interesting things are available on high-performance ECUs. And this is also maybe referring to the previous question. So things that are changing is that you're seeing more and more high-performance chips,

which are running Linux or Linux-like operating systems. You have containers in cars. You have hypervisors in cars. You have virtual machines in cars. And at the same time, you still have the same embedded processes you've always had in order to ensure safety and real-time functionality. So that's a bit of a mixture of things. You also have new concepts of how a network is built.

If it used to be more of a flat network, these days you have a gateway, which is similar to what you would think of as a router or a firewall between different buses. And you have more security boundaries than you used to. Physical access is still going to defeat most of the security like it does in many embedded systems, or is physical access no longer as critical as wireless access? So the question is, at what level of physical access? If you're talking about just connecting to the OBD port,

in many cases today, that's not going to give you much because it's locked behind some sort of a security mechanism. If you start opening up ECUs, you'll get more access and some ECUs are more vulnerable and others will require you to do glitching attacks, which are much more complicated in order to get anything from them. So there are the attacks where I'm plugged into the OBD. There are the attacks where I open ECU, which is a big deal. There are attacks from my computer remotely. And then there are the class of attacks that are, you are near the car, possibly touching the car, but you are not in the car. You mentioned the key security, but there are other ways to interfere with the car being near it, aren't there? I guess it depends on the design.

I guess if the Bluetooth or Wi-Fi are on for some reason, then you might be able to interact with those. But I'm guessing if nobody is actually in the car, most of the time this will be off to save energy. I guess you could also physically try to remove an exterior piece of the vehicle, trying to get to an ECU which is near the boundary of the vehicle, that might be a way in? If I'm a developer and my company has a systems engineer who looks at security, but perhaps

they have a lot to do, can I just use the MISRA standard for compiling and be on my way and say, this is secure enough? Or do I have to think harder about it? So MISRA is a standard for automotive, for coding, specifically C and C++. The C standard is quite up to date. Last time I checked, the C++ one is not and is actively being worked on. But this only covers coding. So this is an important step, but this doesn't cover everything we need from a security perspective from a developer.

We found a lot of bugs in code, which is 100% MISR compliant, but it doesn't take into account the system's perspective. So you can say, you know, the coder is only in charge of his code. But at least the way I see it is you got to understand the system and you got to protect your code from the system itself. Are there guidelines to doing this? So I'm not aware of anything informal, but a lot of this is about working correctly with protocols. I think most of the larger companies in this segment have their own standards and best practices. But a lot of this is just learning over time and improving as you go. I think that would be the main thing. But you also need to have the right kind of personnel, which are proficient in each one of these cases, both at the system level,

electrical engineering, microcontrollers, but also at a higher level of programming. How can someone learn more about or get into automotive security? So I have some recommendations here. So first of all, there is an organization called ASRG, the Automotive Security Research Group. Nonprofit, they have a website, they have webinars up and a Slack channel. There's a lot of discussion going over there. They're slowly building up a lot of open access information. If you want to just learn about what's actually going on in the security incidents, so Upstream Security has very good reports on a yearly basis, and they also have some incident tracking on their website. There is some specific research from this year, which I think is the best way to end-to-end learn about ECU reversing by Willem Melsing. This was, I believe, a power steering system,

which he has a four-part blog on, which I suggest people read if they want to understand how to apply their skills. And then you also have some open-source tooling and testbeds you can look at. For example, you have the ICSIM simulator, which is a simulator for instrument cluster. So you can play around with Canvas a bit. You don't need any hardware. If you do have some time and you want to spend some money, you can actually

build yourself a car in a box. Specifically, Ian Tabor, Mintinetinet has a car in a box called Value Pasta Auto. It's based on a design by Toyota, which he did a cost down for. We actually built one over the last couple of weeks in the office and we've been playing with it. So I think that one is a great one if you have some money to spend and want to learn how to play with these protocols. Yeah, I think those are very good resources to start with. I'll make sure those go in the show notes. What you're saying, I mean, every time we talk to a security expert, I just get so discouraged that there's not a way I can make this work.

It's not just that it's a team sport and I have to have a lot of people contributing and someone checking over this. It's just, it doesn't matter what I do. Someone will always hack it. Do you have any advice for getting over that sort of cynicism? So I don't think it's necessarily the thought of, will it be hacked? The question is, what will be the impact think it's necessarily the thought of, will it be hacked?

The question is, what will be the impact if it's hacked? And also, how easy is it to do? And there are standard methodologies in order to quantify this, I guess, in money, which is usually the main part. So if you get it to the point where the impact to your business is low enough that at this point, it's not worth spending more money, you're at a good point, probably. In automotive, you have these kind of processes set up in order to do this. So you make sure that you're at the point where you want to be from the business perspective. And then I will say good engineering goes a lot a long way. If you cannot explain to yourself why your system operates on a functional level, then there probably will be more security issues.

If you understand very well why you're doing your software update in a reliable fashion, reliability has a good connection overall to security. In a sense, you're doing a checksum on an update. You might be doing this for reliability, but this is also, it's not foolproof for security, but this is also useful for security. And then you go from a checksum to a signature, and now you have better security and reliability. And it gives you a reason beyond, I have to do it, to I want to do it. Yes, exactly. I have some listener questions

since we're getting into the specifics. Many people asked about John Deere and bricking tractors. Should there be kill switches? Is that a good idea or a bad idea? So I'm going to speak on a technical level. I think moral level is a bit not relevant for this discussion at least that we're having um i really think this can backfire i've seen these

kind of mechanisms before in vehicles which on on paper had very good cryptography associated with them and so on but they can be circumvented and in very weird ways. And usually these mechanisms are extremely strong. As you said, you can brick a tractor. So I wouldn't put this in a vehicle. I think it's a bad idea. I think it would be better, at least on a local level, maybe to leave an interface for tracking for police with the consent of the owner of the vehicle. But I wouldn't go as far as to brick remotely vehicles. I think that's a bit stepping over the boundaries. Still picking on John Deere because they've been in the news.

Security, we've talked about how manufacturers would like a subscribe model like John Deere, and that goes against the right to repair movement. Or ownership. Or true ownership. And is there a way to balance the right to repair with the man getting his money well before you answer i would also i'd rephrase that as there's a balance i think between allowing the manufacturer to make changes to something you've already bought and ownership right because what we're talking about is over-the-air updates which will be necessary to keep things secure but they also allow the manufacturer to modify your vehicle in small and large ways um then do you truly own it at that point?

Anyway, sorry. Sure. I think, as you mentioned, the major issue here is more IP and safety and the fact that you want to update the vehicle, not really security. But I would say that I think there is an option to balance this. You've seen over the last year Framework do a different type of product for laptops. I think if a manufacturer was very serious about it, they could figure out the business model which would support this. I don't think that you'll see any of the large OEMs going for this, though, because there's not enough pressure yet.

At some point, there's not enough pressure yet. At some point, there will be enough pressure. And then some small startup will come and decide that they're producing the new open car. It will be very hard to do, but they will try to do it. And it will take a couple of years and you'll see some competing option. I just don't think this is what most people are interested in right now. That might change in the future. But I would be very happy to see that. An open car, one that you could read all of its code? So I don't know if all of its code, but I'd say all the relevant code for maintaining it

and making sure that it works for a long time. And I also don't think this is bad for security. In a sense, I don't think a good security model shouldn't assume that the attacker doesn't know what code is running on the device. What are some other good security models tips? I mean, A, you should assume a white box hacker that they can see inside the code and they know what's going on. What are some other things that are good to remember about security models? So never trust the hardware.

And you must know that from the functional side, but this is doubly true for security. Never trust user manuals, never trust anything which is on a specification. For some reason, things are always very nice on paper, but then they meet reality. Re-evaluate at regular intervals what is possible in the sense of what kind of capabilities attackers have. Because, for example, if we speak about passwords, over time, newer algorithms are out, stronger computers are out. The security advice for passwords a couple of years ago is not the same as it is today. Hopefully, we will be done with passwords at some point,

but the point is similar for many other mechanisms. That goes back to having to maintain these things for a decade or more. I mean, realistically, two decades by the time you start making the product and then finish, the new security features on a car that is older than the one you have because you updated to have faster for the new security and now you have to maintain both? Sorry, that's probably not an answerable question. I think that's the real challenge here. And I think this is what we're going to have to figure out in the coming years of how we do this better. I guess you could design the vehicle in advance with extra performance. You could think about, for example, if we look at quantum security, which I am not an expert at,

but everyone is convinced that we should already be thinking about it today before it becomes a problem. So I would expect that much of the question is, what threats do we think will be relevant over the lifetime of the vehicle? And what can we do today at a reasonable point to prepare for those? So when we need to address those, we are able to. Reasonable is a tough question there, because you do need, it's a judgment call of, we need to be better at security than all of our similarly priced competitors so that people steal those cars and not ours. Well, who's going to steal a 10-year-old car? But I think, well, just from a higher level point of view, security of cars versus security of other things, you have to balance that too, right? You can't say, well, this needs to be as secure as an airliner that's carrying 350 people. Evaluating what the threats are that a vehicle can pose or a collection of vehicles, I think, is important too.

Yeah, collection's important. I agree. I'd also say that it's still easier to get vehicle parts than it probably is to get plane parts. So that's also a large part of the story. Going back to listener questions, Exploding Lemur asks, how should Honda approach fixing their replay vulnerable key fobs? So I'm assuming this is referring to the keyless entry systems we mentioned earlier. So I mentioned before, the industry is migrating to a newer technology called Ultra Wideband.

It's supposed to be more secure. I'm not an expert on this, but there was an attack released called GhostBeak, which has shown that there is ways to shorten the range here. The idea here, if I understood correctly, is that you measure the speed of the communication in a sense, how fast you get a response. You can get a feeling where the person is in relation to the car. And it's supposed to be harder to spoof this compared to Bluetooth Low Energy and other RF interfaces. So I hope that over time they'll figure this out. But I guess this is a question also for physicists and not just for security researchers.

So I'm going to restate that question a little bit. Now that Honda has a replay vulnerable key fob and it's wide in the marketplace, what should their approach be? I mean, one approach is to not do anything and get sued. But what else can they do? Is it a matter of replacing every single entry with this ultra wide band more secure? That's going to be a very expensive retrofit. Yeah, that doesn't sound very feasible. No. I'm not sure, actually.

I think this really depends on the technical details here and to understand if there is something that can be done at the code level and maybe it's possible to maybe not avoid this, but make it harder. But it really would require some more details here to understand. Andre said that he's

seen some, that NXP has some CAN transceiver chips that detect spoofing and will take ECUs off the bus if they detect it. Have you seen or worked with chips like this? So I can't really comment on what we're doing in this space. But on a personal level, I think hardware is definitely part of the solution here. There's many things that are expensive performance-wise or impossible to do in software. So there's a bunch of vendors in this space

which are trying to find hardware solutions or combined hardware and software solutions. And I think these are definitely necessary because Canvas in itself is not very secure. Yeah, I mean, you can put a couple of wires onto your OBD port and see a whole bunch of stuff that's flying by on the CAN bus. I had to do that for work. And it was actually very eye-opening to me because I hadn't really done anything like that before. All this telemetry coming across there, and I assume other things I could inject

in there to do things I probably don't want to do. But yeah, it's kind of remarkable. But the OBD port is usually firewalled, so you can't inject that much. So we mentioned that this depends on the generation of vehicle. Newer vehicles tend to have a stronger firewall there. But then again, if you're going to physically access and connect yourself to a specific ECU or behind that gateway, then you will be able to do more.

Is there anything you are looking forward to in the future for how security will change? So things are getting better overall. When I started, it was much more common to find buffer overflows, like simple ones and all these kind of very simple security issues. Things are becoming more complex. I think there is a long term. This started a long time ago, but a long term shift to open source, reusing code.

We said over there, updates, consolidation, less reinventing of the wheel. Architecture-wise, we're seeing more ARM cores, which are a bit easier to use with tooling. So it used to be much more PowerPC, V850s, Tricore, which are very automotive-focused cores. But you're seeing more standardized ARM. I'd say that we're getting better tools. There's much more simulation tools than there used to be. And there also are better ideas of how to do security. So slowly, a lot of ideas also are migrating from the IT world into the vehicle.

And these were not possible previously. Because the vehicles weren't computers. I think they were computers, but they were not just as strong as they are today, not performant enough. They were 80s computers. Now they're like 90s computers. Well, some of them aren't.

That's true. Some of them are pretty advanced. You've mentioned security overflows and definitely input uh and the the areas between pieces of the system as good attack surfaces are there other good attack surfaces that i should be aware of as a developer or even be aware of as a security researcher? So I think besides what we spoke about, the internal network, I think one of the places or two of the places which should be in focus

is both the connectivity, the telematics units, which has quite a large attack surface from the network coming in from the cellular network, and the infotainment, because you have all of these apps. And of course, this depends on the OEM, but there's a lot of different types of apps running there. Maybe there's even a store. Maybe you can even download an app developed by a third party. So there's quite a bit of attack surface over there, which is really worth figuring out how you secure it.

Do you sandbox it? Do you review the implementation of the modem code running on your device? I was going to say, who cares about the infotainment system? But if you made the screen flash and the radio be super loud, I would probably have a very difficult time driving, especially if you made the screen flash. I agree, but you'd also have a foot in the door in the vehicle and you would be

able to communicate and possibly attack more critical functions in the system. Shouldn't there be like a hard firewall between infotainment and functionality of the vehicle as a vehicle? Yeah, the infotainment should be taped to the dash, and that should be the interface's tape. I'm just going to tape my iPhone to the dash. So there usually is, at this point, a serious firewall. But you want to know, for example, if there's a seatbelt which isn't fastened or a door is open. And many times the ECU, which will be in charge of displaying that, is no longer the dashboard, but rather the infotainment.

So that information has to go there. Therefore, there is some information being passed backwards and forwards. And worse to apps, right? Now you have a phone app for your car and it'll tell you if your door is locked and you can lock it remotely or whatever. Which that seems like the scariest attack surface to me is now exposing internal

functions of the car to the internet. Well, there are more internal functions. Like, I've wondered why we can't open and close our garage door from our phone app, because that's all part of a system to me in my head. You know, it's garage and car are the same. You can. Should you be able to open a garage door from somebody's phone that seems like a... Through their car. Through their car.

That seems like a huge security risk to your house. I figured that it makes more sense to do that through a specific service, which is supposed to tie all these things in together. And maybe then it's a bit better than just putting all your eggs in one basket. But I got to think about it. Everything needs the security piece, and it's easy to circumvent it when you just aren't thinking. You're just trying to make things convenient. Well, I think it's hard for us as developers.

And you've asked this question multiple times in different ways but and i think benny has tried to explain that the fact of security is it's not you've done security and you go on vacation it's security is an ongoing process where somebody tries to break what you've done and you figure out how to correct for that and then somebody comes up with a new way and you correct for that and that just goes on forever and as long as people want things they will want to break into things and that's just an ongoing battle and i think that's the truth of the matter i'd ask you the same thing about other engineering processes when do you know that your system is performant enough when should you stop optimizing well when I made children's toys, we

shipped a masked ROM and you couldn't ever change it again. That was the best. You never had to do maintenance when you have large consumer, like, un-updatable systems. Then it's done. But

for more updatable things, it often feels never quite done. There's always at least one bug. Hey, half the time I've been on products, we've shipped them before the firmware was done. They're on the shelf. And then you have to do firmware update. Yeah. No, you're right.

That sort of model of continuous improvement has bled into other things. And I think that's just the way life is going to be for developers. Benny, do you have any thoughts you'd like to leave us with? So if anyone's interested to learn more about Canvas and automotive security, we have actually a session coming up as part of the SCAR conference, which is an automotive security conference. I think in general, it's very interesting to hear about some of the research going on there. I think that would be it.

All right. I'll make sure that that is in the show notes. It has been wonderful to talk to you. Our guest has been Benny Meisels, lead solution architect at Scimotive Technologies. Thanks, Benny. Thank you for having me. Thank you to Christopher for producing and co-hosting. And thank you for listening. You can always contact us at show.embedded.fm

or at the contact link on Embedded FM, which is where our show notes will be as well as a transcript. And now a quote to leave you with from Mario Andretti. If everything seems under control, you're just not going fast enough.