The Changelog: Software Development, Open Source - Reddit goes dark, Lemmy lights up, OpenObserve, some blogging myths & Jefro on Automotive Linux (News)

Episode Date: June 12, 2023

Reddit goes dark as subreddits protest, Lemmy lights up as disillusioned redditors turn to the fediverse, OpenObserve is a cloud native observability platform, Julia Evans dispels some myths about blogging & Red Hat's Jeffrey “Jefro” Osier-Mixon tells Adam and Jerod all about Automotive Linux at Open Source Summit NA.

What up, nerds? I'm Jared, and this is Changelog News for the week of Monday, June 12th, 2023. This week's episode features one more interview from Open Source Summit, where Adam and I talk with Red Hat's Jeffro all about automotive Linux. So stick around past the outro if that interests you. Okay, let's get into the news. Last week, we covered how Reddit's API overpricing changes caused its third-party devs to announce closures, which caused many of its mods and users to revolt.

Later in the week, Reddit CEO Steve Huffman held a dumpster fire of an AMA, doubling down on the pricing and on his treatment of beloved Apollo dev Christian Selig. This week, the revolt began in earnest. And boy did it. Hello darkness, my old friend. At the time of publishing, over 8,000 unique subreddits have gone dark, representing almost 30,000 moderators for a subscribed user count of 2.7 billion. It's been so bad for Reddit, in fact, that the site and API was down for multiple hours today. That is one big pilot. My word, it's possible that they never fully recover from this.

I've heard it said that history doesn't repeat itself, but it often rhymes. Reddit's implosion has many people looking elsewhere for their not-so-centralized Reddit replacement. Enter Lemmy, a link aggregator for the Fediverse. Yes, the Fediverse strikes again. Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized in the communities, so it is easy to subscribe to topics that you're interested in and ignore others.

Can Lemmy do for disillusioned Redditors what Macedon did during the Twitter exodus? Time will tell, but for now, we'll link you to a guide to finding Lemmy communities and an awesome list of Lemmy instances to choose from. Up next, OpenObserve is a cloud-native observability platform built specifically for logs, metrics, traces, and analytics designed to work at petabyte scale. According to its creators, quote, it's very simple and easy to operate as opposed to Elasticsearch, which requires a couple dozen knobs to understand and tune. With OpenObserve, you can get up and running in under two minutes. It's a drop-in replacement for Elasticsearch if you're just ingesting data using APIs and searching using Kibana. Kibana is not supported nor required with OpenObserve.

OpenObserve provides its own UI, which does not require separate installation, unlike Kibana. End quote. An interesting offering indeed. Here's a couple choice quotes from the comment section. User GetTutaChapa says, quote, I just tried this three days ago. As someone running the home lab and hadn't set up logging yet, it was a great find. I didn't have to learn and combine three plus log technologies. It's just a single all-in-one monitoring server with web UI, dashboards, log, filtering slash search, etc. RAM usage of the Docker container was under 100 megabytes, end quote.

And user Surge Axe says, quote, interesting product. Thank you for the effort. Definitely want to give it a try. For me, though, setting up a system is not the primary pain point today. For what it's worth, signing up for a cloud service is not hard. The problem starts at the ingestion point, end quote. Let's do some sponsored news. Do your users bear the burden of integrating your APIs? That's not great. Or maybe you're manually updating and maintaining a laundry list of client libraries. That's a pain.

Here's a better idea. Build your APIs using industry standards like OpenAPI 3, Postman, etc. and use Speakeasy's pipeline to create idiomatic SDKs for all of the most popular languages and frameworks right out of the box. Here's the best part. Their pipeline operates as part of your CICD. So when your API changes, your SDKs change with it. It even automates publishing to your favorite package managers like PyPy, NPM, Maven, etc. If that sounds like a better world to you, head over to speakeasyapi.dev and join the betas today. Once again, that's speakeasyapi.dev. There's a link in your show notes. Check it out. Julia Evans is one of the most successful developer bloggers out there, so when she decides to write up some myths about blogging,

it's worth paying attention to. Here are a few myths that she debunks in the linked post. need to explain every concept. Myth. Page views matter. Myth. More material is always better. She goes into the details as to why each of these are a myth, and here's a good contrarian one from the end. Myth. Everyone should blog. Quote, blogging isn't for everyone. Tons of amazing developers don't have blogs or personal websites at all. I write because it's fun for me and it helps me organize my thoughts. That is the news for now. On Wednesday, we go deep on pass keys with Anna Poblitz, the head of passwordless at 1Password. And on Friday's talk show, Matt Reier joins us for what will surely be a ridiculously good time. Have a great week, share changelog news with friends who might dig it, and I'll talk to you again real soon. Oh, and if you have some more time, stick around for our conversation

about Automotive Linux. It's a good one. So we're here with Jeffrey Ogier-Mixon from Red Hat. Automotive Linux. I'm so intrigued by this. I know nothing of it. So there's a lot going on in automotive right now with relation to open source. And one thing I want to start out saying is that there actually has been a lot going on

in open source in automotive for quite a while. Okay. About, it was about eight or nine years ago, I guess, there was a project started within the Linux Foundation called Automotive Grade Linux. Okay. And AGL has been very popular. There are a couple of hundred members, I believe. And Red Hat is a member.

We're a bronze member of AGL. AGL has been um very uh influential uh and it's uh it's i think probably the best example of uh an embedded style design for specifically geared forward toward automotive and it's deployed in production cars it is yes toyota has been producing it in production cars. Okay. And I believe there's a couple other manufacturers, too, who have. Okay. But your thing at Red Hat, you're in that AGL group, but you have something else going on, too, as well?

We do. We try to collaborate with the existing communities whenever possible. But the work that we're doing is slightly different. It's basically approaching the vehicle in a different way from the traditional embedded style mechanisms. Normally in a current car, you have these, they're called ECUs, electronic control units. And each one of them is kind of a separate black box that takes input and does output. And sometimes they do some very, very complicated things.

Like the head unit might be considered one ECU. Or the unit that drives the dashboard. The IVI units are sometimes very sophisticated, particularly if there's gaming units in the backseat. Yeah. And that goes all the way down to disc drive, or disc drive, to brake controllers. Sorry, I was just talking to somebody about disc drive controllers. The discs and the brakes. Yeah, disc brake controllers, ignition systems, body control, charging systems now because

of the electric vehicles are becoming very popular. Yeah. So there's a whole lot of different computers inside a car. And the concept that we are starting to approach and that a lot of companies are is more around what's emerging is the name of it is around software-defined vehicles. Okay. It's not to say that AGL isn't a software-defined vehicle because, of course, it's software that's specifically designed for a vehicle. Right. But the approach that SDVs do is to consolidate a lot of these workloads into one larger computer and then to control those individual workloads, usually with containers or possibly with virtual machines to provide some separation.

And there's a lot of challenges with this because every single workload or activity that happens inside a car, they all have different functional safety requirements. Because some of them, like if your disc brake system is obviously, that's the highest. Yeah. It's called ASIL-D for Delta. That one has to work correctly every time. Yeah, yeah, exactly. So the way I think about it is that there's four different levels

of automotive safety integrity levels, ASIL. Okay. A and B are pretty lightweight. If something breaks, that means you're annoyed. If something that's ASIL level C breaks, that means that you're stuck on the side of the road. If something that's ASA level C breaks, that means that you're stuck on the side of the road. If something that's ASA level D breaks, it means that you might have to have a trip to the hospital.

Okay. Or something along those lines. Right. You know, it's just kind of a rough way. A and B is like, ah, my window won't roll down. And then C is like, ah, the car won't start. Car won't start, maybe, or the is is is less functional or it has some kind of a problem and then d is brakes and doesn't run well yeah explosions that's d

explosions would probably usually fall under under acd so all the containers and vms in a car so yeah that was that was the uh yeah that was the origin of the of the start of this is that these workloads all need to be separated because they all have different levels of safety requirements. So there are efforts right now, particularly around Zen and KVM, to do hypervisors, open source hypervisors, that can separate workloads enough to be able to certify them as functionally safe, in the same way that a company would certify one of these black boxes as functionally safe, with the advantage that it's software defined so that it

can possibly be updated in the field, so that it can be orchestrated by some of the amazing orchestration tools that have been developed through DevOps over the last 10 years that currently are not available when everything is a black box. So one thing I've heard, I'm not sure if it's true or not, is that a challenge of many car manufacturers is all these different systems come from different vendors. They're kind of put together like Lego, and so they have different software stacks. They talk different vendors. They're kind of put together like Lego.

And so they have different software stacks. They talk different languages. There's this mishmash of things. And, of course, then they have certain new car companies or integrated things where one company writes all the software across. Is the automotive Linux stuff that you guys are doing with this whole data center style VM thing, is that helping the circumstance of the diverse vehicles, I guess?

Or is that still like a huge challenge despite whatever you're running underneath it? The Tower of Babel challenge is really huge in any industry. In automotive or any industry that has a lot of functional safety requirements it's going to be a real challenge. My feeling is that the way that

we are approaching it the way that Red Hat is approaching it is we're not the only ones doing this. The way that this is being approached within open source communities

is really going to improve that Tower of Babel situation because everybody will be working along the same lines and working towards the same goals even if they then differentiate in other ways and

So there are a number of projects that we are involved with right now The one of the primary ones is Eclipse SDV. It's the Eclipse software-defined vehicle working group, okay? It is I think that there are 42 or 44 members currently and we get new members every month or so. It's a very vibrant project. There are, I believe, either 17 or 19 software projects within this umbrella, within this working group. And they've been contributed by a large diversity of companies, including Bosch, Capgemini. Microsoft has a few projects in there.

And it's been a real pleasure to work with all of these companies, you know, in this area that is typically very proprietary and not very open. Another project that we work on is called SOAFE. It's S-O-A-F-E-E. It stands for Scalable Open Architecture for Embedded Edge. Okay. And this is actually a special interest group that is driven by ARM, the hardware IP company. Right.

But it's not going to be limited to ARM, I'm hoping, in the future. I think that the concepts that it's working on is to create an architecture for embedded edge. So I think that there's an aspiration to do this beyond automotive eventually. But right now the focus is on automotive as a workload or as a functional area. And that also has, I think, about 75 members. There are talks given all over the place, and it's a very vibrant project. How far up the stack does the open source world go? I mean, we're talking about Linux,

so we're talking about operating system-level primitives, making the car function in certain ways, but also every car now is starting to have a huge HUD, a huge screen in the thing that's going to be driven by some software that's also going to be in control of things. Are there efforts that go higher up into UIs and drawing screens and stuff? Yes. Yes, there are. There are some projects that are still in their infancy.

Right now, the main thrust seems to be getting a lot of the plumbing right. So with Sophie's architecture, that really focuses on the lowest end. From really how the operating system talks to the hardware, what do all the interfaces look like, how do you do mixed criticality with these VMs and containers? And then a lot of the software projects that are emerging in the Eclipse software-defined vehicle project

are a bit higher level. They tend to be... I'm trying to think of some now. I should have made notes before I came to talk to you. That's all right, no pressure. I know that Microsoft has one that's called Chariot. It's part of the middleware. We call it middleware.

Okay. I don't know of any actual dashboard projects that are in Eclipse SDV, but there are certainly dashboard projects out there. In fact, AGL actually has an excellent dashboard. Okay. Does Toyota use that, or do they have their own dashboard? They use that as the basis for developing

their own, and that's really what AGL's focus has been, of providing the components that are needed for a car to be created from that software. So I drive a Ford F-250 2022. What's my operating system?

Is it Linux? Probably not Linux. In fact, I can guarantee it's not Linux. Most likely it's QNX, but it depends on which computer you're talking about also. There will be a number of those. Well, Jared mentioned the HUD. I have a big old dash right in the middle there. It handles a lot of the stuff.

Dashboard is very likely running QNX. What is QNS? QNX is an operating system that was created by BlackBerry, actually. It's a proprietary real-time operating system, and it's used throughout the automotive industry. What about other vehicles? How long has Linux been the underpinnings to top of the hardware? Is this new?

This is relatively new. Why? It's developing. Partly because Linux has been historically very difficult to certify as functionally safe. Okay. So that's why where AGL has made inroads has really been just outside the realm of that ASIL structure. So there's another level of ASIL they call QM, which is quality managed. And the requirements for that are less.

And so that would be things that are not dependent on functional safety. So things like the IVI system, or there are some dashboard components that fit into that. Body control is another one. Turning on lights when doors are opened. If you have lights that scale for certain reasons or they're dependent on sensors, those would all be QM level. Or even my mirror sensors that say there's a car next to you. I have sensors around my whole entire vehicle. How close am I to the vehicle in front of me?

You know, is there somebody to the left of me? I forget what that system's called, but there's a name for it. Those are ADAS systems, right? Autonomous Driver Assistance Systems. Right. And very important, if I look left and I see that yellow light, I am not going to start merging left because that's my initial warning to say, hey, Adam, stay in your lane.

Right. You know? Now, there are some ADAS systems that are emerging that are based on Linux. There's a company called Apex.ai that has been developing one. I believe that's based on Linux. I hope that I'm right about that. Right. I know that they use some robotic software called ROS.

And the ROS community is actually very popular within ADAS. And we've been doing some work just recently, actually, to better incorporate ROS into the Red Hat world. So we're restarting the robotics special interest group within Fedora and working with Open Robotics, the maintainers of ROS, to get ROS support in there. Exciting times. So if you were to cast forward to a fully open source based car, soup to nuts, what are we talking? Five years, ten years, never?

I wouldn't say never. I think that the functional safety story is emerging. And functional safety processes generally take two to three years to go through. So you're looking at least probably five years out. Okay. I don't know, maybe less. I mean, I know that whether Linux will be in a car. Linux is obviously in cars now in areas that don't require functional safety.

If you're talking about soup to nuts, though, I think you're probably talking more than five years out. And maybe never. But you think it's going to happen? Because a lot of these computers are still going to exist with real-time operating systems. Disk brake sensors. There's probably three or four sensors within a disk brake system.

Those will be running real-time operating systems. All right, so not soup to nuts. Like, not 100%. No. However, there are open be running real-time operating systems. All right, so not soup to nuts, like not 100%. No, however, there are open-source real-time operating systems. Okay. You know, Zephyr comes to mind and FreeRTOS as well. So where does your work, you personally, fit into this whole ecosystem?

So my work, as a community architect within Red Hat's vertical OSPO, or Open Source Program Office, what I do is help Red Hat participate in all of these different industry groups. And I am able to do some of that participation myself as well. I chair the marketing group for the Eclipse SDV working group as well. And Red Hat has engineers working on this? Yes. How many? You know, I couldn't say at the moment.

I would say, not because it's a secret, but only because people drift in and out. And sometimes people will work on specific projects, and sometimes I never even hear about it. But then somebody will emerge and say, oh, I've been going to those meetings for a month. Well, the dashboard seems like a perfect fit for Linux. You know what I mean? For sure. Let's replace or put in place the most trusted operating system in the world.

When is the year of the Linux dashboard? The year of the Linux dashboard. I love it. 2025 will be the year. I'm not going to put a year on it. But, you know, to go back to the soup to nuts thing, there's another thing that's been really exciting is RISC-V as a hardware. Yeah. RISC-V, of course, is emerging hardware.

You know, there's not a whole lot available yet, but it has been growing significantly. I had the pleasure of working on RISC-V for about two years while I was at the Linux Foundation. Oh, you did? And there is a new automotive special interest group that has formed within RISC-V, driven by imagination. Huh. And so, that's been really exciting, too, and I'm really hoping that all of these

worlds can kind of come together, and we could create a soup to nuts. I mean, imagine it being open all the way down to the transistor level. Wow. Now you're just pipe dreaming. Ugh. Yeah.

That'd be cool. Who cares about car safety? Like, who are these interest groups? Is it insurance companies at all? Because, I mean, they stand to lose, but they also stand to gain until there's loss.

OEMs, tier one suppliers, tier two suppliers, basically everybody who is in the line of responsibility for these components cares about functional safety. The CYA stuff. If my software fails or my choices fail, then it's my responsibility. There is certainly that. That kind of situation. Yeah. But I mean, for every single person, I think, that I've dealt with in the automotive industry, people take functional safety very personal. Yeah.

I mean, it's like a personal, I'm not sure what you'd call it. Value or priority? They're personal values, yes. It makes sense. I mean, nobody wants to build something that's going to hurt somebody. Right. Right? Right.

And I tell you, we've been going through the process. Another project that I'm involved with is ELISA. It's a project within the Linux Foundation that evaluates and provides sometimes patches for the Linux world related to functional safety. Okay. And part of that is going through harm reduction evaluation processes. And boy, those can get to be pretty amazing. There's an automotive sub-professional interest group

in there as well. And yeah, once you start thinking about all the things that can go wrong, it can take you down some pretty weird paths. For sure. Do you have, can you say at least, who the most forward-thinking car manufacturer might be given, you know, this open world, this potential use of Linux, AGL, et cetera, inside of their vehicles? Like, who's thinking about, like, for example, Discover's here, traditional credit card, but they're celebrating how much they're involved in open source and they're coming out, I guess, a bit more in the technical world to share

more of their story. Who is doing that in the car manufacturing world? Well, I know that GM just joined Eclipse SDV and has actually contributed a project into it. So that is certainly a measure of forward thinking. There are a number of European car manufacturers as well. Mercedes has been involved.

And I know that I've talked to a number of other companies who, some of them are starting their own open source program offices. Some of them are starting to participate more in projects even beyond the ones that I've talked about. Some of them are starting to file patches with the Linux kernel.

And that's pretty amazing for a car company to do that, or even a company that, or even like a spinoff software company that spun off from a car company. These are companies that have traditionally been very proprietary and secretive about their work, and now they're really opening up. I think it's wonderful for open source, and it's certainly great for the automotive industry. Yeah. How or where would somebody who's curious about automotive Linux at large,

where would they plug into? That's a good question. There's certainly a lot of written articles about these. One great place to plug in and learn more is AGL. AGL has been around for nine years and there's certainly a lot of written materials out there. And for the method of building automotive software from source, that's really the place to go. For building software based on distros, I would say that the centos automotive special interest group is i have i i full disclosure i actually chair the group so i'm i'm advertising for my own group but it's it's an open group we have meetings once a month uh we have a chat channel on matrix and a mailing list that's very active so um that's a place to go and see, you know, what are we building and why?

You know, we talk a lot about that. There's a lot of documentation on the site as well. And some of these projects like Eclipse SDV is very welcoming. They're very welcoming to people, and they do produce a whole lot of material. Cool. I love it. Linux in my car. Yeah.

Truck. F250. And who knows? You know, maybe your submarine someday. Oh. If I own a submarine, something's really going well in my life, okay? I thought you were going to say something's going wrong. Really well, okay?

If I can own a submarine, we're slaying it, Jared. Bucket list. That's the thing is, a lot of these, a lot of transportation use cases look very similar to cars. Some of them are more fleet oriented. You think about big rigs. There's a whole lot of this stuff, this work is going on with shipping.

Boats, ships themselves. Boeing just started a working group within ELISA to talk about functional safety for avionics and aerospace. So there's a whole lot going on with transportation right now. We talk zero about autonomous cars. Is that a part of this, AGL at all? I don't know that AGL has done a whole lot with autonomous vehicles.

I think there is an autonomous vehicle working group within AGL. There certainly are a number of organizations out there related to autonomous vehicles. I think there is an autonomous vehicle working group within AGL. There certainly are a number of organizations out there related to autonomous driving. Apex.ai is one of them. One of the ones that we work with quite a lot is Autoware.

There's an open Autoware automated driving kit that you can build and install on Linux. And in fact, we have an image for it within CentOS. And it's also very prevalent within the SOFi community. The AutoWare community has been very active. And that's also based on ROS, on the Robot Operating System. Cool.

So many acronyms. There's a lot of acronyms. Holy moly. But I'm happy you say CentOS properly, so congratulations. Which is half the battle. It's not CentOS. Right. It's not whatever else you might say. I don't know all the permutations of CentOS, but it is CentOS, so congratulations.

Thanks. As a Red Hat employee, we get training on that. Is that right? Must say one. Day CentOS properly. First day of Red Hat training is how to pronounce things. Yeah, how to pronounce CentOS. Well, Jeff, thanks for, I was going to say sitting down,

but thanks for standing here and listening to us. We're standing right now, so that's cool. Absolutely. Thanks for doing your great podcast. You opened my eyes to a whole new world.

Absolutely. All right. Let's stop before Adam starts singing a whole new world. You sang it. I didn't. I said, let's stop before Adam starts singing And then you sang it

Nah that wasn't me singing Play it back I can sing much better I'll play it back