Embedded - 349: Open Down to the Transistor

Episode Date: October 23, 2020

Drew Fustini (@pdp7) spoke with us about building Linux, RISC-V cores, and many other things. Links, so many links! Drew is a board member of the BeagleBoard.org Foundation and of the Open Source Hardware Association (OSHWA.org). He is an open source hardware designer at OSHPark (he recommends their blog!). He writes a monthly column for Hackspace Magazine, for example The Rise of the FPGA in Issue 26 and  Intro to RISC-V.  Yocto is a tool to help build a Linux distribution specific to your board and application! Bootlin offers free training material for Yocto and OpenEmbedded (as well as many other things such as Embedded Linux and Linux kernel development). Or there is a video: Buildroot vs Yocto: Differences for Your Daily Job - Luca Ceresoli at Embedded Linux Conference. Or look at Embedded Apprentice Linux Engineer (e-ale.org). Or maybe another video: “Yocto Project Dev Day Virtual 2020 #3: Yocto Project Kernel Lab, Hands-On, Part 1” by Trevor Woerner. RISC-V is an open source processor core. Well, cores. But you can try them out in hardware even if you don’t want to play with an FPGA. The SiSpeed Longan Nano has a GigaDevices microcontroller dev board (with an OLED on board!, more info). Did you know you can run Linux on RISC-V? The cheapest method is emulation and Renode is brilliant for that. Here is Drew using it on the train (twitter). Sipeed boards with Kendryte K210 start at only $13 and can even run Linux (tutorial). There are also affordable open hardware FPGA with free software toolchain support like the ICE40 based Icebreaker and Fomu. For a bit more money, the bigger ECP5 can run Linux. Or look at Greg Davill’s wonderful Orange Crab. For a lot more money but on silicon, the Icicle with Microchip PolarFire SoC is aimed at corporate use. Or you can produce your own physical chips. For free (for a limited time). See the talk from Tim Ansell - Skywater PDK: Fully open source manufacturable PDK for a 130nm process Drew attends a lot of conferences, here are highlights from the past: OHS 2020 wrist badge  OHS 2018 epaper badge  ELC-2018 EALE Buildroot - Thomas Petazzoni  ELC-2018 EALE Bitbake YP - Behan Webster Linux on RISC-V with open hardware and open FPGA tools Sldies for Embedded Linux Conference Video from FOSS North Linaro Connect BoF: gpio and pinctrl in Linux kernel (Slides) RISC-V: How an open ISA benefits hardware security (Slides) (Hardwear.io video)   Here are some future conferences he’s planning to attend: Embedded Linux Conference Europe ($50) October 26-29, 2020 (Virtual) Yocto Project Virtual Summit ($40) October 29-30, 2020 (Virtual) Open Hardware Summit March 13, 2020

Welcome to Embedded. I am Alicia White. I'm here with Christopher White. We'll be talking about Embedded Linux, RISC-V, and FPGA tools. Our guest is Drew Fostini. Hey, Drew. Thanks for talking to us today. Hello.

Could you tell us about yourself as though we met at an open source hardware something conference? Or bring a hack, which I think is where we actually met for the first time. Actually, that's true. Yeah. Yeah. So I'm Drew Fustini. I'm from, originally from Chicago, but live in Berlin now. I'm an embedded Linux developer and hardware hacker. I'm part of the BeagleBoarder Dork Foundation. We create small single board computers with our wonderful community and manufacturing partners. Some of you may know like the BeagleBone Black,

which is probably one of the most successful boards. Going back to 2008, where we had the original BeagleBoard, which is one of the first low boards. Going back to 2008, where we had the original Beagle board, which is one of the first low-cost ARM dev boards. I'm also part of the Open Source Hardware Association, or referred to as ASHWA. Some people may know it. We're the organization that organizes the Open Hardware Summit.

And then we also have our Open Source Hardware Certification Program. I think we'll have the link in the notes there if people want to check out more about that. And then from my job, I work for Ashpark. We're a circuit board manufacturing service in the U.S. and I design hardware projects for them and work with our customers that are doing educational and open source projects. Also, more recently, like in the last year, I started writing a monthly column about hardware in Hackspace magazine. I think we'll have the link there in the show notes. Try and take some topics like FPGA and RISC-V and try and distill them down to like 400 words, like sort of introduction to those topics. Also, one of the people that started Hardware Happy Hour, which now is not really a great time for it.

But, you know, hopefully in the future when things get back to normal and we can meet in person, it was cool. It started in Chicago like three years ago with, with Chris Gamble, who's been a guest many times and another guy from our hackerspace, Andy. So,

and now there's like hardware hacky hour, hardware happy hours all around the world. So that's, that's been a lot of fun. There's been a few virtual ones. And if anyone's interested, there is a Discord for that. So you can get on there and chat with people. But it's not been a great year for meetups.

No. And when I met you years ago, you were at Element 14. Yeah, I was at. And it's confusing because then the corporation i worked for it was called premier farnell um now they're now they're avnet because they're acquired by avnet um i worked like i worked in chicago so we went by newark electronics in europe it was farnell electronics and then while i was there we created a new brand called element 14 which is the which is silicon.

And that may be what people are most familiar with because the products that are manufactured by them, like Raspberry Pi and BeagleBone, were branded as Element 14. So it was always very confusing because depending on where you're from in the world, I had to use one of the different names of the company around the world. That was definitely where I kind of got re-interested in electronics. Like earlier in my career, I'd been a sysadmin and web application developer, and I came to Newark to do that. And then I was doing a deployment one night to like our e-commerce site and started looking through the catalog

and found a microchip PIC, I think it was like the PIC two kit or something like that and i was like oh this looks interesting so i ordered that and then kind of fell in love with um embedded systems you know it was i enjoy the the the more uh visceral feeling of of having your code control something in the in the real world you know um i especially too, because I kind of grew up in the era of like early web development. So everything's very like abstracted away, right? So it's pretty awesome to see your code

control something in the real world. I totally agree. I totally, totally agree. Okay, so you're a listener of the show. So you already know about Lightning Round. Are you ready? Yes, I am ready. Favorite shade of purple? Oh, that's a really hard one because it's so dependent on lightning. I would say

when you look at a Osh Park circuit board, whatever color purple you see there would be my favorite one. What distro are you running currently? I'm currently running Ubuntu on my laptop. There's more to that. There's numbers or something. Numbers. Oh, 2004. So I guess I'm pretty recent. I guess it's October now.

So maybe there, I guess 2010 will be out soon, if not now. Have you found it not traveling after living so much of your time on the road? Yeah, it's been interesting. You know, one of the things is I've actually been pretty busy still attending virtual events. In a way, I think I've maybe even gone into more meetups and events now because I'm not limited by geography, just time zone. But I will say, like, I think I've had more time to concentrate on, like, deeper technical work. Like, I've made a lot of, I've made many more contributions to, like, the Linux kernel this year than I had in any previous year. So I think that's been an upside to it. But looking forward to hopefully someday meeting people again in person. This is a question I don't understand. It's from another listener. I assume there's a private joke. How is Elvis and have you seen him lately?

Oh, I don't. That's a Ghostbusters quote. I don't know. Maybe I'm forgetting something, but I'm not familiar with that. That's a Ghostbusters quote. I think somebody was just being silly. Oh, okay. Okay, then do you have a favorite weirdest Linux computer?

Oh. I mean, I know there was the Hackaday Supercon badge, but something strange like that. Something strange like that. I mean, well, I think Linux running inside a soft core kernel developer, Linus Valege, in Sweden last year. And in his office, he has this museum of old ARM-based systems. And he had a Corel Netwinder PC, which had a strong ARM in it. And one of the things he does, he's a kernel maintainer, and he makes sure that the kernel still boots on these really old arm systems.

So we actually booted the current main line on the Netwinder when I was there, which I thought was really funny. Also, the fact that there was a Corel PC, which I did not know that that was something they did. I think it was during the Net PC sort of era, when your computers are supposed to be like a thin client but it was it was that was unusual the 90s were a weird time uh complete one project or start a dozen i've heard many people answer this and i don't think

i have the discipline to just complete one project uh serially so i i think maybe half a dozen is a better answer for me, because I think maybe I can kind of keep on working incrementally on half a dozen. A dozen is too much, though. Then I would feel bad because I feel like I would not be able to schedule time for each of them. So half a dozen is my answer. Okay. Here's the last one. What technical podcast would you listen to on your honeymoon? I mean, I would say embedded.fm would, would be a good choice since me and my spouse have both will now been on the show.

So I think that that would be a good one. Oh, that was funny because after they got married, they immediately said they were listening to the show, which I just found. Sorry, I'm easily amused by that. I think the last time we listened to it together, we were on a paddle boat on a lake outside of Berlin. It's a fun show to listen to while you're pedaling around the water. Okay, so someone asked for a guest to talk about Yocto, and that made me email you. Because I know you're deep into the embedded Linux side.

So maybe let's start there. Although, once we got talking, I think there's a whole bunch of other stuff we should talk about too. Did you? Was there a question there? There was no question there. What is Yocto? Yeah. So maybe like a little bit more background is probably many people have at this point used maybe a Raspberry Pi or a BeagleBone. And both of those use more general distributions. With BeagleBone, we use Debian. In Raspberry Pi, the most popular distro, I think, is probably Raspbian, which is a

recompilation of Debian to run better on the Raspberry Pi. So those are kind of traditional binary desktop or binary Linux distributions where you have a package manager and you say, like, app get, you know, such and such, and it'll install the dependencies and you don't have to compile anything. And you can still do that on embedded systems like with the BeagleBone or Raspberry Pi. But when people get towards making a product or making a system that's going to do a fixed thing, you know, like you're not doing rapid prototyping, you're making a kiosk or, you know, barcode scanner or, you know, an autonomous vehicle or a CNC controller, then you don't necessarily want all that extra stuff, right? Especially if you're going to make a lot of the product,

you probably want to have like just the memory you need, just the RAM you need, just the storage that you need. So you want to kind of tailor the Linux installation to just what you need. So one of the ways to do that is Yachto. Yachto is a bit of a confusing term because it grew out of another community. So one of the ways that it's been really popular to make an embedded Linux system for many years is called Open Embedded. And this grew out of a community of people that were hacking on old PDAs, like the Sharp Zaris. So out of that came OpenEmbedded,

which is a collection of tools and knowledge of how to build a Linux system that you can put into your little single-board computer that goes inside of a product. Later on, like maybe 10 years ago, the Yocto project started. So the Yocto itself is a project that is a collection of tools that lets you build your own customized Linux distribution. And it uses a lot of the technology that was developed by the Open Embedded community.

Now, Yocto is not the only, Yocto slash Open Embedded is not the only way of doing this. Another popular alternative or another popular way of doing this is called buildroot so i would say for people that are making um their own like linux distro for a single board computer they're probably going to either be using buildroot or yakto slash open embedded and i think if someone was interested in yakto specifically um that may be because over the last maybe five years, it's become very popular for the silicon vendors like Freescale or TI to provide a board support package or BSP in the form of a Yocto layer. So a layer is a collection of what they call recipes for building different packages. And that's one of the reasons why OpenEmbedded slash Yocto is a nice way of building a custom Linux system is because you build up what you need. So you would have a layer for your board support package,

which might include like a kernel, a branch from the vendor, and other specific things to the hardware. And then you'd have layers that build like the applications that you need and the libraries that you need, like maybe Qt, if you're doing a graphical interface. So you kind of, in a nice modular way, build up these layers that give the instructions about how to take all those different projects from source and compile them.

And then either build binary packages like RPMs or DEVs and even produce an image that you could put onto an SD card. So it's a nice way of creating a custom system. And I think the reason why it's become very popular is it's a really nice way for someone like Freescale or TI to provide everything you need for building a Linux system for one of their SOCs. Whereas I remember when I was at Element 14, seven years ago, one of the boards we had was a Freescale-based board, and they just gave a tarball of a bunch of source code that you could compile and put

onto the system, which is not a very friendly way of doing it. So I think Yocto has brought a nice set of tools and infrastructure around how to build from source all the packages you need for a full system. And BuildRoot is a similar sort of philosophy. It is, however, based on tools that people are probably fairly familiar with, which is Make in KConfig. So KConfig is the tool that you use for the kernel configuration options.

And then make, probably all of us have maybe dealt with that at some point, trying to get software to compile in a Linux environment. So Builder is a little bit simpler. So it just produces a root file system. So it doesn't produce like binary packages like devs and RPMs. Um, so I would say like, if you're just starting from scratch, like build words, product builder is probably a little bit easier to wrap your mind around. Um, but, uh, Yocto is something that has a lot more flexibility and also the SOC vendors tend to, um, use that nowadays to provide the support for their chips.

I'm always afraid of building, of even trying this. It seems insurmountable because it seems like I'm just going to make a Linux distribution that won't work at all. I mean... Maybe we should talk about what Linux... Because if you're coming from microcontrollers, you think of a firmware image, it's got, oh, it's got my main, and then it's got my loop, and, you know, it's all in a hex. And maybe you have a second hex that's the bootloader,

but you only have two, maybe three. That's pretty much it. It's a binary image with your entire program on it, including, you know, the ports part. But Linux is much, much different. It's much more of a real computer operating system with layers and things. So what are the things that are... Just real briefly, what are the pieces

that Yocto creates that maybe people coming from microcontrollers don't think about that much? Yeah, and before I forget, we'll have it in the show notes. I think one of the great resources that people can pull up is there's a company called Bootland that does a lot of embedded Linux development and also training.

And they have a lot of training materials that are available under Creative Commons that you can download off their website. And they have a course about building embedded Linux systems with Yocto and also with Buildroot. So we'll have it in the show notes. Check that out. They have the slides and also the labs that you can go through that are based around, I think, the BeagleBone and Microchip Atmol board as well.

The other thing I think we'll put into the show notes is from the recent Yocto Developer Summit back in June, there was a nice tutorial about how to build a system with Yocto using a VM. So this was quite nice. So basically, there's a VM that you can install that has all the tools set up, and then it produces an image that you can boot with QEMU, which is an emulator. So even if you don't have any hardware, you don't want to mess up your hardware, you can learn how to build the system with Yocto and have it boot up and be able to see how,

when you change recipes, see what the interaction is. Oh, nice. But in general, for an embedded Linux system, yeah, there's several things that you'll need, which Yocto can help build for you, also Buildroot. One is the bootloader. And I would say the majority of embedded Linux boards are going to be using uBoot.

That's kind of the de facto one. The one exception to that is the Raspberry Pi is kind of unusual, and the Broadcom SoC is unusual in that the GPU actually acts as the bootloader. But for pretty much everything else, it's going to be using. Wait, what? Yeah, the SoC from Broadcom that's in the Pi was meant for like set-top boxes, like the Roku. Yeah.

So it's called the VideoCore. And the ARM processor, at least on the original one, was just kind of like an add-on. So the main thing that boots the system is part of the VideoCore. Oh, wow okay so it's it's not like the shaders but sure sure sure uh so it's unusual you know raspberry pi is probably the most common arm board out there um but it's atypical in the fact that it kind of has its own proprietary bootloader whereas almost every other board is is probably using u-boot um so that's something that Yocto can help you build. The next thing is the kernel.

The Yocto project itself has a couple of different options. Like they have the stable Linux kernel, which is kind of like the most recent release from Linus Torvalds. And then they have a couple other options, like ones that are optimized for real-time Linux. And then there's also the category of vendor kernels as well. So there might be a branch that TI is developing or maintaining or Freescale is developing or maintaining. Usually that's because they have support for newer hardware in there that hasn't been accepted into mainline yet

which by mainline I mean the Linux repo that is Linus Torvalds whereas there's many different many different separate repositories that other people maintain but when we say mainline we're referring to the main

Linux that Torvalds releases So when you build an application the the embedded developer is mostly going to be working on presumably things in the application layer that are post yachto yeah um so i guess in addition to the kernel um the other things that it does is like the init system um So with Yocto, you have your choice of having like an older sort of system five init system or system D. I think there's a couple other options as well. So you have like the bootloader, you have the kernel, then you have the init system that brings up, you know, the what we would refer to as the user space, which is like the application code

and the different services that might be running on the system, like SSH damer or something like that. And then on top of that, you have system libraries. So let's say, for example, you're building an industrial control panel and you're using Qt or Qt to build the graphical user interface. So like Yocto would be able to build the Qt libraries that you would need. And then on top of that, yeah, for most people, they're probably doing, you know, maybe C++ or C or Python for like the application code that probably going to

provide some sort of like user interface, you know, some sort of networking functionality. So that's kind of, I would say that the way that you compose an embedded Linux system. Okay. Is it better for me to take the kernel I've been given from wherever I got my board and try to strip things out? Or is it better for me to try to recompile the kernel knowing that I'm a little hesitant about getting that right and breaking it all? I think probably the best place to start is for whatever board you have, they're probably going to point you to the repo that they compiled the kernel from.

Usually that's not going to be like the upstream, like the official Torfolds repo. So like, for example, with BeagleBone, we have a Debian image and we do our own kernel builds. And that's something where we take the TI, Texas Instruments has their own repo of the Linux kernel, where they have some additional features that they're working on and like hardware enablement that hasn't been merged

into the upstream kernel yet. So it's not in Linus Torvald's repo yet, but it's in there. So we take that and then we build upon that. So one of the nice things that we have for people that want to then rebuild the kernel is we have like some scripts that allow you to choose which version of the kernel you want to take and compile and then you get all the basically the setup we have which is already like what you're already running on the image but you can basically reproduce the kernel that's already running on the image and the reasons you might want to be able to do that

is you want to turn on a driver or turn off a driver or maybe enable debugging. So there's some debugging options that you can turn on inside the kernel config. So oftentimes someone might be having issue with particular drivers. They want to go and recompile the kernel to turn on some extra debugging options for that driver.

And you can do that by basically, it's kind of a requirement of having Linux on a dev board is you have to point people to the source code that was used to build it. So for us, we have like our own GitHub BeagleBoard Linux repo. And from that, you can check that out, check out the branch that corresponds to the kernel that was on the image that you're using and rebuild that. Now, it can get a little tricky because if you're doing this on your PC, it's going to probably be x86, you know, 64 bits. So you'll need to have a cross compiler.

We solve that by having a little script that goes and grabs the cross compiler. So you can build everything. You can build the kernel for ARM on your PC. Yocto is another thing that does that. So let's say you have, you know, an SOC from Freescale. What is called in the terms of Yocto are these different layers, which are called meta layers, as in metadata. So you'd have the meta Freescale layer that would have the information of how to go and grab the kernel source from their repo. And then you can also modify the kernel config. You can run menu config, which allows you to turn on

and off different kernel features and recompile it. And then like the most vanilla or pure approach would just to go and grab from kernel.org, grab Torvald's repo and just compile that. But that can be a little bit more difficult. One, you have to have hardware that's fully supported upstream. So for something like the processor that's in the BeagleBone, it's been out for like, I think over eight years now. So that's fully supported upstream. So for something like the processor that's in the BeagleBone, it's been out for, I think, over eight years now. So that's very well supported in the upstream Linux kernel. But if you were to take a brand new SoC that came out probably this year,

probably the support's not in the upstream Torvalds kernel yet, so you'd probably want to be relying on the vendor, like Freeskills repo for that. And Yocto, like the Yocto meta layer for Freeskill or TI is just a way of making that easier to do. So does that kind of answer what you were wondering about? Yeah, I think so. Like for me, for example, I do upstream kernel development on some of the drivers that are used on the SoC and the BeagleBone, like the pin control driver and the GPO driver.

And since I want to submit those patches I'm making to the upstream mailing list, I will just go and grab Torvald's repo. But that can be a little bit more difficult because there might be something that's different in the upstream Torvald's kernel than the vendor kernel, like the TI kernel. For my purposes, I'm trying to make them the same thing because I want to make basically it possible to just go and grab Torvald's kernel and build it for the BeagleBone and have everything work out of the box. That's kind of the goal, and that should be everyone's goal, to make sure it all just works with the upstream kernel, so then you don't have to have these instructions like,

oh, you have to go and make sure you grab this vendor's repository of Linux and use these kernel config options. Ideally, for every board, there would just be a configuration in the Torvalds repo that says, okay, this is everything that you need to boot Linux on this board. So it gets a little tricky, though, when it comes to stuff that's relatively new or when it comes to vendors that don't put much resources into getting their contributions or their customizations upstream. You're not making me less nervous about this though. I mean, I still feel like... One way to make this maybe less hypothetical is, is there a particular board that you're

thinking about? Well, I'm usually thinking about the TX2, which I know isn't a board that you have a lot of experience with. I don't think you're supposed to touch that. Yeah. So, I mean, I've not used that a lot. I mean, it's a much more capable board than most of the things I work with. But I think, you know... But, okay, so forget the NVIDIA side, the NVIDIA part aside, and not necessarily this board. But it has Ubuntu. I'm stuck on 16 because I have robot operating system for clients who are

stuck on ROSK and blah, blah, blah. I'm on 16. But I really want, like the networking system is kind of complicated and I know what I want my network to be and I've got three ways to manage it but I just want one because apparently I manage it one way and other people on my team manage it another way and we end up with what you'd expect which is not good so I want to recompile not necessarily the tx2 but if I had this problem on a Beagle board or something else. Sure. Yeah. It's just, it's getting started and then it's having the confidence that I did it right. How do I get those?

Well, I think, I mean, Yachto is definitely the sort of thing that was meant to help in these sorts of situations where you want to be able to like customize parts of the system. And it's one of the downsides to using a full distro like Ubuntu. It's a lot more complicated. So in the case of NVIDIA, there probably is a NVIDIA meta layer for those SOCs. I usually use Freescale as an example or TI because they're really common in the industrial embedded Linux space. But the whole point there is that, you know, from the vendor's meta layer, that that has like the configurations in the repos for the kernel and other things.

But you can do things like you can change the, you know, init system. You can change like the, you know, for a GUI, like what desktop system you want to use, what session manager you want to use. So it gives you a lot more control over that. You know, like maybe you don't want to use system D. Maybe you want to use a different sort of init system or network manager.

I mean, when it comes to network managers, there's a lot of different ways of doing it, right? Yes. You might want to use, like on Beagle, we use something called Conman, which works quite well for us. And it's lighter weight. That's an encouraging name. Conman, yes. With two Ns, which is, I guess, hard to hear when you say it out loud.

But it's simpler than the GNOME Network Manager, which can be quite complicated, or like like system D network D. So there's a lot of different ways of like configuring a network. And when you get like a distro, like Ubuntu, like they typically choose a way, um, and then it can be difficult to, to do it in an alternative way on a distro like, um, Ubuntu. So that's one of the nice things about, um, the Octo is like, let's say, you know, you want to use Conman or do you have a particular like network manager that you are wanting to use? The one that was on the Solaris 5 machine that I did when I was a... Sunos. Sunos, yes. Solaris was the terrible one.

I mean, there might be something that... No, no. I mean, I just want to use a basic one. I don't really care what it is. I just want it to be super simple has semantics like that. No, no, I mean, I just want to use a basic one. I don't really care what it is. I just want it to be super simple. Yeah, look at Conman. That's what we use on the BeagleBone. There's also, like, if you're using a systemd-based system,

like Debian is now by default, and also Ubuntu, like, there's systemd-networkd, which is common for people to use as a network manager. Or you can even not use one at all. Like using a system like Yachto, you have control over that. So you can choose like a layer, which is a collection of recipes for building the code from source that gives you the network manager you want. Or no network manager. Maybe you don't want a network manager,

and you have your own scripts that do it manually. So that's also an option as well. Versus I think with a distribution like Ubuntu, they kind of have chosen a way that they think will work for most cases and maybe for general cases, but it might not be right for like your embedded system. It's more for me, it's often that I want to get rid of things because I have this idea that they add complexity in the operating system. And so I just don't want them and I want to be able to strip out everything. And I, is that, is that a reasonable thing?

Yeah, I think that's, I mean, that's ideally what you should be doing for like an embedded system, right? Like stripping out all this generic stuff, you know, and just using what you know you need, right. For your project or your product. Right. So that is kind of what Yocto and OpenEmbedded give you is the ability to like, they call them layers, which is like a collection of recipes, which are basically like where to grab the source and how to build it and how to configure it.

So, you know, there'll be layers that you could probably grab that have like a minimal network manager, you know, ones that are meant for like no GUI, ones that are meant for like a lightweight, you know, GUI, or maybe like a full, you know, GNOME desktop. So I think if you look at, you know, GUI or maybe like a full, you know, GNOME desktop. So I think if you look at, you know, the Yocto for the board that you're working with,

you'll probably find a lot of different options for doing that. You know, the other thing I wanted to mention before I forget is coming up next week is the embedded Linux conference Europe. And as part of that, there's the Yachto developer day, which is a bunch of both presentations and there's also a Slack chat for it. So it's a good opportunity to like talk to like experts and people that are using Yachto and open embedded to build products. And kind of the upside to the pandemic is the Better Linings Conference, which is put on by the Linings Foundation. It's only $50 to attend it online, which, you know, typically like these Linings Foundation conferences are like might be $500, $600. So plus hotel and travel and all that. So it is a good opportunity to talk to people that are, you know,

doing this on a regular basis, building products around it. You know, it's big enough that you'll probably find someone that's like already working with like the board you are. So that is the Embedded Linux Conference Europe, and that is October 26th through 29th of 2020 and then the yacto project virtual summit is october 29 30 of 2020 so that's what next week is and i would say like a lot of my knowledge of embedded linux other than just using it has come from the embedded linux conference over the year um which is put on by the Linux Foundation.

So they have North America and Europe every year. And eventually they get all the videos up on YouTube. So that's why I would encourage people, if they can, to attend it live because it usually takes several months to get the talks up. And also, we did this

one prior time, which was Embedded Linux Conference North America back in June. And there was a really great Slack instance for that with channels for all the different talks. And it was really great to connect with other people that were working on similar systems and can answer questions that I had. So I think it's well worth it. And in terms of the past 10 years, I think of this conference are all up on YouTube. So for any sort of embedded Linux conference or any sort of embedded Linux topic, you can probably find several talks about it. And because I would say it's really tough to,

well, I mean, you've written a book about embedded systems and I think you kind of had information in it that was not tied to a specific version of software. But we don't have a lot of books that are kept up to date about embedded Linux because it's changing so rapidly. So the conference talks are really one of the primary forms of documentation, I think. So for whatever area of embedded Linux people listening to this might be interested in, I would recommend checking out the Linux Foundations YouTube because ELC, the Embedded Linux Conference, I would say is one of the primary sources of information to figure out how to do things with Linux on embedded systems and also figure out how people are solving these challenges, which we probably all run into at some point in time, trying to get Linux to do what we want on a little board.

So moving on to conferences, you've always been someone who following your Twitter feed, you are here and then there and this conference and that conference. And I just, as somebody who really didn't leave home before it was required uh it's amazing at how many conferences you go to which which ones have made the best switch to remote definitely uh i think the best experience i've had so far is a conference called Linux Plumbers. So kind of a fun take on the idea of like low-level system software. So Plumbers has been going on for, I think, maybe 10 years now.

I only started attending it maybe like five years ago. But it's kind of one of the main tenets of Plumbers is that a lot of the presentations there are supposed to be kind of rather short, and then a lot of it's supposed to be discussion. So you might get a lot of people that are involved in a particular piece of software, and they all get together, and you might have people from the GCC or Clang team, and you might have kernel developers, and they're all in one room, and someone gives a little presentation on something they're working on and they say okay but like what's a way for us to solve this that you know all the different stakeholders are happy with so it's one of those conferences where the

in-person thing was like quite important um it wasn't just like giving like a half an hour long sort of presentation it was meant to be very like interactive um and they put a lot of effort into it this year to try and because they had about six months to get ready because it was in august um and i think they made a decision like by april probably that it was going to be virtual so um they chose big blue button um which is a piece of software that I think originally came out of CERN. It's essentially like a video conferencing slash presentation platform. And it works really well with like multiple people wanting to speak. So the sort of thing where it's like a discussion session and you might have like four or five different people that are going to kind of be talking almost at the same time. It works really well for that.

And then along with that, we had a RocketChat instance, which was good for like kind of more in-depth chat. You could take it over to a channel in RocketChat. And for people that are not familiar with RocketChat or BigBlueButton, the reason why they chose those is their free software. So they were able to set up, I think, on Linode, all this infrastructure. And the entire conference was done with only free software, which I thought was pretty cool. So we weren't stuck on a proprietary cloud system like Slack or Zoom or something like that.

What other conferences have you been going to and what talks have you been giving? So we already had the Embedded Linux Conference North America, which was probably the longest one that I went to because it was a good like maybe six, eight hours a day for four days. I mean, not that you have to do the whole thing, but I enjoy it. So I was on for most of the time in that. That was a real positive experience. You know, in this case, the Linux Foundation was using Slack, and that kind of served as almost the same experience of, like, hanging out in, like, a hallway or, you know, going down and getting some food with people after talks.

Not quite the same, but it opened up the opportunity. So there's many times where I'm maybe thinking about an issue or have an idea how to solve a problem. It's not the sort of thing where I would go post it on the mailing list because it's not really that formal. But when we're in a more relaxed environment, like chatting in person or in this case chatting on Slack, I can say like, oh, I see the maintainer of the GPO subsystem and Linux is on there.

I probably wouldn't email him about this, but, you know, we're just hanging out at, you know, the virtual conference. So I asked him some questions and actually it was really great because it helped me kind of get on the correct path to solving the problem that I was thinking about. And I think, you know, I don't know if I would have like posted to an analyst like a formal thing, like trying to ask for guidance. So it's like those more informal interactions, I think are the really great part of conferences.

And some of them have done a good job of that. Some of the other things I've been involved in are just kind of more one way, you know, where it's like someone giving a presentation over Zoom or like youtube live stream and there's not as much of ability to interact um so it kind of depends um one of the events that i was a part of like about a month and a half ago was called the risk five global forum so it was also really interesting um a lot of good presentations um and for that they also had a chat channel for like each talk where you could go in

and ask questions afterwards. Kind of the upside to that is like normally if you're at a physical conference, like you're eventually getting like shoved out of the room, right? Because the next talk has to come up, you know, in this case, you can continue talking about the subject in a chat channel, you know, instead of having to like all run out into the hallway and maybe get dispersed, you know. So I think it does allow for more continued focused discussions after a talk. I didn't put in the notes, but we can add it in there. It's coming up in December is the RISC-V Summit, which is like the big annual RISC-V conference. So people can register for that if they're interested. I think it'll be quite a good conference.

It was supposed to be in San Jose, but now it's just virtual. Are people attending these conferences in greater numbers, lower numbers? I haven't been because I spend way too much time in front of my computer already. And I look at them and I think, oh, that would be fun.

But then I also think of if I'm in front of my computer, I'm probably not as happy as if I was not. Hard to know. I know for Linux plumbers, it's a smaller event. So it's usually kind of bounded by the space that it has, which is usually only like enough for like 300, 400 people. So I know in terms of Linux Plumbers, I think they had well over like maybe a thousand people register. So it did allow more people to get involved. Some of the other ones, they haven't published the same sort of stats. I'm not quite sure.

But these Linux foundation events are usually pretty large in person like a couple thousand so it's kind of hard to tell but there was always enough people around to have like relevant conversations i do get though like the a lot of people have mentioned like the fatigue thing um that like in some ways sitting at your computer um watching talks for like eight hours is a little bit more draining than like, you know, being in a hotel conference center, you know, being able to like chat with people person to person. Like I definitely after some of those days of the Inventor Links conference, you know, I've been typing a lot and sitting at my computer for like 10 hours. So it can be, you know, a bit exhausting, but I also get really excited when I get to interact with people and discuss ideas. So, you know, I prefer to do it in person, but I think I'm really happy that we were,

that the organizations were able to figure out how to do this online and in a pretty quick fashion. So RISC-V, you mentioned that as both that there's an upcoming conference and something you talked about recently. Yeah. I've been hearing about it for years. Yeah. So you want me to kind of give a bit of an overview of it? I guess so. And why would I want an overview of it? I guess so.

And, and why would I want to look at it? In what cases is it interesting? And in what cases should I be looking at more traditional processor course? Yeah. Um, so just to kind of give a little background, um, it probably people may have seen the term thrown around and a lot of times it's just used like synonymous with like the idea of like an open processor but um it's a little bit more nuanced than that so risk five itself um it's risk with a v uh so it's the fifth but reduced instruction set computer or risk uh architecture to come out of Berkeley. So about 10 years ago, people in the computer architecture research group at Berkeley,

they were looking for a new instruction set to base their research projects on, like looking into custom accelerators and things like this. And they didn't want to go to the bother of licensing a commercial instruction set like ARM or x86. So they decided to just take the knowledge they'd already have from doing like the previous generations of RISC instruction sets and just make a new one from scratch, sort of a clean slate design. An idea behind it was that it was something that would be modular and extensible. So it was

something that scale from like small um small microcontrollers all the way up to supercomputers like the joke from one of the people involved is you know i say like it's one instruction set to rule them all um the idea being is that like the instruction set should not be the thing where you're innovating or differentiating right it should be something that's almost boring um and it should be something that you're able to build upon kind of in layers of functionality as you need them. So, and just to go back to what an ISA is, the instruction set's just basically the specification

or standard of the tasks that the processor can perform. So RISC-V itself is just an open specification for an instruction set. It's actually a group of instructions that specifies everything from a small 32-bit system all the way up to like a 64-bit system with an MMU and can run a full operating system. And even there's extensions for things like vector operations and stuff like that. So you don't need to have all that though. So you can, like the base is just a 32-bit integer ISA. The other important concept is that after something's frozen, it won't change. So for example, the base 32-bit integer ISA for RISC-V is frozen.

So if you go and write a program and compile it for 32-bit integer RISC-V, it'll run on a processor right now. It'll also run on some 128-bit RISC-V supercomputer 30 years from now because it's fixed. But things like extensions, like for vector operations, they're still being worked on, and eventually it'll be ratified. And then if you compile a program for that, it'll still work on a compliant RISC-V core 10, 20 years from now.

But that's just the instruction set. So that's just specifying the tasks that the processor can perform, but it's not the implementation. But I'm really interested in open technologies like open hardware. And in order to have a completely open source processor, we need to have an open instruction set. So I couldn't go and design a processor that implemented ARM or x86 and distribute that to people because

that would be infringing upon those proprietary instruction sets. So RISC-V allows us to build fully open processors, but it doesn't mean that it's open just because it's implementing RISC-V. But if you wanted to get more into like, why would you use it and why wouldn't you um so one of the advantages is you know especially in the embedded space arm is really common in a lot of companies license arm cores like the cortex aa or the a53 um and most companies are doing that are just licensing a specific core they aren't uh they don't have the liberty to do their own implementation of the microarchitecture. Only a few companies do that,

like Qualcomm and Apple. Their architecture licensee is of ARM, whereas everyone else is just kind of like TI's licensing, like a Cortex-A53. But if the company like TI wants to differentiate themselves with changes to microarchitecture, they can't really do that.

Whereas with RISC-V, you're open to implement the microarchitecture the way you want. So an example there is Western Digital is actually fully committed to RISC-V now. In the future, all of their drives, all their storage products are going to use RISC-V based controllers in them. And the reason for that is they wanted the freedom to be able to do their own microarchitecture. In this case, they've actually developed something called the Swerve Core, which is a microcontroller with two hardware threads, which for them allows them to have much greater performance for their storage products. So having an open standard that allows companies to implement it the way that best suits their use case is quite powerful.

The other big thing is that RISC-V is not the only open instruction set out there. There's other ones like OpenRISC that were also open instruction sets, but it's kind of the one that has critical mass of software behind it. So instruction set's not worth a whole lot unless you have a critical mass of software behind it, like tool chains and operating systems. So, you know, we have GCC and Clang and GLIB-C and Linux and all that's all supported on RISC-V. So you also get all the software you need around it, you know, which is the main reason why we're still using x86 is because we just have too much software that was written for it that

we're stuck with this 40 year old instruction set that doesn't make a lot of sense. So is it better for people who are looking at, okay, I want to roll custom silicon and I want more control over the architecture and the ISA, or is this something that's currently a choice that say somebody who's designing in a microcontroller would, would want to explore? Yeah, there's, I think a lot of, um, nice examples of things people are doing with RISC-V in the microcontroller space. Um,

and that's partly because it's-V in the microcontroller space. And that's partly because it's easier to experiment with microcontrollers because they're smaller, they're less complex. So the cost to do the verification and to get chips produced is much less than a full SoC that can run something like Linux. But there are several companies that are doing interesting things with low-power RISC-V-based microcontrollers. Some of the work comes out of ETH Zurich, which is a university in Switzerland.

And they have several of their own RISC-V open implementations as part of a program called PULP, which is parallel ultra low power. So they focus a lot on making power efficient RISC-V cores that then companies are going and putting into things. So there's a company called GreenWaves Technology, and they have some really low power microcontrollers for doing like sensor management. And they're able to leverage these like low power RISC-V,

open source RISC-V cores from the pulp team and then add the things that they need for their own use cases. So it gives them a lot of flexibility in that. I'd say the other kind of practical consideration is we're starting to see a lot of RISC-V microcontrollers coming out of China. And one of the reasons is with kind of political tensions over the last few years, there's been a lot of concern over the ability to continue to license ARM cores. So several Chinese companies are exploring RISC-V as an alternative. Probably the cheapest RISC-V microcontroller you can buy right now

is from a company called GigaDevices. You may be familiar with them because they were the company that did a register compatible implementation of STM32. So you could basically have a really low-cost chip

that had the same register map as the STM32. They have a really low-cost RISC-V core. It's called GD32V, I think it is. So definitely in the microcontroller space, there's a lot of different interesting things coming out. The other place where RISC-V can be really useful is for companies that are taking a microcontroller and then putting a custom accelerator alongside that. So it might be

something, especially for machine learning, where you want to accelerate inference on a smaller device. So one of the other companies as part of the RISC-Vy's um out of taiwan and they do a lot of their businesses making risk five cores that can be integrated into like um custom asics that customers are making where it has like um special acceleration for like sensor sensor fusion or neural network inference and then they also want the risk five core there to do general computation and have a have a core that's well supported by tools. Because before RISC-V, a lot of these companies had their own cores, which means they had to maintain their own tool chains, do their own ports of real-time operating systems in Linux. So RISC-V saves a lot of that work so you can you can have your kind of custom hardware but as long as you are still

using the risk 5 instruction set you have the software um compatibility there okay so linux on risk 5 you mentioned that with respect to the hackaday uh badge are there other ways to explore Linux on RISC-V? Yeah, there are. I would say actually the first thing, it's a little bit boring for us embedded people, but the easiest thing is actually to go get an emulator.

So RISC-V works perfectly well in QEMU and also in another emulator called Reno, which I really like. So even if you don't have any hardware right now, I guess I'll recommend Renode. It's from a company called AntMicro. It's an open source project that's an emulator that basically you can say, okay, I want to pretend like I have this RISC-V development board, and it emulates all the hardware that you'd have on there, including the networking, and you can go and run Fedora or Debian,

which both have RISC-V ports on, like, let's say, there's a company called Sci-5 that has a pretty high-end Linux development board. So you can emulate all that on your PC right now. But that's only limited fun because it's not real hardware. So one of the issues when it comes to Linux on RISC-V is in order to have a system on chip or SoC that's capable of running Linux, it's a lot more of an expensive proposition. So we haven't yet seen a lot of options for that. Microchip has finally launched a system on chip that has a hard RISC-V core in it called the PolarFire SOC.

And they have a dev board called the Icicle. And that just started shipping from CrowdSupply. It's a bit expensive, though. It's $500 because it also has an FPGA in it. I think next year we'll see a lot more options in terms of lower costs. Maybe you're on $100 dev boards that can run Linux on RISC-V in an actual ASIC, so hardcore inside of an SOC. And hopefully one of those will be Altoids-shaped and dog-friendly if things work out.

But for right now, probably one of the more practical options to run Linux on RISC-V is to use an FPGA. So the idea with using FPGA is you can actually have what's called a soft core, where you configure the gates inside the FPGA to represent the implementation of a RISC-V core. So you can actually run inside that soft core Linux inside an FPGA. And one of the ways that this gets even easier is there are now several FPGAs that are supported by open source toolchains. So when I first started using FPGAs, maybe like 10 years ago, I was using Altera, Cyclone. You had to install this awful IDE in Windows and it took up a lot of space

and they're still just as bad. Like Xilinx is like 50 gigabytes I had to install. So there's been a lot of movement in the last couple of years of basically there's this great open FPGA community and they've been working very hard on getting kind of documenting the bit streams and

then producing tools that are open source that can do the same things like the synthesis and the place and route that the vendor tools are doing. The ones that work really well right now are for FPGAs from Lattice. So the ICE 40 and the ECP 5.

The ICE 40 is a little bit smaller, so you can do things like have a soft core for a microcontroller inside of it. And I think we'll have in the show notes, one board for that is called the Icebreaker. And another one is called the FOMU. That's a little tiny one that can fit inside your USB port. Those are both great if you're interested in learning about FPGAs and also don't want to have to deal with the proprietary tools. That's really nice. At Hackaday Superconference and also at CrowdSupply Teardown last year, there was workshops. The nice thing about the free tools is they can just

give you a small USB, you know, thumbstick, and you can put in your computer and get everything up and running. And you don't have to like install 50 gigabytes just to use the FPGA. So that's really nice. Now, in order to run Linux,

we need a little bit bigger FPGA than the Ice 40. So that's where the ECP5 from Lattice comes into play. And more recently, people got free software toolchain working with that. So what was on the Hackaday super conference badge last year was an ECP5 FPGA. So during the course of the conference, we were able to get Linux running on a soft core inside that ECP5. And the project we use in particular, and we'll link to this in the show notes, is called Linux on LIDEX VEX RISC-V.

So it's a bit of a mouthful, but LIDEX is this set of different IP, which is what hardware chip-level people refer to as basically peripherals and libraries for chip design. So things like DRAM controller, Ethernet controller, USB controller,

and then VEX RISC is an open-source implementation of a 32-bit RISC-V core. So those things all together give us basically a full system on chip inside the FPGA that can run Linux. And it's all wrapped up into a nice repo where you clone it and then you run the build script,

and it produces a root file system and a kernel, and even the script that will then go and load that into the fpga so and there are several different fpga dev boards including the um hackaday badge that are supported by it so um you can get one of those boards which i would recommend um a board called the orange crab which is from an awesome hardware hacker named greg dab, who many listeners may have seen on Twitter with his beautiful photos of PCBs. Earlier this year, he came up with a board through GroupGets that's Adafruit Feather Form Factor

with the ECP5 FPGA on there. And that's fully supported by the LIDEX project. So you can get one of those boards and clone that repo. And in a few minutes, you'll have Linux booting inside a soft RISC-V core inside the FPGA using only open source tools. So I think that's really fun and really powerful. The UCP know, not the highest end FPGA. So, you know, the, it's going to be running at like more like 50 megahertz to a hundred megahertz, which is probably slow for a lot of us. You know, it reminds me more of my, um, PC from the nineties.

Right. But one of the cool things when it comes to an FPGA is, well, the soft core might be limited in terms of clock speed. There's a lot of room there to implement custom peripherals. Um, So like, let's say you want to do something where you're reading from a, you know, a sensor in normally you're having to like,

continue to pull some, some value and check to see if it's changed, you know, and you're eating up a bunch of like cycles on the court to do that. Well, you could also maybe implement a peripheral um that did that for you right so um like let's say you have a motor encoder and normally you have to have like interrupt fire and look at the uh if it's changed if it's rotated but you could also implement inside the fpga a

peripheral that did that for you um so tim ansell is someone that speaks a lot about the power of these open source tools and FPGAs. And while they're not as fast as the ASIC, as a hard core, we have the flexibility to kind of tackle the problems from a different angle, right? Rather than having like the software one really fast on a fast core, you might put some of that complexity into the peripherals that are defined inside the FPGA. Like I think the, like a quadrature encoded pulse decoder is a good example of that, where normally you have to make sure your CPU is running fast enough and not getting interfered by other software to be able to like accurately comp the pulses, or you could, you know, define some hardware logic that did that for you. Yeah, and I've done that.

I've done it both ways. And definitely having the chip do it is the way you do it. And having a quadrature decoder on the FPGA and then a counter that the thing reads was really nice. Yeah, so I think those are kind of good examples because if you get a board like the Orange Crab and you get Linux on there, 100 megahertz isn't super fast. But you can now, if you have a bigger FPGA, you can actually have multiple cores.

But it's kind of like maybe instead of having a CPU running at a gigahertz, what you need is like a set of custom peripherals, right? Or things like software PWM, right? You know, like on a Raspberry Pi, you can bit bang PWM or, you know, with a much slower core in FPGA, you could have like a bunch of different hardware blocks that generated the pulses, right? So there's a bit of a different way of tackling the problems.

But I think the key thing here is we now have open source free software tools that allow us to do FPGA development free of like the kind of awful vendor tools. And also now there's more and more open source IP, which is to say the building blocks. So things like, you know, Ethernet controller, DRAM controller, even things now like USB 3 controller and PCI Express controllers, you can drop into that FPGA. And in particular, one of the things that Tim Mansell talks a lot about is using Python for chip design. And that's possible because of this framework called MeeGen. And that's what we were using inside of that Linux on Linux project. So if you actually dig into it, what it is, is actually Python code.

So it's not Verilog, it's not VHDL, it's actually Python code. And that was really useful. So like me and several other people, we sat down at SuperCon and we were like, let's get Linux booting on the batch by the end of the weekend. And I think what really helped us do that was by using LightX and MeeGen.

It was not Verilog that we were looking at, it was Python code. So for example, we had a DRAM chip on the board that wasn't already supported in LIDEX. But we were able to just, like, inherit the Python class for, like, the DRAM chip and then change some of the parameters based on the timings and the data sheet for our DRAM chip. And then it worked. So we didn't have to sit down and write our own DRAM controller in Verilog. And I think, especially for me, I don't come from a chip design background, but I'm very familiar with Python.

It was much easier for me to look at and also leverages like the object-oriented nature of Python. So I think things like LightX and these OpenFPGA tools are really opening up chip-level design to a lot more people. There's so much you've shared. I mean, my brain is exploding with ideas and Linux and FPGAs. But you do have one more major role in your life, and that's Osh Park. You work for them. A, what do you do for them?

And B, how is Osh Park doing? So I design open source hardware projects. Some of the bigger projects that I've worked on over the past couple years was conference badges for the Open Harbor Summit. So back in 2018, we did a fun e-paper badge that had an ESP32 on it. So you could connect to it and change what it said on the screen. And worked with a couple people from the community on that, Michael Welling and Alice Camilo. And that was a really cool example I thought of using online collaboration. So when it started, I just posted on Hackaday.io, which is an awesome site if you're working on hardware projects, saying like, hey, we're going to make the badge for the Open Harbor Summit this year. And started going through some different ideas I had, got feedback from people like, you know, what should I do for the battery?

You know, what sort of, you know, wireless should I use and these sorts of things. And it went back and forth for a while. We thought about using the Regato Nordic module and then ended up going with the ESP32. So it was a really nice way to go back and forth. And on that project, ended up getting several people involved from around the world working on it. So that was a lot of fun and learned a lot and in the end uh ended up having to stay up all night and program like all 300 badges like the before the event um but it was it was like rewarding and fun and everyone everyone was excited about it because one it was e-paper and two we actually had microPython running on it.

So you could actually write little Python programs and transfer them over Wi-Fi onto the badge. Like some person made like a magic eight ball app because there was an accelerometer on there. So you could shake it and then it would give you a different answer. So for the most recent Open Harvest Summit 2020, me and Michael and Alex, we kind of got the group back together, kind of kicked it off at Supercon, got some people into a room and discussed ideas for badges. And this sort of worked out really well. It was like a great collaborative project where at Osh Park we made the boards and Spark Fund did the assembly. But actually it was great digi key actually um because after 2018 um i talked to some people from digi key and they're like if you do this again just let us know we'll give you parts i was like oh cool because that

was the main reason i didn't go with um the the regatta nordic module because it was it was a bit more expensive than the esp but uh so for the 2020 badge, we decided to go with a wristwatch form factor inspired by, in Germany, there's this hacker camp called the Chaos Communications Congress. And every year there's this convention, and every four years they have this big hacker conference or camp. And they had these cool wrist responses that had like bluetooth on them and a little oled so we thought that was really neat so we kind of made our own version

for the open hardware summit unfortunately the open hardware summit ended up being virtual this year um so it was supposed to be the second week of march um so it was in a very like um it was a very difficult time to know if we should do it or not. And we ended up canceling it. So all the badges are still unfortunately at NYU, but eventually we'll be able to send them out to the people that were registered. But yeah, that was a great example of a project where got some people together from around the world.

And we all collaborated on an open Harvard project, which produced like a fun little Nordic-based CircuitPython running wristwatch that people will be able to hack on when they finally get them. The other things I do with Oshpark is I'm very passionate about KiCad. And after a conference earlier this year called FOSDEM, there was a developer day there, and I've gotten more involved in key kind of development. And it's really fun to run the development version and see all the features that are getting added and try and add in some of the things that I've been wanting to see in there. One of which is we have this really fun black circuit board we call after dark. So it's instead of fiberglass, it's normally like tan sort of color. So we were able to find fiberglass that was dyed black.

And then we use a clear solder mask over it, which means you can see all the circuitry. So you see the copper traces everywhere, but they're protected by the solder mask so it's a really nice aesthetic it also it's pretty interesting because you can see all the routing that's on the circuit board so we call this after dark and one of the things i was working on this year was trying to figure out how to render that correctly in key cad because it didn't really have the concept in there of how to differentiate between copper that was under a clear mask, which it looks like normal copper color versus when it's not underneath the solder mask where it's plated with gold finish. So that was an interesting thing to work on.

And eventually, with the help of one of the key CAD developers, it's now able to render that correctly, which hopefully we'll link to it in the show notes. But the Hackaday Super Conference batch that ran Linux, the reason it was able to do that was there was a little board in the back that had a DRAM chip on it. And that board was done in After Dark, so you can see all the little traces between the chip and the connector on it. The other thing that's been new um i was almost going on two years now but it's it's an interesting

journey to learn more about it is a flexible circuit so about two years ago we started offering two layer flex um in small quantity so you can get for like, um, $15, like a small flexible circuit board, which has really opened up, I think a lot of ideas for experimentation. Um, one of the things I've been working on is, is, uh, so unfortunately a lot of my ideas are based around conferences. Um, so one of my ideas was a little, um, LED or a little battery tester. So the idea is you have a little flexible PCB that wraps around both ends of the battery, like maybe a coin cell or a little battery tester. So the idea is you have a little flexible PCB that wraps around both ends of the battery, like maybe a coin cell or a AA,

and then it can light up an LED to show if it has charge left in it or not. So that was kind of interesting because by using cutouts in the PCB, you can allow it to better wrap around the battery. So that's kind of a fun thing that I've been iterating on. Though, yeah, unfortunately, a lot of my hardware projects have been based around business cards or conference badges. So it's not been as much of a need this year for that, but definitely been getting a lot

more involved in KiCad. And I think with version six which will be coming out um towards the end of the year um it's really getting the point now where it's as capable as as professional packages i think like altium and and i think definitely for people using eagle it has all the has a lot of the capabilities that people might be used to in eagle uh, I'm sure that you could go on and I have more questions from listeners and from myself, and hopefully we can link to it in the notes, is it is now possible to design a completely open chip all the way down to the transistor level. And this is because of a project out of Google and a fab called Skywater in Minnesota, where they've opened up the process development kit, which is like the most specific sensitive details of how the transistors are made. So that's all open now.

In addition to that, they also, if you design an open source chip, they will manufacture it for free as part of these free manufacturing runs they're doing. So if he put a link into it, to Tim's talk, Tim Ansell's talk,

but really exciting. You can now make a system that's completely open all the way down to the transistor level, like, you know, the atoms that are constructing the gates level. I think that's really exciting. That's new. We've never seen that before. So I think we're going to see a lot more interesting things happening in terms of open source chip design. Awesome. It has been wonderful to talk to you. Our guest has been Drew Fastini, embedded Linux developer, board member of Beagleboard.org foundation, and open source hardware designer at Osh Park. Thanks, Drew.

Yeah, thank you. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack for some questions and for Drew's mic. And thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm. And now a thought to leave you with, mostly for the US folks, vote. I don't care what

you vote for, just vote. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.