Screaming in the Cloud - Learning eBPF with Liz Rice

Episode Date: May 2, 2023

Liz Rice, Chief Open Source Officer at Isovalent, joins Corey on Screaming in the Cloud to discuss the release of her newest book, Learning eBPF, and the exciting possibilities that come with eBPF technology. Liz explains what got her so excited about eBPF technology, and what it was like to write a book while also holding a full-time job. Corey and Liz also explore the learning curve that comes with kernel programming, and Liz illustrates why it’s so important to be able to explain complex technologies in simple terminology. About LizLiz Rice is Chief Open Source Officer with eBPF specialists Isovalent, creators of the Cilium cloud native networking, security and observability project. She sits on the CNCF Governing Board, and on the Board of OpenUK. She was Chair of the CNCF's Technical Oversight Committee in 2019-2022, and Co-Chair of KubeCon + CloudNativeCon in 2018. She is also the author of Container Security, and Learning eBPF, both published by O'Reilly.She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, competing in virtual races on Zwift, and making music under the pseudonym Insider Nine.Links Referenced:Isovalent: https://isovalent.com/Learning eBPF: https://www.amazon.com/Learning-eBPF-Programming-Observability-Networking/dp/1098135121Container Security: https://www.amazon.com/Container-Security-Fundamental-Containerized-Applications/dp/1492056707/GitHub for Learning eBPF: https://github.com/lizRice/learning-eBPF

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there's a problem.

With Chronosphere, you can shape and transform observability data based on need, context, and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io slash cory-quin. That's chronosphere.io slash cory-quin. And my thanks to them for sponsoring my ridiculous nonsense. Welcome to screaming in the cloud. I'm Corey Quinn. Our returning guest today is Liz Rice, who remains the chief open source officer with isovalent.

But Liz, thank you for returning suspiciously closely time to when you have a book coming out. Welcome back. Thanks so much for having me. Yeah, I've just, I've only had the physical copy of the book in my hands for less than a week. It's called Learning EBPF. I mean, obviously I'm very excited.

It's an O'Reilly book. It has some form of honeybee on the front of it, as best I can tell. Yeah, I was really pleased about that because EBPF has a bee as its logo. So getting a early honeybee as the O'Reilly animal on the front cover of the book was pretty pleasing. Yeah. This is your second O'Reilly book, is not it's my second full book so I'd previously written a book on container security and I've done a few short reports for them as well but uh this is the second you know full-on you can buy it in Amazon kind of book yeah my business partner wrote a practical monitoring for O'Reilly and that was such an experience that he got entirely out of observability as a field

and ran running to AWS bills as a result. So my question for you is, why would anyone do that more than once? I really like explaining things. And I had a really good reaction to the container security book. I think already by the time I was writing that book book I was kind of interested in ebpf and we should probably talk about what that is but I'll come to that in a moment yeah so I'd been really interested in ebf ebpf for quite

a while and I wanted to be able to do the same thing in terms of explaining it to people a book gives you a lot more opportunity to go into more detail and show people examples and get them kind of hands-on than you can do in a 40-minute conference talk. So I wanted to do that. I will say I have written myself a note to never do a full-size book while I have a full-time job,

because it's a lot. You do have a full-time job and then some. As we mentioned, you're the chief open source officer over at ISOVANLAND. You are on the CNCF governing board. You're on the board of OpenUK, and you've done a lot of other stuff in the open source community as well. So I have to ask, taking all of that together, are you just allergic to things that make money? I mean, writing the book as well on top of that, I'm told you never do it for the money piece. It's always about the love of it. But it seems like on some level, you're taking it to an almost ludicrous level. Yeah. I mean, I do get paid for my day job. So there is that. But I feel like that's

the only way to really write a book is in turn to wind up only to just do it for when someone else is paying you to for doing it viewing it as a marketing exercise it pays dividends but those dividends don't in my experience from what i've heard from everyone say pay off as royalties on book payments yeah i mean it's certainly you know not a bad thing to have that uh that income stream but it certainly wouldn't make you you know I'm not going to retire tomorrow on the royalty stream unless this podcast has loads and loads of people to buy the book.

Exactly. And I'm always a fan of having such a snack. I will order it while we're on the call right now having this conversation because I believe in supporting the things that we want to see more of in the world. So explain to me a little bit about what it is. Whenever you talk about learning X in a title, I find that that's

often going to be much more approachable than arcane nonsense, deep dive things. But one of the O'Reilly books that changed my understanding was Linux Kernel Internals, or Understanding the Linux Kernel. Understanding was kind of a heavy lift at that point because it got very deep very quickly. But I absolutely came away understanding what was going on a lot more effectively, even though I was so slow I needed a tow rope on some of it. When you have a book that started with learning, though, I imagine it assumes starting at zero with what's eBPF. Is that directionally correct or does it assume that you know a lot of things you don't? Yeah, that's absolutely right. I mean, I think eBPF is one of these technologies that is starting to be, particularly in the cloud-native world, you know, it comes up.

It's quite a hot technology. What it actually is, so it's an acronym, right, eBPF. The acronym is almost meaningless now, so it stands for Extended Berkeley Packet Filter. But I feel like it does so much more than filtering. We might as well forget that altogether. And it's just become a term, a name in its own right, if you like. And what it really does is it lets you run custom programs in the kernel so you can change the way that the kernel behaves dynamically. And that is, it's the superpower. It's enabled all sorts of really cool things that we can do with that superpower. I just pre-ordered it as a paperback on Amazon, and it shows me that it is

now number one new release in Linux networking and systems administration. So you're welcome. I'm sure it was me that put it over the top. Wonderful. Thank you very much. Yeah, of course. Of course. Now it's writing a book is one of those things that I've always wanted to do, but never had the patience to sit there and do it. Or I thought I wasn't prolific enough, but over the holidays this past year, my wife and business partner and a few friends all chipped in to have all of the tweets that I'd sent bound into a series of leather volumes. Apparently I've tweeted over a million words and yeah. Oh, so I have to write a book 280 characters at a time, mostly from my phone. I should tweet less was really the takeaway that I took from a lot of that, but that wasn't

edited. That wasn't with an overall theme or a narrative flow the way that an actual book is. It just feels like a term paper on steroids. And I hated term papers. Love reading, not the one to write it. I don't know whether this should make it into the podcast, but it reminded me of something that happened to my brother-in-law, who's an artist. And he put a piece of video art on YouTube. And for unknowable reasons, if you mistyped YouTube and you spelt it U-T-U-B-E, the page that you would end up at from Google search was a YouTube video. And it was, in fact, my brother-in-law's video. And people weren't

expecting to see this kind of art movie about matches burning. And he just had the worst comment. Like people were so mean in the comments, but he had millions of views because people were hitting this page by accident. And he had a card of sin of never reading the comments. You never, never break that rule. As soon as you do that, it doesn't go well. I do read the comments on various podcast platforms on this show because I always tell people to insult it all you want. Just make sure you leave a five-star review. Well, he ended up publishing a book with these comments, like one comment per page, and most of them were not safe for public consumption comments. And he just called it feedback. It was quite, quite something.

On some level, it feels like O'Reilly books are a little insulated from the general population when it comes to terrible nonsense comments, just because they tend to be a little bit more expensive than the typical novel you'll see in an airport bookstore. And again, even though it is approachable, Learning EBPF isn't exactly the sort of title that gets people to think that, ooh, this is going to be a heck of a thriller slash page-turner with a plot. Well, I found the protagonist unrelatable. It's not sort of the thing you're going to wind up seeing in the comments because people thought it was going to be something different. I know. One day I'm going to have to write a technical book that is also a murder mystery. I think that would be, you know, quite an achievement. But yeah,

I mean, it's definitely aimed at people who have already come across the term, want to know more. And particularly if you're the kind of person who doesn't want to just have a hand wavy explanation that involves boxes and diagrams but if like me you kind of want to feel the code and you want to see how things work and you want to work through examples then that's the kind of person who might I hope enjoy working through the book and end up with a passable mental model of how EBPF works, even though it's essentially kernel programming. So I keep seeing EBPF in an increasing number of areas, a bunch of observability tools, a bunch of security tools all tend to tie into it. And I've seen people do interesting things as far as cost analysis with it. The

problem that I run into is that I'm not able to wind up deploying it universally just because when I'm going into a client engagement, I'm there in a purely advisory sense, given that I'm biasing these days for both SaaS companies and large banks. That latter category is likely going to have some problems if I say, oh, just take this thing and go ahead and deploy it to your entire fleet. If they don't have a problem with that, I have a problem with their entire business security posture. So I don't get to be particularly prescriptive as far as what to do with it. But if I were running my own environment, it is pretty clear by now that I would have explored this in some significant

depth. Do you find that it tends to be something that is used primarily in microservices environments? Is it to effectively require Kubernetes to become useful on day one? What is the onboard path where people would sit back and say, ah, this problem I'm having, eBPF sounds like the solution? So when we write tools that are typically going to be some sort of infrastructure, observability, security, networking tools, if we're writing them using eBPF, we're instrumenting the kernel. And the kernel gets involved every time our application wants to do anything interesting, because whenever it wants to read or write to a file or send, receive network messages or write something to the screen or

allocate memory or all of these things, the kernel has to be involved and we can use eBPF to instrument those events and do interesting things. And the kernel doesn't care whether those processes are running, you know, in containers, under Kubernetes, just running directly on the host. All of those things are visible to eBPF. So in one sense, it doesn't matter. But one of the reasons why I think we're seeing eBPF-based tools really take off in Cloud native is that you can, by applying some programming,

you can link events that happen in the kernel to specific containers in specific pods in whatever namespace and get the relationship between an event and the Kubernetes objects that are involved in that event. And then that enables a whole lot of really interesting observability or security tools. And it enables us to understand how network packets are flowing between different Kubernetes objects and so on. So it's really having this vantage point in the kernel where we can see everything and we didn't have to change those applications in any way to be able to use eBPF to instrument them. When I see the stories about eBPF, it seems like it's focused primarily on networking and flow control. That's where I'm seeing it from a security standpoint. That's where I'm seeing

it from cost allocation aspects, because frankly, out of the box from a cloud provider's perspective, Kubernetes looks like a single tenant application with a really weird behavioral pattern. And some of that crosstalk gets very expensive. Is there a better way than either using eBPF and or VPC flow logs to figure out what's talking to what in the Kubernetes ecosystem?

Or is BPF really your first port of call? So I'm coming from a position of perspective of working for the company that created the Cilium networking project. And one of the reasons why I think Cilium is really powerful is because it has this visibility. It's got a component called Hubble that allows you to see exactly how packets are flowing between these different Kubernetes identities. So in a Kubernetes environment, there's not a lot of point having network flows that talk about IP addresses and ports,

when what you really want to know is what's the Kubernetes namespace, what's the application. Defining things in terms of IP addresses makes no sense when they're just being refreshed and renewed every time you change pods. So yeah, the Kubernetes changes the requirements on networking visibility and on firewalling as well on network policy. And that I think is, you don't have to use ebpf to create those tools but ebpf is a really powerful and efficient platform for implementing those tools as we see in zillium the only competitor i found to it that that gives a reasonable explanation of why random things are transferring multiple petabytes between each other in the middle of the night has been oral

tradition, where I'm talking to people who've been around there for a while. So I'm seeing this weird traffic pattern at these times of day. Any idea what that might be? And someone will usually perk up and say, oh, is it whatever job that they're doing? Great. That gives me a direction to go in. But especially in this era of layoffs, and as environments exist for longer and longer, you have to turn to a bit of a data center archaeologist that remains insufficient on some level. On some level, I'm annoyed with trying to understand or needing to use tooling like this that is honestly this powerful and this customizable and, yes, on some level, this complex in order to get access to that information in a meaningful sense. But on the other, I'm glad that that option is at least there for a lot of workloads. Yeah, I think that speaks to the power of this new generation of tooling. And the same kind of applies to security forensics as well, where you might have an

enormous stream of events, but unless you can tie those events back to specific Kubernetes identities, which you can use eBPF-based tooling to do, then how do you, you know, the forensics job of tying back where did that event come from? You know, what was the container that was compromised? You know, it becomes really, really difficult. And eBPF tools like Cilium has a subproject called Tetragon that is really good at this kind of tying events back to the Kubernetes pod or whether we want to know what node it was running on or what namespace or whatever. That's really useful forensic information.

Talk to me a little bit about how broadly applicable it is. Because from my understanding from our last conversation, when you were on the show a year or so ago of Memory Serves, one of the powerful aspects of it was very similar to what I've seen some of Brendan Gregg's nonsense do in his various talks, where you can effectively write custom programming on the fly to tell you exactly what it is that you need. Is this something that can be instrument once and then effectively use it for basically anything OTEL style? Or instead, does it need to be effectively custom configured every time you want to get a different aspect of information out of it? It can be both of those things. It depends. My least favorite, but probably

most accurate answer to hear. But think brendan did a really great he's done many talks talking about how powerful bpf is and built lots of specific tools but then he's also been involved with bpf trace which is kind of like a language for a high level language for saying what it is that you want bpf to trace out for you. So a little bit like, I don't know, Ork, but for events, you know, it's a sort of scripting language. So you can have this flexibility. And with something like BPF trace, you don't have to get into the weeds yourself and do kernel programming, you know, in eBPF but also there's there's gainful employment to be had for people who are interested in that ebpf kernel programming because you know i think there's

just going to be a whole range of more tools to come you know i think we're you know we're seeing some really powerful tools with psyllium and pix Pixie and Parker and Kepler and many other tools and projects that are using eBPF. But I think there's also a whole load of more to come as people think about different ways they can apply eBPF and instrument different parts of an overall system. We're doing this over audio only, but behind me on my wall is one of my least favorite gifts ever to have been received by anyone. Mike, my business partner, got me a thousand piece puzzle of the Kubernetes container landscape where this diagram is psychotic and awful, and it looks like a joke, except it's not. And building that puzzle was maddening, obviously.

But beyond that, it was a real primer in just how vast the entire container slash Kubernetes slash CNCF landscape really is. So looking at this, I found that the only reaction that was appropriate was a sense of overwhelmed awe slash frustration, I guess. It's one of those areas where I spend a lot of time focusing on drinking from the AWS firehose, because they have a lot of products and services, because their product strategy is apparently yes, and they're updating these things in a pretty consistent cadence, mostly. And even that feels like it's multiple full-time jobs shoved into one. There are hundreds of companies behind these things,

and all of them are in areas that are incredibly complex and difficult to go diving into. EBPF is incredibly powerful, I would say ridiculously so, but it's also fiendishly complex, at least shoulder surfing behind people who know what they're doing with it has been breathtaking on some level. How do people find themselves in a situation where doing a BPF deep dive makes sense for them? Oh, that's a great question. So first of all, I'm thinking,

is there an AWS jigsaw as well? Like the CNCF landscape jigsaw. There should be. And how many pieces would it have? No, because I think the CNCF at one point hired a graphic designer, and it's unclear that AWS has done such a thing because their icons for services are, to be generous here, not great. People have flashcards that they've built for what services this logo represent. I haven't a clue in almost every case because I don't care in almost every case. But yeah, I've toyed with the idea of doing it. It's just not something that I'd ever want to have my name attached to. Unfortunate.

But yeah, I want someone to do it and someone else to build it. Yes. Yeah. It would need a refresh every five minutes as they roll out a new a new service right because given that given that it appears from the outside to be impenetrable it's similar to learning vi in some cases but where oh yeah it's easy to get started with to do this trivial thing now step two draw the rest of the freaking owl same type of problem there it feels off-putting

just from a perspective of you must be at least this smart to proceed, how do you find people coming to it? Yeah, there is some truth in that, in that beyond kind of hello world, you quite quickly start having to do things with kernel data structures. And as soon as you're looking at kernel data structures, you have to sort of understand, you know, more about the kernel. And if you change things, you need to understand the implications of those changes. So yeah, you can rapidly say that eBPF programming is kernel programming. So why would anybody want to do it? The reason why I do it myself is not because I'm a kernel programmer. It's because I wanted to really understand how this is working and build up a mental model of what's happening when I when I attach a program to an event and what kind of things can I do with

that program and that's the sort of exploration that I think I'm trying to encourage people to do with the book that yes there is going to be at some point a pretty steep learning curve that's kernel related, but you don't necessarily need to know everything in order to really have a decent understanding of what eBPF is and how you might, for example, you might be interested to see what BPF programs are running on your existing system and learn why and what they might be doing and how where they're attached and what use could that be okay falling down that looking at the process table once upon a time was a heck of an education one week when I didn't have a lot to do and I didn't like my job in those days where what is this Avahi demon that's constantly running MDNS

forwarding who would need that and sure enough that tickled something in the back of my mind when I wound up building out my networking box here on top of BSD. And oh, yeah, I want to make sure that I can still have discovery work from the IoT subnet over to whatever it is that my normal devices live. Ah, that's what that thing always running for. Great. And that one case, almost never needed in other cases, but awesome. You fire up a Raspberry Pi, it's why are all these things running when I'm just want to have an embedded device that does exactly one thing.

Wow. Computers have gotten complicated. I know. It's like when you get those pop-ups on, well, certainly on Mac, and you get pop-ups occasionally that say, there's such and such a daemon wants extra permissions. And you think, I am not hitting that yes button until I understand what that daemon is.

And it turns out it's related to something completely innocuous that you've actually paid for, but just under a different name. Very annoying. So if you have some kind of instrumentation like tracing or logging or security tooling that you want to apply to all of your containers, one of the things you can use is a sidecar container approach. And in Kubernetes, that means you inject the sidecar into every single pod. Yes, of course, the answer to any Kubernetes problem appears to be, have you tried running additional containers? Well, right. And there are challenges that can come from that. And one of the reasons why you have to do that is because if you want a tool that has visibility over that container that's inside the pod,

well, your instrumentation has to also be inside the pod so that it has visibility because your pod is by design isolated from the host it's running on. But with eBPF, well, eBPF is in the kernel and there's only one kernel, however many containers we're running. So there is no kind of isolation between the host and the containers at the kernel level. So that means if we can instrument the kernel, we don't have to have a separate instance in every single pod. And that's really great for all sorts of resource usage. It means you don't have to worry about how you get those sidecars into those pods in the first place. You know that every pod is going to be instrumented if it's instrumented in the kernel. And then for service mesh,

service mesh usually uses a sidecar as a layer 7 proxy injected into every pod. And that actually makes for a pretty convoluted networking path for a packet to sort of go from the application through the proxy out to the host back into another pod through another proxy into the application what we can do with ebpf we still need a proxy running in user space but we don't need to have one in every single pod because we can connect the networking namespaces much more efficiently. So that was essentially the basis for sidecarless service mesh,

which we did in Cilium, Istium, and now we're using a similar sort of approach with ambient mesh so that, again, you know, avoiding having the overhead of a sidecar in every pod. So that, you know, seems to be the way forward for service mesh as well as other types of instrumentation avoiding sidecars on some level avoiding things that are kubernetes staples seems to be a best practice in a bunch of different directions it feels like it's a it's an area where you start to get aligned with the idea of service mesh yes that, that's how I pluralize the term service mesh. And if people have a problem with that, please, it's imperative you not send me letters about it. But this idea of discovering where things are in a variety of ways within a cluster,

where things can talk to each other when nothing is deterministically placed, it feels like it is screaming out for something like this. And when you think about it, Kubernetes does sort of already have that at the level of a service. You know, services are discoverable through native Kubernetes. There's a bunch of other capabilities that we tend to associate with service mesh, like observability or encrypted traffic or retries, that kind of thing. But one of the things that we're doing with Cilium in general is to say, but a lot of this is just a feature of the networking, the underlying networking capability.

So, for example, we've got a next-generation mutual authentication approach, which is using spiffy IDs between an application pod and another application pod. So it's like the equivalent of MTLS, but the certificates are actually being passed into the kernel and the encryption is happening at the kernel level. And it's a really neat way of saying we don't need we don't need to have a sidecar proxy in every pod

in order to terminate those tls connections on behalf of the application we can have the kernel do it for us and that's really cool yeah on some level i find that it still feels weird because i'm old to have this idea of one shared kernel running a bunch of different containers. I got past that just by not requiring that workloads that need to be isolated having containers run on the same physical host. I found that, for example, running some stuff even in my home environment for IoT stuff, things that I don't particularly trust run inside of KVM on top of something as opposed to just running it as a container on a cluster. Almost certainly stupendous overkill

for what I'm dealing with, but it's a good practice to be in to start thinking about this. To my understanding, this is part of what AWS's Firecracker project starts to address a bit more effectively. Fast provisioning, but still being able to use different primitives

as far as isolation boundaries go. But on some level, it's nice to have to think about this stuff, but that's dangerous. Yeah, exactly. Firecracker is a really nice way of saying, actually, we're going to spin up a whole VM. When I say whole VM, we don't need all of the things that you normally get in a VM. We can get rid of a ton of things and just have the essentials for running that Lambda or container

service. And it becomes a really nice lightweight solution. But yes, that will have its own kernel. So unlike, you know, running multiple kernels on the same VM where, sorry, running multiple containers on the same virtual machine where they would all be sharing one kernel. With Firecracker, you'll get a kernel per instance of Firecracker. The last question I have for you before we wind up wrapping up this episode harkens back to something you said a little bit earlier. This stuff is incredibly technically nuanced and deep. You clearly have a thorough understanding of it, but you also have what I think many people do not realize is an orthogonal skill of being able to articulate and explain those complex concepts simply and

approachably in ways that make people understand what it is you're talking about, but also don't feel like they're being spoken to in a way that's highly condescending, which is another failure mode. I think it is not particularly well understood, particularly in the engineering community, that these are different skill sets that do not necessarily align congruently. Is this something you've always known, or is this something you've figured out as you've evolved your career that, oh, I have a certain flair for this? Yeah, I definitely didn't always know it. And I started to realize it based on feedback that people have given me about talks and articles I've written. And I think I've always felt that when people use jargon or they use complicated language or they kind of make

assumptions about how things are, it quite often speaks to them not having a full understanding of what's happening. If I want to explain something to myself, I'm going to use it straightforward language to explain it to myself so I can hold it in my head. And I think people appreciate that. And you can get really, you know, you can get quite in depth into something if you just start step by step, build it up, explain everything as you go along the way. And yeah, I think people do appreciate that. And I think people, if they get lost in jargon, it doesn't help anybody. And yeah, I very much appreciate it when people say that, you know, they saw a talk or they read something I wrote and it meant that they finally grokked whatever that concept was that I was trying to explain.

I will say at the weekend, I asked chat GPT to explain DNS in the style of Liz Rice. And it started off, it was basically, hello there, I'm Liz Rice, and I'm here to explain DNS in very simple terms. I thought, okay. Every time I think I've understood DNS, there's another level to it. I'm pretty sure there is a lot about DNS that I don't understand yet. So, you know, there's always more to learn out there. There certainly is. I really want to thank you for taking time to speak with me today about what you're up to. Where's the best place for people to find you to learn more? And of course, to buy the book. Yeah. So I am Liz Rice pretty much everywhere, all over the internet. There is a GitHub repo that accompanies the book. So you can find that on GitHub, Liz Rice slash learning

dash eBPF. So that's a good place to find some of the example code and it will obviously link to where you can download the book or buy it because you can pay for it you can also download it from isovalent for the price of your contact details so there are lots of options excellent and we will course, put links to that in the show notes. Thank you so much for your time. It's always great to talk to you. It's always a pleasure. So thanks very much for having me, Corey.

Liz Rice, Chief Open Source Officer at Isovalent. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you have somehow discovered this episode via Googling for knitting projects. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying.

The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.