The Changelog: Software Development, Open Source - Where DOESN’T curl run (Friends)

Episode Date: June 21, 2024

Daniel Stenberg shares his guiding principles for BDFL'ing curl, gives us his perspective on the state of the internet, talks financial independence, ensuring curl won't be the next XZ & more!

Welcome to Changelog and Friends, a weekly talk show about the life of a BDFL. Thanks to our partners at Fly.io. Launch your app, close your users, too easy. Learn how at Fly.io. Okay, let's talk. What's up, friends? I'm here with a good friend of mine, Firas Aboukadeje. Firas is the founder and CEO of Socket. Socket helps to protect some of the best engineering teams out there with their developer-first security platform. They protect your code from both vulnerable and malicious dependencies. So we've known each other for a while now, Faras. Well, let's imagine somehow I've landed myself a Vercel. And because I'm a big

fan of you, I understand what Socket is, but I don't know how to explain it to anybody else there. I've brought you into a meeting. We're considering Socket because we want to secure dependencies. We want to ship faster. We want everything that you promise from Socket. How do you explain Socket to my team at Vercel? Yeah, Socket is a developer first security platform that stops vulnerable and malicious open source dependencies from infiltrating your most critical apps. So we do that by focusing on real threats and keeping out all the types of risks that are out there in open source dependencies. Everything from malicious dependencies, typo squad attacks, backdoors, risky dependencies, dependencies

with hidden behavior. There's all kinds of risks out there. A lot of reasons why a dependency might be bad news. And Socket can help you as a developer, just keep all that out of your app and keep things nice and clean and pristine amongst your dependencies. I saw recently Dracula. I'm a fan of Dracula. I don't know about you, but I love that theme. Big fan of Zanaroccia. And I saw there was like a misspelling there. And so because Dracula is installed on VS Code and lots of different places, I saw there was a typo squat sitting there that had different intentions than obviously Dracula did. Is that an example of what you mean? Absolutely. Yeah. Dracula, that's a perfect example. It's super common these days to see that type of an attack

where you see a common dependency that you have an attacker just pretending to be that dependency, typoing the name of it by one letter and then trying to get unsuspecting developers to install it. Unfortunately, we're seeing more and more of these types of attacks in the community and they're taking advantage of the trust in open source. As developers, we need to be more aware of the dependencies we're using and make sure that we're not pulling in anything that could risk the data of our users or cause a big breach at our companies. And so part of that is obviously being more careful and asking questions and looking more carefully at the dependencies we use. But also part of that is tooling. It's really a hard problem to solve just on your own as a single developer. And so bringing in a tool like Socket

can really help automate a lot of that work for you. It just sort of sits there in the background. It's really, really quiet. It doesn't create a lot of noise. But if you were to pull in something that was backdoored or compromised in some way, we would jump into action right in the PR or right in your editor,

or even as early as you browse the web. We have a web extension that can actually give you information if you're looking at a package that's dangerous or if you're browsing Stack Overflow and you see somebody saying, hey, just install this dependency to solve your problems. A lot of times even that can be a way to get the attacker's code onto your machine. So Socket jumps in at all those different places and can tell you if something is dangerous and stop you from owning yourself. Yes.

Don't get yourself owned. Use Socket. Check them out. Socket.dev. Big fan of you, Firas. Big fan of what you're doing with Socket. Proactive versus reactive, to me, is the ultimate shift left for developers. It is totally developer first.

Check it out. Socket.dev. Install the GitHub app. Too easy or book a demo. Once again, socket.dev. That's S-O-C-K-E-T.dev. So Daniel, this is your first time being on Teams like on friends. Now we're friends of many years now, but mostly we just talk about curl and i can't help but talk about curl when i talk to you because it's such a big part of your life for one but then back in march just a few days after my birthday you turned 26 so congrats daniel turned 26 no curl I'm 28. I was going to say.

Dang, if curl's 26, Daniel, I'm not going to ask. Good job, Daniel, being 26. Yeah, happy, what do you call that? Post-dated? Belated. Happy belated birthday to curl. Everyone's favorite. I don't even know what you call it.

I was going to call it a command line. Internet fetcher, but that's not what it's called. I mean, it's so much more than that. Daniel, what's your, what do you call curl these days? So many things. Yeah, I usually call it like internet transfer tool or something. Internet transfer tool. Okay.

But yeah. That's not very sticky though, is it? No, it's not. That's why, I mean, I couldn't think of something sticky right off the bat. I was like, what is curl? I know it's like a- Maybe internet plumbing tool. I couldn't think of something sticky right off the bat I was like what is curl I know it's like

maybe internet plumbing tool I don't know HTTP client maybe well it's more than just HTTP of course yes exactly it does so much more

than HTTP that's why I go with internet transfers but that's also a little bit vague and hard to sort of grab your head right

internet transfer tool just doesn't sound so cool I mean everybody knows what curl is though it's kind of part of the substrate of the internet at this point and hard to wrap your head. Internet transfer tool just doesn't sound so cool. I mean, everybody knows what curl is, though. It's kind of part of the substrate of the internet at this point. I mean, it's 26 years old. Yes, it's been around for a while. That's a good way to put it, too.

The internet substrate mechanism thing. I don't know. That's a good thing, too. Substrate. Yeah. So this is your fifth time on our shows. First time on Changelog and Friends. We usually do it around anniversaries, birthdays, 17 years of Curl.

I remember it was one of our episodes. And it's been three years since we've had you on. So why don't you catch us up? What's new in Curl? Yeah, three years is a long time in most aspects. And it certainly is in curl time as well. Even though I often have that reaction, I've probably mentioned it before,

that people say that they used curl 10 years ago and they use curl today. And there's no difference. Has anything happened at all? But I did a quick check the other day and in the last three years, we've added 21 new command line options. Oh my gosh. Do you know how many new command line options. Oh my gosh. Do you know how many total command line options

there are at this time? Yeah, I have a counter. So there's 263 of them. This is why it's hard to define what curl does because I mean, so many things it can do. Yeah, and of course, and then people like to say that, is that a good thing really?

And no, it's not really a good thing to have a lot of command line options because, of course, you get lost among them. And very few people actually use more than, I don't know, 10, 15, 20. But, of course, everyone uses their own subset of them. So they all serve a purpose for some users. But sure, so it becomes a problem to document and for people to understand and find them and so on.

But at the same time, people come up with new things they want to do with this internet transfer thing. And then we have to, how do you add the new thing? Yeah, you have to add a new option. Like we have stuff like, oh, you can set one of them. For example, there's a header field in the IPv4 header. And it's called type of service. You can set it now. It exists in IPv6 too, it's called traffic class. It's just a numeric field in the IP header. Okay, now we can set it with curl because some people like to do that. Most people won't. But of course, we have to add a command line option to be able to set it if you

want to set it, and stuff like that. So there's often these days when we add new options, they're all these niche things for some users. Right. Yeah, it's the eternal struggle, I guess, of somebody who's writing useful software is how to maintain and evolve it over time that continues to serve new needs but doesn't trample down what people came for in the first place. And I think when it comes to command line options, like you said, it's the documentation, it's the man page, it's the help

where it really does get in the way. Other than that, I mean, it's invisible. You know, I use curl daily and I probably use five. Exactly. And I don't know any of the new ones they don't get my way they don't bother me but they're useful to somebody so so when we exactly when we do that properly they're not in the way for those who won't care about them so they'll just sit somewhere and the one day five years down the line when you want to do one of those

things you figure it out and then you find out which option to use and then you do it and then you forget about it again and move on with your life and that's how it's supposed to work right right so you've added a bunch of command line options i know you've been very ahead of the game and continuing to work on http3 stuff what's the state of h3 these days yeah so, so just over these three years, H3 has really grown a lot, both in Curl and in general on the web. So now we support much

more H3 in Curl. And, well, it's still complicated to build Curl with H3 support because of all the different components involved and their different maturity levels and the different APIs and so many different pieces that needs to work together. So it's a juggle.

If you install curl in a Linux jister today, for example, I don't think a single one actually enables HTTP3 by default just because of the weird mix of different dependencies that you need to add up for it to work. But we support it now, as we say, non-experimentally

with a set of dependencies called the ng-tcp2 quick library. That rolls right off the tongue. Yeah, it's a mouthful. The best part is that it then requires the ng-htp3 library too. So that's, yeah, say that fast three times,

ng-tcp2 and ng-htp3. So yeah, and then when you enable http3, you can use the dash dash http3 option, of course, with curl. So then you can actually try it against you can run it against any server really because it'll try h3 in parallel with h2 and h1 oh really so it'll just race them all against each other and the one that wins runs pretty much so you would assume that h3 would always win but i guess not huh no because it's such a problem with the it's but since it's

based on quick which is done over udp there's a certain amount of blockage going on in the world so in a lot of cases it actually doesn't work just because your company organization or something in between you and the server decides that udp is bad we shouldn't let it through. Yeah, just firewalls or whatever it happens to be. Exactly. Or just someone decided along the way that you shouldn't do this. So therefore, it's a lot of that trying out if it works, and then you have to have a fallback and use the older versions if that new one doesn't work.

So you fire off all three at once? Not really. Well, because H2 and H1, you can negotiate on the same connection. So we do both at once. I see, you negotiate that one. You can't negotiate H3 because it's over UDP? Exactly.

And to complicate matters, we also do happy eyeballs. So we do IPv4, IPv6 old, and IPv4, IPv6 new. So we'd actually do four attempts at once that's what they call it old and new yeah not v1 v2 i mean what the world no well i call it like that okay gotcha that's not the true nomenclature no well ipv6 is already hard enough to get i still understand it as a user not a implementer so yeah if they called it old and new, that'd be even worse than V1, V2.

Yeah, well, it's a complicated setup. And then we work with several other backends to enable build Libcurl with other backends too, a little bit depending on so that we can offer them on more platforms and with more different TLS backends pretty much. So it's a circus. We support four different Quake backends. That's what keeps Daniel busy. And we do that a little bit so we don't have to pick a winner. We can support all of them and see a little bit how they develop

and go and become good or bad because once when we pick them originally we don't really know if they're going to be the good one in the end even though the this i mentioned ng tcp2 that that library is actually created by the same guy team that created ng it's http2 library before. Another mouthful. But that's a very successful and very reliable library for H2. Get that guy on marketing team. Help him name some projects so we can pronounce them.

They like their NG stuff, but it's really, really hard to say. Very literal. Is IPv6 arriving? I mean, it's been like the slowest transition in history. Yeah, it's really slow still, and I don't know. Luckily, I don't really have to care. But

from my point of view, that's why we do happy eyeballs, meaning that we do both, pretty much. You say happy eyeballs? Yeah, it's an algorithm. They call it like that. Okay, I thought you said that the first time, but I wasn't sure if I misheard you. Yeah, it's actually an RFC. it like that. Okay, I thought you said that the first time, but I wasn't sure if I misheard you. Yeah, it's actually, yeah, it's an RFC. It's actually accessed in several versions,

but it pretty much means that we actually start the IPv6 attempt first, and then a few milliseconds later, you start the IPv4 attempt. Okay. And the one that wins, that succeeds first, that's the one you go with, and then you cancel the other one.

And it's happy eyeballs because you're watching watching both and whichever one comes back faster your the eyeballs are happier yeah i i don't know where the name comes from it's just a weird name but it's it's just i've sort of used it for so long so i'm just used to it but yeah i don't know why it's called happy eyeballs we need to do that research that's interesting i've never heard hopefully hopefully there will be some happy eyeballs in the end, I guess. I don't know. That's what I assume.

It's like, you're kind of looking at both and then you get happy when one returns a response or something. Could you give us a primer on IPv4 versus v6?

I know that, like, there was a terror that IP addresses will eventually run out and that's why IPv6. Like, how much do you know? Can you give us a primer on that and the state of it?

Well, that's still the case. So there's been a lot of patching and gluing things. So yes, we are running out of IPv4 addresses, and I think they are getting more expensive. So in particular, in certain areas, they really run out. So you have to be creative or you have to pay a lot of money to get IPv4 addresses. So I think that is the real case.

But I think also during this time, people have come up with new ways on how to work around the problem with different kinds of NATs and carrier-grade NATs and everything so that we can keep on doing this. We can extend our lifetime on IPv4 a little bit longer because most people, it turns out, don't really need their own IPv4 addresses.

So we can come up with new ways. But of course, that is then also kind of a blocker for doing new kinds of innovations on the internet. Because back in the at least 80s 90s people thought about doing things peer-to-peer right you knew you had an ip address in your client and you knew the ip address and the server and you could communicate between those two ip addresses nowadays you really can't because nowadays there are so many different layers and translations so you're not so that has but that's just the reality now i think that has made it so that the

ipv4 problem hasn't become that big so we we managed to survive on ipv4 pretty good anyway now we're like layering nets behind each other so you have you know this one ipv4 addresses representing so many networks yeah and and I think also another thing that has happened, especially compared to the 80s, 90s, that nowadays we build everything on top of HTTP instead. So we build so much protocol layers, pretty much much taller protocol stacks now than I thought we envisioned back in the,

well, when did they come up with IPv6? I think mid-90s or something. Yeah, I mean, it definitely radiates me. I remember I was in college, and so this would be around the turn of the century. 01 is when I graduated from high school. Yeah, exactly. And they already had IPv4,

and they're like, but the new thing is IPv6, and everyone's going to be using this because we're running out of addresses. And that was 25 years ago. Exactly. And the story is exactly the same. It's the exact same story. Yeah.

Well, if I was a person issuing IPv4 addresses for a price, I'd kind of like this scarcity. It's helping me out. I can just keep raising my prices and issuing them. It's not too bad if you're holding the keys, right? Right. Yeah, I guess it's more of a problem if you actually want to start something new today

and you really need to have that connectivity to people in the world and you really need IPv4 addresses. Maybe that is a problem. But at the same time, I think the entire internet infrastructure has also changed. We're doing everything over CDNs nowadays that we really did not do in the 90s and stuff like that. So we have changed how we do internet,

internetworking. Well, that was one of the pitches was like every device would have its own address, you know, and not just your house or behind some sort of firewall, but you'd actually have, your refrigerator would have its own address

and it would be publicly exposed. And that would be good for certain reasons, but obviously also bad for other reasons. Exactly. The privacy angle certainly was never discussed back then. So wait a minute, are you going to know that this is my fridge talking to your server? Yeah, we were pretty naive about privacy for a long time.

Right, exactly. Because at least when you're hiding behind a NAT, you don't know if it was my watch or my fridge or my printer that talked to you. Yeah, and especially if you're having multiple NATs. I mean, we have oftentimes an IP address that exposes like an area of a city, but that's all you know.

It's like, well, it's somewhere in Bennington, but we don't know who it is. And sometimes that's problematic if you have actual malicious actors out there who are trying to hide, but it's also really nice for normal people who just want their privacy and don't want to be tracked down to every single thing they're doing all the time.

Eh, they find other ways of tracking us. Oh, they do, yeah. Okay, so there's IPv4 v6, h3 h2, tons of more command line flags. Anything else that's super notable in curl that our listener would be

like oh a cool new thing that i can do now i couldn't do before well i think we've only done minor things really when it comes to the what happens in the end user layer we support tls 1.3 in more ways now for example you can, you can do it, you know, curl ships with Windows since a long time nowadays, right? So it's built into Windows. And then when Microsoft ships curl, they ship it built with S-channel, which is the Windows TLS library native in Windows. And nowadays, for example, we can do TLS 1.3 with that library, which is a fairly new thing.

So stuff like that. But I mean, most users won't notice. They'll just be happy that it'll actually survive longer and work better. But that's just one of those things that you don't really see or know about in the engine. We do more things that we support. Over the last few years, we've done quite drastic refactoring of our way of building protocol chains internally, I would call it. So how you stack different layers of doing protocols, like if you do HTTP, proxies, TLS, or doing protocol, TLS,

protocols, and TLS and proxies in many different layers. So nowadays we can do that in a more flexible way so that we can support more ways of creating protocol. Well, I call them protocol chains, pretty much setting up different, for example, doing different kinds of proxying, different kinds of protocols over different kinds of proxies in more combinations. Because that is also where we're going into the future, because nowadays we have so many different HTTP protocols

versions, and we have a lot of proxies, and people want to do, you know, you want to do HTTP 3 over HTTP 1 or HTTP 2 proxies, or you can do HTTP 1 or HTTP 2 over an HTTP 3 proxy, and, you know, you get a little confused in your head just thinking about that. But pretty much to be able to offer all those different combinations of protocol versions, it becomes an explosion in how to handle that. So we had to change our internals

so that we could build them dynamically in a better way so that the code could manage all of these new ways of building protocol chains. Tricky stuff. It seems like not only has curl changed recently, but the world around curl has changed quite a bit since the last time we talked as well. And I know you have written somewhat extensively

on the effect of LLMs on curl development. I'm trying to find it back in your backlog. The gist being, you're getting a lot more low-quality PRs, issues opened by robots, security fixes that are not useful. Remind me, because it's way back there now. Yeah, exactly. I had this,

I think the one I blogged most about was one particular security report from some user who basically just fed it into an LLM. He claimed that there was a buffer overflow and I asked him for clarification and he just came back over and over with very friendly saying. Yeah.

And trying to, and then gradually changing the nature of the bug over the time and then pretty much me concluding that I'm talking to an LLM here. Yeah, I've had a few of those and I think they're problematic in that. I mean, yes, they're crap,

but they're pretty good crap, at least from the start. They're trickier crap, like they look less crappy. The worst crap you can identify immediately, right, they're crap, but they're pretty good crap, at least from the start. They're trickier crap. Like, they look less crappy. The worst crap you can identify immediately, right? This is crap. Close it, move on. And it's sort of forget about it.

But when it sounds correct and it seems legitimate, you have to actually investigate, spend time and research. What is this? What is it talking about? And as I mentioned then then people then say that well it's me it's it's obvious that it's a i am talking to but i'm also used to talking to people who are using translation services right so maybe i'm talking to a guy who doesn't know a word of english right so he's feeding everything he's saying through in translation thing so i can't really judge his language as, it sounds a little bit

machine-like, but that could just be him speaking Korean into a voice thing that spits this out. I can't really just dismiss him just because his English is not perfect. It's questionable. Exactly. My English certainly is not perfect either. That's pretty good, Daniel. That's pretty good. So therefore, it becomes a challenge. So sure, they say something. It seems, you know, the AI kind of hallucinating. It seems reasonable.

Yeah, it could be right. Maybe, but it doesn't feel right. And then, you know, ask a few follow-up questions. And they're, oh, wait a minute. Yes, you're right. And then it provides more information. But the more information is also slightly off and not actually answering the questions.

And yeah, so it's been a few of those that, yes, they certainly are time suckers because they spend a lot of time and just in the end, it's just worthless crap. One of those were at least good enough that they mentioned that in the first blurb, they said, I asked Google Bard or whatever it is he said, and it found this security problem.

That was at least a very good hint that maybe that's not the best thing to ask for problems. Yeah. I did find the post. It was back in Januaryuary i love the title made me laugh when i read it and holy cow this is on page four you blog a lot daniel so much it's the i in llm stands for intelligence which i thought was a nice turn of phrase and you go ahead and write in detail about how it's basically just causing you nothing but pain right i mean

it's just additional time that you have to spend, and you're now negotiating with a large language model, and you're not sure if it is or not. You have to determine before you write it off. Exactly. Because obviously there's a user somewhere that is copy and pasting this into an LLM, probably. But

it takes a while until you're really sure about it. And also, maybe it is okay to use an Lm if it had been accurate if it had been a genuine problem then i wouldn't have minded at all i mean that would be fine but yeah it was a it's a was a bit of a struggle to get to that point they're just slopping up the curl factory you know they're just throwing their slop into the curl factory yeah it's certainly sort of gravel in the machinery when they do that and i also since security problems tend to be top priority right so then that's when we drop all other stuff and just focus on this because if it's a security problem it's worth sort of dropping the other stuff for right it's

it certainly trumps a lot of other work. Plus there's money in the game now too, right? With a lot of the security stuff. So that makes it more of an attractive target for people to... Exactly. Yeah, that's why they do it. Yeah, they certainly want the money. And of course, I ask for it a little bit by saying that we will give you money if you...

Yeah. I mean, report it. Well, you're incentivizing it because it's important. Yes, exactly. So I think in the end it is a win for us. But yes, we also get a fair amount of rubbish to work. People that just want to, they're basically

fuzzing every bug bounty they can and using whatever tool they can to scale horizontally. And LLMs is a tool to scale horizontally. I can email with Daniel without having to really think about lms is a tool to scale horizontally i can i can email with daniel without having to really think about it you know and i can try to get this bounty exactly i i think it works like that too so they can run their tools against a lot of projects and they just fire off those so if they're lucky they will get some bounty from x percent of those projects so many

negative uses i suppose right that are out there it's like you you want to see the positive but then you also see the this kind of thing which is good when used good but then it's like well you're just inundating folks with crap right yeah you know or versions of it pretty good looking crap too that That's the worst part. Minutia, I don't know, like more to sift through. Yeah, lipstick on the pig. Exactly. It's a little bit of a denial of service attack. At least it would scale up.

Now it's just been, I mean, it's been manageable because it's not gone to a level that is unmanageable. It's early days, Daniel. I mean, this is going to scale. Yeah, I mean, exactly. I can easily see how it could scale up and become a much bigger burden and nuisance than it is now have you considered fighting fire with fire i

mean you have these tools at your disposal yeah but yeah i really i really don't think i can do that in a effective way well that would go against some of the bdfl things you have in place? I'm also very wary about how I appear and what I respond to, because I know that, you know, if I'm being, I don't know, unpleasant or unfriendly, people hold that against me. And, you know, whatever we say on the internet, it'll live forever. So I always make an effort to stay on the right side of how to behave and be friendly and just be accurate and direct. But then, of course, I'll also stop it as early as I can. Well, yeah, maybe just as a higher quality or more sophisticated filter before things reach you. Yeah, exactly. So I think that's's and i think since we're using hacker

one as a service for this they also actually have other i can actually enable more filtering but in the past i've also had problems with that because then suddenly when i've had people actually submit a bunch of legitimate reports in in a fairly high, they got caught in that filter. So I also had the other way. So yeah, but I think it's also one of these things we just have to learn to adapt to. So yes, I think now we get some problems with it. We will add more filtering,

raise the bars a little bit to catch these a little bit better earlier. So we'll see. I'm sure we will find a fairly good balance. I do think some of those things that you could do negatively in that scenario might go against these BDFL guiding principles you've laid out, which, as you mentioned there, you want to come off kind, open, and friendly. That's number one of your 10 guiding principles for Curl as a BDFL.

Does it make sense to kind of go down that list or kind of dig into some of that? Is it important enough to you to sort of put out there? Yeah, absolutely. I think that is one of the things. I want to make sure that I'm not dismissive. I mean, again, because it might not be

an AI. At least in the first post, I don't know that it is an AI. It could just be that guy who just got an AI to help him actually phrase it or produce a proper report. And that's fine. So I don't want to be dismissive. I actually never want to be dismissive. I try to be. I mean, of course, it's a challenge when people are explicitly and deliberately very rude or just being very unfriendly.

And it's hard to not just bite off immediately. But I try to not to and just stick to the facts, answer the actual factual or technical question or details in whatever they're talking about and then not drag it on and stop it there so tell me more what is this uh these bdfl for those who haven't heard bdfl benevolent dictator for life daniel has been and continues to be benevolent and a dictator and yes spending his life on curl yeah full-time now for man for, man, half a decade, I think, something like that, full-time on curl, which is awesome. Yes, exactly. A little bit over five years now.

Yeah, a little over five years. And so as a BDFL, he has, like Adam said, created these guiding principles, which means he's thought deeply about it. What are some of the other ones? So Adam mentioned the first one, be open and friendly. What else do you aspire to as a BDFL? Yeah, what do I do? So of course, because people like to say that, sure, I'm a BDFL, which means sure, I'm the dictator. So I could do whatever I want in the

project, in theory, at least. But of course, there's this difference between being a dictator in a software project and in the country. It's probably that if people wouldn't like the project, they would just go away and do something else instead. Maybe fork it or at least not participate in the project. So there's every motivation to not be a bad dictator for the project. So no, I don't think I have anything that is strange. So sure, I think, for example, the quality of the products that we're shipping is one of the key things. And I know that that's one of the key things that people appreciate about Curl and Libcurl as a project, that we ship products that rarely cause people problems. I've had, I mean, a lot of people mentioned to me

that they never experienced bugs in curl, which of course I think is fun because we fix bugs quite frequently, but still, we still work hard on making sure that it's a good solid product and that we don't break behavior and we don't break user scripts. We don't break APIs at all.

So stuff like that so basically i want curl and the products we ship to be that i want it to be and work like it actually works now right it should be those pillars you should build you should be able to build whatever you want on top and they should just continue to work like this as they have been for a long time and they should continue working like that so that's what we really focus on in the project and i want to want it to focus on too and then of course we're old we're everywhere and then that makes us get a lot of eyeballs on what we do and how we do things so that's also why i want us to then for example do things properly open source wise so i want to want us to make sure that we do open source,

pretty much follow all the best practices you can imagine for doing open source, being open, being transparent, doing things the way you should do things open source. So if I would join a project or participate in a project, how would I want that project to be? And how would that be open source-y the way I like it? So I want Curl to be that. So a lot of that, and also not that be open source C, the way I like it. So I want curl to be that. So a lot of that, and also not only open source-wise, but also code-wise and protocol-wise,

because I know that then since we have been doing protocols and we are doing protocols, a lot of people will then copy the way we do it, either by copying our code or just following the way we do protocols or implement things. So therefore, it's also good and nice to be able to say that if you follow our way of doing it, it is at least thought through and should be a good way to do the communication like we do it, then it should be good. Yeah, yeah. I'd like to commend you on both of those fronts, especially I think on the open source leadership in terms of how you manage the project.

Because, you know, Ab and I have known you for a long time now, and we've been watching your work in the public space for many years. And you really, I think, are a guiding light for so many people because you have been around a long time and you do think deeply about these things and you have been able to dedicate so much of your personal time and now your full time to this project and just maintaining it and sustaining it and like community leadership a lot of times i will just look to say well how does daniel handle this because you've come across like many of the problems that so many people are going to come across it's like well here's how the girl folks handled it maybe it doesn't necessarily apply to you one-to-one but it's a great place to start right and every every project is unique so maybe it doesn't apply to everything right but i think also sometimes it's a big benefit of being an old project and have been around for so long time because we have had time to adapt and adjust and do things the way we should

do because obviously we didn't do everything right from the beginning. I mean, who does, right? But if you just stick around for long enough, you get time and the ability and chances to just fix those and make sure that, sure, that didn't work. Let's do it this way instead because this is the better way. So over time, a lot of things just fall into place and end up being decent ways to do things, decent approaches, concepts and policies and everything.

And you're right about it. That's the other thing is there's probably other people who are also making long-term decisions and managing a project for a long time. But like I said, you blog profusely. Is that the right word?

Maybe that sounds like too much. What's the word I'm thinking of? Profusely. He refuses to stop. No, that's not the word I'm thinking of, but we'll just roll with it. Just documentation. I know that's also,

I'm leading it to the next, or down the field here. That's one of your other principles is the documentation, how much you write about these things. The fact that we know what your guiding principles are because you've taken the time

to write them down. I mean, that's how you lead, right? You lead by example, but you also lead by stating why you do what you do. Exactly. Yeah, that's, yeah. I like being able to also do that.

Not only so that I can inform sort of how I view things, and that's why I think this is so important. And so they sort of go hand in hand. And also, of course, this is not just me saying these things. I actually work on it pretty hard. So I hope that I'm sort of stating the obvious, really, because I've already delivered documentation for this to a pretty ridiculous level sometimes.

So it should be obvious that I'm focusing pretty hard on making sure that everything is documented to a very detailed level, for example. Quick follow-up on profusely. I'm now doubling down. I think that's a great word. Pouring forth with fullness or exuberance. Bountiful.

Exceedingly liberal. Giving without stint. I think you do blog profusely. I think I would pick the perfect word there. Yeah, that fits. Even on accident. I thought it had sort of like a negative connotation, like almost too much. But I'm not seeing that, at least in the dictionary.

Maybe if I go to Urban Diction urban dictionary i'll want to redact that but ah no let's let's stick to that let's stick with it a profuse blogger and you also want to remain independent i think this one might be of all those i mean maybe the hardest right because like this isn't necessarily entirely your decision is it no it's not entirely my decision and i think i've been a little bit fortunate that it has been possible to do it this way. So navigating on how to do open source and how to actually do it for a living, it's not easy and it's not obvious how to do that.

So I'm not judging anyone who's taking decisions on how to do that and making tough sort of choices on how to navigate forward to actually being able to sustain it somehow, because you need food on the table at the same time as you want to produce something. So I think I'm in a fortunate position when I can do it like this, and I'm happy to continue in this way. So I want it.

And when I've managed to do it for this long and for this amount of time, I imagine that I should be able to continue doing it as well. And I think it's pretty good because it makes us, we don't have to obey to anyone. We don't have, not even an umbrella organization or anything, no company, no one actually decides what we need to do. We just decide what we need to do we just decide what we need to do depending on what we think our users want or how the internet goes or what what's an internet transfer really and we can just base it on that and i think that is good because we don't have to bend to artificial whims

well friends i have something special for you because I made a new friend. Tamar Ben Sharkar, Senior Engineering Manager over at Retool. Okay, so our sponsor for this episode is Neon. And as you know, we use Neon, but we don't use Neon like Retool uses Neon. Retool needed to stand up a service called RetoolDB. Tamar can explain it better in this conversation, but RetoolDB is powered by Neon. Okay, they have a service called Fleets.

It is a service that manages enterprise level fleets of Postgres, serverless managed fleets of Postgres, serverless managed fleets of Postgres, and RetoolDB by Retool is powered by Neon Fleets. Okay, Tamar, take us into how Retool is using Neon for RetoolDB at large. So one big problem we had with Retool, we wanted users to have value, production value as soon as possible.

And connecting to a ProDB in a new tool is not something that people will do lightly. But they're much more likely to then dump a CSV into Retool. And so because of that, we said, okay, well, what if we just, you know, host databases on behalf of users, and then they can get spin up really fast. And we really saw that take off. The problem we had is we didn't have a big team, We couldn't set up a new team to support this feature. So what do we do? And so we were looking at, what are the options out there?

And we found Neon. Neon is a serverless platform that manages Postgres DBs. And so like, OK, that's interesting. Let's look further. What's kind of really unique about them is you really only pay for what you use, which is exactly the case that we have, right? Because we want to provide this to everybody.

Not everyone uses it. Not everyone uses it all the time. And so if we had to us manage a bunch of RDS instances, for example, right? Basically, we would have a whole infrastructure team to support, figure out, okay, what are they on? How do we do? Try to have some kind of greedy algorithm to get all the data in the fewest moments as possible.

Right? This is now a hard problem. That's not kind of a core value, right? A core value is kind of providing that database. And we don't want kind of going to we wouldn't know. We're not looking in for team. We don't want to kind of get it get in that game. I think what's really great is that, OK, well, one big kind of risk

when you think of going in third party is is a the cost. We're getting this free to all users. We have 300,000 databases right now, right? Like we can't, especially especially as we were rolling this out to begin users. We have 300,000 databases right now, right? Like we can't especially as we were rolling this out to begin with, right? We like didn't know for sure how it would, how people would respond, right? And, you know, we can't

all of a sudden have like a couple million dollars, you know, at the bank for this without kind of seeing kind of the activation that it has on users. So it's kind of obvious, but what was the appeal of Neon? What was really appealing to Neon, it spins out to zero. And so because of that, it really kind of reduces

the cost. And so really, it's really exactly only what we spend, and there's really actually not a way to actually spend less money even if we're hosting it ourselves. So you can be like removing all the people cost. Because let's say we use something like an RDS, we have to figure out ourselves basically what Neon is doing, right? How to bucket all the instances together,

how to bucket the usages to have as few instances as possible, right? To scale up and down depending on what's going on. And now we sort of don't have to worry about any of that part, but it's like kind of the cost benefit. And so really it kind of was out, you know, it's a win-win. Okay. Win-win. Always a good thing. I like win-win-wins, but okay, fine. Win-win. If it were not for Neon and their offering of fleets of Postgres and how they're essentially your serverless Postgres platform, where would Retool be at with RetoolDB without Neon? Well, we would have to have a, you know, at least a fully staffed team, you know, on call burden would be a challenge. You know, I think we have to spend a lot of time on, you know, making it sustainable. And that's,

you know, a whole, you know, other sets of concerns that are, that we don't have to think about. First of all, like, you know, it's a team of engineers, right? Which is not free. So it's everyone's salaries, right? So let's say probably a team, let's say, you know, eight to 10 people, you know, easily only focus on this. And then it's like, well, like, does the revenue of virtually be offset that the cost, even if just the engineers you know that's step one but I think even before then right like you'd have to set up this team before you even had a product you know databases and you know having them the way that Neon has them right like suspend to zero having you know warm spares that they're you

know ready instantaneously when you like log on to retool those things aren't free and even if we tried to do like an MVP, there's a kind of basic functionality that needs to exist that we all have to start from scratch. And that would be a huge commitment to this. And I think it would have come out like a year later because we'd have to do a lot more validation to know that it would have been worth it right before we started.

Here, we were able to quickly try it out, see that it was effective, and then grow it from there because the cost was very low. And that really gave us a lot of flexibility of also testing out different features and different flavors of it. Okay, so RetoolDB is fully powered by, backed by, managed by Neon. Neon Fleets, neon.tech slash enterprise. Learn more. We love Neon here at changelog we use neon for a postgres database we enjoy every single feature that tamar mentioned for retool db but we use it at a small scale a single database for our application they use it at scale one single engineer propped it up, managed it. That's insane. They would have never been able to do this without Neon.

RetoolDB would have cost more and may not exist without Neon. Okay, go to neon.tech, learn more, or neon.tech slash enterprise. Enterprise. I wonder what makes you do this, like really at your core as a human being. Like I get that you are probably in the best intellectual space to do it. You've done it for so long. But sometimes we do, they call them because. I do this because, right?

Maybe you do this because you want to see the substrate of the internet to be something pliable and usable. But like, really, why do you do this? Yeah, but exactly. To this level of detail and this level of quality. Now, I think the why has then changed over time.

And now, I want to use this platform that I had sort of already reached and I've already done it to this level. Now I want it to remain this. And I want these products and this project to keep on facilitating doing internet transfers and making sure that we do internet transfers properly and that everyone can do them in an easy and good fashion and sort of stable transfers and good way.

So then it just becomes a personal thing that I wanted to continue to be that and to continue to be a really good choice for all of these different services and platforms and tools and devices and languages because it's really cool. And I want it to remain that good and efficient. So it's a little bit, it isn't harder than that.

I just want it to be that good. And I want it to remain at that level of goodness. Yeah. What do you do? I want to be speaking morbid by any means. And your involvement in the project doesn't have to be a morbid scenario. But what do you do and what are you doing if that's your guiding principle personally so that it remains? What are you doing so that you don't have to be the BDFL forever in any scenario? First, I think what I do now is just leading how I want it to be done. So I'm leading by example, right? This is the way I think we should do the project and how we should do protocols and what I think we should do. And then I pretty much just make sure that if I would go on an extended holiday tomorrow,

everything necessary is already available, as in documented, provided, written down. I mean, I don't have any magic handshakes anywhere. There's no secret store that is necessary for anyone. I mean, sure, there are some credentials for logging onto servers and stuff like that, but there's no project secret anywhere. There's nothing hidden.

Everything is out there. Everything is documented, even to the level of how I do releases, how we do things, how we do governance in the project. So everything is there. So that's how I want it to be done, right? Show by example how I want things to run. And if I don't run it, everything is there for someone else to do it the same way or another way. But there's nothing that, if I go away tomorrow, there's nothing that preventing anyone else from taking over tomorrow. Right. What about the last mile of that contingency plan?

Like the credentials, the server logins, the DNS, maybe the password to log into the registrar? Yeah, I have those sorted out too, but at more personal level then. So I have more of a will-like situation so that I have relatives or mostly my brother who's into computers pretty much like I am. So he's most of my next of kin.

So if I would actually die, he would have access to all of that tomorrow. And then is there anybody else on the Curl team or community where it's like they're an obvious, Daniel's gone now, your brother has the credentials. Hey, Daniel's brother is going to pass those credentials onto this person who's going to take over leadership. Is there anything that's obvious like that or is it not so clear? It's not so clear. We don't have any,

there's not a dedicated heir or anything. So I wouldn't- No, I just mean like somebody who you'd have in mind where like, maybe if you're on your deathbed and you're like talking to your brother, like, hey, give the credentials to this person. Yeah, but my brother also has push rights in the project so he's also okay so maybe he's the guy then so he's already a maintainer at in curl as well so so he's i didn't

know that that's cool well he's only you know marginally involved but still i wasn't i give him too much credit you know not give him no but i don't want to dismiss it either. But I mean, he has maybe like 50 commits. I have 18,000. Right. But still, he's around and he knows the project. He knows a lot of things. And he understands these things. Yes, exactly.

Well, that's good to know. So you definitely, you've done your legwork on, I'm calling it contingency planning, legacy planning. I don't know what you call it. I think so. I hope so, yeah. Well, you never know, you know. And so these are things that I think people think about more as we do get older.

We start to think, well, you know. And actually, however silly it sounds, I get this question very often. Do you? Yes. So it's very good to actually have it sorted out because then I have a good answer when people ask me about it. Right. And people ask me about it because, yes, I'm the BDFL. And people, I think, actually, to a slightly larger degree than I deserve, people call it a one-man project or something.

Because I don't think it is because we're quite a number of people actually contribute. I do half of every commit, but there's a significant amount of changes done by others. But anyway, so people still have that, well, they think that about Curl and they think it's me. So if I'm gone, then surely Curl will die, right? So that's sort of the connotation with the question. Can you explain the financial arrangement arrangement like how did you get to financial independence and exactly how it works for you sure it's actually pretty easy so the

curl project is completely separate and standalone so i don't do business with the curl project i do business i support curl stuff but i do that separately so i sell curl services via this american company called wolf as a cell and my primary curl business is just support on curl pretty much a little bit insurance thing i sell a number of issues per year and I have a guaranteed response time. So my customers, mostly actually big American tech companies, they have paid me a yearly subscription basically and they file issues and I help them when they have issues. So I make sure that their curl use is uninterrupted and works well in their products. Usually companies where curl use is deemed important enough for them to do this. Even though I usually, of course, try to tell a lot of my potential customers that it's much better if they pay me to do that and rather spend time for their developers to try to figure out how to fix curl

because I probably do it much faster and much cheaper than they having to waste engineering time on figuring out how to do things, figuring out even just how to or just finding or fixing bugs even more. So that's what I do. And then of course, in addition to just supporting, it's also more feature development and contracts,

more working closely with the product development teams and how they use curl and debugging their applications using curl. Because very often it's not a problem with curl, but maybe in how they use curl or in you know in the area between it's hard sometimes and it's having a you know nda and contract in place makes them feel safe to share their code with me and you know they wouldn't dream to sometimes submit you know extensive explanations in a public bug report because they're scared that their never really special source code would be something special for the rest of the world to

see. So it works pretty good. Still a challenge to sell supports on something that is free because it's free. So why would I pay you when it's free? Well, we had a conversation in Twitter DM about this. I think it's actually the first DM you and I had, Daniel, October 9th, 2021. And I said to you, I said, are you aware of how crucial curl is to the Netflix architecture? And you said, I'm not.

I'm aware they use or use Libcurl for a few years, but I'm not up to speed with their current usage. Now, thanks to a prolific influencer slash developer slash persona in our industry, the Primogen, which I'll link up in the show notes. There's a video that says, Working at Netflix, What My Job Was Like. goes through the netflix architecture and says the tv os they ship submits requests to amazon via http via curl and so i'm like well this is obviously crucial you know i just respond like hey if you check out this you know this video at this, you'll see that it seems to be core underpinning. I mean, if every time I push play on Netflix, there's a curl call to AWS,

that's pretty crucial, wouldn't you say? Yes. Yes, of course. Okay, good answer. But, you know, I've come to the point where, sure, there are, what, 200 million Netflix devices. That's just a drop in the ocean when it comes to curl installations because that right there i don't mean to be sort of get on my high horse about that but sure but then there's also roku and there's apple tv devices they all use curl and youtube has it bundled in youtube app on all your phones. So everything like that is using curl.

And nowadays, every TV has it. Every car has it. Pretty much every printer has it. Every fridge, dishwashers, washing machines, and trains, motorcycles, and keyboards, watches, and robots, and computer games. It's really big in a lot of high-volume games. I guess because they want it to be portable.

I don't know exactly why the games like it so much. So yeah, that's why I say, I mean, actually, I think I underestimate when I say 20 billion installations. It depends a little bit on how we count. But Curl pretty much runs runs since it's not provided as an API in mobile operating systems. A lot of the mobile apps ship their own curl installations, right?

So in an ordinary mobile phone, it's installed like 5, 10, 15 times because a lot of the high volume apps have their own installations. All right, I just bundled it. Well, you know we call that, Daniel, we call that total world domination. That's what we call that.

I mean, where isn't curl? Yeah, where is it? The bottom of the ocean. And we know it was on Mars, right? We knew that much. Yeah, exactly. Yeah, I actually have the discussion sometimes with the commercial or potential commercial customers

as it's actually in the lower end, in the really, really tiny devices, because then they think curl is too big. Are you going to curl light you're gonna have an embeddable yeah i actually have an effort i call tiny curl which is pretty much a way to scale it down as much as possible pretty much disable a lot of well optionally disable as much as possible to make it as small as possible to fit on the just a few megabytes devices well that version hasn't found critical mass or like people don't know about it well how come these people aren't using it then you know do you know yeah well they are using it so i'm actually i'm having

customers using it so it it is happening but but um it's not a then then when i end up in a fun situation usually because when you and when you talk about people writing code for really, really tiny devices, they usually end up with an example HTTP client from some vendor of their development board or something. That's usually pretty much how Curl started 25 years ago. 200 lines of really, really stupid code. And sure, it might work for them, you know, occasionally or mostly. And that's the competition then. Then to say, oh, look at this code. It's only 200 lines. Why is curl 50k when I build on my target, when I can use this 2k code?

So that's the struggle I have then. And then I have to talk about APIs, stability, protocols, blah, blah, blah. Right. Yeah, 48K of security hardening in there and all kinds of things that you're not taking care of. Yeah. And which API do you get and which API will you have in 10 years? Will you have the same API then or won't you? And stuff like that. But you don't have to sell it too hard because you already have total world domination.

Yeah. You already got 20 billion devices. I mean, Adam just told you it's the core of Netflix and it's like drop in the bucket, drop in the bucket. Sometimes it feels like, sure, if you want to argue about going with just a silly example code or you want to go with curl, sure, you go with a silly example code

and call me again in two years when it's broken and you want to have some serious help and you wanted to i mean it's it's it ends up a silly discussion often oftentimes so no i'm not losing sleep over that but my tiny curl effort is basically just at least a a way for me to offer something that is at least in a much smaller footprint than the regular one well the conversation we were having in dms at least it was one bringing to your awareness if you were not aware the level of usage in netflix and i imagine that they're just one of many but they're also a fairly well funded fairly successful uh they're the n in well-funded, fairly successful. They're the N in FANG, whenever you say that term.

We were talking about really how you sell support. And I was asking you, like, are you intimately involved in selling support? You know, Wolf SSL has salespeople. You said, yes, you were. And I was like, well, you know, is selling support a priority? And I think we've talked about this a little bit the last time we talked. Well, SSL was early for you. Now it's sort of three years at least later.

How does it work selling support? How challenging is it? Do you have a large pipeline? What do you optimize for? I don't know. Answer any of these questions you like. There are things in motion that I should not talk about. But selling support in general for curl is in general actually hard

because it's an old, established, very functional, and actually rock-solid product. So a lot of those people, sure, they – and I mean, you mentioned Netflix. Netflix is a fun example. They don't use – I don't think, at least they didn't use to use curl for the actual film movie transfer. They just used it for UI and things related to selecting the actual whatever you're streaming. But there are, I mean, there are other companies, I know at least two, that have much, much, much more higher volume of curl use. Like this company with the blue logo.

I know I talked to some people seven, eight years ago. They did one million curl requests per second on average. And that's a high volume use of curl. You would imagine that those guys would be willing to buy curl support. But so far, I have not managed to get support from them because it just works for them. They think it's sort of, why would they buy support? It's been working for, I don't know how many years, and they think it's good.

So it is a challenge. But yes, I know it's used everywhere. And of course, I'm getting help because I'm worthless at business myself. So I realize my limitations here. I'm not the business guy. I'm not doing most of the sales conversations. I leave that to actual salespeople. But sure, we try to interface and talk to pretty much anyone we know that is using curl in high volumes and in sort of

mission critical situations where where it is really crucial for them that curl continues to work and it's it goes up and down and sure i mean i i still work on curl full time right so it at least pays me so that i can continue doing this but we also explore other ways to charge money for curl related things so there are of course always maybe not just selling support is the only way we should do this and and maybe it's not even the best way and i don't know we have alluded to it i have some other things in motion there will be some other things going forward curl business wise so i think it is a pretty good brand it's a pretty good product i think there should be ways to sustain the business around it so two more of your bdfl principles seem like two sides of the

same coin one is to stay on the bleeding edge and the other one is to keep up with the world and those seem to have different angles, but they sound very similar. Can you explain those two? Yeah, they're really similar. But what I really like, because I think it's a really fun position, is to be on the bleeding edge so that we can be early with development of new protocols,

like we support HTTP 2 really early, actually, and HTTP 3 also very early because it helps. It becomes a good sort of spiraling. We get it in early so that the people who actually implement and deploy servers and proxies can use curl to exercise their code and try their servers, and they can then help us make our code better. Because I think that's fun, and it makes us polish and streamline our implementation early on and sort of make better decisions earlier in the process on how to do protocols, really. And also then, of course, it's just fun to have that position so that we can help others make sure that they get their implementations done better.

So that's what I mean with bleeding edge, because I think it's a fun place to be at. And listening in what the internet wants, it's more, that's what I mean there is I want to make sure that we offer the ways of doing internet transfers that the internet seems to be wanting. The protocols curl supports, I want them to be done in a secure way, in a modern way, and the way we want to do. So if you want to do internet transfers for your application, you want to implement a car infotainment system. You want to transfer data for that. You want it done in a modern way, which means this current protocols,

current security, a current mindset of how to do things. So that's what I mean. That's what I mean with keeping up with how to do things. That means the Cypher suites and TLS and versions with HTTP and how to do authentication,

things like that. Because I think it's important because I think the internet is moving quite fast, quite almost all the time. So if we wouldn't keep up, we could easily just be left behind because I think we have a role and that role should be providing service

and how to do things in the proper way. And also because a lot of users are just using curl already. So we make sure that a lot of things are doing correct and secure internet transfers by using curl. So that's also a reason, right? To make sure that curl does things properly, we help securing the internet, really. What's up, friends?

I'm here with a new friend of mine, Jasmine Cassis, product manager at Sentry. She's been doing some amazing work. Her and her teams over many years being at Sentry, and her latest thing is just awesome. User feedback. You can now enable a widget on the front end of your website powered by Sentry that captures user feedback. Jasmine, tell me about this feature. Well, I'm Jasmine. I am a product manager at Sentry and I'm approaching my three year anniversary. So I've spent a lot of time here. I work on various different customer facing products.

More recently, I've been focused on this user feedback widget feature, but I've also worked on session replay in our dashboards product. With user feedback, I am particularly excited about that. We launched that a few weeks ago. Essentially, what it allows you to do is it makes it very easy to connect the developer to the end user, your customer. So you can immediately hear from your basically who you're building for, for your audience. And you can get basically have a good understanding of a

wide range of bugs. So Sentry automatically detects things like performance problems and exceptions, but there are other bugs that can happen on your website, such as broken links or a typo or a permission problem. And that is where the user feedback widget comes in and it captures that additional 20% of bugs that may not be automatically captured. I think that's why it's so special. And what takes it a step level above these other feedback tools and these support tools that you see is that when you get those feedback messages, they're connected to Sentry's rich debugging context

and telemetry. Because often, I've seen it myself, I read a lot of user feedback, messages are cryptic, but they're not descriptive enough to really understand the problem the user is facing. So what's great about user feedback is we connect it to our replay product, which essentially basically shows what the user was doing

at that moment in time, right before reporting that bug and we also connected to things such as screenshots so we allow we created the capability for a user to upload a screenshot so they can highlight something specific on the page that they're referring to so it kind of removes the guesswork for what exactly is this feedback submission or bug report referring to now i don't know about you, but I have wanted something like this on the front end pretty much since forever. And the fact that it ties into session replay, ties into all your tracing,

ties into all of the things that Sentry does to make you a better developer and to make your application more performant and amazing. It's just amazing. You can learn more by going to Sentry.io. That's S-E-N-T-R-Y.io. And when you get there, go to the product tab and click on user feedback. That will take you to the landing page for user feedback. Dive in, learn all you can. Use our code changelog to get a hundred bucks off a team plan for free.

Now, what she didn't mention was that user feedback is given to everyone. So if you have a Sentry account, you have user feedback. So go and use it. If you're already a user, go and get it on your front end. And if you're not a user, well then, hey, use the code changelog, get a hundred bucks off a team plan for three-ish months, almost four months. Once again, Sentry.io. What's your take on the internet these days? The state of the internet. I mean, do you like it?

Do you dislike it? Where are we? Because we've gone places and now here we are. Yeah, like if you were going to give Daniel Stenberg the state of the internet, like how is it looking out there? I think the...

That's a very... Why did you ask me this question? All right. After an hour, you hit me with this. It's a tricky question, I think, to answer because I think it goes up and down a little bit over time. And I think, I mean, sure, sometimes it seems that we're going the wrong direction,

sometimes in the right direction. So I think it's, yeah, it comes and goes, I think. So I don't think we're necessarily going in a right direction. So I think it's, yeah, it comes and goes, I think. So I don't think we're necessarily going in a bad direction, but it's certainly not always in the good direction either. Yeah. So, I mean, like with encryption and all sort of governments and China and firewalls and whatever happening everywhere. So I think it's, yeah, there are certainly bad signs every now and then,

but we seem to be usually get around them and get around them and kind of come out in the better way. But it seems like a struggle that just continues and keeps on. So we just have to keep an eye on where we're going and stay vigilant to see where we're going like with these you know backdooring encryption algorithms or inventing crazy things because of you know thinking of children or whatever but i'm generally a sort of an optimist so i think in general i think we're going in the right direction one reason i asked that is because we had paul vixie on the show a couple months back who's're going in the right direction. One reason I ask that is because we had Paul Vixie on the show a couple of months back,

who's been instrumental in the DNS area of the internet stack, if you will. And Paul seems more pessimistic. I mean, he described DNS as a series of patchwork that is stuck in the past. I've had my interactions with Paul. I know exactly where. So yes, compared to him, I'm certainly the optimist.

Yes, so he's certainly. Okay, cool. All right. Well, he painted a pretty bleak picture. And Adam and I, we kind of get blown in the wind a little bit on these things. I mean, we have our vantage point as well, but we aren't down in the weeds of any particular thing.

And you are clearly on the bleeding edge of the technical weeds of internet transfer, which is pretty much what the internet is for. And so happy to hear that you're slightly more optimistic than Paul is. But then there's also the social side of it, like the silos of the platforms

and now this revolt against social media. And then we have the rise of the LLMs. There's a lot of things happening at the application layer that are also the internet, but they're not down where you operate the internet. No, exactly. And I think, I mean, that's why I kind of refrain from commenting on that

because that seems like, sure, we can talk about that, but it's not really about internet transfers, really. That's really human social side of things. And I agree, even there as well, we seem to be going both in the right and the wrong direction at times so sure banning social media or not uh yeah have you looked into activity pub are you interested in that protocol or the whole this idea of a federated social network of well i'm i'm pretty much only on mastodon and not on twitter anymore

okay so but but i haven't really looked at a protocol i have a feature request for support Well, I'm pretty much only on Mastodon and not on Twitter anymore. Okay. But I haven't really looked at the protocol. I have a feature request for support for the message signature algorithm they're using because apparently there's an RFC 9D241 or something. And apparently the Fediverse is not using that exactly. Okay, fine. That as far as I've come to the details in the protocol. Okay. And we don't support that exactly. Okay, fine. That as far as I've come to the details in the protocol.

Okay. And we don't support that either. But you can't post to your activity pub stream via curl directly. Of course, there has to be

some sort of a server involved anyways. Yeah, but you can do it with curl, but you just have to do some massaging of some headers and stuff. Gotcha.

Because I've seen people do it with curl. So, yes, it's HTTP in there. So at the social level though, I mean, we respect your opinion at all levels, Daniel. I think I was kind of reading into that with keeping up with the world.

I was kind of thinking like, stay on the bleeding edge. There's your technical side. Keep up with the world. That's kind of more of the social side. Maybe that's not the way you were framing it. But in terms of the Fediverse idea, do you think that's a good idea are you are you like a proponent are you just like a a user who's like well i don't want to be on twitter and

so i'll use this and i'm not really happy no no i i really i really think it's a good idea i think it's uh when it comes to like social media it is certainly one way to manage the load and how to manage just people and content in general so that we don't have to have those central silos deciding what is right and wrong and what is truth or false or you know what is allowed and not allowed so we can spread it out just like we've done with emails in the beginning of time so i I really believe in that, even if I understand that having things like this is also probably not the business-y side of things. So I figure it's going to be hard to do that as a business. That's why no business does it, because they like the silos and building walls around it so that they can extract the maximum amount of money out of it.

But so I guess that's also why it's going to be this struggle between the technical benefits of doing it like that and the financial benefits of doing it the other way. So which way is the best way? I mean, you also have to have money involved because it needs to be run, right? I think we're already seeing that because if you read at least between the lines, some of the bigger Mastodon instances now have a huge bill to pay every month because they get a lot of traffic. And how do you finance that? It was possibly easier when you had one silo called Twitter, but it's going to be harder when everything is federated and distributed.

Yeah. That's why the threads integration is interesting because it's such a big business. The thing that's interesting to me with Fediverse stuff right now is how a lot of the small publishing platforms are adopting. So you have Mozilla, you have Flipboard, for instance, Medium. Then you also now have Ghost and these are like small businesses in relation to big tech

but then you have this gargantuan giant thing that is threads glomming itself on and you're like some people love it, some people hate it maybe it's validation, maybe it's evil empire and we don't necessarily have to get our opinions on that aspect of it but there is the business side kind of coming to the Fediverse you're right, yeah

and that is interesting I guess the challenge is to see how that goes over time yeah, exactly Threads versus Fediverse I guess it's fine as long as they are the elephant and the Fediverse is the ant they don't care about the ant here because it's just an ant on an elephant.

But maybe if the growth of the Fediverse explodes and threads isn't, maybe it'll change the equation. Or maybe not. I don't know. You're right. It's an interesting balance there. They're at least trying it out. I'm just thinking about your thoughts there on the business coming to the Fediverse. What does that actually look like?

Given that it's pretty much been disparate users there, really folks revolting against big tech social media. Is business or that, is Threads welcomed back in there? While they may or may not support different protocols with it, how does business then sort of hop into this fediverse world with welcomed arms will it actually improve it you know is that is that what's required to get to the feature set that does improve the fediverse for mass adoption beyond where it's at and how do you get the business to i mean how do you spend a lot get business in there really focused without sacrificing something, right? Yeah. I mean, you're not going to see anyone wanting to add ads in there.

But the business is... I mean, sure, you can get some kind of marketing value of being there and everything. But as the network is growing, it's going to be more and more expensive for everyone involved. So somewhere, someone has to get money in there. Right. Well, businesses go where eyeballs are too, right? That's true. That's where I think the precipice happens. Yeah, but not only

you want those eyeballs to do something. You want to convert. Exactly. You want to convert them somehow. Well, maybe you can do that by posting. I don't know. Ads. Gosh, they're just everywhere. Yeah, I mean, whether you're selling... Even these podcasters ads. Yeah, maybe you can do that by posting. I don't know. Ads. Gosh, they're just everywhere. Yeah, I mean, whether you're selling- Even these podcasters' ads. Yeah, whether you're selling socks, right?

Or a subscription to a journal or a author, you want people to know that you have things to offer. And the internet was a great way of doing that for a long time. And then social media came around and Facebook said, actually, the eyeballs are over here. Come build your following.

And people did that, right? And then they sold you access back to the following that you built. And then we all got burned by that. And that happened at Instagram and that happened at Twitter and that happens at LinkedIn and all these other places. And people are just sick of that. They're just like, if I build up some reach so that I can tell people what I'm up to, and then they can visit my website and buy some socks, I don't want someone intermediating that relationship and then just

selling it back to me. We don't want to get burned again. We got burned once, and so let's not do that. And so I think that's attractive for businesses the question is can the fediverse actually provide the reach to the people who are there to see what you're up to and to actually talk with you in those things and so far it's been part of it is it's it's it's not siloed but it's federated and so there it doesn't doesn't work the way the silos work that's why they have the better user experience right exactly so and i guess in one way it can't be prevented right right since you can just invent a way to just nag about your socks all day long if the if the other instances just think you're too annoying they will just block you or just not federate with you and that's fine but so yeah i think it'll be interesting to see if if the businesses and

brands will show up to a significant amount well the gateway is always content right if you create compelling content that matters to people and you take it to the people right rather than saying hey i have a website come check out my website content's here, you take the content in unique ways to the places where your future potential possible buyers would be. And you do it in a way that is not just, hey, I make socks come by, but it's more like, wow, we work directly with these cotton farmers in XYZ, and this is how we sustain that region, and this is how we've enabled an uplifting community there, or whatever it might be, and that's their brand story. That, I think, is how brands engage, obviously, in meaningful ways. I think you're right. But I think, given what we've seen on the web, is that, sure, that works for a subset

of the content producers. They think like that. The other guys, they think, hey, SEO is a good thing. We should just pepper our website with weird keywords because this is the way they will find us, not by producing content because just spewing out keywords is much cheaper and faster than actually doing that good content. Absolutely. And you can use an LLM to generate those too. Like, hey, I have a business in XYZ sector. Can you give me keywords? And it'll give you keywords all day long.

And then in the end, if you then use your LLM and send more socks than the guy who did that nice content for his socks, what kind of incentive is that? Call that an S-DOS. That's a sock denial service. That's what that is. Yeah, so there we go.

We're kind of like kind of good, kind of bad, right? Kind of promising, also fraught. I mean, none of it's straightforward. And that's why I think we're at a very interesting point in the internet history, because we have had a bit of a revolt. I think many people's eyes are open. You look at the, the backlashes on Reddit, the backlashes on Stack Overflow to these content deals the users of the internet are not very happy with their platforms right now and so people are willing to maybe suffer a little bit of reduced user experience and maybe like a a fractured reach in order to have more autonomy and a little bit more ownership and so maybe there's a chance i don't know yeah and i i think at least my impression is that people have at least started to learn that maybe we don't need to be everyone at the same place either, right? So maybe it's better because we might get better content by not trying to be there where

the entire population of the world is, because it's not going to scale. It's not going to work, really. Right, it didn't work. It hasn't worked for so many people. I mean... Right, every time someone tries it'll it fails eventually all right well there's your state of the internet report we dragged it out of it we dragged it out daniel you know i'm somewhat optimistic as well i think that there's obviously problems some seem insurmountable some are larger

than others i think when you have a worldwide network and then you have individual countries operating on a worldwide network, you obviously have a lot of concerns that don't cross international borders, but they cross the network borders. And so that's always problematic. And it was from the start. It's just that we've matured into

where the stakes have risen to where everybody cares more. And so nation states and large tech and these companies are starting to like try to establish themselves and make their rules fit everywhere. And it's messy out there. Yeah. And I think we see some troubling tendencies there sometimes when you try to make your borders mean more on the internet. Suddenly within our country, this is the rules for how we do things because it's going to be really, really messy if we want to try to enforce that. Sure, China is an example, but even more when the EU comes up with things, the US does.

If we're actually going to try to enforce having different rules with it for different countries it's going to be really hard to to do things so maybe but at the same time i think sometimes also see this as a danger but it hasn't happened as much as we maybe think it might do but well good examples eu with apple and the way apple approached their compliance with a lot of the eu regulations was localized to the EU. And of course, I'm in there as an engineer, I'm in there with the Apple engineers thinking like, how fractured is this code base now where they have very specific

and completely different rules based on region? You know, how many if statements are in this code base that don't need to be there? They're just there for regulatory purposes, right? Exactly. And it begs the question then, so sure, they did that for EU, but are they going to do that for more areas as well?

Or are they going to hold out on that? Because you would otherwise imagine that you would pretty much, like the GDPR pretty much affected how they would do business in the whole world instead. Yeah, because that's just more convenient. Or like they had to do with their hardware, right? Like with the USB-C port.

Oh, right, exactly. They just put it in all the phones. But when it comes to software, they're like, no, we're not going to do it for everybody. We're going to do it just for the EU. I guess that tells us where there is a lot of money involved. Yeah, it costs less to put on all the if statements than it does to actually do it for the whole world.

It's worth complicating the software to that significant amount just to keep it. Good old-fashioned tech debt. Potentially, do you have time for maybe a can of worms, potentially? Hit me. Well, Curl is verified. You know this because you wrote the post and you've done the work. But I think given the install base and the know the fact that curl is pretty much everywhere crow could be at this point people will question whether or not there's ever a possibility of curl

being exceed i don't think so given the things you've done but i think what's important to talk about is the ways and maybe some of this is in your bdfl manifesto or ten commandments or however you want to frame it i don't think so necessarily but in what ways are you preventing or working against securing bolstering your security to never have a even a social engineering attack you know how do you protect yourself how do you protect curl and what it it is? First, of course, the XZ attack was pretty amazing in both sort of the engineering that it took to do it, but also in how they selected the target, both in a project that was mature for their attack,

but also technically the correct project in that they had payloads in in git for example that they could in fact with like encrypted binary stuff so they could hide a payload in git so they had a lot they did their due diligence really good when they selected that target so not only were they a skilled team they selected the perfect target for that. I don't think curl is as perfect target for such an attack. So if we start in that end, so one way that XZ was good is that they could insert huge payloads that were just encrypted because they wanted binary blobs to test their compression or failed

compressions or whatever it was. And so one way we should then make sure that we don't have anything hidden in Git. We don't have binary blobs in Git. Everything should be motivated and understandable, right? There should not be any big binary blobs that no one can understand. So you should be able to review everything. And ideally, then someone actually reviews everything. I guess that's sort of the Linus law, right?

Given enough eyeballs, all bugs are shallow. But how many eyeballs do we actually have? Probably not too many, because we all just think that someone else is doing the reviewing. But at least we've had a few security audits of curl. We actually had a few recently even. So we've actually had within the last few years have had two security audits. So actually someone at least independent security professionals actually reading a lot of curl code and figuring out if we have any hidden issues

or problems somewhere. So at least we have that. And then we just do the normal things like making sure that every change is done as a pull request on GitHub. We review as much as possible. I think I review every pull request someone else provides. I have to admit that I merge code that I write that no one

else actually says okay on because that's just me because I want to move faster than someone else is actually saying thumbs up. So I'm just reviewing my own code and that's a vulnerability in the process really. But I trust myself to do that. And also, in addition to that, we have a lot of test cases. And then if the test cases actually work and they prove that curl works to this degree, so you actually have to do a pull request that actually runs through all those test cases.

And someone did the math and we do indifferent and mutilations and somewhere around 140,000 tests per pull request. So there's a lot of testing. So when all of those tests go through green, we know that all the functionality that we test for is there. So it's really, really, really hard to plant a backdoor or land unintended functionality

there. Bugs, sure, stuff that we don't test could obviously land every now and then because we fix bugs all the time. But actually, generally land a backdoor in this code, I insist, is really, really hard. I'm talking about what people ask me. People ask me that quite often, if I've ever have

detected anyone trying to land the back door in curl and that might just be the because i'm stupid because i've never detected an attempt to land the back door in curl and i think that is because it's so hard to actually do that so i don't think people in general try that. I think they did that with XZ because that was a different project. They could do it. I think it's much easier if anyone actually wants to attack curl, it's much easier for them to find a bug or exploit something that we already did unintentionally,

like a security vulnerability, find that and exploit that. I think that is hard, but I think it's much less hard. And then in the end, so if everything we land in Git is tested and it gets reviewed, there should actually be a very, very slim risk that we actually land something there that is bad. And then in the XZ case, they added attack code when they produced the release tarball, right? So they actually added code that wasn't in Git. And that's the final step we do that we do releases the same way.

We actually generate files that we put into release tarball. And those generated files are not present in Git. So we could actually do the same kind of attack with Carl if I was a malicious release manager. And that way, we pretty much make sure that we have a reproducible process to do those release tarballs. So I pretty much nowadays, I do them with a Docker image. So anyone else can produce the binary identical copy of a release tarball. So ideally, hopefully someone verifies my release by doing a identical copy of the release

to make sure that the process actually works. And if every single step there holds, then there should be no room. And sure, I mean, every process, of course, have flaws. I mean, we can do mistakes and we're human involved, so humans do mistakes every once in a while. But hopefully we have enough checkpoints,

enough process and procedures to make it really, really hard. And if one of those tiny things go wrong, it won't be enough to actually land a backdoor. So I think we have a lot of checks, a lot of processes, and so it should be hard to land a backdoor. I will never say never, but it's going to be a challenge. There's going to be, they need more ingenuity and effort than they did for XZ to land it in curl.

It's about putting those hurdles and breakpoints and just frustration points in there that if there's enough in the chain, then it becomes less and less of a desired target because it's just generally hard. So we talked to Jacob DePriest, VP and Deputy Chief Security Officer

at GitHub. Actually, if you're listening to this, it's in the feed already. So we're releasing it tomorrow officially from our record date. But in terms of ship date of this podcast you're listening to, it's already in the feed, so go check that out. But one thing he talked about was this idea of attestations, artifact attestations.

And I know that you play, I guess, in the GitHub sandbox to some degree. They may even talk to you early on. Are you involved in any way with that? Are you familiar with that feature set they're bringing out? It's in beta, but are you planning to use that? What do you think about it? I have just sort of read up about it. So I don't know.

Right now, I haven't sort of been missing any feature in that area. So that's why I haven't really kept up with exactly what it means and how we can use it. Because I think I have my T's crossed and my checkboxes crossed pretty well. And I do all this and I sign my releases. I have the same signature and my keys published since a long time. So I think I have a lot of these already covered.

I'm not sure exactly what else they offer in this regard. So I haven't really followed it. So I can't comment too much. Can you say the word attestation? Attestation. Oh, good. Because we found that to be the hardest part of it was the lack of stickiness of the word attestation.

Yeah, it's a hard word. I think the idea, though, is pretty straightforward. And the fact that they're leveraging GitHub Actions as a part of it, it's basically a way to generate and verify signed attestations for anything you make with GitHub Actions. So I think you're deeply on, obviously, GitHub, and you're probably using GitHub Actions in many ways that I know for sure. I do, but we don't do releases with GitHub Actions.

We don't do releases with any CI. So I do, that's why I don't know for sure. They do, but we don't do releases with GitHub Actions. We don't do releases with any CI. So I do, that's why I don't need it. In our case, all our CIs or all our jobs we run there, they're just, you know, throw away machines. They just run tests and verify. They just say green checkbox, then they're done and we throw them away. So we never use anything that they produce.

So therefore, if you attack our CI services, it doesn't matter for us. Sure, you can make our CI jobs fail or fake that they work. That could possibly be an attack vector, but you cannot affect or sort of inject anything in our releases because we don't build releases that way. I build them locally in a Docker image on my machine, but in a documented way way and then i sign them and i upload them to my server that's why you're the best daniel personal hand cut releases after all these years he's still he's handcrafting his software exactly 259 of them or something. Wow. Good stuff. Good stuff. Well, is there anything burgeoning,

anything upcoming, anything that you're up to that you want to make sure that our listener knows about before we let you go? I don't think so. When it comes to curl stuff, we're just chugging along, adding things.

Since we do releases every eight weeks, we always just keep adding small things. It feels like we never add big things because we always do these iterative, smaller things on and on and on and on and never ending. And since we don't break APIs, we don't break users' scripts,

so we just continually add and polish things. And that's what we're going to do continuously. So yeah, that's what I'm planning on continuing doing going forward as well. I guess we'll talk to you in three years and see if you're still doing just that thing. Yeah. Or sooner or later.

I don't know. We got to have him back soon. Now that we have ChangeLogging friends, we have another episode every week that we can actually just bring you on. I feel like we have to interview you. We got to get you on more regular, Daniel, because every three years, I mean, you're better than that.

Yeah, we can certainly do a different cadence. Exactly. Maybe annually at least. Yeah. Come on. Cool. Well, we always appreciate your perspectives on everything, man. Oh, right.

I forgot to mention, by the way, what also happened during these three years is that I added a new command line tool to my agenda. So nowadays, I also make a tool I call TrueREL. Talk about hard to pronounce. Okay, TrueREL. T-R-U-R-L. T-R-U-R-L.

Completely impossible to pronounce, but I pronounce it TrueREL. I have to add an E. Yeah, throw an E in there. But it's just a small command line tool for doing URL manipulations on the command line. Okay. T-R-U-R-L.

It's just another command line tool just to fiddle with URLs because it's really hard when you write shell scripts and you want to do things with URLs, like change the query parts or change host names, change... Yeah. I see.

They're tricky to manage in shell scripts, so now I have a tool for that very cool very much fitting in the unix philosophy here you can just use tr url and yeah and and and also it's another more important underpinning to that is that one of these things that happens over and over there's been actually several times during the last decade, people have pointed out written papers about the problems with having different URL parses in different layers of your system. Pretty much if you write an application and you use another transfer library or third thing,

and they all parse URLs. And since URLs are such a weird beast, so there's no good spec for what a URL is or isn't. So all URL parsers have their own definition or opinion what a URL is. So basically, that means if you're using two different ones, it's a big risk. So there's a big risk that one of them treats it slightly different than the other. And that's one way to sometimes smuggle things through because what's the host name in one

might be an invalid host name in the other and vice versa. So that's also one way why I wanted to have this tool to manage URLs because it uses the same parser as curl does. So if you want to write a shell script that understands URLs, it understands URLs the same way that curl does. So if you want to write a shell script that understands URLs, it understands URLs the same way that curl does. So it sort of removes that friction that you might actually handle

the URLs differently in different parts of your setup, pretty much. Gotcha. Now, is TrueRail just as easy to get your hands on as curl in terms of apt-get, dev-install? Yes, it's getting there. It's only a year old, so it's not completely everywhere,

but it's getting deployed in most of those popular package managers now. Well, we'll link that up so folks can check that one out. Neat tool, terrible name. Yeah, but it works out because it ends with URL and it works with URLs and it's also a little bit like the tr tool as a tr, you know, the shell command for transposing or whatever it's called. And it's similar in spirit to tr. So that's TR URL. It works sort of. Yeah, but maybe. I like the way it looks on GitHub slash curl slash,

I would call it trurl. Yeah. I can't pronounce that. So TR URL. It rhymes with curl. I can see why you selected it. Right. But you know, naming is hard.

Naming. It is. It really is. We'll come back soon. We'll talk more. Yeah. Thank you. know, naming is hard. Naming. It is. It really is. We'll come back soon. We'll talk more. Yeah. Thank you. And that's all for now.

Bye, friends. Bye, friends. Naming things is hard. Names we imagined up for this episode include BDFLing Curl, everywhere Curl could be, and ultimately, where doesn't Curl run, which we ran with. You know what else is hard?

Finding your next favorite podcast. Help your friends and colleagues by sharing changelog shows with them. They'll thank you later, and we'll thank you right now. Thanks. We appreciate you spreading the word. We also appreciate our partners at Fly.io,

our Beat Freakin' residents, Breakmaster Cylinder, and our friends at Sentry. Do yourself and us a favor and use code CHANGELOG when you sign up for a Sentry team plan. 100 bucks off. Next week on the Changelog, news on Monday,

a deep dive on semantic versioning on Wednesday, and Kaizen 15. It's a deep dive on semantic versioning on Wednesday, and Kaizen 15, it's a good one, right here on changelog and friends on Friday. Have a great weekend, leave us a five star review if you dig it, and let's talk again real soon.