The Changelog: Software Development, Open Source - Securing the web with Let's Encrypt (Interview)

Episode Date: April 7, 2020

We're talking with Josh Aas, the Executive Director of the Internet Security Research Group, which is the legal entity behind the Let's Encrypt certificate authority. In June of 2017, Let’s Encrypt celebrated 100 Million certificates issued. Now, just about 2.5 years later, that number has grown to 1 Billion and 200 Million websites served. We talk with Josh about his journey and what it's taken to build and grow Let's Encrypt to enable a secure by default internet for everyone.

One of the big discussions in the standardization process for H2 was, should H2 require HTTPS? Should it require TLS? For me, that felt like a no-brainer. Like, yeah, obviously H2 should require TLS. We know that not using TLS is a huge problem. It's 2011. Why in the world would we ever create a new protocol that's not secure by default?

That seems just crazy. Being With Your Change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Changelog because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com slash changelog. Linode makes cloud computing simple, affordable, and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale you need to take your ideas to the next level. We trust Linode because they keep it fast and they keep it simple.

Check them out at linode.com slash changelog. All right. Welcome back, everyone. This is the ChangeLog, a podcast featuring the hackers, the leaders, and the innovators in the world of software. I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog. Today, we're talking with Josh Ose, Executive Director of the Internet Security Research Group. That's the legal entity behind the Let's Encrypt Certificate Authority. In June of 2017, Let's Encrypt celebrated 100 million certificates issued. Now, just about two and a half years later, that number has grown to 1 billion and more than 200 million websites served. We talked with Josh about his journey and what it's taken to build and grow Let's Encrypt to enable a secure by default internet for everyone.

Securing the internet is obviously a big deal. Back in May of 2013, you thought so as well, and you started the Internet Security Research Group. But since then, Let's Encrypt has issued, in all caps, 1 billion certificates. And that's a big deal. So that was an announcement you made February 27th. Talk about that moment for you. What was that like to pin that post? Still pretty clearly to remember sitting around debating with our staff whether we've issued 1 million certs in the first year or not. Like when we started, we had no idea what the scope of this would turn into.

So a billion is a big number and it's amazing to get here it's so many ideas you know never turn out to what you wish they would turn into yeah and it's pretty exciting that this team has built something that turned into what we wanted to be which is serving so many websites around the internet. We're getting close to around 200 million now, and that's fantastic. It's often interesting because you can take for granted what's right there in front of you today. And so I kind of look at it like new developers coming into the scene in the last two years, let's say since the inception of Let's Encrypt. And just the idea that it's there and it's fairly easy now to request a free certificate.

We're in a day now where I suppose the security of the internet is much more important as we all become more dependent on it, and it's more prevalent in our everyday lives, especially in a day where right now we're using a Zoom chat, so we're assuming that this is encrypted. I'm not sure we're seeing anything that's concerned. Well, we're recording it ourselves, so it's not too private. The point is that not all these Zoom calls are ones you want to put onto the internet.

So obviously security is a pretty interesting thing, but we came from a day where SSL certificates were very difficult to get, generally expensive, and just the process was very cumbersome. And now it's a fairly easy process. People take that for granted. Yeah. On the one hand, we sort of want to be in a position where people can take us for granted because we want the service to just happen for people. Ideally, you could just set up web servers and not even know that you have an SSL certificate, let alone where or how you got it. We're all about automation and removing humans from the loop

so people have to do less. We'd love to get to a world where every time you spin up a server it just gets the certs you need in the background installs them correctly everything just works people need to know about us and the flip side we do want people to know about us because we're a non-profit and we need people to know about the great work that we do and help fund our work so yeah every day go out and try to put ourselves in a position where people can take us for granted, but it does have some negative consequences. It's kind of a catch-22.

What are your plans around that? We continue doing what we've been doing, providing great service, building things that people can rely on, and make the internet more secure. And then on the technical side it just happens but we've got great communications people and they go out and talk about what we do talk to potential funders and uh so far it's working out great i always think of wikipedia and their opportunity to

throw jimmy wales's note up there once a year usually usually in December, and say, hey, we're a valuable resource. Send us your money. And I was thinking, what's the Less Encrypted equivalent of that? It's probably a bad idea. You've got 200 million sites, but you don't exactly want to be injecting anything into that experience. Like you said, you want to be as seamless as possible.

So do you beat the drum mostly on the blog, or are there campaigns? What do you guys do to let people know what you're up to? We certainly don't want to go out there and inject messages into systems that people depend on. That's a very bad idea. I didn't think so.

We've got social media presence, a blog. We also have a lot of contacts and people who understand what we do and support us and we meet with them all the time. We go to companies, explain what we're doing. We're here for you. Talk to open source projects. Anywhere that we think understanding what we do can help. It's just a lot of work we go there and do.

Sometimes, you know, the media picks up good things like a billion certificates. That's really helpful to get press coverage. Unfortunately, sometimes people find out about us when there's a problem, right? Like when you have a service that so many people depend on, when things go wrong, that's when people notice you. That's when they stop taking you for granted. That's not obviously something that we strive for. We'd prefer that that never happened.

But software is complicated. Systems are complicated. Every system has problems once in a while. So some people learn about us when we have problems and that sort of does inject a message into their life, which hopefully is like, look, this is something I depend on. And when there's an issue,

you realize how much you depend on it. And the best way to help out is to support us. What are some of the bigger problems you've had to mitigate over the years? Mostly just comes down to, you know, normal software bugs. So we have a software stack called Boulder that is sort of the core of the certificate authority and it's written in Go. It's non-trivial in size, although it's mostly been developed by, let's say, an average of three engineers over five years. So it's fairly complex, and it's non-trivial in size, and like all software, every once in a while there's a bug there,

and those bugs can cause either stability or compliance or security issues we haven't had too many serious security issues but we have had some stability and compliance issues just yeah bug pops up usually we're very quick to fix it um but when you're serving you know 200 million websites yeah any little thing becomes sort of a big thing. Yeah. But I'm really proud of our track record here. We have a great track record for stability and reliability, security. And when incidents do come up,

I'm particularly proud of how well we deal with them. We typically fix issues, you know, a couple of hours max. And we go out and talk about it with public reports and detailed public information and lots of transparency very quickly. Almost always within a few hours. We really work hard to follow up and make sure that that type of problem doesn't happen again. Use those things as a learning experience.

Well, given the opportunity and potentially even by design, the ability for people to take Let's Encrypt for granted for the uninitiated, how do you describe Let's Encrypt? Like Let's Encrypt today You mean how do I describe it to people who don't really understand what SSL certificates are? No, let's take it from a developer's point of view Somebody who kind of gets it but doesn't They've heard of let's encrypt

they don't know all the bits and bots they don't know all the details of what let's encrypt is what do you do yeah so if you're a developer you want to set up an hdps site you're going to need a certificate and normally you know without let's encrypt in the world you would have to go find some place to buy a certificate and decide what kind of certificate you want. You have to decide how much you want to pay. You'd probably have to create some sort of certificate signing request or fill out some form containing a bunch of details about what exactly you want in your certificate. It can be a pretty time-consuming, costly, and complicated process, and it's frankly just really confusing.

I think it's the biggest reason that people didn't deploy HTTPS prior to the existence of Let's Encrypt. So Let's Encrypt really just tries to do away with all that. We try to make things as simple as possible. So we have an API. You just submit a request using some API client software. You don't need to write it yourself or know how it works. You just download some software for whatever platform you want to use. That software knows how to talk to Let's Encrypt. You just tell it what domains you want

certificates for. Typically, the software will just go out and get the domains, complete the challenges, do what you need to do to get the cert, and then sometimes they'll even install it for you. And all this doesn't require you to know anything about how certificates work or how you get them or what's in them, and it doesn't require any payment. One of the most important things about not requiring payment is that it's not necessarily just about the amount of money involved. It's not free just because free equals zero dollars. It's free because if you're sitting in some big company and you want to set up a site really quickly and you want to use best practices and deploy HTTPS,

if you've got to go to accounting and get a credit card and get approval to spend money and set up recurring charges and things like that, even if you're charging $0.10 a month, it's pretty big friction for you, right? You're just going to say, I'm not going to go through this whole silly process. I'm just going to set up the site without HTTPS. And now you've got another insecure site on the web. So not charging money is about not creating friction in the process

and not requiring humans to be involved any more than necessary. So yeah, Let's Encrypt is a really automated, easy-to-use, free way to get a certificate. So we first covered Let's Encrypt on the Change Log back in March of 2017, episode 243. We had Jacob Hoffman-Anders from the EFF on the changelog back in March of 2017, episode 243. We had Jacob Hoffman-Anders from the EFF on the show. Probably a fun episode to go back and listen to in light of the success y'all have had

because that was very much near or shortly after the kickoff. And so from there to a billion in three years, under three years, it's pretty amazing amazing i think free is a huge aspect of that but i'm just curious from your perspective what did y'all get right in addition to making it free which is a big aspect of course to get the spread on i mean you spread like crazy which is amazing what did you do right to get here well first of all let me recommend going back to that episode to anybody who wants to. Jacob is our lead software engineer, and he is brilliant.

And I will never regret listening to him. Yeah, he was a great guest. Like most things in cryptography, adoption is all about ease of use. You can come up with the most brilliant security or cryptography mechanism you want, but if it's not easy enough to use, it's not going to see deployment. Not at scale anyway. So from the beginning, it's been all about ease of use. And for us, that really means automation and just making it so people don't really have to do anything.

So in order to automate, you just want to have computers do the work so on the let's encrypt side we got a bunch of computers that do the work on our side it's a little more complicated than that but you know computers do most of the work and then on the client side for people requesting certificates we needed client software that works for everybody and people use a lot of different stacks out there some people are on linux some people are on Windows, BSD, some people are using Apache, Nginx, whatever. There's a lot of different deployment environments out there. And there's no way that we could write client software

for all of those environments ourselves. It's just not possible. But we came up with a really well-documented and standardized protocol. And then our community, which is amazing, has gone out and written possible. But we came up with a really well-documented and standardized protocol, and then our community, which is amazing, has gone out and written hundreds of clients that work with this protocol. So now no matter what your application stack is or your software stack, there's almost certainly a Let's Encrypt client out there for you to use. And all you need to do is just find that client, install it, and it will do most of the work for you to use. And all you need to do is just find that client, install it, and it will do most of the work for you.

So that's something we really couldn't have pulled off without our community. Did you bootstrap any of that? Like, did you say, well, we'll do the Apache integration or the Nginx integration and get it started, get the ball rolling? Or was there immediate community support post-announcement and maybe the publishing of, is it a spec, of the actual way it works?

Yeah, the protocol is called ACME, and it's an IETF specification now. And before that, we just published the spec for it. We did originally create a client that we developed not for very long because we realized that, you know, us putting resources into one client ourselves just does not cover enough use cases. There's so much out there. So we really need to focus on the community building clients and not us doing it. And for that client, which has now been renamed to CertBot, the Electronic Frontier Foundation, EFF, they were volunteering most of the work to make that client anyway. It wasn't sort of the Lutz and Krupp server-side engineers doing it. So we decided pretty early

on that it makes more sense to just turn that client over to EFF and make it an EFF project, since they were doing most of the work anyway. And we wanted to focus on supporting the strong community instead of building clients ourselves. So we did sort of try to bootstrap it by building this client early on and it served its purpose well, but it's had a much better home at EFF

and it's really grown into an amazing client over there. When we go back to the inception of things, which I don't want to go too far back, but just enough to sort of understand the, I guess the crux of the problem. Obviously an unsecure web is an issue, but what was the biggest thing that made sense to move forward with the internet research group and that being the foundation behind Let's Encrypt? What was the biggest problem happening that sort of was like, this has got to stop? Yeah, there's a few different people involved in starting ISRG

and I think they all have their own personal motivations for why they wanted to get into this. So I don't want to make clear this might not necessarily be true for all of them. For me, at the time, I was running the networking group at Mozilla. So the group that does all the networking code in Firefox. And one of the most frustrating things about running that team is there's nothing you can do on the browser side

to make sites use HTTPS. So if the site doesn't use HTTPS, you're just stuck doing completely not secure networking. And there's no amount of code you can write. There's no clever code you can write. You're just stuck. So if you're sitting there, you know, in charge of the networking stack for a major browser, it's very frustrating that there's nothing you can do about this. You can't improve the situation. So we started thinking about what's the problem here? Why are people not using HTTPS? And the biggest problem seemed to be that getting and managing certificates was too complicated or too costly or too time-consuming. For whatever reason, people didn't want to do that.

And everything else is pretty easy. Like, if you want to turn on TLS in Apache or Nginx, it's pretty easy to config flag. The software's all there. Easy to turn it on. You just can't do it without a cert. And this really came to a head when I was participating in some of the discussions in the IETF about standardizing HTTP2, or that's kind of a mouthful,

so I'm just going to call it H2 from here on out. We do as well. One of the big discussions in the standardization process for H2 was, should H2 require HTTPS? Should it require TLS? And for me, that felt like a no-brainer. Like, yeah, obviously H2 should require TLS. You know, at that point, I think it was like 2011 or 2012 when this conversation was happening, it seemed like, yeah, we know that not using TLS is a huge problem. It's 2011. Why in the world would we ever create a new protocol

that's not secure by default? That seems just crazy. Yeah. And there was a lot of pushback on that idea. In part, part of the pushback came from proxy providers. So basically people whose software and jobs depend on intercepting web traffic. So that's sort of the expected pushback. But other than that, it just seemed like a no brainer to me. But there was another objection, which was that if you make H2 require HTTPS,

you are effectively going to make it pay-to-play and you're going to make it much harder to deploy because you won't be able to deploy H2 without buying certificates. And you won't be able to deploy H2 without going through the certificate obtaining and managing processes.

So the idea was that requiring TLS would make H2 deployment handicapped, and pay to play and that frankly felt pretty reasonable and it felt to me like if i'm going to continue with this position that it should require tls and that it you know it's crazy to not to then i need to be willing to deal with the criticism here that's like, I need to have some answer to this. And at the time, I didn't really have an answer.

I mean, they were right. One of the great things about the web is that a lot of things don't require you to pay. And if we required TLS for H2, you would effectively be putting a financial tax on anybody who wanted to play H2. That didn't sit. And likely people wouldn't play. Right. So what's the point of even creating the tech or the new protocol, the new SPAC,

because less people would use it, except for, say, the ones who can obviously pay. Yeah, you hamper its adoption and you also hamper the people who can't afford to adopt it, right? Yeah, I mean, the big companies would do it. You know, Google, Facebook, whatever. That's a lot of traffic on the web, so it still has a benefit. But I don't think we should be designing major protocols like H2, you know, primarily just for the biggest players out there.

I think, centralized as the web has become in many ways, I think it's still important to pursue some of the ideals of accessibility and making sure everyone has the ability to participate on the web. So I thought those criticisms of requiring TLS were legitimate. And if I was going to run around telling everybody that they have to use TLS all the time, then we need to deal with that problem. So I went back to the team I was working on then, and we talked about a lot of different possible solutions. But it was hard to find any solution that was going to solve the problem and solve it in a reasonable amount of time. Like there were ideas where it's like,

well, if we do this, then maybe in 10, 20 years, you know, the situation is different and we can do this, but that's way too far. Like we can't design that. We should have done this 10 or 20 years ago, not be planning to do it 10 or 20 years in the future. So the only solution we came up with that we were pretty sure would work,

and that might work in a big way in five years or less was that there just had to be a new certificate authority that was public benefit really easy to use doesn't cost money and available all over the globe you know available everywhere to everyone without that we just didn't see how we're going to get out of this. So to be honest, I don't think anybody involved in those discussions was thinking like, man, I'm excited to spend the next five years of my life building a CA from scratch and like dropping all these other things that I wanted to do in my career and like, you know, build a CA. It wasn't like the most attractive project, but it felt like this is what's got to happen. We don't do this.

The web is going to be not secure for a long time. So we did it. We went out and started a new CA. At the time, I knew nothing about how to build or run a CA. So it was a lot of learning for me. And I think everybody involved, I don't think anybody, we had some advisors who had some experience, but no, I don't think anybody actually building the CA had ever built one before. So it was a big undertaking, but that's why we did it. And I think here we are five years later. And when we started, I think 39% of all page loads, so not websites, but 39% of the time you loaded any particular web page, it would be encrypted. and other big properties and everything else wasn't. Here we are five years later in the United States, I think we're approaching 92%. Globally, we're over 80 now

and those have a great trend line up. So five years later, we've encrypted most of it. We've got some more work to do, but we got there. In this new world of remote first, more and more teams are looking to build video into their apps. Everything from media publications, education and learning platforms to communities and social platforms. If you're trying to build video into your app, you're probably deciding between having full control by building yourself or faster dev cycles with an out-of-the-box platform. Well, Mux gives you the best of both worlds by doing for video what Stripe has done for payments. In a world of complicated encoding, streaming, multiplexing, and compression,

Mux simplifies all things video to an easy-to-use API to make beautifully scalable video possible for every development team. Mux lets you easily build video into your product with full control over design and user experience. Videos delivered through Mux are automatically optimized to deliver the best viewing experience, and you don't have to deal with the complaints about rebuffering or videos not playing. Get started at get.mux.com slash changelog. They're giving our listeners a $50 credit to play with.

That's over an hour's worth of video content that you can upload and play with and check out all their features, including just-in-time publishing, watermarking, thumbnails, and GIFs. To get the credit, just mention changelog when you sign up or send an email to help at mucks.com, and they'll add the credit to your account. Again, get.mucks.com slash changelog. So I can imagine starting a CA from the scratch is an undertaking.

You mentioned that you had some advisors obviously giving you some advice, that's what advisors do. But the majority of everyone involved in co-founding the Internet Security Research Group had no clue how to do some of these things. So how did you get a clue? How did you do this? What's involved in building a certificate of authority? Well, we got some advice from people, like you said, sort of laid out some of the basics of how it works. There is a document out there called the Baseline Requirements, which is a document built by the CAs and the browsers combined, sort of come together in a forum called CAB Forum, CA Browser Forum, and they create a document of all the rules and requirements that all CAs are required to abide by. And you can figure out a lot about how a CA is going to have to work based on what those rules are.

So we read those very carefully. We hired some auditors who audit us every year to make sure that we're compliant with those and some other rules. Our auditors helped us figure out some stuff. But yeah, mostly we just consumed all this information and started drawing up plans for how things work and then we'd iterate on them until it all seemed like it would work. And then we went out and bought the hardware, signed the agreements.

Just got to work. Some things had to be iterated on a couple times, but pretty much how you figure out anything else you don't know yeah if i grab the right document it's 68 pages i can imagine that's quite a read for one two who gives the authority for a c like who do you get the authorization from to to move forward yeah so you can start a ca and you can do whatever you want as a CA. The question is, who trusts you? If you start a CA and you do whatever you want, pretty much nobody's going to trust you,

so it doesn't matter that you're really running a CA. If you're running some sort of private CA, you need your clients to trust you. If you're running a public CA like Let's Encrypt, basically what that means is that the general public trusts you. What that comes down to is the browsers trusting you. So if you want to start a CA, you need at least all the major browsers to trust you.

So today that would be Google, Mozilla, Microsoft, Apple. Those are the big ones. If any one of them doesn't trust you, then this whole thing falls apart. You can't have a website that works in three of those browsers, but not on iPhones or something. So when you talk about becoming a publicly trusted CA, what you're really talking about is getting those four browser makers to trust you. And they all run what are called root programs inside their organizations. And those root programs decide who they trust and then follow up on compliance from everybody

they've already decided to trust. So when you start a CA, you need to build your systems. Then you need to get them audited against the current auditing guidelines for CAs. Then you take that audit report and you include it with an application to each one of those root programs. So you're submitting at least four applications to four different root programs, and they all have their different ways of applying. Some of them are relatively simple emails or bugs to file, and some of them are longer applications. But you apply to all four of them, and then you wait to get accepted.

And that can take anywhere from three months to three years to get accepted. And then once you're accepted, you need to wait for them to actually put that trust into the browsers. So for something like Chrome or Safari, what that means is you're waiting for them to ship a software update that includes your root of trust in it. And until that happens, until a user gets that update, their device still doesn't trust you. In the case of Microsoft, it's more dynamic. They don't do it through software updates necessarily. If they don't understand, they will query a server and check. So trust in the Microsoft ecosystem can be done pretty quickly. Once you're in, you're in. The real problem is stuff like Android in certain parts of the world. People have old Android devices that

they don't get updates anymore and they never get rid of them. In some cases, they're still manufacturing Android 4 devices and those things are never going to get an update. So if you want to get those devices to trust you, you're really just talking about waiting for them to leave the ecosystem. So the point of this is, between the time that you apply for trust and get approved and then all the devices in the world actually trust you, you're talking about a period of six to ten years. Wow.

So commitment is required, I suppose. I mean, without saying so much. You think about some people's plans for new ventures, whether they're small ideas or big ideas, any sort of itch that's scratched. So we talk to a lot of people who scratch itches around here and do something about it. And you have to think about your commitment level

to said mission, right? If you have a horizon of, a year or two years and you're building a CA, maybe you need to stretch that quite significantly to like five or six or maybe even further. What was your horizon for this? Were you like 10 years, 20 years? Did you kind of know all this beforehand or was it sort of learned as you go? Because you mentioned a lot of this was learning as you go.

So what I just described is the basic process of getting trusted from scratch. And that does require a big commitment. It requires quite a bit of money to get set up to a point where you can pass audits and even apply. And then every year, while you're waiting for all this stuff to happen for six to 10 years,

you have to stay compliant compliant get audited every year so you're talking millions of dollars and six to ten years before you can even be a publicly trusted ca in any meaningful sense that's the basic process there is a way to make a shortcut which is how lots and crypt was able to start without waiting six to ten years first. So we did go through the process that I just described, building up our own rooted trust from scratch. But the world right now does not really rely on that yet because it has not been long enough. So somewhere around mid to late next year,

we're going to switch over to our own root that's trusted from scratch. But from our inception through now, we have what's called a cross signature, which means we knew we didn't want to wait six to 10 years to start offering lots and crooks services. So we found another CA that understood what we're trying to do and was willing to help. And they had a route of trust that was already trusted and what essentially happens is you know we create a contract and an agreement between us and then their root of trust essentially lends its credibility to us so they issued a certificate that our root is trusted by their root and their root is trusted by the browsers basically so. So that's called a cross-signature.

So we acquired that before we did much of anything because without that agreement in place, there's really no CA. So one of the first things we did was get that agreement because without that agreement in place, there's no point in buying hardware and doing anything else because we're not going to sit around and do nothing for six to 10 years. So that was a really critical agreement.

So we got that in place with a company called Identrust, who's been a great partner for a while now. So we're trusted through Identrust today. And mid to late next year, we will stop using that cross-signature and just be trusted on our own. That's a big deal. Yeah. Well, it'll be transparent, that trust, I suppose,

since it's still you, it's still trust, still the same browser, unless you're running an old Android device. That's right. Yeah. Their root is really widely trusted, and that's been fantastic. And it's trusted all the way back into early Windows XP

and all the Android devices. The problem with that root is, the longer a route is around, the more trusted it is, but eventually it expires, and that route expires next year. So I don't remember the exact lifetime on it, but I think that's a 20-year-old route, at least, if not more than 20 years. So the advantage of an old route

is that it has really widely trusted status, but the disadvantage is it's going to expire soon. So we will be switching from an old root that's very well trusted to a younger root that is also very well trusted, but admittedly not quite far back as identrust. But if're still running windows xp on the public internet you might have bigger problems than your certificate probably the same goes for really old versions of android you'd mentioned the dollars involved to uh for the six to ten years and i from just grokking past blog posts from you or others at Let's Encrypt, it's primarily a people cost.

So staff, is there a lot of cost aside from that when it comes to, say, the fast track that you took or the non-fast track? Or the cost primarily people? Certainly the cost for us to run the CA today is primarily people. Roughly speaking, paying staff is about two-thirds of what we spend every year. There's startup costs and then there's ongoing costs. For startup costs, usually those cross-signing agreements do cost money and it's a non-trivial amount of money.

So when another company agrees to trust the new CA, they are responsible for that trust. If the new CA that they're trusting messes up, it's on them. So in exchange for the liability they're taking on by trusting, you know, in our case, just like some guy from Mozilla walks into the office and says, you know, I'm going to quit my job and start a new CA and it's going to do all these amazing things. Like, you should trust. Hypothetically, right? Yeah.

You should trust us and, you know, put your business and your reputation on the line. It's not an easy ask. So these cross-signing agreements, there's a lot of liability involved. And for that reason, they end up being non-trivial amounts of money. So that's a big startup cost for anybody. Then going forward from that, that's probably the biggest startup-specific cost, aside from maybe initial capital.

You're going to need to buy servers and HSMs and things like that. Ongoing, we have to buy a certain amount of hardware every year. We do use some cloud services. We use some external services. But mostly, aside from people, it's about the data centers and what's in them. Publicly trusted CAs are not allowed to operate in the cloud, so we can't run our CA systems on AWS or GCP or Azure

or something like that. We have our own hardware in special secure rooms inside data centers, and they're not even normal data centers. They're special walled-off rooms in data centers with a bunch of extra biometric access and stuff like that. And that stuff is a non-trivial expense and you've got all this hardware that goes inside. You've got to have a lot of redundancy. So we pay for that stuff and that's where a lot of the rest

of our budget goes. In a world where Let's Encrypt is ubiquitous, which is what we're getting to, right? Like you'd mentioned, you know, two and a half years ago, 100 million certificates issued, you know, a month or two ago, a month ago, a billion. It's quite massive growth. In a world where Let's Encrypt is ubiquitous, what's the point of other CAs? Like, I'm just thinking, like, why would anybody not use free? There's a lot that other other cas do that we don't

do so for example we offer one specific type of certificate you can't change very much about it it's you know we think shorter certificate lifetimes are better so we don't let anybody create a certificate that's longer than 90 days in lifetime. And we also don't offer human support. So there are a bunch of reasons to choose other CAs. For one thing, we don't sign the sort of normal contracts. Some people have procurement requirements where they want certain things in contracts from their vendors, and we don't do that.

We don't provide support. You can't pay us for like 24, 24, seven phone support. If you know, if you don't like the type of certificate we offer, you want something else or you want it configured in some special way. We don't do that. So we're sort of one size for everybody. And if that doesn't work for you, then there's luckily a lot of commercial CAs you can go to and they'll be happy to help you, I'm sure. So what you're saying is Let's Encrypt is for everybody, but not for everybody. Yeah, it's a pretty basic option. I mean, we try to be, I think it's basic for two reasons. First of all, we try to be a really efficient organization. So

as complicated as running a CA is, we try to scope that complexity and limit it as much as we can. So we don't want to offer a ton of choices to people because complexity just leads to more bugs, more costs, things like that. The other thing is we're really focused on best practices. So we tend to do whatever we think is the best practice. And an example of that is the cryptographic algorithms that we offer or the certificate lifetimes that we offer. Or you can only get it through certain validation methods the way that we do it because we think those are the only secure ones or something like that. But other people have other opinions.

So between trying to be efficient and trying to focus on best practices, we offer a pretty limited service that I think works for a lot of people, but not everybody. And the web is huge. We may be serving 200 million sites right now, but there's a lot more than 200 million sites out there and they should all be using HTTPS. So there needs to be a robust ecosystem of CAs in the world so that people who need something else besides what Let's Encrypt provides have a place to go. I mean, given the requirements to even be a trusted CA, it doesn't seem like something that there are just handfuls of people listening to the show saying, I'm going to drop what I'm doing today and become a CA. It's just such a long road. I almost think you have to really be invested, I suppose, in securing, and I guess that's a whole different kind of problem or different kind of business.

But given the amount of effort it takes to get there, to start a business today, you have to kind of get to product market fit. Create a product that people want and people will buy. In this case, you can do that as a CA, but still not be able to sell it because it takes you so long to become a valid resource for giving it. The trust, you know, the trust is such a big deal. It's an interesting business to create.

Yeah, creating a CA is not a decision to take lightly, but the way that most people create CAs now is not to wait for that process to play out. They do the cross like you've done. A cross signature or you just go acquire another CA. You just buy another CA. Wow. There are at least a hundred existing publicly trusted CAs.

I'm not sure exactly how big the list is. They buy and sell each other all the time. Sometimes they go out of business. I don't remember how many publicly trusted CAs there are, but there's at least a hundred, right? And if you want to start a CA, like if you're going to start a serious business, one way is to go spend, you know, X millions of dollars on the cross sign, or you can spend X millions of dollars, just go buy a CA that, you know, most of these CAs you've never heard of. They're very small. I mean, they're not, you know, under the couch money, but if you're starting a trucking company and you need to go buy like $10 million worth of trucks, right. That's something people do all the time.

You can probably go buy a CA for, I don't know, something on that order of money. Yeah. So in some ways it's not really that different from starting any other business. do all the time you can probably go buy a ca for i don't know something on that order of money yeah so in some ways it's not really that different from starting any other business i would imagine i mean i only have experience with let's encrypt in the cross sign i've never actually bought another ca but i don't think that the startup requirements are really that different just because most people don't wait for the from scratch process to play out you mentioned the other certificate types i'm just curious on your thoughts on extended validation certificates and the idea of wrapping up identity into encryption like establishing a

secure connection and then you also have this extended validation, so you know at least the CA trusts that the person who owns the certificate is who they claim to be. What are your thoughts on that style? I know you don't offer it, but is it worthwhile? I don't know if I can say whether it's worthwhile for any particular, you know, some people have specific needs or regulatory needs or whatever, so I don't know if I can say whether it's worth any individual in general should do it. But I do have some thoughts.

First of all, trying to include the identity of a legal organization in a cert does not affect encryption at all. You can't tie the two together. You can put them in the same cert next to each other. But EV certs and OV certs, which contain this legal identity information, the encryption is no is, well, there's a bunch of them. First of all, browsers are increasingly not showing that information to users. So there's no point in having an insert if the browsers aren't going to show it to them. And the reason the browsers aren't showing it as much anymore is that most research just shows that people either don't look at it at all

or don't understand it when they do see it. So you're not going to build a secure system that relies on the average user on the internet looking at information and making informed decisions. That's just not how security works. If that's your plan, it's not going to result in generally increased security for anybody it only works when it happens automatically and doesn't require people to to look into it individually so you know there's a bank out there called usaa right and if you look at an ev cert from them it says united something automobile something you know like the spelled

out name of the business is very long and nobody knows what USAA stands for they just know the bank has USAA so when you see that kind of information pop up in an EV cert how can you possibly expect anybody to make a reasonable decision about that so I don't think that it's very useful to put identity information in a certificate that's used on the general internet. And the research backs that up and browsers tend to agree and are dropping that from the UI.

So yeah, that's true. I don't have any stake in this. Like we don't issue it. So in some ways I don't really care, but it seems not very helpful and probably it's not going to have a strong future on the internet no i was just curious your thoughts on it i think at one point it was interesting because it was different like not all certificates you got uh gave you the option so i can recall a day whenever you go to

github.com and it would say, you know, like separately twice almost. It's almost like double branding even. You know, like heavy on the brand side. With a big green background and it showed. Yeah, it seemed official. It seemed cool. It seemed secure. And so I would think, which I don't say,

I don't know all the research behind it, but from a UI perspective, it's probably cumbersome because it's redundant. But it looked different than, say, someone who didn't pay anything for their certificate or didn't buy a certificate that offered that. And it was unique. So you would see it happen on people who would want to pay for it, I suppose. Yeah. The problem there is that once you see it,

it might possibly seem like a good thing to you. Although, again, the research shows that people don't really react to it in a useful way. The problem is you don't notice the absence of it. Like, if GitHub just didn't have that, you wouldn't say, hey, it doesn't have that thing. I'm going to leave. I mean, I've been to GitHub today

and it doesn't have it today. I'm going to leave. I mean, I've been to GitHub today, and it doesn't have it today. I don't care. Right. So it just turns out not to be very useful. And also, there's a lot of issues with how it's validated. Like in domain validation, where you're just proving control of a server before you issue a cert, there are pretty clear and strong ways to do that validation.

When people do identity validation, it basically involves phone calls and faxing around document copies of your articles of incorporation and copies of driver's licenses and stuff like that. It's easier to mess with. And pretty famously recently, somebody registered a Stripe Inc. in some state that's not where the normal Stripe payment company is. And since registration of businesses is by state, you know, they had an EV cert that said Stripe Inc.

And obviously that is not what you'd expect, but it's not a bad cert. They legitimately did own Stripe Inc. in, you. in North Carolina or whatever it was that they did. Right. It's like a namespace conflict, but it wasn't inappropriate. The actual CA that issued that certificate could have went out to their business and got their articles of incorporation and all the stuff in the state that they're in. So it's completely valid. There's technically nothing wrong with that cert you know then they sort of arbitrarily revoke the cert because they say like well you know it may not be it you know we just don't like that cert you know it brings a lot of

arbitrariness into it and yeah that is a cool party trick and it demonstrates some problems with ev certs but the real issue is that nobody seems to care what's in the cert anyway, so it doesn't matter if you... Nobody really looks at it or makes security decisions on the basis of that stuff anyway, so it doesn't matter if your namespace conflicts or something are sort of a second-order issue. Right. I think it's interesting that it was, for a time,

almost a status symbol amongst technology companies to have that. It was like, we've arrived or we have the browsers, when the vendors said, yeah, let's just go ahead. No one looks at it except for nerds. Most people don't even look at the address bar. They don't even know it exists, which is why the number one thing people Google is Google or Facebook. They Google Facebook to go to Facebook.com when they could just type it into their address bars because people don't look at the address bar,

let alone is the background green or does it have the thing. So really the browser vendors made that not a thing. So I mean, that's super interesting. They've kind of like, because that was an advantage of a certain certificate author

or it was kind of an upsell. Isn't it always an upsell? Hey, get the extended validation cert. And it's like, just the movements of the web and the decision making of the browser vendors basically just like quashed the value there because it was really only in the status symbol like you said

yeah i mean the advantage is sort of a quote-unquote advantage right it's not really an advantage because it doesn't actually mean anything um just takes up a bunch of ui space yeah again it's kind of fascinating if your plan for security is to show average users some information and then expect them to make a really good decision based on that information, that is never going to work. It doesn't work for EV. It doesn't work for anything else. What up, nerds?

I've got some pretty awesome news to share with you. Pluralsight is totally free for the entire month of April. I'm not kidding. Seriously. Head to pluralsight.com slash changelog and skill up while you stay at home. For the entire month of April, you'll get access to over 7,000 courses from experts in software development, security, cloud, and data. There's never been a better time to skill up. Head to Pluralsight.com slash changelog.

Again, Pluralsight.com slash changelog. So Josh, we've been talking about Let's Encrypt's success over the five-ish years you've been doing this. A lot has changed since the beginning. A lot has changed since 2017 when we had Jacob on the show saying, Let's Encrypt the Web, mostly extreme amounts of adoption. You have some stats in your Billion Certificates blog post that in June of 2017, approximately 58% of page loads used HTTPS globally, 64% in the U.S., and today that's 81% of page loads use HTTPS globally.

I think you mentioned that earlier in the conversation. And we're at 91% in the United States. I wanted to reiterate that. That's a massive number, 91% in the US. So you guys played a large role in that. And I'm curious because there's also been like the web has changed alongside you and the trends are changing

and security is more important and all these things. So I'm curious how much you feel you've been pushing this up the hill and how much you feel like maybe you've been riding a wave in the last couple of years. It doesn't feel like pushing it up a hill so much. I think there was a lot of demand, right? I think developers understand that using HTTPS is a good thing.

They understand that without it, you're not secure. I don't think it's hard to convince most of them to do it. I think they're ready to do it if they have a reasonable option for doing it. And by reasonable, I mean very easy to use. So, you know, we put our service out there and it's not that hard to convince people to use lots of encrypt. We don't really market or engage in too many activities around really trying to convince people to use Let's Encrypt. Most of our efforts revolve around trying to get people to give back for using Let's Encrypt

and keep stuff going. But yeah, it definitely doesn't feel like pushing something uphill. It feels like people want to do the right thing, they just need the tools, and now they have them. So I would say that I think the developer mindset, as my own personal opinion and experience, has changed probably from, you should encrypt anything that's important.

I'm talking like three to five years ago, like that was kind of the ethos. Was anything important, like if you're signing in, if you're, obviously if you're making e-commerce transactions, those things should all be encrypted. Taking passwords, et cetera. But that's, those things should all be encrypted. Taking passwords, etc. But that's pretty much what needs to be encrypted. I think nowadays, generally speaking,

the ethos is just encrypt it all. Encrypt all the things. The thing that gets me about the first argument that only important things should be encrypted is that people need to remember that when data is not encrypted, not only can it be read by other people need to remember that when data is not encrypted, not only can it be read by other people, but it can be modified. So any unencrypted traffic can have stuff injected into it. And it doesn't matter whether that traffic is important

or not. So if you're on your banking website in one tab and you think, oh, that's important, that needs to be encrypted, and you're over in another tab looking at memes or something and you think, these things are not important. These are just some mass media gifs flying around the internet. Why does that need to be encrypted? The problem is that unencrypted traffic can be modified. So you can have malware or some kind of exploit loaded into the traffic for that tab that exploits your computer and now does stuff with your banking info because they owned your browser through the unencrypted traffic in the meme tab. Don't go changing our memes.

It is really not a good idea to try to draw distinctions between what is important and what is not because it's all exploitable in the same ways. And that line just never gets drawn in the right place. Yeah. Which makes celebrating a billion certificates all that more important because you know you started out at the bottom and now you're here to use a rap song very wisely you know and that's that's the thing like use that rap song very 2017 100 million now we're at a billion five years later that's that's a big deal and that means that so much more traffic and so many more people uh are not getting advantage taken over them or the opportunity to get the advantage taken over them because of being secure.

And in a day prior to this, it costs money to enter. Not that the money factor was a thing. It was just a barrier to the entry to using SSL. One of the Jareds and the me's of the world before said, hey, only important traffic needs to be encrypted. And now it's like, well, everything for those reasons. And that's a big deal. I mean, but you mentioned that you haven't done much to get there. So I mean, going from a zero to a hundred million

to a billion, no marketing, not much involved. It's mostly community work. You know, when you have to account for how you got here, how did you get here? What are the things you did to do that specifically? Well, like I said, there was a lot of pent-up demand, and we gave people the tools, and we made them easy to use. And that's really the gist of it. And then some people start doing the right thing.

You get the numbers high enough, and then the mindset of the world switches from, you know, HTTPS is an optional thing that you can have if you want to spend time doing that, to HTTPS is the standard thing that you need to do all the time. And if you don't do it, you have a problem. So one of the biggest accomplishments, I think, for us and for everybody, you know, it's not just Let's Encrypt. We're not the only reason the web is where it is today. There's lots of different people working on different great projects around the world that have helped promote HTTPS.

But one of the big accomplishments of that community is that HTTPS is considered the standard today. You set up a website. If you expect people to visit it, you need to have HTTPS. That's a huge mindset change. Sometimes I think about ways in which the internet has changed similar in the past for other technologies. It is hard for me to imagine, or not imagine it. I don't know everything about the history of the internet, but I don't remember any other thing fundamentally changes how almost all this traffic flows across the internet in less than five years

like i can't think of another watershed change to how the internet functionally works that played out that quickly i mean i'm a huge fan of ipv6 but that's like that transition has been dragging out for a long time transition of all time you know and when we started lots and Crypt, we were thinking this cannot be an IPv6 trend line. It can't be that way. We've got to make sure this happens much faster than that.

There's a lot of other improvements to make the internet too. I hope this serves as an example of if we want to make change, we can do it. There's a bunch of other stuff we should fix. It is possible to change major parts of how the internet works in big ways in a few years with, you know, under the right circumstances, with the right plan. What you're sharing here reminds me of this idea that I haven't quite

verbalized yet, but it's this cog mentality. So if you've ever heard of Seth Godin, he wrote a book called Be a Linchpin. I think it's just called Linchpin, but the idea is to be a linchpin. And I think in many ways we try to, as individuals, be really important, right? And that kind of goes against the idea of cog mentality, which means that you're just a very sharp, very specific, very purposeful thing as part of a much bigger, much more grand whole machine. And so if it weren't for the use of the world, Josh, doing Let's Encrypt and all the effort here, then the browsers wouldn't be able to do its thing and then the site developers wouldn't be able to do their thing.

And so all these things are sort of in concert, a system. So this idea of a cog mentality really rings true here. Yeah, well, we're happy to do what we do. But like I said, there's a lot of people who play in this. I mean, running CA servers and providing the API is certainly an important part of this. But we wouldn't be anywhere near where we are today if there weren't hundreds of people out there

writing Acme protocol clients that work with lots of crypts so people can just download software for whatever they want it works yeah the browsers have done a great job incentivizing moves to hdps by limiting new technology to hdps connections and some ui work things like that so it's been the browsers has been open source community lots of people involved. Even within Lots Encrypt, there's so many people involved in it. There's the engineers that work on it. But our sponsors, our funders, that's huge. We don't go anywhere unless somebody decides to write a pretty big check.

People who make those decisions to write those check, I feel like they often don't get enough credit because it's not like fun and open sourcey, but that's a big deal, you know? So the fact that there are people out there and companies who understand what we're trying to do and they're willing to write those checks and, you know, stand up and really make the internet better,

that's where it starts. Are there any standout organizations that have been supporting you either in big ways or for a long time that you'd like to give a shout out to? Because like you said, they don't get much credit, maybe a logo on a webpage somewhere. But do you have any major supporters?

Like, well, it wouldn't be here if it weren't for this company or this organization. Yeah, we've got over 70 corporate sponsors, so I'm definitely not going to be able to list them all here. But our Platinum sponsors are our biggest supporters. They write the biggest checks, and they've been fantastic. And companies don't make the decisions. People inside those companies make the decisions,

and I'm so glad that those people understand what we're doing and get it done. Our Platinum sponsors right now are mozilla cisco electronic frontier foundation ovh a company that i think not a ton of people in the u.s have heard of but they're a great huge cloud provider in europe and they have just been fantastic since very early on and uh google and specifically with google chrome team one thing that's amazing is what you've done with that money so in that same post you talk about how you're serving forex the websites that you were back then and of course now here you are at 200 million so even more websites and your budget hasn't increased 4X or anywhere near that, right?

You went from a $2.61 million annual budget in 2017 to $3.35 million. Now from 11 staff members to 13 staff members. So only adding two staff in the course of three years and 4Xing your website serve, that's pretty good numbers. Yeah, internally we are obsessed with efficiency. Like I said, it's a really big deal for people to entrust us with millions of dollars every year. There's a lot of good in the world that money can do. So when that money comes to us, it's our obligation to make sure that we use it wisely and do the most good we can do with it. And that means delivering the best service to the most people that we can.

We take that responsibility to be good shepherds of that money really seriously. So whenever we talk about a new service or a new feature or some way in which we're going to expand our service, we have a whole bunch of things that we think about to make sure that we're being efficient. One of the most important things is, does it require any people to be involved with anything anywhere in the chain? So, you know, one of the reasons we don't offer phone support is if we did, we would have to fill a skyscraper with people sitting by the phone, right? So when we think about delivering a feature that feature cannot require support we need to make sure that it's so easy to use and so easy to document and so easy to automate that the people consuming the feature should not need support even the people who are the least technical

it should just happen and if they do need support it should be as simple as reading a very easy to find bit of documentation. So ease of use is, again, hugely important for efficiency. If it's easy to use, then people need to talk to you as much. You know, if even some very, very small part of 1% of the people using Let's Encrypt needed to actually talk to us on the phone about something, that would just be overwhelming. We can't do that. So everything has to be very easy to use. Internally, we think a lot about how much data do we store. We're basically allergic to data, right? We only really hold on to what we need to hold on to for either compliance purposes or to debug our own systems. But aside from that,

we don't want to have more sensitive information than we need. We don't want to be sitting on piles of information where we have to pay for storage servers and things like that. We tend to just do what we need to do and not hold on to tons of data. When we need to use an external service, we often find partners who are willing to provide the service free of charge to us, essentially as sponsors or donors. Yeah, we're just very concerned about efficiency.

So there were only 13 people today. And, you know, we have a lot of specialized systems, but it's probably, I don't know what people imagine we run when they think about like, what is Lots and Crips actual hardware? But, you know, it's like about three racks full of hardware. It's not a ton of hardware. It's all very carefully maintained in some ways.

But modern servers are crazy powerful, right? You can fit a lot of stuff in there. You don't need a lot of physical space. So, yeah, we've got a couple different data centers, maybe three racks of hardware between them, and that's triple redundant, right? In theory, if we needed to, we could just run the CA out of one rack of hardware, and that would serve all 200 million sites pretty easily.

So if you automate everything and get computers to do all the work for you, you can be pretty efficient. I mean, it still requires, you know, I think this year we're going to spend a little under $4 million, but that's really not that much money. I'm fairly confident that there are, you know, Fortune 500 companies out there that spend more than $4 million on their internal PKI systems. Right. So you mentioned earlier that you weren't sure you wanted to even spend the next five years of your life doing this, doing a CA,

but you felt like somebody had to do it and you were well-positioned and willing to. Here we are. You've done over a billion certificates, 200 million sites served, all these big numbers. And I think more importantly, those global trends, which from the very beginning y'all have said you wanted to encrypt the whole web right like the global trends i think are probably more

important to you than your let's encrypt footprint on that 81 globally 91 in the u.s do you feel like let's encrypt has accomplished its mission is there still a lot left to do? Well, in the US, there's still 9% of page loads are not encrypted. Globally, it's still 19. And I think if you're an engineer, you probably understand what I mean when I say something like 90% of the work is involved in finishing the last 10%. You know, the 10 or 20% around the world that haven't moved to HTTPS yet. They're almost by definition, the ones that are hardest to reach.-EPS yet. They're almost by definition the ones that are hardest to reach. They either don't know or they don't have the tools or they have some reason why they haven't switched. And it's the people for whom it was easy to switch have mostly

already done it. So I think that last 10% is going to be pretty, it's not going to be as easy as the 10% before that. And also, this service needs to continue going. I mean, I don't know how long Let's Encrypt is going to need to be around, but it's quite possible that it'll be around 10, 20, 30 years. I have no idea. But it's not like once you encrypt it in site, your work is done. You've got to continue to issue new certificates to them on a regular basis. So we need to be around for that.

And in order to be around for that, we need to stay on the top of our game in terms of compliance and security. And at the end of the day, people need to trust us. That's what it really comes down to is we're never done because trust is never done. If at any point the world loses confidence in us, we can either lose trust technically where browsers don't trust us like on a cryptographic level. If our donors don't trust us, browsers don't trust us like on a cryptographic level if our donors don't trust us we don't get the money we need to continue so the job is certainly not done we need to maintain high standards and stay trusted for a long time now and he said i wasn't exactly like thrilled about the idea of spending a huge chunk of my life building a CA and running it, but it turned out to be really great. It puts me in contact with so many people that are really

passionate about making the web a better place. And that's something I am very happy with. I love working with our board members and our partners and our community. It's like I can have enthusiasm on tap anytime I want it, just call people up and talk about what's happening with us so it turned out to be great and our staff are wonderful so as a job I really couldn't ask for more. The benefits of a job often outweigh the job itself sometimes you don't really care for the job itself or the mission not so much the mission mission but the fact that you get to interface with so many people who care has got to be uplifting for you.

In light of what Jared asked you, I'm sure that this will be a little easier for you to answer. Maybe not. But what's on the horizon for you? What's the big picture? You mentioned 10, 30 years down the road, so you've got to have some sort of idea. Give us a snapshot, maybe one to two years in the future. What's something nearest on the horizon not many people know about that is something you can share today?

Well, we're going to keep grinding and finishing encrypting the rest of the web that we've already talked about. There's so many more things that need to be worked on. I don't know that, you know, Let's Encrypt's mission is pretty well scoped. We issue certificates and our goal is to get to 100% encryption and be entrusted while we do that. And in some ways that's a fairly narrow scope and it's part of why we have done well, I think. But I think in doing this work, we've realized where there are a bunch of other issues on the web that we need to solve. A couple of the ones that are top of my mind are, there's a protocol called Border Gateway Protocol,

BGP, and that is the protocol that's used to decide how and where traffic gets routed around the internet. So if you're going to send a packet from Seattle to Philadelphia, what exact route is that going to take to get there? And that's all determined by BGP. That protocol is not secure. It's very vulnerable, and I think the only reason it hasn't been exploited more is it's not very popular. People don't know about it.

The attackers don't know about it. They've also had easier targets. But for as many security problems as we have, we've done a pretty good job working on them. And I think a useful way to think about the next 10 or 20 years of security is that I think we're going to keep pushing attackers down the stack. So you improve application layer security, and then maybe the next step down the stack is network layer, like transport layer or something like that. HTTPS, you encrypt that,

and then the attacker's got to move on from there. And right now, I think the next layer down that has not been exploited to its full potential yet is BGP. I think of that as the soft underbelly of the internet. Attackers are going to take notice of it and they are going to get better at it. And they can cause massive outages by doing that.

They can reroute traffic wherever they want. So I'm concerned about BGP and that has some pretty direct impact on Let's Encrypt in that certain types of BGP exploits can be used to mess with certificate issuance processes. That's true of any CA. It's not specific to Let's Encrypt. It's just sort of a part of our risk profile as an industry. But it's a hard thing to secure. So I'm very interested in what we can do about BGP security going forward. That's going to require a lot of the big companies

that operate the major pathways on the internet to change how they do things. So that's one thing I'm interested in, and we do some work around that at Let's Encrypt to mitigate the problem right now and also try to invest a little bit in the long-term solutions there. Another thing that both affects us

and that I'm personally pretty passionate about is memory safety. So in the same way that it seems crazy to us now that you would start a major website and not use HTTPS, we know so much about the risks of that, and it just seems crazy to do that now i think we're also going to come to a point where we feel like it is crazy how people say stuff like well i'm running a bank and i need to do some reverse proxying so i'm going to spin up uh you know an instance of nginx or ap Apache and do my reverse load balancing. Because what you're really saying there is,

why don't I just stick several million lines of C code on the edge of my network? And that'll probably be fine. That code is not safe. We certainly should not be writing any new code in C and C++. The opportunity for vulnerabilities, I mean, it absolutely will have vulnerabilities and that we need to get that type of code away from our networks to start with, and then probably away from most other things too. So I would hope that in 10 or 20 years, we think it's crazy to be deploying,

you know, major or maybe even minor pieces of software that are written in languages that are not memory safe. So we're trying to remove code written in C and C++ from our infrastructure at Let's Encrypt. I think that's just a basic part of diligence of providing secure infrastructure. If your stack is like some giant pile of C++ or C at the network edge, followed by OpenSSL written in C, followed by Linux kernel written in C, GLibc. Your whole pathway has got all this code that is like, you just know it's full of security holes.

It absolutely is. You just can't claim that those are even close to secure systems. They're absolutely not. And we're going to look back on this and say, that was crazy, and we have better options today. So we're trying to remove that kind of code from inside Let's Encrypt because it's a huge liability, but we're also, we slash I am looking into ways that we can try to move the needle

on this problem in software in general. What's your first step, you know, we slash I am looking into ways that we can try to move the needle on this problem in software in general. What's your first step, you think? What are some of your early insights on moving that needle? The first step is don't write any new code in C and C++ or any other memory unsafe language. That should just be a given. You know, if you can tolerate a garbage collector, if that's fine, then you have a ton of options. Java, Go, whatever. If you want a memory-safe language that doesn't use a garbage collector, go use Rust. You have that option

now. Seems like a next step would be having viable replacements for a lot of the software that's already out there. Yeah, the next step is we need to rewrite all the software that we already wrote in C and C++ and replace it it and when i tell people that they the most common reaction is like you can't possibly expect us to rewrite the world that's so unreasonable you're not a realistic person when you say that and you know i really strongly objected that reaction. We're in a world full of talented people who care, and we can absolutely accomplish that if we want to. If your goal is to rewrite a major web server

or a major proxy server or a major library or whatever in Rust, let's just do it. Yeah, it'll take five years. It'll introduce some logic bugs along the way that will get fixed. But in the end, this software is going to be around for a very long time. And we need to eliminate that massive class of bugs because vulnerability scanning and audits and static analysis, pen testing, that stuff doesn't even begin to deal with the problem like it's a good thing to do if you're stuck with cnc plus plus but it's absolutely not going to eliminate the bugs that stuff's not going to go away until you rewrite it what we're doing right now where

we just you know spin up giant piles of cnc plus without thinking about it is, we should not be doing that. We can't be doing that 10 to 20 years from now if we want to try to have a more secure of the world than we have now. So I think we need to think bigger. We just need to think like, yeah, let's rewrite the world. Rewriting a big web server or something is a big project, but I'm sure there are teams that any number of companies that could accomplish it on their own without help if they just decide to do it. It would be five years, but whatever. Five years

from now, put in some effort. Now you've got a much more secure software system. So I'd like to just see some more ambition and some more optimistic thinking about this stuff. I think it's really important. I don't want to be suffering from buffer overflows in everyday software that sits on the network edge 10, 20 years from now. My guess, Josh, is that you're well-positioned to encourage, considering what you've done in the last five years.

Going from zero to a billion certificates issued is a big deal. You've found a way to create a CA in a world where it's very difficult. Obviously, there's protocols by, was it the cross-signatures, is that what it's called? Yeah, cross-signature. Cross-signature. I mean, just that alone, I mean, that was a smart play, and you've been able to do so much.

So I think you've probably piqued our interest and as well many listeners listening to the show by saying so and we need we need you out there petitioning for this and encouraging those out there that can do this to take on this mission to do so and not look at the five or ten years that it might take to do it lightly because we see such blowback when we don't consider the large scale costs over many, many years. If this software isn't going to go anywhere in the next 10 or 20 years or 30 years, then we're going to rely on it. And just securing the web is more important than it has ever been.

Having secure software that doesn't have memory issues or unsafe memory where you can do these things. It's, it seems so clear to me. Yeah. It's got to happen. Josh, thank you so much for, for your mission. Thank you so much for let's encrypt and the work you all have done and to you and the team. I know you're not a lone soldier in this mission, but the many behind you enabling this,

but without you and many others doing this, you know, we would have 51% less internet secured. So thank you very much for that. We appreciate your mission. We appreciate you. Thank you, Josh. Thank you so much. Alright, the next step is to head to the comments and let us know what you think about Let's Encrypt and their awesome mission to secure the

internet. Head to changelog.com slash 389. Of course, you can comment on all of our podcast episodes at changelog.com slash 389 of course you can comment on all of our podcast episodes at changelog.com head to the show notes and click discuss on changelog news we'd love to hear from you and we get asked this all the time how can you support us the easiest way is to tell your friends that's the best way podcasts grow word of mouth send a text send a tweet send an insta story whatever works we appreciate it and as you know, Jared and I host The Change Log. Our beats are produced by the Beat Freak Breakmaster Cylinder.

And we're brought to you by some awesome partners, Fastly, Rollbar, and Linode. And of course, you can subscribe to all of our shows with a master feed. Head to changelog.com slash master to subscribe or search for Change Log Master in your podcast app. You'll find us. Thanks again for listening. See you next week. Bye.