The Changelog: Software Development, Open Source - Let's Encrypt the web (Interview)

Episode Date: March 18, 2017

Jacob Hoffman-Andrews, Senior Staff Technologist at the EFF and the lead developer of Let's Encrypt, joined the show to talk about the history of SSL, the start of Let’s Encrypt, why it’s important to encrypt the web and what happens if we don't, Certbot, and the impact Let's Encrypt has had on securing the web.

Bandwidth for Changelog is provided by Fastly. Learn more at Fastly.com. I'm Jacob Hoffman-Andrews, and you're listening to The Changelog. Welcome back, everyone. This is The Change Log, and I'm your host, Adam Stachowiak. This is episode 243, and today we're joined by Jacob Hoffman-Andrews. Jacob is a senior staff technologist at the EFF. He's also the lead developer on Let's Encrypt, which is the free and open automated certificate authority. We talked about the history of SSL, the start of Let's Encrypt, why it's important to encrypt the web and what happens if we don't, their project CertBot, and the impact Let's Encrypt has had on securing the web.

We've got three sponsors today, Linode, TopTow, and Rollbar. First sponsor of the show is our friends at Linode. We host everything we do on Linode servers. Head to linode.com slash changelog. Get one of the fastest, most efficient SSD cloud servers, SSD storage, 40 gigabit network, Intel E5 processors. Use the code changelog2017. 20 bucks in credit. Head to leno.com slash changelog.

And now on to the show. All right, we're back. We're talking to Jacob Hoffman Andrews. And Jared, this is a show we've wanted to do for a while. We've been using Let's Encrypt, featured it in ChangeLog Weekly. Yeah, just a huge, ambitious project coming out of EFF, Mozilla, and others. One that we're all behind and we are grateful for and huge supporters of the Let's Encrypt initiative and all the technology behind it. So, Jacob, thanks so much for joining us on the show.

Yeah, thanks for all the kind words. You know, I think one of the best things about the Let's Encrypt project is just how much support we've gotten from the whole community out there. So far, I mean, we would definitely define it as a, so far, a massive success. I would, at least from my perspective. Of course, what is Let's Encrypt? It's, you know, free and automated certificate authority. And what do you do when you want something to be a huge and massive success? Well, starting off with free is a great way of helping that along. Jacob, why don't you just give us real quick for the audience sake, an idea of your role with the project and what you do at the EFF? Yeah, sure. So, you know, I wear two hats.

One is as the technical lead for Let's Encrypt itself, which is an independent nonprofit. And my other hat is as a senior staff technologist at EFF. So I wrote code for Boulder, which is the custom certificate authority software we wrote for Let's Encrypt. So are you coding in your role often? I know you said you wrote Boulder. Is that your main thing, your only thing, or you have other things that you're coding?

Yeah, it's a bunch of different stuff. And I'll correct that I didn't write Boulder entirely myself. It's definitely been a collaborative effort from the very beginning. You know, most of my day is coding generally, although it varies quite a bit. Lately, I've been doing some work on documentation, both user-facing documentation and certificate authorities have to publish this kind of very formal document called a CPS that details all the ways in which they validate certificates and issue certificates. And so we've been revising that so it's easier for everyone to read. And some of my time is also spent engaging with standards communities. One of the interesting things about Let's Encrypt is it's based on a new IETF RFC, which to break through the jargony acronyms there, it's an internet standard and it's getting close to standardization.

And the cool thing about that is, you know, someday other CAs will hopefully implement the same protocol and you'll be able to use software that works with a number of different CAs interchangeably. Let's go back a bit and learn about the history. We always love to find out what's the inception point or the origin story for projects like these, especially when they're big, ambitious projects. We just had a show all about the Atom text editor, and that's Atom, not Atom.

And Atom and I were talking with Nathan about that, and we just asked, it's such an ambitious goal when you're starting a tech side pretty much from scratch. It's a long term thing and so how do you, how does it start and then how do you plan out? I would say that this initiative to encrypt the entire web and then like the machinations of how you actually go about promoting and doing that is even more ambitious. So can you tell us how it all started? Well, I'll tell you my personal story of how I joined the movement to encrypt the web.

I had been a follower and supporter of EFF for many years. And EFF had been promoting this notion of encrypt the web, the whole thing, it's got to be encrypted all the time to make users safe for a while. And I'd been kind of following that along. And in October 2010, I was working at Twitter at the time, and Eric Butler published this Firefox extension called FireSheep. And it did this clever thing. It took an attack that everybody already knew about and made it really accessible and obvious. So the problem with HTTP,

which is the non-encrypted version of HTTPS, is that somebody with network access can sniff your web browsing as it goes by. And that includes not only what pages you're visiting, but also your cookies. And your cookies, you know, when you're logged into a website, are what authenticate you to the site. So if you can copy somebody's cookie, you can become them. So FireSheep demoed this by, you know, you would install the extension

and it would sniff the local network for people visiting Facebook, Twitter, a number of other sites, you could even write your own rules and it would make it really easy and it would pop up not only people's handles but their avatars and you could say oh who's this? This person's on the network, I want to

become them and you just double click on it, it sets your cookies and suddenly you're in their account. So I was thinking, you know, wow, this is a really good demonstration of why we need HTTPS. And we're at kind of this unique tipping point in the history of the web where HTTPS has gotten cheap enough to implement. You know, I think Gmail had just recently enabled it for Gmail access and had kind of published a blog post saying, hey, this is actually pretty cheap and easy. And, you know, combined with that,

there was this demonstration of why it was so important. So I started working within Twitter to make all of Twitter HTTPS. And, you know, I expected that to be a three-month project. It wound up taking about a year in the process.

I moved on to the security team and became a security expert, which I wasn't before. Why did it take three times what you thought or four times what you thought? Yeah, so there are a number of interesting obstacles. I think for any site moving to HTTPS, one of the tricky things is what's called mixed content. So when you load a web page, it's not just the one URL. It includes a lot of other stuff. It brings images, iframes, video, JavaScript, CSS. So you might have an HTTPS URL for a page, but an HTTP URL for the images or the JavaScript or the Flash even.

Back then, Flash was more of a thing. So browsers treat that specially because they want to be able to say, this page load is securely encrypted or not. And if the main page is HTTPS but the JavaScript isn't, it's not safe. Because somebody can just man in the middle that JavaScript and replace it with something malicious. So if we had just gone straight to HTTPS right away, we would have gotten a lot of these mixed content warnings. So first we had to fix up all the internal URLs

so they would also be HTTPS. But the big problem then was there was a lot of third-party content. So if you tweeted a YouTube video, for instance, or a Flickr image, the page would load JavaScript and images from this third-party source. So we had to make sure all those third-party sources

had HTTPS. And some of them didn't yet, so we had to go bug them. So that was kind of on the mixed content front. The other interesting thing was we were deploying a new load balancer at the time. And there's some concern about how load balancing HTTPS to the old front ends would work.

So we waited on the deploy of that so we could really scale it up and be faster. You mentioned the cost had become low enough to implement SSL or to implement HTTP2. What does involve HPS?

Sorry, not to my bad. What's the, what's the actual cost behind that to make it? So your example was Gmail and they said that it become low cost enough to actually implement. What's the cost to bear when doing this at that level? Yeah. So there's, I think, let's say, maybe three main costs I like to talk about. You know, the one that used to be a big deal that people thought was a big deal was CPU cost.

So, you know, encryption was expensive. And if you enabled it for all your users, you know, you might see a big spike in the amount of CPU you used, which would mean you'd have to buy more machines sooner. But computers were getting faster. And so one of the key figures that Gmail published in their blog post was 1%. They found that the CPU usage went up only 1% after turning on HTTPS for all their users.

And one of the interesting things about cryptography is the computers getting faster means that they can do the encryption faster. It also means that somebody trying to break encryption can go faster and try to break it faster, but not at the same rate, right? So it's not like oh encryption got twice as weak when the uh computers got uh twice as fast i mean in some very technical way you can you know brute force twice as fast but the margin of difficulty is high enough that

uh we're still safe yeah because i mean the whole point the whole point of the process is it's easy to calculate on the one side and it's really hard on the other. And so your processor speed, your wins are bigger than your losses in terms of that. The one that I always thought about was the network cost or the connection cost. Is that your number two? Yeah, so actually that's a good point. I should bump the list to four. Um, so, you know, in terms of performance, uh, you know, what somebody sees when they're loading your web

page, uh, there's a bit of a performance cost, uh, for the handshake. So normally when you connect to a website, uh, you do a, a regular TCP handshake, um,, which costs about one and a half round-trip times, or RTTs. So that can be commonly on the order of 150 milliseconds, depending on your connection. When you connect to HTTPS, on top of the regular TCP handshake, you also need to do a TLS handshake. TLS is the encryption protocol that underlays HTTPS.

So that adds another one or two round trips to that first page load. That can be mitigated somewhat for reconnects. So if you're connecting to a web page for the second time, you can take advantage of some cached information and bring that down to one round trip time. So there's, you know, a slightly increased time to load cost for HTTPS on the order of, you know, tens to hundreds of milliseconds, depending on how bad your connection is. Uh, but you mentioned HTTP two earlier, and I think, uh, this is a really good, uh, time to introduce what HTTP two is. Um, do you want to talk a little about HTTP two?

Sure. Yeah. Um, so, uh, one of the big problems with HTTP one is that you only get to request one resource at a time. So you have to request it and wait for the full response and then request it, request another one, wait for another full response. And so in order to load pages faster, browsers started opening multiple connections to the same host. And so they would kind of get this simultaneous download advantage. But this has a number of flaws, and I won't go into the flaws in a ton of detail.

I think you could do a whole show on HTTP2. But the gist of it is we needed a protocol revision that would let browsers request a whole bunch of stuff and servers to kind of send it back as it came in. And that allows a really big performance win. So that protocol was originally called Speedy. It got standardized and it's now called HTTP2. And one of the tricky things about it is

it's such a radical departure from how HTTP1 was encoded that it wasn't really practical to deploy over HTTP because you have a number of what's called middle boxes. So these are machines that sit on a network and try to intercept traffic, especially HTTP traffic, and do something to it. They might be caching, they might be siphoning off a copy for inspection, they might be scanning for viruses, but a lot of these middle boxes screw up HTTP in various ways, or they don't cleanly pass through things they don't recognize. So the implementers of HTTP2 decided that it would only be supported on encrypted connections because the middle boxes can't generally interfere with those.

It was the only practical way to deploy it. So what that means is today, the incentives are a little flipped. You actually get a performance win from using HTTPS because if you do, the browsers accessing your site can upgrade to HTTP2. And so you get all these great multiplexing wins. So that was a very nice summary, by the way. We do have a show, Adam, I didn't get a chance to

pull it up, but Ilya Grigorik has talked to us on a show about HTTP2. So for those interested, want to go deep into that, Ilya knows tons about HTTP2. I don't have a show number. So just, we do have a search function now on our website, which people have been asking for. Just go there and search that. You'll find that episode. And it's very good. So you were going through your list. Oh, you get the number?

Cool. Episode 161. So changelog.com slash 161. Cool. Did you use our search function? I did. Very nice. Very nice.

That's dog food. The only problem is I'll say here on air is that I had to put the slash in there. So there's two searches. Technically, you could do HTTP two or slash two. You know, so that's the people that named its fault. Yeah. So you're going through the costs. And I assume the next one is going to be cost, you know, to purchase a certificate.

That's got to be in there. Right. So continue down that path. And then we's got to be in there. Right. So, uh, continue down that path and then we're going to get close to our break. We want to get to the point where you've, you know, joined the initiative. So get us there. Tell us about, tell us about that cost because that's, that's a big one for not for the Googles of the world, but it's big for, uh, small and independent site owners. Totally. So, you know, uh, it used to be the case that you'd have to pay a certificate authority. You know, there are only for-profit certificate authorities.

You'd have to pay them to get a cert. And when SSL, which was the predecessor to TLS, was first developed, certificate authorities were rare and it was uncommon to even get a certificate. And so they charged a lot, could be on the order of hundreds of dollars. You know, in 2010, you could get a certificate from a commercial CA for on the order of $15, which is a lot less, but it's still, you know, it's not nothing. And, uh, especially if you're running, say an open source project and you're not paid for it, uh, you balance that against, do I really want to bother?

In 2010, there were a couple of other, not a couple of other, there were a couple of commercial CAs that did have a free product. So you could get a free certificate from Start SSL. Actually, I think Start SSL was the only one at the time, and WoSign started up later with their free offering. But it was really difficult to use, and that's kind of a nice segue into the fourth and what I think is the most important cost of implementing HTTPS, and that's human time. HTTPS and TLS configuration is a really arcane specialty. And, you know, most people don't have that specialty and shouldn't have to learn it necessarily. But with older CAs, you had to learn a certain amount about what even a certificate is and how to install it correctly and how to configure your cipher suites.

And there's a lot of ways you can mess it up in ways that are just really hard to figure out what's wrong. And it just, it takes time. You know, even if you spend the time to learn it, you know, even for an expert, we actually did a couple of informal experiments. We had, you know, an experienced developer try to set up a certificate on their website

during the lead up to launching Let's Encrypt. And it took them, I think, three hours. And so at a certain point, you just start to think this is not worth it. It's a difficult task, that's for sure. Yeah. So this was all in the context of, we started this branch of the conversation

talking about working at Twitter, you know, we had dozens of certificates we had to manage and, you know, we had a pretty big engineering staff at the time, but, you know, even then we would screw it up from time to time, you know, would forget to renew a certificate or would misconfigure one. Uh, and it consumed a lot of time from some skilled engineers. Um, so, you know, those are the things that kind of got in the way. But eventually we turned it on. And the cool thing now is if you use Twitter, you don't even have to think about HTTPS.

It just happens for you. So you spent that year at Twitter getting them on HTTPS. And then at a certain point, the Let's Encrypt initiative and the EFF either came to you or you came to them. Get us there. Yeah, so I came to EFF. I was thinking, you know, I really want to reach out and do more for the web in general.

And at the same time I had been working on HTTPS, some folks within EFF and within Mozilla had independently been trying to get a certificate authority started for kind of the same reasons, right? They saw the need to encrypt the web. They saw that the lack of free certificates from a non-profit automated CA was an obstacle and they wanted to improve that. So I joined up and started working on the project. And in 2015, we launched. We started a beta, I think, in September or October, a closed beta, and then we opened it up

to the public in December. And since then, it's just been kind of like a rocket. On fire, yeah. Yeah, we've issued, well, you know, the main stat we count is not the number of certificates we've issued, but the number of certificates active. And there's some reasons for that around certificate lifetime. But the number of active certificates issued by Let's Encrypt about 30 million at this point wow that's huge that's 30 million sites or and or servers that previously were not being encrypted and previously subjected to this potential man-in-the-middle issue whether it's the people in the service obviously but

they're at risk yeah so some of some of those. That's true. That's true. They could have been using others, but obviously. Still. Maybe half aren't. Well, actually, a lot more than that. A colleague of mine actually did this analysis. We can talk about CT logging in a little bit, but basically there's enough data available

publicly to make an estimate of what percent of these certs are to sites that are newly encrypted. Wow. And he found that it was over 90%. Wow. That's huge. Yeah. That was right, Jared. Excellent.

Pretty close to right. For the first time ever. It's in an order of magnitude. Pretty close. Pretty close. Cool. Well, we're hitting up against our first break. On the other side, we'll get into the nuts and bolts.

What's it take to be a certificate authority? What is Boulder? We'll talk about the Acme spec and CertBot right after this. Our friends at TopTile are longtime supporters of this show. If you've ever had to quickly scale your team, you know how hard it is. You have to go through all this hassle of writing job descriptions, adding them to your website, or maybe you have to hire somebody just to go out there and find the candidates for you. That's a lot of work, a ton of work that you don't have to do.

If you call my friends at TopTile, they do all the work for you to find the right candidates for your positions. Plus, because they have a very rigorous screening process to identify the best, you know you're only getting qualified candidates for your positions. Plus, because they have a very rigorous screening process to identify the best, you know you're only getting qualified candidates for your open positions. Head to toptal.com to learn more. That's T-O-P-T-A-L.com. Tell them Adam from the Change Law sent you. If you'd like a more personal introduction, email me, adam at changelog.com. And now back to the show. All right, we're back with Jacob. And Jacob, I'm assuming it's a lot of work to set up a certificate authority. Is that fair to say? And take us through it. What all does it require? And what work have you and everybody involved put into it? Yeah, so you're definitely right. It's a lot of work. And I'll go through a bunch of the stuff.

And I should clarify, you know, a lot of this was not done by right. It's a lot of work. And I'll go through a bunch of the stuff, and I should clarify, you know, a lot of this was not done by me. There's a whole team that spent a lot of work doing this. So the first question is, you know, sort of to be a certificate authority, you know, it doesn't mean much. You could become a certificate authority today. You can create a CA certificate on your own machine

and issue your own certs, but nobody would trust them, right? If you loaded that on a webpage, people would get a warning when they reached it. So you need your root CA certificate to be in trust stores. So that's in operating systems or browsers or things like Java. On Linux, there's usually a package called CA certificates that ships down a bunch of these trust anchors. And the problem is, even if you were to convince all of those, so we call them root programs, the programs that manage who gets to be a CA and who doesn't.

Even if you convince them all simultaneously to let you in today, there's all the older software that doesn't yet trust your root CA certificate. So the usual way around this is what's called cross-signing. who's already a CA to sign your root CA certificate or an intermediate and essentially vouch for you and say, this organization knows how to safely issue certificates and based on our existing presence in these root stores, certificates from them should be trusted. So that's step one, finding somebody to cross-sign for you. So Let's Encrypt was very lucky to find Identrust, who is willing to cross-sign for us.

And so even if you are running an operating system that was built years before Let's Encrypt existed, your browser will still load Let's Encrypt certificates successfully because of that cross-signature. Now, in order to get that cross-signature, you have to meet a certain number of requirements and pass a certain number of audits. So first, obviously, you have to generate your keys and your certs. You want to generate those keys in an HSM or hardware security module. And the whole point of a hardware security module is to store crypto keys in a way such that

even if somebody hacked you to bits, they still wouldn't be able to make off with the keys. They might be able to trick your HSM into signing something incorrectly, but they wouldn't be able to go off with the keys. They might be able to trick your HSM into signing something incorrectly, but they wouldn't be able to go make a copy of your private key. So once you've generated those certs safely and written down all the steps you took,

or rather, first you write down the steps you're going to take, then you follow them all in what's called a key ceremony. You do it all on video, and you have autotest check that you did it all. Really? It's all videotaped? Yep. It's like a wedding. Yeah. The key ceremony,

the videotape. That's true, Jared. I was telling our ops folks who performed the ceremony, you know, they should have robes and candles, but I don't think they did. Might have set off the fire suppression system in the data center. It seems that this process could get political, too, since there's such a.

Almost like a good old boys. I live here in Texas, so that term is pretty common around here. Good old boys or insiders or this protected ring or circle of trust, so to speak. Is it is that the case? Is that is it is, is it, is it purely technical, you know, like you had mentioned, you know, the different processes to do it? Is it, is it political? Is it technical? Well, so, you know, I think there's a couple of questions, you know, one is, you know, can you get a cross signature? And two is, you know, can you get into the root programs,

right? So, you know, it's definitely possible to start a CA and not get a cross signature, but just apply directly to the root programs and wait. But, you know, you're going to have to wait many years before your certificates can be used in practice. So usually it's a matter of, you know, buying an existing CA or paying a CA to cross sign for you. So I think definitely, you know, that's it's a barrier to starting up a new CA, especially if you want to be broadly trusted right away. And what about the capitalistic pressures from the other vendors, the other CAs who are selling certs? And now here comes one that's going to give them all away.

Surely there was some pushback or perhaps political machinations around the fact that here comes a free one, right? Yeah. We have seen some criticisms from other CAs and actually I think more frequently from certificate resellers, blog posts with lists of reasons why Let's Encrypt is bad. Some are inaccurate uh and sometimes they are you know legitimate disagreements about you know what a ca's role

is or you know what's valuable you know one of the big differences between let's encrypt and a lot of the larger commercial cas is we only do what's called domain validation. We don't do extended validation. So DV and EV for short. So domain validation is where to get a cert from Let's Encrypt, it's a domain validated cert. You just have to prove you control the domain for what, for which you want the cert. If you wanted an extended validation cert from another CA, you'd have to not only prove you control the domain in the cert, you'd also have to prove who you are, what sort of business entity you are, where you're based in, show them a lot of documents and so on. And the advantage of that one is browser. I mean, the browsers change how they do these things all the time, but at a certain point, at least,

they would make a big green thing in the address bar as opposed to a little yellow lock symbol. And so it's kind of the incentive for a site owner to go through the extended validation is to improve their – make them look even more trustworthy to browsers. But you guys only do the domain one because it's too much manpower to do the EV? Yeah, that's essentially it. Our whole thing is about being free your passport to this particular phone number. Yeah. And, you know, then we'll call your lawyers and they'll check at the Better Business Bureau

if you actually exist. That's that's not easy to use. That's not hard to automate that. Who needs that kind of level of assert, though? I mean, it's there for a reason. Who needs that? So I think opinions differ on the role of EV and how important it is. I think for folks who really believe in EV, I think the claim there is it helps with phishing, right? So one of the

problems with phishing is basically if you type your password into a site whose domain name doesn't exactly match the domain name you thought you were typing it into, it's game over. Somebody else has your password and they can access your accounts. But people are notoriously bad at comparing domain names, right? You get a dot out of place, you get an L that looks like an I, and it's really easy to spoof people. So with EVE, folks want to emphasize like a business name, you know, and kind of natural language. And, you know, I think that's, I think users tend to be relatively insensitive to additional UI bits like that. Are you talking about how in the URL bar the business name will show up? Yeah, exactly.

Okay, so GitHub uses that then. I mean, that's common for all of us. So they have an EV. Yep. I believe they do. Definitely, yeah. So I would say for most sites, I wouldn So they have an EV. Yep. I believe they do. Definitely. Yeah. So I would say, you know, for most sites,

I wouldn't really worry about EV. It doesn't affect the quality of the encryption that you have on your site. And it also doesn't impact the quality of the validation of the domain names in the cert. So it's kind of the same process in terms of validating the domain names and that ownership that's done for both EV and DV. Most users do not even know that there's a way they can identify a site as encrypted or not encrypted, let alone as the EV versus non-EV

encryption. And so the goal is let's get all the sites encrypted because people won't know one way or the other and so the EV I think it's a cloud thing I or it's like a it's kind of like you know pimp in your ride if you're big business forget who makes sense it looks cooler you know is there that's the only reason why I think it would be attractive to me because like like Jacob said the encryption is the same it just shows that shows that you went the extra mile to be audited more thoroughly than other sites did. So one of the interesting points there, Google Chrome has done a lot of really cool user research on usability of security indicators. And one of the results they've found, and I think other researchers have found, is that users don't tend to notice the absence of a security indicator, right? Like people don't notice that the green lock is absent. They do notice when something is present that says, oh, this is actually bad. Like

if the safe browsing page pops up and says, this site might be phishing or this might be malware, they notice that. And so the long-term goal for uh chrome and i believe also for firefox uh is something called http bad so instead of marking https as better the mark https as normal and http as unsafe i saw this article on Wired. I linked it up behind the scenes to Jared. This may be something you're familiar with, Jacob, but it was talking about this user experience of exactly what you're talking about, whether or not it's present and how they can best go about making general web users aware of whether the site they're on is encrypted or not or secure or not. Yeah. So both Chrome and Firefox recently rolled out kind of phase one of HTTP bad, which is

specifically if a page has a credit card or password entry form and it's HTTP, it'll get that new marking that says, hey, this is unsafe. There's times, too, I actually put in like a username and password into a site that I've done the check i know that the url is correct but for whatever reason they don't have that page secure and i'm like well i mean i look at the value of what the service actually is and it's like what do they what do they have of mine that i gotta worry about this so like what do you say to something like that where even when the little pop-up comes up and says hey this is not encrypted you're

putting your password across the wire. You sure you want to continue? And you're like, okay. You know, I think you're wise to think twice about entering passwords in sites like those. I think, you know, I'd recommend anyone who has need of a site that doesn't encrypt its passwords should definitely send email to their support staff and say, this is a really big security problem. You need to fix it. That's why let's encrypt is in place is to. There's no more excuses.

There's no more excuses. I mean, get your cert. Do it. Let's move on a little bit from the CA requirements because they are many and interesting for sure. But just to keep it moving along, let's talk about Boulder. Jacob, do you want to go into Boulder? Because it's your guys' implementation of the certificate authority. There may or may not be interesting things to talk about there.

Up to you. Yeah. So to talk about Boulder, I think first we have to talk about Acme. So we had a number of goals with Let's Encrypt. And one of the ones that gets the most attention is, of course, free certs and automated certs. But, you know, we also want to bring about interoperability. Right. So, you know, if Let's Encrypt becomes the CA that everybody uses, it's a single point of failure. Right. You know, if for whatever reason it fails, it's terrible news for the internet.

And instead, we want an ecosystem of software that's able to automatically issue certs and keep them up to date and can get them from a variety of CAs. And so that's the birth of the ACME protocol. It's a backronym for Automated Certificate Management Environment. It's also a nod to the Roadrunner cartoons. Oh, yeah. All of Wile E. Coyote's stuff was ACME, right? Yeah, exactly. All of his materials.

A backronym, that's when you started with ACME and said, what can we figure out that makes this work? Yep. Yeah. That's fun. So, you know, Acme was developed alongside Boulder, and it's still in standardization at IETF. You can join the Acme working group and give feedback. We're in working group last call right now, which means we think it's pretty much done and we're kind of sanding off the rough edges. So because Acme was a new protocol, of course, we needed to write new code to support it.

The most popular CA software out there in industry is called EJBCA, or Enterprise Java Beans Certificate Authority. But we decided to start from scratch, largely because we're implementing this new protocol. We wanted to use Go for both memory safety and performance reasons. Certainly, we didn't want to write in C. Java would also have been an option, but Go felt like the right choice at the time, and I'm glad we made that choice because it's been a real pleasure to work with.

So Boulder takes care of, okay, first, in the Acme protocol, you have to create an account. This can be created automatically by software running on your web server. Then you say, I want a certificate for these domains. And the server says, okay, great. Here's what you need to do to prove that you own or control those domains.

And there's three ways you can do it. They're called challenges in Acme. So one is the HTTP challenge. So it says here, basically, this is a file I want you to create, and I want you to put it at a certain path on your web server under slash dot well known so well known files are this you know newish spec to you know you know about robots.txt right right so everybody knows where robots.txt is but you know we don't want to keep adding files at the top level of web servers and everybody has to know they're special and has to prevent randos from uploading them. So now that place is slash.wellknown. So when protocols need to register a certain path,

it's under there. So that's the HTTP challenge. Is there anybody else doing that besides you guys? It sounds like you're becoming standardized, but is there anything else that goes in? Because this is the only occurrence for me personally of seeing anything get put in SlashDoc well-known. Yeah, there are a few. It's starting slowly, but definitely I've seen other new protocols establish that.

It's a handy thing. There was actually a talk I just watched from Brad Hill at Enigma about better account recovery. And part of the protocol he was developing involved putting a certain file at a well-known URL. Very cool. We should use that for our repos on GitHub instead of the file files, you know,

like gem file, Docker file, star file. We should just have a dot files or something and put everything in there. You know, it's your junk drawer. It sounds like a good idea. Definitely for web servers, though. Continue. We cut you off.

OK, so there's there's two others. There's the DNS challenge, which says, you know, put this special token value in a DNS text record under your domain. And then there's the TLS SNI challenge, which is take this token value, wrap it in a temporary cert, and we're going to attempt to connect over TLS to your site and ask for that. And we want you to echo back that value plus an additional validation value in the cert itself.

So once you've proven control of your domain or domains, Boulder will then sign a certificate and send it back to you. Boulder also incorporates some of the other important aspects of what a CA does. So OCSP is one of the important things we have to do. It stands for Online Certificate Status Protocol. And that's how you can find out

if a certificate has been revoked. So a CA will sign a blob of data that says, this certificate's still good or this certificate's been revoked. And you can request that via HTTP. So Boulder offers OCSP services and it also sends out expiration mail when your cert's about to expire. Very cool. So you have Acme, which is the specification, you have Boulder, which is,

I would think it was server-side, it's the certificate authority software. And then on the client side, the ones who are requesting certificates, you have CertBot. Yeah. Yeah, absolutely. I would say, of course, there's actually dozens of clients, which is one of the really cool things about Acme is that anybody can go. Yeah, because of the spec. So anybody can read the spec

and just implement the spec, and there you go. Yeah. But yeah, we're especially proud of CertBot. It was the first client because it was co-developed with Acme and Boulder. And I think probably the most ambitious. Most was the first client because it was co-developed with Acme and Boulder. And I think probably the most ambitious. Most of the clients tend to be focused on get your cert and

then it's hands off. CertBot really wants to eliminate some of that human cost of getting and installing a cert and in particular installing it. So with CertBot, it knows about the config formats for both Apache and Nginx. And so you can install CertBot and just say, give me the certs. I want the certs. And it will work with Apache and Nginx.

It'll actually reconfigure them to answer those challenges. And then once it's got the certs, it'll stick those certs in the Apache or the Nginx config files. So you can have just a single command, run it, and it works. That's pretty magical. That's awesome.

And including the updating, like the 90-day key updating? Yep. So certbot has this command renew. And so you just put certbot renew in your crontab. Or if you're using a packaged copy of certbot for your operating system, usually it'll install the crontab itself. And so that'll run actually twice a day and say, okay, is there anything that's expiring 30 days from now?

If so, get it again, renew it. Yeah, that's excellent. Certbot has a homepage at certbot.eff.org. Just have to mention, I have a question about this. I just wanted to say there's a, there's a little like wizard around installation and where you pick which software you're using, uh, which operating system you're on, and it will tell you the instructions for getting Serbot all set up specific to your web server and your Linux distribution, or if you you're on Unix or what have you. Very cool.

We talked about how successful this effort has been and is being. At the top of the show, I said, well, you make it free, that's a good start. Free plus easy is usually a recipe for a win. And so CertBot, as the first client, it started off not so easy, right? Like, of course, private beta, public beta, things are buggy, things are more difficult.

But it seems like you guys have really gone the extra mile, especially with like you hit this page and you know exactly how to get CertBot set up for your specific scenario. It seems like you guys are really trying to make this easy on people. Yeah, that's job one, right? Just make it super easy, allow more people to do it i could be just an idiot but one time i did try to use let's encrypt on a digital ocean server when we were running wordpress so this is probably about 12 months ago at least about a year ago and i was using a digital ocean tutorial to do it and i got stuck and i could not do it. So I don't know if I'm the idiot or if the tutorial was just dated, but I was lost. So this is helpful now, the cert bot.

Yeah, I think, you know, uh, there's definitely, there's always more work to be done on cert bot. Certainly there's a lot of places you can get stuck. And, uh, definitely there are a lot of tutorials from the early days of cert bot uh where like a lot of stuff has changed since then um and yeah they have a lot of page rank because people were really excited they linked to them a lot yeah but yeah i think can backfire sometimes yeah i think now probably the best tutorials are on the cert bot page but you know uh if you find one that you think is better send us a pull request and make

our documentation better too let's talk a little about uh the expiration and the renewing so um on the easy side sometimes it's easy to get set up but then you have this 90-day renewal and if you don't have software that automates everything because you want it to be automated but there are circumstances today for instance uh if you're on heroku if you're using s3 like static web hosting you still have to take the challenge upload the file to the well known directory and get that renewal and because it's every 90 days instead of perhaps once a year it's actually for certain of our sites that i manage has kind of increased my my overhead of how much time jared has to put into it,

which is fine for now because I'm assuming eventually those will be automated. But tell us about the 90 days. What's the reasoning behind such a short window? And then let's talk about automation a little bit. So, you know, one of the things that's really unique about Acme, of course, is that it allows automation and, in fact, encourages it. So it's more feasible to do something every 90 days. Then there's also this question of, is 90 days better, right? So one of the things that really motivated us in thinking about this was thinking back to 2014 when Heartbleed came out. So Heartbleed was this terrible vulnerability in OpenSSL that would just leak memory out to random users on the web.

And it was figured out that this could include private keys. And so that meant that most of the private keys on the internet were potentially compromised at that time. But it was kind of impossible to know which ones specifically were. And a lot of sites rotated their keys at that time, but a lot just didn't get the memo. And so for a pretty long amount of time, there were still keys available on the web that had been served using vulnerable versions of OpenSSL.

If you have a large ecosystem of Let's Encrypt clients that are reissuing every 60 days, you know, with that 30-day buffer, you know, the window of exposure to one of these kind of internet-wide bugs is a lot less, assuming they're rotating their keys at the same time they renew. So that's a big benefit to 90 day starts. And it also encourages automation. So, I think what we've seen from a lot of listing providers is that, they have wanted to help automate this for their customers

because it's really a big win. Whereas I think, if we had chosen the traditional year plus time window, you would see a lot of hosting providers saying, oh, we don't really care. You can go through this really tedious process once a year of uploading the file and running the client yourself. And that's really not the experience we want for people.

We want hosting providers to just turn it on for you. We want most people to never have to think about HTTPS. And I think we're for people. We want hosting providers to just turn it on for you. We want most people to never have to think about HTTPS. And I think we're getting there. I think more and more hosting software is incorporating that. I know WordPress has a Let's Encrypt plugin now. And also WordPress.com, the paid hosted version of WordPress, they're a Let's Encrypt sponsor. And they automatically issued for everyone who has a hosted domain of WordPress. They're a Let's Encrypt sponsor. And they automatically issued for everyone who has a hosted domain with them. So if you use their service, you have HTTPS.

You didn't even have to check a box. And that's, I think, the future of the web is it just works. Yeah. What about at the web server level? During the race, we were talking about Caddy, which is a Go web server that is integrated Let's Encrypt. We've had shows with Matt Holt on Caddy, both on the changelog and on GoTime. caddy as your web server but what about the big ones like can you get it integrated into nginx could you get it inside apache so there's you don't even have to have a separate client that integrates into them you just have it inside of your nginx yeah caddy is great i really think like that's how we want the let's encryption experience to work you know it's hdbs first

and that's considered a core part of the web browsing experience. We've definitely talked to folks from the Apache and Nginx projects, and they're supportive. So far, I think the approach has been, this is most appropriate as this external configurator for now. I think there's some obstacles in the Apache module system that make it relatively hard to give as smooth an experience as Cadi would. But I think in the long term,

I would like to think that's a possibility that the major web servers could just integrate Acme support and it just works. Well, we're coming up on our next break. So when we come back, let's talk about next challenges being faced, maybe even some public awareness in terms of like how the community can kind of step in and help this mission of encrypting the web. So we'll take this break.

We'll be right back. Rollbar puts errors in their place. Full stack error tracking for all applications in any language. And I talked to Brian Rude, the CEO and co-founder of Rollbar deeply about what Rollbar is, what problem it solves and why you should use it. Take a listen. How do you build software faster? Like how do you build better software faster? And there are like, there are tons and tons of aspects of that. Like in Ruby is like, you have a better language, you can have better frameworks that help you be more expressive and more productive. So the flip side of that is like after you've built something that works, or at least mostly

works, how do you like go about getting it from working to like in production and actually working? How do you cover the edge cases? How do you find things you missed? How do you iterate on it quickly? And that's kind of where what we're trying to do comes in. So we're trying to say after you've shipped your software, you're not done. You know, you saw there's still work to do.

And we want to help make that process of maintaining and polishing and keeping things running smoothly be really, really, really easy. So like developers spend roughly half their time debugging, right? So anything we can do to make that process better is going to have a huge impact. All right. That was Brian Ruse, CEO and co-founder of Rollbar. Sharing with you exactly why it fits, why it works for you. Head to rollbar.com slash changelog. You get the bootstrap plan for free for 90 days. That's basically 300,000 errors tracked totally for free. Give Robar a try today.

Again, head over to robar.com slash changelog. All right, and we're back with Jacob Hoffman Andrews going deep on this idea of securing the web, and what a fun mission. And sometimes, Jacob, to know where you're going, you kind of got to know where you came from. And in the breaks, I asked what I thought was a dumb question, and it turns out it's actually not that dumb. And it goes a little something like this. Why didn't we start with security first? Why didn't we start with, I know you don't like the term SSL because that's going away,

but why didn't we start with a secure internet versus an insecure internet? Cool. Well, actually, you've got two questions there. I actually have no problem with the term SSL. I tend to use TLS myself, but I think, you know, in the battle of convincing people to change their language, I don't think we're ever going to win that one. And I don't think we're ever going to win that one and i don't mind um let's encrypt the website we call it free tls slash ssl certificates so people can actually find us with the yeah you gotta do that for google searches right because

people are going to be searching for ssl even if that's not necessarily the protocol being used nowadays but you know i also try to say https more than either of those because that's like the thing that people who are not technical recognize more. It's the thing that shows up in their URL bar. If I'm being honest, the reason I don't say it is because I can't say it. It just never comes out right. HTTPS, it's always

jacked up. So I just speed through the two T's and the S is sort of like slurred. So that's why I'm happy about H2. You can shorten HTP2 to H2 and there's no version of that for this. So about the past, you know, I think so when the first web protocols and other Internet protocols were developed, you know, it was in this very trusting environment. It was like a relatively small group of people in academia, you know, the idea that somebody would be malicious was, you know, I think not unconsidered, but it was relatively distant

in most people's mind. But I think the, you know, much more interesting question is like, why didn't the point we're at today, why didn't we get here a lot earlier? Um, you know, SSL was developed in the 90s and later became TLS. And so why didn't it just take off like wildfire? Some of it was CPU costs, but I think a big portion of the blame lands at the feet of the US government and something we call the crypto wars. Now we have to call them the first crypto wars. So NSA and GCHQ and other spy agencies were kind of used to having a lock on cryptography. They were the cryptography experts. Ordinary people wouldn't have access to encryption.

They wouldn't even know the algorithms. But with computers really taking off, you had a lot more academics researching these questions of how do we make things safe? And they started discovering good crypto algorithms. Some had already been secretly discovered inside spy agencies. Some were new. And so the US government decided, these secrets, these abilities to keep secrets are too dangerous for us. We don't want people outside the US to be able to keep their stuff safe. So we're going to pass arms regulations. We're literally going to say cryptography is a munition and it's governed under ITAR, which is Arms Export Rules.

And so if you write crypto software, or even if you write about cryptography, you can't export that from the United States unless it's intentionally weakened to the point where we think the NSA can break it wow so if you downloaded netscape in the 90s you had this awful experience where it's like do you want to download the safe netscape with all the strong cryptography in it or do you want to download the unsafe netscape with the bad crypto algorithms in it and if you want to download the safe one you have to like really be sure you're in the u.s and you're allowed to do this and so this created obviously a ton of uncertainty and doubt in the software community of whether they were legally safe to implement crypto algorithms. And if they did, would that limit their market? So it really held back implementation for a long time. And it also meant that a lot of protocols like SSL built in these non-secure fallbacks, you know, so you could negotiate a really weak cipher suite with SSL.

Because you might have to if you were talking to a web server that wasn't allowed to use a strong crypto. Right. So this was actually a really big issue area for EFF in the 90s and to this day. And we took the position that code is speech, and it's actually a violation of the First Amendment to say you're not allowed to tell people outside the U.S. how to do cryptography. That's crazy. Yeah. I've never heard that story before. It's always the government's fault.

It's always the government's fault. It's always the government's fault, right? Not always, but in this case, for sure. Is this new news to you, Jared, or is this something you've heard of before? No, I've heard of this. I studied information assurance in college, so I got a little bit of the history of the whole InfoSec thing, cryptography. Read a couple books, so it's definitely – I haven't thought about it for a long time, but as Jacob was saying it, I was like, yeah, you know what? I remember all that. Yeah. So we fought a really important court case, Daniel J. Bernstein versus the US government. I say we, I wasn't at EFF at the time, but EFF fought this case and won some victories. It wasn't the perfect victory we would have wanted,

but now you do have the ability to export strong cryptography. And that really opened a world where people can develop open source software and include strong cryptography and not worry about violating these rules. And so that's been really important. And I think that's a big part of why now cryptography has been burgeoning. It's really a remarkable thing, having lived through that time, to realize that what we have in our web browsers is close to as good as it gets. These are some of the strongest algorithms out there. We believe they are strong enough to withstand government level decryption attempts, which is why when you look at, you know, leaks of how people are,

you know, attacking communications, it's often we're going to hack into somebody's computer. So we get the bits before they're encrypted. Yeah. So I mentioned, you know, OK, now we have to call them the first crypto wars. Yeah. So you tease that at the beginning. I was like, oh, that people can send stuff that we can't read is scary. And so we want to put restrictions on that and we want to, you know, limit cryptography in various ways. Most likely, you know, the second time around is not going to look like the first. It's probably not going to be a matter of arms restrictions. We don't know yet exactly what that attempt will look like but we're definitely ready to fight it and fight for people's right to keep their stuff safe yeah the bad part i guess i guess it depends on

your perspective of seeing it's a bad part when i say that so don't take that as a really bad thing but the point i'm trying to make is that it could be the good guys or it could be the bad guys right that want to keep their stuff safe, want to keep it hidden. So it's really hard to be. It's a really fine line of saying, I think everybody should have access, obviously, to security and secure Internet and secure transmissions of their messages. It just sucks whenever it's the bad people trying to hurt you. You know what I mean? Right.

So that's like a very tough position for any government to be in, not saying that they're right or wrong in their convictions or reasons for doing what they do. They have a very tough job to do to keep a nation safe, whether it's our nation or another. Yeah. The way I like to think of it is as the right to whisper, right? You know, you can always like lean over and whisper to the person next to you and have a relatively private conversation. You know, we would never dream of telling everybody, look, whenever you have a conversation, you need to have it through a megaphone just in case, you know, you might be a bad guy and the government wants to collect data of your wrongdoing. You know, I think there are plenty of avenues besides weakening cryptography that are available to the government in law enforcement activities.

Yeah, agreed. Let's talk about the future a little bit. So this ambitious goal, encrypting the entire web, recently saw an article, and I haven't found it again, but I believe it was within the last two weeks, that maybe we're about 50% there, 50% of sites are HTTPS. Jacob, do you have other data than that, or does that sound about fair? Yeah, that's correct.

So if you go to the Let's Encrypt website, we have a stats page, and there's two main graphs that we care the most about. One is how many certificates we've issued and for how many domains. That shows how many people have helped. But the other one that's arguably our most important metric is what percent of page loads happen over HTTPS. And we get that from Firefox telemetry data. Arguably, our most important metric is what percent of page loads happen over HTTPS. And we get that from Firefox telemetry data. So Firefox users can say, yes, I want to share some data with Firefox to help them make the product better. And one of those is how many of their page loads are over HTTPS. And so we've been tracking that for a while now, and it finally crossed 50%.

I think in February or January, it's now up around 51%, which is really heartening. Someone asked me the other day, how will you know when you're done? How will you know when you've succeeded? And easy answer, when that number reaches 100%. Yeah, exactly. May I live that long. Compete saturation. Yeah. That's why you do it but you know i think that

that rate of change has been going up and i like to think let's encrypt has had a big part in that well i mean to get to that kind of position though to even get to 100 or even 50 percent uh you know it takes a lot of effort and i guess a question behind the scenes here is like how does this happen financially like how do you fund making this possible is that there are donations non-profit you know collaborative partnerships with companies like mozilla or others that you mentioned in this call here how does this what powers let's encrypt what makes it happen yeah so let's encrypt is uh an independent non-profit uh the non-profit name is actually ISRG, Internet Security Research Group. But basically, ISRG and Let's Encrypt are synonymous.

The same. donors and from sponsors. So we started out with some pretty big sponsorships from large companies, Akamai and Cisco got us off the ground. Once we launched, a lot more companies came in and have been really supportive and have given us big donations. And so that's where a lot of our money has come from. But we also, we don't want to be solely funded by companies and web hosts. We love them. We love how they're broadening HTTPS adoption. But we really want to also be accountable to users. And so we do also solicit donations from individuals.

We ran a crowdfunding campaign or a fundraising campaign this winter, raised a whole bunch of money from individual contributions. And we're planning to keep doing that on an ongoing basis. Well, on that note, you can actually go to let'sencrypt.org slash donate. If you'd like to do that now, it's like a lot of options. $25, they're $2,500. Who's ever got that kind of money? Or a one-time custom, which could be a million bucks. Who knows?

Yeah. Especially if previously you were paying annual fees for certificates. You know, you could just take that money and now you're saving it every year because of Let's Encrypt. Give a portion or all of it to them instead. And it's a win-win. Yeah. I would say it's a win-win-win because there's a lot of winning going on here.

Who else is winning? Everybody. The users are winning as well. That's right so 100 let's talk about 100 real quick because i have like a little side topic with regards to like the small sites so uh you have a home page i have a home page you know they both have you know blogs and kind of programmery home pages right, right? Yours is encrypted and mine is not. Now I'm a, I'm an advocate for encryption. Uh, what's the upside? So like jerasanto.net, it's got some, you know, some tutorials, you know, maybe an about section. There's nothing really there. We're not taking logins. There's no private and there's nothing, right? It's just my homepage. Yeah. What's the,

what's the upside for me to encrypt that um you know i think uh there's there's a number of reasons in terms of upside you know access to h2 we talked about earlier in the show can make your site faster and you need encryption for that if you want to use advanced features like uh or powerful features they call them. For instance, full screen in HTML or geolocation data, browsers are starting to require HTTPS for that. And there's also an aspect of reading privacy. If you want people to be able to visit your site and click around and not necessarily have spies know what they're reading, you know, HTTPS helps protect that. But I think, you know, the larger idea here is, you know, kind of herd immunity, right? I think, you know, there may be a lot of smaller sites for which it may not be worth the effort right now to implement HTTPS,

but for the health of the internet ecosystem, we'd like, you know, the vast majority of websites to implement HTTPS because that allows us to treat HTTP as less secure. But of course, in order to get there, we have to lower all the costs of HTTPS practically to zero, right? We have to make it so when you're weighing cost and benefit, you're like, eh I might as well check the box that says HTTPS. So we want to make it zero effort. Yeah, I think that's exactly on point. Just thinking about my specific website. It's a dream host thing, shared hosting.

And I have SSH access, so I could go in there and get a free certificate with CertBot and get know get it all set up with I believe it's Apache but I don't even know or care I might do that I haven't yet so apparently like those goals you know those upsides aren't strong enough for me to do that if I got to renew it every 90 days myself I'm definitely not doing it that being said if I could just go into my DreamHost control panel and click the box, now what's, you know, like you said, the cost is zero to me. And so I really feel like once you guys get a lot of these middlemen, the hosting providers, the Rokus of the world to integrate the Acme protocol into their services, it's going to really help that 50% get to a hundred. Yeah. Well, I happen to know that a DreamHost actually does have an Acme integration. Sweet.

They do have a box you can check. I don't know if that's, you know, it sounds like you have a VPS with Shell login, so it might not work on that. It might only be for the full host. I guess it's not shared hosting. I have a VPS. So that being said, that's a great step. I mean, even just, I'll go check.

I'll see if I can just turn it on. Cool. Sounds good. Might just be a little toggle there, man. Oh, so easy. I mean, that's the way you've got to make it easy for, you know, mom and pop web host or, you know, website owners that run a small business and all they're doing. Well, my site's just a brochure, so there's nothing special happening on there. It's just, you know, how do you get to my location and who are we and can you see

pictures of our shop or whatever like they're not commerce so and they don't even know what the web is really they're just like we have this thing on the thing and it's the cloud or whatever and you can go there and find out about us and it you got to make it so much easier for those people so there's one other feature topic i'd like to talk about if you guys feel like we have time. Do it, man. Open it up. So one of the things I see is the most exciting thing in the future of HTTPS is called CT, certificate transparency. One of the things we've seen in the past is, you know, sometimes certificate authorities screw up. You know, they usually assert that they shouldn't have or they get hacked. And how do we know about that? The CA ecosystem is a little

funny because CAs can issue a certificate independently. They don't have to tell anybody. And it's trusted by billions of browsers all around the world. And it's possible that if you were issuing a malicious certificate, a fake cert for, say, addons.mozilla.org, you could serve that to only a small set of users such that no one else in the world would ever see it. And for those users, you'd be able to man in the middle what should be a secure connection and serve them some malicious software or steal their passwords.

And so one of the big examples of this, this actually happened. Somebody issued some malicious search for add-ons.mozilla.org, mail.google.com, I think back in 2010 or 2011. And there have been any number of misissuances since then. And they're caught in various ways, but kind of by chance and happenstance, or in some cases because Chrome has telemetry that will notice certificates that it doesn't expect for certain sites. But ideally, we want to generalize that, right? And when you issue a cert that's going to be publicly trusted you should have to tell the world about it so certificate transparency is based on the scheme of logs so when a ca logs a certificate when a ca issues a certificate, they can log it to a number of CT logs.

And those logs are kept honest by essentially a blockchain. They use a Merkle tree internally to give themselves this append-only property so that if a log is malicious and they try to pretend that they received a cert but not actually show it to anybody, they, in theory, would be detected. So this is relatively new. I mean, it's been in the works for a few years now. There are CT logs out there, and logging to them is voluntary for most CAs. For CAs that want to be trusted in Chrome for EV,

it's required. And for a few other CAs that have had recent mis-issuances, it's also required by Chrome. As of October 2017, Chrome is expanding that requirement. So if you want to issue a certificate that's trusted by Chrome, you will have to log it to multiple CT logs and include proof of that in the certificate itself.

Or not necessarily in the certificate itself. There are a couple of ways to deliver it, but you'll have to be able to prove that that certificate was logged. So I assume this will be coming to Boulder real soon, if not already? Yeah, so Let's Encrypt really believes in transparency. It's one of our core values.

And so from day one, we were voluntarily logging to multiple CT logs. One thing we don't yet do is we don't embed the proofs in the certs so that a browser can verify, yes, the cert was definitely logged. And so embedding that proof is something we're planning to do. Very cool. Man, blockchain, FTW, recently people have found so many different uses for blockchains. It's very, I mean, this is a great, great example of that. Great time for it.

I mean, we have a much bigger need for it now. And now that the care and technology around blockchain is getting more and more mature, it's a great time for it. And I'm going to super nerd out here for a minute and say it's not exactly a blockchain. It's like a blockchain. Like a blockchain. Blockchain-esque. We're running short on time, but do a real quick diff for us between blockchain and like a blockchain. Yeah. So I think fundamentally both use Merkle trees underneath. That's the kind of construct that enables them both.

Logs are kind of single operator. So they're just maintaining a list of things for themselves that are append only, whereas blockchains are usually distributed consensus. Oh, it's not distributed. Right. So you have multiple logs, but each log is a world unto itself. Oh, I thought that all these CAs would be sharing one common. It would be so much cooler as a blockchain. Yeah, there's some tricky trade-offs there around availability.

Yeah, for sure, for sure. But as long as a log gets you where you need to go, then. That's right. That's a weird. I never thought I'd say as long as a log gets you where we talked about that. We talked about how you're moving funding forward in terms of support to Let's Encrypt. You have corporations supporting you. You have an option for the caring public to donate to support you. But for the developers out there listening to this show, they want to step in.

They want to help. They care enough. They want to step in and make an impact. Aside from simply donating, what other avenues do you have available to those listening to the show that says, hey, I love this mission. What can I do to do to make this go further? Yeah. So, you know, first off, you know, we're an entirely open source project. You know, that's part of the reason we're on this show. Boulder, CertBot, you know, a lot of the stuff, all the stuff we rely on is available on GitHub.

We have help-wanted tags or good volunteer tasks tags on both repositories. So jump in. If you've used CertBot and found it lacking or difficult, file documentation bugs or bugs on how we can make it better, that's always super welcome. And if you maintain a website and it's not yet HTTPS, go now and do that.

And if you use a website that's not yet HTTPS, email them and say, why aren't you HTTPS yet? I really want to see this. And if you use a hosting provider that doesn't offer easy automatic HTTPS yet? I really want to see this. And if you use a hosting provider that doesn't offer easy automatic HTTPS, tell them to do it. And especially if you work for a hosting provider,

enable HTTPS, make it an automatic thing that's on for everybody by default, and we'll get there. So advocacy is a huge part then. Exactly, yeah. Just sharing the message. Yeah, I mean, Let's Encrypt has played a big role Um, and we'll get there. So advocacy is a huge part then. Exactly. Just sharing the message. Yeah.

I mean, the, you know, Let's Encrypt has played a big role in getting us this far, but way more than that, it's just all the hundreds of people who have advocated for HTTPS over the years. Well said. All right, Jacob. Well, let's close there, man. Thank you so much for coming on the show, man. It's been a blast.

I really appreciate all that you do. Don't stop. Keep doing it, man. Thanks very much. It's been great being on the show, man. It's been a blast. I really appreciate all that you do. Don't stop. Keep doing it, man. Thanks very much. It's been great being on the show. All right, that wraps up this episode of The Change Log. Join the community

and Slack with us in real time at the changelog.com slash community. Follow us on Twitter. We're at changelog. Special thanks to our sponsors, Linode, TopTow, and Rollbar. Also, thanks to Fastly, our bandwidth partner at the facet.com

to learn more also huge thanks to breakmaster cylinder for the awesome beats we'll see you again next week thanks for listening Bye.