Bankless - Did Ethereum Break? with Preston Van Loon & Terence Tsao

Episode Date: May 18, 2023

Did Ethereum break or what actually happened?  We brought on Preston Van Loon & Terence Tsao, two Ethereum Protocol Developers, to help us answer that question.   ------ 📣 CONSENSYS | DILIGENCE FUZZING https://bankless.cc/diligence-fuzzing-pod  ------ 🚀 Airdrop Alpha is waiting for you on Bankless.com  https://bankless.cc/Alpha   ------ BANKLESS SPONSOR TOOLS:  🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://k.xyz/bankless-pod-q2   🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask  ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum  👾STADER LABS | ETHX LIQUID STAKING https://bankless.cc/Stader  🗣️TOKU | CRYPTO EMPLOYMENT SOLUTION https://bankless.cc/Toku  🎮IMMUTABLE | GAMING ECOSYSTEM https://bankless.cc/Immutable  🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle  ------ Timestamps: 0:00 Intro 7:53 Did Ethereum Break? 8:22 What Happened?  14:50 Slot Visualization & Connectivity  20:30 Did Users Notice?  23:57 Why Was This Worrisome?  26:19 Preston & Terence's Emotional Response 35:55 Ethereum Resilience  39:30 Was This Good For Ethereum? 41:20 Solana vs. Ethereum Liveness  44:40 Closing & Disclaimers ------ Resources: Is Ethereum Breaking? https://twitter.com/BanklessHQ/status/1657122512516575232  Preston Van Loon https://twitter.com/preston_vanloon  Terence Tsao https://twitter.com/terencechain  ----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures 

in the night with finality on the Ethereum blockchain last week. And we want to go all, we want to go over that today with some experts who can help us answer the question as to whether Ethereum broke or not. What does it mean to break? What happened? What does non-finality mean? I think we'll talk about that. And what are the risk conditions that we are exposed to as a result of this. How did Ethereum handle this event is really the central question in my mind? David, who do we have on today? We have Preston Van Loon and Terran Stow of the co-founders of Prismatic Labs now in Arbitrum, now part of OffChane Labs. And we brought Preston and other members of Prismatic onto the show before to understand the world of clients, because it is a complicated world, but not so complicated that we won't be able to understand it here on the show. So some very important questions with some very smart people who have been with the Ethereum proof of stake effort since it was once upon a time called Ethereum 2.

But before we get into the show, we've got to talk to our friends and sponsors at Consensus because we've got to talk about this thing called diligence fuzzing. It's a new code auditing tool. Just fuzz it. Yeah, with a sick new name called diligence fuzzing. So what's diligence fuzzing for smart contracts, solidity devs, auditors, enterprises, engaging with Web3 that need to ensure that their smart contract is secure before proceeding to main net.

Diligence fudzing, it's an audit-grade security tool that serves as an automated way to find code vulnerable abilities for people without hosting their own infrastructure or writing complete complex test cases. So there are many ways to do automated testing, but fuzzing is a way, brand new way, with quite a lot of sophistication, according to consensus, that you can go and access right now. So this is an extra layer of protection for your smart contracts in addition to and above and beyond typical code audits. Humans are messy and can miss things, but diligence fuzzing is diligence by robots.

Fuzzy robots, apparently. There's a link to the show notes to get access to this. David, can you imagine releasing a smart contract and not fuzzing it? I just can't even imagine. I prefer fuzzy smart contracts. It's fuzz it. All right. So there was an event that happened last week,

and this was an event that happened after the roll-up, and I'm wondering if we could kind of tee this up before we bring Preston and Terrence on. So what happened? I think maybe the first sort of, mini event was on Thursday and then again Friday, if I'm recalling correctly, all my weeks are blurred, David. But give us the recap before we get in here. Yeah, it happened twice. And so Ethereum failed to finalize, which is an interesting sentence. What does that mean?

Blocks were still propagating, transactions were still going through, but finalization was not happening. And just at a high level, finalization is just something that happens when enough clients, enough validators have given the thumbs up about a certain epoch, epaques, which are, I believe Preston's going to come in here and give me the whip. I believe 64 blocks, but it's a collection of blocks. And then if enough validators give the thumbs up, then that epoch finalizes and that is permanently embedded into the blockchain. When a epoch is finalized, it requires slashing conditions to go back and unwind those blocks. And so if a block isn't finalize, it's not as permanently embedded into the blockchain as it otherwise could be.

Except with this event, which happened twice, Ethereum failed to finalize because blocks were being missed. Validators were not doing their jobs. I am only able to explain it to some degree at a high level. And so we're going to go talk to Preston and Terrence to get a deeper dive into what this world of finalization means. Should do a disclosure before we get in. Of course, now Prismatic is owned by OffChane Labs, which is a sponsor of the bankless podcast, very supportive of what they're doing in layer two and client development work. Also, David and I are notorious holders of the asset called Eiff. So you should know that podcast that we are bullish on Ethereum generally and are interested in finding out what happened

today. So guys, we're going to get right to the episode with Preston and Terrence to talk about whether Ethereum broke or not. But before we do, we want to thank the sponsors that made this episode possible, including our number one recommend exchange, Cracken. Go check it out. Cracken Pro has easily become the best crypto trading platform in the industry. The place I use to check the charts and the crypto prices, even when I'm not looking to place a trade. On Cracken Pro, you'll have access to advanced charting tools, real-time market data, and lightning fast trade execution, all inside their spiffy new modular interface. Cracken's new customizable modular layout lets you tailor your trading experience to suit your

needs. Pick and choose your favorite modules and place them anywhere you want in your screen. With Cracken Pro, you have that power. Whether you are a seasoned pro or just starting out, join thousands of traders who trust Cracken Pro for their crypto trading needs. Visit pro.crakhan.com to get started today. Mantle is a brand new high-performance Ethereum Layer 2 network, built differently from the other Layer 2 you may be familiar with. Mantle is a modular layer 2 built on the OP stack but uses eigenlayer's data availability solution instead of the expensive Ethereum layer 1. Not only does this reduce Mantle's gas fees by 80% compared to other layer 2s, but it also reduces gas fee volatility. Mantle has a decentralized sequencer set,

eliminating the risk of downtime and censorship on the network. And because Mantle implements multi-party computation nodes, layer 1 settlement execution is shortened from 7 days to as low as just one or 2. Mantle is the first layer 2 built by a Dow and is backed by one of the biggest Dow treasuries in the world. Bit Dow. Mantle already has subcommunities from around web 3, 3rd. onboarded to help the growth of Mantle, like Game 7 for Web3 gaming, or EduDow for the world of D-Cai, and ByBit for TVL, liquidity, and onramps.

Check out Mantle at mantle.xyZ, and follow them on Twitter at ZeroX Mantle. Immutable is at the forefront of Web3 gaming, on a mission to bring digital ownership to every player, offering the world's best games and game development platform. Immutable lets game builders and players focus on great gaming experiences. So, build your next Web3 game on easy mode, with Immutable's leading full-stack Web3 gaming platform.

It's built-in Ux features like the Immutable Passport are designed for games to scale to the next billion players coming to Web3. With Immutable, players can sign up with an email, pay with a credit card, and experience a frictionless purchase flow inside of games. So, discover your next favorite game and explore a network of 150 games building on Immutable, including such titles as Gauzin Chains, guilds of Guardians, Illuvium, Embersword, and Metalcore. So join Web3's largest ecosystem of games and players.

Build, play, and connect at immutable.com. Bankless Nation, Preston Van Loon, and Terrence Tau are the co-founders of Prismatic Labs, one of the organizations that maintains the PRISM Ethereum Consensus Client. Prismatic Labs was one of the earliest client teams that was formed to start building what was then called Ethereum 2, which we now just call Ethereum. And they've been very key to Ethereum development every step of the way. Preston, Terrence, welcome to Bankless. Hey, happy to be back.

Thanks for having me. Yeah, okay. I'm going to look forward to being educated about some of the nuances of finality and security. But first, did Ethereum break, Preston? No, it did not break, right? Oh, okay. We did see Ethereum actually work. We have a time where Ethereum has difficulty finalizing, and we got to see it actually work in action, right?

So this is not like a problem with Ethereum that specification did not break. It actually worked and it worked in an unfavorable condition and we got to see that happen in Mainnet for the first time. So how would you summarize what actually happened? And so I did my best to explain it in the intro. How would you describe to the layman? Explain it like I'm five, what actually happened here. And how many blocks are there in an epoch, Preston? Most importantly.

There are 32 blocks in an epoch, not 64. At least you got a power of two. so that's nice. What happened was multiple clients, Prism specifically, Antecu, had this problem where if it received these sort of old attestations, attestations being what the validators are producing to say, this is what I think the head of the chain is, they received these old attestations,

which caused Prism in particular to replay a bunch of states, right? It had us to go back in time to evaluate, whether or not this attestation was valid or how to count this vote, it ended up being a pretty expensive operation that somehow bypassed or filled the caches that we had. So we weren't able to, you know, keep that work in the client. It had to re-comput it every time. So essentially it was an accidental, I guess, like kind of denial of service where the clients were so busy doing these computations. they weren't able to respond.

This is the beacon node part of the client. The beacon node was not able to respond to what the validators were asking to do. The validators said, here's the block I want to produce. Here's the attestation I want to produce. These weren't getting out the door because the client was so bogged down with trying to process these valid but untimely attestations. That's the gist of it. This happened to two clients, Preston? Yeah, well, two clients have released.

updates specifically to handle this scenario. That's Prism and Teku. Lighthouse already does it really well. So a lot of us have sort of taken inspiration from their model of how they handle these old estate stations. In short, they basically dropped them to say this attestation is not going to be very valuable to how we're going to update the fork choice. It's not worth recomputing all this all this effort. So we sort of copied their model of just excluding that attestation from consideration, not doing all the work. And so why did this happen so suddenly? Well, we're still trying to track down the exact, you know, trigger to the event. But what we think had happened is that some clients, namely Lighthouse, when they lose a connection to the execution layer, the layer that

the client that handles transactions and the state of Ethereum as we think of from, you know, the application layer that being like Go Ethereum or whatever you use for your execution client, when we lose access to that, some clients will behave, well, they'll just withhold these attestations and then later produce those, just sort of later release them. So what Prism does is they just don't, we just don't attestering. at time because we can't really come up with, we can't advance our node. So we're not, it just wouldn't really be attesting there. Other clients will produce this sort of old, valid, but, you know, not valuable attestations. And that was what actually triggered the event.

It's not a common path that happens. But what we have learned is that it wasn't something that was nefarious. It was just some sort of unexpected behavior, valid behavior. but unexpected that Prism and perhaps Techu didn't consider or handle very well. I think a good like a question here is like why it didn't happen on a test net, right? Like why it didn't happen before. I think there are several reasons to it. The first reason is that there's a lot of validators now. There is 600,000 validators and validator matters.

Like the number of validator matters, the more valid you have, the more hashing that you have do on a state and that's slower. And our test net only has 400,000 validators. So even the test net is lower than the main net, which is obviously a flaw here. That's something that we're taking back and we're going to revisit how do we do this type of test net. The second thing is there is a lot more deposit now, right? Because now we enable withdraw and then the deposit key with and then people are kind of getting their top of Eve, they're going to re-deposit it. And then with so many deposit, there's a lot more hashing that you have to do. They also increase the latency. So those are the two big factor. And once we reach those two bowling points and we kind of just

create this death, that's spiral. Okay. So to recap what I'm hearing, both there were our new conditions about the state of the beacon chain, a higher number of validators are increasing the resource requirements of nodes. And then also from what, from what Preston was saying, Prism and Lighthouse shared similar properties because Prism just borrowed from Lighthouse's code base to do this one part of the client in Prism. And because that part was shared, this showed up this problem that was triggered by the increased computational requirements and bandwidth or networking requirements of the more and more validators coming in. That was triggered. But then because of the shared property of Lighthouse and Prism that showed up in two clients, which is how it actually showed up as a non-finalization event. in Ethereum. Is that recap accurate?

Sort of. So we, we, the fix was to replicate what Lighthouse handles this behavior. It was Prism and Teku that didn't handle this scenario very well. So when the outage happened, right, there were something like 60% or more blocks were missing. So that's an indicator that more than one client is having issues. And we believe that to be Prism and Tekew because both clients have really sort of emergency updates or critical updates. or critical updates since the incident on Friday and Thursday. Okay. I have a question in general about what this looks like for Ethereum to not finalize.

So here is a beacon chain visualizer we've shown before on bank lists. I think specifically when the hard fork upgrades happen, you sort of look for these things to go from red to green, right? These are proposed blocks that we're looking at. So when the blockchain doesn't finalize, what does when Ethereum didn't finalize on, I guess, it was Thursday once for a period of time, and I don't know how long that was. And then also Friday, first of all, what does this chart look like? Well, the main indicator is that the green bar you see at the bottom of each epoch,

justifying or finalize, that's not going to be greater than two, You needed to be greater than two thirds to justify or finalize. In the case of the incident last week, we were missing a ton of blocks. So you can see a couple of red spaces here in the blocks. Those are missing blocks. This is a proposer, whoever they may be, failed to produce a block at that slot. And because there were so few blocks, you know, more than half of them were missing, it wasn't possible to pack in all of the attestations.

and also very likely the same nodes that we're suffering from not being able to produce the blocks. We're not able to produce the attestations either. So two parts to it. You have to have enough blocks to include attestations, and you have to have enough attestations that agree with each other, that being more than two-thirds, to finalize and just to finalize an epoch. And how long were these periods of time where this sort of degraded finalization service was happening?

And the first time on Thursday, I think it was for four or maybe five epochs that we had a gap in finality. Usually we're seeing every epoch is finalizing, you know, two epochs later. So in this screenshot, you can see epoch 63 is finalized. When the current epoch 65 finishes, we're going to see the 64 will finalize. And we just have this sort of continuing, rotating, you know, the epoch two epochs that goes finalizing. Well, when it wasn't finalizing on Thursday, it took five epochs to get back on track. And then on Friday, it took even longer. I don't recall the specific number.

Maybe Terence knows. But what was significant about Friday is when it goes longer than five epochs, we start to incur an additional penalty that everyone gets penalized. and it's called a liveless leak, I think. Terrence, do you want to explain in detail? Sure. So we have this in-activity dig. So this in-activity dig is essentially one-made Ethereum World War III resilience, right? So imagine we cut the internet in half.

And then there's this, say, Asia and then USA, and they cannot talk to each other, right? In that case, we want either side of a validator to dig their balance faster, to lose their balance faster, so we can finalize faster on each side. So you will begin to lose balance faster if we don't finalize for more than a 4-to-5 epochs. And then eventually, after 17 days, you will lost probably around 16 if, then you will get ejected from the pool. So this is the first time this in-activity, in-a-tivity big happen. So I did some math earlier. So we ended up burning about 28 Eve.

And I don't know if this is something that we should be proud of just everyone loves to spend eat, but this is slightly different. So on average, every validator, if today, if you are offline at that time, you lost about $1. So, yeah. Okay. Just to double down on that mechanism. And the idea here is Ethereum's philosophy is to withstand World War III. So if we imagine the internet as a gigantic cell and that cell gets cut in half,

Ethereum should find a way to bridge the gap between the division of the internet. And the way here is that if there can't be consensus between the nodes across the world, the assumption is that there's going to be at least a few nodes that can span that gap. And for all the nodes that cannot span that gap, they will get inactivity leaked and down until they are kicked out of the validator set. And then the few remaining nodes that can span east and west or whatever division of the internet there is, they will be the ones left standing who are not getting inactivity linked because they are the ones still propagating the blocks. And that's the mechanism that entered.

Is that correct? Yep. That's right. Well, at least we got to test the inactivity leak, right, guys? There's a silver lining here. I don't think we've ever tested that in the wild. And so this inactivity leak came to play for those who were staking with, was it? Is it just Prism and Techu and Lighthouse kind of carried on?

Was it just sort of a subset of client users? I think it's hard to tell. I think every client had some issue at that time because of just high CPU usage. So they're sending their attestations, but their attestations may not be received, may not be processed. So I think like every single client, unless today you run your client on a very, very powerful machine that you are, likely break even or you'll lose a few cents. I got it. Okay.

So what happened? That's the view of sort of from the client in high-level blockchain perspective. And we've seen that. What about the average user? What happened during this time? Were they able to get transactions through? Or what happens to users using the network? Did they even notice?

So I guess put it this way, right? If today it wasn't for a pitching chain explorer, if it wasn't for that site, will we ever find out the issue? Maybe yes, maybe not. But I don't think we'll find out the issue at that time, right? We'll probably find out the issue like a few hours later when someone was using this finality notion before, right? So with that, it basically, we can say that, yeah, it probably doesn't affect most people. If today you're doing a swap on uniswap, you're buying some NFTs, those still goes on. But if today, say today, you're actually using finality, which is a pretty rare set of application, I will say maybe

there too, maybe like it's changed as a big amount of withdraw, then that would be affected. But that's a kind of a small set of the use case. I mean, was it acceptable? No, I don't think it's acceptable for Ethereum to be this global internet of money, which we cannot have this, right? But I'm really happy that I'm really happy that lifeness still goes on because that's probably like, I would say close to 95% of the use case all there. The transactions were going through completely. It wasn't as if there was a pause in sort of user transactions. It was just this one finality piece that some sort of you call them settlement applications might use. It might be a layer two or it's an exchange, maybe doing a big batch process with a lot of money.

They want to wait until full finality is achieved before they settle it. Those are the kind of apps that were maybe affected, even though that's even hard to see because this is so brief, but users weren't affected at all, like the regular users and liveliness was preserved through this event. Preston looked like you were going to add something to this. Yeah, I wanted to add that the real impact for the end users was that there was a diminishing amount of block space, right? So when we're packing transactions into blocks, what drives the gas price of Ethereum is the demand for block space and the availability. So I did notice that, at least on Thursday, gas prices were slightly higher than the daily average. It was something like

120 way, at least on the epoch that I checked, which, you know, in terms of the scale of Ethereum gas pricing is kind of a negligible increase. Maybe it was. was 100 to epoch before and out's 120. We've seen a lot of, you know, DAPs push the gas price much higher for different reasons. So it didn't seem like anyone really would have noticed, right? It wasn't astronomically high. It's just slightly higher than average during that time due to less available block space for transactions. But everything was still going through pretty in a timely fashion. You may have noticed your your transaction took up to a minute, but where it normally takes, you know, 12 seconds,

but that was really, really it. You didn't really feel that it was down or broken in any way. So in that case, what's the big deal here? Like, why do we care about finality? What would, would, what would have been, what could have been a worse condition? What, what, could we play out a scenario where maybe this happens again and then something actually bad happens? Like, what's the big worry here? Well, the concept of finality in proof of stake is one of the main features, right? It means that we have finality in the blockchain and that we can be assured that that point in time is final and can never be rewritten or we can have go back in time, build on a prior block and sort of exclude this history that's happened. As Terrence was saying, that's very important for some transactions.

So if you're doing maybe a very large purchase, let's just say, you know, we're at a point where you can purchase property in the real world. You say, I want to close on this property. I'm going to send you my ETH and then or whatever it might be my tokens. And we're not going to consider it done until there's finality. Right. So there is that aspect where we wanted to be final cannot be rewritten. It's also part of the entire security of Ethereum, right? If Ethereum aims to be the global settlement layer, you need to have this consistent finality in a timely fashion.

And it can also affect, you know, layer twos or other networks that are leveraging Ethereum security. I was imagining a scenario or what if this played out for a very long time, right? We have optimistic roll-ups out there with a seven-day, I think a seven-day window for the challenge period. So you can imagine that if this incident went longer than seven days, what does that mean for the challenge period? Even though it has resolved, it's not entirely final because it's not final until it's finalized. And I thought about that as being sort of like the medium term worst case. I mean, the obvious worst case, this goes on longer and longer until we have so much inactivity leak that a lot of validators are rejected. Eventually, Ethereum recovers.

L2s would recover and those challenges would eventually be finalized. but you could see that would be a major disruption if it went on longer than seven days. Can you put us into the headspace of Preston when this was happening? Like, were you freaking out? Where you're like, oh, that's a curiosity. And what just like sometimes, yeah, we can talk about like the technical details of finality. But what about the emotional response? I can give you guys the response of people on Twitter, which was mass panic, freak out.

Oh, my God. Ethereum is broken. And that was the response on Twitter. But yeah. Yeah. So this is, when I happened on Thursday, it was around like 4 p.m. my time. And I had been started my day very early. And I was like, I'm going to go, I'm going to go take a nap.

Like I'm kind of tired, right? And then I just check right before I go to make sure nobody needs me for anything. And I see this message I never want to see again from Nishant. He's on our team, Nishant Das. He sent me these a couple times. he says, Mainnet stopped finalizing. And I said,

oh my God, there's no way I'm going to be able to sleep for days. Those three words, those three words. Maynett stopped finalizing. And this is how I found out about it the first time.

So this isn't the first time Mainnet has had a finality delay or finality incident. It happened a few years ago where Prism was unable to produce blocks and Prism was a super majority thanks to client diversity that were not,

have that risk anymore. But the same message he sent me, he said, may not stop finalizing. And at that time, it took us, I think, three days to figure it out, to figure out what the issue was. And we were sleeping in rotating shifts, like maybe a few hours at a time. So when I thought, okay, I'm going to go take a nap. And then he said those three words, I said, great, I'm just never going to sleep again. So emotionally, that was challenging. Well, I think this does a very good job illustrating why client teams are so important and why client team maintainers are heroes that should wear capes. So Preston, Terence, thank you for always wearing a cape, even though you don't actually legitimately, literally wear a cape. There's a lot more that I want to ask about.

But first, we have to talk about some of these fantastic sponsors that make this show possible, especially Madamask. If we have used jargon in this episode, that is confusing to you, I bet you, MetaMas. Learn has an explainer for you on your product. There is a link in the show notes. Here we go. Learning about crypto is hard. Until now, introducing Metamask Learn, an open educational platform about crypto, Web3, self-custody, wallet management, and all the other topics needed to onboard people into this crazy world of crypto. Metamask Learn is an interactive platform with each lesson offering a simulation for the task at hand, giving you actual practical experience for navigating Web3. The purpose of MetaMask Learn is to teach people the basics of self-finding

custody and wallet security in a safe environment. And while Metamask Learn always takes the time to define Web3 specific vocabulary, it is still a jargon-free experience for the Crypto-Curious user. Friendly, not scary. Metamask Learn is available in 10 languages with more to be added soon, and it's meant to cater to a global Web3 audience. So, are you tired of having to explain crypto concepts to your friends? Go to learn.menomask.io and add Metamask learn to your guides to get onboarded into the world of Web3. Hiring people, worldwide, paying them in crypto, providing them access to benefits, it all so complex. But it doesn't have to be.

Complying with labor laws, payroll rules, tax obligations, and crypto regulations in every country that you employ someone is difficult, time consuming, manual, and costly. And it's drawing more and more attention from regulators and governments. But there is good news. Toku is here. Toku is the first employment and compensation platform for the crypto industry that makes this easy. Toku helps you hire employees or contractors and pay them in fiat or. crypto legally,

compliantly, and with all the taxes handled, in over 100 different jurisdictions. So whether you're an early stage company with just a team of two, or you're an enterprise with 200, Toku has a solution that meets your needs. Toku is already working with the leading companies in the space. Protocol Labs, Hedera, Gitcoin, and many more. So transform your employment and token payroll operations with Toku.

You can reach out to Toku at Toku.com slash bankless or click the link in the show notes. Introducing ETHX from Stater. ETHX is a liquid staking token designed to maximize rewards, all while securing Ethereum. With Stater, you can run an Ethereum node with just 4Eath, an 85% lower capital requirement versus the 32 ETH required for solo stake. With Stater's 4Eth nodes, you can get a 35% average higher yield, since you charge fees to those who use your node to stake their ETH. By running a node with Stater, the ETHX staking derivative token can get minted on your validators

and can flow into the world of D5, which Stater is actively building integration and partnerships into to increase the utility of ETHX. Stater allows for both permissioned and permissionless nodes to join the network, maximizing its potential scalability for ETHX, while preserving the values of decentralization and openness behind its liquid staking token. Go to Stater Labs.com slash ETH and sign up to get access to the Stater Staking protocol. Arbitrum 1 is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web 3 landscape.

Hundreds of projects have already deployed on Arbitrum 1, producing flourishing defy and NFT ecosystems. With a recent addition of Arbitrum Nova, gaming and social daps like Reddit are also now calling Arbitrum home. Both Arbitrum 1 and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM compatible. On Arbitrum, both builders and users will experience faster transaction speeds with significantly lower gas fees. With Arbitrum's recent migration to Arbitram Nitro, it's also now 10 times faster than before.

Visit arbitram.io, where you can join the community, dive into the developer docs, bridge your assets, and start building your first app. With Arbitrum, experience Web3 development the way it was meant to be. Secure, fast, cheap, and friction-free. And we're back now. Terrance, I'd like to turn to you. What was your story? What is it like from your perspective to hear the words Maynett stopped finalizing as a client team deaf? Yeah, Stephanie not great hearing that on Thursday afternoon.

typically I stopped all my meetings after Thursday so I was excited to get the coding have some personal time read some paper but so I yeah so like Preston did I heard from the news I heard from Telegram and to lose 65% of participation it's it's a little crazy like when I first read that I just couldn't believe that I was like just like how would that happen right it's like imagine like 65% of the note just the just disappear, right? It has to be like, this sounds very like catastrophic if you really think about it. So in my mind, I just like started piecing everything together quickly, right? Because like 65% then I know it's not just one client because like thanks to client diversity, like every client is pretty

spread even like present as, sorry, Prism has 33%. My house has like 33%. So I know it's not just one client. so it's not a client bug. Then I thought, okay, maybe it's like a, maybe it's like a target attack, right? And then maybe it's like literally like someone is attacking the network. Finally, someone is actually attacking the network. And then, but luckily like the whole thing like went back to normal after like three to four epoch, which is like 20 minutes. So they kind of gave us more time to think about what the issues is, right?

And then I thought, okay, well, if it's a target attack, like why did this? stop. I mean, if they could have done for longer, like, why stop there? So I, so I didn't really think it was a target attack at that point. Then I thought, okay, maybe there's like a really big exchange or staking pool that run something on GCP and AWS. And then they shot down or something like that. So that was another reason. But I luckily, I think the whole thing took about like four to five hours until we came to the gist of it. That's basically our software was a little bit under optimized. So how hard of a problem was this to fix as soon as so it sounds like diagnosing wasn't too difficult. But what was the actual solution process like? It wasn't hard to fix.

We could have fixed the problem within the first 30 minutes, but we kind of wanted to do the perfect fix, right? Because everything is trade-off. Client software is a trade-up. If you optimize one thing, you cannot lose another thing. And this is why client diversity is nice, right? So I'm going to shout out client diversity again. Just lighthouse would fine because of the trade-up they made, right? Today, network is under another scenario. Lighthouse may underperform, like, versus prison, just everything is a tradeoff here, right? So we kind of, we kind of picked the fix that we thought had the right tradeoff that were we like. And then again, this is a team effort, like probably like five to 10 of us, like we got together, we talked about it. And then we're like,

okay, this is the fix that we want. We also share our fix, our idea with five other awesome client teams as well, right? And then because we're all learning from each other, we are all study each other's design. So, yeah. One interesting observation I have about this is this is, this is sort of a finality event, but it was also like a non-event, right? Like when we asked the question at the beginning of this episode, Preston did Ethereum break?

And you were just emphatically, no, it didn't break. And so this is maybe a demonstration of the resilience of the Ethereum network. And I, what came into play? Like, how did the Ethereum network show resilience during this event like that? Terrence, you just talked about multiple clients. Was that the key thing? Is that the thing that saved our ass, I guess I would say? Or are there other elements that demonstrate the resilience of Ethereum?

First to you, Terrence. That's one thing, right? If today, it was three years ago when Prism was a majority client, I will be really, really worried, right? but today we're just like a piece of the pie and I'm so therefore I feel like I can sleep a lot better. The second thing is that we really need to thank like this thing is called LMD Ghost. This is designed by Vitadeg and a few brilliant researchers from the Ethereum Foundation. So LMD Ghost is this consensus mechanism, is this four choice rule that we use for E2 or order of Bittgen chain basically. This is so essentially the Bican chain Ethereum Pro-Ferceasing

consensus is made up two part. The first part is LMD goes essentially it favors likeness. It basically ensured today if we don't have safety, we don't have finality, the blockchain keeps going. And this is very important, right, as it turns out, because before I always question what do we need this. Why can we just have safety? That's it. What can we be like a tandem in chain? What can we be like maybe like the other blockchain that just have safety, right? And this is literally the reason. This is why because like if things go wrong, if we don't have finality, we don't want to affect the use. We don't want to affect the UX, right?

We want people still be able to buy NFTs, they'll buy coffees, they'll be able to buy coffees, they'll be made trade. So essentially to answer your question, this LMD goes for choice rule was the one that saved our butt. It prioritized liveliness. That's what it did specifically. Preston, what would you add to that? What saved us? Yeah, I think Terrence said it well. It's the specification of Ethereum, L&D Ghost, and these things that make Ethereum

the way it is combined with client diversity, right? Like the specification worked here. When there's an incident, it can recover, it can live on. And because there was an incident and we're having client diversity, it was minimized to a subset of the network, right? It could have been worse if it was more of a majority prism.

It brings me back to the days when we had these early test nets like Madasha, which was 100% prism and it was just on fire. It was terrible when there were incidents. And, you know, thankfully, client diversity, we've done a really great job over the last two years or three years, however long to smooth that out. And we could see it here today that there was an incident and it recovered on its own without intervention within minutes. And that we were able to release mitigations and hot fixes and client updates within days to say, okay, now it's not going to happen again.

So I think that went really well. It's a great test in Maynett. So I'm going to pull in something from my Bitcoin or world. They always love to say this is good for Bitcoin. And the spirit of that response is that anti-fragile crypto economic systems, what doesn't kill them makes them stronger. So Preston, was this good for Ethereum? Yeah, I think, you know, like we're saying, this is the first time we've seen it in May net in production. We knew it would work, we've done it in test nets, but actually see it in action.

It just goes to show the resiliency that when things are going horribly wrong, it can still be okay. it still eventually finalizes and that the mechanisms and the specification and the design of Ethereum proof of stake is resilient. So I thought that was great. And also, yeah, Robert. So like Ethereum is an evolving protocol as we know, right? It is still constantly heavy under heavy R&D. And then we definitely came out here with like a very better understanding on just like very little detail of things that we want to improve of even like future research spec as well. Just I heard your podcast on MEP Burn, right? That's a very good podcast.

But because of this, right, we're kind of looking at MEP Burn. We look at PBS to see how we can do better there as well because everything is interconnected. Like they all, they basically all touch each other. And I definitely feel like we definitely got all of this very strong. And not only that, but clients. have discovered a weakness that is now removed, correct? Yep. Cool.

Nice. I love anti-fragile systems. My last question is this, and this goes back to, Terence, something that you were saying about kind of the design of Ethereum proof of stake with LMD ghosts. It's like way in the weeds for people, but like the priorities that the Ethereum network makes. As we all know, it's not high transactions per second. on the base layer, on mainnet, right? Instead, Ethereum has made other tradeoffs.

And one of those tradeoffs is a liveliness, a liveliness tradeoff. And I want to just focus for a minute on some other outages that have happened in crypto with other layer one blockchains. And we don't have to name names. It's like I don't want us to, like anyone to feel like, you know, Ethereum's punching down, that kind of thing. Okay. So like, but like...

Just think of the first name that comes to your mind about a blockchain. But also just like just the facts. Okay. So like I think when people were talking about, and by the way, bankless included, we're sorry, everyone listening for the tweet headline. All right. We've made amends on that. That happened this week.

But when people equate Ethereum breaking to sort of the downtime and the breaking or temporary halting of other blockchains, it's much different. So I'll just throw the name Solana out there just for a minute. So we can talk about that as a just the facts kind of use case here. When Solana was down, because it did not prioritize liveness when it had some issues, it was actually offline, right? Like you couldn't get a transaction through for a period of time. I'm wondering if you guys could just weigh in on that briefly because it is a useful contrast point.

Okay, Theorem doesn't have the highest transactions per second on the main net, but it does have this feature of liveliness. Terence, could you talk about that for a minute? Yeah. So I just wanted to add, right, like there's nothing like what we are doing today, right? We're working on open source software and this is the open network. That means anyone can read any code that we write. We're like chefs or cooking in the kitchen, right? And anyone can join the network, right? So it makes it very like vulnerable to a text. Therefore, right? Therefore, I think favoring lifeness is very important just because like you want your life.

to keep going, right? Say today you have a country that's built on top of Ethereum, I want to buy coffee. And then I mean, today, Ethereum is done, cannot process transaction, I cannot buy my coffee, then I will be very upset, right? So I would say lifeness is a very important property, probably the most important property. If we want higher food, there is layer two, right? And that's what layer two is for. But Ethereum as a base layer, definitely lifeness. Yeah, you know, I think that's what makes a blockchain great, is to have liveness, right? It's not useful if you can't use it. So to have liveness is, I think, one of the most important aspects of Ethereum, more so than transactions per second, right?

We have solutions for that. We can abstract to layer two or other clever mechanisms. But if you don't have liveness, then none of that works. You have zero transactions per second. So it is the most important aspect of Ethereum. Preston, Terrence. Yeah, thanks for summarizing this. Thanks for jumping on the bankless podcast, explaining this to the bankless community and the

crypto community. Did Ethereum break? The answer is no. This was maybe whoever tweets that is dumb. Yes. This is maybe Ethereum's biggest liveliness test yet, though. And so it was good to come out on the other side and see a resilient network. Pretty unique.

I think something that's very special in the space and worth preserving. And as always, got to thank you. Whenever we have client devs on the bankless podcast, we would be remiss, David, if you and I did not express our thanks, both to you, Terence, and to you, Preston, and also to any other client dev that is doing this hard and essential work

to build this incredible freedom technology network that we care so much about. Thank you, sincerely. Open source superheroes for... We appreciate you guys going on. Yeah, it's our pleasure to get to work on it. Really, it is an exciting opportunity. We know that Ethereum's going to change the world, and to be working on it at the protocol level,

there's nothing better that we could imagine doing. So it's our pleasure to be here. Plus one to what person said. Thank you for having us. Recent disclaimers, guys. Of course, got to let you know, Ethereum is resilient, but ETH the asset is risky. So is all of crypto.

So is Defi. You could definitely lose what you put in, but we are headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the bankless journey. Thanks a lot.