Radiolab - Bit Flip

Episode Date: May 8, 2019

Back in 2003 Belgium was holding a national election. One of their first where the votes would be cast and counted on computers. Thousands of hours of preparation went into making it unhackable. And when the day of the vote came, everything seemed to have gone well. That was, until a cosmic chain of events caused a single bit to flip and called the outcome into question. Today on Radiolab, we travel from a voting booth in Brussels to the driver's seat of a runaway car in the Carolinas, exploring the massive effects tiny bits of stardust can have on us unwitting humans. This episode was reported and produced by Simon Adler and Annie McEwen.  Support Radiolab today at Radiolab.org/donate.  And check out our accompanying short video Bit Flip: the tale of a Belgian election and a cosmic ray that got in the way. This video was produced by Simon Adler with illustration from Kelly Gallagher.

Wait, you're listening. Okay. All right. You're listening to Radio Lab. Radio Lab. From. W. N. Y. C.

See? Yeah. Anyhow. So what have you been up to? Annie, do you need a microphone? I think we're going to share. Because what else are we going to do? I think it's good.

It's fine. And he's going to have to sort of strike like a viper. Hey, I'm Chad. I'm Robert Crowlwich. This is Radio Lab. go like this. Yes. I'll be like...

And today, that's like that. A story about when things from far, far away. It's a combiner head. Hit very close to home. Just put your hand on his forehead. Or over my mouth, that's fine.

Comes to us from producers Simon Adler. And Annie McEwen. Okay. Okay. So you're going to tell us something about something. Should I start us? Yeah, yeah.

You start. Okay, so we've been exploring what happened on May 18. 2003 in the great city of Brussels. Specifically in Scarbeck. Scarbeck. Scarbeck. One of the municipalities in Brussels. Story starts with this guy. So it was around 11.30 p.m. and I was with one of my colleagues at a restaurant to have one of the good meals of the day.

This is Emmanuel. Emmanuel Willems. I am the one everybody calls when shit hits the fan. Yeah, let's put it that way. And that evening, Emmanuel, along with much of the rest of the country, was waiting around for that day's national election results. The results for all the different parliamentary assemblies. But Emmanuel's mind was on something else. Because across the country that day, millions of people had just voted on computers. Belgium was in the early days of experimenting with electronic voting. And so instead of filling in a paper ballot, in the voting booth was a touchscreen computer.

and voters would walk in, insert this white credit card-sized card into the machine, and then vote by selecting names on that screen. And it was a manual's job to make sure that all these hundreds of computers worked properly. I'm an IT guy.

So something like that. And at that point, at 1130 at night, after the polls had closed, besides the usual computer glitches... It seemed like everything had gone well. And so we were... There were two of us... He and one of his fellow IT guys.

waiting for the others, I guess, to talk about the day. Pat each other on the back for a job well done. And that's when the... That's when his phone rang. It was unexpected. And it was one of the officials of the ministry. They said that they had a problem, that they had detected a problem in Skalvik. And so I said, okay, we're coming over.

And so you pay the check, you hop in the car. I hope we paid the check. We dashed out of there. We just didn't leave like that. 15 minutes later. They arrived. They get a check. leather black escalades, as I like to imagine, and they enter the polling station.

The townhouse of Skarbak. This ornate building with big, tall ceilings. It's a very old building. Filled with this crowd of confused-looking people. Well, they explained to us that going through the numbers, they noticed one of the contestants. This relatively unknown Communist Party candidate, Maria Vindivogel, had a very high number of votes from this one polling station. Okay.

And taking a closer look, they noticed that the number of votes for this contestant was impossible. Wait, impossible? Yeah, mathematical impossibility. Essentially, she received more votes than there were voters who could possibly vote for her. And so, as Emanuel's hearing this, I'm curious. Is this just a fluke? Is there an explanation? Fraud.

You're thinking fraud? I'm smelling fraud. It could have been tampering. It could have been a bug. It could have been a hardware glitch. Could have been a... It could be anything. Yeah, absolutely.

So first things first, they got all of the ballots, those white plastic magnetic cards that the voters had loaded their votes onto. And recounted, reinserting every single magnetic card one by one. This took several hours. So now it's like 1.30 in the morning. And they print out the new recounted report, hold it up against the old one.

And they are the exact same results for every single contest. to the vote, except the contestants, which had had this abnormal number of votes. This time around, Maria had far fewer votes. And at that point, one of my colleagues did the math and said the difference between the number of votes she had in the first count and the number of votes she had in the second count. It's exactly 4,096 votes. She had 4,096 fewer votes. Does that mean anything to you?

4,096. No, it means nothing to me. Well, 4,096 is not a random number. It is a, it's something about the number. Tell us. I don't know, I don't know the answer, but I have this, a deep sense. I'm feeling tingly. So if you guys were IT people and you heard the number 4,096, you'd be like, great, Scott! Would we?

Yes, you would. Yeah, absolutely. To every IT guy in the room, the number 4,096 was very, was very. very meaningful. In fact, it was a clue, a clue to a cosmic culprit capable of swinging our elections, and switching our cars, and reminding us just how small to... Crashing our cars. ...and-away Toyotas taking off on their own. We're going to wreck. We're going to wreck.

And reminding us just how small and at the mercy of the universe. We really are. So wait, so why is 4,096 so important? Okay, well, to explain, I'd like to back up... Yeah? To the morning of the election. Okay. So, polls open, and at some point,

someone slid their white card through that card reader, and Maria Vindervogel received her first vote. He registers vote number one for Mary. Yeah. Now, when that happened, inside the computer, in its memory, a zero switch to a one. Okay. And I mean that both in terms of her vote total and also in how a computer counts. The way computer works is that everything is stored in memory. And memory is binary. It's either a zero or a one, either on or off.

Zero is off and one is on. And I think we all know this, right? Or we've heard it at least that deep down computers are zeros and ones. Yeah, yeah. And so in this specific case, what that means is all these individual votes that Maria would get throughout the day, they were all stored as just ones, as these individual ones. Okay. Now, when you're limited to counting with just ones, a way to do it is to tally, like you're doing groups of five on a chalkboard. Right. Diagonal line.

Bundling it. Yeah, exactly. But when that tally starts to get high, it becomes incredibly inefficient to count that way. So instead, to save time, energy, and computing power, computer. computers do a little bit of math. They count using powers of two. Powers of two. You remember powers of two?

Maybe. Something squared is a power. Exactly. Two squared or two to the second power, two times two is four, two to the third power. Two times two times two is eight, two to the fourth, 16, et cetera, et cetera, et cetera. So how is it that a computer is doing that? Right. Well, so to help you visualize this, imagine inside a computer's memory, there's this string of 13. light bulbs. Light bulbs that IT people call bits. Each one representing a different power of two.

Just stick with me here for a second. Yeah, okay. So that first light bulb represents two to the power of zero, which is one, because anything to the zero power equals one. Don't ask math. Second light bulb. Two to the first power. Which is two, the third bulb. Two to the second power. Or two times two, four, et cetera, et cetera, et cetera. Now, when Maria received her first power, which is two, two, the third bulb. first vote. A little bit of electricity was sent to that first light bulb. Two to the power of zero. Turning it on. Two to the zero is one. Exactly. Now when she got her second vote, this time some electricity was sent to that second ball. Two to the first power. Turning it on representing two votes. And because two votes are stored in that second light bulb,

what the computer does simultaneously is turn off that first ball. So to be clear, when she got her second vote, the first bulb was turned off, and the second bulb, which represents two to the power of one, was turned off. So it's now using only one bulb to represent two votes. Oh, okay. Now, in order to represent numbers that aren't powers of two, all the computer does is turn on multiple bulbs at once. A sequence of power of twos. For example, when Maria got her third, it turned on the first and second light bulbs. Because one plus two equals three. When she got her fourth, it turned off the first and second light bulb and turned on the third. Two to the second power.

Fifth vote turns on the first and third. The power of zero. When she got, say, her 17th vote, it turned on the fifth bulb. Two to the fourth power. 16 plus the first ball. Two to the power of zero. One, equaling 17. And so on and so on, adding up these powers of two.

And with that respect, 4,096 is a power of two. I see. So that number is one that they're like, blink. Is that what that is? Yeah. It's the 13th bit of memory, which meant that... At some point during that date, something somehow turned that bit on. It flipped what should have been a zero to a one. Yeah. It was a binary flip. Your mind just went there immediately.

Oh, yeah, absolutely. Yeah. I love that all the IT people are like, oh, and all the non-IT people are like, what's going on? Well, to each is trade. Yeah. Well, okay, so you know that it was just one bit that flipped. What does that make you? So we knew that the result could be traced back to the 13th bit flipping.

What we didn't know was what caused the bit to flip. Okay, so that's your next question. What could cause this bit to flip? No idea. What were the options? Well, it could have been any number of things. And so we each, in the following days, we got access to absolutely everything. First thing they checked.

It could have been software. So we carried out random tests on the source code. Looking for any bugs that could have caused this bit to flip? But there was nothing major. So we were fairly confident that it wasn't a software error. Okay, so if it's not software, it's hardware, right? Well, we got the machine that was actually used in that polling station on the day of the election.

The computer that had done all that tallying and counting, and they ran a bunch of memory tests on it. Memory testing to see if something came up. To see if some defect in the physical memory could have caused this bit to flip. What came back on that front? Nothing. Not a single error. So the memory on that computer was working fine.

Well, so what the heck are you left with? Not much. As far as we were concerned, this one computer had had a glitch, which caused it to give a wrong answer. But isn't that sort of an unsatisfying answer? That's the way things go. You know, we just simply had no proof of what caused the bit to flip. We just don't know.

That's the way it turned out. And so over the next couple of days, Emmanuel and his team sat down to write up these findings, half expecting that once finished, this report would be filed away and never mentioned again. However, hello, hello, Brussels, can you hear us? Hi, is that Simon, it's Natalie here in Brussels. How are you? That was not the case.

I have your guest here, armed with plenty of tea. I'm just going to let me have a seat and have a chat with you. Thank you. Yeah, thank you, Natalie. No worries. Okay. Merci. Hello?

Hello, is this Zoe? Yeah, yeah, it's me. Hello, Zoe. Hello, hi. All thanks to this lady. So I'm Zoe Ginoe. I'm member of the parliament. And back in 2003, young and freshly elected to parliament. I was 28 or 29.

Like half the age of most of the MPs. Yeah, it's that. She was generally aware of what went down in Scarbeck, that there was this computer problem, someone got some extra votes. It was really strange. And for me, it was really a concern. Because our most concern is that the people can. can trust the electronic system. And with this sort of problem,

we are not sure the people can trust the system. So we waited for the report of the specialist. And finally, about three weeks after the election, it came out. Yeah, we only, well, we ended up writing that at some point during the day, the 13th bit had flipped. That's all we wrote, basically. Means nothing. It means nothing.

Well, then that's what she's thinking. No, no, no, no. There must be more. Nobody can understand this sort of vocabulary. And I was confused with explanation. Because way down at the bottom of the report in the final paragraph, the only thing they said about a possible cause was, and I quote, this phenomenon is extensively described in the specialized literature.

Yes, we did write that. And what were you referring to there? Nothing in particular. I said, what's this? Why did not explain what's really? So Zoe went to her fellow parliamentarians and was like, There was a problem. We don't know what really happened. Like, we've got to look into this.

What can we check? Who can check? Who will check? But the people, the other member of the parliament, they say, oh, yeah, Zoe, I know she's there with the strange style and she's there speaking about problems. Basically, forget about it. It's done. We are elected. election are gone, there was no problem. But I said, no, no. We must find out what was really the problems and speak about it.

So Zoe contacted me and she did send me the paper report. So she reaches out to this guy named David Glode. He is a member of a group called... Our name is Poreva and we are fighting for ethics into electronic voting. He hates. Electronic voting. Basically, we are against voting machine, yes. But interestingly, he's also like a hot shot IT guy, works for the European Parliament.

Yeah, and as an IT people, I tell you, don't trust us. And don't put everything in our hands. Anyhow, Zoe had known him for a long time. And so she reached out and asked, can you help me understand, what are these random bit inversions? Like, what is this? So the report, I read it. I said, hey, wait, I did computer science studies and bits are not flipping. What are they talking about?

So I needed to find out. So I guess I started to Google to ask what could explain this. Initially, he came across some articles or blog posts, arguing that these sort of things could always be traced back to some software bug. But they did not find the bug, so they blame it on something else. Not totally satisfied. He tried some new search terms. Blah, blah, blah. Bit flipping.

And came across a couple articles that made this strange. They said the cosmic ray could flip a bit of memory. Wait, what? They said, yes. If the cosmic ray touch your computer the right way at the right spot, it could change a bit from zero to one. Gamma rays aimed at Belgium in favor of a particular Walloon. We're not going to be able to use any of this tape.

This is great. Wow. So you're saying that what created this effect in Belgium on that day was a zing from deep space? That was the theory that David came across. And my first reaction was to say, hey, hey, cosmic ray. Hey, what a joke. But then it became a bit bizarre because I found out about some IBM research.

Freeing man's mind to shape the future. An IBM paper from the 70s, and in it... Electronic Data Processing Center. They made a lot of studies about random bit-flipping and problem with computer memory. The report explained how in the 60s, NASA had noticed these random bit-flips in their satellites. And then more recently, how IBM itself...

Problems by the computer can be displayed in various ways. ...had studied their computers at higher altitude. So closer to a space, let's say. Closer to the sky. Closer to God. And they found out that they were more bit flipping up there than at the sea level. And the only explanation they could come up with for what could be causing this was cosmic rays. Yes.

When they investigated, they found out that it could have been cosmic rays causing problems. Oh, okay. Okay. Because cosmic rays essentially a charged particle. And so electronic systems. can get affected. This, by the way, is Professor Barat. Barat Buo. I'm a professor at Vanderbilt University in Electrical Engineering and Computer

Science Department. And while Barat wasn't involved in that specific IBM paper, he has essentially dedicated his life to studying the effects of cosmic rays. So we started noticing on the surface of the Earth. This problem started around 2001-2002 time frame. Recently. Right. And they have big ramifications. Is this a thing that people know about?

I mean, is this like a... Well, yeah. This is the first time I hear of it. Well, it's... I don't know if you should continue on the story or go into cosmic rayland. Yeah, what do you want? Yeah.

Do you want narrative or do you want cosmic ray land? Oh, I love that that's a choice. We get to decide. I'm torn. Well, I don't know. Maybe we just go with the story. I don't know. I think we dive into...

We give some of this. Okay. Okay. Oh, yeah. Okay. So according to Barat, to understand what happened that day in Belgium, we have to go back to a long time

go, wait for it, in a galaxy far, far away. I'm going to generalize way too much. No, no, that's good, that's good. Way too much. Brat says in this distant galaxy, there was a very, very old star that exploded. And when the star explodes, it sends out a huge amount of material. All sorts of little, tiny charged particles. discharge particles are thrown into space.

And these particles are going to travel through space for millions of light years. Simon. Right. Sorry. These particles are passing through other galaxies. They get deflected. They get trapped.

Going around suns. The gravitational pools. Until finally, on May 18, 2003, one of these particles with lots and lots of energy. enters our solar system. And not long after that... Comes into the atmosphere.

And it crashes into an earthly atmospheric particle. And in that moment... Energy is transferred to the atmospheric particles. Essentially, it hands off a little bit of energy, causing that newly energized atmospheric particle to start descending towards Earth. That is correct. But on its way down,

it smacks into a couple other particles, passing some of that energy on again. causing those particles to fall, and crashing to other particles. And this happens again and again as these particles and this cosmic energy catapult towards the surface of the Earth. And so you get this cascade. It's a shower of all these particles. From the top, it will look like a Christmas tree.

Now, down at sea level, down on the surface. A divided country, Belgium was going to the polls. The electronic system that was counting the votes in Belgium. zooming along, humming along in its own way. Two to the six power. Tallying the votes by turning on those light bulbs. Now, the switch that turns on those light bulbs is called a transistor.

The transistor turns on or off, and that creates binary ones and zero. When it's on, it sends out a little bit of electricity switching on a bulb. But the thing is, these transistors can be tricked into turning on. So imagine... Suddenly, this particle from outer space... comes zooming down and enters that voting machine, hitting precisely the transistor that controls that 13th bit. And that instant...

The transistor is tricked and sends out a burst of electricity to that 13th position. To the 12th hour. And the bit flipped. A zero was changed to a 1, and from the heavens, those 4,096 votes appeared. You what the hell? What a chair? David glowed. I don't know. I mean, it's like if I talk about Cosmic Ray, on one side,

I'm afraid that people will take me for somebody crazy. But, well, you can read it too. If you Google those words, you will find a lot of scientific stuff talking about that. And so, not long after, David and Zoe met up, and David

told her what he'd found. That it's possible, who knows, but it's possible that the vote was changed by the cosmos. I said, a cosmic ray got down in the computer and it makes a mistake. I said, it's not possible. You are really sure. I mean, I've got no clue if that's the true explanation or not. And there is no way to prove that it's a cosmic ray.

But, I mean, it's the only thing which matched their explanation. I said, oh no, I will never can explain that in the parliament. They're all going to laugh. Aware of all of this, at the next session of Parliament, I decide to ask a question to the new minister who was responsible for the election. So the session gets underway, and eventually the agenda turns to Zoe. Now, there's no audio recording of this session, but based on transcripts, it went something like this. She walks up to the podium at the front of the chamber.

and begins. So, Mr. President, Mr. Minister, at the occasion of the last election, we have

been in Skarbeck, and then explained what David had told her. That this beat inversion, it's a

cosmic ray who caused a problem in a computer. The room was there, joking, ha, ha, ha. Zoe Geno,

come here with the little green man. Zoe, what a dumb-dum. And the minister... They say, yeah, yeah, don't worry. It was a small error somewhere, but we see it. There's no problem with the result of the election.

With that, the minister sort of motion to move on. But before her time could expire... Zoe jumped back in and basically said, like, wait a second, think about what this could mean. If this is true, like, can we try? trust these machines. So I asked, are you sure there was no other cosmic race somewhere who changed some of the result of the election?

And the minister? After pausing for a moment. Admits, he was not sure. And when the minister say, yeah, perhaps there are other problems we don't know, there was a strange silence. and nobody can't laugh anymore. Democracy and stars do not get along, apparently. I mean, ipso facto dingo dango.

I think someone here needs to hold the skepticism. I'll be the skeptic. You be the skeptic. What are the odds? It's fun to, supposing this is the truth. There's no question that this does happen, that a cosmic ray can hit a computer and goof things up. I mean, the only thing I can say is that...

I mean, even a manual will concede that. Yeah. Cosmic radiation can have an effect on electronic computers. And Barat and his team have actually synthesized this happening. If we suspect a bitflip, we would actually put the computer in linear accelerator. As usually, it's a cyclotron. Which is basically this mini-controlled supernova. Pretty close to it.

It's all magic to me. But what they do is in cyclotron, they can get you the type of... They put the computer in this concrete bunker, point the particle beam at it. Have the electronic systems working. we drag all the wires out of the room. Then there are a few buttons to push three, two, one. The light comes on, alarm sounds. Beam time.

It's called beam time. That is correct. Oh, wow, that's great. That's so great. Yeah, beam time. They blast the computer with these tiny particles. A million particles for a second. Just like those cosmic rays flying through space.

And as this is happening, Barat says, you can sit there and watch the bit flips. And given this fact, what Zoe was really asking was... If the extra vote were not 4,096, but a small bit flipping, then would you notice? Like what if a candidate just got 32 extra votes and that invisibly changed an outcome? That, yeah, that's the point. I mean, it's only when the computer produced an impossible result that you can tell that there was something wrong. And we call that the tip of the iceberg.

You want us to just throw it a break, or do you guys want to come in and toss? Let's hear what you do, and then we'll come in and ruin it. Classic. Music's well, music's well, music's well. And we're like, so buckle up. When we come back, we are driving full steam ahead into that iceberg to figure out just how deep it goes. What an interesting metaphor. That's a mixed metaphor.

You're going to drive a car in the internet iceberg. You can drive a boat. I love it. I love it. Oh, my. You have to ask Mrs. Vanderbilt, your third grade teacher, to go back to the metaphor section of the lesson in the next hour. This is Angela Babiars from San Jose, California. Radio Lab is supported in part by the Alfred P. Sloan Foundation, enhancing public understanding of science and technology in the modern world.

More information about Sloan at www. W. sloan.org. Okay. Eddie? Chad. Robert. Simon. Annie.

Radio Lab. So, all right. So we just heard that a single cosmic ray from deep space flipped a particular election result. I guess the next question is like, you know, is this happening all the time, some of the time? Well, if you mean like how often do those charge particles hit Earth? Every day, every second. We are bombarded by these particles.

Quite often. Because it turns out these cosmic rays. They're coming to get you, Barbara. They're coming. Stop it. They are? Yep. They're coming for you.

Look, that comes one of them now. Right now, right through your body, you are getting thousands of these particles going through. And even if you are inside a building, you are still going to get all those particles, because most of these particles just cannot be stopped. They just go through anything and everything. So Brat seems pretty sure of himself. Yes, yes. But Simon and I were like, wait, we got to see this for ourselves.

We should turn off that screen because we've got to make it dark in here. And apparently, although these particles are just too small to be seen with the naked eye, we are currently in the studio, blacking the place out. Barat did tell us there's something we can build, something called a cloud chamber, which is basically just an upside-down cup you fill with vapor. Yeah. That might let us see the trails these particles leave behind. If you can make one, that would be really, really good.

Okay, here's what you're going to need to make your very own at-home Cloud Chamber. Now, it turns out there's a lot of YouTube videos showing you how to do this. Hi, I'm Joanna. And I'm Steve. Okay, Cloud Chamber. One starring a kid who looks about four years old. So, we thought we had a pretty good chance of making it work. Whoa.

Yeah. Be careful. That's cold. We got dry ice, cast iron pan from my kitchen, flipped it over. And we are now putting the cast iron over the chunks of dry ice. The metal cooling down. That was amazing. The metal pan was singing.

It was such a pure tone. Good job. Metal pan. Next, take a see-through plastic cup. 24-ounce. Some felt. Our felt is purple. It's like if Prince had a little bit more red in his purple.

Stick that felt in the bottom of the cup. Hold it in place with some magnets on either side of the felt. Magnets, Chad. Soak that felt in rubbing alcohol. What do you think? Soaked. That seems pretty thorough. Turn the cup upside down onto the now very cold cast iron pan and seal it in place with modeling clay.

Overhead lights off, flashlight on, aim it at the base of the cup, and then wait. Let's watch for it. How is this thing going to work? What are we trying to see here? So what we're looking for, what should happen, is that that cold metal is going to make the alcohol vapor inside the cup condense and like turn into a cloud. Which you're going to make a cloud. Yes. Kind of foggy looking in there. That's what you want, right?

And through that cloud, where we're hoping to see are these tiny little particles shooting through. And we won't be able to see the particle itself, but we will be able to see the trail they leave behind.

And it's supposed to be very similar to the trail that an airplane leaves behind in the sky. A con trail, those things are. Conrail, exactly. So what we're hoping to see is these tiny little airplanes

with their tiny little contrails inside this cloud. And that will be evidence of these very particles. Exactly. So all three of us are on our knees in the studio. Pitch black at the moment.

The only light is emanating from a headlamp on Annie's head. But maybe it's one of those like watch pot never boils things. Nothing happened. I don't think there's enough fog in there yet. Come on, little part of those. After about an hour of this, pulling up YouTube on our phones. So jealous at this four-year-old.

Check our scientific method. I don't know. Chad and Simon had to get back to work. Good luck. They had things to do. Okay, I'll be back in 10 minutes. Okay. I stayed behind, just in case.

Why is it working? Alone in the dark, with a little beam of light. Oh. I saw one. You saw one? Yeah. Simon. What is it looked like?

Like a tiny little... He's very, very tiny white, thin, thin threads. There, do you see that? Materialize. There, I'm seeing a bunch of them. Really? Suspended in the cloud, holding their shape for just a moment. Oh, there's a long one.

Wow. Before disappearing. Oh, Jesus. Once we started seeing them. Oh, they were just like, there's little backers. We couldn't not see them. Oh, that was a big one. There's like three there.

That one was coming right through me. Okay, I can now tell you not just because someone told me, but because I saw it for myself, these things are definitely real and there are definitely a lot of them. And so I guess the next obvious question is, like, if we know they're here, besides that voting booth in Belgium, like, what else are they messing with? Like, has my phone ever, is the probability such that my phone has likely been goofed by a bitflip caused by a cosmic ray? Absolutely. So all our computers, all our systems are getting bitflips on a daily basis. Okay.

For example, the routers that we have on the internet. Did parts of the internet get shut off by these things? Yeah, routinely. What? Even today. Whoa, whoa, whoa, whoa. Yes. People just assume that, oh, my internet went down for five minutes.

All the internet companies, like Cisco systems, they are very keenly aware of this problem. Apparently, in the early 2000s, Google's indexing system was down for five months. No kidding. What else has been goofed by these? A lot of manufacturing plants have been goofed by this. Really? There is a custom manufacturing plant, which made people. custom machines and one of the bitflip.

So they were making things wrong. Wait, what's a real example of that? Not sure I can discuss this on the air. Oh, really? Can you say what kind of machine? Not sure. Oh. Sorry.

Something that Barat was able to tell us is that the higher up you are in our atmosphere. The number of particles with higher energy is higher. And that means, the chances of a bit flip are also high. The electronics in airplane is always bombarded by these particles. And one time the plane is flying, it's on autopilot, and suddenly one of the bit flips that takes it out of the autopilot. What?

Yes. Who? What airplane? Where? This was a Qantas Air Flight in about 2008. 13 passengers and one crew member were seriously injured when the Qantas Airbus nose dived, plummeting more than a thousand feet.

Jesus. Oh, my God. They essentially dropped like a rock. And it was all because of a one bit flipping. Which they believe was caused by a cosmic ray. Wait, it just nose dived for a thousand feet and people like hit their heads on the ceiling. There was something that was on just turned off and there was nobody. Oh my God.

Because of a bit flip. Holy shit. Now, we should say that, like even though bit flips are happening, frequently. The vast majority of those flips? They are benign. We don't even notice them flipping. Those bit flips matter only if it's right there at the right position. So, for example, in that parliament, that bit flipped.

But that bit flipped mattered only if they were taking a vote at that time and actually telling the vote. So the likelihood of a bitflip having any consequence at all? It's rare. Okay. Hello. This message is for either Grover or Barbara. And that's what made this next thing, Brad told us. They're coming to get you, Barbara.

It's a strange call to receive, I'm sure. Even crazier. I'm sorry I didn't answer the phone initially. I'm just tired of people who want to reduce my credit card interest or give me something for my pain. This is Barbara Walton. We reached her in Boiling Springs, North Carolina. Little one stoplight town. Where she lives with her husband, Grover.

We've been here since we got married 53 years ago. We're hoping to be able to understand what it was like to be. in one of those cars. Oh, my gosh. Oh. Do you mind just like telling me what you were doing the morning of the accident? We live in North Carolina. We have a son that lives on the coast in South Carolina, so we were going to visit.

It's about a four-and-a-half-hour drive takes you through some pretty good-sized towns. The area we were in was more or less a rural area. They're driving a 2008 Prius. Grover is behind the wheel. And I was in the backseat because I was on my computer and I didn't like to do that in the front seat because of the possibility of the airbag coming out and throwing the computer in my face. And what were you, if I can ask, what were you doing on your computer in the back seat?

What would I have been doing? Probably solitaire, something to occupy my time. I usually snooze or, you know, we've made that trip many times, so I've seen the scenery. For the first three and a half hours of the drive, everything is totally normal. Grover calmly at the wheel, Barbara on her laptop, and then Barbara, looks up from her computer. And I noticed my husband is speeding. And I said, what are you doing?

And he said, I can't slow down. And I said, well, you have to slow down. He said, I'm applying the break. It is not slowing down. Do you guys remember exactly how fast you were going? He's a much better judge of that than I would be. Let me ask him. G.H.

When our car ran away with us, how fast would you say the car was going? Probably 65. I'm sure it was more than that. Probably he's saying 65, I'm saying 65, 70. Okay. How are both of you feeling? I can imagine feeling like really panicked. Mr. Walton doesn't panic.

Mrs. Walton panicked. I am freaking out. At this point in the drive, they're on a two-lane highway cutting through a huge wetland area. So on both sides, it's just swamp and water. And I didn't want to go down in the swamp. That's Grover Walton, Barbara's husband, the driver. Every time I attempt to break, as quick as I took my foot off the brake, it would start accelerating again. I finally just basically stood on the brake.

The way they described it, it was almost like one of those Hollywood movies, where you're like careening down the highway. We are not slowing down. Zipping by cars. Seat belts on. Weaving in and out of traffic. Computers down. Praying that this was going to turn out all right.

For somewhere between 10 and 15 minutes, Grover fought with the car. I can't remember if it. How did you stop it? I know we crossed that swampy bridge. Well, I wore the brake pads and the rotor out. Okay, so that's what stopped it. By just applying the brakes so long and so hard, it wore the brakes and the rotor out,

and that's how the car finally came to a stop. They called their son, who did some quick research online. And apparently there were already complaint. The driver of a Toyota Priya says he couldn't get his car to stop. Terrifying account of a runaway ride in a Toyota. Turns out all over the country. It just kept speeding out. Even with his foot on the brake, people were losing control.

Flunged over a 100 foot cliff. He had more than 90 miles an hour. He couldn't stop his Toyota. Lexus cars, Toyota cars, Toyota trucks. Of their Toyotas. This is a serious problem. I remember this. This was a big thing.

This was a big news story, yeah. From the driver point of view, what was happening was that the car would take off, sometimes at low speeds, but sometimes at high speed. And the drivers would say, my foot was on the brake pedal, but my car kept going faster. This is Phil Copman. He's a professor at Carnegie. Mellon University and co-founder of Edge Case Research. And my specialty for all these years has been fault-tolerant computing.

And Phil told us that around the same time these accidents started happening in the early 2000s was the same time that Toyota cars and all cars really were starting to become computerized. That's right. Before this moment, cars are mostly mechanical and you could like, you know, pop the hood and basically figure out what was going on. But after this moment, it's mostly computers and not just in the engine. Your car has dozens, or in some cases, more than 100 little boxes that have electronics inside it. How small?

Oh, sort of like small book size, like a paperback book. Embedded within the wall and floor and ceiling of the car? Wherever they can find room. They do all sorts of little things. When you take out your keys and you press the button and your doors unlock, there's a computer inside that key sending a message to a computer inside the car telling the computer at the lock to open the locks. Three computers are conversing about that unlocking moment? At least three. And they're responsible for some pretty important things.

Like now when you press the gas pedal, instead of it being this direct physical link to the throttle, which makes the car go, that pedal is now... Basically just an input to a computer. Why do they want to do that? Well, you can get better fuel economy. You know, it's a more pleasant drive. So that part is great. It's a better driver experience. But you just trusted your life to software.

that's a new thing. Was that talked about or known, or was this sort of a silent transition that you as the driver would never really know unless you looked it up in popular mechanics or something? I don't think that you would probably have known as a regular driver. And so... When this whole thing broke... Because most people didn't know that their cars were now computers, and because most of the victims... Oh, gosh. We're on the older side. We might be older, but we're not stupid people. They were basically blaming the driver for all of these. Barbara says when they took the car to the Toyota dealership. I'll never forget this smirky grin he had on his face telling us he didn't find anything wrong with the car.

If you're on the car company's side, you say, so you press the pedal and it kept going faster. Which pedal does that? I told them, I've been driving since I was 15, and I knew the difference in the accelerator and a break. And so these random accelerations sort of flew under the radar. Until... 9-1 emergency, ready to report. August 28, 2009... Yeah, we're in a...

I'm sorry, your cell phone's cutting out. A 911 call came from a passenger of Alexis Toyota. We're going north-125. Mm-hmm. And I'm sorry? I show it a step. We're on 125 or take. Okay, northbound 125.

Where are you passing? We're passing up. We'll be passing. We're on 120 Mission Gorge. We're in trouble. We can't, we're just no break. Okay. Okay. And you don't have the ability to, like, turn the vehicle off or anything? We're approaching the intersection.

Okay. We're approaching intersection. Hold on. Great. Okay. All shoes. Hello. Not one of the four people in that car survived.

It's a 45-year-old male California Highway Patrol officer, Mark Saylor, whose specialty is vehicle inspector. And this guy can't stop his car. And this just broke that driver error narrative. Toyota has a problem. To pieces. And so that's what launched all the inquiries and the recall. A record $33 million in fines. Toyota settled a huge $1.6 billion lawsuit. No electronic based cause without admitting any blame. But after that, for at least 2,000 reported cases. A four-year federal criminal investigation. With the death count rising to 89 people, a queue of 500 death and injury cases started to form. And when the experts, Phil being one of them,

were given full access to Toyota's software and hardware, they found a number of things wrong. But the key in all of this was, you guessed it. If you had a bit flip, the task that controls the throttle position could die. Causing it to stick in some open position. That would happen. They reproduced that in a car. Well, and in the first of those death and injury cases, what happened? The jury said that they, well, I have to make sure I only say things I'm allowed to

say. This is actually a big deal. Oh, okay. Yeah, so they, Toyota went after one of the experts for contempt, so you have to be careful. Wow. Okay. Sorry, I'm going to look it up. No, take your time. So the jury found that Toyota was liable for the death, and the only theory presented in the case was that the computer killed people. And the jury agreed. Toyota settled that case immediately and began to settle all 500 other cases in the queue.

Rip the Band-Aid off here. Pay everybody, let's not drag this thing out and put it through the public. I can't speculate as to the motives, but I think the facts speak for themselves. God, I mean, this is... Yeah, and like one quick, crazy side note, this ruling actually led to the freedom of a Minnesota man named Kuafong Lee

who had already spent two and a half years in prison, charged with vehicular homicide for killing three people in the Toyota he was driving when it suddenly accelerated at a control. So this ruling got this guy at a prison. But he was in prison for two and a half years for this, you know, this acceleration thing. That's nuts. I know. Wow. When said like in your opinion and your experience, do you think this specific case and other cases were in fact caused by just this crazy cosmic chain of events that led to.

a tiny little bit of electricity grazing a circuit board? As amazing as it sounds, these type of events are almost certainly caused by that. Now, was that the primary contribution to the Toyota UA? I don't know, and probably no one will ever know. I feel like the way you're painting the world almost just looks like it. They're just smoking wrecks everywhere, like crashed planes and cars into light poles and the internet's not working. How are we functioning?

I don't even understand. So as soon as we found out, this is the problem. We went back and we started fixing the problem. Once we know that autopilot is the one that's causing the problem, then we fixed it. How did you fix it? So you get three copies of everything. The same electronic system, there are three of them, and at least two of them have to agree to do anything.

Oh, smart. So today, like, tripling and doubling redundancy is the norm in planes and cars and like any other life-threatening technology. And actually, the same thing was done in Scarbeck after that incident in 2003. So now when they have an election, each vote is actually counted twice by the computer. And if one of those tallies doesn't match the other, they know something's gone wrong immediately. So that's cool. But here's the thing. This doubling and tripling is just too expensive to do with most things. And as our electronics get smaller and smaller and smaller.

These bitflips are just going to happen more and more. Why would that be? Well, the smaller, the transistor, the less energy you need to flip that bit. Now we are at a point where you only need about thousand electrons to represent a one or a zero. Wait, how many electrons did you need back in the 70s? Oh, probably million. Oh, and now you need a thousand. Wow, okay. And transistors made today are like tiny, tiny.

Your hair is 50 micron in diameter. So take your hair and slice it thousand times. So that gets you down to 50 nanometer. And we are looking at 7 nanometers. So you also have to pretty much slice it down seven more times. And what you are left with is the size of transistors that are coming in the market in the next two years. Will they be in cell phones and laptops and things like that? They will be everywhere.

So as we get better and better and better at miniaturizing our technology, we are more and more and more at risk to these tiny flying projectiles. Right. Through our technological advancement, we have invited this problem upon ourselves. That is one way to look at it. If you go back to those big cell phones, which are weighing 5KG each and about 4,000, long, then you won't hear this problem. Thanks, Annie, Simon?

Yeah. Actually, we have one more thing for you. Sure. Well, we've been doing some research into this, and we've talked to a bunch of electrical engineers, and this is going to sound very, very strange. But what these researchers are saying a likely cause was, was a cosmic ray hitting one of the computers

could cause the driver to lose the ability to control the gas pedal. Oh, my Lord. Well, I am not a scientific person. Anything's possible. Now, you're saying we're bombarded by cosmic rays all the time. If that's so, perhaps. Maybe we just got zapped.

I don't know. Do you think Grover would be? I'm just curious to hear his reaction to the possibility that your car except. accelerated because a cosmic ray hit it. What does he think about that? You said an engineer came up with that there? A bunch of electrical engineers.

Very smart people. The electrical engineers have come up with a theory. You know, we're bombarded with cosmic rays all the time. With cosmic rays all the time. And it's possible that a cosmic ray, if it hits just at the right time and the right place, it can zap on board computers and, possibly zap the accelerator control. He wants to know how you feel about that.

Somewhat open-minded, but at this point in time, I, you know, I question that, but it's not that big a deal. Okay. Well, it's home and quite wine time. Well, we couldn't agree more. And so what we've decided to do is provide you with a small entertainment. for you to sip your white wine with. Simon has made a video to accompany this episode

that must be experienced. It is at RadioLab.org. Watch it. Very short and very weird. Yeah. This piece was produced and reported by Simon Adler and Annie McEwen. Special thanks to Lucas de Raker

and to Vedrik de Booker and to Francesca Constanzo and Kelly Gallagher, Natalie Higgins, and the Hailstone Ice Company in Crown Heights, Brooklyn. Okay, you want to go now? Yeah, I think so. I think I'm going to go now.

Well, it's not going forever, just for the moment. Yeah, I feel reluctant to leave the room for everything. I felt that in your voice. I'm Chad Abumrod. And I'm just going to stay here forever. Okay, well, yeah, he's Robert. And we'll see you in the next one. It's so much fun, so much fun.

My name is Tim, and I'm calling from Cleveland, Ohio. Radio Lab was created by Jad Abumrod and is produced by Soren Wheeler. Dylan Keefe is our director of sound design. Susie Lechtenberg is our executive producer. Our staff includes Simon Adler, Becca Bressler, Rachel Cusick, David Gebel, Bethel Hapta Hapta, Tracy Hepta, Nora Keller, Matt Kielder, Robert Krollwich, Annie McEwan, Latif Nasser, Melissa O'Donnell, Sarah Kari, Ariana Wack, Pad Walters, and Molly Webster. With help from Shima Oliai, Audria Quinn, and Neil

Denesha. Our fact checker is Michelle Harris.