99% Invisible - 170- Children of the Magenta (Automation Paradox, pt. 1)

Episode Date: June 24, 2015

On the evening of May 31, 2009, 216 passengers, three pilots, and nine flight attendants boarded an Airbus 330 in Rio de Janeiro. This flight, Air France 447, was headed across to Paris. Everything proceeded normally for several hours. Then, with no … Continue reading →

This is 99% Invisible. I'm Roman Mars. On the evening of May 31st, 2009, 216 passengers, three pilots, and nine flight attendants boarded an air-france flight in Rio de Janeiro, Brazil. The flight was headed across the Atlantic Ocean to Paris. A warning. This story might not be a good one to listen to if you're sitting on a plane right now. That's producer Katie Mingle letting you know that the closest exit could be behind you. The plane that took off that evening from Rio was an Airbus 330.

The takeoff was unremarkable. The plane reached a cruising altitude of 35,000 feet. The passengers read and watched movies and slept. As they flew over the Atlantic Ocean passing the equator, there was a lot of thunderstorm activity quite typical, nothing really unusual about that. That's William Longavicia. He wrote an article for Vanity Fair about this flight and has written extensively about aviation. He also used to be a pilot. So I spent many, many thousands of hours in cockpits.

Which means he's particularly well-suited to tell the story of Air France flight 447. It was a long flight. So they had two co-pilots, known as first officers and one captain. They would fly the plane in shifts. Two of them flying, well one captain. They would fly the plane in shifts, two of them flying while one slept. We know that the flight proceeded normally for several hours. And then? Then, with no communication to the ground or air traffic control, it suddenly disappeared.

Some pieces of the plane and several bodies were found days later floating in the Atlantic, but it would be two more years before most of the wreckage was found deep in the ocean. All 228 people on board had died. When the wreckage was found, they also recovered the cockpit voice recorder and the flight data recorders. And they were in good shape. The recordings were able to tell a story about how flight 447 ended up in the bottom of the Atlantic Ocean. And the story they told was about what happened when the automated system flying the plane suddenly shut off,

and the pilots were left surprised, confused, and ultimately unable to fly their own plane. Automation in flight has been around for a long time. The first so-called autopilot was invented by the Sparry Corporation in 1912, and it allowed the plane to fly straight and level without the pilot's intervention. In the 1950s, the autopilot's got better, so that you could program an autopilot to follow a route not just to keep the wings level. This was making the pilots job a lot easier. But there was still a lot to think about in the cockpit of an airplane.

Controlling the electric systems, controlling the hydraulic systems. And because of all this, most planes had three people in the cockpit, two pilots and an engineer. Flight engineers sat behind the pilots and they manipulated and managed these increasingly complex systems. But by the 1970s, all of those complex systems were also automated and flight engineers lost their necessity and their jobs.

By this time the jet engine had replaced the piston engine, making planes much more reliable. And studies were showing that most accidents were being caused not by mechanical error, but by human error. In a French company called Airbus, thought that they could make planes safer by letting automation carry an even bigger part of the burden of flying. Airbus was a leader. Airbus has always been quite radical in its design philosophy. Led by a guy named Bernard Ziegler, Airbus set out to design what they hoped would be the safest plane yet.

A plane that even the worst pilots could fly with ease. Bernard Ziegler, he famously said that he was building an airplane that his calls Sierge could fly. Ziegler's plane not only had autopilot, it also had what's called a fly-by-wire system. So this can get a bit technical, but basically an autopilot just does what a pilot tells it to do. Fly-by-wire is a computer-based control system that can interpret what the pilot wants to do and then do it smoothly and safely. So if the pilot pulls

back on his or her stick, the fly-by-wire system will say, oh, I see that you want to pitch up and here, I'll do it for you at just the right angle and rate. Importantly, the fly-by-wire system will also protect you from getting into an aerodynamic stall. So in a car, a stall happens when the engine stops turning, but in a plane, stalling is different. Basically getting so slow that the wing's no longer function correctly and the airplane loses altitude, put it that way. A stall in a plane can happen when the nose of the plane is pitched up at two steep and

angle. A steep angle can cause the plane to lose lift and start to descend. Stalling in a plane is not good. Stoffer long enough and you will crash. But fly-by-wire automation makes it impossible to do as long as it's on. Unlike autopilot, the fly-by-wire system

cannot be turned on and off by the pilot. You can't turn this automation off. But here's the thing. It can turn itself off. And that's exactly what it did on May 3, 2009, as flight 447 flew through the night sky over the Atlantic Ocean. At some point, having flown normally for a few hours, the captain had gone back to sleep. This meant the two co-pilots were in control of the plane.

They encountered some weather. It wasn't rough. They never hit significant turbulence. So far nothing out of the ordinary. And then suddenly, they lost airspeed indications. A pressure probe on the outside of the plane had iced over. The automation could no longer tell how fast the plane was going. At this point, autopilot disconnected.

At the same time, the fly-by-wire system degraded by one step. So the autopilot turned off and the fly-by-wire system shifted into a different mode, a mode that did not protect against aerodynamic stall. None of this was ideal, but it also wasn't a reason to panic, according to William. The plane was still basically flying straight and steady. The airplane was not upset. Had they done nothing, they would have done exactly what they needed to do, nothing.

But that's not what happened. The copilot in the right seat put his hand on his control stick, a little joystick-like thing to his right. And he pulled it back. Not just a little. He pulled it three-quarters of the way back. Why he did that is a major question. Perhaps he was startled or reacting to some turbulence.

It's important to note that he did not tell the other pilot in the cockpit, the guy in the left seat, what he had just done. If he had, maybe his co-pilot would have known the dangerous situation they were about to be in. In an air bus, the two pilots have separate control sticks that move independently of each other. In other commercial airplanes, the two pilots have separate control sticks that move independently of each other. In other commercial airplanes, the controls would move in unison so that you would actually feel the moves your co-pilot was making on their controls.

This design variation may also have contributed to what was about to become a very bad situation. In any case, as the pilot in the right seat pulled back on his stick, the airplane pulled its nose up and started to soar upward. When stall protections are in place, the plane won't just warn you not to pitch it at such an extreme angle, it physically will not let you do it. But remember, the plane had shifted into a mode in which it wasn't offering that underlying protection.

As the angle got steeper and steeper, the air couldn't flow smoothly across the wings and the plane began to stall. The plane was angled up, trying to gain altitude, but it actually began to lose altitude. A warning began to sound. The plane couldn't prevent a stall, but it could still warn that a stall was happening. It said stall, stall, stall.

It would have sounded like this. And you'd think that if you heard this alarm, someone would say, hey, I think we might be stalling, but this never happened. At one point, one of the copolytes did say, get your nose down. But he didn't say we've stalled, and the man in the right seat who was flying the airplane to junior copilot, he would put the nose down a couple of times and bring it right back up again. The copilot began to ring frantically for the captain to come into the cockpit, at one

point one of them says, where is he? The airplane was shaking, all kinds of alarms were going off. One minute and 38 seconds after the episode started, the captain came into the cockpit and asked, what's going on here? The copilot did not say we're in a stall. Instead one of them said, quote, we've completely lost control of the airplane and we don't understand anything. We've tried everything. The captain had walked into a difficult scene to understand. Everyone was confused, were they descending or climbing, at times they couldn't seem to figure it out.

Finally, they seemed to realize they're in a fast descent, losing thousands of feet per minute. But why? They couldn't figure out why. The captain never assessed correctly what was going on. He was obviously trying to, he wasn't there to drink coffee or suck Islam, but he was never able to figure out what was going on, even though it should have been amply obvious what was going on.

But for whatever reason, it wasn't obvious to them what was going on. They didn't know they were in a stall. If they had realized it, William says the fix would have been clear. It's basic. It's pilot school 101. The recovery would have required them to put the nose down. Power doesn't matter, just get the nose down, get it below the horizon, regain a flying speed, and then pull out of the ensuing dive. The plane continued falling down at a jaw dropping pace, losing several

thousands of feet per minute. But since they started at 37,000 feet, for a while there was still time for a recovery. So for quite a long time, a normal crew would have been able to recover, and at the very end at around 16,000 feet, the best crew in the world only probably would have been able to recover. And below that altitude, no crew would have been able to recover because they simply didn't have the space beneath them to execute the dive before hitting the water. But in any case, they never tried to do it.

And so they rode this airplane down, expressing confusion the whole time, and finally expressing certainty that they were going to die. Four minutes and twenty seconds after the incident started. They pancaked into the water at a very high descent rate, and of course it killed everyone instantly. 6 years after the crash of Flight 447, what's clear is that the pilots didn't understand what was happening to them. The big question is how. How could they have a computer yelling stall at them and not understand they were in a stall.

There were various things that contributed to the crash of 447, but automation, which has overwhelmingly made airline travel safer, also played a role in this accident. For however much automation has helped the airline passenger by increasing safety. It has had some negative consequences. In this case, it's quite clear that these pilots had had experience stripped away from them for years. William actually did the math on this. The captain of the Air France flight had logged 346 hours of flying over the past six months, but of that time, there were only about four hours in which he was actually in control of an airplane.

Takeoffs and landings. The rest of the time, autopilot was flying the plane. That's four hours in six months. The co-pilot would have had even less time at the controls. This is not to say pilots do nothing when autopilot is on. They still have an important role to play. What do good pilots do in flight? They watch the systems.

They handle navigation and communication. They occasionally program things in, flight path changes. They deal with anticipation of whether they think about fuel management. Increasingly, William says, pilots have become automation managers, with fewer and fewer chances to actually fly the plane. He believes this lack of experience at the controls left the pilots of 447 unprepared to take over when the automation turned off.

The pilots were hideously incompetent. That's, I think, no one would disagree with that. So I have certainly heard that position from other people before. I think we all have to be very careful. None of us was there when this actually happened. That's Nadine Sardar, a systems engineer at the University of Michigan who's studied a lot of aviation accidents related to automation. My approach to things is different, I guess. These pilots were in that airplane with the passengers and I'm sure they had every motivation

to keep that flight safe. And yet they didn't. So then I tried to understand, well, why did they not? Did they not have the right information? Did they not have the proper training? When one of the co-pilots hauled back on a stick, he pitched the plane into an angle that eventually caused the stall. But as possible, he didn't understand that he was now flying in a different mode, one

that would not regulate and smooth out his movements. This confusion about how the fly-by-wire system responds in different modes is referred to, aptly, as mode confusion. And it's come up in other accidents. The plane switches into a different mode and suddenly pilots find it difficult to understand what's going on at that point. They do not exactly know whether the system is still taking care of the airplane or whether they are in charge now. They don't necessarily understand whether all of it has disconnected

or just parts of it have disconnected. A lot of what is happening is hidden from view from the pilots. It's buried. Or the airplane starts doing something that is unexpected. And the pilot says, hey, what now what? And now what's it doing? That's a very, very standard comment in Cockpits today. What's it up to now? The pilots of Air France Flight 447 never quite say

what's it doing now, but they say similar things. William wasn't the only person to point out that what's it doing now is a commonly heard question in the cockpit. It seems to be almost a running joke in the industry. A joke that everyone agrees is a serious problem. What's the most often asked question in our cockpits? What's it doing now?

That's American Airlines Captain Warren Vanderberg addressing a group of pilots in 1997. People have been talking about this issue way before the crash of Flight 447. It's just that no one's really figured out how to solve it yet. But you see, we have become what I call children of the magenta. The children of the magenta are too dependent on the guiding magenta colored lines on their screens. You know, we think we have to have those magenta lines

on the map and that magenta V bar that's steering us toward that line. The children of the magenta are too dependent on automation in general. And if this is you I'm talking about, do not be defensive. In the industry we created you like this. Vanderberg recommends in certain situations to turn off the autopilot and fly the plane yourself. When you deem the situation is right, to practice your skills, turn off your autopilot and your auto throttles, fly your planes,

maintain your skills. Nadine Sardar agrees that this could help. She also recommends more and better training, using simulators to let pilots explore different kinds of emergency scenarios. And then of course, there are people who think the problem can be solved with more and better automation, but this is a paradox of automation. Automation accommodates incompetence, in fact it's designed to do that. The Airbus was designed after all, for Bernard Ziegler's concierge to be able to fly.

We appear to be locked into a cycle in which automation begets the erosion of skills or the lack of skills in the first place. And this then begets more automation. Nadine Sardar thinks it's a mistake to throw more automation at the problem. The position I would adopt is one that has been termed a human-centered automation, where we see what you need to do instead is make the automation that you have, make it A, maybe smarter, but importantly make it what we call a team player. And part of that is to provide better feedback. Part of it is have the automation communicate more effectively with the pilot.

No one I talk to is advocating going back to the battle days before automation. So automation in general, the fly by wire systems, all of those certainly do increase safety I would argue. And even though a lot of the accidents we see these days have to do with the interaction between pilots and automation, William says you have to remember. One of the reasons for that is that because of automation, there are hardly any other accidents. And as far as how we should feel about air France flight 447 and other accidents involving automation, should they be seen as an acceptable loss in an industry that's become mostly safer through automation?

I think as a society, the answer to that is yes. For the individuals, there's no such thing as an acceptable loss. These are terrible trash. Terrible. And for the pilots who died as well. Airbus planes are not any less safe or any more safe than any other plane on the market. Plains have accidents. But it should be said that they're pretty safe too. The accident rate for air travel is very low.

About 2.8 accidents for every 1 million departures. And there are plenty of pilots out there that know how to take control in an emergency. So breathe deep everyone, just relax. It's going to be okay. It's true, there are lots of good pilots out there. But in a far off future, we might be flying in pilotless planes. We're a long way from that, but the technology is definitely coming and we will, I mean, this will happen and it just makes a lot of sense. I don't think anyone is sort of loudly saying great, that's just going to be a wonderful thing. But I think reasonable people are also not

tearing their hair out about it. Because by the time it happens, the automation will be so good and so reliable that humans, with all of our human emotions and human fallibility, the children of the magenta, will really just be in the way. 99% invisible was produced this week by Katie Mingle with Sam Green span Avery Truffleman and me Roman Mars. We relied heavily on the reporting and expertise of William Lungavisha for this piece, so special thanks to him, he went above and beyond for us, and if you want an even more in-depth minute by minute analysis of the crash of Flight 4447, check out his piece in Vanity Fair. It's called the Human Factor.

And on the next episode of 99% of Asphalt, we're going to take this idea of the Automation Paradox and use it as a lens to view the next great design challenge in automation. When we all potentially become the children of the magenta in our self-driving cars, it's our first two-part series, so stay tuned. Now, last week I mentioned that we have a new addition to the radiotopia collective, and I'm pleased to announce that Nate Dimaio's The Memory Palace is joining the gang. When I was first thinking about starting 99% of his role, Nate was one of the first people I called because I really love what he was doing with his show featuring these poignant historical

moments and I wanted to do something in the same spirit with stories about design. And so more than any other program I can say, if you love 99% of visible, you will love love the memory balance. Nate and I are friends and he once recommended a history book to me, and on page 325 I noticed a couple of sentences, just a couple of sentences, that were a tiny but really striking digression from the main narrative, and I'd like to think that I would have noticed the charm of this historical footnote on my own, but the real reason it stood out to me was because I knew it sparked an episode of the memory palace. Most memory palace pieces start on page 325 of a simbook.

That voice you just heard is Nate D'Ameo. The Memory Palace has a special way of zeroing in on these moments in history that light up your imagination and hit you in the gut or make you smile. I had noticed that there were these historical things that would jump out from long Ken Burns documentaries or the one moment in some historic home tour

when that object on the mantel piece actually jumps out and says, wow, that's amazing. Look at that beautiful thing. And so often, I noticed that the things that moved me were these focused things. They were recognizing the single object. They were the incredible twist in someone's story. And I wanted to find out on some level if there was a way to cut to the chase.

If there was a way to focus so tightly, it was almost just an aesthetic artistic challenge. Like, is there a way that I can replicate this feeling of being moved or having my head blown off or having my heart broken? This thing that I have once felt is there a way that I can get people to feel that same thing. Like, can I retell the story in such a way that they don't have to take the whole historical home tour, or they don't have to sit through all nine and a half hours of this documentary series? Is there a way to kind of crystallize it and turn this sweeping historical story into a pop song. But the memory palace is never just a simple historical anecdote.

There's always something more going on. Like there is a subject and then there is meaning. Like there are themes which is different than plot. At some level I'm not merely trying to tell a historical story, I'm trying to conjure it, trying to have it sort of like live in the listener's mind for a bit, and then melt away. I really like his show so much that at a hard time, selecting which episode to present.

So I asked Nate which episode really stood out to him as the one that did all the things he was hoping to achieve in the memory palace, and he picked this episode. Enjoy. This is the memory palace. I'm Nate Demetre. The world loved a world's fair.

And the exposition universel in Paris in 1889, just killed. The exhibits in the grounds were unparalleled and impeccable. And at the center of it all was this audacious steel structure that managed to be imposing and elegant and the tallest thing on earth and unmistakably French all at once. Paris had the Eiffel Tower and the men planning the next fair, the Colombian exposition in Chicago, just a couple of years away, needed something that good, something Eiffel Tower good. And that wasn't easy to find.

The proposals ran from the ridiculous, like a structure that would soar more than a thousand feet above the land of Lincoln made entirely out of stack logs, in top with a replica of Abe's boyhood at home, to the extra ridiculous. Something so tall that visitors would take an elevator to the top of a slot, that they'd ride down until it dropped them off in New York or San Francisco. The Ferris organizers were panicking. They demanded that America's designers and engineers step up. In a man named George Ferris, step forward.

When his Ferris wheel was completed that summer. It rose 264 feet above the ground, which was a lot shorter than the Eiffel Tower. But whatever, that thing didn't even move. The idea that something's so massive, but that looks so fragile, like a bicycle wheel whose spokes look too thin to keep a bike up, was thrilling and pretty terrifying. Despite what the engineer said,

despite the math and laws of physics, despite the many assurances of the fairs organizers, there were people who were sure that this ludicrous machine was going to be a disaster. The people were going to die. Not only was there no way that that flimsy thing could stand up to a prairie wind or a gale off the lake. Even if it did, the prospect of tumbling through the air in a cage 20 stories above Chicago was full on craziness. Who in their right mind was going to want to ride that thing? Hundreds of thousands of people did. Despite the fact that during its first test run, hundreds of bolts in loose parts rained

down on spectators below. Despite real stories of panic riders trying to escape through the windows when they realized exactly how high 200 feet was. Despite apocryphal stories of suicides and severed limbs, this ferris wheel, this thing that is basically a kitty ride today, was a bigger thrill ride than any quintuple loop open car reverse twist rocket coaster they might have at six flags. The papers, even some in France, said it was the marvel of the age, better than the Eiffel Tower.

That almost mundane sensation we have now of looking down from above, of moving through space, up and out, and down and back around. No one had ever felt those things before, and of course now we can't really feel them again. We've gone around too many times, and we've looped too many loops. But back in 1893, you could pay your 50 cents and climb into a car right after sunset during the golden hour. And experience something entirely new. You could rise up above the world's fair, where down below Americans were eating hamburgers for the first time.

We're Buffalo Bill and Frederick Douglass and Mark Twain were sightseeing. We're entire villages from Egypt and Algeria had been brought and reconstructed. We're people who had ragged time for the first time. Sohuli dancers and belly dancers. And you could come back to Earth and walk out and be among the first people in human history to walk

around at night. We light on. But fairs end, they shut down, they pack up and leave, and the Ferris wheel did too. They moved it up to a different park in the north side of the city, and after a while the novelty was gone. And the Ferris wheel became just a Ferris wheel and they tore it down. A salvage company bought it for 8 grand and blew it up with dynamite and sold the pieces for scrap. That's the memory palace from Nate to Mayo, now from radio toopia. After a couple years of not putting the show out very regularly, TV shows, books, and

stuff, I'm pleased to say he's releasing weekly episodes through the summer, so get on board. You're going to want to set aside five minutes or so to listen to it. The moment each new episode comes out, that's what I'm going to be doing. You can find the show and like the show on Facebook. You get little hints of what's coming up on the show if you pay attention to Facebook. We're also on Twitter, Instagram, Tumblr, and Spotify too. But if you want to learn all the things there is to know about 99% of visible, the best

place to do that is at 99pi.org Radio Tapio From PRX