Everything Everywhere Daily: History, Science, Geography & More - Survivorship Bias

Episode Date: September 27, 2021

During World War II, the US Army assigned statistician Abraham Wald the task of statistically figuring out where extra armor should be added to American bombers. After analyzing the evidence and sharing it with the Army, he recommended the exact opposite of what the Army assumed. The reason was that the Army had engaged in a logical fallacy. Learn more about survivorship bias and how it manifests itself into everyday thinking, on this episode of Everything Everywhere Daily. Learn more about your ad choices. Visit megaphone.fm/adchoices

During World War II, the U.S. Army assigned statistician Abraham Vauled to the task of statistically figuring out where extra armor should be added to American bombers. After analyzing the evidence and sharing it with the Army, he recommended the exact opposite of what the Army assumed. The reason was that the Army had engaged in a logical fallacy. Learn more about survivorship bias and how it manifests itself into everyday thinking on this episode of Everything Everywhere Daily. Do you ever climb into bed ready to sleep, only to have your mind start racing the most moment your head hits the pillow? Thoughts bouncing around, replaying the day or jumping ahead to tomorrow? That is exactly why Catherine Nikolai created Nothing Much Happens. Each episode is a

gentle, cozy bedtime story where, well, nothing much happens. No drama, no tension, nothing you need to follow closely. Just soft narration, calming repetition, and soothing sensory details designed to help your mind slow down and your body relax. It's not about entertainment, it's about rest. And millions of listeners around the world use it every night to quiet their thoughts and finally fall asleep. If you've ever struggled to shut your brain off at night, this might be exactly what you've been missing. You can listen to Nothing Much Happens wherever you get your podcasts.

Episodes are every Monday and Thursday. This episode is sponsored by Scotty Vest. Autumn is here, and that means cooler weather and jackets. If you're looking for a jacket this season, I highly recommend checking out Scotty Vest. They have pockets to hold all of the gear that you carry with you. I'll often go to a cafe to write. and work, and I can take everything I need in the pockets of my Scotty vest jackets. It can easily hold my MacBook Air, charging cables, an iPhone, and my AirPods in its pockets, and no one is any

the wiser that I have anything on me. In addition to jackets, they also have a great selection of shirts, pants, and dresses, all of which have extra pockets. You can get 15% off all ScottyVest products by going to ScottyVest.com and using coupon code, Everything Everywhere, all one word at checkout. Once again, that's Scottyvest.com, coupon code, everything everywhere. Let's start this discussion with the definition of survivor bias. The simple definition is that it's considered a logical fallacy or statistical error or a cognitive shortcut, where you don't look at an entire population when making an inference because some part of the population didn't meet certain criteria.

The classic example which illustrates the point is the example that I mentioned in the introduction. In World War II, the Army Air Corps wanted to provide additional armor to their bombers that were flying missions over Germany. Bomber crews had the highest fatality rate of any type of service group of the Allies during World War II. In an attempt to reduce their losses, the Americans wanted to put more armor on their bombers. The problem was, armor is heavy, and the more armor a bomber had, the fewer bombs they could drop. The solution was to just provide armor to the most vulnerable parts of an aircraft. The army began a study to analyze where the planes were being hit. When a plane returned back to base after a mission, all of the bullet holes would

be documented and then they would be tallied. A pattern soon emerged. When all the bullet holes were mapped out, there were clear clusters of them on the tips of the wings, the back part of the fuselage, and the tail. The army determined that those spots on a plane with the most bullet holes were where the armor should go. Before they went ahead, they let us see. statistician by the name of Abraham Wald, who worked at Columbia University. He saw the same data and came to the exact opposite conclusion as the Army. He noticed that there were no bullet holes recorded on the engines or the cockpit. He realized that they were only recording the bullet holes on the planes that made it back to base. What they weren't counting were all of the planes that were

shot down. Bullet holes on the tips of the wing or on the tail didn't indicate that those places should get extra armor. What it meant is that you could get shot in those places and still make it back to base. The army was only getting data from the planes which survived, not all of the planes which actually flew. This became known as survivor bias. Survivor bias can be found all around us, and it can result in coming to the wrong conclusion about things because you aren't looking at all the data. There are several examples of survivor bias that creep up in business. One is trying to use success stories to guide your decisions. Bill Gates, Steve Jobs, and Mark Zuckerberg were all college dropouts that went on to become billionaires. Many people have used

their examples to justify dropping out of college or never attending college at all. Now, it is true that these men did drop out of college. However, just focusing on these exceptions ignores the thousands of people who dropped out but didn't achieve the same level of success. In fact, if you look at the entire population of people who have dropped out, overall they do more poorly than those who stay in school to completion. The Dow Jones Industrial Average is claimed to have positive returns most years, and that if you invest in the companies which make up the Dow Jones, you'll most probably see decent returns over time.

What this ignores is that the companies that actually make up the Dow Jones Industrial average change every few years. If you remember back to my episode on the companies that made up the original Dow Jones, if you bought stock in them when the index was created, many of them, like the United States leather company, went bankrupt. These underperforming companies are removed from the Dow Jones before they can ever bring the index down and are replaced with more successful companies. The same is true with the S&P 500 and other indexes.

In the 1982 bestselling book In Search of Excellence, Tom Peters and Robert Waterman analyzed 40 companies to distill eight rules of what makes a successful company. Since the book was written, of the 35 publicly traded stocks, most of them have underperformed the market. When creating their rules of success, the book ignored all of the companies which have failed. As a result, the rules of success turned out to not actually bring about success. Survivor bias can find its way into scientific studies as well. In one classic study conducted in the 1930s, a researcher by the name of Joseph Banks-Rine

claimed to have discovered extrasensory perception or ESP in some people. His study had people try to guess the order of cards that appeared on the other side of a screen. He tested a large number of people and then eliminated those to perform poorly, indicating that they didn't have ESP. He eventually kept narrowing down the group until he had a group that he was quite sure exhibited ESP. What really happened is that he started with a very large number of people, and then some of them kept getting lucky. If you had a million people flipping coins, the odds are that one of them would toss 20 heads in a row. However, finding that one person doesn't mean that they have any special powers. This same technique can be found in the famous investment newsletter scam.

The way this highly fraudulent scam works is really simple. Send a free investment newsletter to 10,000 people. Tell half of them a particular stock will go up and the other half it will go down. Once you have the result, send out a similar letter to the 5,000 people with the newsletter who got the correct prediction. Do this six times and you'll eventually have a list of about 150 people. Those 150 people will then think that you've correctly picked stocks six times in a row without error. You can then ask them to invest their money with you because they think that you are an investing genius. One of the most stunning implications of survivor bias might be the conclusion reached by Stanford medical researcher John Ionitis,

who concluded that 85% of published scientific research findings might be wrong. Studies that find a negative result are almost never published. They aren't sexy and they don't generate funding. or attention. Most published studies usually can't be replicated by other researchers. His conclusion is that there is an abnormally high number of false positive results being published. Much of this, but not all of it, is due to the fact that negative results not showing something is true are usually ignored. Survivor bias also comes into play with how we see the past. Music from certain periods often seem better the further away we get from them. The reason is that we only hear good music,

and most of the bad music has been forgotten. Music from the 18th and 19th centuries is dominated by a few composers like Mozart and Beethoven. However, there were a lot of composers back then. We never hear their music because it isn't played anymore because they weren't as good. The classic example of this can be seen in the movie Amadeus, where Salieri lives long enough to see his music become forgotten. Whenever someone talks about a golden age of anything, it's usually because they're only focusing on the good stuff which survived,

and they're ignoring all the bad stuff which was eliminated or forgotten. Sometimes you can have a sort of reverse survivor bias. Instead of ignoring part of a population, you can start counting it and come to another wrong conclusion. World War I was the first major conflict that saw the widespread use of helmets. One of the things which medical staff noticed right away was that the number of head wound cases actually increased when people started wearing helmets. Most of these injuries were due to shrapnel. This initially led many top military officials to consider getting rid of the helmets. However, it was eventually pointed out that the increase in head wounds was due to men not dying from head wounds,

which would otherwise have happened if they weren't wearing a helmet. Dead soldiers don't go to the medic and don't get counted in head wound statistics. In this case, the increase in injuries, which is normally a bad thing, was actually a good thing. They only knew it was a good thing by looking at the entire population of soldiers, not just the ones who went to the hospital. I've experienced people engaging in survivor bias many times. Many people say that the only way to be successful in podcasting is to have started years ago. They look at the most successful podcasts, and indeed they started years ago.

However, they're only seeing the podcast that survived, not the podcast that started at the same time, but are no longer around. Survivor bias is something that you should always be on the lookout for, even if it's in your own thinking. because if you don't, you might end up putting your armor in the places that don't need it. The associate producers of Everything Everywhere Daily are Peter Bennett and Thor Thompson. If you'd like to support the show, please join the list of patrons over at patreon.com. And also remember, if you leave a review or send me a question, you two can have it read on the show.