Hidden Brain - Guessing Games

Episode Date: June 27, 2017

Pundits and prognosticators make predictions all the time: about everything from elections, to sports, to global affairs. This week on Hidden Brain, we explore why they're often wrong, and how we can all do it better.

This is Hidden Brain, I'm Shankar Vedanta. We're surrounded by people who tell us they know what's going to happen in the future. A lot of people have no idea that Trump is headed for a historic defeat. Bear Sturz is fine! Don't move your money from there! That's just being silly! These predictions have a few things in common. The commentators have complete confidence in themselves. We as the audience love to hear them make a complicated world seem simple. And finally, no one ever pays a serious price for being wrong.

Donald Trump wins the presidency. The country is win the Super Bowl! Brady has his ship! Trump wins the presidency. Their stirs in the bargain bin sold to rival JP Morgan Chase for just two dollars a share. On today's show, our fascination with making predictions and why we may need a revolution in the way we make them. If you play the game the way it really should be played, the forecasting game, you're much more subject to embarrassment. We look at how some people actually are better than others at predicting what's going to happen in the future. Ironically, these aren't the people you usually find on television

blow-vading about what's going to happen next week. They're ordinary people who happen to know a very important secret. Predicting the future isn't about being unusually smart or especially knowledgeable. It's about understanding the pitfalls and the way we think and practicing better habits of mind. Phil Tetlock is a psychologist at the University of Pennsylvania. Over several decades, he's shown that the predictions of so-called experts are often no better than what he calls, dart throwing chimpanzees. After spending years criticizing forecasts and forecasters, Phil decided to look at people

who were very good at predicting the future. In his book, Super Forecasting, The Art and Science of Prediction, Phil explores how we can learn from these people to become better forecasters ourselves. Phil, welcome to Hidden Brain. So Phil, lots of people watch television at night, and millions of people feel like throwing things at their television set each evening as they listen to pundits and prognosticators, explain the day's news, and predict what's going to happen next. Of all the people in the country, you probably have more cause than most to hurl your coffee cup at the television set because starting in 1984, you conducted a study that analyzed

the predictions of experts in various fields. What did you find? Well, we found that pundits didn't know as much about the future as they thought they did. But it might be useful before we start throwing things at the poor pundits on the TV screen to consider their predicament. They're under pressure to say something interesting. So they resort to interesting linguistic gambits. They say things like, well, I think there's a distinct possibility that Putin's next move will be on a stonia. Now, that's a wonderful phrase, distinct possibility.

It's wonderfully elastic because if Putin does move into a stonia, they can say, hey, I told you, there was a distinct possibility it was going to do that. And if he doesn't, they can say, hey, I told you, there was a distinct possibility. It was going to do that. And if he doesn't, they can say, hey, I just said it was possible. So they're very well positioned. Now, if you play the game the way it really should be played, the forecasting game, and you use actual probability.

So you play it the way Nate Silver plays it. And you wind up with, say, 70% probability that Hillary will win the election a few days before the election in November 16. You're much more subject to embarrassment. If he said there's a distinct possibility that Hillary will win, he could have been very safely covered. Because when you ask people to translate distinct possibility of the numbers, it means

anything from about 20% to about 80%. I want to talk about how and why people often arrive at the wrong forecast. In your book, you tell the story of a doctor in 1956 who went in to see a dermatologist because he saw spots on the back of his hand. Tell me the story of Archie Cochran. Well, Archie Cochran was one of the early pioneers of evidence-based medicine. He was a great believer in running experiments, randomized control trials, and he was very skeptical of his fellow physicians. He was a great believer in running experiments, randomized control trials, and he was very skeptical

of his fellow physicians. He thought that they were systematically overconfident in much the same way that my data suggests that political pundits are systematically overconfident. So he was a skeptic by nature. Yet when he did get this cancer diagnosis from an eminent specialist. He didn't wait for the biopsy results to come in.

He just allowed them to go right into surgery immediately before the biopsy, which was a shame because it turned out that he did not have emoligrancy. We use that as an example of the bait and switch heuristic. So Archie Cochran didn't know whether he had cancer, but he was looking in the eye of a very renowned specialist who looked exactly like the sort of person who would know. So he took a really hard question, do I have cancer? And he substituted a somewhat easier question.

And that is, does this guy I'm talking to right here look like the sort of person who would know if I have cancer? And the answer to that was a resounding yes. So he took the answer to the easier question. He assumed it applied to the harder question and he forged ahead and got massive surgery that proved to be unnecessary when the biopsy results came in.

Archie Cochran didn't make that mistake again. He went on to have a very distinguished career where he developed a series of techniques to help doctors guard against the pitfalls of following their intuitions. He said, look, you might be very smart, you might know a lot, but there's something you have to do

before you act on your intuitions. This did not make him popular with his fellow doctors. Right. And Cochran now is honored, but I think during his lifetime, a lot of his colleagues, Tom, is a pain in the butt. He would run experiments that would repeatedly show that the intuitions of his colleagues were off base. There was a debate, for example, among cardiologists about how long heart attack patients should stay in the hospital after the heart attack, and whether the mortality would be higher if they went home or if they stayed in the hospital. And the conventional

wisdom among cardiologists was that it was better for them to stay in the hospital. But Cochran said, look, we have to test our intuitions. People would be happier, it would be less expensive, and so forth, if people were resting at home. And they thought that would be a terribly unethical experiment to run, because it could kill people, right? If it showed that the people getting the home rest were actually dying at a higher rate. But Cochrane persisted.

He wasn't a guy to take no for an answer, and they ran the experiment. And he also had a mischievous streak. So he presented the results to those two colleagues, and he flipped them around. He showed them a chart that indicated that the heart attack patients who went home were dying at a higher rate, just as the conventional wisdom predicted. And the doctors were outraged.

They said, Archie, we told you you're a fool. Look what you've done. You've caused great harm. You've got to stop this experiment immediately. And then he paused and he mischievously smiled. caused great harm, you've got to stop this experiment immediately. And then he paused and he misdivously smiled and he said, oh, I got the data turned around. It turns out that the people who are going home are living longer. The truth is, making predictions is difficult, but many biases also get in the way of making

accurate forecasts. When we make a prediction and it turns out wrong, most of us don't remember that we predicted something different. In fact, the hindsight bias prompts us to believe we'd gotten it right all along. We also hail people who make predictions that turn out right, whether in this talk market or politics or sports, but that keeps us from seeing the role of luck. Many people who get the right answer are just lucky, and some people who get's wrong are just unlucky.

Over time, the laws of probability mean luck can only take you so far. One reliable way to check if someone's success at predictions is driven by skill or by luck, is to have them make lots of predictions and see how they turn out over time. A few years ago, the federal government launched such an experiment. They conducted a forecasting tournament where thousands of people logged into computers to make thousands of predictions.

As time passed, the forecast could be checked for accuracy. It was a remarkable thing for the professional career government officials to do, to sponsor a project that had the potential to be embarrassing. We were one of five academic research teams that were competing to pull together the best methods of making probability estimates of events that national security professionals cared about. What kind of questions were they asking? All over the map quite literally. security professionals cared about. What kind of questions were they asking?

All over the map, quite literally. So there would be questions about violent clashes in the east or south China Sea. There would be questions about the Syrian civil war, about Russian-Ukrainian relations, about the Iranian nuclear program, Columbia Narco-Trafikers, literally all over the map. So you found, at some time, that people who were not necessarily trained experts in a specific domain were actually able to perform as well, or maybe even better than people who were the experts? Yeah, that's essentially right.

David Ignatius broke the story, I think, in 2013 in the Washington Post about how the best forecasters in the Good Judgment Project are outperforming professional intelligence analysts who had access to classified information, and we're working on an internal prediction tournament. How could this be? Are these people smarter than the rest of us? More knowledgeable? When we come back, I'll talk to Phil about what makes a super forecaster a super forecaster. Stay with us. If you were asked to pick someone to answer a difficult question about the

economy or foreign affairs, you might turn to an Oxford-educated public intellectual who writes a column for a very important newspaper. You probably wouldn't turn to a retiree in Nebraska who spends his time bird watching. But Filtaht Lock says, maybe you should. He was the opposite of Tom Friedman in the Superforecasting book. Tom Friedman, of course, being an eminent New York Times columnist, a very, very elegant writer, best-selling author, well-known for his explanations. But nobody has any idea how good a forecastry is.

And Bill Flack is an anonymous, retired irrigation specialist working in Nebraska, working out of the public library, or out of his home, and doing a fabulous job, making probability estimates in the good judgment project in the intelligence community forecasting tournament. So you helped discover Bill Flatt. Was he part of your team that you entered in this forecasting tournament? He was indeed. He was one of the very best. The very best forecasters in the tournament were called out each year.

The top 2% and we put them into elite teams called super forecasting teams. And we let them work together. And they did just a fabulous job. Super forecasters like Bill Flack turn out to have some things in common. Tell me about the kinds of philosophies they have and the kinds of thinking styles that you seem to find in common among many of these super-forkasters. I would say the most distinctive attribute of the super-forkasters

is their curiosity and their willingness to give the idea a try. And when I say the idea, I mean the idea that forecasting is And when I say the idea, I mean the idea that forecasting is a skill that can be cultivated and is worth cultivating. Because it doesn't matter how intelligent you are, or how knowledgeable you are, if you believe that it's

essentially impossible to get better at these kinds of tasks, you're never going to try, and it's never going to happen. It's as simple as that. So a big necessary condition for moving forward is having the attitude that this is a skill that can be cultivated and is worth cultivating. SuperForkasters tend to gather information and update their beliefs in a very particular way. Filthat Lock points to Aaron Brown, the chief risk officer of the hedge fund AQR. Before he was a big shot in finance, he was a big shot in the world of poker. He was a world-class poker player. And we quote him as saying that you can tell the difference between a world-class poker player

and a talented amateur. Because the world-class player knows the difference between a 60-40 bet and a 40-60 bet. In any pauses and said, oh, maybe like 55, 45, 45, 55. Distinguishing more degrees of maybe is an important skill. Why is that? Well, the very best forecasters are well calibrated. So when they say events are 80% likely, those events happen about 80% of the time.

When they say things are 90% likely, they happen about 90% of the time. So it makes a difference how frequently you update your forecasts. If you don't update your forecasts reasonably frequently, you're going to fall out of phase with events. And that means often making adjustments that are relatively small. You suggest that forecasters should do something that doesn't seem to be very intuitive. Instead of looking at the particulars of an individual case, you say forecaster should zoom out and figure out how often something has happened historically. So Daniel Conhamen is probably one of the greatest psychologists of the last hundred years,

and he calls that the outside view. And he says, people rarely take the outside view when they do forecasting. They normally start from the inside and they work out, but there's a big advantage to you as a forecaster from starting with the outside view and working in. Take another example, let's say you're out of wedding and you're sitting next to somebody who has the bad taste to ask you how likely to think it is this couple is going to stay married? And you look at the person, it's bad taste and all that, you see how happy the couple is and you can see it's a joyous occasion, you say, I can't imagine these people who are so happy together getting divorced,

I think maybe a 5% chance they're going to get divorced. Now, if you ask that question of a super forecaster, they'd say, well, what's the, let's look at the sociodemographics of the couple and let's see, what's the base rate of divorce within this socio-demographic group? Let's say it's 35 or 40 percent over the next 10 years. Okay, I think it's about a 40 percent chance to get divorced in the next 10 years. Now, that's not the end of the forecasting process. That's just the beginning. The real value, though, of doing it this way, of starting from the outside and working in, is it puts you in the ballpark of plausibility right away. 40% is a much more plausible number than 5%. Now then you could start adjusting the 40%. So if you discover things about the couple that suggest they really are

deeply bonded to each other and they've known each other a long time and they really understand each other and they've done great things for each other, you're going to lower your probability. If you discover that the husband is a sociopathic philanderer. You're going to lower your probability, if you discovered that the husband is a sociopathic philanthropist, you're going to raise the probability. Those are inside view sorts of pieces of data that would cause you to adjust. Or you might just see that having a small fight and say, well, OK, I'm going to move from 40 to 41%.

And that's one of the interesting things of a super forecast is they do a lot of updating in response to relatively small events. And most of the news, one of the interesting things of a super forecast is they do a lot of updating and response to relatively small events. And most of the news, most of the time, is what statisticians would call low diagnosticity news. It doesn't change things dramatically, but it does change things a little bit. And appreciating how the news gradually builds up toward one conclusion or another is a very valuable skill.

I'm wondering if one reason many of us start with the inside view rather than the outside view, is that just at an emotional level, that's how our minds think, that you see a couple, and you put yourself in the shoes of that couple, and you try and imagine what's happening in their lives, and we think in stories, and we imagine what life must be like for that couple, and we're trying to see

how that story will turn out, and we're trying to follow the narrative where it leads rather than do this much more, you know, abstract remote process of saying, let me start with the rough estimate of how often something happens. There's something about that in some ways that requires us to step outside this narrative frame that we often use to understand the world. I couldn't put it better. I think that's right. We're quite readily seduced by stories. Another example might be, it comes from research and this is again related to Daniel Coniman and his work on the conjunction fallacy. It's another source of error and forecasting. Let's say I ask you, how likely is it that in the next 10 years there'll be a flood in North America that

kills more than 1,000 people and ask you to make an estimate on that. Let's say I ask another person to make the estimate. How likely is it that there'll be a flood in California that will be caused by an earthquake cracking a dam leading to a massive outflow of water? Now, when I put the two questions together like that, it's pretty obvious that a flood

anywhere in North America, due to any cause, has got to be more likely than a flood in California caused by an earthquake cracking a dam. The California event is obviously a subset of the more general North American flood thing. But people don't see it that way. The California earthquake dam story is more like a story. It's like a story. You can actually put it together in a more meaningful way, whereas a flood anywhere in North America is kind of abstract and vague.

So people can transport themselves into the world, and you can imagine it's like a movie, like a Hollywood movie playing out. And they can see it happening. Yes, I can see that happening. And in fact, I can see it as you're speaking right now, Phil. And that pumps up the probability and that screws up your forecasting track record. So again, think of forecasting as a skill that can be improved with practice. When you're making a prediction, start with the base rate, the outside in view. Beware of the risks

of storytelling. Finally, amateurs make three kinds of predictions. Yes, no, and maybe. Professionals have many gradations of maybe, and the attached specific probability estimates to their predictions, allowing them to go back and learn where they went wrong. But even if you do all these things, I asked Phil how you can be really sure that predictions that turn out correct are because of good technique. You know, there's an old trick that they play in business schools to talk about the role of luck where they divide the class into pairs and they say, you know, have a coin toss between each person and then the winner of each of those coin tosses competes against another winner

and after 12 rounds there's one person who has who has declared the winner and of course that person in a business sense might seem to be extraordinarily good but really all that's happened is they've happened to win 12 coin tosses in a row they've just been very lucky. How do you distinguish between people who are lucky and people who are actually very good? Well that is indeed the $64,000 question and it comes up in finance too. I mean there are some finance professors out there who would argue that the really famous super investors were in Buffett or Ray Dalio and people like that. Aren't some sense like coins that come up heads 20 or 30 or 40 times in a row.

We have a lot of people competing in financial markets, and they're making a lot of predictions over long stretches of time. So you're going to expect some streaks. And when we get really streaky performance, we declare we found a genius. So the skeptics would say, well, Phil Tetlock is doing essentially the same thing here. He's anointing people who are essentially lucky.

So we built in a lot of statistical checks, but you can never be 100% sure. One of the other critiques of Phil's work is that the kinds of questions that super forecasters are answering are not really the questions that people want answered. Most of us are interested in the big questions. Who's going to win the current standoff

between the United States and Russia? Super forecasters tend to answer much more narrow questions. Is Russia going to invade Ukraine in the next 18 months? I think that's a very fair criticism of the first generation of forecasting tournaments that we put all of our effort into improving forecasting accuracy. Now, I'm not going to say the questions that people were trying to answer were trivial because they were by no means trivial. As a way, no means trivial, whether Russia was going to invade the Ukraine or the Syrian civil

war was going to drag out as horribly long as it has and so forth. I mean, the questions were really important things. But were they about the things that matter the most from a public policy perspective? Could we have made the questions more relevant to deep policy questions? I think the answer is yes. Science proceeds incrementally. You do one thing at a time. So what these first generation of forecasting tournaments generated is that there is a skill that is worth cultivating and can be cultivated.

Okay, that's step one. Now I think the next generation of tournaments should be focusing on exactly what you're saying here. I think we should be focusing as much on the insightfulness of the questions as the accuracy of the answers. I'm wondering if some questions just are too hard to forecast. We were having this conversation in 1917 rather than 2017 and someone would tell you, you know, 40% of the country right now is in farming and in 100 years from now, 2% of the country would be farming.

It would be reasonable to conclude that a good chunk of the country would be out of work that our unemployment rate would be 35%-40%. And that would be a perfectly reasonable conclusion to draw. And of course, they just would be no way to fully understand how that would not have come to pass. And I'm wondering in some ways, is the point of super forecasting to break questions down so that they're so simple, so that they're answering questions with accuracy, but not necessarily questions that are of importance. old joke of the guy who loses a coin on the street and he searches for the coin under the street lamp because that's where it's brightest, not where he actually dropped the coin. Is that what we're doing with super forecasting?

Is it possible that we are asking the kinds of questions that are the easiest to answer, the easiest to which we can attach these probability estimates? But the moment you go a little bit further out, we essentially are just guessing. Well, you've raised a host of interesting questions, and I would be the first to agree with you that forecasting tournaments can degenerate into games of trivial pursuit. And I'd also agree very much with your premise

that forecasting accuracy falls virtually to chance, to the dark throwing chimpanzee, when you try to forecast more than about 10 years out on geopolitical and geoeconomic questions of consequence. So both of those things are true. There's a limit on the forecasting horizon. And it's true that forecasting tournaments, if you don't pick the questions carefully,

so that they add up to something important, they can become trivial. So both are true. I don't think that they mean we shouldn't be doing forecasting tournaments. I think that we should be doing forecasting tournaments as well. I'm going to ask you one final question. And this is also I think a potential critique of super forecasting, but it comes in the form of a forecast that I'm going to make. The reason I think many of us make forecasts or look to prognosticators and pundits to make forecasts is that it gives us a feeling like we have a handle on the future.

It gives us a sense of reassurance. And this is why liberals like to watch the pundits on MSNBC and conservatives like to watch the pundits on Fox. A more cautious style that sort of says, the chance that Donald Trump is going to be impeached is 11.3% or the chance that you're going to die from cancer is 65.3%. These estimates run up against a very powerful psychological impulse we have for certainty that we actually want someone to hold our hand and tell us you're not going to die. We don't want a probability estimate.

We want actually an assurance that things are going to turn out the way we hope. So here's my last question for you. If someone advises people to do something that runs against their emotional need for well-being and reassurance, I'm going to forecast that that advice, however well-intention, however accurate, is likely not going to be followed by most people. What do you make of my forecast field? Well, I think there's a lot of truth to what you say. I think people, when people think about the future, they have a lot of goals, and they want to affirm their loyalty to their ideological tribe.

They want to feel good about their past commitments. They want to reinforce their preconceptions. So those are all social and psychological goals that people have when they do forecasting. And forecasting tournaments are very unusual worlds. We create a world in which only one thing matters. It's pure accuracy. So at somewhat analogous to the sort of world

is created in financial markets, or London bookies, or Las Vegas bookies, all that matters is the accuracy of the odds created in financial markets or London bookies or Las Vegas bookies. All that matters is the accuracy of the odds. I would say this. I would say people would be better off if they were more honest with themselves about the functions that their beliefs serve. Do I believe this because it helps me get along with my friends or my boss, helps me fit in, helps me feel good about myself or do I believe this because it really is the best synthesis of the best available evidence. If you're playing

on a forecasting tournament it's only the latter thing that matters, but you're right when people sit down in their living room and they're watching their favorite pundits, they're cheering for their team. It's a different kind of psychology. They're playing a different kind of game. So all I'm saying is you're better off if you're honest with yourself about what game you're playing. Psychologist Phil Tatlock is the author of Superforecasting, The Art and Science of Prediction. Phil, thank you for joining me today on Hidden Brain.

My pleasure. MUSIC This week's episode was produced by Raina Cohen, and edited by Tara Boyle. Our team includes Jenny Schmidt, Maggie Penman, René Clarre and Parth Shah. Our unsung hero this week is Portia Robinson-Migas from our marketing team. Portia helped us brainstorm many ideas in the early days of hidden brain and has remained a steadfast friend. Thanks Portia. You can find us on Facebook, Twitter, and Instagram

and listen for my stories each week on your local public radio station. I'm Shankar Vedantam, and this is NPR. you