Everything Everywhere Daily: History, Science, Geography & More - How to Lie With Statistics

Starting point is 00:00:00 Mark Twain once said, there are three kinds of lies, lies, and statistics. The reason why he placed statistics into its own category is because it's possible to use numbers to misrepresent the truth, distort reality, or outright lie. However, if you know what to look for, you can catch misuses of statistics, and if you really pay attention, you can find these misuses almost everywhere. Learn more about how you can lie with statistics on this episode of Everything Everywhere daily. What if your perceptions about the past were wrong? Throughline is a podcast that takes you back in time to uncover the parts of the story that may have gone unnoticed. It effectively turned day and tonight. And how it shaped the world now. Time travel with us every week on the ThruLine podcast from NPR.

Starting point is 00:01:05 In the past, I've done episodes on subjects such as logical fallacies, correlation versus causation, survivorship bias. This episode is in the same ballpark insofar as it deals with critical thinking, but it's a bit different in that it's more nuanced. There's some overlap with the previous episodes I've done, but for the most part, it's different. The reason why it's different is that lying with statistics isn't necessarily lying, which I suppose makes the title of this episode a lie. It certainly can be lying, but more often than not, it might just be putting a positive spin. on something or interpreting data in such a way that it supports a previous hypothesis. The goal of this episode is to give you an idea of some of the things to look for when you hear

Starting point is 00:01:55 the results of studies, surveys, or polling data. This is important because if you pay attention to the news, you will hear statistics reported all the time. And there are big problems with a lot of statistics you see reported in the media. Sometimes it could be an honest mistake, and other times it might actually be a case of fraud. So let's start with the most extreme case, that of fraud. Statistics is ultimately about collecting data and then trying to interpret that data in a meaningful way. Whether it's a poll, survey, econometric data, or nutritional research, ultimately there has to be trust in the researcher than that they honestly collected the data. However, there have been cases, far too many cases, of outright,

Starting point is 00:02:42 research fraud. In 2023 alone, over 10,000 research papers were retracted from various academic journals. Not all of those retractions were fraud, but many of them were. The Wiley Corporation, a publisher of academic journals, had to shutter 19 different journals because the retractions were starting to get out of hand. In January of this year, the Dana-Farber Cancer Center, an affiliate of the Harvard Medical School, had to retract six studies and change the data on another 31. The papers were published by some of the most senior and distinguished researchers at the Institute. I don't want to belabor the point of research fraud because that really isn't the focus of this episode, but you should be aware that it is a thing. The vast majority of published

Starting point is 00:03:29 research is not fraudulent, but it does exist. If the data is bad, then it doesn't matter how good your statistical analysis of it is because you will still get a fraudulent result. So let's assume that the data wasn't completely made up, which is the case the vast majority of the time. Even then, there are a whole bunch of other things that can go wrong. And let's start with P hacking, also known as data dredging. If you have a data set, most researchers are looking for a strong correlation between two things, and this is known as a P value. A P value is sort of the statistical probability that an observed result is true.

Starting point is 00:04:10 Usually a P value of 0.05 is what a researcher wants, which means that there's a 95% probability that the results are correct. And I should note, there's nothing magical about 0.05. It's just a number that's been traditionally used in research. There are some very good arguments that a more stringent p value, maybe 0.01 should be used, but that's for another episode. P hacking is when you find a correlation first and then make up a hypothesis to fit the data. If you have enough data, then it isn't hard to find something that will correlate with 95% confidence. There is an XKCD comic that makes a joke about this. A scientist claims that jelly beans cause acne. After finding no correlation, they check 20 individual colors of jelly beans before finding one of them

Starting point is 00:05:01 with a p-value of 0.05 and publishing the results that green jelly beans cause acne. Something similar to P-hacking is cherry-picking data. Unlike P-hacking, where you go through the data to find a hypothesis to fit the data, with cherry-picking data, you just ignore data that doesn't fit your hypothesis. This could be considered a form of data fraud insofar as you're committing the crime of omission rather than of commission. Let's suppose you have a hypothesis. You collect your data and everything looks great. Except for the fact that you have some outlier data that messes everything up. You could conveniently just leave that data out to make your data set look better. Cherry-picking data happens all the time.

Starting point is 00:05:48 If a politician wants to show that crime is increasing or decreasing, you just have to pick the right crime to look at. One crime might be going up, while others might be going down. You could also pick the right time frame to illustrate a point. By picking the right start and endpoints, you could make a stock, for example, look bad or good. You could say that a stock has gone up 5% over the last month, which is good, and totally ignore the fact that it's actually down 80% for the entire year. Another statistical problem that creeps up quite often is having a sample size that's too small. You may have seen the TV commercials that say, four or five dentists recommend sugarless gum for their patients who chew gum. Well, how many dentists did they ask? If

Starting point is 00:06:34 they literally only asked five dentists, that's a very small sample size. This is always a big problem when polling for elections. People with similar political opinions tend to clump together. They may congregate in the same city, same occupation, same religion, or social circles. If you take too small of a sample size, you run a high risk of only sampling people from a particular group. One area where you see small sample sizes is always in presidential elections. Every presidential cycle, you'll see people who are trying to show trends in presidential elections. The problem is that presidential elections don't happen that often. There are only 20 of them that take place each century.

Starting point is 00:07:18 Going back even 10 presidential elections, and much of the electorate wasn't even born. So trying to glean trends from such a small sample size is difficult to impossible to do. Certain types of medical research can't avoid the problem of small sample size. sizes. Certain rare diseases only affect a small number of people, making studying the disease difficult. And in such cases, there isn't much you can do about it other than not use certain statistical techniques. One of the most common misuse of statistics is misleading percentages and proportions. And this happens quite frequently when talking about risks. Let's assume that we're talking about something that has a risk of causing cancer. And just for the sake of argument, let's assume that the risk is real.

Starting point is 00:08:02 One thing you might hear is that something increases the risk of cancer. Knowing that something increases the risk of cancer actually doesn't tell you very much. Leaving the house increases the risk of getting hit by a car, but people do it all the time. We do it because although the risk is very real, it's also very low. Knowing that something increases your risk of cancer doesn't mean much without knowing what the absolute risk is. Let's say that there's a type of cancer that someone has a one in 100,000 risk of contracting, and you do something X that increases that risk to two in 100,000. That can be described in three different ways.

Starting point is 00:08:47 You can say that doing X increases your risk of cancer, and that would be true, but it doesn't give you the magnitude of the increase or the size of the risk. You could also say that doing X increases your risk of cancer by one. 100%. And that sounds a lot more dramatic and many people confuse a percentage increase in risk with absolute risk. Going from 1 in 100,000 to 2 in 100,000 is indeed a 100% increase, but many people confuse it as a 100% risk. And the other way you could describe it is to say that doing X will increase your risk of cancer by 1 in 100,000. And that's also true, but it sounds much less ominous than a 100% increase. In all three of these examples, the way I described it was

Starting point is 00:09:37 technically true. But how you would describe it would depend on whether you're trying to play down or magnify the risk. Another big problem with any sort of data collection has to do with leading questions. Many people who make up surveys know that you can get the results you want by just asking the right question. Let's say you're doing a customer satisfaction survey and you want to show that people were satisfied with your product or service. The question you would want to ask is, how satisfied were you with our service? Logically, you could phrase the question as, how dissatisfied were you with our service? But that would be phrasing the question as a negative instead of a positive. The first question assumes that you were satisfied, and the question

Starting point is 00:10:24 is only how much. This is very important when it comes to election or polling issues. One to look for in any poll is to see if they provide the question that was asked to the people that were answering the survey. Most major presidential polls provide this information, as well as providing data on whether the interviewees intend to vote or is a registered voter. Nutritional surveys often run into problems with data collection and questions. In these surveys, the problem isn't with leading questions per se. The problem is asking questions that people either can't remember or feel they have to provide the correct answer for. Large-scale nutritional studies will give people food surveys that ask them what they've eaten over the last six months to a year.

Starting point is 00:11:08 Most people can't remember everything that they've eaten over that long of a period of time. The number of cookies, pork chops, and apples that they've consumed is at best a guess. The problem is that people will often give the answer that they think they should give. They will over-report the number of apples, for example, and under-report the number of cookies because they want to appear to be eating right. Conducting statistical surveys or doing polling is a very difficult thing to do. And I've only covered some of the more obvious problems

Starting point is 00:11:39 with statistics which are easy to observe. There are actually many more technical issues that can come up in statistics that can only be detected with full access to datasets and the techniques used. And full datasets are often not made available. Whenever I hear research shows or something to that effect in the news,

Starting point is 00:11:58 I always take it with a grain of salt, at least initially. It might be true, but one survey or poll is usually not enough to establish anything. It's only through replication of the results, which often never happens, that you can really determine if a statistical finding is in fact true. The executive producer of Everything Everywhere Daily is Charles Daniel. The associate producers are Benji Long and Cameron Kiever. I want to give a big shout out to everyone who supports the show over on Patreon. including the show's producers.

Starting point is 00:12:33 Your support helps me put out a show every single day. And also, Patreon is currently the only place where Everything Everywhere Daily merchandise is available to the top tier of supporters. If you'd like to talk to other listeners of the show and members of the completionist club, you can join the Everything Everywhere Daily Facebook group or Discord server. Links to everything are in the show notes.

Everything Everywhere Daily: History, Science, Geography & More - How to Lie With Statistics

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.