Stuff You Should Know - How Election Polling Works and Doesn't Work

Episode Date: September 17, 2020

Election polling had a pretty good rep until 2016. But it turns out they weren't far off even then. It's really the media driving the narrative. Learn all about how election polling works today. Learn more about your ad-choices at https://www.iheartpodcastnetwork.comSee omnystudio.com/listener for privacy information.

On the podcast, Hey Dude, the 90s called, David Lasher and Christine Taylor, stars of the cult classic show, Hey Dude, bring you back to the days of slip dresses and choker necklaces. We're gonna use Hey Dude as our jumping off point, but we are going to unpack and dive back into the decade of the 90s.

We lived it, and now we're calling on all of our friends to come back and relive it. Listen to Hey Dude, the 90s called on the iHeart radio app, Apple Podcasts, or wherever you get your podcasts. Hey, I'm Lance Bass, host of the new iHeart podcast, Frosted Tips with Lance Bass. Do you ever think to yourself, what advice would Lance Bass

and my favorite boy bands give me in this situation? If you do, you've come to the right place because I'm here to help. And a different hot, sexy teen crush boy bander each week to guide you through life. Tell everybody, ya everybody, about my new podcast and make sure to listen so we'll never, ever have to say. Bye, bye, bye.

Listen to Frosted Tips with Lance Bass on the iHeart radio app, Apple Podcasts, or wherever you listen to podcasts. Welcome to Stuff You Should Know, a production of iHeart radios, How Stuff Works. Hey, and welcome to the podcast. I'm Josh Clark, and there's Charles W. Chuck Bryant over there, and we've got the scoop.

Jerry's around here somewhere. And this is Stuff You Should Know, off to a great start. She's in her office. She is, she's got this remote thing going on. Yeah. It's like the COVID special. That's right. And this was, this has been one I've been wanting to do

since 2016, and it seems like the fire kind of went down on it, and now it's the fire is back up again in election season. I thought, no better time than to talk election polling and this weird sort of black magic, which is really not black magic at all. And as we'll see, the polling wasn't even really that off in 2016.

No, it was great. There was a furious, we'll talk about it in a second, but there was a furious reaction by the media just left polling and pollsters out to dry, saying like, you, you're terrible, your whole craft is useless. You lie to us. As the pollsters went back after election night on 2016,

which by the way, was a bit of a surprise to everybody involved. I think so, including the president. Yeah, when the pollsters went back and looked at their stuff, they said, wait a minute, no, this is, this is all fine. It was you guys media, you screwed up. You don't know what polling is or what it does. Or how to talk about it, most importantly.

Yeah, and then you public, you have no idea what's going on. You just see some percentages and you automatically leap to some conclusions and this is way off. So it's in part that the media was misrepresenting it. Some polls weren't very good. And then the public in general just needs to be a bit more educated on statistics to understand what they're hearing. And that's what we're here for.

Cause I took statistics three times in college, the same course. At Georgia? At Georgia before. I took one of those classes. I hated it. Intro to statistics, right? Yep, boy.

I hated that class. The professor finally, I walked up to her on the last day of the third time was like, please. And she bumped my D up to a C and I was, I never looked back. She'd say, you have a one in four chance. And you're like, but what does that mean? Right. What is four?

But so if I can understand this after doing some research, then anybody can understand at least the gist of it enough to understand polling and not be taken in by bad representation of what poll results are. Yeah. So if you remember in 2016, there were pollsters saying, or I'm sorry, and I'm going to say that wrong over and over again, you had media saying that Hillary Clinton is going

to win in a landslide. She's got an 85% chance to win. Some said as high as 95. She's going to win the popular vote by 3 percentage points. All the battleground states in the Midwest, she's going to win those narrowly. And it did not work out that way. And like you said, there was a furor over how could everyone

be this wrong with the polling. And there's a man named Nate Silver, who everyone probably knows at this point, who has made his name as a data specialist and runs the 538 blog and said, you know what? Polling is flawed, and that's probably the first thing that everyone should understand, is all polling is a little bit flawed. State polling is definitely a little more flawed

than national polling. But here's the deal, everybody. These polls from 2016 were not only not so far off, but historically, dating back to since 1972, they actually performed a little better than a lot of elections. Yeah, and the state polling, while worse than average, wasn't that far off from the average error rate.

So what do you want? So there's a lot of stuff. Like we said, there was a lot of postmortem that was done on the 2016 polls and what was gotten wrong and what was gotten right. And we'll talk about that later. But the point is, is that overall, it wasn't that far off. And so the idea isn't that the polls fail

or that there's something inherently flawed with polling or that there's even something inherently wrong with the media. Like I want to go on record here, especially in this climate. The media is not our enemy. Like any healthy democracy needs a vital, robust, independent media, as free from bias as an objective to reality and good injustice as possible. But there's also such a thing as a 24-hour news cycle,

and you've got to fill that. And that's given the rise of opinion news and pundits and basically trying to capture as much market share as possible, which is definitely the wrong track for media in general. But I just want to go on record while we're going to be kind of beating the media up a little bit. That does not mean that the media is inherently flawed or evil

or seeks to kill you and your family and your family dog. So Silver goes back and a bunch of people go back and look at history and what went wrong here in 2016 as far as the polling goes. He says, we went back for the past 12 presidential cycles since 1972. And he said the polling error was 4.1. He said, in 2016, that national polling error was 3.1.

So technically, by a full point, it was a full point better. He said, we predicted that she would win the popular vote by 3 percentage points. She actually did win the popular vote by 2 percentage points. The state polls were the real difference maker. They actually did underperform at a 5.2 error rate. And that doesn't sound like that much. I think the overall error rate for state polls since 1972

is 4.8. So 4.8, 5.2 doesn't sound like much. But if you're talking about a percentage of error and just a handful of swing states that can make something look like a landslide, even though you lose a popular vote, that's exactly what happened. Right.

That's exactly what happened because you got to remember, Trump didn't win the popular vote. He won the electoral college and it came down to those swing states. But the fact that they were off just by 0.4 points from the average for the error rate goes to show you just how close that race actually was, which again is the opposite of how it was being broadcast

throughout the election. It was supposed to be a landslide like Hillary Clinton might as well just be taking measurements for curtains in the Oval Office right now. It was just that set. So it was presented one way when in reality, if you really looked at the polls and the polling results, if you looked at them with a sober face,

it was a much closer race than it appeared or than it was being broadcast. I haven't had a sober face since that night. So we should talk about the margin of error in polling. Anytime you see a poll, they talk about the margin of error. It's usually plus or minus 3 or 4. And that is on each side. So for each candidate's poll, in other words,

it could be a potential like 7 to 8 point swing and still be within that margin of error. So when Trump is winning states by a 0.2% margin or a 0.5 or a 0.7% margin, that's well, well, well, well within the margin of error. Right, right. So that margin of error, by the way, is just built in. We'll talk about it a little more in a little bit.

But it's like there's just no way around it. To get around any margin of error, you would have to literally go through and interview every single voter in America and then compile the evidence or their data perfectly without any miskeys or anything like that. And it's just impossible. So everyone accepts that any poll is

going to have a margin of error. But you want to keep it within plus or minus 3 points. Right. Maybe 4. Yeah, so a little history of polling. We've always been pretty spellbound by polls in this country. We put a lot of stock in polls, especially the presidential race.

The word straw poll, if you've ever heard that, that comes from the idea that you hold up a piece of straw to see which way the wind is blowing. So a straw poll is kind of like, here's how things stand today on something. This is the way the wind is blowing today on this matter. Yeah, and there's this kind of informal. They used to take them on train cars.

A journalist would ask people who they were on the train with, who they were going to vote for. Nothing formal or anything. But it does kind of reveal how longstanding our fascination with polls really is. Yeah, it got pretty serious in the 1930s, specifically the 1936 election, where a literary digest, it was a pretty big magazine at the time,

polled its subscribers. It's just kind of funny even seeing the Senate's. They predicted a landslide win for Republican Alph Landon over FDR. So if you've never heard of Alph Landon, you know why, because Alph Landon did not beat FDR. And the magazine's editor said, you know what? We didn't even think about the fact

that we just polled our subscribers and that they're wealthy people, or at least wealthier, on average, and they're probably going to vote Republicans. So Alph Landon was their man. Right, so if you go out even today and just interview Republicans and say, hey, who you're going to vote for, and then take that results and apply it to the entire population of the nation,

you've got a flawed poll. And that's what literary digest did. But in doing so, they established this kind of, they pointed out a real design flaw that now is just one of the first basic things that anybody conducting a poll gets rid of. That's right. Gallup came on the scene.

They galloped onto the scene. So sorry. And they were one of the first big polling companies to say, all right, we got to get this right. We got to get a representation of all of America here. So we're going to send our people door to door. We're going to go to every zip code in America. And they did that from 1935 to 1984

and got, basically, within about 3 percentage points, doing a pretty good job. But it was really expensive. So in the 80s, in the mid-80s, they switched to calling people on the telephone. Yeah, which I mean, that's still today. That is the gold standard is for a human being to dial up another human being and ask them some questions.

And we'll talk a little more about it. But what Gallup does and what Pew does and what a few others do is it's called randomized sampling or probability sampling, which is where you basically leave it to chance that any registered voter in America is going to get a phone call from you. So what Gallup is doing and what Pew does

is called randomized sampling or probability sampling, where any voter in America has an equal chance of receiving a phone call from Gallup or from Pew and being asked these questions. And it worked pretty well for a while when they moved from in person over to the phone because they were still asking people questions and they could still get their answers and harass them,

which is a big thing as we'll see about this type of sampling. The problem is when people started to use caller ID, they stopped picking up the phone as much. And so the response rate went down dramatically. Yeah, so they would call people using random digit dialing, which is a computer system where it fed in an area code and then the first three digits and then randomly dial the last four.

So you've got a pretty good start there on the random sampling, but even then, they said, you know what? Women tend to answer the phone more than men. So to truly randomize it, whenever whoever picks up the phone, we have to then follow up and say, we want to talk to the person in the house who's had the most recent birthday.

Further randomizing, I got kind of a laugh about this because I don't know that I've ever, literally ever seen my father pick up a telephone in his life or at least growing up for the first 18 years of my life. I don't think I ever saw him answer the phone. It's all ham radio, huh? Not, no, he went into that.

But just literally, not one time, he would just let it ring if no one was around, if my mom wasn't around to answer it. And granted, it was usually never for him. No one ever called to talk to him, but I picked up on that. And my friends used to get really frustrated back before texting that I would just never answer my phone.

And I always just thought it was an option. Like when the phone rings, it doesn't mean you're obligated. It just means now you have an option. You can answer it or not. Well, technically, that's true. I mean, like it depends. Nope, you don't have to answer the phone, but it depends on who in your life

could possibly be calling. Well, I didn't think it was rude or anything. I just thought it was literally like, you know, I'm gonna hedge my bets here that one of my friends isn't stuck on the side of the road. They can leave a message, and if they are, I'll go get them. So what you're talking about, Chuck,

is what's called a non-response. And that's factored into the response rate, which with phone polling from 1980 until the 1990s, it was manageable. I think the response rate peaked at 36% in 1997, which was good. Not bad. Now it's down to like 9% because like I said,

people have caller ID and if some unknown numbers calling, you typically don't answer. And that actually affects things because there is a certain kind of person who answers the phone no matter what. And they are not like every single American. And that actually factors into

the kind of poll you're conducting. Plus also you want like a certain amount of responses. I think out of a sample size, you want a minimum of 800 survey responses. And back in the day, when you got a 36% response rate, meaning 36% of those people you called would answer the phone and go through all of the questions

and answer them fully and complete the survey. Since it was down to 9%, you went from having to call between 2,000 and 2,500 people to up to 9,000 people now, just to get 800 surveys completed. And that made the whole thing a lot more expensive. On the one hand, because it was expensive, it meant that there were fewer and fewer companies

that could conduct these polls, which meant that the polls you were seeing were more and more legitimate. But on the other hand, it also usually decreased sample size a little bit because as Gallup pointed out, you can kind of fiddle with the numbers a little bit with a smaller response rate and smaller sample size. Yeah, and it also led to robo calls because of expense,

because of people not answering their phone as much. And those systems, I mean, I love how Dave Roos put it. He said they range from okay to terrible and how well they work, online polls and these other new techniques. But I think we should take a break and then talk about what I found the very interesting way that they further randomized this thing

from this point forward, right after this. ["The Star-Spangled Banner"] On the podcast, HeyDude the 90s called David Lasher and Christine Taylor, stars of the co-classic show HeyDude, bring you back to the days of slip dresses and choker necklaces. We're gonna use HeyDude as our jumping off point,

but we are going to unpack and dive back into the decade of the 90s. We lived it, and now we're calling on all of our friends to come back and relive it. It's a podcast packed with interviews, co-stars, friends and non-stop references to the best decade ever. Do you remember going to Blockbuster? Do you remember Nintendo 64?

Do you remember getting Frosted Tips? Was that a cereal? No, it was hair. Do you remember AOL Instant Messenger and the dial-up sound like poltergeist? So leave a code on your best friend's beeper, because you'll want to be there when the nostalgia starts flowing.

Each episode will rival the feeling of taking out the cartridge from your Game Boy, blowing on it and popping it back in as we take you back to the 90s. Listen to HeyDude the 90s called on the iHeart radio app, Apple Podcasts, or wherever you get your podcasts. Hey, I'm Lance Bass, host of the new iHeart podcast,

Frosted Tips with Lance Bass. The hardest thing can be knowing who to turn to when questions arise or times get tough, or you're at the end of the road. Ah, okay, I see what you're doing. Do you ever think to yourself, what advice would Lance Bass and my favorite boy bands give me in this situation?

If you do, you've come to the right place because I'm here to help. This, I promise you. Oh, God. Seriously, I swear. And you won't have to send an SOS because I'll be there for you. Oh, man.

And so, my husband, Michael. Um, hey, that's me. Yep, we know that, Michael. And a different hot, sexy teen crush boy bander each week to guide you through life, step by step. Oh, not another one. Kids, relationships, life in general can get messy. You may be thinking, this is the story of my life.

Just stop now. If so, tell everybody, yeah, everybody, about my new podcast and make sure to listen so we'll never, ever have to say bye, bye, bye. Listen to Frosted Tips with Lance Bass on the iHeart radio app, Apple Podcasts, or wherever you listen to podcasts. Learning stuff with Joshua and Charles,

stuff you should know. All right, so we've already talked about the fact that they've randomly called someone and then they take one further step on that call by saying, let me speak with whoever had the most recent birthday, even if it's, I guess, your three-year-old. Right.

And one other thing, I kind of made mention to it that I have to interject, dude. Like harassing people, like if you've been picked by this computer, if your phone number's been picked, they're going to keep calling you and calling you. And that is because as a person who doesn't normally participate in phone surveys, you are a specific kind of person

that you can't be left out of the population because you represent a large number of people and they want your opinion. So part of this phone standard of calling people is to call them over and over and over again to basically harass them into participating to get their answers for this survey because it's as important, if not more important sometimes,

than the people who are like, oh yeah, I'd love to answer this phone survey. Two totally different kinds of people. Yeah, absolutely. And I was totally kidding, by the way, to the listener when I said they will speak to a three-year-old. They asked the most recent birthday of someone of voting age, obviously.

Right. So then you've got a pretty decent random sampling to begin with. And then you have to start the process of waiting, which comes in a lot of different forms. If you want an example of a really good political poll, it's gonna be paid for by a neutral source. It's not gonna be a CNN poll or a Fox News poll

or a Super PAC or anything like that. You're gonna have a random sample of the public, which we just talked about. You're gonna be dialing cell phones and landlines these days. That's a big one. So they'll ask you if you have a cell phone and a landline, and if you say, yes, I have both,

they're going to adjust your response based on the fact that you had a higher chance of being selected because you have two numbers that the computer could have picked. Right, and another thing is, like you mentioned, they're gonna keep calling you. The best ones use live interviewers still. And then what they wanna do,

and this last one is really important, is they're gonna try and improve the accuracy of the results by waiting the response to match. What they wanna do is just match a real world demographic, age, race, your income level, your education level, and all of that stuff is factored in, and all this stuff is weighed out because, well, we'll talk about it,

but there are many different kinds of Americans, and if you want a really good sampling of different kinds of Americans, you're gonna, like you said, have to fidget with the numbers to make it a true representative population. Right, so because even if you just get it exactly right demographically and weighted, which like you said,

we'll talk about some more in a second, you still have that margin of error, and again, that's that 52% plus or minus three points, and that means that it could be 55% or it could be 49%. They don't know, but somewhere between that, most of your answers are going to be, the correct answer is somewhere in there. That's what that means with that margin of error,

and the reason that that's built in is because it is basically impossible to perfectly represent the larger population through random sampling. You're just not going to pick everybody correctly just by the fact that it's random and it's a sample. Yeah, and that's important because, like that's why you hear so much hay being made

over a double digit lead in a poll, which Biden had sort of semi recently. I know it's gotten a lot tighter since then, but when Biden was up by, I think, like 10 percentage points, people were flipping out because, like we said, it's plus or minus four for each candidate, so that's a total of eight, and so basically the press started screaming like,

he's outside of the margin of error, everybody. Like, nothing can beat him. Right, right, yeah. But now things are back within that margin. I saw on PBS NewsHour, they interview Mark Shields and David Brooks, Brooks is a New York Times columnist, and I think Mark Shields is an independent columnist, and one of them actually said, and this is in July,

America has clearly made up its mind on who's going to be the next president. And I was like, this is July. Did you not learn anything from 2016? Or I couldn't believe that those words came out of their mouth. And they said it's so matter of factly. Yeah, it's irresponsible, and there's been studies about this too

that have suggested that words like that, that polls that say 99% chance of winning, that this kind of stuff like actually has a negative impact on the leader because it makes people think, well, I don't need to go out and vote, everybody else is gonna go vote, and the turnout might be lower than otherwise.

There's also people who just, well, there's people who dispute that. They say, yes, it makes sense intuitively and anecdotally, but we've yet to actually see genuine data that says clearly that this has this effect, but it's something that's still being studied right now, whether it actually does or not. Well, and I also saw an article the other day

about the quote, unquote, silent majority, and that another reason those polls were so wrong back then, and they're saying are probably wrong now, is because there are, they say that there's a substantial block of voters who vary privately and secretly vote for Trump. Yeah, the term for them among pollsters is shy Trump voters.

Yeah, saw that. They won't admit that they're going to vote for Trump, but they're going to vote for Trump and that that affects polls. I saw that that's actually not been proven to actually exist, but I think it was a Pew, there's a really great Pew article.

If this stuff is speaking to you at all, go check out Pew's key things to know about election polling in the US, and it has a bunch of great links that you should follow in there. And there's also SiLine. They have surveys and polling, which is a guide for journalists to polling,

but I found out you don't actually have to be a journalist to read it online. So if you want to go check those out, they have some great, like just some breakdowns of some of the stuff we're talking about, but also about how to read polls and what to trust and look for in general. And then little known fact,

Pew was actually originally called Pew Pew until 1976 when Star Wars came out, and they were like, we got to change our name now, guys. Yeah. Can't do it. Man, it is. Dad O'Rama today with you, huh? So back to the waiting thing.

And by the way, we should mention that Gallup said if they wanted to increase that sample size and actually get the margin of error down to like plus or minus two, that they could do that, but that would be like a literal 100% increase in the cost. So like everyone just please live with plus or minus three or four points.

Yeah, and everybody generally does. And Dave uses this really good example. Dave Roos helped us out with this. And he said, this margin of error is best understood where if you selected 100 marbles, there's a jar of 500 red and 500 blue marbles, and you pick out 100 of them, you might pick out 50 of each one time.

Wait, 500? What? You said 500 marbles? Oh, no, I'm sorry, 1,000 marbles. I've lost my marbles. Is it 1,000? Yes, there's 1,000 marbles, okay? And 500 are red and 500 are blue.

Your task, Chuck, is to pick out 100. So you go to the trouble of picking out 150, red, 50, or blue, and I say do it again. And this time it's 47 and 53. And you keep saying again, again. Right, and I smack my riding crop on the desk that you're sitting at. And I do it 100 times.

And you get super turned on. Yeah, I do it 100 times because dear leader told me to. Right. And at the end, you get a little bell curve and basically a plus or minus four. Right, so yeah, almost all of them, this is what's a 95% confidence interval, almost all of them are going to fall in that bell curve.

There's going to be some outliers. There's going to be that one time where it was just absolutely insane you actually picked 100 red marbles randomly blindfolded from this jar. That's so insignificant statistically. It's just such an outlier. But almost all of them are going to be in there.

So when you're polling like this large group of people like American voters and 95% of them are falling within a couple of percentage points of either side of this middle, you can pretty much feel confident about that. And that is the basis of election polling, of political polling, of all polling really, that they have this built in margin that they know exists,

but everybody can live with it. The problem is, is when you're hovering around that 50% mark and you're talking about a two-party system. Yeah. One of them has like 51% and the other one has 49%, but there's a plus or minus of like two points. That means flip a coin, America. It means we have no idea.

And some people would say, well, why even do polling? Because what you're showing there is not who's going to win. That's not the point of polling. The point of polling is to take a snapshot of how America or whoever you're polling is feeling that moment about who to elect, about what laws to pass, about religion,

about the Cleveland Indians. It doesn't matter, right? Like the, that's what a poll does, but you can pervert polls into making them talk a different language and say, hey, look at this percentage. You take these polls and you convert them into something else. Now you have something like a 95% chance

this person is going to win. Go shout that, Wolf Blitzer. And Wolf Blitzer goes and shouts it as loud as he can. So we need to talk a little bit more about waiting. I mentioned earlier that there's other things they do to sort of tip the scale and that sounds like a bad term. So I guess I shouldn't say it that way, but things they do to make it equitable

and a true representative of the American population, for instance, African American voters make up 12% of voters. So if they did a poll and in the end, they only got 6% of respondents that were African American, then they just double it basically. If the respondents were overwhelmingly Caucasian, they would wait that down to their true representative number,

which is about I think 66%. Yeah, the electorate is white. And if 80% of white people respond, or 80% of the people that respond are white, then they're gonna kick that down. And again, this is just adjusting the poll to the proper weight. So you have a really legitimate snapshot.

And if it sounds crazy that they're using 1,000 people's responses and drawing that out to the size of the voting population of America, it is, but if you're a statistician, it isn't. You know what I mean? Yeah, it reliably works as long as you present it with plus or minus this margin of error. It sounds crazy as just an average Joe on the street.

To be like, they ask 1,000 people and we're supposed to know and extrapolate that. And a statistician who are number wonks and data wonks would say, yeah, that's exactly what that means. Shut up. That's really all you need. But it really is a testimony to the power of those statistics and that data

and the analysis of them. Yeah, weighting is really important. It goes far beyond just like age political party. I think Gallup uses eight different variables. The New York Times, the Inna College poll uses 10. And they include things like marital status and home ownership. Pew uses 12 variables.

They ask things like, do you have home internet access? Do you have volunteer or do you engage in volunteerism? And all of these things have been shown to be associated. So like if you're a white woman age 65 to 75 who volunteers twice a month. And lives in the suburbs. You're a very specific person where you, there's a group of people out there

who vote like a certain way and you represent like all those people with that. So they'll weight the results based on these additional questions that you're answering. They don't just ask you, are you going to vote for Trump or Biden? And there's also built into that question a really important point.

Are you going to vote? Yeah, that's it. That's a huge thing we haven't talked about. It's one thing to poll registered voters. But here in America, somehow presidential elections only get about 60% turnout still, which is just shameful and crazy. But that's another podcast.

But so most of the really, really good polls drill down and to get a real, real good representation of what might actually happen, they try to drill down to whether or not you're most likely to actually vote. Right. Because who cares what your opinion is if you're not going to vote.

And I mean, they generally take your word for it that you're telling the truth, you know? Yeah. But they do have like nine, I think Pew, yeah, Pew has nine questions that they basically use to establish that you are planning on voting. Like you're actually going to vote,

you're not full of hot air, you know? Yeah, I don't know what those questions are, but I imagine they have to do with, do you know where your polling place is? Do you have transportation stuff like that? I was thinking they were going to be like, are you really, really going to vote? Was like question three.

And they just kept adding release. Right. So you've got these people who've been called and they have answered these questions and they have participated in this survey, whether they wanted to or not. And they've finally done it. Built into that margin of error,

built into this poll is that understandable margin of error that just comes from the fact that it's a randomized sample, right? But what Pew and any other legitimate polling group will point out is that the margin of error is actually greater than that. That the margin of error for the average poll according to Pew is that it's something more

like six points, not three or four. It's actually six. And the reason why is built on top of that margin of error that's automatically part of the poll just by the virtue of it being a randomized sample are things like the person typing in the wrong key accidentally, those kind of things add up. Or that the question isn't worded clearly enough

that anybody who hears it knows the intent and knows what their answer is, that there's some sort of miscommunication involved. There's also things that they can't control for like people who have pseudo opinions who don't wanna sound dumb. So they just answer yes or no based on something they really don't care about either way.

And because they don't actually have an opinion that actually that weights things the wrong way. So when you add all these stuff, these things up, you have these additional errors that lead to like a bias overall in the poll which can affect the outcome. But again, the companies that have the money to conduct like these genuinely big gold standard polls

are they know enough to know how to kind of factor or control for those as much as possible. But still what Pew says is if you're listening to a poll and somebody says plus or minus three points, you should probably go ahead and double that in your mind. Double it in your mind. Double your pleasure, double your fun.

Double your margin error. So let's take a break and we're gonna come back and talk about what exactly they think went wrong with those state polls right after this. All right, welcome to the show. Brent, then in this episode, we're gonna introduce you to David Lashen. David Lashen is you're music and he's an excellent singer.

guilty of being in trouble and now nobodyères in anyway öz and we're gonna introduce thereafter back into the decade of the 90s. We lived it, and now we're calling on all of our friends to come back and relive it. It's a podcast packed with interviews, co-stars, friends, and non-stop references to the best decade ever.

Do you remember going to Blockbuster? Do you remember Nintendo 64? Do you remember getting Frosted Tips? Was that a cereal? No, it was hair. Do you remember AOL Instant Messenger and the dial-up sound like poltergeist? So leave a code on your best friend's beeper,

because you'll want to be there when the nostalgia starts flowing. Each episode will rival the feeling of taking out the cartridge from your Game Boy, blowing on it and popping it back in as we take you back to the 90s. Listen to, Hey Dude, the 90s, called on the iHeart radio app,

Apple Podcasts, or wherever you get your podcasts. Hey, I'm Lance Bass, host of the new iHeart podcast, Frosted Tips with Lance Bass. The hardest thing can be knowing who to turn to when questions arise or times get tough, or you're at the end of the road. Ah, okay, I see what you're doing. Do you ever think to yourself,

what advice would Lance Bass and my favorite boy bands give me in this situation? If you do, you've come to the right place, because I'm here to help. This, I promise you. Oh God. Seriously, I swear. And you won't have to send an SOS,

because I'll be there for you. Oh man. And so, my husband, Michael. Um, hey, that's me. Yep, we know that, Michael. And a different hot, sexy, teen crush boy bander each week to guide you through life, step by step. Oh, not another one.

Kids, relationships, life in general can get messy. You may be thinking, this is the story of my life. Oh, just stop now. If so, tell everybody, yeah, everybody, about my new podcast and make sure to listen, so we'll never, ever have to say bye, bye, bye. Listen to Frosted Tips with Lance Bass on the iHeart Radio app, Apple Podcasts,

or wherever you listen to podcasts. Learning stuff with Joshua and Charles, stuff you should know. All right, so, I think it's generally acknowledged that 2016, the, and again, I want to say the polling was off, but apparently the polling wasn't off, but the way it was reported on was off. But what really happened in 2016, what was off, was the state polling and what they think they've,

like you said, gone back and obsessed over these polls since then, you know, because they were already statistical walks, but when something like this happens, they really sort of get worked into a dander and have to get to the bottom of it. I mean, people were calling for the end of polling. You said it was a failed profession, basically.

Your pee was like, I'm getting rich off this, man. Yeah. We can't end polling. Jimmy Pugh was like, stop, stop talking like that. So what happened in 2016 is, they think, is that a lot of non-educated white people came out in big, big numbers for Donald Trump. And that was a sort of a new, not a new factor,

because they'd always talked about college education, but a new factor in how outsized of a factor that was. It had never been that outsized. And so all these state pollsters, they didn't wait it and they didn't adjust their polls to reflect this fact that college educated people are more likely gonna respond to these surveys. So their polls were just off.

Yeah, and they knew that college educated people were more likely to respond to the surveys. That wasn't news to them. What caught them sleeping was that they had not picked up on the fact that this group of people, non-college-educated white voters, were going to go to the polls in numbers like never before

and that they were going to vote for Trump. They did not pick up on that. That was brand new. Like that didn't exist before. Trump basically brought up a new electorate that helped get him elected, especially in battleground states like Wisconsin and Michigan and Indiana.

Although I think Indiana, he was a shoe-in because of Pence. But this group of voters that did not exist or this line between college-educated and non-college-educated white voters, that partisan gulf hadn't existed before Election Day. The pollsters didn't pick up on it and so they didn't wait those responses because they'd never had to wait the responses before

based on college education. Yeah, so suburbs, ex-urbs, and especially the rural vote counted like it had never counted before, which is obviously why you see what's going on right now. Like a very hard push by the Trump campaign to get these same people out again in the way that they do that. That's the nicest way I can put it.

Genuinely is. So, yeah, so the idea that there was, this was already kind of a close race. A closer race than was being broadcast. That these electoral, huge electoral battleground states that got flipped, that was basically the reason that Trump was able to take the electoral college. But the idea is that these voters kind of came out

of nowhere and voted for Trump. And that there were some other things that happened too, that the pollster didn't anticipate. One, that the undecided voters, people who'd said I'm legitimately undecided at this point a week before the election, from what I read, they broke hard in favor of Trump on election day.

When they made their decision, they voted for Trump. That hadn't been predicted. That was another big one. And then one of the other things too is that the polls were just doing what polls do, which is sometimes they're right, sometimes they're wrong. Polls gonna poll. But polls had gotten so good in the 2000 aughts

that people came to be overconfident in their ability to predict and pick winners. And the 2016 race reminded us like, polling's not perfect, let's stop pretending it is. Yeah, and a lot of it has to do with, like we've been kind of harping on the way the media presents it. And then a lot of it has to do with just our conditioned,

how we're conditioned to look at things like underdogs. And it's different in politics. And I remember when these aggregator, especially at 538, they had these predictive models and they started talking about the fact that, and I think the Washington Post even wrote a good comparison to sports. And if someone has a real big underdog

going into like a Super Bowl or a World Series and they end up winning, people don't get angry and go after the people who said they had a 15 or 20% chance of winning. They just said, wow, what a story, the underdog won. But there are so few presidential elections, one every four years, that it's the same thing, but people just look at it differently.

Like an underdog, like Trump was an underdog that supposedly had like a 15 to 30% chance of winning. Some people said one. Yeah, well, that's ridiculous. But a 30% chance of winning is a real shot at winning. For sure, yes. But the way it's framed, it doesn't seem that way in politics.

No, and so that's one thing. But another thing is that we shouldn't even be talking about presidential elections with like 15% chance of winning, 99% chance of winning. Like that is not how we should present it. And that's not how we used to present it. We used to present it saying like this poll found that Clinton was going to lead Trump 52% to 48%

or something like that, plus or minus two points. And that would have shown you like, oh, okay, well, this is a really close race, way closer than I think. And that's that. There's my information, and the problem is that you can take that same statistic, 52% chance of winning, plus or minus of four point margin of error.

If you convert that to a normal distribution, you come up with an 84% probability of a win. That's the problem is that the statistics that are being the data that's being produced by these polls are being converted in ways that they shouldn't be. And then that's what the media jumps on. That's what the public laps up

because that is the horse race statistic in 84% chance of winning, a 15% chance of winning. That's what we think about. That's what we look at. And so rather than realizing that actually, this is a close race, 52% plus or minus four points, we see 84% chance of winning. And that's a foregone conclusion

that that person's going to win. That ultimately is where the media and the public are culpable for this. Yeah, I don't think they were meant to be extrapolated like that to begin with. No, they weren't. And that, you know, polls are valuable, but like I haven't looked at any polls

and partially because of the way 2016 went down. And in fact, for the past week, I've taken a complete internet news and social media break. And it's been pretty great, actually. Oh man, it's so liberating. Yeah, I mean, I literally haven't looked at a single news thing.

I very sadly found out that Chadwick Boseman passed away like three days afterward. Oh wow. Like that's how dark I've gone in not looking at the internet unless it's something that brings me joy, which is to say, you know, old Led Zeppelin and Van Halen YouTube videos.

I was looking up classic Mad Magazine covers of the 80s the other day. That's all I've been doing is if it doesn't bring me joy on the internet, I'm not doing it. That's good. And you know, I got to break that soon because I do think you should be active

and involved in the know, but... Yeah, but taking a break fairly regularly is definitely mentally healthy for sure. But that aside, I haven't, I'm not looking at any polls. I don't care what any poll says. Well see, so I was thinking very similar stuff too. And like what's the point of polls?

I don't know. Okay, well I finally found it. If you look that up on Google, there's just very little on it. But I found somebody who explained it pretty well, I thought, that polls aren't meant to tell you who's going to win. They're not forecasting models.

Like I said before, they're meant to be like a snapshot of how whoever you're polling feels at the moment, right? And in doing that, because you are sampling American people and these are independent news organizations typically who are carrying out these polls, you get to tell everybody else how America's feeling rather than the leaders saying, I'll decide how you're feeling.

I can decide what you want and what you need and what you think is important. Polls prevent that from happening by telling the rest of the people, hey, this is how everybody else is feeling right now too. And in some ways, it is kind of sheepish where the idea is like, oh, is that supposed to sway my opinion

that everybody's going to vote for this person and not for that person that should have no bearing or impact on your vote. And it feels like that that's how polls are used sometimes. But if you step back and look and see that they're actually kind of an important part of sharing what other people are thinking rather than being told what we're thinking

or what to think, then they actually are pretty legitimate in that sense. Yeah, well, I say take your polls and sit on it. Well, one more thing we cannot talk about polling and not talk about internet polling real quick. This is a completely different style of polling than's ever been done before rather than a randomized sample.

You actually just say, hey, you want to take the survey and people click it. So it's called opt-in surveying. And very specific kinds of people take surveys on purpose on the internet. So they really are, because they're new, they're really now figuring out how to weight these things or not

and how to use them because they can produce legitimate data. But it depends on who's conducting the poll, how, if they know what they're doing, that kind of stuff. But just like everything else, the moving things online is democratized polling. And so anybody can conduct a poll now and basically enter the news cycle.

That's how Kid Rock almost became a senator in Michigan for a second there. But so on the one hand, it's good, but it's also we're in a big period of disruption as far as polling is concerned. So for you, the polling consumer, either go like Chuck and just stop listening to polls altogether

or look for things like transparency. Do you recognize the company or the name that produced the poll? Are they sharing their data? Like how the questions exactly were worded, what their population size was, how they weighted it, all this stuff. If all that stuff is included,

you can probably trust the poll. And then beyond that, just remember what you're looking at, that this isn't a predictor of who's gonna win. It was a snapshot for a very brief moment of a very specific sample of America to show how people would vote right then. And it was right then too,

this is not election day we're talking about. Yeah, and I wanna be clear, I'm not poo-pooing polls. I just, they're valid and useful, but I just don't care to look at them right now. I understand. Yeah, that's my jam. Well, you got anything else about polls? Nothing else about polls.

Well, if you wanna know about polls, start looking around and go check out Pew's stuff and Cyline's stuff and all that stuff. And since I said stuff three times, here comes Rumpelstiltskin. Or Candyman. So this is from Keely Price and Keely says this, hi guys, I'm writing today

not only to confess my unending love for stuff you should know, but also to share a link to some black-owned bookstores. It would be so cool if all of your listeners purchased your book, she should just say period, comma from a black-owned bookstore. Couldn't agree more, by the way. Yeah.

A couple of podcasters that I listened to while I wait for stuff you should know have books out and coming out soon and they encourage their listeners to support black-owned businesses through the purchase of their book. It's a win-win. I don't know why it's taken me so long

to think to write this to you guys. I blame it on corona madness. But last but not least, I'll say I love the end of the world with Josh Clark and Movie Crush as well. Any chance to hear you guys talk is a chance worth taking. When we get a COVID vaccine and you guys can do your live shows again,

please come to Nashville. Oh yeah, for sure. I think we planned on Nashville. Yeah, Nashville got scuttled by COVID this time around. We were going to come, now we might not ever be able to come. No, I know it's super close to Atlanta. I'd lose my mind if I got to see you guys here.

All the best Keely Price. And so Keely sent a link to a handy website that lists black-owned bookstores near you. I made a little URL shortener to make it easier on everyone. Oh, let's have it. So you can go to bit.ly slash S-Y-S-K-B-L-M. Nice.

And find black-owned bookstores near you to purchase stuff you should know an incomplete compendium of mostly interesting things. At the very least, we like to encourage people to go to ndbound.org and support ndbookstores. I don't know if there is an actual black-owned ndbookstore website,

but I would imagine most of the black-owned bookstores are ndbookstores. Yeah, probably. So check it out. bit.ly slash S-Y-S-K-B-L-M. Go out and buy our book, everybody. You're gonna love it. It's really great.

And thanks for that, Chuck. And thanks for setting us up for that too, Keely. Much appreciated. We'll see you in Nashville. I guess Keely will be the one, like she said, losing your mind in the crowd. If you wanna lose your mind on us via email, we love that kind of thing, kinda.

You can send it off to stuffpodcastsatihartradio.com. ["I Heart Radio"] Stuff you should know is a production of I Heart Radio's How Stuff Works. For more podcasts from I Heart Radio, visit the I Heart Radio app. Apple podcasts are wherever you listen to your favorite shows.

On the podcast, Hey Dude, the 90s called, David Lasher and Christine Taylor, stars of the cult classic show Hey Dude, bring you back to the days of slip dresses and choker necklaces. We're gonna use Hey Dude as our jumping off point, but we are going to unpack and dive back into the decade of the 90s.

We lived it, and now we're calling on all of our friends to come back and relive it. Listen to Hey Dude, the 90s called on the I Heart Radio app, Apple podcasts, or wherever you get your podcasts. Hey, I'm Lance Bass, host of the new I Heart podcast, Frosted Tips with Lance Bass. Do you ever think to yourself, what advice would Lance Bass

and my favorite boy bands give me in this situation? If you do, you've come to the right place because I'm here to help. And a different hot, sexy teen crush boy bander each week to guide you through life. Tell everybody, ya everybody, about my new podcast and make sure to listen so we'll never, ever have to say, bye, bye, bye.

Listen to Frosted Tips with Lance Bass on the I Heart Radio app, Apple podcast, or wherever you listen to podcasts.