That Neuroscience Guy - The Neuroscience of Understanding Science

Episode Date: November 28, 2022

Some individuals may have told you to "do your own research" when discussing current events, especially those related to scientific findings. But what is the correct way to do that research and understand modern science? Today's episode of That Neuroscience Guy provides a primer on finding, reading, and understanding scientific research. 

Hi, my name is Olof Kregolsen, and I'm a neuroscientist at the University of Victoria. And in my spare time, I'm that neuroscience guy. Welcome to the podcast. So I was inspired for today's episode because over the past couple years, I've seen people for today's episode because over the past couple years, I've seen people quoting and interpreting research, but they really didn't know what they're talking about. Of course, I'm referring to the COVID pandemic, but there's been any other number of cases where I've seen people say they're quoting research and they're discussing research, but they're clearly doing it wrong. So on today's podcast, a primer on how to interpret scientific research. So what I'm going to do is give you a bunch of steps. It's the way I'm going to handle this. Things that you have to consider if you're

going to interpret scientific research. Now, probably the very first thing to discuss is whether you're getting your research from a primary source or a secondary source. So primary sources are academic peer reviewed journals. Now, what I mean by that is they're academic in that they're scientifically motivated to publish. They're not publishing for commercial reasons. You can't go down to the local bookstore and buy a copy. You can't even order a copy online. They're academic journals. And I'll talk a little bit more about how to figure out whether a journal is academic or not in a second. Now, that's a primary source, a research study in an academic journal. And the reason I say that is sometimes in academic journals, you get what's called a review paper. A review paper is when they discuss a bunch of studies and they sort of give

you a summary. Even then, you have to take that with caution. Even though the review paper is in an academic journal, you should still go find the primary source, the original research article. Now, what are secondary sources? Well, it's basically everything else. Wikipedia, Google, Facebook, these are all secondary sources because no scientist publishes their findings in these sources. And the problem is that someone might transcribe the source, if you will. They might take the information that's there and give you their own version. This has happened to me many times. I've been quoted in popular magazines. I've been quoted on TV and I've been quoted in the press and they tell the story, but they kind of make a few

mistakes. And then if someone else tells that story that they read, say in a popular journal, then a few more mistakes are going to get put into the process. It's kind of like when you go around the circle at a camp. If you remember when you were a kid, and one person whispers the story in one person's ear and they whisper to the next. And by the time it gets to the end, no one recognizes the story. So that's why you have to go to a primary source. Now to see if it's a primary source, you have to talk about academic journals, because we publish in what are called journals, You have to talk about academic journals because we publish in what are called journals, fancy word for magazine, versus predatory journals versus non-academic journals.

So an academic journal is where scientific research should be published and it's the only place you should truly go. Predatory journals have come about in the last 10 to 15 years and these are journals that look scientific, but they actually either give you money to publish your paper or they charge you to publish your paper. In both cases, it means that the scientific process is being circumvented. And then finally, there's non-academic journals. These are all of the journals that you would see on a magazine rack. So how do you know something's an academic journal? Well, the best thing to do is go to a website called PubMed. All right. PubMed is a free website for anyone to use. And if you can find the

article in the journal on PubMed, it's legitimate. It's something that's being vindicated by the scientific community. And it is the only true place to figure out if a journal is real or a paper is real. If it's in a non-academic journal, it won't appear in PubMed. And if it's in a predatory journal, it generally doesn't appear in PubMed either. I wouldn't use Google Scholar. Check out PubMed. Now, it's not a great place to read articles, but it's a great place to look up a journal or article and find out if it's real or not. The other thing you can

check really quickly is if you actually go to the journal's homepage, every journal will have a homepage. You'll find out that if they're charging enormous fees to publish, it's either predatory or it's not good. Most academic journals, it's free to publish in. There's a few small exceptions, but that's a general rule. So primary sources that are in academic journals that you've verified on PubMed. Now, the next thing you can do is look at the authors. Google the authors. See where they are. Authors on true primary source academic papers are basically working at universities like I do.

That's where research scientists generally live. If they're working for a company in their publishing, you have to think about bias. You know, if a researcher is being hired by company X to do research, well, you probably got to suspect that the conclusions they draw are a bit biased by company X. And same goes for think tanks and other things like this. The best sources are academics who are working as research scientists at universities. Now, there are some exceptions to that, but it's a good rule of thumb to have. Now, let's dive into the paper itself. So, let's say you find the paper. Now,

I admit these can be hard to find because generally the way it works is university libraries pay the journal companies for a subscription. And a lot of journals allow you as a consumer to buy an academic article, but they're generally not up there for free. That's actually technically a no-no. So generally, if you're finding the PDF of the article for free, it's because someone's sharing it and they probably shouldn't be, or it might be in one of these predatory or non-academic journals. Now, if you do find an academic paper, you're going to find that it has an introduction.

And the introduction is important because it sets the background. Why are you doing the study? But the most important part is at the end of the introduction, the scientist will have hypotheses. What are they predicting will happen? What are they trying to show? Then you get into the methods. The methods are generally really complex if you're not an expert in the area.

And the reason for that is that the standard in academic publishing is that you should be able to replicate the study with just the information in the paper. So I could read a paper by someone else, go out and follow their methods and get the same result. Now, we're going to come back to the results in a second, but the next section is typically results. And then finally, there's the discussion. The discussion is basically an opinion piece. It's where you say, well, this is what I found and this is what I think it means. I'll be honest, I don't read the discussion most of the times when I read a paper because I look at the

results and I draw my conclusions based on what I know. Now, let's get back to the results, because I said that that's an important part. What are good results? You might know about what are called p-values, and I'll get back to them in a second, but it's a way that we prove that the scientific result is there or not. Now, let's come up with a simple study.

I don't want to talk about COVID and get people angry, so let's go with something simple. I'm going to be biased towards my home country, And I'm going to say people are taller in Canada. They're taller than the people of the United States. Now, how do we know that's a true statement? Well, there's two ways to do this. One, we could measure the height of everybody in Canada, and we could measure the height of everyone in the US. And if we measured every single person, every single person, then you wouldn't need to do anything else because you would know the truth. You could just look at the average height and go, well, people are taller in

Canada or people are taller in the United States. But the way scientists work is we have to use a subset. We can never test everybody. So we might test 50 people. We take 50 people from across Canada, 50 people from across the United States. And that's who we use as our subset of people that we test. And that means that we have to infer that there's a true result. So we basically want to test a subset of people and say, hey, the subset of people from Canada have a taller height than the subset of people from the United States. And this is where inferential statistics and p-values come in. And really, I'm not going to dive into it too deeply. There's plenty of sources if you want to learn about p-values and inferential statistics. But the

general idea is pretty simple. You want to infer that the people in Canada are taller than the United States. So you run a statistical test that gives you a p-value. And that p-value is a bit complicated to explain. But the simple version is that if it's less than 0.05, so 5%, so let's say people are taller than Canada in my sunset than the United States, and my p-value for my statistical test is less than 0.05, or 5%, I would say, yes, that's actually true. We believe this to be true for all people in Canada and all people in the United States. And if the p-value is greater than 0.05, so it's 0.06 or higher, so 5.1% or higher,

then we would say it's not true. People in Canada are the same height as people in the United States. And the reason the average is different between the two subsets, or samples as we'd call them, is just random chance. the same height as people in the United States. And the reason the average is different between the two subsets or samples, as we'd call them, is just random chance. So you need to run these statistical tests to make sure it's not a random chance. Now it's a bit more complicated than that even because you really have to know a little bit more detail to interpret it. But the first thing

you do is look for the p-values because that that's the truth statement. Now, there's newer alternatives. There's something called the new statistics, where people plot what are called 95% confidence intervals. And basically, that just means that they are 95% certain that the mean of the sample reflects the population mean that the lies within that range are a population average. And there's Bayesian statistics now, which are even more complicated, but they're more conservative, which is good. And the reason for this is that there is what we're calling a reproducibility crisis, where there's a lot of published papers out there where people can't replicate the result. So someone managed to pull the study off once and it never happened

again. Now, what do I mean by that? Well, imagine I do test 50 people from Canada and 50 people from the United States and the p-value is less than 0.05 and my average values indicate people in Canada are actually taller. Well, that's just one study. In principle, you would want to run that study again and again and again. And this is what we call replication. If you did that with 10 samples across Canada and they were all statistically significant, and that means a p-value less than 0.05, then you'd say, hey, these things are actually different. And if it happened once and it didn't happen 19 times, then the one time it happened was by chance. And the way you can think about this simply is,

imagine you go to the shopping mall and you're going to determine the average might of the shopping mall just by picking one person. Well, what if you pick Shaquille O'Neal? You can't say that Shaq is representative of the height of all of the people in the shopping mall. Now, you might want to look at the figures. You know, I always tell students that are reading research papers, if it's really important, a really important result, there'll be a figure or a table.

So you want to make sure that the figures have error bars because that shows you the confidence of the result. And there's even types of error bars you want to look for. Basically, the 95% confidence interval is a gold standard. Other types of error bars you want to look for. Basically, the 95% confidence interval is a gold standard. Other types of error bars are statistically valid, but they're generally not as conservative as a 95% confidence interval. You also want to check the axis scales. It's a really simple trip. If you have two bars on a graph and you want them to look different, just play with your axis, the y-axis scale, the axis on the left, and keep changing it until all of a sudden it looks like there's a massive difference.

So take a look at that and use some common sense. And probably the last lesson before I summarize is the most important one. If the sample size of a study is less than 30 people per group, so let's say less than 30 Canadians and less than 30 Americans, you can't trust the results. If they're comparing eight people to eight people, basically any result they could find is by chance. Now that's for lab-based research. If it was a survey, I would tell you they have to test a whole bunch of people. And again, you have to think of bias. If they test, if you do a survey, you survey, who's answering the survey? Is it just people

that happen to be at home during the day? Well, that rules out a large percentage of the population where everyone has to work during the day. So I just wanted to give you a little primer on how to actually evaluate scientific research. I think the key points to understand is always find a primary source. Use that website I told you about, PubMed, and find academic papers and try your best to read them. Do not rely on secondary sources like the internet, it's the worst one out there, or non-academic journals or those predatory journals I mentioned. And when you do find the paper, make sure that they have large sample sizes. Make sure they've tested a lot of people so that you can believe the results.

And last of all, if you really got the little lesson in there, look at the results and try to decipher what it means. If you use a bit of YouTubing, you can find all about p-values. In fact, I have a couple videos up there about what p-values are. And see the actual results for yourself as opposed to relying on someone else to interpret them for you. All right, I have lots of opinions on Facebook and social media, but they don't reflect my scientific opinion. They're just my opinions on things and usually they're things I don't know much about. You know, I've said for

years Quiznos is better than Subway. Do I have any scientific evidence to support that? Heck no. Anyway, that's it for today. A little primer on how to interpret scientific results. Just as a bit of a preview, that ADHD episode is coming out next week. I promise we've already started to record. And then we're going to do a really interesting one. A friend of mine was asking me about the impact of sunlight in the Northern Hemisphere. We're heading into that time of year where it's dark out all the time.

And I did a lot of reading about how that impacts our brain. And I think you're going to find it's pretty cool. And of course, there's going to be a couple of neuroscience bites out there. Remember, of course, you can check out the website, thatneuroscienceguy.com, where you can see our links to Patreon, where you can support my graduate student, Matt Hammerström, who does the sound production directly. And you can also

buy shirts from the Etsy store. We've sold a couple recently, thanks to the people that bought them. Again, all that money goes to Matt. And please, if you've got ideas, we're already thinking about season five. I'm not going anywhere. Neither is Matt. So follow me on Twitter, at That Neuroscience Guy, and DM me some cool ideas. A lot of the ideas for season Season five, I'm not going anywhere. Neither is Matt. So follow me on Twitter at that narrow side guy and DM me some cool ideas. A lot of the ideas for season four have come from that. And last but not least, thank you so much for listening to the podcast.

Please subscribe. It really does help. We're trying to climb the Apple rankings and the rest of it again, and a little bit of a ploy to get money for Matt. Anyway, big shout out to Matt as well, because he's going to be editing the sound on this one, and he's done such a great job. He's made the podcast entirely possible.

That's all I got for this week. I'll be back shortly with a neuroscience bite in another episode, the one on ADHD. My name is Olive Craig Olson, and I'm that neuroscience guy. Thank you so much for listening, and I'll see you soon.