The Peter Attia Drive - #188 - AMA #30: How to Read and Understand Scientific Studies

Episode Date: December 20, 2021

In this “Ask Me Anything” (AMA) episode, Peter and Bob dive deep into all things related to studying studies to help one sift through all the noise to find the signal. They define the various types of studies, how a study progresses from idea to execution, and how to identify study strengths and limitations. They explain how clinical trials work, as well as the potential for bias and common pitfalls to watch out for. They dig into key factors that contribute to the rigor (or lack thereof) of an experiment, and they discuss how to measure effect size, differentiate relative risk from absolute risk, and what it really means when a study is statistically significant. Finally, Peter lays out his personal process when reading through scientific papers. If you’re not a subscriber and listening on a podcast player, you’ll only be able to hear a preview of the AMA. If you’re a subscriber, you can now listen to this full episode on your private RSS feed or on our website at the AMA #30 show notes page. If you are not a subscriber, you can learn more about the subscriber benefits here. We discuss: The ever changing landscape of scientific literature [2:15]; The process for a study to progress from idea to design to execution [4:15]; The various types of studies and how they differ [7:30]; The different phases of a clinical trial [19:15]; Observational studies and the potential for bias [26:30]; Experimental studies: Randomization, blinding, and other factors that make or break a study [44:00]; Power, p-values, and statistical significance [56:15]; Measuring effect size: Relative risk vs. absolute risk, hazard ratios, and “Number Needed to Treat” [1:07:45]; How to interpret confidence intervals [1:17:30]; Why a study might be stopped before its completion [1:23:45]; Why only a fraction of studies are ever published and how to combat publication bias [1:31:30]; Why certain journals are more respected than others [1:40:30]; Peter’s process when reading a scientific paper [1:43:45]; and More.

Hey everyone, welcome to a sneak peek, ask me anything, or AMA episode of the Drive Podcast. I'm your host, Peter Atia. At the end of this short episode, I'll explain how you can access the AMA episodes in full, along with a ton of other membership benefits we've created. Or you can learn more now by going to PeterittiaMD.com forward slash subscribe. So without further delay, here's today's sneak peek of the Ask Me Anything episode. Welcome to another Ask Me Anything episode, AMA number 30. And once again, joined by Bob Kaplan. In today's episode episode we discuss all things around studying studies. If you listen to this podcast or read any of my weekly emails you know that I place a

large emphasis on being able to sift through the noise and find the signal when it comes to various studies and papers that are printed both as the studies themselves and unfortunately a lot of times with the media reports on them. So if you follow the news you know there's no shortage of articles that either contradict each other, seem too good to be true, don't make any sense. It can be hard to understand this. So what do you do? Well, we've, as some of you may know, written a series on this called studying studies,

but we've also tried to tackle some of the bigger things in this AMA. And we come through a lot of the questions that many of you have been asking over the previous months and years that relate to this topic. And I think we have enough of them here that we were able to put together a solid episode. This episode's a bit longer than normal. We contemplated breaking it into two because it's so long, but I think our audience can handle this. At least now you have in one place the all-singing, all-dancing discussion of how to study studies. In this discussion, we talk about a bunch of things. What are they? What is the process for a study to go from an idea to a design to execution?

What are the different types of studies out there? And what do they mean? What are the strengths and limitations of each of them? How do clinical trials work specifically for drugs? What are the common pitfalls of observational studies that you should be looking for? What questions should you be asking about a study to figure out how rigorous it was?

What does it really mean when a study is statistically significant? Why do some studies never get published? What is my process for reading scientific papers? So if you're a subscriber and you want to watch the video of this podcast, you can see it on the show notes page I think that's valuable for this episode because I do refer to some tables though This is still certainly amenable to audio only if you're not a subscriber You can watch the sneak peek of this on our YouTube page. So without further delay I hope you enjoy AMA number 30.

Hey Bob how are you man looking pretty studious there in the library today. Hey Peter thanks very much. We're just getting some reading in before the podcast. This is going to be a pretty good one because as you may recall about, I don't know, four or five months ago, maybe longer, I was on a podcast with Tim Ferriss. And I don't know how it came up, but I do remember somehow it came up that we had spent a lot of time writing this series, studying studies. And God, that's been four years ago, I think. But we didn't really have something more digestible for folks on how

to make sense of the ever-changing landscape of scientific literature and how to kind of distinguish between the signal and the noise of the research news cycle. And I remember after that, Tim and I went out for dinner and he kept pressing me on, well, what can I do to get better at this process? Are there newslet. I see me subscribing to and things like that. And while I'm sure that there are, I didn't know what they were off top of my head. And so I think what we've done here, when I say we, I mean you, what you have done here is aggregate all the questions that have come in over the past year, basically, that pertain to understanding the structure of science.

I looked through the questions last week and I was pretty excited. I think it's gonna be a sweet discussion and I hope this serves as an amazing primer for people to really understand the process of scientific experiments and everything from how studies are published and obviously what some of the limitations are. So anything else you wanna add to that, Bob,

before we jump in? I agree, I think it's a fun topic. We get so many of these questions that we end up early, sti-do, or to the website, where we'll point readers to one of the parts of the studying studies, but I think sometimes just talking about it and explaining it can help a lot. So I think this will be really useful as far as like a question and answer session rather than just reading a blog. I don't think this displaces that other stuff.

I think we go into probably more detail on some things there, but I also think we're going to cover things here that aren't covered there. So depending on how you like to get your info, this could be fun. So where do you want to start? We have again, a lot of questions, but I think this question gets to the core of, I think what we're trying to do here, which is, how can a user or a person who has no scientific background better understands studies that they read in the news or in the publications to know if the findings are solid or not, especially in today's age where

you can easily see two studies that contradict each other. Coffee's good, coffee's bad, eggs are good, eggs are bad. So I thought we could run through a bunch of questions with the first one that we got here is, what is the process for a study to go from an idea to design an execution? This is a great question. In theory, it should start with a hypothesis. Good science is generally hypothesis driven. I think the cleanest way to think about that is to take the position that there is no relationship between two phenomena. We would call this sort of a null hypothesis. So my hypothesis might be that drinking coffee makes your eyes turn darker. So, I would have to state that hypothesis, and then I would have to frame it in a way that

says, my null hypothesis is that when you drink coffee, your eyes do not change in color in any way, shape or form, and that would imply that the alternative hypothesis is that when you drink coffee, your eyes do change color. You can already see, by the way, that there's nuance to this, because am I specifying what color it changes to? Does it get darker? Does it get lighter? Does it change to blue? Green? Does it just get the darker shade of whatever it is? But let's put that aside for a moment and just say that you will have this null hypothesis and you will have this alternative hypothesis. And to be able to formulate that, cleanly is sort of the first step here.

The second thing, of course, is to conduct an experimental design. How are you going to test that hypothesis? As we're going to talk about, a really, really elegant way to test this is using a randomized controlled experiment. If it's possible to blind it, we'll talk about what that means. You'll have to decide, well, how long should we make people drink coffee, how frequently should they drink coffee, how are we going to measure eye color? These are the questions that come down to experimental design. You then have to determine a very important variable,

which is how many subjects will you have, and of course that will depend on a number of things, including how many arms you will have in this study. But it comes down to doing something that's called a power analysis, and this is so important that we're going to spend some time talking about it today, although I won't talk about it right now. If this study involves human subjects or animal subjects, you will have to get something called an institutional review board to approve the ethics of the study. So you'll have to get that IRB approval. You'll have to determine what your primary and secondary outcomes are, get the protocol approved, develop a plan for statistics, and then pre-register the study.

All of these things happen before you do the study, and of course, in parallel to this, you have to have funding. So those are kind of the steps that go into doing an experimental study. And what we're going to talk about, I think in a minute is that there are some studies that are not experimental, where some of these steps are obviously skipped. Yeah, one of the questions we got was, what are the different types of studies out there, and what do they mean? For example, observational study versus a randomized controlled study. What are the different types of studies? I think broadly speaking, you can break studies into three categories. One would be observational studies.

We'll bifurcate those or try for Cate those in a minute. Then you can have experimental studies, and then you can have basically summations of and or reviews of and or analyses of studies of any type. Let's kind of start at the bottom of that pyramid. I think you actually have a figure that I don't like very much, but I was going to say that was one of your favorites. Yeah, I can't stand it. I'll tell you what I like about the figure. I like the color schema, because my boys are so obsessed with rainbows that if I show them this figure, they're going to be

really happy. So let's pull up said rainbow figure. Okay, got it. Okay, so you can see these buckets here. And again, at the level of talking about them, I think this makes sense. What I don't agree with the pyramid for Bob is that it puts a hierarchy in place that suggests a meta-analysis is better than a randomized control trial, which is not necessarily true.

But let's just kind of go through what each of these things mean. So looking at the observational studies, an individual case report is first or second paper I ever wrote in my life when I was in medical school was an individual case report. There was a patient who had come into clinic when I was at the NIH. This was a patient with metastatic melanoma and their calcium was sky high, dangerously high, in fact. And obviously our first assumption was that this patient had metastatic disease to their bone and that they were lysing bone and calcium was leaching into

their bloodstream. It turned out that wasn't the case at all. It turned out they had something that had not been previously reported in patients with melanoma, which was they had developed this parathyroid hormone-related-like hormone in response to their melanoma. This was they had developed this parathyroid hormone-related like hormone in response to their melanoma. This is a hormone that exists normally, but it doesn't exist in this format. And so their cancer was causing them to have more of this hormone that was causing them to raise their calcium level. It was interesting because it had never been reported before in the literature, and so

I wrote this up. This was an individual case report. Is there any value in that? Sure, there's some value in that. The next time a patient with melanoma shows up to clinic and their calcium is sky high and someone goes to the literature to search for it, they'll see that report.

And it will hopefully save them time in getting to the diagnosis. You're mentor and friend Steve Rosenberg. I think of him when I think of individual case reports. I think if you listen to the podcast, he talks about this, but a lot of what motivated him early on, I think we're just a couple of cases. I think it gets back to that first question too about the process for a study to go to an idea to design execution, which is to have a hypothesis you need to make an observation.

And so you make an observation, you say, hmm, that's strange. And I think that that's what individual case reports can represent sometimes. This is an interesting observation. It's hypothesis generating for the most part, but it really might kickstart a larger trial or it might kickstart a career. You never know. Exactly. Now, of course, it's not going to be generalizable.

I can't make any statement about the frequency of this in the broader subset of patients and obviously I can't make any comment about any intervention that may or may not change the outcome of this. So that gets us to kind of our next thing which is like a case series or set of studies. So here you're basically doing the same thing, but in plural, effectively. You wouldn't just look at one patient, you would say, well, I've now been looking back at my clinical practice, and I've had 27 patients over the last 40 years that have demonstrated this very unusual finding. Another example of this going back to the C. Rosenberg case would be

one could write a paper that looks at all spontaneous regressions of cancer. Obviously, spontaneous regressions of cancer are incredibly rare, but there are certainly enough of them that one could write a case series. So now let's consider cohort studies. So cohort studies are larger studies and they can be retrospective or they can be prospective. So I'll give you an example of both. So a retrospective observational cohort study would be let's go back and look at all the people who have used SONAs for the last 10 years and look at how they're doing today

relative to people who didn't use saunas over the last 10 years. So it's retro perspective. We're looking backwards. It's observational. We're not doing anything right. We're not telling these people to do this or telling those people to do that. And the hope when you do this is that you're going to see some sort of pattern. Undoubtedly you will see a pattern. Of course, the question is, will you be able to establish causality in that pattern? Cohort studies can just as easily, although more time consumingly be prospective. So you could say, I want to follow people over the next five years, 10 years, who use sonnas, and compare them to a similar number of people who don't.

And now, in a forward-looking fashion, we're going to be examining the other behaviors of these people, and ultimately what their outcomes are. Do they have different rates of death, heart disease, cancer, Alzheimer's disease, other metrics of health that we might be interested in? Again, we're not intervening. There's not an experiment per se. We're just observing, but now we're doing it as we march forward through time. So this brings us to the kind of the next layer of this pyramid, which are the experimental studies. Divide these into randomized versus non-randomized, and of course, this idea of randomization is going to be a very important one as we go through this.

So a non-randomized trial sometimes gets referred to as an open label trial where you take two groups of people and you give one of them a treatment and you give the other one either a placebo or a different treatment, but you don't randomize them. There's a reason that they're in that group. So you might say, we want to study the effect of a certain antibiotic on a person that comes in the ER,

and we're going to take all the people that come in who look a certain way, maybe they have a fever of a certain level or a white blood cell count of a certain level, we're going to give them the antibiotic and the people who come in, but they don't have those exact signs or symptoms, we're going to not give an antibiotic to, and we're going to follow them. That's kind of a lame example. You could do the same sort of thing with surgical interventions.

We're going to try to ask the question is surgery better than antibiotics for appendicitis or suspect that appendicitis, but we don't randomize the people to the choice. There's some other factor that is going to determine whether or not we do that. As you can see, that's going to have a lot of limitations because presumably there's a reason you're making that decision and that reason will undoubtedly introduce bias. So of course, the gold standard that we always talk about is a randomized control trial where whatever question you want to study, you study it, but you attempt to take all bias out of it by randomly assigning people into the treatment groups, the two or more

treatment groups. We'll talk about things like blinding later, because you can obviously get into more and more rigor when you do this, but before we leave the kind of experimental side, anything you want to add to that, Bob? I would add, so non-ranomized controlled trials, maybe another example, a lestrative example, I think, with non-ranomized controlled trials might be, you have patients maybe making a decision beforehand, which will get into selection bias, but they might want to go on a stat and, let's say, and then you give them a choice. The other ones might want to go on some other drug like a Zedemib.

They're basically selecting themselves into two groups, but you could compare those two groups and see how they do, but it hasn't been randomized. There's a lot of bias that can go into that. There could be a lot of reasons why one group is selecting a particular treatment over the other. And so that's why I think when we get to randomized trials that shows the power of randomization. Yeah, exactly. We don't need to go back to the figure, but people might recall that the top of that pyramid was systemic reviews and meta-analyses. Let's just talk about meta-analyses

since they are probably the most powerful. So this is a statistical technique where you can combine data from multiple studies that are attempting to look at the same question, basically. So each study gets a relative weighting and the weighting of a study is sort of a function of its precision. It depends a little bit on sample size, other events in the study, larger studies,

which have smaller standard errors are given more weight than smaller studies with larger standard errors, for example. You'll know you're looking at a meta-analysis. We should have had a figure for this, but I'll describe it the best I can. They usually have a figure somewhere in there that will show across rows all of the studies. So let's say there's 10 studies included in the meta analysis. And then they'll have the hazard ratios for each of the studies. So they'll represent them usually as little triangles. The triangle will represent the 95% confidence interval of what the hazard ratio is, which we'll talk

about a hazard ratio, but it's basically a marker of the risk. And you'll see all 10 studies, and then they'll show you the final summation of them at the bottom, which of course you wouldn't be able to deduce looking at the figure, but it takes into account that mathematical weighting. So on the surface meta analyses seem really, really great, because if one trial, one randomized trial is good, 10 must be better. I know I've said this before, probably three or four times over the past few years on the podcast, but as James Yang, one of the smartest people I ever met when I was both a student and fellow at NCI once

said during a journal club about a meta-analysis that was being presented, he said something to the effect of a thousand-sous ears makes not a pearl necklace. And that's just an eloquent way to say that garbage and garbage out. So if you do a meta-analysis of a bunch of garbage studies, you get a garbage meta-analysis. It can't clean garbage. It simply can aggregate it. So a meta-analysis of great randomized control trials will produce a great meta-analysis. They try to control for garbage, the researchers and the investigators, but I think to your

point, with the Pearl necklace, imagine if you had, say, 10 trials and nine of them are garbage, one of them is really good, really rigorous randomized control trial. And you're looking at the top of the pyramid and you're saying, well, meta-analysis is the best. We should be looking at this meta-analysis. Meanwhile, you've got that one randomized control trial that actually is worth its salt, its rigorous, et cetera, that I would say, if you had the option, I think you probably would rely more on that one randomized control trial, which is lower on the pyramid. So I think that's probably, I think

you've told me, one of your hangups with the pyramid, because it's not necessarily top of the pyramid is going to be some meta analysis of randomized control trials. That's right. Yeah. I don't want to suggest meta-analyses are not great. What I want to suggest is you can't just take a meta-analysis as gospel without actually looking at each study. You don't get a pass at examining each of the constitutive studies within a meta-analysis. It's really the point I think we want to make here. There's one thing in here that isn't represented, but we had a few questions about it. I think a couple. People are asking about what's the difference between a phase three and a phase two or a phase one clinical trial.

You know what's going on there? Yes. So here we're talking about human clinical trials. This phraseology is used by the FDA here in the United States. And typically, the world does tend to follow and lockstep, but not always with kind of the FDA's process. So if you go way, way, way back, you have an interesting idea.

You have a drug that you think is, or a molecule that you think will have some benefit. Think of it as a cancer therapeutic. You've done some interesting experiments in animals, maybe started with some mice and you went up to some rats and maybe even you've done something in primates. And now you're really committed to this as the success of this and the safety of this in animals looks good. So it's both safe and efficacious in animals and you doubt decide you want to foray into the human space.

Well, the first thing you have to do is file for something called an IND, an investigational new drug application. So after you do all of this preclinical work, you have to file this IND with the FDA, and that basically sets your intention of testing this as a drug in humans. And the first phase of that, which is called phase one, is geared specifically to dose, escalate this drug

from a very, very low level to determine what the toxicity is across a range of doses that will hopefully have efficacy. These are typically very small studies, usually less than 100 people. They're typically done in cohort. So you might say, well, the first 12 people are going to be at 0.1 milligrams per kilogram, and assuming we see no adverse effects there,

we'll go up to 0.15 milligrams per kilogram for the next 12 people. And if we have no effects there, we'll go up to 0.15 milligrams per kilogram for the next 12 people. And if we have no issues there, we'll escalate it to 0.25 to the the the the notice Bob. I said nothing in there about does the drug work? These are going to be patients with cancer. If this is a drug that's being sought as a treatment for colon cancer, these are going to be patients that all have colon cancer. They're often going to be patients who have metastatic colon cancer. So these are going to be patients who have progressed through all other standard treatments and who are basically saying, look, sign me up for this clinical trial.

I realize that this first phase is not going to be necessarily giving me a high enough dose that I could experience a benefit, and that you're really only looking to make sure that this drug doesn't hurt me. But nevertheless, I want to participate in this trial. If the drug gets through phase one safely, then it goes to phase two. And the goal of phase two is to continue to evaluate for safety,

but also to start to look for efficacy. But this is done in an open label fashion. What that means is they're not randomizing patients to one drug versus the other typically. They can, but usually it's now we think we know one or two doses that are going to produce efficacy. One or two doses that are going to produce efficacy, they were deemed safe in the phase one. We're now going to take patients and give them this drug and look for an effect. And a lot of times, if there's no control arm in the study, you're going to compare to

the natural history. So let's assume that we know that patients with metastatic colon cancer have, on standard of care, have immediate survival of X months. Well, we're going to give these patients this drug and see if that extends it anymore. And of course, you could do this with a control arm, but now it adds the number of patients to the study. So again, typically very small studies can be, you know,

in the 2030, 40, 50 range, maybe up to a few hundred people. And that one, Peter, I think is a probably a good example of if you have the non-randomization, this might be a case where say it's an immunotherapy and people know about the immunotherapy and it's been really effective. It gets approved for a particular cancer, let's say. And there are a lot of people that know about it and there are cancer patients that know about it and they want to get that treatment, but it's not approved. They're talking to their doctor, they maybe they're online. They might enroll in one of these trials because they really want to try the drug and

maybe they might believe in it more than some other treatment. Yep. There are lots of things that can introduce bias to a phase two if it does not have randomization. Again, the goal would be to still randomize in phase two because you really do want to tease out efficacy. So if a compound succeeds in phase two, which means it continues to show no significant adverse safety effects, which by the way, it doesn't mean it doesn't have side effects. Every treatment has side effects.

It's just that it doesn't have side effects that are deemed unacceptable for the risk profile of the patient. And it shows efficacy. So really you have to have these two things. You then proceed to phase three. Here, a phase three is a really rigorous trial. This is a huge step up.

It's typically a log step up in the number of patients. You're talking potentially thousands of patients here. And this is absolutely a placebo controlledcontrolled trial, or not necessarily placebo, but it can be standard of care versus standard of care plus this new agent, but it is randomized. Whenever possible, it is blinded, and with drugs, that's always possible. And these are typically longer studies. Because you have so much more sample size, you're going to potentially pick up side effects that weren't there in the first place. And of course, now you really have that

gold standard for measuring efficacy. And it's on the basis of the phase one, phase two, and mostly phase three data that a drug will get approved or not approved for broad use, which leads to a fourth phase, which is a post-marketing study. So phase four studies take place after the drug has been approved. And they're used to basically get additional information because once the drug is approved, you now have more people taking it. And they may also be using this to look at other indications for the drug. We talked about this recently, right? A phase four trial with semi-glutide being used to look at obesity versus its original

phase three trials, which we're looking at diabetes. The drug's already been approved. This study isn't being done to ask the question, should semi-glutide be on the market? No, it's on the market. It's basically expanding the indication for semi-glutide, in this case, so that insurance companies would actually pay for it for a new indication. But given the size and the number of these studies, you're also looking for, hey, is there another side effect here that we missed in the phase three? Right. And it might be the particular population. It might have a different risk profile.

You might have a different threshold. That's right, because you're not doing this in patients with type two diabetes. You're doing this in patients who explicitly don't have diabetes, but have obesity, different patients. Can we are going to see something different here? So yeah, so anyway, that's the long and short of phases one, two, three and four. Okay. So going back to observational studies, are there any things that you look for in particular that will increase or decrease your confidence in it,

whether it's a Pearl necklace or a garbage? Thank you for listening to today's Sneak Peak AMA episode of the Drive. If you're interested in hearing the complete version of this AMA, you'll want to become a member. We created a membership program to bring you more in-depth, exclusive content

without relying on paid ads. Membership benefits are many, and beyond the complete episodes of the AMA each month, they include the following. Redeculously comprehensive podcast show notes that detail every topic, paper, person, and thing we discuss on each episode of the drive. Access to our private podcast feed, the qualities which were a super short podcast typically less than five minutes, released every Tuesday through Friday, which I like the best questions, topics and tactics discussed on previous episodes of the drive.

This particularly important for those of you who haven't heard all of the back episodes becomes a great way to go back and filter and decide which ones you want to listen to in detail. Really steep discount codes for products I use and believe in, but for which I don't get paid to endorse, and benefits that we continue to add over time. If you want to learn more and access these member-only benefits, head over to peteratiamd.com forward slash subscribe. Lastly, if you're already a member but you're hearing this, it means you haven't downloaded our member-only podcast feed, where you can get the full access to the AMA and

you don't have to listen to this. You can download that at peteratia-md.com forward-slash-members. You can find me on Twitter, Instagram, and Facebook, all with the ID, peteratia-md. You can also leave us a review on Apple Podcast podcasts or whatever podcast player you listen on. This podcast is for general informational purposes only. It does not constitute the practice of medicine, nursing, or other professional health care services, including the giving of medical advice. No doctor-patient relationship is formed. The use of this information and the materials linked to this podcast is at the user's own risk.

The content on this podcast is not intended to be a substitute for professional medical advice, diagnosis, or treatment. Users should not disregard or delay in obtaining medical advice from any medical condition they have, and they should seek the assistance of their healthcare professionals for any such conditions. Finally, I take conflicts of interest very seriously. For all of my disclosures and the companies I invest in or advise, please visit peteratiamd.com forward slash about where I keep an up-to-date and of such companies. you