Speaking of Psychology - How ‘open science’ is changing psychological research, with Brian Nosek, PhD

Episode Date: June 16, 2021

Is psychology research in a crisis or a renaissance? Over the past decade, scientists have realized that many published research results, including some classic findings in psychology, don’t always hold up to repeat trials. Brian Nosek, PhD, of the Center for Open Science, discusses how psychologists are leading a movement to address that problem, in psychology and in other scientific fields, by changing the way that research studies get funded, conducted and published. Listener Survey - https://www.apa.org/podcastsurvey Learn more about your ad choices. Visit megaphone.fm/adchoices

We talk a lot on this podcast about the fascinating research conducted by psychological scientists. Today we're going to do something a little different. We're going to explore how that research gets done and how it might be done better. Over the past decade, there has been discussion of what critics have called psychology's reproducibility crisis. The fact that some published research results, including some classic findings in psychology, don't always hold up when other researchers reproduce those experiments. Now psychologists are leading a movement to address that problem by changing the way that research studies get conducted and published. So what exactly is reproducibility and why is it important? If the goal of science is to increase our understanding of the world,

then is the way science is conducted and scrutinized right now helping researchers achieve that goal. And how much can we, as readers and consumers of science information and news, trust published research results? Welcome to Speaking of Psychology, the flagship podcast of the American Psychological Association that examines the links between psychological science and everyday life. I'm Kim Mills. My guest today is Dr. Brian Nosek, the founder and executive director of the Center for Open Science in Charlottesville, Virginia. The Center for Open Science is a non-profit organization that aims to increase the reproducibility of research by building tools to encourage scientists to share data and conduct their work more transparently. Dr. Nosek is also a professor of

psychology at the University of Virginia. Much of his own lab research has focused on implicit cognition, thoughts and feelings that occur outside of our conscious control. Broadly, he's interested in the gap between values and practices, between what people intend to do and what they actually do. Thank you for joining us today, Dr. Nosek. I'm delighted to be here today. Thank you for having me. Let's start with how you became interested in the reproducibility of scientific research. I know there were a series of events around 10 years ago that spurred a heightened scrutiny of psychological research and led to a lot of the work you're doing now. And I'm thinking in particular of cases such as psychologist Deutrick Stappell and Daryl Bem. Can you tell us what happened?

Yeah, in the early part of the 2010s, there were a number of eliciting events that brought to four some issues that have been known for decades, but never really had an impact on daily practice. one was the emergence of a massive amount of fraud perpetrated by Dieddrich Stoppel that came to light because of whistleblowers of his own collaborators and students that had worked with him starting to be suspicious of the origins of the data that were behind some of the papers. But in the end, more than 50 papers of his were identified and then ultimately retracted from the literature as being illegitimate in some way, largely invented data. And the shocking part of it was that these were many of these papers were high-profile results,

results that people knew about it, that people were citing and using to inform their own research. And yet no one over the decades of those papers being written and published, No one had identified any of them as potentially fraudulent, leading to people to say, oh, my gosh, how did we not know this? How did we not recognize and building on that work that actually there was very little basis for at least no actual evidence? Maybe they were true, but perked by chance. The second event was Daryl Bem, a renowned social psychologist, has done wonderful work

throughout his career, got interested later in his career. career in the potential for ESP and other paranormal types of paraps of psychological types of things as possibly being true, and so he started to study them. And he published a paper in one of APA's flagship journals, the Journal of Personality and Social Psychology, showing evidence across multiple experiments of ESP. And this created an outcry, reaction among psychologists because they said how could, you know, a leading journal in our field publish something that is so obviously flawed that can't be true. And then others on the other side saying, well, look at what he reported. It looks like every other paper that we publish in this journal.

It follows the same standards of evidence. In fact, it does so in a really remarkably great way compared to many of the papers that are published in this journal. So what it created was this tension of either we need to now believe that ESP is true because the paper follows all of our rules for what it takes to provide evidence, or we need to question our rules because they allowed something that seems to be a fantastical conclusion to come to light. And then the third event that comes with these two around the same time was a publication of a paper called False Positive Psychology by Laif Nelson, Joe Simmons, and Uri Simonson, where they essentially consolidated this many decades of insight of ways that we might create false positive results, results that aren't actually true through our practices of how we get our data and how we analyze our data. And it's a rhetorically brilliant paper because what it does is it gives the reader,

who has some experience with analyzing their own data, a sense of here's something that you might do, which are very common kinds of behaviors. And here's the implications that has for increasing the likelihood that your findings are no longer credible. And just through a sequence of examples, they show that what would a third? ostensibly be something that might be, create 5% false positives, right? Seeing signal where there isn't anything might actually end up being 60% or higher false positive rates. Like, whoa, most of my findings are not actually there because of how I analyze my data.

So each of these created shockwaves in its own way of something is not right here, and we need to do something about it. Shouldn't the peer review process prevent such flawed research from being published? And sometimes if a study's conclusion are too convenient or too good to be true, aren't they accorded a more stringent level of scrutiny? It would be great if that were true. And the challenges with why it isn't true are a couple of fold. One, peer review is hard, right? I am reading what you did in your research and I'm trying to understand it, trying to recognize limitations.

of it, trying to say, oh, here's some things that you might trust, might not trust. So just that work itself is hard because you had to write down what you did. I have to interpret what you did, and I have to come to some understanding in the process. But let's say I'm really good at it as a reviewer. It's also really hard because it isn't transparent. All I have to review of what you did is what you wrote down. So taking that example of, of these many different analyses that you might do and just report the ones that make the finding come out but actually increase its false positive rate,

if you don't show me all those other analyses that you did, for me to then say, hang on a second, you analyze this 50 ways, you only showed me the one way, like what happened to the other 49 ways? If I don't see that, there's no way for me to catch it in peer review. Right, I didn't like the way those turned out. Yeah, right, right. In the Diedrich Stoppel case,

well, he knows how to write a paper. And how am I to know that the data aren't there, aren't real, if I can't access the data? So one of the big limitations of the peer review process, besides not in being imperfect, and that's just part of the system, no system like that will be perfect because science is hard, and it takes a while to figure things out. But one of the big barriers for effective peer review and ultimately self-correction is a lack of transparency. If you can't see what I did, how I did, can't see the data that I generated, then there's very little chance that you can effectively evaluate the quality of my work.

So your organization is called the Center for Open Science, and I'm hoping you can explain for our listeners, many of whom are not psychologists. In fact, most of whom are not psychologists, what you mean by that? How does it relate to the reproducibility problem that we're talking about right now? Yeah. So the Center for Open Science is a nonprofit technology and culture change organization that has a mission to increase openness, integrity, and reproducibility of research. And it spun out of my lab in 2013.

Jeff Spees and I launched the organization. Our goal was to say to recognize that this is a culture challenge. This isn't a matter of just teaching people how to do it. research in a better way, having them read Uri Leif and Joe's paper and say, okay, don't do those things and your research will be fine. That's helpful, right? We need education and training about how to do the best research we can. But really, as a researcher in the field, I have two major constraints that make it so that the work that I do might be less credible than I want it to be. One is the limitations of my own mind.

I intend to do good work. I intend to find out things that are true. But I have skin in the game. If I find something super exciting, that advances my career. I'm more likely to get a job. I'm more likely to get tenure at my institution. I'm more likely to get awards. And if I find things that don't work a lot and things that are kind of messy,

well, that's not going to do as much for my career. So I have a conflict of interest with my own results. And so I may bring to bear lots of not reasons, but rationalizations for how it is I can justify reporting this finding rather than that finding. Why it is I would take these two studies that looked really good for my phenomenon and decide that those four studies that didn't find evidence for it were actually just bad studies. So I shouldn't report them at all.

So the bias that I bring to the table, I may not recognize in how it is I do my work. And so we have that constraint is how do we help ourselves be more credible by being more transparent and confront our own biases, just like we study biases about how we make judgments of other people in everyday life and implicit bias. Same sorts of principles about scientific evidence. The second major constraint is the structure of the system, how it is that I'm rewarded, right? Because I need to get those exciting findings, though that constraint, that part of the structure of the system, what universities do to decide who to hire and who to promote, what journals

do to decide which findings to publish, what funders do decide who to give grants to, those create a system that constrains my behavior as much as I want to do things well. In order to be in that system, I have to align to some degree to those reward systems. So what the Center for Open Science exists to do is to confront both of these and try to facilitate a change in the research culture. And it does so by providing technology for researchers to make it easy to be more transparent, to show you what my plans were in advance, to show all of the things that I found,

not putting anything, hiding anything away, to making my data available to you to reanalyze it. And we work on the structure, working with publishers and funders and institutions, to try to nudge those policies and incentives so that what researchers are rewarded for is doing rigorous, transparent research, rather than producing exciting findings that may not be credible.

Your center encourages several things, open data, open materials, and pre-registration of studies. What are each of those practices and how are they helping with science being more credible and more transparent? So with open data, if you want to be able to analyze my data to see if I found, what I said I found, simple reproduction, you can't do that unless you have access to my data. If you'd like to see whether my findings are robust, if you change an analyst this way, analysis that way, all different reasonable choices you might make and see if the same findings recur, you need access to the data. Or if you want to combine the data I generated with similar kinds of data by many other researchers to see how robust is the phenomenon when you look in Virginia,

where I am and participants in Iowa and participants in Israel and participants in Russia, then you need to be able to aggregate that data. And you can do some of that metanalytically with just the outcomes, but it's even more powerful to be able to look at the raw data in aggregate. So that open data is part of where we started earlier when you asked about the peer review process, is if we want to be able to self-correct effectively and assess credibility, having access to the original data is very useful. For open materials, you want to see how is it that I generated that data.

You want to know what was the exact format of the survey or the behavioral intervention that I did or what it is that the people did in the setting when I was trying to elicit their behavior. If you don't have access to all of the actual content of the study, how the study was conducted, it's hard for you to then assess the credibility of the data. It's also hard for you to say, you know what?

I want to know if that replicates in this other context. Sure, he showed it in his lab under these very limited conditions, but really if that's meaningful evidence, it should also occur in a corporate environment or it should occur in everyday life in some way. And so I want to test it over in this population or that population. Not having access to materials makes it harder for you to assess how it is I did it and then to redo it. So sharing that, again, provides better, effective,

self-correction, extension, and building on each other's evidence. The last one, pre-registration, has two important roles. The first, what pre-registration means is when I plan to do a study, I write down some information about what the study is going to be and how I'm going to analyze the data, and I post that in an independent repository, so that in principle, anyone can discover all of the studies that I plan to do. And so the two problems that solves is, first, the publication bias. We talked about earlier where I do 10 studies, and I only show you the three that worked. Well, now, if I've pre-registered all of my studies in advance, you can find the other seven. you can say, oh, wait a second, Brian.

Is the evidence as credible as you say it is? Like, what happened to these? And I say, oh, no, no, no. Those were bad studies, you know, and you might agree or you might disagree. But at least you can see the evidence. The other reason for pre-registration is that it makes very clear what was planned beforehand and what I discovered after the fact. And both of those modes of research are super important.

We have confirmatory modes of research where it is I have a hypothesis in advance. This theory makes these kinds of predictions. And so I'm going to collect some data to test the viability of my theory. See where it holds up. See where it's weak. And then revise our confidence in the theory as a consequence. That confirmatory part, it's really important to say in advance, how are you going to test it? And so then afterwards, I don't say, oh, actually my prediction is a little bit different.

and I think it only alternatives. So I don't do that sort of reconstruction after the fact. The other part of that is that the evidence itself becomes easier to assess because then once I've seen the data, I am free to go into it and explore and discover new things and find out what's in the data that might shape new ideas, might generate hypotheses for me to test next. And that's where a lot of the new discovery in science occurs, right? It's not from what we're testing from existing hypotheses or theories. It's when the data go totally awry and send us in a whole new direction.

Like, wow, I didn't think that was going to happen. And that is super important. But it's also very important to recognize that it's more uncertain. When data go in a different direction than expected, what we would ordinarily want to do is follow up on it, develop some hypotheses and test those hypotheses for why it went in that new direction. So pre-registration helps keep clear when we're in these two different modes of research so that we can respect and value both of them.

So if I pre-register a study and I say I'm going to conduct these 10 experiments as part of my study, does this commit me to using all of them or maybe I find that six out of the 10 confirm the hypothesis and the other four don't. But I've told everybody that I did it. I mean, can I still submit the paper and leave out those four? How committed am I to including everything? Pre-registration doesn't force you to do anything at all. It just makes it so that it's transparent what it is you did.

So this happens all the time, right? I plan a number of studies. I pre-register them in our lab, and then a couple of them go sideways, right? Like, even just implementing the study, I had a plan for how we were going to do it, and I can't do it that way. So, like, it is a broken study. But many times it's more like what you described. Like, it just doesn't turn out the way that I thought. And so I can say, well, these are the ones I'm going to report.

But the value of pre-registration is in those other ones are also available. They're also accessible for you as a reader or for anybody in the community to say, well, I want to, want to consider that evidence too. And so the goal of pre-registration is not to say, here are these strict rules that you now have to follow. It's here is a way to articulate and make plain what your plans were and compare that to what you did. And I can tell you why I had to make those changes. We planned to do this study with 1,0006-month-olds to measure their attention towards this versus that get into the study. Turns out it's hard to do research with six months old. We thought a thousand that was crazy. We're going to do it with 20, right, or whatever it is. And so at least I can

then follow up and say, here's what we had to change. This is why we had to change it. And this is what we think implications it has for how confident you should be in our results. And then you as a reader get to say, hmm, I'm glad I can see that. I'm glad I can understand how it is he got from here to there, and I either agree with his explanations and the reasons that he changed or I disagree. But all of that is enabled by the transparency that pre-registration provides. It does not replace reasoning in any way. These are pretty radical changes to the way that research is being done. And I'm just wondering, are these practices catching on? Are researchers agreeing, going along and saying, yeah, you know, this is what we should have been doing all along.

I'm all in. Yeah. So you opened saying that the critics have firm called this a reproducibility crisis. And I think the term is wrong. And there are many others that have said this as well, that really it's a renaissance. Leif, Joe, and Yuri, the ones who did that initial paper, they have a great paper recently called the Psychology's Renaissance. And it's a very optimistic paper in that it argues, that what these troubling episodes produced was of self-reflection in the psychological community about psychological processes, about how humans and the cultures that they inhabit are themselves influencing sometimes for the negative, but also potentially for the positive, the quality of science and how we produce and understand findings.

So as a consequence of that, there has been a very, broad engagement in the psychological community by bottom-up researchers themselves saying, we need to change this, and so we're going to start to adopt some of these new behaviors, sharing data, pre-registering research, and otherwise, and some top-down, where publishers and editors, funders, and institutions are saying we need to change our reward systems, how it is that we encourage the most credible research that we can. Now, it's 10 years since some of those eliciting events occur that may brought this really to the fore. And it's, you know, culture change takes a long time, but a lot has changed.

So just as one example, we operate the open science framework. This is part of what the Center for Open Science does, is this technology where researchers can use this as a collaborative management service, the OSF, where you can share your data and your materials, you can pre-register your designs, etc. The number of users of the OSF, which is open to any discipline, not just psychology, but is more than 300,000 now since it opened in 2012, right? It's been increasing in exponential rates every year. The number of registered studies is in the many, many thousands at this point. Millions of data files have been shared on the OSF.

And all of these show this exponential growth, and that is largely driven. by the psychological community. And I think one of the great benefits of psychology confronting this is that really the challenges that are trying to be solved are psychological challenges, right? Challenges with the individual in their behavior, challenges with the system and how individuals react to that system. And so as a consequence of psychology leading the way on a lot of this work, there's a lot of similar sorts of self-reflection and change happening in other disciplines. And so the reproducibility

movement has spanned into biomedicine, into ecology, and evolutionary biology, into pre-clinical disease research, into education research, and business research. So that, I think, is going to be the real, the next 10 years is really the full embrace of these challenges across different scholarly communities. So replicating experiments that have already been conducted isn't seen as a career enhancing move in the sciences. What, if anything, is being done in an attempt to change this, to encourage people to try to reproduce studies? Yeah, replication is an important part of science because it's how we establish the reliability of findings. We rarely do a study to, to make a statement about a historical fact, like, did this thing happen or not?

That isn't usually what a scientific hypothesis or claim would be. Often scientific claims are about if these conditions occur again, then this kind of thing will happen again sometime in the future, right? We're making statements that have general application, make predictions about observations that haven't been made yet. Even if they're about events in the past, we haven't observed them yet, the hypothesis would make an anticipation of what's going to a way. occur. So replication helps with assessing whether those predictions come true when you try to do them

again in similar sorts of contexts as the previous research. But it is entirely valued because of those reward systems that we've been talking about focus on innovation and novelty at the expense of verification credibility. I gain more stature by having a phenomenon with my name on it. The Nosec phenomenon of X, right? Whereas the replication is me trying to assess the credibility of somebody else's claim, usually, could be my own too, but isn't as valued. So there are some things that are changing. One is just a growing appreciation of the role of replication in research. So journals are changing their policies to be more welcoming of replication studies, recognizing that it is important contribution to science. That opens the door. It is becoming easier to do replications

because researchers are sharing their materials and their data to a greater extent. So now if someone wants to replicate somebody else's findings, they can get a leg up on figuring out how they actually did their research more quickly and easily because of greater sharing. And now there are also new publishing models that lower the barrier to trying replication studies. And one of them is called registered reports. And the idea of registered reports is to really fundamentally change how is it that researchers get that key reward, the publication. In the standard model, I do all of my research, and then I write it up and I submit it

at the end. Here are all my findings. And then the peer reviewers evaluate whether it's credible and whether it should be published. In registered reports, what I submit to the journal is my plan. Here's the question that I'm going to ask. Here's the background material. Here's some initial evidence to say that this is worth studying. And here's my study design and how I plan to analyze my data when I get it.

And the peer reviewers evaluate, is that an important question? And is the methodology an effective test of that question? And if they agree, then the journal commits to publishing my findings regardless of what happens. and this fundamentally changes what my goal is as the author. Instead of my goal being make the findings as flashy as exciting and as innovative as possible, my goal is ask the most important questions and design the best methods I can to test them, and that will get me the publication. And then the results are the results.

I just do the research and then I publish whatever happens. And that changes a lot of the things that we've been talking about, right? It eliminates publication bias because the studies are going to get out there, whether they come the way I hoped they would or not. They're all going to be reported because they've been pre-committed in advance. They're going to get reviewed in advance so the rigor can be enhanced compared to reviewing it afterwards where you say, oh, well, you screwed up X, Y, and Z. And I say, great, well, the studies are all done. I can't change that now. But the relevance to replication, which is where you started in asking, is that it also means that I can propose.

to do replication studies and get commitments from the journal that they're actually interested in publishing it before I do all of the work to do the replication. And that's a lower-barred entry, right? I can say, well, it's fine. I don't know if they want the replication, but at least I can just write up a proposal for it. And then if they don't want it, then I don't have to do it, save myself some time. So all of those are shifting some rewards for replication, but then also overall how it is people are rewarded in general. Are some of these open science ideas filtering down to training yet? I mean, are undergraduate and graduate students being taught these new methods? Yes, very much so. And that's one of the more exciting elements is that that's really how these sorts of things

get sustained is where the training that gets integrated into curricula at the graduate and undergraduate levels really starts to embed these new practices. And that's happening in formal cases like methodology instructors at their universities, building them into their coursework. And it's also happening in informal ways where people who are advocates for open science in the community, and it's a very robust, lively community are developing materials and making them available for broad dissemination. So, for example, there is a society called the Society for the Improvement of Psychological Science, SIPs. It's improvingps.com. And the goal of that is to be a very much of a community-based ground swell of ways that we can do better.

That might be creating training materials, it might be delivering training, it might be ways to try to change some of these reward systems or anything else. And that community has been fabulous at generating lots and lots of different new training materials or education devices or ways of explaining some of these challenges. There are projects like the Fort Project or the Open Scholarship Knowledge Base that aggregate these materials and then make them more easily available for people to search and discover to bring into their class or to learn themselves. So that's really a big part of the engine of what is helping to sustain this change

is the community providing training for itself about these issues. So another fascinating thing that you're exploring at the center is faster ways to predict which research results are likely to replicate and which aren't. And I know you've looked at something called prediction markets and also at using artificial intelligence to predict reproducibility. Can you talk about those tools and how they work? Doing a replication is a lot of work. So if we want to know whether a finding is trustworthy or credible, saying, okay, we've got to run it again and get all, another set of participants and spend another year running the study and analyzing it, wow, that's a lot of effort to assess credibility.

And that's essential and important and will continue to be part of the process for as long as we can envision what science will be. But it would be useful to have other indicators that gave us a more rapid assessment of the credibility of finding. So peer review is supposed to provide that, but that still is a lot of effort, right? the reader has to read it all and decide do they think it's a credible finding or not. Wouldn't it be awesome if we could have machines read the papers and decide whether those findings were credible or not? Yeah, that'd be great. Then it would be a lot less work. It'd be a lot

faster. But of course, can machines actually do that? I don't know. So that is sort of the goal of this program that we are part of. And there's hundreds of collaborators involved in this program. It's called score. The idea is to see if it's possible to create algorithms, artificial intelligence, that will score papers and particular findings or claims from papers on their credibility. And if that works, then it provides an initial heuristic that readers, like you or me, can make about papers. So if I pass my own paper through the machine to say what's the scores on it, I might learn something about where the evidence, at least from the machine's perspective, is stronger or weaker. And then that might help guide me for saying,

oh, I need to bolster that with some additional citation evidence or other evidence that already exists. I might need to do another study there, or I might need to sort of look and at least look closer. Right, you as a reviewer might do the same. Or as an editor might say, oh, it would be really useful to make sure we have an expert on that topic because the machines gave an initial indication of this is an area of high uncertainty. So that's the goal is to create some tools like that to help facilitate and direct resources for the deeper dives into the evidence credibility. So why do we think we can accomplish that? Well, there's actually some evidence that it might work. So the first bit of evidence is that we in collaboration with a number of others, particularly

a great group of economists, have run prediction markets where people bet on whether findings would replicate or not. It gave them money and said, you can buy shares. You know, buy a yes share. If it succeeds in replication, you get paid. If it fails, you lose that money. You buy no shares. Then you're betting against it, et cetera.

And those markets were quite successful at anticipating replication outcomes when done in these systematic studies, suggesting that people have information that can anticipate whether findings are likely to replicate or not. And similar findings that observe with surveys and other kinds of elicitation techniques to see assess credibility. And then a few different teams have tried with machines, is train machines on reading, papers, give them a training set, and then give them outcomes of whether those successfully replicated or not, and then test them with papers that have replications associated with them that the machines had never seen before. And they performed similarly to the humans in predicting which papers were likely to produce

replicable findings and which were likely to fail to replicate. So those initial studies were sort of the basis of doing this project at some. scale, the score project, which is doing this across the social behavioral sciences and involving thousands of papers and hundreds of replication studies and prediction markets and elicitations of confidence and multiple different artificial intelligence groups, all trying to see if we can really create robust automated prediction of credibility. And if it works, then it has lots of potential applications and lots of potential risks of overuse and inappropriate use. So it'll be a very interesting challenge to navigate of how one would use these appropriately, responsibly,

and effectively to help address research credibility and improve it. Last question for our listeners who, as I mentioned earlier, are just people who are interested in the science of psychology, not necessarily scientists or researchers themselves. How can they think about the issues that we've discussed today? Is there some way for them to know whether a paper has gone through the process of open data, open science, registration? How are you getting the word out there to more of a lay audience, particularly people who just get a lot of their science by reading more popular publications, for example, or listening to other podcasts? Yeah, no, it's a big challenge is how do we create effective signals of credibility for different degrees of engagement with the research finding. I am a practicing scientist and I only read a very, very tiny portion of the research literature, right?

So like an everyday person, most of what I know of science, I also get through the same mechanisms, right? Reading in the newspaper or otherwise. I can't read all of those papers. There's millions of papers. So that it is true not just for the layperson, it's true for every person that we need more effective dissemination of the credibility, the confidence that we should have in different findings. Part of that is education about how science actually works, right, which is initial findings are always uncertain. When that first study that comes out and says, coffee is terrible for you, you know that next year of study is going to come out and say coffee is the best thing. you could ever drink. Don't stop drinking coffee. And then on and on and on, right? Because it's

really hard to figure it out. You can't randomly assign people. You're going to drink coffee now is the only thing you're going to drink for the rest of your life. And you're going to drink, not coffee, for the rest of your life. And then we'll see who lives longer. And you're just, the kinds of studies to actually unpack these phenomena with confidence aren't going to be done. And so the evidence accumulation for a lot of the things that we read about is slow and uncertain and subject to change, and that's fine. In fact, it's ordinary. And so the one caution, I think, that just any of us should have is not expecting more from science than it can deliver in the short term. Science works on long time scales, and the confidence that we have is accumulation over time and building of evidence

And I shouldn't say over time necessarily, it can be done very quickly with lots and lots of resource investment, right? COVID vaccines is a perfect example of that. The vaccines themselves, the amount of evidence gathered in a short period of time is astonishing. Of course, it's billions of dollars got put into making sure that we got that evidence. He got it credibly very quickly. and those particular vaccines are built on 30, 40 years of accumulated research on the basic processes of how it is that we could develop those more quickly. So it feels like it happened all the sudden,

but it actually happened over many, many, many years of accumulating process and insight and evidence. So I got a little bit off track there, but the basic point, like, who can't think about COVID these days, right? Right, right. But the basic point is that just is caution, is science is slow, and so take evidence as tentative. And the more signals that you have about transparency, the better. And that doesn't mean that it's right. It does mean that it's possible that you can self-correct or obtain accurate assessments

of confidence more easily. Well, this has been really interesting. I appreciate you're talking to us today, Dr. Nose. and I learned a lot about the renaissance that we're experiencing right now in scientific research. Thank you. Good. Thank you for having me. You can find previous episodes of Speaking of Psychology at speakingof psychology.org

or on Apple, Stitcher, or wherever you get your podcasts. And please leave us a review. If you have comments or ideas for future podcasts, you can email us at speaking of psychology at APA.org. That's speaking of psychology, all one word, at APA.org. Speaking of Psychology is produced by Lee Wynerman. Our sound editor is Chris Kondyin. Thank you for listening. For the American Psychological Association, I'm Kim Mills.