Science Friday - What To Do When Your Hypothesis Is Wrong? Publish!

Episode Date: July 1, 2024

Most scientific studies that get published have “positive results,” meaning that the study proved its hypothesis. Say you hypothesize that a honeybee will favor one flower over another, and your research backs that up? That’s a positive result.But what about the papers with negative results? If you’re a researcher, you know that you’re much more likely to disprove your hypothesis than validate it. The problem is that there aren’t a lot of incentives to publish a negative result.But, some argue that this bias to only publish papers with positive results is worsening existing issues in scientific research and publishing, and could prevent future breakthroughs.And that’s where the Journal of Trial and Error comes in. It’s a scientific publication that only publishes negative and unexpected results. And the team behind it wants to change how the scientific community thinks about failure, in order to make science stronger.Guest host Anna Rothschild talks with Dr. Sarahanne Field, editor-in-chief of the Journal of Trial And Error, and assistant professor in behavioral and social sciences at University of Groningen.Transcripts for each segment will be available after the show airs on sciencefriday.com. Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

Why is it a bad thing if scientists only publish studies with positive results? It was so bizarre for researchers to think about actively submitting their negative results to a journal that was open to receiving them. It's Monday, July 1st, and you're listening to Science Friday. I'm Cyfry producer Deep Peter Schmidt. On a show like this, we feature a lot of studies that have positive results. Say you hypothesize that a honeybee will favor one flower over another, and your research backs that up.

That's a positive result. But what about the papers with negative results? Researchers know that you're just as likely to disprove your hypothesis than validate it, but there's not a lot of incentive to publish failed experiments. And some argue that this bias to only publish positive results hinders scientific progress. And that's where the Journal of Trial and Error comes in. It's a scientific publication that only publishes papers with negative or unexpected results. And the team behind it wants to change how the scientific community thinks about failure

in order to make science better. Here's guest host Anna Rothschild with more. Dr. Saranfield, editor-in-chief at the Journal of Trial and Error, an assistant professor in behavioral and social sciences at the University of Hrningen in the Netherlands. Welcome to Science Friday. Thank you. I'm so happy to be here. So you're a researcher. For those listening who aren't in science,

how much more likely is it that you end up with a negative than a positive result at the end of your study? Oh, honestly, I would say about one in every two studies are negative. Right. I mean, is that frustrating as a researcher? No, because for me, honestly, it's expected. This is part of the scientific process. It's iterative. It's not linear as some people might expect. It doesn't go from hypothesis to method to result to lovely finding on social media as simply as one might expect. There's a lot of trial and error involved. And so having an unexpected or maybe a disappointing finding is kind of expected in and of itself. Right. And also, I mean, this is something you're devoting your life to.

So even a negative result is a new piece of information. Absolutely, which is why we're doing what we're doing. We want to make sure that we can learn from the mistakes that we make in science, because a lot of the time that's what they are, the mistakes that we've made there, mistakes that we can learn from, they have information value. And just for our audience to know, you could spend years of research on a study and get a negative result. Absolutely. Yep. So you've said you want to highlight the ugly parts of science. I like that. Why is that? Well, the gap between what is researched and what is published is much bigger than what the lay public might expect. So we do loads and loads of studies. And only a portion of those.

the ones that come out really pretty are published. But what that means is that we end up with a scientific literature that is all the pretty stuff and none of the stuff that went wrong. And what's a problem with that is that we can't learn from the things that went wrong. So you have researchers that are going out and doing the same studies over and over again when actually other people have gone before them and have failed to find results, but they just didn't publish them. So people are wasting time and money, which in research are very scarce resources, to try and do studies that will never come out right anyway. And so it's a shame that we can't learn from those. So we attempt to close this gap between what we research and what we publish. Right. In the past, have you,

as a researcher, tried publishing negative results and run up against pushback? No. And I am very rare in saying that. The good thing about the discipline that I am in, which is meta-science, meaning that we use the scientific method itself to study science, we're very familiar with the things that can go wrong, and we're very open to publishing negative results and unexpected results compared to a traditional older discipline in science. So I'm actually really lucky, and in fact, I've published no results quite successfully and easily, so I'm very lucky. Yeah, that's quite rare, right? Amongsts. your colleagues, maybe in different disciplines, how often are they sort of turned away from publishing

in a journal if they want to publish something negative? Oh, constantly, constantly. And the majority of disciplines have this issue, which is exactly where our journal comes in, because, you know, there are so many good studies, and I mean, good quality, well-designed studies that should be informative, that can't be published simply because they don't support the hypothesis that was in the study. And so there's an enormous, enormous amount of literature that should be out there that's not. So this is very common. Why don't journals tend to publish negative results? They're not sexy. They don't get as much readership. If you read, okay, social media, you see a really cool looking study that says something along the lines of, oh, drink a glass of red wine,

you don't have to go to the gym today. That's so much cooler than saying, have no idea if red wine has any positive effect on the body. That's so much cooler to read, right? The former, I mean, rather than the latter. And it's also just the case that reviewers can make a little bit more sense. So reviewers are the people who look into these papers and check them for mistakes and that kind of thing. Reviewers are much more able to make heads or tails of a study that looks clean and tidy and linear. Because sometimes when you have a null result or an unexpected, result, you kind of go, huh? What does this mean? You know, and it takes a lot of work to kind of analyze and understand what you can learn from that. Whereas if you see a nice, easy, shiny study that went

exactly as planned, it's like, oh yeah, we know what to conclude from this. Right. I mean, and then there's this issue on the flip side, right, where researchers also don't want to publish negative studies. Why is that? Absolutely. So you're talking about what's called reporting bias. And indeed, it's just a matter of most people know as a researcher that it's so much harder to get published. So you conduct a study, it went pear-shaped to one reason or another and you think, oh, God. Okay, so I can either spend two years of my life trying to get this thing published, which may never happen, or I can just turf it. Start over again, give it another go, try a different variable, and then you're going to waste a lot less time because the thing is, a lot of us are having trouble getting tenure and making

sure we have job stability. And so we're really incentivized to have a good CV and what makes a good scientific CV at this point, publications. So you want to spend your time working on something that's going to get published rather than something that's going to just cost you time and effort and never get on the scientific record anyway. Right. And it takes a long time to actually pitch and submit to a journal only to then be turned down. So you want to sort of like send out the stuff that you feel like has a good chance of actually making it, right? Absolutely. What's your pitch to researchers to go ahead and try to publish those negative studies anyway? These kinds of negative studies, they have information value. So what I would say is if you have a good quality study, a study that's been designed well, that has a good sample. So that means a lot of participants, because that's going to be more informative than a small participant pool. And it just went pear-shaped for one reason or another. Write it up properly like you would a normal study. Provide a good reflection.

on what might have gone wrong, and then send it right on over to the journal of trial and error. So that's what I would say, but let me just provide a quick caution. We don't want to work with authors to produce bad studies, right, that went wrong, but no one knows why. We're not like Rumpelstiltskin, where we turn straw into gold. We do want to work with something that's high quality, so I would just provide that as well. But not only do we have the journal of trial and error, we also have another journal that operates as well called the, I believe, the journal in support of the null hypothesis. So that's

another outlet you can publish in if you're thinking about this. Well, like, let's talk about some of the use cases in the future of this. So there's been a lot of talk about integrating machine learning and AI into research, which can in theory, you know, analyze a vast trove of data and pull away new insights that would have been too laborious to do by hand in the past. And there's a lot of promise here for big breakthroughs, you know, in medicine, for example. How accurate can machine learning models be if they're only trained on positive data? It's a real concern because, okay, so say you're part of the lay public and say you have access to the literary record, the studies that have been published, what you would theoretically learn is a portion of the

full story, right? You would be learning about a subset of what's really going on. And the exact same thing is the case for any kind of machine learning methods where the model is being trained on what's available. That's exactly what you'd get. A completely biased data set is what's being basically trained on. And that's a massive problem. That concerns the pants off me, to be honest, because I can see so many implications of this and I don't know how to get around it, but it's a massive problem. Could you maybe give a specific example of a study that would maybe use machine learning in this way and where negative results would be really useful? I think anything that has to do with, for example, drug trials,

there would be a potential, I think, for using very large-scale data to get some insights from existing drug trials that the FDA will eventually end up approving. And as it currently stands, so many drug trials are conducted and loads of them are false, but then you get, say you conduct five drug trials on a particular drug, one of those will be positive and it will show efficacy of the drug and the other four do not. So I can imagine in a case here, you would need that model to really learn as much as possible about the true state of certain drug efficacy and it just cannot because the information is not there and so it gets a completely false sense of what drugs are actually have efficacy,

simply because of missing information. I mean, if you think about drug trials, that's such a heavy thing with such huge importance and impact for the public. And then the likelihood of it actually being working is so low, but we just don't know that. Yeah, they're just huge gaps. I understand that when the Journal of Trial and Error first started in 2008, even the editorial team were having trouble getting researchers to submit negative results to the journal.

Has that changed? Yes, it definitely has. Part of the reason that they had so much trouble in the start, and I say they because I didn't join until 2021, they had so much trouble because it was so bizarre for people to, for researchers to think about actively submitting their negative results to a journal that was open to receiving them. It's a matter of getting the message out and saying,

hey, where this outlet that actually wants, your studies that went wrong, not that you'll have to fight tooth and nail to get even a review. It's just so outside of what literally decades of a developed research culture kind of says you would expect. So it's a very, very, I would say, avant-garde in that way. And it still is, back in 2018, we were only about seven years out of the crisis of confidence or the reproducibility crisis. And so it wasn't until that really started to blow up that we started to say, hey, there's value in no results, there's value in unexpected findings. And then from then to go, hey, we can actually publish them and learn from them. So it's a matter of, you know, academia being slow on the uptake and just getting word out, hey, there's an outlet that does this.

Are more mainstream journals now also encouraging people to submit their negative results? Absolutely. Yeah, we're seeing that a little bit as well in, certainly in some of the meta science journals. We have a lot of journals that also accept the registered report format. I'm not sure if that's come up on SciFRI before. But registered reports are a type of article in which you've pre-registered your plans for a study, and that plan has been peer-reviewed before data is being collected. What that means is that if you can do or if you plan a really good study and you get

unexpected results, they'll probably get published anyway. we actually give a guarantee called an IPA. Sounds like a beer, but it's even better. It's, we will publish your study regardless of what you find. What that means is that a lot of journals now are publishing null findings simply because those authors had IPA. So those authors had a promise, we will publish this study if it's of good quality despite the results.

And so that just ends up by default than more null results and unexpected findings are actually being published and in good outlets too. That's so great. Yeah, it really is better than a beer. Yep. This was one example of a sort of change in the publishing industry. What other sort of larger structural forces or incentives in scientific publishing need to change in order for more negative results to be published? I think one thing that really needs to change is kind of how we see the research life cycle.

So like I said earlier on in the piece, we, and certainly the public thinks of it this way too, we think of science in a very linear way. Like we have theory and hypotheses and that flows into a plan for a method and then you collect data and that's your results and then you interpret the results and then you publish it. That's how it kind of looks like. That's how, at least at the start of our careers, we're kind of taught it goes. But in fact, the whole story in real life is very different. There's a lot of stuff that goes wrong, a lot of stuff that just doesn't quite work out,

a lot of choices that get made and changed throughout. the process. And what this means is that the final published study that is supposed to kind of reflect that process is very static. It's very linear and it gives a very kind of stiff quality to a process that's actually quite living, dynamic and very iterative. And so what I think needs to change and something that is changing is the publishing format itself. So instead of having these, you know, 20, 30 page PDF that we read, There are other options for publishing. For example, modular publishing is an example of this where you publish modules.

So literally chunks of a study that are all connected together. And that's your study. So they each have DOIs, they're versioned. And they kind of break up this static block of text into sort of a more living and dynamic structure that reflects better the research process. So that's an example. So your publication calls itself an open access journal. redefining failure. What have you personally learned about redefining failure over your years of being

the editor-in-chief? Do you look at failure differently now? I absolutely do. You know, I've been in academia for about 10 years now, and I was part of the group of people, that's the majority of scientists who were sort of worried that studies wouldn't pan out the way I'd like them. and although working in meta science in that time has kind of helped me come away from that mindset now that I'm actually face to face, so to speak, with some of these failed studies and having to really draw information out of them, I'm really starting to embrace the messiness of science instead of kind of shying away from it. I'm going, hey, this is actually really cool, it's fun, it's interesting, it's challenging.

I love that. Yeah. For scientists who are listening, when you see initiatives like ours, like the Journal of Trial and Error, if there's any possibility to donate to these causes, that's really important because we're a Diamond Open Access Journal, for example, in our pace. And the only reason that we don't have to get anyone to pay anything is that we're helped by donations. So if you're a scientist and you see an initiative that you think is really important,

consider seeing how you can support that initiative to help it keep running. Because that's one thing that we really struggle with in, you know, in meta science and in these kinds of things is to get the support to continue doing what we're doing. Well, thank you so much for taking the time to explain this to us, Saran. No worries, not at all. Dr. Suran Field is editor-in-chief at the Journal of Trial and Error. And that's all the time we have for today. Lots of folks help make the show happen, including Kathleen Davis. Diana Plasker.

Beth Rami. Danielle Johnson. Tomorrow we'll learn about why kids are getting their peer. earlier than previous generations. But for now, I'm Cyfry producer D. Peter Schmidt. See you next time.