Freakonomics Radio - 572. Why Is There So Much Fraud in Academia?

Episode Date: January 11, 2024

Some of the biggest names in behavioral science stand accused of faking their results. Last year, an astonishing 10,000 research papers were retracted. We talk to whistleblowers, reformers, and a co-author who got caught up in the chaos. (Part 1 of 2) SOURCES:Max Bazerman, professor of business administration at Harvard Business School.Leif Nelson, professor of business administration at the University of California, Berkeley Haas School of Business.Brian Nosek, professor of psychology at the University of Virginia and executive director at the Center for Open Science.Joseph Simmons, professor of applied statistics and operations, information, and decisions at the Wharton School at the University of Pennsylvania.Uri Simonsohn, professor of behavioral science at Esade Business School.Simine Vazire, professor of psychology at the University of Melbourne and editor-in-chief of Psychological Science. RESOURCES:"More Than 10,000 Research Papers Were Retracted in 2023 — a New Record," by Richard Van Noorden (Nature, 2023)."Data Falsificada (Part 1): 'Clusterfake,'" by Joseph Simmons, Leif Nelson, and Uri Simonsohn (Data Colada, 2023)."Fabricated Data in Research About Honesty. You Can't Make This Stuff Up. Or, Can You?" by Nick Fountain, Jeff Guo, Keith Romer, and Emma Peaslee (Planet Money, 2023).Complicit: How We Enable the Unethical and How to Stop, by Max Bazerman (2022)."Evidence of Fraud in an Influential Field Experiment About Dishonesty," by Joseph Simmons, Leif Nelson, and Uri Simonsohn (Data Colada, 2021)."False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant," by Joseph Simmons, Leif Nelson, and Uri Simonsohn (Psychological Science, 2011). EXTRAS:"Why Do We Cheat, and Why Shouldn’t We?" by No Stupid Questions (2023)."Is Everybody Cheating These Days?" by No Stupid Questions (2021).

Just over a year ago, Francesca Gino was, there's really no other way of putting it, she was a superstar. An academic superstar, at least. She was at the center of everything. Being a prestigious faculty member at Harvard and all of her public speaking and her books. Her reputation was perfect. She was synonymous with the highest levels of research in organizational behavior. She's just a giant in the field.

The field in which Gino is a giant, where her reputation was perfect, is variously called behavioral science or decision science or organizational psychology. According to her website at the Harvard Business School, where she has been a professor of business administration, Gino's research focuses on why people make the decisions they do at work and how leaders and employees can have more productive, creative, and fulfilling lives. Who wouldn't want that? Gino became a superstar by publishing a great number of research papers in academic journals, as well as a couple of books.

Her latest is called Rebel Talent, Why It Pays to Break the Rules at Work and in Life. She produced the kind of camera-ready research that plays perfectly into the virtuous circle of academic superstars. A journal article is amplified by the publisher or university into the mainstream media, which feeds a headline to all the firms and institutions who are eager to exploit the next behavioral science insight. And this, in turn, generates an even greater appetite for more useful research. The academic who is capable of steadily producing such work is treated almost like an oracle. There are TED Talks to be given, books to be written, consulting jobs to be had. Francesca Gino, for instance, gave talks or consulted for Google, Disney, Walmart, for the U.S. Air Force, Army, and Navy,

and many more. But that's all over for now. In July of 2023, Harvard Business School, responding to an analysis by academic whistleblowers, investigated Gino's work and found that she had, quote, intentionally, knowingly, or recklessly committed research misconduct. Gino was suspended without pay. She then sued Harvard and the whistleblowers. Those same whistleblowers have also produced evidence of what they call data fraud by an even more prominent behavioral scientist, Dan Ariely of Duke. Ariely has enjoyed the spotlight for many years, going back to his 2008 book,

Predictably Irrational, The Hidden Forces That Shape Our Decisions. Duke is said to be finalizing its investigation into Ariely, although that's been going on for a while now. And when it comes to academic fraud, universities have a habit of downplaying charges against their superstar professors for the obvious reason that it reflects poorly on them. Meanwhile, Dan Ariely's book lives on as the basis for a new NBC crime drama called The Irrational. It stars Jesse L. Martin as a professor who uses behavioral psychology to help solve crimes. Hey, what was it you did back there? Paradoxical persuasion. I overly embraced his idea to force him to think it through enough to realize it was

a terrible idea. And how did you know he wasn't going to pull the trigger? It works about 95% of the time. And the other 5%? There's always our liar, Marissa. Dan Ariely and Francesca Gino have both maintained that they never fabricated data for their research. Neither of them agreed to record an interview for this episode, but one of their co-authors did. Certainly, I felt a moral obligation to correct the record.

Today on Freakonomics Radio, we'll hear from him as well as the three data detectives who blew the whistle on Gino and Ariely. So I would say I have, on the falsity of the findings, I don't have reasonable doubt. But this is a much bigger story than two high-profile cases in behavioral science. We will get into the incentives that produce academic fraud. If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat. We will hear what's being done to change that and, most important, why this matters. Because the research fraud in academia extends well beyond academia, and it has consequences for all of us. The first of a two-part series begins right now.

This is Freakonomics Radio, the podcast that explores the hidden side of everything with your host, Stephen Dubner. I rarely do this, but today I'm going to start by reading a couple sentences from Freakonomics, which Steve Levitt and I published in 2005. Cheating, we wrote, may or may not be human nature, but it is certainly a prominent feature in just about every human endeavor. Cheating is a primordial economic act, getting more for less. So when you think about it, why shouldn't we expect cheating even among scientific researchers? Consider this. Today, it is thought that Ptolemy, the second century Greek astronomer, faked his observations to fit his theories. And a new study in the journal Nature found that last year, more than 10,000 research articles

were retracted, easily breaking the old record. While a lot of the recent headlines are about scholars at big-name American universities, the countries with the most retractions were Saudi Arabia, Pakistan, Russia, and China. Fraud has existed since science has existed, and that's primarily because humans are doing the science. And people come with ideas, beliefs, motivations, reasons that they're doing the research that they do. And in some cases, people are so motivated to advance an idea or themselves that they are willing to change the evidence fraudulently to advance that idea or themselves. That is Brian Nosek, a psychology professor at the University of Virginia. In 2013, he founded the Center for Open Science, a nonprofit that tries to improve the integrity of scientific research. Just of private sources such as the John Templeton Foundation, Arnold Foundation, and many others.

And that diverse group of funders, and it's quite diverse, I think share the recognition that the substantive things that they are trying to solve won't be solved very effectively if the work itself is not done credibly. In other words, the stakes here are high, higher than just individual academic researchers trying to advance their careers. If your goal is to improve medicine or transportation or immigration policy, any area where decisions are based on academic research, you don't want that research to be compromised. There are specific cases where a finding gets translated into public policy or into some type of activity that then ends up actually damaging people, lives, treatments, solutions. One of the most prominent examples is the Wakefield

scandal relating to development of autism and the notion that vaccines might contribute. And that has had an incredibly corrosive impact on public health, on people's beliefs about the sources of autism and the impacts of vaccines, et cetera. And that is very costly for the world. There's also a local cost in academic research, which is just a ton of waste. So even if it doesn't have public downstream consequences, if a false idea is in the literature and other people are trying to build on it, it's just waste, waste, waste, waste. There's also the idea that as much as universities worry about their students cheating, like using chat GPT to write a paper, what kind of example are their professors setting? And there's one more big reason this story is so frustrating, and that has to do with the standards

of academic research. The general view, at least this is the view that I have long held, is that academic research exists in a special category. It is a fact-finding coalition that operates under a set of rules built around the accurate gathering and analysis of data, with the entire process subject to fact-checking and peer review. Good journalism operates under similar rules. The New York Times has an old mission statement I've always liked. It goes, to give the news impartially without fear or favor, regardless of party, sect, or interests involved. I've always thought this mission applies

to academic research as well, that it's meant to be not only accurate, but free of personal or financial interests. These research papers aren't being written by some political official or management consultant or equity analyst. They're being written by someone so devoted to their field of research that they went through the hell of getting a PhD in order to spend their days doing that research. But the fact that Brian Nosek has been kept very busy with his Center for Open Science suggests that my faith in academic research has been misplaced. I asked Nosek to walk me through how he went from being a researcher himself to being a new sort of referee. Yeah, I have always had an interest in how to do good science as a principled matter. And in doing that, we in the lab would work on developing tools and resources to be more

transparent with our work, to try to be more rigorous with our work, to try to do higher powered, more sensitive research designs. And so I wrote grant applications to say, can we make a repository where people can share their data? This is like 2007. And they would get polarized reviews where some reviewers would say, this would change everything. It'd be so useful to be more transparent with our work. And others saying, but researchers don't like sharing their data. Why would we do that? And why would researchers not want to share their data? I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh, my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career. Gosh, I do feel so naive because I've spent time in this ecosystem for many years now. And what you're saying just sounds wrong.

And it sounds selfish. And worst of all, it sounds like it goes against the mission of the scientific goal, which is to discover and distribute knowledge to the world. And that kind of offends me, I have to say. Yeah. discover and distribute knowledge to the world. And that kind of offends me, I have to say. Yeah, and here's the irony, is that almost every academic would say, of course, science is supposed to be transparent. Of course, we're doing research for the public good.

Of course, this is all to be shared. But come on, Pollyanna, we live in a world, right? The reality here is that there is a reward system and I have to have a career in order to do that research. And so, yes, we can talk all about those ideals of transparency and sharing and rigor, reproducibility. But if they're not part of the reward system, you're asking me to either behave by my ideals and not have a career or have a career and sacrifice some of those ideals. You can see how these incentives create a problem. If a system has a built-in bias against transparency, not only will there be less transparency, but also more opportunity to cheat.

Nosek and his colleagues set out to address this by trying to replicate the result of papers that had already been published in academic journals. They called their idea the Reproducibility Project. And in the end of that, 2015, we published The Findings, which was a 270 co-author paper of 100 replications of findings from three different journals in psychology. We got a little less than half of the findings successfully replicated. You did not mishear Brian Nosek. That's what he said. Little less than half of the findings successfully replicated. So he's been running large-scale replications ever since, and not just in psychology.

A year and a half ago, we published the results of the reproducibility project in cancer biology, doing the same kind of process, and found very similar results. Less than half of the findings in preclinical cancer research replicated successfully when we tried to do so. It's important to point out that what Nosek says here may not be as bad as it sounds. Here's how he thinks about that high number of studies that don't replicate. That doesn't mean that the original finding

is necessarily wrong. We could have screwed something up in the replication. Successfully replicating doesn't mean the interpretation is right. It could be that both the findings have a confound in them, but we just are able to repeat the confound. There might be other explanations too, legitimate explanations.

So we shouldn't equate a failure to replicate with fraud or even misconduct. Still, NOSIC's numbers do suggest that a lot of research happening today may not be producing science of lasting value. And we're talking here about the research being done at the most elite universities and academic journals. If you are a fan of science, any kind of science, this should concern you. Fraud is the ultimate corrosive element of the system of science, because as much as transparency provides some replacement for trust, You can't be transparent about everything. So the ideal model in scholarship is that you can see how it is they generated their evidence, how they interpreted their evidence,

what the evidence actually is, and then independent people can interrogate that. And so to the extent that fraud intrudes and actually the evidence isn't there, it isn't actual evidence, then the whole edifice of that scholarly debate and tangling with ideas falls apart because you're actually tangling with ideas that aren't based on anything. How familiar are you with the Joachim Bolt situation? I'm not recalling that name, but I may know the case if you describe it. Yeah, this was the German anesthesiologist who had almost 200 papers retracted. Oh, yeah. I gather there were people actually dying as a result of this faulty research.

So I am curious to ask you, in which academic fields or disciplines do you think fraud or sloppiness is most prominent? We can't say with any confidence where it's most prominent. We can only say that the incentives for doing it are everywhere. And some of them gain more attention because, for example, Francesca's findings are interesting.

They're interesting to everyone. So, of course, they're going to get some attention to that. Whereas the anesthesiologist's findings are not interesting. They put people to sleep. Until they kill you. Well findings are not interesting. They put people to sleep. Until they kill you. Well, yeah, I guess they put people to sleep and then they kill you.

But it does seem like your field of training, psychology, and especially social psychology, is a hotbed of, I wouldn't say fraud, but certainly controversy and overturned findings over the years. Yeah, I would say that there is a lot of attention to social psychology for two reasons. One is that it has public engagement interest value. That's a way of saying that people care about your findings. People care about, at least in the sense of, oh, that's interesting to learn, right? But the other reason is that social psychology has bothered to look.

And I think social psychology became a hotbed for this because the actual challenges in the social system of science that need to be addressed are social psychological problems. What do you mean by that? I mean like the reward system, how it is that people might rationalize and use motivated reasoning to come to findings that are less credible. A lot of these problems are ones that social psychologists spend every day thinking about. I asked Nosek why he thinks that presumably honest people might, over time, come to behave dishonestly. The case that people make to say that this is a bigger problem now is that the competition for attention, jobs, advancement is very high, perhaps greater than it's ever been.

Do you think that's been driven by the shrinking of tenure positions at universities? Yeah. So there are many more people and many fewer positions, which is an obvious challenge for a competitive marketplace. And there are now pathways for public attention that have even bigger impact. Academics, by and large, didn't think about ways to get rich. They looked for ways to have time to think about the problems that they want to think about. But now they have pathways to get rich. Those pathways have benefited many people, myself included, even though I am not an academic. Thanks to my partnership with

Steve Levitt, who's an economist at the University of Chicago, I have had more opportunities than I ever would have thought possible, including this show. I have wondered why there seems to be so much less shady research in economics than in psychology and other fields, when you talk to economists, they'll give you several reasons. Economic research is very mathy, and it comes with a lot of what they call robustness checks. There is also a tradition of, let's say, aggressive debate within academic economics. Long before you publish a paper, you typically present it to your peers and elders at seminars, and they are only too happy to point out any possible flaw and call you an idiot if you disagree.

I'm not saying this is the best way to conduct business, but it certainly makes shaky data more costly. Also, economists tend to work with big data sets, much bigger than in the rest of the social sciences, and it's often publicly available data. So no cheating opportunity there. This doesn't mean there is no controversy within economic research and no overturned results. There are a lot.

And if you hang around with economists, as I tend to do, you will hear whispers about a few researchers who are suspected of fudging their data. But it does seem that most researchers in econ are mostly honest. And for all the honest researchers out there in whatever field, there's one more twist. When you are playing a game by the rules, but you see that the people winning the game are cheating, you feel like a sucker and no one enjoys feeling like a sucker. But it's bigger than that. If the cheaters are winning, that means non-cheaters get smaller rewards and it means all their hard work may also be viewed with suspicion. So what can be done about that problem? This might require something more invasive than a reproducibility study. This might require interrogating the data or research methodology on suspicious research papers and making public accusations of fraud.

And this brings us to the whistleblowers we heard about earlier. They are co-directors of the Credibility Lab at the University of Pennsylvania, and collectively they run a blog called Data Colada. My name is Leif Nelson, and I'm a professor of business administration at University of California, Berkeley. Uri Simonson, I'm a professor of behavioral science at the Asada Business School in Barcelona. Joe Simmons, I'm a faculty member at the Wharton School at the University of Pennsylvania. Nelson, Simonson, and Simmons are, let's call it, mid-career academic researchers. They've been at it for a while, and they hold high status positions within the ecosystem they are concerned about. They've all published widely in top journals of psychology and decision science.

So they're coming at this not only from the inside, but with inside knowledge of how academic research works and doesn't work. So they focus on examining the methodology used in this kind of research. What motivated our whole journey into methodology was that we would go to conferences or read papers and not believe them. And we would find that whenever a finding didn't align with our intuition, we would trust our intuition over the finding. And that was, it sort of defeated the whole purpose. Like if you're only believing things you already believe, then why bother? This guy, Daryl Ben, published a nine study paper with eight studies worth of statistically significant evidence that people have ESP. And most people were like, what is going on?

Like this, this cannot be a true finding. So the idea was like, how do we show people that you can really very easily produce evidence of anything? So we thought, let's start with something that's obviously false. We said, okay, something that's quite hard to do is to make people younger. We've been trying forever. We never succeeded. So let's show that we can do that in a silly way. So we decided to show that we can make people younger by listening to a song by the Beatles. The song was When I'm 64, correct? That's right. And so the idea is, if we can make anything significant, one way to prove it is to say, I'm going to show you with statistical significant evidence that people got younger after they listened to When I'm 64.

Will you still need me? Will you still feed me when I'm 64? They ran real lab experiments with real research subjects who had real birthdates and played them real songs, When I'm 64 and two others. A control song was called, I believe, Kalumba by Mr. Scruff. And then we had another song that was meant to go in the other direction, and it didn't work, so we just didn't report it, which was Hot Potato. They essentially manipulated and cherry-picked their data to produce the absurd finding they wanted, that listening to When I'm 64 does lower your age. By a full year and a half, it turns out. They published their article in Psychological Science,

one of the top journals in the field. Their piece was called False Positive Psychology. Undisclosed flexibility in data collection and analysis allows presenting anything as significant. They wrote, these studies were conducted with real participants, employed legitimate statistical analyses, and are reported truthfully. Nevertheless, they seem to support hypotheses that are unlikely or necessarily false.

So we were surprised when the paper got accepted. And then the immediate aftermath was shocking. Like there were so many people for whom this resonated. And then there were lots of people who were really not very happy. Like, why are you giving the field a bad name? But Simmons, Nelson, and Simonson felt their field already had a bad name. Yeah, we thought it was very bad. We started our blog in late 2013.

We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part, but would definitely not be sending signals of self-seriousness. So we tried out a few things and somewhere in there, Data Collado was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name, he thought it rhymed, which still tickles me and Joe, because for him, it's Dada Kalada. The Data Kalada team, or Dada Kalada, or maybe Data Kaleda, they weren't going to look just for cases of outright fraud. They were concerned, as Brian Nosek had been, that the pressure to

publish interesting results might produce unreliable findings even if the researcher had mostly followed the rules. Consider, for instance, a practice they came to call p-hacking, with the p standing for probability. This is Leif Nelson. The classic forms of what we'd characterized as p-hacking, they're not quite errors. They're decisions that are accidentally self-serving. It's if you measure multiple things but only report the one you like the most. Or you run a study where there's three treatments, condition A, condition B, and condition C. But in the end, you drop Condition B, and you don't even talk about it. You just compare A to C.

That, remember, is one of the things the Data Colada guys did in their pranky paper about when I'm 64. They just left out hot potato. And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers. And you say, we should eliminate those outliers, or we should Windsorize the outliers, which is basically truncating them down to a lower high number. Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just, if you have a consideration of reporting

one variable or the other, and one variable makes your hypothesis look good, and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self-serving or honestly, because you'd say like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good. And that one looks good. It's probably the better measure. And here's Simonson. There's a few approaches. Some of them that we've done, like just do statistics and said, this is statistically impossible. The other is you see associations in the data or lack associations of the data that they're not mathematical properties,

but just anybody looking at data who's familiar with that would realize this is not right. Imagine that you have data on weight and height and you correlate it and you find zero correlation. That cannot be right. People who are bigger are heavier. And so if you found zero correlation or a negative correlation,

you think maybe these are not real weight measures. Another one is you see rounding or precision that is suspicious. You see rounded values where there shouldn't be any rounding or absence of rounding where there should be. So for example, in one case that we worked on, there was data supposedly when people were asked, how much will you pay for this t-shirt? And the very curious thing is there was no rounding. People were equally likely to say $7, $8, or $10. But if you've ever collected data like that, you know that people round. People say 10 or 20. They don't say 17.

There is another category that you might call convenient errors. Here is Nelson again. These will be things that can be as simple as a typo, where someone's writing up their report and the means are actually 5.1 and 5.12, but instead someone writes it down as 51.2. And you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally a typo might end up in print. And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data. Do you think that the first set of conditions that you described

are quite likely to lead to fraud? I mean, how slippery is the slope? Is the kind of person who's willing to do those things willing to go ahead and commit fraud, especially if they've gotten away with it for a while? Well, Stephen, you were asking a question that is pretty heavy and one that I'm not particularly well-equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect 10 measures and I report one. That's slippery slope. But making up data feels qualitatively different. And I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop some measures or drop a condition or you remove the outliers. And then also you take participant 35 and you change their answer from a 7 to a 9. You're like, whoa, that last one doesn't sound the same. But maybe there's some psychology for whoa, that last one doesn't sound the same.

But maybe there's some psychology for that, that it feels like it's an extension. So these three self-appointed data detectives looked into the work done by their peers and they publicly posted their findings. The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual. I want to go read that paper. And so I read the paper and then

looked at that data set. And that one, it had collected data on a nine point interval scale. So people can answer one, two, three, up through nine. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy. Once you open the data set, you can then close it and say it's broken. The paper Nelson was describing had been published by four Taiwanese researchers in 2012 in the journal Judgment and Decision Making. After the data collada investigation, the paper was retracted. Although as far as we can tell, the researchers weren't sanctioned or punished. So how much fraud is there? I asked Yuri Simonson. I would estimate the share of fraud in the order of, say, 5% of articles.

What's the difference between, let's say, high-profile academic journals versus mid-tier or lower? Is fraud more likely to be prominent in the higher or the lower? I don't read really low-tier ones, so I don't know. Sometimes I will, but if I come across fraud there, I will ignore it because the cost is so high for seeing a case of fraud that it's just not worth it. If it's a paper that has seven citations after three years and it's published in a journal that nobody knows, I just let it be. And I'm sure other people do that too. Simonson, Nelson, and Simmons kept on with their investigative work as a sideline to

their regular research and teaching responsibilities. Within psychology and behavioral science circles, Data Colada became well-known and a bit feared, but they didn't have much reach beyond those circles. That changed a couple years ago. They published a post called Evidence of Fraud in an Influential Field Experiment About Dishonesty. The fraud they claimed to identify was in a paper published years earlier in a top journal known as PNAS, or the Proceedings of the National Academy of Sciences. The paper was called, Signing at the Beginning Makes Ethics Salient and Decreases Dishonest Self-Reports in Comparison to Signing at the End. Let's put that in English.

The paper claimed that if you are asked to sign a form at the top of the form before you fill in the information, you are more likely to be truthful than if you sign at the bottom. You can see how that might work, right? There are four things you might want to know about this paper. One, the central finding had proved to be extraordinarily popular. A lot of firms and institutions started putting the signature line at the top of tax statements, insurance forms, things like that. Number two, the article had been edited for PNAS by Danny Kahneman, perhaps the best known living psychologist and one of the most highly regarded.

Number three, two of the five co-authors on the paper were among the best known people in this field, Dan Ariely and Francesca Gino. And four, there was already evidence that something was up with the original paper because its authors had published a second paper saying that their original findings didn't replicate. The failure to replicate, as we heard earlier, does not necessarily mean fraud. But now, the Data Colada investigators claimed to have proof that, yes, there was fraud in the original paper. That original paper was written by Dan Ariely, Francesca Gino, along with Nina Mazar, Lisa Hsu, and this man. I'm Max Bazerman, and I'm a professor at the Harvard Business School. After the break, Max Bazerman takes us through the collaboration that triggered a crisis.

Also, Freakonomics Radio had a great 2023. You are one of more than 2 million people who listen every month. Let's make it 3 million for 2024, okay? If you like the work we do, please tell people about it. That is the best way to support the podcasts you love. Thanks in advance. We will be right back. Max Bazerman, a professor of business administration at the Harvard Business School, is considered an elder statesman in the field of behavioral science.

For decades, he has been publishing well-regarded research papers and books, and he's known as a wise and caring mentor to younger scholars. That last bit, it would seem, is the best explanation for how Bazerman wound up being a co-author on what would turn out to be a very, very, very problematic research paper. This is the signing at the top paper we heard about before the break. The paper published in PNAS in 2012, which claimed that you are more likely to be honest if you sign a form at the top before you fill in the information than if you sign at the end. This paper actually started out as two separate research projects. Here's Bazerman. So 2011, Lisa Xu and Francesca Gino and I had a working paper that got rejected from a couple of journals that basically claimed to show if you sign a document before you fill it out,

you're more likely to tell the truth. Our studies were done in the laboratory. Lisa Hsu was a doctoral student at the time. Bazerman was her advisor, one of the chairs of her dissertation committee. As for Francesca Gino? Francesca started visiting the Harvard Business School as a doctoral student from Italy. By 2004, she was attending my doctoral seminar, and we started to interact pretty regularly. And eventually, I was on her dissertation committee and played a pretty active role in advising her. Bazerman liked Gino a great deal. In fact, nearly everyone liked Gino and admired her intellect and work

ethic. We once even got to the point of our two families making an offer to a developer on a project to have houses connected to each other. Their signing at the top paper was based on data from two lab experiments from the University of North Carolina at Chapel Hill, where Gino taught before coming to Harvard. In those experiments, research subjects tried to solve a number of puzzles with a financial reward for every right answer. And afterward, they filled out a form to tell the experimenter how much money they had earned. Some of the research subjects were asked to sign an honesty pledge at the top of the form, others at the bottom. In the experiment, the ones who signed at the top were more truthful about their earnings.

At least, that's what these data said. They were brought to our threesome from Francesca, and the exact role of Francesca and the lab manager remains somewhat unclear, but it's certainly safe to say that Lisa Xu and I had little to do with the collection of the data. So this goes back to a time when Francesca was developing an excellent career, but she was not the superstar that she came to be. And I was a senior person and probably doing the least amount of work and working more after the document came to fruition. Do you think that most people, let's say the median American who maybe holds a decent opinion of university life and academic research, which maybe that's not the median person,

maybe the median person. Maybe the median person doesn't hold such a decent opinion. But for someone who might read an article that's based on an academic study and say, oh, that's interesting. I'm going to file that away as a useful, probably true piece of information. How surprised do you think that person would be to learn that a senior colleague like you, who's co-author on a lot of papers with junior colleagues, that you personally don't interact at all with the original data? How surprising do you think most people would find that? So I wouldn't say don't interact with all.

I certainly would read the results section pretty carefully, but I would read it with the intent of seeing if there was any error along the way. But how can you tell if there's error if you're not in the, you know, it's like this whole issue reminds me a little bit of being, let's say I'm a chef in a restaurant. And I'm given the ingredients to cook, but I'm not allowed to examine them. So I don't know if they're rancid or fresh or even fake. So I like that example. So instead of being a chef of a restaurant, let's imagine that you're the owner of 12 different restaurants and you have a head chef in each. And that head chef is going to be what I think of as the most senior of my more junior colleagues on the project. And over time, I've come to trust that they're going

to be doing a really good job of overseeing the ingredients that go into the research process. And by doing so, there's other things that I can do. I can work on, you know, making sure that we have the funds available. I could, you know, work on whatever particular problems come up administratively. I could work with more young scholars because my time is more available. So there's lots of good by this efficiency of trusting the assistant professor on the project or the head chef at a particular restaurant so that I'm not examining the specific ways in which the sausage is made. Now, remember, this original paper was rejected by multiple journals.

Bazerman says they got feedback suggesting their argument about signing at the top would be more believable if, in addition to the lab results that Francesca Gino provided, they had some results from the real world as well. That's what researchers would call a field experiment versus a lab experiment. As luck would have it, another researcher, a friend of Gino's, no less, apparently had some good field results. We collectively heard that Dan Ariely was presenting a very similar result based on a field experiment having to do with an insurance company.

And Francesca reached out to Dan, and we basically combined efforts to pull the three studies into one paper. Ariely's data included the number of miles the customers of this insurance company reported having driven in a year. If you think about how insurance works, a customer might have an incentive to under-report their mileage in the hopes of lowering their insurance bill. Ariely's data showed that customers who were asked to sign this mileage statement at the top reported having driven more miles than customers who were asked to sign at the bottom. Again, suggesting that signing at the top makes people more honest. Max Bazerman now received the first draft of a new paper that combined the Ariely and Geno studies. I'm reading the insurance field experiment for the first time at this point.

And as I'm reading it, I have some questions. I wasn't remotely thinking about fraud as an issue. I just thought there was something wrong. And what I saw as wrong was that we were reporting that the average driver in the database had driven between 24 and 27,000 miles per year. And I just looked at that and I said, that seems off. It seemed off to Bazerman because the average American only drives around 13,000 miles a year. So I asked some questions about it. And Ariely, who was the point person for that data, the origins of which are not completely clear, sent back a very quick email saying the

mileage is correct. And I continued to say, well, we need to clarify what's going on here. It seems off that people have driven so many miles, particularly when you're talking about tens of thousands of drivers. And eventually, Ariely comes back with, the drivers are senior citizens in Florida. Sounds like they should drive even less than 24,000 miles then. Exactly my thought. So my questions continue, and I don't get very good answers. And literally, this goes on for months, and I'm seriously considering taking my name off the paper. At the time, Lisa Hsu is a doctoral student on the job market, and she's presenting this work. And how concerned were you about damaging her prospects?

I was very concerned that if I dropped off the paper, that there's something suspicious about Lisa's presentation. So I keep on asking questions, but I don't withdraw. And by early 2012, I'm attending a conference, and I arrive at the conference, and in the big hallway, I run into Lisa, who's my advisee, my friend, my co-author, somebody who I like a lot. And she's with Nina Mazar, who I had never met before. So Lisa introduces us, and I believe I was expressing my unhappiness with the lack of clarity on this mileage issue. Nina Mazar is Ariely's collaborator on the insurance study, correct? So that's a little bit unclear. Okay. So when we asked Ariely to join forces, he said, fine, but Nina would be part of the project as well.

So I always thought she was part of the insurance study. Later on in life, Nina claimed she had no more connection to the insurance study than I did, that she first connected to it when this five-author paper comes together. But in any case, at this conference, she assuaged you to some degree. Yeah, exactly. So she basically pleasantly and openly pulled up the database on her computer. And I said, so what's going on? And she said, I think what's going on is that we don't know that the period between time one and time two for assessing the number of miles driven was one year. We know when time two was collected,

but it may have been more than one year for when time one was collected. And in my mind, what becomes clear is that that makes our study noisier. But as long as a real experiment was run, this is actually pretty good news. All we need to do is correct the presentation in the paper, which we did. The paper's submitted, it's published. I develop a belief that this effect is true. And people love this result. And from a theoretical standpoint, it's a shockingly simple idea. From a practical standpoint, it's just perfect. It's so simple that organizations can easily implement it. And who did implement it?

A lot of people implemented it. You know, I think Lemonade Insurance, under Ariely's advice, implemented it. Lemonade Insurance, by the way, didn't just implement Ariely's advice. They hired him as their chief behavioral officer. And many government agencies implemented it, including the U.S. government. Indeed, the first sentence of the paper that Bazerman and the others published in 2012 reads like this. The annual tax gap between actual and claimed taxes due in the United States amounts to roughly $345 billion.

Now, imagine working for the IRS or any tax agency anywhere in the world and learning that these brilliant academic researchers from Harvard and Duke had found that if you simply have people sign their tax forms at the top rather than the bottom, that millions, maybe billions of extra dollars will suddenly flow your way. And by 2016, I get an email that's actually critical to the whole evolution of what happens later. The email is from Stuart Bazerman, only he spelled his name with an S instead of a Z. And he basically is a pretty low-key guy who is working on an insurance startup to do insurance online. And his wife, Sue, encouraged him to email me because he was working on the problem of how do we get people to tell the truth online? And he had read the 2012 signing first paper, and he said, this looks like I might even be related to a guy who knows how to get people to tell the truth.

So Stewie emails me, and we develop a very good relationship. Turns out he's a fifth cousin, according to 23andMe. And I also develop a consulting relationship with Slice Insurance, the company that he was developing. Now, sorry to ask the rude question, but does that seem like a conflict of interest at all in retrospect? To work for my cousin. Well, to go be a consultant for an insurance company based on the findings of a paper, which turns out to have been fraudulent, that you were a co-author on. I didn't know it was fraudulent in 2016. I even believed it by now.

Yeah. Are you still a consultant for Slice? No. Did you leave because the finding was fraudulent? No. No, I have a terrific relationship with Slice. Okay, so going back to 2016, Max Bazerman wanted to help his newfound cousin learn whether signing at the top would be as effective in an online setting, the way it seemed to work with paper documents.

So Bazerman and some junior colleagues set out to test that question. And how was Bazerman feeling at the time about the original sign-at-the-top finding? We know it works. We know the effects are big. We know the world is intrigued by it. Seems perfect. Were you worried about a placebo effect at all, which is to say, if enough people have heard through media descriptions of this signing- first phenomenon, that if they encounter a forum where they're asked to sign first, they will now know that they are under some kind of scrutiny, perhaps, and therefore they're more likely to be honest because they know about it?

That's a terrific mythological critique. So what you just said logically makes sense. I think as of the time we were doing these online studies, and there are many of them, I don't think that there was widespread public awareness of the signing of first effect. You know, Ariely and Gino and I talked to lots of executive classes, but I wouldn't say it was a well-known social phenomena. But I could be wrong. So your methodological critique could be viable. But anyhow, signing first with or without a placebo doesn't work online.

We got nothing. How surprised were you? Very. And I kind of said, well, let's take a look at what we did. Let's see sort of how we might have messed up the design. Let's try again. So we make some changes. We do it a second time. We make some changes. We do it a third time. Still, no effect, no effect, no effect. And recall, the 2012 paper not only has effects,

it has effects across three different studies that are all statistically significant, and the effects are large. So the project clearly transforms somewhere between replication failure three to five. It's transforming from how do we get people to tell the truth online to a massive replication failure of a pretty visible academic effect. And so after we fail six times, we decide, well, let's go back and do a large-scale replication of one of the original lab studies. Bazerman and the collaborators he had brought on to do the online replication work now convened with the authors of the original paper, including Dan Ariely and Francesca Gino, and they set out to replicate one of that paper's

lab studies, but using more than 10 times as many research subjects as in the original. One feature of many academic studies, especially in a field like psychology, is that they often use a small set of research subjects to run this kind of experiment, often just students from their own university. A small research pool is cheaper and faster. And if the goal is to produce a lot of publishable research, fast is good. But small sample sets are more likely to return a skewed result. So now, with a bigger sample and much more scrutiny, they get no effect. Signing at the top doesn't seem to do much of anything. So certainly I felt a moral obligation to correct the record. And not just moral obligation, but as you've told us, there are institutions,

government institutions and firms that are using this research. Did you feel, I mean, I don't want to put a word in your mouth, but was it a sense of guilt or panic or fear, anything like that? Certainly, I would feel some sense of, maybe guilt is the right word, that my name's on a paper that people are using when I no longer think that they should be using it. But, you know, I didn't think that I was doing anything wrong. And quite honestly, I'm still not thinking fraud at this point. I just don't know what's going on. I'm thinking of cleaning up the record. In 2020, Bazerman, along with all the original authors and his two

more recent collaborators, published a follow-up paper in PNAS, the same journal where the original piece was published in 2012. This paper was called Signing at the Beginning vs. at the End Does Not Decrease Dishonesty. Now, from the outside, you may think this is not a very brave position to take. You are simply putting out a new paper saying that the paper you put out eight years ago, the one that got so much attention and advanced so many careers, that that one didn't actually work. On the other hand, you could say,

this is how science is supposed to work. You have a hypothesis, you run experiments to test your hypothesis, you gather and analyze the data, and you present your findings. If new information comes along that overturns your finding, well, that's what needs to happen to correct the scientific record. But it's worth noting what the original authors didn't do. They didn't retract the original paper, nor did the journal retract it, at least not at this point. So from the outside, this looked like a story of science that was perhaps conducted

sloppily, but not a story about fraud. And at least some of the original authors hadn't given up on the original finding. Ariely and Nina Mazar both continued to argue, well, it works some of the time, it doesn't work other times, and we need to do more studies to find out when it works and when it doesn't. And my attitude, so I won't speak for other co-authors, was to basically say we have plenty of evidence to conclude that we should tell the world that we have no faith that this effect works. So we don't retract, and life moves forward. And in June of 2021, I believe, I get an email from one of the members of the Data Colada team saying, Max, can the three of us meet with you on Zoom

to talk about something important? After the break, what it's like to be on the other end of a Data Colada Zoom call, and how much does academic fraud contribute to the public's ever-sinking opinion of universities in general? I'm Stephen Dubner. This is Freakonomics Radio.

We'll be right back. Okay, so Max Bazerman, a well-regarded senior researcher and professor at the Harvard Business School, gets an email asking him to meet with the Data Collada team, and they say it's important. So there's a Zoom, and the first part of the Zoom is the Data Collada team showing me the evidence for fraud in the insurance paper. And it's kind of overwhelming. These guys are careful and they're thorough. And like, they convinced me that there was fraud in this study. This insurance study was one of three studies in the sign at the top paper that Bazerman had co-authored some years back. The Data Collada researchers had scrutinized the data that Dan

Ariely had used and found several things suspicious. The most obvious one was a data chart called a histogram showing the number of miles driven each year by the people in his study. For data like this, a histogram will typically look like a bell curve with a lot of people clustered around the center and then some outliers sloping off toward the upper end and lower end. But this histogram showed a nearly uniform distribution of drivers from zero miles to 50,000 miles. This is not what real data look like, the Data Colada team wrote on their blog post,

and we can't think of a plausible, benign explanation for it. And what's Max Bazerman thinking now? I'm just kind of overwhelmed with the fact that I'm the author of a fraudulent paper. And later, there was new information from the insurance company that had given Dan Ariely the data. They told the Planet Money podcast that the data Ariely published was significantly different from what they had given him. And in their original data, there was no difference between those who signed the forms at the

top and those who signed at the end. Although Ariely declined to do an interview for this episode, he did send a written statement. As someone who spent many years studying dishonesty, he wrote, I appreciate the irony of being accused of dishonesty. There's no question that the data underlying the 2012 study I co-authored with four other researchers about dishonesty was, well, dishonest. I have looked diligently to figure out what went wrong, but given that this took place more than 15 years ago, I can't tell for sure what happened. He added, all five co-authors of the study in question participated in review sessions with people from the insurance company and asked

questions about the data. Ultimately, we all felt satisfied with the answers we got and collectively decided to move forward with the paper. Max Bazerman, when we shared with him Ariely's statement, replied, I am extremely confident that I never participated in any such review sessions and confirmed this with conversations with Lisa Hsu and a thorough search of my email records. The final part of Ariely's statement reads, The circumstances that led to the data being falsified are being investigated by Duke University. I am confident that the investigation will find no evidence to suggest I was responsible for any data manipulation. I am sure that this matter will be behind me very soon, and I will be resuming my research

at Duke at full speed. That's the end of Ariely's statement. As for the PNAS Journal, they did finally retract the original paper. Meanwhile, on that Zoom call, the Data Colada team had some more news for Max Bazerman. So after they presented the insurance evidence to me, they said, and now for worse news. That's when they introduced the allegation of data fabrication in one of the lab studies. One of the lab studies, meaning one of the separate studies in the same paper, whose data had come from Francesca Gino. Data Collada said they had found serious problems with her data too. And to make matters worse, there was evidence of data fabrication in three other projects that Gino was a co-author of. And when you say they said, now for the worst news, I'm assuming, but tell me if I'm wrong, they're saying that because they know that you have had a long and close relationship with Francesca Gino? Yeah, and I was clearly more closely connected to those lab studies.

And you're a co-author on more than one of the papers. I was only a co-author of one of the four papers that they were showing me, but I, by then, had been a co-author of eight different empirical papers that have Francesca Gino as a co-author. So what happened next on the Zoom call? So they provide the evidence. And so I'm now aware that there's what Data Collada later called cluster fake, that there's at least two frauds in the same paper, or that that was likely to be the case.

And then Data Collada basically said, so Max, you're the Harvard professor. We think that Harvard should have access to this information. Are you the person who will take it to them? And I said, no, thank you. Because why? Because I certainly thought that Harvard should be aware of what I was looking at, but I didn't want to play a central role in making that happen.

Data Collada had begun investigating Francesca Gino after they received a tip from a graduate student named Zoe Ziani and another anonymous researcher. In addition to the sign at the top paper, the Data Collada team wrote, quote, we believe that many more Gino-authored papers contain fake data, perhaps dozens. Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud. That, again, is Leif Nelson, one of the three Data Collada researchers, along with Yuri Simonson and Joe Simmons. Bridging the gap between those two positions is this other entity, Harvard University.

And they have said, although we, Joe and Yuri and I, have not had access to any of their internal investigation documents, we only know what they've said outwardly, which is that they've put her on administrative leave and they've recommended the retraction of those four papers, or the retraction of three plus an amendment to a previously retracted paper. And how confident are the Data Collada researchers in the accuracy of their analysis? Here is Simonson. So I would say I have, on the falsity of the findings, I don't have reasonable doubt. Not long after Harvard Business School placed Francesca Gino on leave, she filed a lawsuit. So we were sued together with Harvard for $25 million. We were sued for defamation. Again, Francesca Gino declined our request for an interview. On a website called Francesca v. Harvard, she wrote, I absolutely did not commit academic fraud. Harvard has ruined my career wrongfully. The only way to right this wrong is for me to sue Harvard.

As for Data Collada, Gina wrote, the decision to sue Data Collada was more difficult. I have long admired Data Collada's work. I have particularly respected its commitment to sharing any negative findings with the author before going public with the charges. Yet in my case, Data Collada changed its procedure. Francesca Gino also claimed in her lawsuit that Harvard Business School discriminated against her on the basis of sex. And later, her allies wondered why Gino's punishment had been so

prompt and severe when, for instance, Harvard president Claudine Gay hadn't been disciplined immediately for outright plagiarism in her research. But then, as you likely know, Gay was recently forced to resign from the presidency. Yet more academic fraud in the headlines. I asked Leif Nelson how it felt when he heard that Francesca Gino was suing him and the other members of Data Collada. Certainly scary. Scary because it's so unfamiliar. I found out from talking, basically I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now given the lawsuit, would you like to add a new comment? And I was basically like, what lawsuit are you talking about? And so it's like this devastating thing to be like,

oh my God, it's like the whole house is collapsing and no one told me. Look, I definitely had moments after this where I was scared for myself and my family. And that is Joe Simmons. Like just the amount of money involved. I didn't quite appreciate that at the very first moments. I mean, this is not the way to adjudicate these things. There's a million chances to show we're wrong. A million. Like earlier. A lot.

There's a lot of chances. And this is what you're going to do. You're going to sue three individuals for $25 million. That seems not great. The Data Colada team found out it's expensive to defend yourself in a lawsuit like this. Some colleagues set up a GoFundMe campaign. Here is Simonson. Within 24 hours, they had $200,000.

We found a First Amendment expert lawyer who's representing us. We've learned a lot of the boring stuff that happens with lawyers that you don't get from TV shows, like the timelines and the language and how long they're judged. Like, things take forever. I mean, it makes academia seem expedient in comparison. It's not just nice to have money, but it's also nice to know that thousands of people are willing to, at least somewhat publicly support what you're doing.

So that was a big boost. My very first thoughts were like, oh my God, how's anyone going to be able to do this again? If you can get sued for conducting these kinds of investigations, and we are business school professors at very good institutions that have a lot of resources.

We're definitely in a position to bear the brunt of this more than, say, the average person in the field. And so just the chilling effect on scientific inquiry and criticism, it's like, is everything we've been working for for 10 years, is that now gone? Our field doesn't have a culture of open criticism. It's not considered okay.

And that is Samin Vazir. If I name a specific theory or finding, that's considered a personal attack on the people associated with that theory or finding, even if I don't talk about the people behind it. And that's considered not okay. Vazir is a psychology professor at the University of Melbourne. She's also the new editor-in-chief of the top journal Psychological Science. And she has already been a central figure in the push to reform behavioral science. It was Vazir who set up the GoFundMe campaign for the Data Colada team after Francesca Gino sued them.

I think one thing we can say for sure is that humans are quite good at self-deception. There's a lot of reasons why researchers want to believe that they've found the answers to these problems, right? One is a pro-social reason. They want to help solve these problems. They want to help people. Another one is more self-interested, that they want a seat at the policy table. They want attention for themselves. They want to promote their theory and their brand. And it's also kind of a survival thing to stay in academia, to be able to continue to do the research you need to have successes. And those successes often mean selling your work and sometimes overselling your work. We want to be taken seriously as scientists and be scientific. And that means being calibrated and careful and not

exaggerating. But at the same time, the people who do exaggerate are probably going to get more of those successes that get them attention, get them a seat at the table, get them the next grant, the next job, and so on. When you describe the incentives like that, it sounds to me as if those incentives conspire against the scientific method. Do they not? Yeah. So if you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat. I think that is absolutely the against the pure pursuit of knowledge even within academia. I just think if people like you, fellow scientists, can't successfully evaluate these claims, what's the public supposed to do? What are we supposed to think when we hear a claim that is often amplified through media? Doesn't it just make everyone skeptical

or even cynical about everything? I think that maybe that's appropriate for fields like psychology, where we're tackling really messy, complicated things that are multiply determined, that have many causes. It's hard to measure even one of the causes. And so I think if you hear a claim that sounds too good to be true or sounds too simplistic, I think it's appropriate to use common sense to be skeptical of it. Generally, I think we're taught that science can overturn common sense and you shouldn't just not believe science just because your common sense goes against it. But I think that should be different for different sciences. And psychology is just a lot harder to do and it's a relatively young science. We haven't

perfected even the measurement of many of these concepts, much less all the other steps to studying them. Do you think of your field as being in crisis? I do. What kind of crisis? I mean, we hear the phrase replication crisis, which is a specific response to a fear that research is garbage, but I assume it's broader than that. Yeah. I mean, I think to our credit, it's broader than that. I think that our field is really in a period of intense self-examination. It's going to tell the world, I think, how committed we are to

scientific values, how we deal with this crisis. There's so many branches to this crisis. And I think it's just a crisis of, I don't know, integrity or credibility or whatever the most fundamental word you could use to describe what it means to be scientific. There was a lot of debate in the early days of the crisis of whether we should air our dirty laundry in public. The people who were arguing against it, I lost a lot of respect for them. But the other side won. We did air it in public. And I think that we deserve some credit for that.

I don't think we should rest on our laurels there. Identifying the problem is not the same thing as changing our practices. We should do some soul-searching about how we got there and how we can prevent that from happening again. Max Bazerman has got a head start on the soul-searching. You know, I've been connected to so many research projects where, as the most senior member of the team, I never looked at the database. So, you know, does that make me guilty of something? I think yes. I think that it makes me complicit that I didn't do more verification that I thoroughly trusted. And I think that I should be somewhat accountable for the fact that I didn't

do a better job of verifying, not just in the signing first paper, but in my research more broadly. And when, you know, in the signing first paper, I was bothered by some aspects of the data, and I asked a lot of questions back in 2011, but I eventually got an answer that I wanted to be true, and I accepted it. And I never looked at that database at that time either. Do I wish I did? Absolutely. I'm not sure that this story would be unfolding today had I looked at that data in 2011. There's so many positive influences that social science can have, from getting us to eat healthier foods, to exercising more, to saving for our retirement. And social scientists have been remarkably good at helping figure out how do we get people

moving in the right direction. And the vast majority of that work is honest, credible research that we should pay attention to. And if we now end up having sort of the big story being fraud and social science, then all the credible stuff is going to have less value and less impact than it should. Coming up next time on the show, we are going to keep this conversation going, but from some different angles,

including the money. You may have heard of diploma mills or puppy mills, but how about research paper mills? It could be anywhere from hundreds of dollars to even thousands of dollars per paper. And they're publishing tens of thousands and sometimes even more papers per year. So you can start to do that mass.

We will start it, maybe even finish it. That's next time on the show. Until then, take care of yourself. And if you can, someone else too. By the way, while we were finishing up this episode, someone very close to me turned 64. And in response to the questions posed by Paul McCartney in that song, the answer is yes. Freakonomics Radio is produced by Stitcher and Renbud Radio. Our entire archive, going back to 2010,

can be found on any podcast app or at Freakonomics.com, where we also publish transcripts and show notes. This episode was produced by Alina Kullman. Our staff also includes Eleanor Osborne, Elsa Hernandez,

Gabriel Roth, Greg Rippin, Jasmine Klinger, Jeremy Johnston, Julie Kanfer, Lyric Bowditch, Morgan Levy, Neil Carruth,

Rebecca Lee Douglas, Ryan Kelly, Sarah Lilly, and Zach Lipinski. Our theme song is Mr. Fortune by The Hitchhikers. So if I were to rephrase your answer, you basically said podcasts are better than academic journals, correct? As a former podcast host, I were to rephrase your answer, you basically said podcasts are better than academic journals, correct? As a former podcast host, I have to agree. The Freakonomics Radio Network, the hidden side of everything. Stitcher.