Plain English with Derek Thompson - Megapod: Why Is There So Much BS in Psychology?

Episode Date: November 27, 2024

In the last decade, several major findings in social psychology have turned out to be hogwash—or, worse, even fraud. This has become widely known as psychology's "replication crisis." Perhaps you have heard of power poses—based on a study finding that subjects reported stronger “feelings of power” after they posed, say, with their hands on their hips for several minutes. But that finding did not replicate. Or perhaps you have heard of ego depletion—the more famous assertion that, when people make a bunch of decisions, it exhausts their ability to make future decisions. Again: did not replicate. “There’s a thought that’s haunted me for years,” social psychologist Adam Mastroianni has written. “We’re doing all this research in psychology, but are we learning anything? We run these studies and publish these papers, and then what? The stack of papers just gets taller? I’ve never come up with satisfying answers. But now I finally understand why.” Today’s episode features two interviews. First, I talk to Adam about his big-picture critique of his own field: how psychology too often fails as a science, and what it can do better. Second, we speak with journalist Dan Engber from The Atlantic, who has been reporting on a billowing scandal in psychology that has enveloped several business school stars—and raised important questions about the field. What is psychology for? What would progress in psychology mean? And how can this field—which might be the discipline I follow than any other in academia—become more of a science? If you have questions, observations, or ideas for future episodes, email us at [e-mail]. Host: Derek Thompson Guests: Adam Mastroianni and Daniel Engber Producer: Devon Baroldi Links: “Is psychology going to Cincinnati?” by Adam Mastroianni  https://www.experimental-history.com/p/is-psychology-going-to-cincinnati "I’m so sorry for psychology’s loss, whatever it is" by Adam Mastroianni  https://www.experimental-history.com/p/im-so-sorry-for-psychologys-loss#footnote-anchor-3-136506668 “The Business-School Scandal That Just Keeps Getting Bigger” by Daniel Engber  https://www.theatlantic.com/magazine/archive/2025/01/business-school-fraud-research/680669/ Learn more about your ad choices. Visit podcastchoices.com/adchoices

Did you know that scientific studies have found most people lie once every 10 minutes? In my new podcast, Truthless, I'm talking to people about the lies, they tell, from faking illnesses in high-pressure moments to making up stories on national TV. From Spotify and the Ringer Podcast Network, I'm Brian Phillips. Listen to Truthless on Spotify or wherever you get your podcasts. Today's episode must start with a confession. I adore psychology. I wish I had majored in it.

Some of my favorite episodes that we've done on this show have been with psychologists who I think do interesting, important, and true work. But for a long time, my love of psychology has been intertwined with this other feeling, a darker feeling. But sometimes when I'm reading a book or a paper that is, is filling me with that best feeling, that gosh-wow sense of wonder, at the majesty of the human mind,

the thing that I'm reading is not real. In the last decade, several major findings in social psychology have turned out to be hogwash, according to a phenomenon known in the field as the replication crisis. For example, perhaps you've heard of power poses based in a 2010 study finding that subjects reported stronger feelings of power after they posed, say, with their hands on their hips for several

minutes. When psychologists tried that experiment again, it did not replicate. Or perhaps you've heard of ego depletion. Ego depletion, this even more famous assertion that when people make a bunch of decisions, it exhausts their ability to make future decisions. In follow-up research, that finding, too, did not replicate. Now, science is hard. Digging out fundamental truths about how our brains work is difficult stuff. And I think psychologists deserve a certain amount of latitude, as all scientists do. Researchers who are doing their best can reach conclusions that aren't as solid in psychology as, say, the laws of physics. But the problems in this field go deeper than replication.

In the last few years, a crisis has been billowing at some of America's most prestigious business schools, where the work of academics that has made its way into TED talks and best-selling books is slowly being revealed to be based on not just mistakes, but in some cases, outright fraud. Adam Astriani, a psychologist who writes at the site Experimental History, put the crisis in psychology this way. Quote, There's a thought that's haunted me for years.

We're doing all this research, psychology, but are we learning anything? We run these studies and publish these papers, and then what? The stack of papers just gets taller? I've never come up with satisfying answers, but now I finally understand why. End quote. Today's episode features two interviews. First, I talk to Adam, a social psychologist who does research and also writes at this

newsletter, this wonderful newsletter, experimental history, about his big-picture critique of the field, how psychology too often fails as a science and what it could do better. And second, we speak with the Atlantic journalist, Dan Engber. We talk about his reporting on a billowing scandal in psychology that has enveloped several business school stars and raised big questions about the future of the field. What is psychology for? what would progress in psychology actually mean? And how can this field, which might be the discipline that I like more than any other in academia,

how can it become more of a science? I'm Derek Thompson. This is plain English. Adam Astriani, welcome to the show. Thanks for having me. So we are about to engage in what I hope will be a really fun, and adventurous critique of the field of psychology. But I want to begin by saying some nice things about psychology.

Starting with, why did you become a psychologist? Was there a moment in a classroom, reading a book, watching people at a high school party where you thought, holy shit, this is it. This is how I have to spend the rest of my life? Yes, I remember the moment very specifically. I was taking a social psychology class. We were reviewing for the midterm and playing a Jeopardy game to review.

And I answered one of the bottom row questions, right, like the ones that are supposed to be the hardest. And the graduate student who was, like, leading this review session looked at me and was like, you're pretty good at this? And I was like, me? Good. And then I thought like, oh, I do work pretty hard at this. That must mean that I like it. Like, oh, that's what self-perception theory would say, that you can infer your preferences from looking at your behaviors as if you're observing yourself from the outside. And I was like, oh, maybe I should. Maybe I should be a psychologist. And around that time I got an email that like there's this psychologist named Dan Gilbert who's looking for research assistants for the summer. And I just read his book, which is about how we're really bad at predicting what makes us happy.

And I was like, this guy lives on earth. You can go hang out with him. And like you can do this stuff with him. And I did. And it was so much fun to run studies about like, oh, you think this is how you're going to feel. well, we'll find out if that is really how you feel. I mean, it felt like opening a backdoor to the mind. Like, who, I don't understand how anybody does anything else.

What do you think is the most important thing that the field of psychology has done for human knowledge? I think it's done a really good job at overturning our folk psychology. So, like, before there was science, humans had to make sense of the world somehow. And we made sense of it with what psychologists now call like folk intuitions, folk theories. So you got your folk psychology, your folk biology, your folk physics. And I think our folk psychology of all of these is the thickest and the deepest because we need to know the most about how other humans behave and how they work and how ourselves work as well. You don't need to know that much physics to get through life. You need to know enough to like not fall off a cliff or like, you know, run your SUV into a bunch of pedestrians.

but other people are really complicated, and there's a lot you need to know about them. And so we develop all of these rules of thumb or all these sort of like general theories about other people that now that we have a thing called science, we can test. And this is where I think psychology has been really successful, is taking these things, is eliciting these beliefs that we have and putting them to the test. I mean, that's what brought me in in the first place is like, oh, we have this theory that when bad things happen to us will be sad forever. And then it turns out that's not true, that remarkably quickly you go back to feeling the way that you always feel. That to me felt

like a revelation, and it felt like a permanent upgrade to my life, to know that. And so I think that's what we've done really well, and that's how we've benefited humanity. You write that if we had to boil down everything that psychology, and in particular social psychology, has figured out about the human mind, all that knowledge really fits into, two buckets that you call proto-paradimes. The first proto-paradigm is cognitive bias. What is it that we've fundamentally uncovered here? And how long have we been at it? So I think we've been at this generously for like 50 years. So a lot of people would start this history with Danny Kahneman and Amos Tversky publishing their landmark papers in the 70s about heuristics and biases. And that

created a whole field. And that went on to win basically two different Nobel prizes, not even in psychology, because there isn't one. They won Nobel Prize. These ideas won Nobel Prizes in economics, because that's the closest thing. And that work, I mean, is really cool. And you can tell because everybody knows about it, right? Like, people, people know those words. People know about, like, the availability, heuristic. Like, these are things that if you wanted to entertain people by telling them about them, they'd be like, we've heard all this before. Do you have a favorite? Do you have a favorite

Connman's Versky bias? Yeah, the ones that I like the most are the ones that you can replicate in the course of a conversation. So like availability, do you think there are more words that start with R or that have R in the third position?

And people are like, more words that start with R. But it actually turns out there's more words that have R in the third position and it's because it's easier to think of words that start with R than it is to think of words that have R as the third letter. And this is a nice little demonstration that one of the rules of thumb that we have

for estimating how common something is is by how easily it comes to mind. And this shows up not just in something silly like estimating the number of words, but also estimating the danger of various things, like how safe is it to get on an airplane versus to get into a car? It's easy to think of plane crashes, hard to think of car crashes, even though there are many, many, many more of the second one than there are the first. The second proto-paradigm that you say psychology has gifted us in the last half century is the concept that situations matter. What does this mean? It's funny because now it seems almost silly that that would be interesting.

And I think it's because of the success of this work. So there was a moment, I mean, maybe the entirety of human history up until we started doing this work, where people had a reasonable theory that like there are just certain kinds of people in the world. and certain kinds of people are capable of something and other people aren't. And what this work did was essentially through a series of elaborate pantomimes show that you can create situations where people do things that you thought that only the craziest kind of people were capable of doing. The classic one here is the Milgram Shock experiments from the 1960s where people aren't familiar with these. Basically, you come in and you think you're teaching someone in a number. other room, how to learn words, and you shock them when they get them wrong. And through some stage

managing and sleight of hand, it actually turns out you're not delivering shocks, but two-thirds of people in this study shocked someone basically to death, or so they thought. There have, by the way, been some attempted debunkings of that work that, although lots of that work from that time has been debunked, these, I think, have survived those attempts. And so those, I think, really stuck around because they are such incredible demonstrations of the power of the situation. Oh, you thought that only an evil person was capable of shocking a stranger to death?

No, it's your neighbors. It's you. In the right situation, you could do that. And the fact that now that doesn't seem so surprising is because that work did so well at demonstrating that. So in addition to having some nice things to say about psychology, one really fascinating theme of your work is that you think this field has actually failed in many ways

to achieve the kind of progress that other fields of science have achieved. And from reading your work, I would say that there are three strikes that you make against the field of psychology, three big picture critiques. And I want to go through those one, two, three. The first is that many of the most famous findings, including findings, in these two buckets we just talked about,

cognitive bias and situations matter, just don't seem to replicate. That is to say that when you do these studies over and over again, they come to different conclusions in terms of what happens or what the strength of the finding is. And the proper context here to set you up is that nine years ago in 2015, Brian Nozik at the University of Virginia published, or excuse me, persuaded several hundred psychologists to redo 100 published psychological experiments to see if they would get the same result. And many of the studies failed to replicate.

And this became famously known as the replication crisis. in psychology. We've been living in the world of the replication crisis in psychology now for nine, almost 10 years. What do you think is the most underrated consequence of the replication crisis for psychology? Why is it important? I think there are, if I can do two, there are two, one good and one bad. The good underrated consequence, and you can see some people saying this now, but it takes a while for everyone to catch up that's not like watching this very closely, is that we're much better at doing this now, that there are things that you could do, even 10 years ago, that would get you laughed out of a room now if you tried to put them up in terms

of, you know, dropping people from your studies because like, oh, they're outliers or testing your effect as it goes along and then stopping if it's significant. These are things that nobody, I shouldn't say nobody, that most people thought were fine things. to do. And through some really effective demonstrations, we know that now we shouldn't do them, because it inflates basically how easy it is to show that something is statistically significant. That's the good news. The underrated bad news is I think we've learned the wrong lesson from the replication crisis, that now the first question that we ask of any work is, is it true? Which is a sensible question to ask, but we should instead ask a different question first,

which is, would it matter if it were true? And the thing that I think we haven't learned about the replication crisis is most of the work that doesn't replicate didn't matter in the first place. Like, I didn't care that it happened the first time, so I don't care if it happens the second time. And I think a piece of evidence for this is in that study you just cited, you know, these hundred studies we try to replicate, nobody can tell you which ones they were, which you might think would be like, well, wouldn't we be interested to be like, oh, man, is it one that I rely on? And it's not because nobody really relies on anything. Like we each create these little research worlds unto ourselves. And that is, I think, the underrated bad consequence that we've learned to ask the questions in the wrong order. This is such a subtly important point.

And it was fairly mind-blowing to read it in an essay published several months ago. And I just want to pause and dilate on it because it is really significant. If in the field of physics or chemistry, there was an attempt to... to replicate 100 of the most important studies, and 40 to 60 of them failed to replicate, the world would change, right? We would say, like, oh, my God, like chemistry doesn't work the way we thought it did.

We thought that we were poisoning the atmosphere with carbon dioxide. It turns out that we were just actually emitting stuff that wasn't carbon dioxide at all into the atmosphere, and there's no global warming. Or, oh, all of the ideas from chemistry have been totally changed. That's not steel at all. There's no difference between steel and cardboard. They're actually the exact same material.

Like, these are the kind of things that would follow a revelation that half of what we assumed in the fields of chemistry and physics turned out to be false. But instead, dozens and dozens of conclusions from the field of psychology were found not to be true. Things like social priming or subliminal exposure, right? Like, you know, having little flashes of the word like, you know, rats in an ad that's trying to get you to hate Democrats and get people to associate Democrats. the rats, ego depletion, the idea that we have a limited supply of willpower. So if I go to the

grocery store and my wife makes me order 17 things rather than two things, I'll have less energy when I come back to the house, all of these things turned out to not be necessarily real. And it seems to have changed the average person's life not one bit. And that has to, I think, at some level, create some kind of crisis of confidence or existential despair in psychologist to think, what if our entire life's work is zeroed out and no one's life is changed because of it? Do you buy into this rant that I just did? Do you think that it is a little bit of an existential crisis for psychology to have this replication crisis happen? And most people's lives just move on exactly as they did before? I think it should be, but it generally isn't.

I mean, I grew up basically as a psychologist while this crisis was unfolding. And certainly the that we all had was like, oh, wow, the stuff we learned in Psych 101, like, we were lied to, basically, like, how dare they? But I think no one ended up asking the second question, which was like, why were we, why did we care in the first place? Like, a lot of these are cool little stories, but what is writing on the fact of, like, does holding a pen in a certain way in your mouth make you think jokes are funnier versus less funny, which is one that people have really spent some time trying to figure out whether it's true or not? Like, what were the... I just want to pause. This is, because this is...

a real study. It was a real study that was done of whether or not a joke is funnier depending on whether the pen is held in your teeth or by your lips and not touching your teeth. And this was overturned? Yes

and no. I think the most recent version, there may be something more updated, but the last I checked on that finding, it was, we think it might be there if you're not recording people. So they ran a study where they like had a camera or not. And like, oh, maybe we're making people self-conscious.

And then other people were like, actually, you look at that after the fact. Like, this is just trying to make the effects show up. So I don't know if we've settled this one. And I'm not that interested to know how it settles. Because, like, what is this really about? This isn't some parlor trick. This was meant to be about, like, well, can people's physical, like, posture or the way your face moves influence the way you feel about something? Well, like, I certainly believe it can.

Does it in this situation? Well, like, maybe, maybe not. But that doesn't really change whether or not I think it is at all possible to construct any situation in which someone's like physical arrangement could change how they feel about something. It's like it's not, it makes no sense to put so much on that one demonstration. If we tried everything we could and could never find a situation in which like moving someone's body around makes them feel different, I would both be very surprised, but I'd also be

like, well, I guess that reasonable sounding hypothesis is wrong, but it can't be solved by being really sure about whether this pen thing works or not. The second major strike that you make against the field of psychology is that in addition to this replication meaning crisis, many assumptions that we hold deeply in psychology actually don't seem to hold up against what you call folk wisdom, you have a very entertaining example of personality tests. There's a theory of personality in psychology, the Big Five theory, that has been used in hundreds, thousands of papers. But when you put it to the test against personality tests that people just pull out of their butts, it doesn't seem to do that much better in terms of predicting behavior,

predicting personality. Can you unpack that point a little bit for us? First, explain what is the Big Five theory? And then also for people who, you know, might hold on to the Big Five theory as being their way of expanding person out of themselves, what is it that we've started to learn about the Big Five theory not actually working? Yeah. The Big Five is like the Honda Civic of personality psychology. It's solid. Everybody thinks it's great. You know, some people would like a different model that's a little fancier or a little less fancy, but like this one satisfies. And the idea is that everything about human personality comes down to these five factors. Openness to experience, conscientiousness, extroversion, agreeableness, and what used to be called neuroticism

and is now more politically correctly referred to as emotional stability so that no one has to feel bad about scoring high on it. And the idea here, and the thing that it has worked out empirically over and over again in studies, is that whatever you measure about someone's personality ends up falling on one of these five factors. So you can reduce all of that data down to these five things. And we've run these studies for decades. This has been like a great achievement. As you said, thousands of studies, probably millions of dollars. And one reason it's stuck around is because it does a good job at predicting other things.

That if I get you to take one of these big five questionnaires, I'm good at figuring. I am good at finding out like, oh, are you married? Are you likely to have this kind of job versus that kind of job? Are you likely to commit a crime? So that's one way of testing whether it works. Another way is testing it against basically a personality test that was not developed scientifically. Like, for instance, the Enneagram, which comes out of a spiritual tradition.

And no offense to what they're trying to do a different thing, right? They're trying to help people better understand themselves. they weren't trying to run, like make an empirical scientific instrument. But they have this questionnaire, and it will tell you you're one of nine types. And so this group called Clearer Thinking earlier this year tested these against each other. They had people take a Big Five questionnaire and also take an Enigram. And by the way, take a few other, the Myers-Brigs inspired a test because the Myers-Briggs is actually proprietary, as well as they figured out their astrological sun sign.

And they just tested like, okay, which of these does the best at predicting other? things like, are you married? Have you bought a house? Have you been convicted of a crime? These life outcomes that we might be interested. And when you put them head to head, the Big Five and the Enneagram perform just as well as each other, which is a humbling moment, or at least it should be for us, that this thing that we worked on really hard and poured a lot into, like, it should have blown the other test out of the water. But in fact, we are performing on par with people who had no intention even of performing well in this way, and certainly didn't refine this with years of study and evidence.

So that should make us sweat a little bit. So up to now, we've established that too much psychology, A, doesn't replicate, B, ask questions that don't matter. C, fail to predict human behavior better than ideas more or less made up by spiritual teachers. And God bless spiritual teachers, but that's not the field of academic psychology. The final strike that you make against the field of psychology is that it has struggled to turn its knowledge into technology. And that might seem a little bit ethereal to some listeners at first.

But I do think it's important to remember that the reason to study something, the reason to glean science from the cosmos should ultimately be to make things that wouldn't otherwise be possible, right? That you build bridges with actual physics. This is your point. I think I'm stealing. You build bridges with actual physics, not with folk physics. That's why the bridge stands up.

But you show that it doesn't always work out that psychology produces so-called technology that works. Tell us about nudges, what nudges are, and what evidence you have that they haven't worked out in the real world as much as some psychologist and behavioral economists would like. So a nudge is the idea that you can make a small change to someone's environment and get a big change in their behavior. And so, like, a classic example is you change how the food is laid out in the cafeteria, and as a result, people eat more vegetables because you put them in the, you know, closer to where people enter or something like that. This has been hot for now, like, 10, 15 years. and there's been a lot of research on it and a lot of people showing like, look, we can nudge people this way

and we can nudge people that way. And I think really the way to see whether this approach works, whether we've nailed it, whether we've figured it out, is to look at like, well, what's the state of the art? The state of the art right now

is this approach called the mega study where they test a bunch of interventions at the same time to see which ones work. There was a paper that came out two years ago that was trying to do this to get people to go to the gym more often, which is already like people already go to the gym, people want to go to the gym, right? This isn't like, this isn't smoking cessation.

This isn't trying to get people to switch their vote. Like, this isn't trying to get people to levitate or go to the moon. This is like very low level, like trying to help people do something they already do and want to do more. It's sort of like if nudges should be able to do anything, they should allow people to align their preferred life. I go to the gym all the time with their lived life. I actually do go to the gym all the time, right? Yes, yes. Yes, we should be able to unblock the dam between people's intentions and their behavior.

And these are people who did this who are like they are top of the field. Like, this is as good as it gets. And what approach did they take? They were like, look, we don't know which approach is going to work. So let's try 53 different interventions based on our best guesses. Half of those interventions did not produce detectable effects. They did not help people go to the gym more often. And in that same study, they had experts in the field, behavioral scientists and public health experts,

try to predict which interventions were going to work, and they couldn't do it. Not only did they dramatically overestimate how effective the interventions would be, they couldn't predict which ones would work and which ones wouldn't. And in fact, in another mega study that was about vaccine uptake, actually non-experformed experts in terms of predicting which interventions would work. So, like, is it, can we in any situation use nudges to, like, help people live better lives? Yes. But when it really comes time to, like, okay, in this situation, what should we do?

We really don't know. And not even to mention, like, okay, we did this with this gym. Would it work for getting people to return their library books on time? Would it work for a yoga studio instead of a gym? Would it work for helping people stop smoking? Like, for all those, we don't know. Like, we'd have to run another mega study. You have a very funny line in the piece that says,

this isn't super reassuring. This is your reaction to the mega study where only half of the 53 nudges worked and it was difficult to predict which ones. You say, quote, this isn't super reassuring. If I hired a plumber to install a toilet in my house and he was like, sure thing, I'll just install 53 different toilets

and then check which ones flush. end quote that's that's a very good um relation of here's uh here's a physical world property that we understand uh water go down and here's another physical world property we don't entirely understand which is behavioral nudges to get people to go to the gym and lift weights i think we've done a really good job here of establishing the baseline critique that too much psychology doesn't replicate ask questions that fundamentally don't matter fails often to predict human behavior better than folk wisdom, and finally fails to translate into effective policy at better than a coin flip 50-50 rate. So you publish these articles. They're read by psychologists, by people who have,

like you, devoted their life to hopefully not entirely failed enterprise, but at least a difficult feel to push forward in terms of human progress. What's been the feedback? I mean, surely some folks have said, you're a jerk, you don't know what you're talking about. What's the feedback been like? Yeah, I gave a talk at a conference of social psychologist a couple weeks ago. And the reactions spanned from people nodding along and going like, yes, yes, it's all a lie. And like, hey, man, what are you going to do? You know, you got to make a limit somehow. And other people who were livid, you know, people would come up to me afterward and go like, no, no, we're really good at making technology with psychology. I mean, just look at, you know, social media. Those are,

like, those are made by psychologists and they're controlling our minds. And I'm like, I understand, I understand this critique. What knowledge from psychology are they using? Like, can you point to the papers that they are like then taking and feeding into the algorithm or they're using to, to like control our minds? And he's like, well, we can't know because they won't tell us. And I'm like, but these are, we trained these students. We taught them everything they knew. Like, what is it that they know that we don't know? And I think in fact, like, that's been very much overblown as to, like, the usefulness of findings from social psychology for, like, making social media more addictive, that I bet,

I mean, we don't know for sure, but I bet that most of what makes social media work, they've discovered from their own testing. Because, again, like, these people come from our field. We would know, like, we, they publish papers. So if it was something like that, we would know. Just as an example here, years ago, there was this Cambridge Analytica scandal where basically the upshot was like, there's this company that's scraping your Facebook data illegally. And they're using it to like micro target ads to you that then mind control you and make you vote for Trump. And I think this was like a really useful scapegoat for people who were surprised and dismayed that Donald Trump won the election.

And then some people tried. Like some academic psychologists were like, hey, can we do this, by the way, this whole micro-targeting thing? And they published a paper where they're like, yeah, it doesn't, well, they couldn't really get it to work very. Like, micro-targeting helped like a little bit. But it actually turns out to be really hard to persuade people of things. And like, yeah, it doesn't matter that you know which Hogwarts house they're in

because they took a Facebook test and now you like stole their data. That it just turns out like, yeah, knowing someone is a Slytherin versus a Hufflepuff, like it's maybe a little bit better than just guessing, but like it doesn't, you know, open up their skull and allow you like muck around in there. So let me give you my own feedback. I am persuaded by so much of what you've said,

but there's two bumps that I have when I hear your argument. The first is that even if psychology isn't a hard science that produces mathematical truths like in physics that you can use to design structures, like nuclear power plants, the truths of psychology are still true enough. That is to say, an advertising department that understands cognitive bias really can make a commercial that's 5 to 10% more effective. And a salesperson who understands social psychology can be a 5% to 10% better salesperson

or a listener who understands psychology can be a 10% better friend. And so even if psychology is doomed to messy truths, those messy truths are true enough to be useful. The second is that in clinical psychology, because my wife's a clinical psychologist, I know that, for example, cognitive behavioral therapy, CBT, has been shown to work very well, not just in the U.S., you know, weird, Western, educated, industrialized, rich countries, but also around the world. in 2022, Christopher Blatman and several other researchers tried CBT in Liberia and found that it had strong effects among the population there in terms of reducing anxiety and depression. So how do you feel about those two fronts of a critique? Number one, that psychology might not be physics,

but it's certainly better than nothing, and so it can absolutely help some people. And number two, that there seems to be some advances in terms of turning the science of psychology into the technology of something like CBT. Yeah, for both of these, I would say like totally possible. But we're not sure enough that I think the costs are clearly outweighed by the benefits. So to the first one, yeah, it's possible that like advertisers and marketers do a little bit better because they know about things like social norms or social proof or they've read some studies about persuasion. At the same time, like we actually have, I mean, this is purely an anecdote, but I think a representative one where this year, there was this big project trying to basically get Democrats and Republicans to like

each other more. It was called the Strengthening Democracy Challenge. And people could send in the interventions they thought were going to do this to like, you know, get people after a short intervention to say that they like the opposite party more. One of the interventions that worked the best was a Heineken commercial from 2017. That was like not made, as far as I'm aware, like based on they read a bunch of literature. They just showed people making and like a piece of IKEA furniture together and then having a beer. It's a great commercial.

It makes me want to buy Heineken beer. It also, I guess, makes people feel a little nicer toward the other side. But that was not made by psychologists like trying to do that. That was made by, you know, beer marketers trying to move units of beer. And so, and so like, yeah, it's possible they had a head start because they were, you know, they had read a book of psychology. but the head start doesn't seem to be very big. To your point about CBT, so I'm a social psychologist, so I don't treat patients,

and so this is outside of my direct area of expertise, but as some of what I read for writing this piece, there is an ongoing, there is a, I would say, significant minority of folks who make the point that like it kind of seems like all psychotherapies work equally well. It's at least hard to tell the difference between them. In fact, there was one paper I read where the author was like, we should stop developing new therapies because it seems like they all work. And what we really need to do is just get them to more people.

And look, I don't know enough to weigh in conclusively on one side of that or the other. But it does seem like the margins are close enough that it's not a slam dunk. And so again, I don't think it's that psychology has made zero progress. But what we're arguing about is whether the confidence in interval includes zero, I think we should be worried about the state of the field. What is your grand theory here? Do you think social psychologists are dumb, or do you think that social psychology is young, that this is a difficult field, and people aren't atoms, and psychological truths are in many ways much more complicated than those in chemistry or physics,

because people are incomprehensible, including to themselves. And so this field is, you know, like my daughter, just learning to walk and talk and use utensils. This is just an infant discipline. Yeah, it's exactly the way I feel that I know all this can read as like, psychology is so dumb. You guys are such idiots for doing it. It's not the way I feel at all. I actually feel like we're just getting started. but the real mistake that we could make

is thinking that we're done thinking that we're just crossing the T's, dotting the eyes, and what we figured out so far is like, we'll discover some, a few more things, but like this is pretty much the way that it is. And you know,

like paradigm shifts, that's a thing that used to happen and doesn't happen anymore. Like, that's the reason why I care so much about making people dissatisfied with it because you need to be dissatisfied before you start to entertain

the sort of strange ideas that might bring about that paradigm shift. So yeah, this isn't like, you know, lecturing a 13-year-old that they should be 40. Like, it's fine to be 13 years old. It's not fine to be 13 years old and to think that you're done growing because that makes it a lot harder to grow.

I think a good way to move a field forward can sometimes be to ask questions that are so fundamental that they might even seem stupid. And you have a couple of these questions at the end of a recent essay, including the following. The average American watches 2.7 hours of television a day.

We write this off as leisure as if that's an explanation. But why is it fun to watch someone make a salad on TV? Question mark. Why are you interested in this question of why it's fun for people to watch salads being made on TV? Because it's weird. Why do people like doing that? What are they getting out of it? You can't eat the salad.

And they spend so much time. I'm doing it. Right. Like, it's obviously not everybody's watching a salad get made. But, you know, when people are bored at work, what they long to do is come home and watch people do things on a screen. Why? What is that? Like, any good explanation in psychology will have to explain how people spend the majority of what we call their leisure time. When people can do whatever they want to do, they like to do this. Why this? And this is the kind of mindset that I, that I want to inculcate to myself and others. because I think it's how psychology got started in the first place. Like, you know, there's a lot of things that make psychology complicated. You know, you don't have to ask the permission of protons to, like, smash them together. You do have to ask that of people, whatever. But I think it's really interesting that it took us so long to start at all. Like, you know, if you think of Galileo as the first physicist, we're talking like 1600,

the first experimental psychologist, like the first actual person that we might consider a psychologist, not until 1860, 250 years later. Why is that? I think it's because people actually felt pretty satisfied with what they thought they knew about how their minds work and how other people's minds work. And like the first guy who like got this started, he only got started because like it's sort of a long story,

but he stared at the sun for too long because he was trying to study like after images. He was a physicist, but first, he stared at the sun too long. He went blind, get really depressed because he's like bedridden. his sight's gone. At one point, he undergoes moxibustin, which is where they put this root called,

or this weed called mugwort on your skin and set it on fire. It's maybe a psychedelic. Maybe he's like tripping. He eventually gets his sight back. And he's like, what I need to do is start a new religion. He writes these books about his new religion and they don't go anywhere. And they're like, what I really need to do is provide a scientific basis for my new religion. And what he starts doing is what we now call psychophysics, which is like,

you know, you show people like a flickering candle and you see, like, can you tell that it's there or not and from what distance and how much intensity? And he lays down the first laws of psychology. This guy, Gustav Fechner, who by the way, like, trains, like him and his colleagues train the first generation of psychologists. I can trace my academic lineage back to that guy who like went blind, got high, invented a religion, and then started doing experimental psychology. Like, why did it take that? I think it took that because he had to be alienated from what he thought he knew about how the mind works. And like, I'm not going to stare into the sun or set myself on fire, so instead I have to ask, why do people make salads on TV? I do love this idea that sometimes the most interesting questions can be asked when you try to see the world. Like you say like an alien. Sometimes I think of it as like a tourist, right, as if all these things that are familiar to you, are instead pieces of a foreign country.

And what if you went about the world thinking like a tourist? It can be a really interesting way, I think, to come up with novel questions. Or actually, to see the novelty and familiar questions. One of the question that I'm very interested in, and I would love to close on any thoughts that you have here, is I have always been so interested in the science of creativity, while nonetheless recognizing that the, quote, science of creativity is not much of a science.

It is an attempt at a science. I think people are doing their best. But how people come up with new ideas and new scientific paradigms, new novels, new salad recipes to use one of your favorite examples. This is an amazingly interesting mystery. And the implications are so profound,

not only because, It would be amazing if there were some kind of formula for creativity, but also because some of the most powerful people in the world right now are trying to build creative machines. And I think in many ways failing to do it, in part because what they understand how to do is how to take a synthesis of all of the knowledge that exists and mix and match it in a kind of pastiche to create next tokens. And if there were some signs of creativity, the things that we could do with these machines be really, really fascinating and magnificent.

How would you even think about pushing forward the frontier on something as oversubscribed and yet difficult to make progress on as the science of creativity? I would start by banning the word creativity from my mind. Okay. I mean, the great piece of advice that I got from my advisor was study verbs, not nouns. You know, there's no thing in the mind called creativity. It's a thing we use to describe what people do. So focus on what is it they do.

Well, what they do is they come out of dreams and they write the song yesterday. At least that's what Paul McCartney said he did. He said it came to him in a dream. We got mathematicians saying the same thing. And so I would start with the fact that like it really seems like whatever is going on here is beyond the reach of people's individual consciousness. and so like what's going on there? Is it like an offline subconscious recombination of experiences that people have had? In what sorts of situations?

Like, can we make those dreams happen more often? What was it that caused that to happen? Like, did you need to be thinking of songs all day before you go to sleep? And then sometimes you come out with yesterday? Or is it you need to, you know, hear some song and kind of half forget it and then you rearrange it? Yeah. I don't know how to solve the mystery other than to begin by embracing it and like to look for the parts of it that are the most mysterious, which is we appear to be, you know, riding atop of this vast subconscious that seems to be doing all the recombinating and generating and then it feeds it back up to the conscious mind.

And that this is both true for musicians and for mathematicians. And so maybe those things that result in very different outputs are, in fact, the same process. which would be really interesting if in fact the mind treats those things similarly, although we think of them as totally different. So that's where I would start. Adam, thank you so much. This was really fun. Thanks for having me. That was the social psychologist Adam Astriani. Next up is our conversation with the Atlantic journalist Dan Engber and his reporting on a billowing scandal in business school psychology.

Dan Engber, welcome to the show. Thanks, Eric. You have a story to tell us, and while I think it's a great and important story, it's not a simple one. So I want to try to go as simply and chronologically as we can here. Why don't you start us off by telling us about Francesca Gino and the blog post that began this whole sort of affair? Sure. So Francesca Gino is one of those very famous people you may never have heard of. she is or was a superstar business school professor at Harvard she wrote a bunch of books one of them was called Rebel Talent her research was on kind of dishonesty and rule breaking and how those relate to creativity and success in business and life

she had a very successful career doing you know, public speaking, corporate gigs, TED talks, that kind of thing. And in summer of 2003, two things happened. One, she was placed on administrative leave by Harvard Business School. Behind the scenes, there had been a long investigation of her work. And the other thing that happened was a group of her business school colleagues published a blog post. That group is called Data Collada.

That's just their kind of blog name, where they laid out evidence of fraud in four of Gino's papers. And it turned out that the internal investigation at Harvard Business School was looking at the same four papers. In fact, the Data Collada bloggers had tipped off Harvard to their work, and then they'd held back the blog post until the investigation was complete. Okay, so this was a huge story in, the world of scientific fraud.

Scientific fraud is discovered on a fairly regular basis, but this one was so juicy, mostly because of the fact that Gino was herself an expert on, as I said, breaking rules and lying and also, you know, very successful. So this was, it made international headlines. It was a big deal. And you're right, Derek. This is such a complicated story because there's like another part.

There's like another twist. We haven't scratched the surface here. Yeah. There's another twist just in the setup here, which I mean, I could go right into, which is that one of the four papers that had been flagged for apparent fraud, the same group of bloggers had already identified potential fraud from another data set used within the same paper, from another famous, you know, superstar business school professor.

or Dan Ariely, who's written, you know, best-selling books of his own. So you had this totally just, I mean, unprecedented situation where here was a very important paper in the business school psychology literature that had, you know, many prominent academics as co-authors. And two separate studies within that paper had separately been identified as being potentially fraudulent. So it was, I mean, for me and I follow these kinds of. scientific misconduct stories for a long time. It was mind-blowing. So that was June of

2023. That's the starting point. People will be listening to this show initially during Thanksgiving week, and it occurs to me that there's a famous dish popularized by the NFL coach and announcer John Madden called the turduckin, which is turkey that has within it a duck that has within it chicken. And this is sort of like a turduck and a fraud, right? Because the paper had fraud in it, but it contained within it data sets that had been separately identified as being fraudulent. And to your point, I'm using your language here, you summarized this scandal by saying she was accused of faking data for a study about when and how people might fake data. And Dan Ariely, one of the co-authors or contributors to this piece, is a celebrated expert in the psychology

of telling lies. So this scandal is already fully in the realm of if you turned in a fiction manuscript to a publisher, they might reject it for being too heavy-handed. So this is already a famous scandal within academic psychology, and people who might have seen it reported the New York Times or, for whatever reason, are interested in scandals and academic psychology, they might have already begun to have heard about this chapter one of the story you're telling here. What did you initially want to add to the story? What made you initially interested in reporting out your own angle here? So it was what came right after that, how the community responded to what had happened.

So one of the things that, you know, was obvious from the start to people in the note, people have thought about this kind of thing is, okay, if you've got, we know there's fraud in science, there has to be, just in the nature of things. If you had this coincidence of two potentially fraudulent data sets within the same paper, what does that tell you about the base rate of fraud in science generally within this field, within this particular community of academics who work together within this field? And that's a really important question. And you think it's one we might know the answer to, but we simply don't.

And the reason we don't is because there's never any kind of broad audit of the the scientific literature in any field to figure out what the base rate of fraud might be. So we had this tantalizing and disturbing clue that that base rate might be much higher than people generally believe. So, I mean, that was just very interesting to me as someone who's followed these issues for a long time. But what happened next is that Gino's colleagues and Ariely's colleagues in business school psychology decided, okay, we've got a problem here, which is We have four papers from this one researcher that seem like they could be fraudulent. She published a lot.

I mean, she was incredibly prolific. She has something like 138, I think, co-authored papers with other, you know, stars in the field, but also young academics. And the problem is, okay, we just don't know how many of the other 134 are problematic. We don't know, you know, on some of those papers, maybe she was just, just like the fourth author out of seven authors or something and didn't really contribute anything meaningful to the actual research, maybe we should consider that paper, you know, totally fine. We shouldn't be suspicious of it at all. But if there was another paper where she was, you know, a more significant co-author, let's say she had actually run some of the studies herself,

well, what do we do with that? No one's gone to the trouble of investigating it for 18 months behind closed doors, should we just assume that that study is sound, unless we have specific reason to think otherwise or what? So this, you can imagine how this is just a problem for everyone. It's especially a problem for Gino's younger colleagues who, you know, each paper, each one of these papers could be playing a pivotal role in their shorter CVs in terms of getting grants and getting jobs. So it's kind of a crisis on a.

a very practical level for a lot of people in this field. So what they decided to do was something, again, I've never seen this before. They came together and they said, we're just going to do kind of a giant audit of all of our work. So it was called the many co-authors project, and it referred to the many co-authors of Francesca Gino. The idea was, okay, we're going to set up a central database. And actually, this project was sort of spearheaded by one of the original data collata bloggers who had identified the signs of fraud in the original four papers. And we're just going to ask every single co-author on every single paper to just tell us some basic facts.

Okay, did Francesca Gino run any of the studies? Did she have access to the data? Did the co-authors have access to the data while the paper is being put together? So even just that step, we'll get some, you know, basic questions answered about this. full body of research. And then in a perfect world, there would be further investigation of each of these papers. We want to know, can these data be trusted, again, from 134 papers. So this was both kind of just interesting and inspirational to see happen, how everyone came together to try to test and buttress the integrity of the field. But I also realized, in a sense, this is a sense,

This was something like an audit of, you know, this was not the kind of audit you would want to understand the base rate of fraud within psychology broadly. This was decidedly non-random, right? This is only papers that have Gino as a co-author. But it's kind of like a large-scale look at, you know, scores of papers. And maybe you could get a sense of like, well, how deep do the problems go? So I thought that was kind of a side benefit of this, that it was almost like accidentally, a way of testing a larger sample of academic papers to see how many of them were, you know, deeply flawed. So the reporting on Gino's potential fraud inspires this project to determine whether or not she's a bad apple or whether or not there's a rot inside the barrel here. And for folks thinking, you know, what kind of?

kind of studies are in this many co-authors project. We're talking about research about whether, say, procrastination makes you more creative, research on whether you're better off having fewer choices, an idea that's sometimes called the paradox of choice and is related to this other phenomenon that you've reported on called ego depletion, research suggesting that you can buy happiness by giving things away. These are not studies that change the world necessarily, but they are often very famous studies. They're the subject of viral TED Talks that have millions of views. They're the subjects of airport bestsellers. And as you point, speaking of self-auditing, nearly every single business academic named in this many co-authors project has either been

quoted by or cited by The Atlantic, our employers. So these ideas are very much in the ether. And one of the people who's helping to run this many co-authors project, this huge audit of business school psychology is a woman named Juliana Schroeder. And when you start reporting the story and you reach out to Julianna Schroeder, she just seems to be a paragon of scientific rigor. I mean, she seems to have a fantastic reputation. And the folks you're talking to about her are calling her exceptional and a model researcher. But here's where we have one of the many interesting twists. Just days into the creation of this initiative, she discovers that a paper that she helped to write

suffers from some major data problems. Tell us about this paper, which is called, for those who want to look it up, don't stop believing, rituals improve performance by decreasing anxiety. What's this paper, and what did she find? Okay, I'm sorry.

I would have just back up for one half step just to say that some of those big topics you talked about, paradox of choice, for example, and procrastination, creativity, Those are not specifically represented in the Medico Authors Project. Those are representative of this field, which I'm calling in my story, business school psychology. So these are people who come out of psychology programs often, are using the methods of behavioral science that are common in campus psychology departments.

They're publishing in the same journals as regular psychology departments on campus. but they're located in business schools like Harvard Business School or Wharton, you know, or at Berkeley or the University of Chicago, and they're getting paid a lot more money to do the same kind of research. And it's from that sort of world of business school psychology that you get this hugely influential kind of gee whiz research that shows up so often in the Atlantic and the New York Times and many other publications. and on television, et cetera. So I just wanted to be clear about what was actually included the many co-authors project. I'm very grateful that we're practicing the art of self-audits, even within this interview. I'm not going to pretend that I screwed up that set up purposely, but I'm glad that we had you as a paragon of keeping everything straight on the show.

So, Dan, please continue. Okay, so Juliana Schroeder is someone who had co-authored a bunch of articles with Gino. So obviously, she was very disturbed to find that there was now this evidence of fraud in Gino authored papers, right? So she immediately got involved in the MediCo Authors Project and became one of the leaders of it. Not only that, but she took the work of the many co-authors project to a place that other people who got involved did not. So many people were like, thought it was important. They gave the, you know, the basic level of information about the paper they'd worked on. Oh, yeah, you know, study three of that paper.

Francesca collected the data, but I have no, you know, I've never seen the data. I don't have any particular reason to think that there are problems there, but they would just report those basic facts. What Schroeder did was she said, you know, I'm going to go back and I'm going to audit every study in every paper that I worked on with Gino. and I'm going to post everything to OSF, which is like a platform for open science and sharing data and methods. And, you know, one of the Uri Simonson, one of the data collada bloggers,

he even told me he thought this was kind of overboard. Like he thought the point here was just to, you know, get a handle on what Gino herself had done. Schroeder was treating it more like, no, this is a way of looking at the, you know, the field at large, really trying to see how much of this work stands up, how much of this work is potentially reproducible, but also how much of this work is just clean

in the sense of being without obvious glaring flaws or potential corruption. So, yeah, so she's coming into this as a leader of this push to really test the literature for signs of rot, basically. Yeah, yeah, so thank you for, I think that that step back was really fantastic context. I'm particularly glad that you corrected me on exactly what papers are in the many co-authors project versus what papers are representative of the field that is being studied within the many co-authors project. That's very important.

So let's go to the study specifically. Again, this is a study about the degree to which performing a ritual can decrease your anxiety. and in study 1B of this paper, they are measuring the heart rates of people before and after they perform a ritual. Tell us a little bit about study 1B and what Schroeder and maybe other folks

see in here that is so clearly representative of fraudulent data. Okay, so yeah, very quickly getting into it. Francesco Gino and Harvard Business School, colleague me, Michael Norton, had done this important work, seemed important at the time, let's say, about what Norton would later call the ritual effect. The idea that performing rituals is sort of like performance enhancing and improves your life in many different ways. So a lot of people were excited about that idea in the mid-2010s. Julianna Schroeder was a graduate student at the time.

She was started working on this. There was a graduate student at Wharton named Allison Woodbrooks. She was working on this. They kind of realized they were working on the same thing. They came together. They did this paper. And what Brooks in particular had been doing were these studies using a karaoke machine. And she would have people try to sing Don't Stop Believing, which is pretty hard to sing, I can say from personal experience. And then she would have them do a ritual before singing, don't stop believing, to see if that helped them do a better job, made them less nervous, et cetera.

So the study that we're talking about, Brooks apparently contributed, although there's some dispute about this later, these studies to that paper where people would, again, perform this arbitrary ritual. They would draw scribbles on a piece of paper, crumple it up, and pour salt on it. And then they would perform, don't stop believing. And she would test their heart rate or whoever ran the study would test their heart rate before and after they'd, perform the ritual. And the basic finding of study 1B, which you mentioned, is that if you perform this ritual, it made your heart rate lower, basically. The idea was just like it, it helped you to stay calm and then, you know, you could infer that this would then improve your performance. So, so that was the idea. It's, you know, this isn't going to change the world. But it's, you know,

if you're speaking to, let's say, a business audience, you have people who are making presentations. Maybe that's something that they could use, right? They could go to the bathroom, perform some ritual in front of the sink, and then go back and load up their PowerPoint. I don't know. Anyway, that was the paper.

But when Schroeder started looking at it, she gathered what data they had. They didn't have the raw data anymore, but these data spreadsheets that had been produced after the fact. And they just looked wrong, especially the data spreadsheet for 1B, although there was other ones that looked problematic as well. But in particular, so this is all public now.

So I was able to download these spreadsheets for myself and just be like, well, what looks wrong? And there's just one thing where you do not need a PhD to understand what is wrong with this data set. And it's might be hard to describe in audio form. Can I try to describe it? I want to try to describe it. First of all, we're going to link to your piece,

obviously, in the show notes, because people should read the piece. But this is a graph showing the heart rates of a bunch of people that have just done this ritual of pouring salt on a piece of paper. And if you look at the middle of this distribution of heart rates, it's basically a straight line going up. Now, typically, you would think

that if you measured a bunch of people's heart rates, you'd get a bunch of variability. You know, it's like it's 120 here, it's 50 here. If you line them up randomly, it should look very, very spiky. But this isn't spiky at all. And I was trying to think, like, what's a way to represent this for listeners? It's a little bit like, Dan, tell me if you agree with this. There's a fire drill at the Atlantic.

And everybody at the Atlantic has to line up outside the office spontaneously. And there's a researcher there who comes by and measures our heights. And the results show that 70 of us just happened to line up in perfect order from short to tall. that's about the same as the odds of this data that you're looking at in study 1B of don't stop believing being real. Like, there's just no way that 70 people at the Atlantic just happened to line up outside the office in perfect order of short to tall with every difference being like the exact same, you know, one centimeter. So you look at this data, you look at this graph, and it's clear that someone is essentially just drawing a line and placing a heart. heart rate measurements on it rather than studying a realistic distribution of heart rate measurements after some crazy ritual. Does that sort of, in an audio way, kind of translate to a loose

description of what we're looking at in this data set? Yeah, I think that that's good. I think it's, you know, one thing that you have to keep in mind, it's very tricky. So this forensic data analysis is super subtle stuff. So I don't want to pretend to be an expert, but you could imagine in a situation where you look at a data set like this, and you do see ordered data in this way. And one thing you have to be careful about is, well, maybe there was a reason why a researcher would sort by heart rate, and maybe they, after sorting by heart rate,

assigned, you know, participant numbers to all the people who did the study. You could come up with a story where this sort of happens, but it doesn't make sense in the case of this dataset because it's not like every point, data point in the study, was in this order, it's just a run of about a hundred of them in the middle. So it's just one of the red flags that kept coming up when I was talking to data experts about this study was it's just, it's so hard, I would say impossible to come up with a narrative of how this happens, right? And so that's the people who, the fraud detection experts, that's what they

look for. They're constantly thinking, I mean, of the null hypothesis, maybe, if you want to use science language. They're constantly thinking about how could you explain this other than someone made up data? And this one is just a study where it is so hard to come up with an alternative explanation than data were made up, right? So, but your description of the fire drill is good. So, I was like fishing for a compliment there, but I really did spend time like looking at this graph and thinking, how the hell do I describe this on radio? That's the best I could partially do. I want to keep the story going because there's about to be another turn that's really, really important to the narrative. So Schroeder has already, in the process of combing through

all of this work, revealed that there are major problems with this paper on the ritual effect. But that's not all. You report that as she continues to review papers associated with Francesco Gino, she comes around to looking at a paper that she's worked on that is also about the power of rituals, which came out in 2018 in the Journal of Personality and Social Psychology. And this is where the story starts to get very naughty. Tell us what happens here. Sure. But can I just make clear what the significance of the previous finding was to you? Oh, absolutely. Yeah, sure, please. Again, when they started this, people thought, oh, you know, Francesca Gino was associated with

four papers that had, you know, suspicious data. I wonder if there are more. That looks like a pattern of behavior. But when Trotter found this problem that we're just talking about with the run of data and the don't stop believing study, she had no reason to think that Gino had ever touched those data. And so just going back to where we started with the idea of, you know, this fraud turduck in, it was, I guess we had like a fraud turduck before, just a duck in a turkey. And now it's like, okay, we've just added another layer. There may be a third independent source. of made-up data just in this, you know,

specific community of scholars within this specific field of research within business academia. So, I mean, it just starts to get weird, right? Like, we knew there were already two sources of suspect data. Now there's maybe a third independent source. Like, how far does this go? Okay, so that was, when I started reporting on this,

that was the point I was interested in. I was like, wow, we're up to three, right? How many more could there possibly be? So then the next turn in the story, I guess, is that Schroeder finds another paper with her name on it, and Gino's name on it, as you say, about rituals that has another kind of suspicious-looking data set. In this case, though, it's Schroeder's own data set. There's no dispute.

With the Don't Stop Believing Paper, there was disagreement among the co-authors who actually handled those data and they don't, to this day, don't agree on that question. So it's hard to say if those data are flawed, who created the problems in it. That's an open question. But in this other paper that came out in 2018, it's, you know, Schroeder's like, these are my data. I did this analysis. So, but if you look at those data, they are quite suspicious. Basically, this was a study of whether you can use rituals. We already talked about you can use rituals maybe to sing don't stop believing better.

She wanted to know if you could use rituals essentially to lose weight. And the way the study was set up, women were recruited at a gymnasium at the University of Chicago, women who wanted to lose weight, and then about half of them were told to do some kind of ritual before every meal. They were supposed to organize their food

on their plate in some way, like cut the food up into pieces and then separate it on their plate. And then they wanted to see if merely doing this ritual helped these women who are all trying to consume fewer calories in general to actually consume fewer calories. The study found that it worked, right? The women who did the ritual ate less. But what Schroeder discovered when she was auditing, again, her own work, she found that a bunch of data points had been flipped, potentially, from one condition to the other. So women who had been marked on the log sheet for the study as being in the ritual condition,

somehow it seemed like maybe they had actually been in the control condition based on other data from the file. And then similarly, some data were flipped the other way. Okay, so you had some, this is often described fairly or euphemistically as a coding error. Okay? So you had some kind of coding error where some of the data was misattributed by condition, potentially. Now, if you flip it back, this is naturally what you would do and what Trotor did. Let's flip it back and see what happens to our finding. And actually, if you flip it back, the finding vanishes completely, of anything.

The women who were doing the ritual ate a little bit, tiny bit more than the women who didn't do the ritual. So that's already disturbing, right? There was a coding error that just happened to create the effect you were looking for. And if you, again, Schroeder has made all this public. You could download the data. If you look at the data, as I did, you see that the particular data points that flipped are not just, they don't appear to be a random set of, people from the study. They seem really specific, right? It's people who either ate a whole lot and were in the ritual condition or people who ate very little and were in the control condition.

And so just on first principles, it looks deliberate. Now, I talked to some, you know, data fraud experts who tried to sort of formally look at this. They evaluated the odds that if you were just making these coding errors at random, what are the chances you would end up with like a publishable effect? And, you know, they were coming up with basically one in a million. So you've got this deeply suspicious, you know, a quote unquote mistake in this paper. But I just want to emphasize Schroeder, who was responsible for this study, is 100% the one who identified this problem and brought it to everyone's attention. So this was, you know, the first. I guess the first twist in the reporting of the story. I noticed this, you know, basically

last spring when I had already been reporting on the other paper. There's a very interesting and very tense section of your essay where you confront Schroeder with a weighty accusation and an accusation that I think flows from a logical way of thinking about everything you've reported on. You ask her if she's engaged in purposeful data tampering itself. And even more tensely, you suggest that maybe she started this entire project, or you do it in a very clever way. You say, you know, essentially how might one, couldn't one potentially think that you've started this entire project as a way to own the narrative of fixing previous errors rather than have outside critics, like those from data collata, identify the data tampering.

before you brought it to light. Tell me about that conversation and what were conversations that you had with her where you're trying to work out, you know, Professor Schroeder, what really happened here? What were your motivations in making public an error

that some data experts I'm talking to say is a one in a million chance? Yeah, I mean, it was a, it was, as I was reporting, story this was really perplexing to me, right? It was inconceivable that Schroeder herself had been responsible for switching the data on purpose, right? She's the one who brought it to light. She identified the problem and publicized the problem. On the other hand, her best explanation or what she deems the most likely explanation for what happened is that her research assistance

must have altered the data. And in her telling, they did this. because maybe they'd made a mistake earlier and we're trying to fix the mistake. It was like well-intentioned in some way. But of course, if that's true, what they did is so completely out of bounds for science to look at the data and start moving, you know, switching conditions. So, you know, that was her explanation. It's possible that that's true. RAs make mistakes.

RAs commit fraud. On the other hand, it's also, you know, often what professors say when they're accused of fraud. they say, oh, you know, I don't know what happened there. My RAs must have messed up with the data. So I just had this really, I was faced with what seemed to me like a situation that was impossible to explain, right? It didn't make any sense that Juliana Schroeder would have altered these data and then identify the alterations herself.

On the other hand, I found the, you know, quote-unquote, most likely explanation for what it happened, very unsatisfying, especially because, you know, we don't know what these RAs have to say for themselves. I tried to find them and failed. Schroeder said she knows who they are, but, you know, didn't want to involve them in the story, so their take is totally mysterious. So I just, at a certain point, I started to wonder, is there a narrative here that makes sense? Like this goes back to what we were saying about, can you come up with a story here that explains the fact?

So is there a story here that could explain why someone might have done something like this and then called attention to it? Is that really so inconceivable as I had originally believed? And it occurred to me, as you said, that maybe there's a certain logic to it if you think that, you know, you know that this is. big audit of Geno's papers is going on. You know that people are looking closely at this stuff. You have a sense that problems are going to emerge on lots of papers. Maybe there's a kind of logic to, okay, let me just be like the most rigorous integrity-oriented researcher there is. Let me just make myself totally unimpeachable in this whole story, let me just look closely at all of my own work, just throw it all out into the air, and then let me just cop to whatever problems appear

and try to move past them. And maybe, you know, it'll be okay. I'm the one who've identified the problem. Certainly that itself is laudable, right? And it's over and done with. The papers retracted or that there's a correction or whatever, and you move on with your life. It also occurred to me, you know, what if this is something, it was not a pattern of behavior, but just like a one-off thing. So you can just kind of imagine getting it out there and behind you. So I'm not saying that this is necessarily what I believe happened.

It's just something. I began to think of this as an alternate narrative that could. potentially explain what happened. And I just felt like I had to present that to Schroeder and, you know, see what she thought. So that's the, you know, set up for this final conversation we had. And she was very, you know, calm in hearing me out. She, you know, she's, she, at many business school professors, I feel like are, are trained in and teach kind of these interperts interpersonal exchanges negotiation and stuff.

And I felt like there was some of that going on too with like she told me she heard me and understood me. She said my name a lot while we were talking. And of course, she absolutely denied ever having gone in and altered data herself. She acknowledged that it does look bad and how weird it is. But we really got to this point where we almost took a step back from the specifics and talked about what the field should do.

And I thought it was quite compelling and moving even that she said at the end of that conversation that she thinks the whole field, that there could be a lot more empathy involved in how these situations are handled. And I won't try to interpret, overinterpret what specifically she meant by that, but that resonated with something that I definitely feel about this process, which is, again, let me just say this is not, I don't know that this is what Julianne Schroeder believes.

So I'm fully in the range of my own take here. But I do think one major problem in business school psychology and in dealing with fraud, in science more broadly is that there is no room whatsoever for grace, right? The stakes are so high if you are found to have committed scientific fraud, you're done, right? Your PhD will be stripped. You'll lose your position. And that's it. And so there is this incredibly high bar for concluding that that's what's happened.

in the Gino case, she was placed on administrative leave after this. The report is now public. It's like a 1,200-page document. It is, you know, it's like the 9-11 commission about this one Harvard Business School researcher and four of her papers, right? And even that is just a total mess now because Gino has sued Harvard. She also sued, initially sued the bloggers who first published the evidence about those papers for defamation.

So who even knows what the upshot of that is going to be? But it's like, I mean, in taking a step back, it just seems like a mess, right? It's a gigantic mess. What do you do in a case where it's not someone who's potentially a prolific fraud, but one paper that might have, you know, a little bit of something fishy in it? There's just no middle ground where you could kind of address that head on, expect people to be honest and have some kind of, you know, moderate consequences. If that paper, for example, were, if it were true that that paper contained fraudulent data,

I think, and, you know, the University of Chicago agreed with that, again, it's the ultimate punishment. Like, I think the Ph.T. would be nullified. So I just, whatever happened with Julianna Schroeder aside, I just became very aware of the fact that by the heavy constraints on action in dealing with the problem of fraud, like there is just, there's no room to operate. If someone did, you know, do something bad at one point in their career, there's no way for them to come clean. There's no way for them to say, even to serve as a role. model of how you could, you know, stray as a graduate student and then become, you know, truly a hero of open science. That path is not available.

And I guess, you know, maybe that's, maybe there are people who believe that's how it should be that data tampering is the ultimate crime in science and so forth. But I just, I find it quite distressing and the outcome of this super high-stakes system is terrible, right? Most cases of suspected fraud are just swept under the rug because, again, it's such a mess to even open the door to looking at it. Ironically, you're posing an incredibly rich topic for something at the intersection of moral philosophy and psychology. Maybe it's organizational psychology, but how do you design a system that punishes bad behavior, enough to discourage the bad behavior, but not so much. much that everyone is disincentivized to report on their own bad behavior, right? You're trying to

strike this incredibly knotty middle ground where you don't want people to commit fraud, but also you want to encourage light being shown onto frauds committed in such a way that people have some incentive to say, oh, I did something wrong, and I'm sorry, and I hope this doesn't destroy my entire life because at the end of the day, you know, maybe misreporting a single cell in an Excel chart in a study on ego depletion should not, in fact, ruin the rest of your career. I mean, I don't have a strong feeling about this, but I'm very willing to hear that argument. You make a really interesting comparison in the essay with steroids in baseball. And I'm reminded that Barry Bonds, if I recall this correctly, allegedly started using

steroids when he realized that players who were clearly inferior to him were using steroids and putting up higher numbers, were hitting more home runs. And so the use of steroids justified more steroid use as a kind of social contagion, or maybe like a keeping up with the Jones's effect, if that in fact is an effect, maybe it's subject to its own fraud. But you make the point that maybe if you're a young social psychologist and you want to be published in some of the most prestigious journals in social psychology or quote-unquote business school psychology. And everyone around you is putting up these, you know, steroidal numbers with these incredible whiz-bang effects that are like,

oh my God, didn't even realize that sprinkling salt in a piece of paper turned me into the lead singer of Journey, you become motivated to maybe mess with your own numbers just a little bit in order to get the whiz-bang factor up to a point where a journal on social psychology will publish your work. So maybe just dial it on that last point a bit, because I thought it was so interesting, the degree to which the expectation of researchers that fraud exists in their field

serves as a mechanism for them to create their own minorly or majorly fraudulent work, because, again, they're playing in a league where everyone is using steroids. Yeah, I mean, I think that's right. does seem to me that fraud would be contagious in that sense. I think there's another issue that we're touching on here, which is super important, which is that below the level of fraud is, you know, this whole world of what are called in the business, questionable research practices,

QRPs is the term of art. And that's, you know, your listeners probably have heard of P hacking. right? This is this big problem in psychology, social sciences, and truly sciences broadly for a long time where there's just these ways of kind of just tiptoeing up to the edge of what is, you know, what is sub fraudulent statistical analysis. You know, look at your data. There's some data points that don't really help you and you come up with some reason why, you know, I'm, Honestly, that guy when he came into the lab, he was acting strange, maybe he was even drunk. Like, I think we should discard that data. And you can, I mean, it's obvious, right, how this could be abused. But I think it was commonplace as recently as 10 or 15 years ago to have no guardrails whatsoever on data exclusion, just to pick one example that comes up in my story. So that's common.

If you can just kind of, in this freewheeling way, exclude data, even if you kind of, you know, you don't think you're doing wrong, that itself is going to lead to this inflation of stats. So there are, you know, and that just pushes the incentives in the wrong direction already. So just to go back to the baseball analogy, you could imagine, like, short of taking steroids, there's a norm in baseball where, you know, it's okay to steal signs when you're on second base, right? I think this is, the Astros got in a lot of trouble for a sign-stealing operation some years ago that involved technology.

But I think just doing it by eyeballing is kind of like part of the game, right? So maybe that's analogous to bee hacking. I might be stretching here, but I'm just imagining that there's like ways to get kind of an advantage in science. in terms of producing the results you want, that everyone is doing and everyone accepts. And so already you have these unrealistic expectations on rigorous science to produce publishable findings. And then you can keep tiptoeing closer and closer to the line.

Like, P-hacking can get more and more egregious until it's really in a gray area where you're like, you probably know that what you're doing isn't scientific per se, right? and then take another few steps, and now you're actually just changing data. So the case is often made that maybe there's a wide swath of research practice where fraud is indistinguishable from questionable research practice. I don't really buy that myself so much. I think everyone knows intuitively that there's something different, but there's a meaningful difference between, you know, sloppy application of statistics and just making up numbers.

But I do certainly think there is a lot of blurry stuff in between. And I do think the creation, the existence of that blur is not going to help and contributes to that kind of spread of cheating, right, that we were talking about. Like I think that that makes it only that much harder to get published. So it increases the incentive to cheat. and also gives us sense that there's just kind of like, as I said, a lack of guardrails, really, truly. I want to round toward a conclusion in just a second, but you spoke to several academics for this story, a bunch of people in and around this business school psychology space. How would you describe their general attitude?

Are they in denial? Are they disillusioned by the revelations the last few years? Are they in despair over the state of their industry? I guess I used 3D words there, but which would you choose? Denial, disillusionment, or despair? Denial, disillusionment, or despair. Actually, feel free to use literally any word. You do not have to choose the three words that just randomly pop to my mind.

I just generally, how would you describe? I can only, this is a psychology experiment, I can only think of words that begin with D. I'm going to go with demoralized. Okay. Yeah, I mean, I think there was denial for a while, right? I think the Gino scandal has made it harder to be in denial in this particular field, right? But, yeah, I mean, I think Schroeder's experience itself is so demoralizing for her and for anyone who's watching it. because she has done so much to try to correct the record, right?

And even in just identifying the flaws in that first paper we talked about, like she is running afoul of very powerful people, right? Including a very senior academic, as I described in the paper, kind of tried to tell me on the slide that he thinks Schroeder is a fraud. Just to give a sense, like this is someone who, you know, Schroeder's tenured academic at Berkeley, but someone who is a much more senior person

is telling a reporter that she's a fraud. Like that's, that just shows, I think, how dangerous it is for anyone in this world to even try to, like, cast a light on this stuff. Now, we've talked about how Schroeder's case is complicated, but I do think it's incredibly dispirited. I'm just going to lean into the dees.

It's incredibly degrading to see how in this, what I described earlier is this inspiring project, the mini co-authors project, has both shed light on the scope of the problems, but also been so damaging for so many people. And yeah, it's brutal. Things are not set up to get better. as they are right now. I do want to offer at least the possibility of an optimistic interpretation of the story,

which is that 10 years ago, 20 years ago, business school psychology existed. The field of social psychology was a frothy field, best-selling books for being published. None of this was happening on the positive side. There was no many co-authors project. There was no data collada. There was no public awareness of P-hacking

such that there could be an ability to problematize the practice. Many of these iffy ways of dealing with data might have been widespread, but it existed entirely in the shadows. And the process of pulling these practices into the light, while an incredibly painful one for the people in the field, could optimistically lead to a social psychology discipline. that is much more disciplined in the next decade, in the next few years. And so we are, in a way, hopefully somewhere close to the emotional nadir of this general

experience. We are realizing how much fraud has been in the system. But that very realization is going to lead to different outcomes, the same way that, you know, to jump back to the stairway experience of baseball, no matter how you feel about the evolution of baseball since the early 2000s, and I have a lot of fun. complicated feelings. Nonetheless, you can't argue with the fact that there is a steroid testing regime, and people are banned from the sport for taking substances that were probably absolutely widespread 20 years ago. And theoretically, something similar could happen to this field of psychology.

To the extent, do you think that optimistic interpretation is completely polyanish-naive versus potentially realistic? No, I mean, things are changing. quickly. So I don't think that's totally unreasonable. I think about it, you know, in a way there's two approaches. What are we going to do about

this? We can continue to try to diagnose the problem where we can intervene. And obviously, intervening, fixing is better than diagnosing. Diagnosing doesn't get you anywhere on its own. And some people are trying to fix it, right? Yuri Simonson from Data

Kalata is working to to come up with a system that you can kind of track the provenance of data, basically. So not just share your raw data with people, but actually create a technological fix where people can see who worked with the data. Like just the full, you know, think of it sort of like cryptocurrency. You can follow the life history of the data. And that kind of openness, if it were adopted broadly, would make a big difference because there would be, you know, there's no way to hide what happened if there's a full data trail on the data itself, right?

But in order to get those fixes adopted, those fixes are expensive in terms of time and money. And the costs of such interventions are disproportionately borne by young research. and researchers at less well-funded universities, right? Because if you're at Harvard Business School or Wharton and you're a senior person, someone else can handle all this data tracking stuff for you or whatever else, what other interventions are put in place, if you're just starting out at a different school,

that's on you. That administrative burden is fully on your head. So you have to be really careful about what fixes you put in place because they're these disproportionate harms from the fixes themselves. So there is both reasonable opposition to these interventions and there's unreasonable opposition. People don't want these fixes because they don't want to, you know, as one person told me, they don't want to kill the golden goose. If there is no fraud or p-hacking in business school psychology, you know, that's not great for the people

who are, you know, making successful careers doing this work. So you have this opposition to it, reasonable and unreasonable. And I think to overcome that opposition, you need the diagnosis. Like you need to understand baseball. At a certain point in baseball, I remember this, you know, the Mark McGuire, Sammy Sosa home run race, where you're just like watching the game and you're just like, this is absurd. Like these people don't look normal. They're not, their performance is abnormal.

Like there was just widespread agreement that there was just rampant cheating in baseball. I don't think you have that in science. And so these moments, like this moment of the genome papers being identified in 2023 and now, again, with two more suspicious papers being identified, each one, I think, it doesn't tell us the big picture. We still don't know the base rate of fraud in psychology or in science broadly, but it just sort of dials up the pressure. a little more to do something about it. And I suspect, I have no idea, but I suspect that we still need more of that. We need more diagnosis before everyone acknowledges. Not that there is a problem, everyone does acknowledge that, but like the true scope of the problem. I think it's still possible to go, okay, well, you know, this is one particular group of researchers. Maybe the fraud rate

is high there or something. But there's so much business academia that's so wonderful. Granted, that's true. I'm just saying I think we do need to know how much fraud exists. I wish we could have some kind of broad random audit. That would be a lot of work, but I think it could create the conditions that would be necessary for truly fixing things. I want to suggest one more conclusion of your really, really interesting reporting.

it points us back to an argument that Adam Astriani makes in the first interview that we're going to have on this show where he talks about the fact that, you know, psychology hasn't produced workable technology the same way that, say, physics or chemistry have. And in part, psychology is a very different field because you're fundamentally working with people and people are messier than atoms and molecules. And whereas in a field like chemistry or physics, there's a paper and it reaches a conclusion. And that conclusion becomes a technology. And the technology lives in the real world. And it either works or it doesn't. Either quantum mechanics makes the transistor and the semiconductor possible or it doesn't. And the transistor doesn't work. And software doesn't

exist because quantum mechanics, it turns out, was just wrong. Whereas in psychology, those falsifying feedback loops in the so-called real world outside of the lab don't necessarily exist in the same way that they do and other so-called hard sciences. And so maybe one thing that we're seeing here is that it's easier to get away with little flubs and little frauds in psychology, because it's so much harder to falsify the fundamental question of, does a ritual like pouring salt on a piece of paper make you better at presenting a PowerPoint?

Like, that's a very, very hard thing to say for the entire human population given how many variables are present in each and every PowerPoint. And this is simply a necessary correcting mechanism for a field where the so-called world of technology, the physical world, wasn't going to serve as that kind of correcting mechanism,

the same way that it can serve as a correcting mechanism in chemistry, where you reach a conclusion, you try to build something based on that conclusion, and it just falls apart in the world of atoms because it turns out the chemistry was wrong. And they're making sense here, like basically the physical world falsifies the hard sciences. We needed some other mechanism of falsification in a so-called soft science like psychology, and a movement like this, therefore maybe just had to be inevitable to serve as that mechanism. I mean, I agree with that, basically.

I think Adam is brilliant, by the way, and hilarious, and everyone should reach his substack. I do think these falsification loops may not be as robust in other fields as we would like them to be. Like, sure, of course, right? Like, power posing, if it doesn't work, who's going to notice? Whereas, you know, if a plane doesn't fly, you're going to have a cabin full of angry passengers. Yes, obviously true. However, I think there's a lot of middle ground there where, take, for example, fraud scandals and Alzheimer's research, right? Like a leading paradigm for Alzheimer's is false, right?

It's based on some key papers that are fully unreliable has been revealed by reporting by Charles Pillar and science, among other people. Right? So I don't think that, for me at least, and I'm guessing for you too, our experience isn't like, well, of course, this became obvious because a lot of people with Alzheimer's

took a drug that didn't work and thought, you know, it was like the plane didn't fly. No. In fact, there was just huge quantities of research dollars. I mean, I could be wrong on the order of magnitude, but my understanding is hundreds of millions of dollars. At any rate, a lot. And years and years of research from pharmaceutical companies,

those are people who are interested in the falsification feedback loops, right, went into this work that was essentially for nothing. So I just don't think there is a clear demarcation between psychology research where it's really hard to tell when something's bogus and say, you know, I'm going to biomedical research where, you know, you can look at, you know, patients in the hospital and their outcomes. Like it, you, it would be nice if, you know, it was easy to tell when biomedical research was, was bogus, but it's just in practice, I don't think that's true. So I think, I think the problem exists outside of psychology. And so you need

the same kind of diagnosis and intervention across fields, basically. That reminds me I really want to do an episode on the amyloid hypothesis in Alzheimer's because you glossed it, but it's so fascinating. And to your point, billions of dollars were probably spent around the world. Billions of dollars are probably spent. I mean, when you think about the money that is sloshing around for pharmaceutical companies to cure a disease as widespread as Alzheimer's, my uninformed guess is we're talking about billions of dollars chasing a theory about neuroscience that turns out to be totally bunk.

Science, it turns out, is very hard, and I think you're one of the best chroniclers that we have at exactly how hard it is. So Dan Engber, thank you for your work, and thanks for walking us through this particular story. We appreciate it. It's my pleasure. Thank you so much for having me on the show. Thank you for listening. Today's episode was produced by Devin Beraldi. Our schedule for plain English for the next few weeks will be one episode a week on Fridays. We'll see you next week.