Decoding the Gurus - Autism, Microbiomes, & Mice Burying Marbles with Kevin Mitchell

Episode Date: November 18, 2025

This week, we are joined by Kevin Mitchell, Associate Professor of Genetics and Neuroscience at Trinity College Dublin, who has committed the unforgivable sin of pointing out that an entire academic and media hype cycle might be built on… well, very little actually. His new co-authored paper in Neuron politely dismantles the highly promoted link between the gut microbiome and autism, which turns out to rest on flawed studies, contradictory findings, creative statistics, and a touching faith in mice burying marbles.Kevin walks us through the joys of observational studies that don’t replicate, mouse experiments that don't make sense, and clinical trials where there is no blinding and no control wing, and shockingly, everyone reports feeling better. Meanwhile, journalists and wellness gurus eagerly report each new “breakthrough”, unburdened by any concerns about the strength of evidence or methodological robustness.In the end, the microbiome–autism connection looks less like a sturdy scientific stool and more like three damp twigs taped together by optimism and marketing departments.We finish, naturally, by dragging Matt back out of his panpsychism phase and asking whether consciousness is really fundamental to the universe or just something that happens in podcasters who haven’t slept enough.LinksMitchell, K. J., Dahly, D. L., & Bishop, D. V. (2025). Conceptual and methodological flaws undermine claims of a link between the gut microbiome and autism. Neuron.Kevin Mitchell's Website

Hello and welcome to the Hello and welcome to recording the gurus interviews interviews. edition. As for usual, Matt, the psychologist of sorts, is there. And Chris, the anthropologist of sorts, is here, me. But today we have returning guest, Mann also from Ireland, Kevin Mitchell, who is associated with Trinity College in Dublin and an associate professor of genetics and, no, wait, genetics and microbiology Institute of Neuroscience, that's your affiliation, right? But maybe

your title is slightly different. It's all. It's much of a muchness. Yeah, so we get genetics and neuroscience and microbiology. Those are the important things that there. And much as we look forward to later

in the episode, discussing panpsychism with your Matt's recent adoption of that philosophy, The reason that we have you on today is that you have a forthcoming paper along with Darren Daly and Dorothy Bishop, the title of which is conceptual and methodological flaws undermine claims of a link between the gut microbiome and autism. So that's what we're going to talk about. But in general, Kevin, thanks for coming on. And it's good to see you.

Yeah, yeah, it's my pleasure. And thanks so much for having me on. It's great, you know, to see some interest in this paper, which is sort of a null finding, in a sense. And, you know, those kind of papers don't often get the same attention that the positive ones do. So, yeah, I'm really happy to be on to be able to chat about it. So what's the background to this, Kevin? So we've got this connection between, purported connection, between the gut microbiome and autism. And it's got a popular discourse aspect to it.

and it's also an active area investigation in the academic literature. So how would you describe it? Yeah, well, I think you hit both nails on the head there, right? So it is within science. It's a very active area that tracks a lot of funding. There's loads of papers, there's loads of reviews and so on. And then it's also the kind of topic that makes its way into the New York Times and the Guardian. I mean, there was even a Netflix special on it recently.

and tons and tons of online discourse about it in various groups. And so, you know, I think the rationale here for studying this, right, for thinking that maybe something about the gut microbiome has something to do with autism or contributes to risk comes from a few areas. One is just the idea that there's an epidemic of autism, right? So what we've seen over recent years is rates of autism diagnoses have gone up and up. And so people point to that as evidence. that there must be something in the environment that's causing that,

something that's changing, that's causing this increase in the rates of autism. And, you know, you'll hear RFK Jr., for example, point to that all the time, right? There's evidence that there must be environmental factors at play. It turns out that, you know, while there's an increase in the rate of autism diagnoses, that doesn't mean there's any difference in the actual underlying biology or symptoms that people have. And in fact, there's really, really good evidence that it's just that the diagnostic criteria have expanded and loosened so that it's quite possible to get an autism diagnosis with much less severe symptoms than it would have been 10 years ago or much, much less than

20 years ago, and so on. So there's a ton of research actually showing that the apparent rise in diagnosis doesn't reflect any change in underlying biology. It's just convention. And actually, you know, psychiatric diagnoses are a matter of convention at some point, right? Yes. So that's the first, but nevertheless, that's the impression that people get. There must be some environmental cause, so let's look for one, right? And then the second sort of line of evidence is that many children, in particular, with autism, also have some gastrointestinal symptoms, right? So there's some GI disturbances that are more common in people with autism.

And so there's a sort of a link there. It's like something's going on in the gut, something's going on in, you know, behaviorally, maybe those two things are linked together. And of course, there's lots of ways to think about that, right? And one of them is just to say, well, okay, maybe there's a link that goes from a disturbance in the gut causes behavioral symptoms, which is sort of the narrative that's being proposed. But it could also be maybe something in the behavioral symptoms causes a gut disturbance, right? So maybe having a really restricted diet and being a very picky eater affects your diet and affects your causes, kind of. constipation or diary or other concerns, right? And then the other possibility is just that they're

completely unrelated, right? So yes, it's true that children with autism have higher rates of gastrointestinal symptoms, but so do children with every neurodevelopmental disorder. I mean, so do children with Down syndrome, right? Nobody thinks that the gut disturbances in Down syndrome are causing everything else, right? They're not causing intellectual disability or anything else, right? And in fact, you can also say, well, look, you know, many, children with autism also have seizures, much higher than the general population. Again, nobody thinks the seizures are causing the autism. They're just two manifestations of a syndrome, basically. And so, you know, I don't think either of those lines of evidence is actually really strong.

You know, it's fine to say, okay, we can have this hypothesis because of these ideas. I don't think there's a very strong, just sort of preliminary grounding for the hypothesis. But But nevertheless, those are the two main lines of evidence. And then the third is sort of a negative, which is to say we've really, really good evidence that autism is very strongly genetic. If you look across the population at variation in who gets it and who doesn't, about 80 to 90 percent of that is explained by genetics. And so what that means is that there should be genetic differences between people that are

causal for a risk of autism. and we should be able to find them if we look for mutations. Now, finding them turns out to be really difficult, right? You need very large numbers of people. Because we all carry loads and loads of genetic variants, it's hard to pinpoint which ones are which. So we've made tremendous progress in identifying lots and lots of high-risk genes where a mutation can confer a high risk of autism or other neurodevelopmental conditions,

but there's still loads and loads of cases unexplained. So there's two ways of looking at that. One is to say, okay, we're just getting started. We need to figure that out more. We need better ways of finding these things. And the other is to say we've hit the end of the road there. There's something unexplained. Some other factor must be at play.

So those are the sort of the three lines of evidence that prompted, I think, people to start looking at this hypothesis. And Kevin, so the paper just for, you know, obviously the people listening to Munt have read it. And I would classify it as a critical review of the literature, right? So people may or may not be familiar with meta analyses, but in general, meta analyses are collecting the statistical results of a selection of studies and trying to combine them together with ways to qualify how rigorous they are and then looking at whether they can find overall effects or not. And your paper is actually above that kind of level because it's looking critically

at the various different meta-analyses that have been conducted. So kind of taking a top-down look across the whole literature and the quality of the literature they're in, right? Yeah. And you cover three lines of evidence, which I think we should get into.

But I just want to mention given that that is such like a kind of bird's eye view of it. So you, as we mentioned, you know, associate professor of genetics and neuroscience, your co-offer Dorfie Bishop, I recognize from a well-known figure within the replication crisis, our open science, methodological reform movement. Your other co-author, Darren Daly, I'm not familiar with, but I'm wondering how the three of you ended up deciding that,

obviously the things that you just mentioned are all like the contextual factors, but how did you guys decide that you're going to take it on and look at this literature? Yeah, so maybe I should just briefly say, say what the literature sort of entails, because this was what we were looking at, right, is that these reports started coming out. So people had these kinds of hypotheses to test, and then they started using these technologies that allow you to basically sample the gut microbiome from, basically from stool samples, right? So you can take someone's stool, you can sequence the DNA that's in it, and it will contain loads of the microorganisms that are in your gut. And then

you can use this sort of bioinformatics techniques to figure out which sequence belongs to which species and you can determine the relative abundance of different species and phyla and so on from one person to the next, right? So this tool became available that people could do these kind of analyses. And then it was, you know, then you could just ask, well, do people with autism have a different profile in their microbiome from people without? And when we started to see these studies, like the first one is about, you know, 15 years ago, I think, in 2010, I was sort of interested because I work on the genetics of autism and neurodevelopmental disorders more generally, so always interested at possible factors that could be contributing to it.

But when I looked at those studies, I found, you know, they were just really small studies. They didn't seem to be well done, statistically speaking. There were loads of what we now know are research practices that basically will generate false positive findings, right? And so we know that from our history in genetics, our history in neuroimaging, history in psychology, you guys have talked about it before, right? All these things that have led to the replication crisis where we've identified methodological practices that just are not going to be robust. And those were the kinds of things that I was seeing in these papers. So it made me highly skeptical of the findings. And Dorothy, you know, also has been well known, as you said,

in the field of reproducibility and the replication crisis and so on. She's written lots and lot about what those factors are, what these bad research practices. But she's also an expert in autism and neurodevelopmental disorders, speech and language disorders and so on. So she was quite interested. And then Darren is a colleague here in Ireland who also is very interested in the replication reproducibility sorts of things, an advocate of open science and good, robust science practices, and also a particular expert in the design of clinical trials. And And what we started to see in the field was that people were doing these studies, claiming to see differences in the microbiome between people with autism and without, and then they

started to move to other lines of evidence, first of all, experimental work in mice, but also these small-scale trials in humans, where they were testing probiotics or even doing fecal microbiome transplants into children with autism and then claiming that there was some amelioration of symptoms. And so, you know, surveying the whole field, I think the three of us were, first of all, we were skeptical based on the methods and what we knew about robust methodology. But we were also, you know, a little dismayed at the hype, frankly, because these claims are not, you know, when they're published, they tend to be published in a quite sort of stark kind of a way. Like the headline claims are very strong. And then they get picked up by the newspapers. And so it becomes what we, took to be, you know, a lot of effectively misinformation. And, you know, so that's what prompted us to actually dive in here and see, well, I mean, you know, maybe our impression was wrong.

Maybe actually something robust was coming out of the field. But it took it, it takes a deep dive to really assess it and evaluate it. And that's what we did. Yeah, Kevin. I think the context he attaches on like a few very common sort of public misapprehensions or misconceptions about how things work. And one of them, of course, is the reification of diagnoses, assuming that they are a hard type of category when any psychiatrist or psychologist knows that they are diagnostic conventions which change as society changes. What's true of autism is also true of ADHD and any number of other conditions. And the other aspect of course is, I mean, I first heard about the gut microbiome

in the public discourse when we covered Gwyneth Paltrow. It came up again with... Michaela Peterson. Michael Peterson, of course. And I've forgotten whether Andrew Huberman is into it. Has you talked about it, Chris?

Oh, yeah, of course. Yes. Of course he has. Yeah, so there's obviously this huge public appetite for for ideas like this. And I think, I mean, Personally, it's a bit psychoanalytic of me, but I think we all have obsessions about what goes into us

and what comes out of us, right? There's those certain kind of obsessions that people have. But what's kind of most concerning potentially, and we'll get to your results in a second, what you found, is the degree to which the academic and scientific literature, which, you know, as we all know, is voluminous and of widely varying quality, and bits and pieces of it that's sort of,

especially the ones that maybe are a little bit free with their abstract or their titles or their conclusions, how that can get picked up and functionally serve as misinformation, even if that's not the intention of the people writing the papers. Yeah, no, I think that's exactly right. And it's weird, like, it's funny to think what makes this topic so appealing. And I think there's a few things that go into. One is that, like, even for scientists, it's sort of,

sexy, right? It's so novel to think, oh, like, what if I told you, you know, everything you thought about your personality and the way your mind works is actually being affected by the gut, you know, the bacteria in your gut. It's just a cool kind of sounding hypothesis, right? I think people find it sort of attractive for that reason. It's just a bit novel. Scientists are not immune to trendiness, right? And then the other aspect for something like autism is that what this topic promises is a simple explanation for a really, really complex condition. And so that's always appealing. It's simple. It's straightforward. If you're a journalist, you can write this paper. You don't have to do any background or, you know, nobody needs any background to understand it.

You can just say autism is caused by the excess of these kinds of bacteria or the imbalance between these different phyler genera bacteria in your gut, and you don't need anything else, right? Whereas to explain the genetics of autism, what we know about that, it's so complex, right? There's so many risk variants that we know of in different genes that do different things. They combine together in highly complex ways. There's effects of genetic background. There are risk factors for other conditions as well. You know, even just what I just said, right, your eyes start to glaze over.

It's like, shit, that's like, I can't tell that story. That's a shit. That's a terrible story to try to tell, right? whereas the autism is caused by this one thing, and it could be the microbiome, it could be Tylenol, it could be fluoride in the water, it could be any number of things that people have run through COVID vaccine. COVID, of course, right? Absolutely. I forgot the COVID vaccine. I forgot vaccines. I know. I forgot vaccines in general. And of course, you know, there's a link between the the original Wakefield's fraudulent paper that claims vaccines were at play with

leaky got this sort of really vague nebulous sort of idea but that was very much at the at the core of the his initial claims and so there is a link out there I'm not claiming that the people that the scientists in the field working on it are making this claim but out in the literature out in the broader discourse there's very much an idea that there's some kind of link potentially between vaccines and the microbiome and autism and so on. So, yeah, I mean, it, like you said, it becomes functional misinformation, even if the scientists writing the things don't intend that. But I'm not going to let them off the hook entirely, Matt,

because actually I think people do have a responsibility to be somewhat cautious and circumspect in the way that they talk about their findings, the title, in the abstract, especially in press releases, where, you know, you'll see these individual studies, usually pretty small-scale studies that people find some positive association or something. And then they get huge press. Well, that's not a coincidence, right? That means that the people involved have made a press release, and they put it out there. And they're promoting things and spinning it in a certain way that, you know, I find, let's say, could be more circumspect and more cautious.

That's a polite way of putting it. And I will say that reading this paper reminded me so much of a whole bunch of unrelated literatures that I've looked into where you see similar kinds of issues around small sample sizes and kind of overhype claims. I mean, mindfulness, meditation, for example, has similar sorts of issues around it. So basically, anything that gets a lot of hype often comes with these kind of. concerns. But to speak about some of the specifics more, so you, you cover three lines of evidence, human observational studies, pre-clinical experiments in mice and human clinical trials.

And just for my personal preference to start, you can veto this caveat if it doesn't flow coherently. But the mice studies, I came across these because when Matt and I covered a character, Dr. K, you may or may not. with, but he referenced the fecal transplant studies, right? Yes, because he wants to link it actually to Ayurvedic notions about diet and these kind of things. So I went down a rabbit hole into the studies that you're discussing in the paper. And I remember attempt, like one noting the very small sample size, but I lack, you know, the grounding in the relevant animal study methodologies to know how bad the

low sample size is. But the issue that I kept coming up against, I was like, is this just the norm here? Is there was no pre-registered studies? Absolutely zero. Like open, it's like open science didn't seem to have occurred in this field. So anyway, I'm curious about like all three lines of evidence. Maybe we start in the non-human lineage first. So this is really interesting, right? So people have tried in trying to understand medical conditions, um, People often use animal models. So, for example, if you find a mutation in some gene that causes epilepsy in humans,

you can make that same mutation in an animal. And then you can see, do they develop seizures? Then you can have a model where you can do some experiments. You can test some drugs. You can work out a mechanism and so on. So that's the rationale for making animal models of conditions like that. Now, for seizures, it's pretty obvious if an animal

having a seizure, right? Because it looks like they are and you can record from the brain and you can see that that neural activity is happening in that way, right? The question is for psychiatric stuff, like what do you look for when you make an animal model? And this becomes really tricky. You know, there's all sorts of things that are taken, all sorts of behaviors that are taken to be a mouse analog of depression or psychosis or mania or things to do with autism. Now, for some of those conditions, like for psychosis, for example, there are some lines of evidence that back that up, and one of them would be, like, you can give people some drugs that make them psychotic acutely, and you can give those drugs to an animal, and you can see what

happens. And then you can say, here's my model of psychosis in an animal. It doesn't matter if it looks like psychosis, you know, superficially. It's that there's some underlying neurobiology that you can presume is the same, right? And then you can say, well, actually, I also know I can give these other drugs, these antipsychotics, to humans and stop psychosis. And if you can treat whatever the emergent behaviors in an animal are with those same drugs, then you've got some good confidence that you're in a realm that has some relevance to psychosis in humans, right? For autism, there's nothing like that because we don't have any drugs that treat the symptoms of autism itself, the core symptoms, which is, you know,

delayed language development, differences in social cognition, differences in social interaction, repetitive behaviors, and, you know, narrow interests. Those are the things that, those are the core symptoms of autism. And we don't have any drugs that treat those. And we don't have any clear analogs in mice, or at least what people claim is an analog in a mouse is just based on what's called face validity, which is a sort of a superficial similarity. Basically, you can use some of the same words to describe what's going on in your mouse with what's going on in the in humans, right? So one of those would be, you know, the most obvious one would be social interactions. You can see how much interest a mouse has in other mice. And that's a, you know,

an area of science in mouse behavior. It's a perfectly valid area of science, but it's a leap to say that indifference to other animals for a mouse is a proxy for whatever is going on. I mean, you know, it's much more complex sort of social cognition in humans, right? Yeah. And the burying marble is, okay, so there's a couple other ones that are taken, right? One of them is you can measure ultrasonic vocalizations, which is basically how much my squeak. And some people will say, well, less squeaking is like a language deficit. Again, you can see what a leap it is to make that claim.

And in fact, some people say more squeaking is also, you know, a proxy for autism. So it's like either way, there's some relation. But the marble-bearing one is the one that really, yeah, I think is kind of baffling. So there's this if you put a bunch of, it sounds funny even describing this, right? If you put a bunch of marbles in the bedding of a cage that has a mouse in it, they'll tend to kind of dig at them. And as they dig the bedding, the marbles will tend to submerge. So it's not that they're trying to bury the marbles. It's a consequence of how much they like to dig around in the bedding.

And so people have said, well, that digging around in the bedding is like this kind of compulsive, repetitive behavior. So maybe that's like repetitive behaviors in autism. Right. And again, like it's such a leap to make that. But people have sort of taken that to be the case, right? Do they measure how many marbles are like underneath the bedding? This is the way the test. So somebody looks in after.

20 minutes or half an hour and counts the marbles. Yeah, that's the nature of that. At least that should be very easy to pre-register. It would be. So, right, so this is the other issue, right? It's like if you're doing these tests, well, let me say the basic idea of the test is that there was all these human observational studies that purported to show a difference or many kinds of differences in the microbiome of people with autism versus without.

Now, just from observational studies, you can't make any causal claims or inference. right? It's just an association. It's just a correlation. So that prompted people to do some causal interventions in animals where they say, okay, what we're going to do is manipulate the microbiome in some way and then see what, see if we get any effects that we think are related to autism. And there were sort of two designs. One is to generate animals that you think reflect the ideology of autism in some other way and then see if manipulating the microbiome affects their symptoms. And then the other is to take some animals that you think are kind of just a baseline and actually put in the fecal microbiome of people with autism and see if that

makes them exhibit these autistic-like symptoms. Turns out all of those designs have various problems with them. You refer to, first of all, the lack of pre-registration. There's many, many sort of researcher degrees of freedom in the kinds of behaviors you can look at. Usually they weren't corrected for all the tests that people do, you know, the more things you look for, the more likely it is something by chance is just going to pop up. The constructs that people used in the first kind of design where they said, we have a mouse that's sort of autistic, let's manipulate the microbiome, we're also based on epidemiological factors that have no validity. So things like maternal immune activation, there's a whole literature around that. But like a lot of epidemiological findings,

the supposed association that you get from really small studies just goes away when you look at really, really huge studies in Denmark or Sweden or someplace that has national registers. It's like when you do the epidemiology properly, these risk factors kind of evaporate. And we saw the same thing with Tylenol, exactly the same dynamic, some small studies claimed in association, and then bigger study properly done, it evaporated. One of the other mouse models was maternal high-fat diet, and that, that is supposedly this link between maternal obesity and autism, which, again, comes out with the really small studies, and then when you do it properly, and especially when you control

for potentially confounding factors in your design, within, you know, if you do a within family study, you don't see any effect, right? So when you do the epidemiology studies properly, those kinds of effects tend to disappear. So anyway, this is a long background, but this is the problem like it gets really technical to actually get into the details. You actually have to dig in and see what's going on. I mean, this would be familiar to DTG listeners because we've covered before a kind of pattern where when, especially in the context of meta-analysis,

where it can kind of look like there's a lot of evidence there. Like there's a lot of smoke. So it sort of feels like there's got to be fire. But when you look at the details, you see a pattern, which is the kinds of studies that are supporting the proposal tend to be very weak, have lots of flaws, you know, small in, all kinds of stuff like that. And then when you look at the studies that are stronger, bigger in, all of that stuff, they tend to not find the results.

But if you kind of look at all of the studies naively, then it can seem like there is, you know, there is evidence for an effect. Like hevermectin. With like Ivermectin. Yeah. And Kevin, maybe you were about to cover it, but one of the things that struck me in this section was the notion that even if you accept, like, I think to me the clearest analog is, okay, like if there's a relatively okay way to measure my sociality, which

is questionable, but let's grant it, right? Then, okay, so this is something that is usually disrupted by severe autism. And then you want to test about these fecal transplants as a possible way of curing, or at least reducing the symptoms. But that relied on, as you described in the paper, the notion that like a human, because like the human gut biome is not the same as a rodent, god biome. So you've got that going across and then that that will cure because that would mean that like essentially the autism is such a core component of mammalian. Physiology. It doesn't even matter that mice don't eat the same things as humans generally. There's so many sorts of underlying conceptual problems. This is one of the things that's really frustrating. This is really loose movement or sliding from human observational studies where

basically you're just finding something is different in the microbiome. It's not consistent at all as we hope we'll talk about. And then it's like, okay, well, let me test that in the mouse. But like conceptually, what are you doing? What do you think is going to have Ben, what kind of an effect size do you think you're dealing with that manipulating the microbiome could cause this effect in mice? And so it's a bit baffling, especially as you say, because the gut microbiome in mice is really different from that in humans. Like it's something like 85% of the species that are there and humans aren't there in mice and vice versa. And when you actually implant things, some of the species will engraft and stay there, but many of them won't.

Like you can put human microbiome in a mouse. It doesn't. mean it will have a human microbiome. Most of those don't live there, right? So there's a sort of fundamental flaws there and then, or questions at least, let's say. And then the other thing that's funny is like these experiments, many of them start with germ-free mice. So these are mice that have been raised in completely sterile conditions for generations. They have no bacteria in their gut whatsoever. And then the ideas you populate them with bacteria from either children with autism or without. And then you ask what are their symptoms, right? So, you know, that's quite a drastic thing to do to an animal that has never had any bacteria in its gut in its life and to populate it like

that and then say, you know, what happens behaviorally? It's not really a surprise if something happens behaviorally. I mean, you might get a shock if someone introduced a whole microbiome to your system that you'd never had before, right? Well, actually, given, I have a small technical question here. Yeah. So if that was their intervention, was their control to just do nothing, or was it to inject the mice with healthy non-autistic human? Right. So in one of these studies, it was the microbiome from healthy, sorry, neurotypical donors versus from other ones. And, you know, but this particular study, which is cited, I don't know, like over a thousand times, right? I mean, the influence of this one study is amazing. Pretty quickly. And so it came out in 2019, almost immediately, people online started going, wait a minute. what's this now? Because the data just didn't seem kosher. There was like something off in the statistics. And it turned out that the experimenters had made a simple, but super important statistical error. So they used samples from five patients with autism and three neurotypical patients. And then

for each of those, they transplanted the microbiome into a bunch of different mice. So the number of mice was a lot higher than that. And they used that number as the baseline for their statistics when looking for significant differences. What they should have been using is the five and the three, right? Not the number of mice because those were, they treated them as independent experiments when in fact they're replicates, right? This is a hierarchical design, Chris, by the way. I know that, but okay.

So people pointed this out, like really, really immediately. And the authors, you know, to their credit, made their daily available. Other people reanalyzed it and the effects go away. I mean, let me say, I should say what the claim is. The claim is that the mice that are given the microbiome from the patients or people with autism showed some of these symptoms, right? So marble burying and some social interaction thing and other things. Actually, it's interesting, just as an aside, to illustrate the flexibility in approach here.

When they did the basic social interaction test, they didn't find any difference. Then they did another one, right? So then they said, okay, let's try this other social interaction test. And then they claim, well, maybe this one's better. that was the one that showed a positive difference, right? So you can see there's a kind of what people call P hacking, which is basically torturing your shifting the goalposts, shifting endpoints, torturing your data, big red flags.

Right. When you don't pre-register, even out loud to yourself, what you're going to do, there's a temptation to just tweak it a little bit until you see something, right? I think in that context, it's like for people who don't run experiments, just to think that imagine the kind of. actual world where the experimenter gets the result in the first measure, would they then go, okay, we need to run another measure because maybe that wasn't the good one. Right. Like, no, so the only time that it becomes a good measure is going to give you the result. And this is also,

yeah, yeah, the case where field replications and stuff. They're well conducted when they replicate the result, but when they don't, there's too many things change. Exactly right. This is how research or degrees of freedom work is that you keep looking until you find the thing that you want to get. And it's why these experiments that set out to prove a hypothesis, a hypothesis, are much more dangerous than ones that set out, you know, in a disinterested fashion to test a hypothesis. And here, there's just strong motivation that this is a thing, right? And so that's kind of what has happened. And anyway, to finish this particular paper, when the data are reanalyzed properly, basically all of the effects go away. I think there's a very small remaining effect on marble burying. which, you know, we already talked about the question will... And the fact size is very small. And it's tiny, right?

In that case. So basically all the claims from the mouse literature that we've looked at just don't stand up. I mean, I think there's nothing solid there that you can say definitely mouse studies have shown manipulating the microbiome causally affects autistic-like symptoms, which is how it's described. I mean, that's the amazing thing, is that in the literature, people cite these studies based on claims in the abstract or the titles. And then other people see, I've cited the study positively in my introduction or something,

and then Matt, you're reading my paper, and you go, well, Kevin knows where he's talking about it. This must be real. And then you cite the citation. And then it just becomes lore, right? It becomes a fact. Yeah. And so few people go back to the original studies

with their own evaluative powers and say, that doesn't make much sense at all, right? Yeah. I actually was working on a paper today where I've been the victim of this in that we wrote a paper a long time ago had a catchy finding I knew it was a bit weak it's actually one of

our most highest cited papers and we've tried to correct we've discovered it's it wasn't totally wrong but we've got better refined estimates and we'll publish that and they just keep getting ignored because the the sort of the original one just has too much impetus behind it

but yeah so yeah it's a bit it's a bad thing a lot of this people reminded me and Matt you'll probably note the parallels as well, but Dorsa, Amir and Chas Farristone, we had them on talking about the visual allusions in the Mueller-Layer literature.

And there you have a massive literature, and they did exactly what you're talking about. They just went back through it all and found that it's not the Mueller-Layer illusion itself, but some of the claims made around cultural variation on it. Yeah, but there are massive literature

there. So, like, actually going back and looking critically at it, you can often find things that you don't want to find like in the Stanford Prison Experiment. Oh, yeah. Exactly, right? And that's, yeah, so those are the kinds of things that become lore. You know, Stanford Prison Experiment is a great one. And there's a load of

well-known psychological experiments. And, you know, I always think these days, I guess I've become more skeptical or even cynical, is that if an experiment has a name, if it's like Dutch hunger, winter, something, don't trust it. Right? If it's a one-off kind of thing,

it's just far more likely to be wrong. And what we should do is look at bodies of evidence that consistently builds towards a consensus, right? The salient test, that's still okay, right? I mean, that's a body of literature. That's not a one-off. Yeah, that is a body of literature. Yeah, exactly.

Okay, Kevin, so we've covered the mice. Yeah. But there are other lines of evidence that you reviewed with this. Yeah. Yeah. So the lines of evidence that prompted the mouse work are basically human observational studies. And this is basically epidemiology.

So what you do is you have people with a condition, people without, and you look for some factors, some exposures that are more common in the people with the condition than without. Those might be causal, right? So that's how we found out that smoking is associated with lung cancer, for example, because in people with lung cancer, the rates of smoking are much higher than in people without. And then you can kind of invert the logic and say, what is the extra, what is the increased risk of having lung cancer if you're a smoker? But the original data are the other way around. So the design here would be to say, okay, well, maybe something about the microbiome is different in people with autism versus people without.

Let's look and see what that could be. Now, the problem is when you, when you sequence the microbiome, this is massively high dimensional data, right? You've got tons and tons and tons of variables that you can add. analyze in lots of different ways. So if you look at bacteria, you know, they're organized into, there's different bacterial species, but they're related to each other in families and in a genus and in a phylum. Okay, so there's like a dozen major phyla of bacteria. You can analyze those. You can group them in that way, or you can go down a level to the genus. That's nest at my mind. Yeah, that's nasty. Exactly. And you can down another level to species and

so on. You can do it any which way, right? And if you don't, again, if you don't pre-register it or say what you're going to do, then you've got this tremendous kind of flexibility. You can even come up with measures like, let me measure the ratio between these two types of species, right? People do that. Or let me measure the overall diversity. Maybe it's not one particular species. Maybe it's the diversity of species that's different. So that's what the literature basically involves, taking people and then analyzing the microbiome looking for some differences. Now, the first studies that were done, And we sort of surveyed kind of maybe a dozen seminal studies that are given as the foundational

evidence for this claim that something is different. You know, they had sample sizes of 10, 20, 30 people, that kind of thing. Now, many of them reported some significant differences in some aspect of the microbiome between their samples. But what quickly became apparent was there was no consistency, right? I mean, you might publish your study, Matt, and then I published mine, and both of them could say, look, there's an association with the microbiome. And if you don't dig in a little bit, you don't realize that we've actually contradicted each other. I mean, each of us has published a failed replication of the other person's results effectively. And so if you look across these studies, what you see, and we have a sort of a figure in the paper that shows, I love a study that says bacterioids are higher, and you'll have one that says it's lower, and Chris have one that says there's no change. And then... It's different, though, but each of us has found a difference, right? And so what has happened in this literature is that you get this kind of apparent replication, which is actually contradictory evidence.

And then people have shifted a little bit from claims about individual species, which just haven't held up, to claims of dysbiosis, which just means there's a change in the pattern, and something about the new pattern, is pathological. That's the implication of it. But your dysbiosis could be different from mine, right? It just hides the inconsistency. So there was that set of small studies. And basically what we know from other fields is that you shouldn't be doing studies with 10 or 20 people that are looking at thousands of variables, right? You're just going to get noise, especially when we know it's a noisy

measure. And we know that because if you measure the microbiome from one person from one day to the next, it's super variable. Like, it's a really noisy measure. Kevin, just to check, wasn't this like, I think you covered this in the paper, but wasn't this like an issue that cropped up in other fields when people came across technology, that allowed them? Maybe it's gene-wide association studies or genetic sequencing, but they were dredging the data and showing, okay, this gene might be associated with this, but then it turned out that all of that literature was like. Absolutely. This is the sort of frustrating thing, I guess, is that we've been through this, right? We know this is not how to do it from genetics. And this was like a painful experience in the field of genetics. People would take a set of genes that they were interested in. They'd analyze the genetic variants that were present in those genes in people with one condition versus another, whatever it is, you know, schizophrenia, autism, whatever it is. And they would find some differences. And then they'd publish those and then someone else would publish and, you know, so on and so on. But.

these small studies just generated noise. And it wasn't until people started to realize that was happening that we fixed the problem by making these huge consortia. Like, we could no longer do this cottage industry. It's just not okay to have even 100 or 200 people in your samples. We really need like 200,000. I mean, it was that kind of scale, right? And then the whole field had to change the way that they worked

from each of us doing individual things to pooling all of our resources and, you know, doing things on a huge scale. And then that really, really robustly showed real findings, but also showed the initial sort of stuff was all noise, right? And then, you know, the other field is in neuroimaging, right? I mean, if you take a scan of people's brain, there's so many parameters in there that you can compare between groups. And there's vast literatures claiming to have found a neuroimaging biomarker of depression or autism or whatever it is, right? In, you know, 20 patients. versus 20 others.

If you took 20 Gemini's and compared them to 20 Libra's, you would get differences in their brains, right? So, yeah. And so, again, what the field has figured out is actually you need sample sizes in the thousands, not in the dozens. And so a couple of papers came out recently for the microbiome stuff, which showed the same thing. To find even the most robust differences that we know about.

So effects on the microbiome due to diet or age or smoking, or eating fruits or diabetes, right? Those are all factors that we know have a big effect on the microbiome. In order to robustly detect those effects, you need samples of like 1,000, 2,000 people, not 10 or 20 people. So it's very clear that you can't do these studies

with these small numbers of people. And actually, we don't have to spend any time figuring out or thinking about what to make of those data. You can just, with confidence, ignore them now. We just don't have to talk about them anymore, right? Now, what people have done has gone on and done larger studies, right? So they have done studies with, at least in the high hundreds of people. And those studies are also not consistent with each other. Each of them finds something, right? But they're not consistent with each other, except in that what they can

look at is the overall amount of variance in the microbiome measures that's attributable to the autistic diagnosis. So in your sample of people, you've got people with autism, people without, but they also vary in lots of other ways, diet, age, whatever, right? And you can say how much of the difference in the microbiome is due to the fact

that these people are in these two different groups. And those differences range from zero percent to five percent. So in some of the studies, up to five percent of the variants in the microbiome diversity was due to this autism diagnosis, right? And then, okay, so first of all, like, that's a really tiny effect, which is important for thinking about your design of your mouse studies or clinical trials, right?

But secondly, that is still just an association, right? We don't know what is going on causally. And it turns out that some of these studies that control for things like diet find that that association is driven by diet, right? It's driven by the different behaviors of the people. So it's a small effect in the first instance. It's not consistent. And when it's there, it's most likely driven by a confound, not by the causal arrow that people are claiming.

You know, this makes me think, Kevin, that again, something we've come across in people talking about gutball ailment, also talking about toxic mold that's occurring. Oh, yeah. Okay, that's current. It's taken out Jordan Peterson. But in that case, there's a whole cottage industry.

And this includes perfectly credentialed people, you know, with medical degrees and so on, where people will talk about that they've got tests. Yes. That show that, you know, that they were impacted by toxic mold or that the gut biome is out of whack. And when we were covering Gwyneth Paltrow, she also had a doctor on who was talking about how they have, you know, their own bespoke ways that all our tests don't pick up the differences. and it felt like there's a very big industry around telling people it is the gut biome

and there's so many different markers like you're indicated because it's a complex system that you can always detect something, right? Like if you get enough past. Yeah, and the concern is like some of these studies now are moving to machine learning as a way to find patterns, right? Now what machine learning is going to tell you it's going to find a pattern, right?

It's going to find something that's different between your groups, because that's what it's designed to do. And so it's a sort of a hyper-powered way to generate spurious findings that are now uninterpretable because you don't know what's gone on in the black box. Like, it's not coming to save us. It's just a terrible, terrible way to analyze your data, right? If you could take a terrible problem, which is this massive ratio of the dimensionality of the variables to the end, and you go, let's make it much, much worse.

much worse. By feeding it into a much more flexible AI, your idea. Matt and Kevin, imagine you had someone, you know, like some listener or what I, who didn't know much about machine learning and you needed to explain to them basically what that involved. What kind of thing would you do to explain and why that's a problem? Not me, you know, just a listener. Do you want to have a go with that, Matt?

Yeah, I'm happy to. So, I mean, if you compare a machine. learning model fitting some data compared to your normal boring linear statistics, then essentially they're much more flexible. So if you've got a bunch of data points that are on an XY plot, you know, if you're fitting a straight line to it, it's pretty constrained. It's only got two degrees of freedom to fit it with. If you fit it with a really wiggly curve, then you can fit any data that comes along. You just make the line wigglier. So that's probably the way to think about it. Yeah, exactly. You basically explode your data to a latent space that's massively

multidimensional and then your degrees of freedom explode exponentially but also you already have too many degrees but also i mean the other thing is that you we as researchers then don't know what the what the thing has done right at least when you do a regression you can probe it and say which are the bits that are causing it right but the the black boxing effect here is really is really an issue so this is the opposite of a multiverse analysis kind of approach where you're running like and saying, okay, when we ran this 1,000 times, this was significant twice. I don't know if it's the reverse fit, but it's definitely not the same as it. And, you know what?

I mean, also there's another sort of issue here. So there's a couple of papers that have come out claiming that they do, they train their machine learning model on the patterns in the microbiome of their sample with autism and sample without. And then they claim that they can get a predictor, right? So their machine learning model has learned, what a microbiome signature looks like, and it can predict autism in other people with some greater degree of success than chance,

not up to 70% or something like that, 80% in some cases. The problem is, and then they propose that this could be a diagnostic tool, which to me is just such a head scratcher. It's just like a category error, because autism just is the name that we give a cluster of behavioral symptoms. And the way that we give people a diagnosis,

is by asking if they demonstrate that cluster of symptoms. And it's not like you're going to give somebody a microbiome test who doesn't have those symptoms and then say, you know, we've got news for you. It turns out you're autistic. And they're like, well, I don't seem to be autistic. Well, nevertheless, that microbiome says you are. Right.

It's just kind of a weird or vice versa, right? It's like someone who has all those symptoms. And then the, what are they going to say? No, you're not autistic. Your microbiome doesn't say that. I feel you're not cynical. enough, Kevin, because I guess my goal happened there is people would

get the positive test, I would have none of the actual symptomology and then say, well, I guess that means, true. I'm autistic, so yeah. So anyway, I mean, like, you know, overall, looking at this human association study

literature, we see this pattern where there's load of small studies, they generate lots of noise, it's really inconsistent. The sample sizes are like a hundred times smaller than they should be. And effectively, there's nothing there in that literature that actually supports this claim that there's any real association going on, or if there is any tiny association that it's important, or that it's going in the causal direction that people claim and, you know,

not confounded. And in fact, you know, one of the really important things that came out of this, the study when we looked at these things, is that when people do a study with control, within the same family, like sibling controls, then they don't find these differences, right? Which really suggests that there's confounding with other familial factors that is driving the small differences that people have seen. So, I mean, the bottom line for us is that, look, you know,

Matt, you said it earlier, you can see that all these studies are coming out. And even if they're a little inconsistent, it's like, well, look, there's so much smoke, there must be fire. But actually, sometimes there's just loads of smoke. right? And that's what we've seen in these other fields, like the neuroimaging studies of biomarkers of depression or whatever. It's just smoke. And the same with these candidate gene association studies, just smoke. There's nothing there at all. Right. So I think there's like one other line of evidence that you considered. And is it human trials? Yeah. CTs, that kind of

thing. Yeah, exactly. So RCTs. Yeah. So basically there's three, there's three legs of the dueled. At least that's the way that people talk about it. So when people are doing the human association study, they'll often say, we know from a mouse work that such and such is the case. And when people are doing the mouse stuff, they'll say, we know from human work that association studies are, you know, associations are valid and so on. And then the clinical trial people will say, we know from human association studies and mouse work that these things, there's a causal implication. Let's test that in humans. And the idea is to test either probiotics, which are it's a weird term basically means bacteria that are taken to be good somehow, some beneficial

bacteria, defined in some rather nebulous fashion, and we'll give those two patients with autism who are enrolled in these small-scale clinical trials, and then we'll see if they improve, right? The other one is to actually give a fecal microbiome transplant, and this has been done in different ways. One of the studies that's been really highly cited took a fairly drastic approach where they actually gave children like a two-week antibiotic purge that got rid of their own microbiome, and then they transplanted in the microbiome of healthy people and asked, do their symptoms improve? And they claimed to find that there were some improvements in symptoms, right? So that's the headline, is fecal microbiomes from neurotypical people can improve the symptoms of autism.

That's the take-on message. That's how it's been presented. Now, when you look into it, what you see, is, first of all, there were 18 people in this study. Secondly, there was no control arm. Everybody got the thing. It was open label. Everybody knew what was happening, right? So very, very prone to placebo effects. I mean, there's a reason when we do clinical trials that we give some people the treatment and some people a placebo. And the reason is when people get any kind of treatment, they tend to report their symptoms improve. In this case, their parents reported their symptoms improved. So it's a kind of a manifestation. of wishful thinking sort of effect, but very, very strong.

So what has happened in these studies, if you look at them overall, is that a bunch of studies have reported some positive effects, but they tend to be ones with these open label, single arm, no placebo control, and small numbers of people. And then when the studies are done properly with randomized control trials, where you have two arms, one people getting the treatment, one set getting the control placebo, and the people are randomized between them, then there basically are no effects, right? So across these different studies, you know, what we're seeing is small samples, again, lots of research degrees of freedom, problems with controls,

and the smaller studies are the ones that show something positive, and then the bigger studies that are done more rigorously don't show anything. So again, in looking across all these studies, and, you know, I want to emphasize, this isn't just us, you know, the three authors of this paper who are being critical, it's people in the field, you know, are making, like we quote a bunch of meta-analyses of these studies that basically have negative conclusions. They say there's just no evidence for any efficacy of probiotics for autism or anything like that. And nevertheless, you can find, like if you search online, what are the best probiotics for autism? You're going to get a bunch

of sites that are offering you various concoctions that you can give your child to treat their autism. So this is definitely just out there in the public perception. I mean, what's interesting is, like I said, these three lines of evidence, people point to them as providing support for each other. And I would say, first of all, that they're actually not, they're just not commensurate with each other. They're not doing the same types of things. It sounds like they are because they're using the same words, but they're actually just premised on very different ideas. So, you know, I don't think of them as three legs of a stool. They're just like sticks on the ground. And so they're really not related to each other. And then when you pick up

any of the sticks, it crumbles in your hand, right? There's just, there's just nothing left. So, you know, I think the idea that there's just this very robust connection that has been proven is just not the case. Yeah. The thing that struck me when, like, reading through, apart from in general, just my eyes boggling about all the various. Every time I come across a review that it's like this and the low standards in studies from recent years. Because, like, to me, the notion that you would take a clinical trial and you give someone a treatment, and they know they're getting a treatment, and they're likely, in many cases,

to be patients who are actively self-selected into that because of they believe in the treatment, right? So there might be parents who think that it's about the gut biome. Exactly. They're being treated by a doctor who believes the gut biome is the key. And then you've got people doing the analysis who think that it's gut biome. and you don't have a control group. Yeah.

Like a proper control group. It's just to me like, well, that's like a kind of recipe for this. And we know it's a recipe for disaster, even with the best intentions of like everyone involved. So yeah, that that was just kind of shocking to me. It's very frustrating that this kind of thing still goes on, right? Because, yeah, everyone knows that's not a good way to do things. And it's a weird sort of dynamic that emerges that people, People think, okay, well, I think something might be going on here.

So let me do a pilot trial, right, as a sort of the way that these things are pitched. But then, like rather than saying, okay, I have some preliminary data, now I should do the real trial at scale with the proper conditions, they publish the thing and then they make claims off of it, right? I mean, I don't have any problem with doing a small scale exploratory thing, but just don't hype it up, you know, as proof when you know that everybody knows that this is not the way to do clinical trials. files. Well, Kevin, I was going to ask you about your conclusion after going through all of his evidence, but I think we already heard it. We don't have a sturdy spill. We've got three flimsy sticks lying on the ground, decomposing slowly. I saw it through them a little bit. But, you know, two, I think there's, I mean, I can think of a couple of takeaways, but I want to ask you what the takeaways are. But for a member of the public who sort of casually

looks at this kind of scientific literature, you look at that citation list. It looks incredibly impressive, and it looks like there's really something there. Even a casual academic, unless they do a lot of work like you've done, could easily get that oppression as well. As for Andrew Huberman or Jordan Peterson, they have no chance whatsoever. So, like, that's one of my takeaways. But what for you? Like, what do you come away from doing this kind of exercise feeling? What are the implications for our scientific culture?

Are you okay? I just want to drink. You just want to go lie down for a bit. But what are the implications for scientific culture and also what is the implications for the public understanding of science? Yeah. I mean, I think there's a few things. First of all, that, you know, the work that people in some fields have done around

reproducibility and robust research practices and so on clearly has not been socialized to all fields and needs to be. So we do need to keep banging that drum and talking about, you know, good ways to design experiments, good ways to do stats, and bad ways, right? And we need to stop doing the bad ones. I think, you know, what's interesting in this field, you can ask, well, how, like, how is the public supposed to evaluate these, or how are other scientists not in the field supposed to evaluate these when they're peer-reviewed and they're published in really high-impact journals, right? So those are seemed to be, like often taken to be two markers of quality

where someone who isn't an expert in the field can say, well, look, I don't know what going on, but I feel like I should trust it. Some people in the field clearly thought this was good. The reviewers thought it was fine. The editor thought it was fine. This journal thought it was fine, right? So I guess one of the keys, well, one of the questions here is like, what's the dynamic within this field in particular that leads to this? And I think there is a kind of a shared conflict of interest in the sense that like if the three of us are working in this field and, you know, I published a paper and it's sent to you two as reviewers, well, you know, you might quibble with some of the things. The findings might not be exactly what you've seen, but it's still

in your interest for that paper to get published because you can point to it when you put in your next grant proposal or when you put in your next paper to say other people have found something like this in the field, right? There's a shared interest in just the phenomenon being a thing in the first place. And, you know, what's interesting in this literature is that all the literature is just continually trying to show that the phenomenon is a thing. It's just doing that over, it's like it doesn't go anywhere else. And that's a real red flag is when you see some literature making some claims, and then 15 years later, there hasn't been the normal kind of follow-up where you would say, okay, here this study came out and showed X. They replicated that

in another study. They dug a little deeper. They found this mechanism. We're getting closer to what's going on. That's the way normal science happens. And instead, what we see in this field is like, oh, this study shows X. And then, well, this other study shows, eh, X prime or Y, but it's sort of like, and then this other study shows something slightly different, but it's continually trying to prove that this thing exists. And so I think, you know, as consumers of the scientific literature, I think there are some red flags there that we should be wary of. You know, there's small samples, there's this dynamic of conflicts of interest, just academic ones, but then, of course, there's also commercial conflicts of interest, right?

So many of the proponents of these claims and some of the authors of the study, that we look at have declared conflicts of interest, where they have some commercial interest in the probiotic that they're using or the model that they're using or the biomarker panel that they think that they've found. They have relations with big food companies in some cases with other sorts of entities, biotech companies and so on. So that's a flag that you should take into account, take into consideration, when you're looking at these kinds of data.

And I think that dynamic is interesting in the general public because there's always this red flag is like, oh, you know, you're working with big pharma, right? And there's a huge big pharma industry and they're really, you know, this terrible evil people who just want people to have diseases and, you know, so that they can give them drugs. And I mean, I'm not here to defend big pharma.

There's so many problems with big pharma. But what's missed in that is like big web, I mean, the wellness industry is like a billion, billion dollar industry. It's incredible. So there's a sort of a blind spot that some people have there. Sorry, Matt, I forget what your question is. No, those are all really good. Kevin, you give like a whole bunch of them. And I would also say that like, again, this makes me think of all our literature is like the things that are currently around psychedelics. And there you have conflicts of interest. with industry as well. We're going to speak to someone soon who made a report about that. But then one point that just struck me when you were talking about that is like it actually is a little bit of a hard ask for someone because, you know, when you're talking about a literature that's kind of stalled and it's just invested in repeatedly justifying its existence, I think it's hard for someone who isn't like grounded in methodology and statistical analysis

and stuff. They understand, well, what's the difference between that and when you're talking about that we need replications and robust demonstration of effects. But I was thinking about very recently there was a study, maybe you've seen it, the many babies study where they tried to replicate the pro-social effect, then very young infants where they prefer pro-social objects. And in that case, that's a huge study, you know, it's a many lab studies. So it's like 37 or whatever labs involve, thousands of babies. and they're just replicating one methodology.

It took five years, you know, they go through and the results are null. Yeah. Like the results overall are null. And we know that those kind of things, pre-registered, large multi-labs studies, without the people that are, you know, or even adversarial collaborations if you want to deal with the issue about conflict of interest, these are solutions. So I think one thing I just want to add as an addendum is that even while it is hopeless

and there's a lot of things in academia that are like bad incentives and bad studies are still appearing, we do have solutions that we know work or at least help and they could be done. Absolutely. Yeah, yeah. No, I mean, so towards the end of the paper we conclude with

this sort of, okay, what should we do now? Section and we sort of put it in two ways. One is like if you still think that there's potentially something there, that is a hypothesis that's still worth testing, then, you know, do it properly, like do it at scale,

pre-register or at least delineate your hypotheses beforehand so that you don't have this huge garden of forking paths that you could follow in your analyses, have a replication sample, do it across different labs, and, you know, have thousands of people. That's the right way to do it, and then you'll have an answer. Okay, so if people are still invested in this idea, despite, as I say, you know, there's not a really good rationale for it

and there's effectively no evidence in support of it whatsoever, in my view. But, you know, if you want to, at least, you know, just don't do it methodologically poorly over and over and over again, right? You know, just face up to the fact, as I said, you know, genetics faced up to this fact that they couldn't just keep doing small scale studies and had to pool their resources. The alternative is to say, actually, we've, we have explored this and there doesn't seem to be anything there, so let's do something else, right? Let's stop working on this, in a sense. We don't necessarily expect people to do that, but it's interesting, like, to say,

what would make you stop? What kind of evidence, if you keep doing this, what's the stopping rule, right? You know, what would make you stop doing this? And if, you know, if they don't have one, then it's not an honest endeavor, right? You know, it's not testing a hypothesis. It's really trying to prove it. And that just becomes bad science. and, you know, something that I don't think people should engage in. You know, so I think researchers in the field should think about this. And again, like all of the problems that we pointed out here are not just ones that we've pointed out from outside the field. People inside the field have pointed out the issues with animal models, with small scale trials, with open label, small sample sizes, you know, all of that kind of stuff.

So it's not just us throwing cynical bombs from the sidelines. I guess the other question then is, like, what should funders do? because their decisions are often the ones that drive things, as well as like, you know, editors of journals and reviewers and so on. It's like, you know, what standards should they hold things to? And there, you know, there's a lot of funding that goes into this. I just saw there's a new program just been announced by Welcome. It's called the Welcome Leap Foundation in the States of $50 million specifically for this topic.

And they cite some of the animal studies that we just went through that, you know, are really, really not any kind of strong evidence for any further follow-up. But clearly, this is still, like, it's still out there. It's still a very, very live hypothesis. And so, yeah, I think people should be cautious in interpreting it. And I think funders might want to rethink where they're putting their investment. Yeah. Yeah, I think I got one final sort of philosophical takeaway for everyone.

Because, you know, not, especially psychologists, we often, we have got people who listen to it doing their PhDs or doing postdocs, whatever. They don't necessarily have a million dollar grants to do really strong, powerful studies. But, you know, I think there is still something you can do, both as a producer and a consumer of perhaps relatively small scale research, which is try to be genuinely dispassionate about the outcomes that you're looking for, right? It's great to have an interest in knowing whether or not something is true or not and to be passionate about investigating it. But it's a difficult thing to do. It's a discipline. You have to kind of force yourself.

You have to try not to be invested in the answer and recognize your own motivations, both as a researcher and a consumer. If you feel that the gut microbiome, that feels right to you, feels real sexy, it feels something that you're jived about. Then, you know, hold your horses and go with caution because we've seen this in some. many psychological fields, like some of these things, like mindfulness training, meditation, like the entire field, positive psychology. I mean, all of those sexy results that you kind of know from social psychology. They're all stuff that you want to be true. And it's true of the

researchers and it's true of the readers of the research. So we have to try to be too passionate. Yeah, I think that's right. And, you know, in particular for the microbiome stuff, you know, I should say I've been really critical, obviously, of the way that the research is done. I don't want to be critical of the motivations because the people who are doing this, you know, they're motivated to want to help people with autism, right? And if there's a possibility that the microbiome was really causally involved, then that's great because it gives a treatment possibility, right? So, you know, there's a positive motivation there. It's just that you're right. You have to be dispassionate about it and put that to one side and say, okay, I would love to be able to help people with autism. Maybe this is

true. But, you know, if the evidence continues to show it isn't, then at some point you just kind of say, okay, well, that didn't pan out and try something else. So, yeah. It reminds me as well that, you know, in general, I think we're, all of us here are on board with it, but the notion that like, ideally, I'm for science of progress, you want to be doing severe hypothesis testing, not just saying, we'll find a difference from the null. Yeah. And so many studies seem to be doing the opposite or any, any, change from baseline that can be detected is a hit, right? And there's another dynamic here, which is, which I see all the time, which is that the more

sort of extraordinary the hypothesis, even outlandish the hypothesis, the lower the evidence bar to get this published in really, really high-impact journals, right? So it's exactly the inversion of what, you know, it should be extraordinary claims require extraordinary evidence, but if you've got something which is like, oh, this is a paradigm shift, this is going to blow your mind to totally change things than much easier to get it into high you know you know we we don't have time to talk

about it now Kevin but I mean at some point I want to convene a round table of skeptical people to figure out what the fuck is going on with the with the journals and the publication treadmill and so on because some things some things not right

yeah we just get Eric on board but um you know Kevin we we've took up a lot of your time, but we can't let you go without one final question. And, you know, it's important we've been talking about your paradigm shifting ideas

and thinking, Matt's recently become very interested in pan-psychism. He's kind of like an acolyte you could say. And I'll summarize his position, but I just he doesn't believe me that there's an issue. So I thought I'll ask you because I think you can, you know,

steal a man and respond. So as I understand, understand it, right? The pan-psychism, the notion that, like, consciousness is a fundamental component of the universe, right? Like, matter is not the unit, it's consciousness, and consciousness exists in everything, including, like, things with no brains or no agentic possibility within them, right? Now, from my perspective, my non-philosopher, non-neural science-informed perspective, I regard consciousness as a process that emerges from like a genetic biological units, as described very nicely in your book, which I read. But Matt assures me that panpsychism

is a deeply serious and very coherent approach. So I'm just asking, is this true? Have I been misled that panpsychism is actually, there's a lot to it scientifically? I see Matt shaking his head there. Like he doesn't agree with this characterization of what it said. But, you know, I'm used to this, Kevin. I'm used to this. So you go ahead and respond to that. I'll heap some more abuse on you then, Matt. Yeah, so panpsychism for me is one of these sort of baffling

ideas that doesn't do any work. So it just, it says here we have this mystery. Some things are conscious or consciousness is a property that we see in the world of some things, where could it come from? It's just this really, really hard problem, right, as it's called. And one way to solve that is to say,

well, maybe everything's a bit conscious. Maybe consciousness is at the root of everything, and therefore, like, you don't solve the problem. You're not explaining any of the phenomena that you started with. You just dissolve it. You just kind of push it under the rug and say, you don't need to worry about that. And it's just a weird sort of,

premise or conclusion because you could say the same thing for life, right? You could say, well, we don't understand life. It's this complex, mysterious thing. Some things seem to have it. Some other things don't, but we don't really understand why or where the transition is. So maybe everything's a bit alive. You know, maybe electrons are a bit alive. And then when you put them together in certain ways and atoms and so on, they get more alive. It's like, well, no. It's just It's just not doing any work, philosophically speaking. It's just saying you don't need to worry about the problem. So for me, it's not very helpful and not very interesting because it just doesn't work.

I mean, the phenomena that we are looking at is like, first of all, we're having subjective thoughts and experience, right? But also, sometimes we don't. Like, when I'm under anesthesia, I don't. So there's a contrast case that's interesting. Now, pan-psychism doesn't do anything to explain what's happening there, right? It doesn't explain what happens when you're unconscious. It doesn't explain having subconscious psychological processes that you're not consciously aware of and can't really access. So all of the sort of phenomena of consciousness that make it an interesting thing to study are just sort of pushed under the rug by pan-psychism.

They just say, don't worry about that. Or they offer no explanation for it. So if you want to think spoons are conscious, fine. Like, go ahead. I know you have your dinner to get, though. That was it. That was a good explanation. I just want to correct this really rather evil mischaracterization.

As an offhand comment, I forget what we were talking about. What was it in? I was contrasting it with. There was something. Oh, I was saying Alex O'Connor, because Alex O'Connor became a pun, or I was talking about how pan-psychism is appealing. Yeah, we'll talk about something. There was some other sort of thing that there was even sillier.

And I say, well, you know, at least pan-psychism is kind of coherent. And sort of, like, it has a simple, like, a simple kind of elegance to it on the face of it. Which is, which is exactly what you said, right? Which is, if you say that there is, you know, even the example you gave, sometimes we're very conscious. Sometimes humans are less conscious. and there are other animals if you admit that they're, you know, conscious to some degree, maybe a bit less than us, who knows, and you can go all the way down and there's, and you sort of, most people would have to admit there's a continuum, and the argument behind panpsychism

is just, well, if there's a continuum, then the continuum goes all the way down to spoons, right? Yeah, exactly. So on that incredibly limited, you know, in that incredibly limited way, I say, well, at least it's kind of elegant or coherent or whatever. I don't like pan-psychism. I don't endorse it. I am not, and I have never been a pan-psychist. I don't associate with very coherent.

I just triggered Chris by throwing them a barred. I think the label is attached and it's going to stick. I'm sorry. He's a pan-sikist now. I'm a pan-sikist now. Kevin, I know I promised I let you go back. I know you need to go get your dinner. But I have to ask you one other question where I have you here.

I mean, I can DM you it, but. Why not do it? Go ahead. So there's another debate that Matt and I have been having. I think you will actually side with Matt here. Okay. So I'm just curious. Right.

It's not a pronunciation debate, is it? No. So I'm definitely not citing with this. No, it is. It's not because I know I'm right on all those. But in this case, I think you might have more sympathy from, but I'll see. I mean unconvinced. Even after reading your book, okay?

I've read your work. I've listened to you to debate, Sapolsky, and I've read Chalmers and Daniel Deanna. I'm not an expert in it, but I've read around the topic. I just, I lack the bit where I'm kind of finding consciousness and subjectivity, this mysterious amazing. He lacks the ability to understand that there's even a problem to be solved or even a question. I do.

I like, I lack this thing because I feel like, well, but. you know, just things that have brains and whatnot, and they get more complicated and we are agentic beings that like to imagine other futures and stuff. So like self-consciousness in humans, not a problem. And then in other animals that have nervous systems and, you know, reactions and like in your brook, it kind of makes sense to me that this would be, you know, on the continuum of thing. But where's the first of history? I agree up to a point, right? So if you think of consciousness as a mode of cognition, right? It's a way of doing cognition where you have, you know,

you're not just sort of, you have like an internal model of the world that you can kind of run simulations over. That just turns out to be a super good way to do behavioral control in the world. And you could imagine building, you know, robotic systems that have multiple levels of cognition with a sort of a highest level that is running those kinds of simulations, sort of figuring out, you know, if I do A, if I do B, what are the outcomes going to be, what should I do, integrating all this sort of data with your

knowledge of the world and so on. So consciousness as this sort of highest level of your control system is actually not mysterious at all. What's mysterious is why it feels like something. Yes, that's what I'm saying. It's a subjective. That's what Matt says. Right, right. But why isn't mysterious? Okay, so that's the bit that I like the insight. Because I kind of feel like why, but it... Good luck, Kevin, because I've been trying to... Because, like, to me, okay, right?

Like the experience of echolocation for a bath, like this common example, it likely feels like something. I have no idea what it would be like, it would be an experience, but like, because Matt and the philosophers like to say, ah, but what if you could have a system where you could produce all the outputs and you have all the things, but you don't have any, the sensation of the yeah the zombie argument which I don't

which I don't like because I think if you produce all the things you would have the subjective feeling it's just but that's not an explanation of why it feels like anything and why it feels the way that it feels right those are the two aspects of it because imagine Chris you built this robot that I was

talking about that has these levels of cognition it's doing its simulation it has a map of the world maybe it's map of the world derives from echolocation maybe derives from vision or addition whiskers, whatever, right? But it's a sort of a relational mapping of where the robot is in the world. It uses it, navigate around, does all the things that we do with our conscious thought.

The question is like, if you built all those things, would it feel like something to be that robot? And it's just like it's really sort of nebulously defined. What does that mean? It feels like something. So when people are talking about consciousness, it's the sub-reliable. objective experience of it that becomes really hard to understand. And of course, like, everyone just takes

it for granted. I think why it doesn't feel like a problem is because we live it. You're just in it all the time. So the idea that it requires explaining, I think, you kind of have to pull back from it. That's why the zombie argument was made. I don't like the argument, but the underlying idea

is, like, let's get a different perspective on it to see the problems. Okay. In that case, though, the only, and I promise, I'll about like this after I say this. So the, like, if you built the, you know, the hypothetical computer where it did all the things and that it, let's just grant the kind of P-Zambi thing where it didn't have an internal experience, right? But it produced all the outputs. Like, to me, it would then just be a case of the potential for, you know, like convergent evolution. You can build an eye

in lots of different ways. So you might be able to build something that can, in whatever mechanism, do a version of conscious activity, but it doesn't have the subjective experience. But a human made of flesh and blood and genes and all that kind of thing, it just produces that kind of sensation from being made of that material. So, like, either way,

I kind of feel like while we deal with an n equals one planet with one self-conscious thing, there's nothing but thought experiments as a counter thing where there's something that's conscious that lacks our subject. I think we're getting, you know, we're getting to at least a, stage with AI and robotics where we can imagine,

it's not pure science fiction to imagine a scenario where we are going to have to wrestle with this problem. I think there's also the question of where the quality of sensations come from. If you're a baby, right, and the first time you feel something painful, it hurts. Like it feels like something. It's not just a signal, oh, I should move my hand away from this thing that hurt me. It's not just a robotic control signal

It has a feel to it has a raw feel to it And the question's like where does that raw feel come from Or like you know if you ever seen videos of people giving babies a lemon And they taste a lemon and they clearly have this experience of tasting something sour And they haven't learned it from anywhere right And they make the sour face And you know you can do it with dogs and it's very funny But it suggests there's some raw feels to experience

that, you know, it's just really tough to explain where they could come from, why they feel different from each other, why they, like, why does sourness feel that way? Right. It's just weird. It's really hard to explain. I like that answer because I still lack the irrelevant. But that speaks too. That's an nice balance. This is that Chris is a pea zombie. I was just going to say. I think we're edging, we're edging toward that's that conclusion all right but I like this because that means you know that basically

Matt Panzakis study is like he has vindicated that you know there is some sort of mystery that I just lack the ability to comprehend so that's good that's balance we both got to win we both got to win that's good yeah thank you Kevin

yeah you're welcome it's a great paper Kevin and we'll point to it is there a preprint there's not There's not a preprint, but it's going to be open access in neuron. Okay. Yeah. Yeah.

Okay. So we'll point to it whenever it's there in any case. But it's a great paper. Maybe we'll cover it, Matt, on the coding academia. That would be fun to get the people to, like, have a look at it. But, yeah, so great work. Oh, thanks. And really appreciate you explaining it to us in the audience.

Yeah, no. Great. Thanks a million for having me on. I appreciate it. Thanks, Kevin. See you. Thanks. Thank you.

Thank you. Thank you.