Maintenance Phase - The Worm Wars

Episode Date: June 28, 2022

In 1998, a researcher rolled out an innovative approach to education. And in 2015, Methodology Twitter had a big fight about it. Support us:Hear bonus episodes on PatreonDonate on PayPalGet Maintenance Phase T-shirts, stickers and moreLinks!The original 2004 paperThe 2015 re-analysisThe 2015 meta-analysisThe Tanzania dissertationDoing Good BetterWorm Wars: The AnthologyDeworming DebunkedWHO advises blanket anti-worming treatment for children despite lack of benefitDeworming Delusions? Mass Drug Administration In East African SchoolsUnderstanding Heterogeneity in the Impact of National Neglected Tropical Disease Control ProgrammesWhat Has Been Learned from the Deworming Replications: A Nonpartisan ViewMass Deworming Programs in Middle Childhood and AdolescenceIs it worth treating everyone who might get worms?New research debunks merits of global deworming programmesShould the WHO withdraw support for mass deworming?Why Don't People Take Their Medicine? Experimental Evidence from KenyaDear journalists and policymakers: What you need to know about the Worm WarsThanks to Doctor Dreamchip for our lovely theme song!Support the show

[♪ OUTRO MUSIC PLAYING [♪ Give me two seconds, I need to close the windows. I was like, it's so fucking hot in here. And I was like, I was trying to do it with the windows open, but there's like, shit loads of bird sounds. Yeah, totally. All right, no birds. LAUGHTER

This is how no No Birds podcast. You have never sounded more like Jerry Stiller. No birds. No birds. What have you got for us? Hi everybody and welcome to Maintenance Phase, the podcast that's crawling out onto the sidewalk after a nice long reign. Oh, it's a worm, it's a worm opener.

All I had was worms. You had to go back. You had two words, worm wars. Sure, man, I'll take my inspiration from my morning walk the other day. I like a hugs. I'm Aubrey Gordon. If you would like to support the show,

you can do that at patreon.com slash maintenance phase. And if you wanna just keep listening, just keep listening. We're so happy to have you. You're here with us. Today we're talking about worms. Yes, so okay, I'm so excited. Because I like telling you stories

and I like fucking with you. And so I told you, I told you nothing other than the name of this episode, Worm Wars. This is the controversy that Tor methodology Twitter usunder in 2015. And I want to tell you all about it. Okay. Tell me. So the hardest part of this episode is going to be getting through the first five minutes.

Oh. There's just some like gross stuff like body horror that we have to get through. Okay. Are you familiar with intestinal worms? Uh, I mean, not really. Okay, so I assume they're like parasites, yeah? Yes. Okay, so somewhere around two billion people in the world

have intestinal worms, which are like exactly what they sound like. They are little tiny worms that live inside of your digestive system. Okay. Some intestinal worms have lifespans of like one to two years. And they'll literally just like attach themselves to the inside of your intestine and just like be there for years and then they die. Okay. Most people have light to moderate infections which don't really have any symptoms. But sometimes you'll get like much more heavy infections. And so this causes various symptoms that it it usually happens through giving you anemia, it makes you tired, it gives you brain fog, you have to sleep all the time, it can stunt kids growth in the developing world, it can make them too skinny. I have never had a worm infraction or

parasite or any of that kind of stuff, but I have been anemic before and boy, oh boy. It doesn't seem like it's gonna be that rough and then it's pretty rough. Dude, for no matter what, passed out in a Starbucks because she got anemia. It's just, you just can't get your energy up like at all. Yes, so okay, so here's the really gross part. So the way that worms spread is you get these worms

in your intestine lining and they lay eggs. They lay like hundreds of thousands of like little microscopic eggs and these eggs end up in your poo and then you poo out the eggs. And so the reason why worm infections are endemic throughout the developing world is because oftentimes there isn't running water. So in some areas people will kind of use soil

to wash their hands after they go to the bathroom which kind of grabs and like dry soil from the ground. Sometimes they use poo as like fertilizers in gardens and then it ends up in their food or it's like they're walking around and they have a cut on their foot and there's like a little tiny egg that like ends up getting into the cut and then like swims its way to their intestinal tract. Sure, sure, sure. So it's just like the poop human interface is like how people end up getting worms. And it's like this rolling snowball where like the more worms there are in the soil,

the more likely kids are to get it, the more likely kids are to spread it, et cetera. It used to actually be a really big problem in America. Here is what I appreciate about this whole episode is I didn't have to bring up anything related to poop. I know. You just fully picked a topic and you were like, you know what I could go for right now is a little poop research.

I saw a methodology fight and I was like, oh, it's going to drive me into the poop world, isn't it? I can't have to go inward. So the protagonist of this episode is a guy named Michael Kramer, who is currently a Harvard professor. Okay. Before all of that, he had spent some time in Kenya as a kid and he was really fascinated by like why are people poor in this country?

Like what are we doing to alleviate global poverty? He eventually gets his degree in economics and so he decides to do his graduate work in Kenya. So he moved to Kenya, and he starts befriending people in the development sector there. He meets like World Bank people and NGO people and charity people. And what he finds is that the field of international development

is like in kind of what other people will later call a statistical dark ages. This is like the early 1990s, and the way that people do development, it's kind of on like the live aid model. Right where it's like a celebrity does something, and there's like this big fanfare, and then like no one ever follows up afterwards.

Yeah. And if there's any like statistical rigor or like studies being done, they're oftentimes before and after studies, so they're like, we gave everyone a go. And then a year later, they were doing better. And it's like, well, was just everyone doing better in the country that year?

Because there was like a really good harvest or something, like you're not comparing them against a control group. Or was there another policy intervention? There's like so many things that it could be. Think about the last year in your own life. And think about if someone from the outside tried to determine what was the cause of any changes

in your way, that would be kind of a fool's errand, right? And a lot of the projects are done in this sort of like common sense way. This is oftentimes how you get bad development projects where people are like, oh, it's just common sense to give everyone a bicycle or whatever. It's like, well, did you ask them first if they needed a bicycle? If you give everybody in the village a bicycle, is that going to change the dynamics of the

village? It's just being done in this really unsophisticated way. Michael Kramer has a friend who works at a Dutch NGO and they're about to implement a program where they give kids and Kenya textbooks. a gamer has a friend who works at a Dutch NGO and they're about to implement a program where they give kids and Kenya textbooks. So at the time, there's like 30 kids in every classroom and there's one textbook per classroom. So again, common sense. It's like, well, if everybody had a textbook, this is obviously going to make like their grades better. And like school performance attendance, everything's going to get better if everybody has their

own textbook. Obviously, right? So Michael Kramer suggests to his friend. He's like, no, no, no, don't do it before and after study. Instead of just giving this to seven schools, take 14 schools, and give seven of them textbooks, and the other don't give them textbooks, and then you can actually compare the effect of the textbooks. So they do this, the project goes for a couple years, and low and behold, there's essentially no effect to the textbooks. So after the textbooks don't work, there's this period of kind of testing stuff. It's like, well, that's weird that the textbooks don't work. So then they try like chalkboards, like a lot of the schools don't have chalkboards. So again, they split into two groups.

Turns out that doesn't work. They then do a thing where they hire more teachers at half of the schools. That doesn't work. So it's like they're kind of going over these sort of obvious, quote unquote, education interventions. And it turns out that like education interventions are not working to improve education. They're like, hmm, so one night Michael Kramer

is talking to a friend of his who works at the World Bank and he explains these symptoms of worm infection to Michael Kramer. And he tells him that like, there are treatments available for worms, like there are pills for basically every kind of species that's endemic in that part of Kenya, and like they're really cheap,

and diagnosing the kids is actually like really expensive, because you literally, like you literally have to go through their poop and like measure how many eggs there are. And so it's such a pain to diagnose this and it's so easy to give the kids these medical treatments. Like, they don't really have any major side effects. They're pretty cheap. They're already available in that part of Africa.

And he's like, I wonder what would happen if we just gave all the kids these deworming tablets because some of them are probably infected with worms and just don't know it. So they do the same thing. They find a district with 35,000 students, 75 schools, and they do the same thing. They separate it into control groups and treatment groups. They roll out this mass deworming program where they just give every kid deworming pills, right? So I'm going to send you an excerpt.

This is from a book called Doing Good Better, How Effective Altruism Can Help You Make A Difference by William McCaskill. Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote,

Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote, Quote,

Quote, Quote, Quote, Quote,ing reduced it by 25%. In fact, every child treated, spent an extra two weeks in school, and every $100 spent on the program provided a total of 10 years of additional school attendance among all students. Enabling a child to spend an extra day in school therefore costs just five cents. It wasn't merely that deworming children quote unquote worked at getting children into school. It worked incredibly well. It worked incredibly well. I mean, fucking yeah.

Yeah. If you have something uncomfortable to painful, to debilitating going on with your body, when you feel better, I bet you do better in school. Yeah, so I mean, it's a very appealing story also because they've tried all of these educational interventions. And it turns out that there's this sort of like zigzag solution that maybe it's not an education intervention.

Maybe it's like there's something broader and bigger at play that is just invisible to people that are in the school and also development aid workers. Yep, totally. So the main headline from this study is that they reduced absenteeism by 25%. That's the thing that ends up in William McCaskill's book. That's like the kind of the main thing.

It's in the abstract, but there's also some other effects. So one of them is like, there's health benefits. So the kids who got the deworming were slightly taller, they're slightly heavier. It was actually nice to read a study where it's like, the kids are fat. Yeah. It's like, it's like just once. What a win! They also had lower rates of anemia, and one of the really interesting effects of the

study is these spillover effects. So because worm infections are an infectious disease, right? One person gives it to another person. Once you blast hundreds of kids in these schools with deworming pills, they then don't spread it to other people. So what they find is that there's actually attendance effects in schools like three kilometers away and six kilometers away. There's all these other benefits that come along with basically just like breaking the chain of infection. And they also find long-term effects. This is from McCaskill's book. When Kramer's colleagues followed up with the children 10 years later,

those who had been dewormed were working an extra 3.4 hours per week and earning an extra 20% of income compared to those who had not been dewormed. In fact, deworming was such a powerful program that it paid for itself through increased tax revenue. Wow. So it's like you can then even fast forward 10 years and you find that like the dewormed kids are doing better big if true

Big if true Aubrey first five minutes of the episode You know everything I tell you at this point is true everything goes according to plan the nice white guy from Harvard Yeah, definitely right and don't worry. It's going to come from his actions stop remembering the way that I tell you stories Stop remembering the premise of our show. The premise of our show. So over the next decade, basically, D-Worming becomes the hottest idea in international development. I was working in international development at this time.

I remember all of the D-Worming national plans that were coming out. It paralleled the rise of this effective altruism thinking, where in Michael Kramer's original paper, which was published in 2004, a lot of the paper is actually about the cost benefits of this, right? That when it comes to something like hiring more teachers, well, like every single teacher costs a lot of money and there's payroll, there's all these other costs associated with these educational interventions, whereas this one, The deworming pills cost 30 cents each.

The benefit of this versus the cost of doing it is just unprecedented international development. So in 2007, Michael Kramer found a NGO called Deworm the World, which helps governments implement their national deworming plans. There's also this charity Give Well, which basically has this model of like, we're gonna look at all the evidence,

and we're just gonna tell you that like, if you have a hundred bucks to give away, here's probably the most effective thing to give it to. You know, if you want to help people, this is what you should do. And that's like one of the few charities that is ranked by Give Well is this D-Warm, the World Initiative and other kind of D-Worming Charities.

There was like a D-Worming strategy by the WHO, the World Bank was really big on D-Worming. It was like D-Worming was like the hotness, right? Kenya has a national plan on the basis of this 2004 paper. India declares February 10th National D-Worming Day and has a program to D-Worm 140 million kids a year. Whoa. This is huge. According to the Gates Foundation, at one point they put out a paper

calling this the largest public health program ever attempted. Wow. Huge full court, like decade long, full court press on deworming. There's part of this that feels very like Ted Talkie life hackie. This one simple trick fixes everything. Aubrey, am I getting ahead of myself?

I love that you're at the point with the show where you're like, wait a minute, there's a good story here. It's probably fake. I don't trust it. Like, boom. So, okay, I'm about to send you a couple paragraphs

from a Guardian article in 2015 that kicks off the worm wars. Woo. This is one of the longest excerpts that we've ever read in the show, but I wanted to get, I wanted you to get like the flavor of this. Great.

The headline says, New Research Debunks Marits of Global Dewarming Programs. The headline says, new research debunks merits of global deworming programs. And I say, fucking called it. Twist it, twist. Quote, deworming children, once ranked by Nobel laureates as the fourth most effective intervention to solve the health problems of the whole world, offers very little benefit despite the millions of dollars spent on it, according to a reanalysis of the evidence. The World Bank and the UK government are among the big funders of deworming programs that

were claimed to improve not only children's health and growth, but also their educational attainment and ultimately their country's economic prosperity. However, much of that optimism rests on one particular study carried out in Kenya in 1998 to 1999, which reported that deworming improved attendance not only in schools where children were treated and taught about avoiding worms, but also in other schools because of an apparent reduction in transmission. Researchers from the London School of Hygiene and Tropical Medicine have now re-analyzed the trial data and published two papers in the International Journal of Epidemiology. Researchers used methods common to health research,

rather than those used in economic research, as the original paper had, god damn it. Good sign, the epidemiologist finally looked into it 10 years later. There had been no increased attendance in the schools where children had not been treated, as the original study said. That was down to calculation errors and was wrong. The researcher also found missing data and some evidence of bias in the finding

that school attendance improved. Shocking quiz. Who could have seen that coming? What is your overall impression? What is the plain language paraphrase of what you've just read? The fucking epidemiologist never looked at an epidemiology issue,

and it was studied by fucking economists. What are we doing? And then like whole fucking nations built their interventions around what a fucking economist thought was good for public health. At this point in 2015, I've seen various numbers going around. One of the numbers is that one billion D-warming pills are being distributed annually. What? Under programs like this. I assume you're gonna walk me through this. Yes. It sounds like what they have debunked here is the idea

what they have debunked here is the idea that school attendance did not improve in schools where kids were not treated. That's a very good catch, yes. It's not the whole thing is garbage. It's just this idea of school attendance going up in other schools is not true. That's what I'm taking away from this paragraph. That was a very good read that like a lot of people

did not notice at the time. You just felt it's now. Fun. So now we get misunderstandings. So this is the thing. So we had to talk about what they actually did and what they actually found. So as you know, there's like in psychology and other sciences,

there's now this like this thing called the replication crisis. Essentially, the idea is that like there's all these canonical studies that have affected policy and have affected public understandings and they're basically rerunning them. And in a lot of cases, they don't pan out. So things like the infamous marshmallow test. What happened here is that people looked at this study from 2004 in Kenya being used to justify deworming 140 million children in India and we're like, wait a minute, like a lot is resting on this paper. Let's just look into the findings.

So like they didn't redo any studies. This wasn't a replication. What they did was they went to the original author, so it was Michael Kramer and his co-authors on the original paper, and they said, give us all your data. Like give us the software that you ran, give us the original like the questionnaires you used in the school, like just give us everything.

And all we're going to do is just run it again. Okay. So to their great credit, Kramer and his coauthors, like give them everything. Okay. So these reanalysis researchers spend months going over everything and basically publish like a reanalysis of this paper. So the biggest thing that they find is there's data missing and there's data that's miscalculated. You know that like spreadsheets have all these

conditionalities where like one cell refers to another cell. It's like they basically got one of these little codes wrong and that ends up cascading down to all of the other codes in this way that was like invisible to the original researchers. So it's essentially like the level of like a transcription error. Exactly. So as a result of all of these errors, everything of kind of recalculating the data, all of the health effects of deworming disappear.

What? The anemia link is gone, the kids, difference in heights, gone, the difference in weights, gone, and all of them. Essentially all these spillover effects. So like all the stuff about like the indirect, three kilometers, six kilometers, those effects, the six kilometer effects are gone,

and the three kilometer effects are much, much, much weaker. To me, the main thing that comes out of this statistical reanalysis, I mean, this is something very well known in studies of school attendance in sub-Saharan Africa and really throughout the developing world. That like if you rely on school registers, they're not very accurate. Teachers write kids down as being here when they're not really here. So if you want to do a study that actually measures school attendance,

you have to send people to the school to like physically count the kids, which to their credit they did in this study, like they sent people to like, okay, one, two, three, four, five in every school when they wanted to do it. And they did it at random times, right? You wanna get an actual like an accurate kind of the kids. What they found in this reanalysis was that the deworming schools got way more visits from the counters, from the auditors and the schools that got visited more had better attendance

rates. It could be just the fact that someone was checking that's doing this and not actually the warming. So this is what turned me into the fucking Joker on this. Once there's all this like statistical stuff going on and this extra scrutiny on the original paper, people also go back and just like read the original paper. The first thing that jumps out at you

is that this wasn't a deworming program. It was a deworming and education program. So the school that got the deworming pills also got like classes about like how it's important to wash your hands like better sanitation techniques how to prepare food the kids and their parents in some cases were getting information along with the deworming pills that was related to not getting reinfected with worms. No, but it's definitely those pills.

But it's not I mean this is the thing is like the, the, the researchers say that when you survey the kids about like how important is it to wash your hands and stuff, there's actually no difference between the kids that got the education stuff and the kids that didn't. Wow. As we talk about on this show all the fucking time, like public education campaigns don't really work. Yeah. Knowing that washing your hands is good doesn't give you running water in your house. Yeah. The researchers are like, oh, well, it doesn't matter that we also gave them education programs because the education programs didn't work,

so we can scale up the pills without scaling up the education. But the thing is they didn't check the effect of the education on the school attendance. There's literally no way to separate those things. You're doing two things at once, and then school attendance gets better. I am just really breathing through feeling my blood pressure rising. You don't have to be like the world's greatest fucking scientist

to figure out that if you give somebody deworming pills and also like a bunch of like educational resources and support for their lives, you can't then say, everything that's changed is just because of those pills. We've been through this so many times as a field. This is as a former development aid guide. This is what drives me nuts is that what you have over and over again

is you have a technical intervention, something like a medical intervention or like better lighting in schools or whatever, something that's easy to scale, right? But then you also have all these social things that go around it, right? Like getting to know the kids, having them attend these educational seminars, having probably just like more adults in the school generally if they're participating in a study, right? And then it's like when it's time to upscale, we're like, oh, this is a proven development intervention. So we're gonna upscale the technical parts.

Yeah. But you're not gonna upscale all the fucking social stuff that made the technical stuff work. God damn it. That's not the purpose of randomized control trial. The entire point of all of this was to be like, we're gonna do it like pharmaceuticals do it, right? We're gonna do it in a way that allows us to say

with certainty that like this is the intervention that mattered. Yeah. And then they do this like 101 shit where it's like, oh, we're going to do two things at once and then upscale one of them. Yeah, totally and also like this does feel sort of related to because this is all happening at the site of education as a field. It also feels totally fucking wacky to and deeply predictable to have a bunch of economists really and be like, teachers, you've been fucking it up, you're welcome, we're

here, we figured it out. But then, okay, are you ready to explode? Oh no, I guess. So the other thing that people start like bringing up but the original paper is that that number of it reduced absenteeism by 25% is a little bit misleading. If you imagine a school with like a 98% attendance rate and then you switch the vending machines from like Coke to Pepsi and then they have a 99% attendance rate. You could say that they reduced absenteeism by 50%. Oh god dammit.

That sounds really impressive, but what you've done is you've gone from 98% to 99%, which is not that impressive. No, this is the like fucking diet marketing of like your twice is like lead a lose weight on our diet, though it's on this other diet. And you're like, okay, so you lose one pound instead of half a pound and a year great. Your twice is likely to win the lottery when you buy two lottery tickets. That would have said, I mean, you're gonna win the lottery. The actual attendance rates at the schools

that didn't get deworming pills, the attendance rates are 75% at the schools that did get deworming, it's 82%. Oh, God. Is that nothing? No, like it's a 7% bump in attendance rates. But also a 7% bump in attendance rates. But also a 7% increase in attendance. Much less like we must scale this

to 140 million children immediately. Another thing that again is in the paper that was published in 2004 is that the dewarming pills did not have any effect on cognitive ability or school performance. So nothing. Nothing. So nothing. Nothing. So exam scores, they did some IQ,

test-e, pseudoscience, bullshit on the kids, nothing. So it only affected attendance. I feel like I'm resorting to fucking Lamaz breathing to get through this shit. I feel like I'm this close to like, to like deal with my fucking rage about this. Well, the whole narrative of this intervention doesn't really make sense.

Yeah. The theory of change that dewarming is supposed to do, right? It's like you give kids the dewarming pills, their health improves, their school attendance improves, their test scores improve, and then their life gets better, right? That's like the chain of events, right? But what we have now is we have, you give the kids the warming pills,

it doesn't seem to affect their health, it does affect their school attendance, but doesn't affect their school performance. Yeah. It's just kind of a mess. I don't know, man. This feels like an indicator of the depths of two major appetites. One is how fucking hungry we are at all times for a silver bullet thing.

Mm-hmm. And how much more fun it gets to people and exciting when it's like an unexpected thing. Like, oh, if you get this one pill, you get better at school, imagine. Totally. And appetite number two that is sort of shaping this is the unending appetite of white people to be like, we found the thing, we solved it, you're welcome, you couldn't do it yourself,

a weed, and it for you. Exactly. Not a bad impulse to want to help, a terrible fucking impulse to want to save people from themselves. I actually got radicalized on this. I think a decade ago now where I was writing an article about how big ideas were ruining international development.

I called up one of the charities that was doing one of these projects in India. They were expanding the deworming, nasty, warming programs to India. And one of them was Kafka-esque interviews I've ever had, I was asking him like, okay, how's the project going so far? And they're like, it's great.

We've distributed 10 million pills or whatever it was. And I was like, okay, great, like what are the results on school attendance? And they were like, oh, we're not measuring that. What? Oh, because it's proven. Like it's proven that it affects school attendance. So we don't really have to measure that anymore.

Oh. And I was like, well, it's proven in like it's proven that it affects school attendance. So we don't really have to measure that anymore. And I was like, well, it's proven in 75 schools in Kenya 10 years ago. You're doing this in India, which is a completely different educational context of schooling and like the government's role in school, everything else. And like, so you're just not testing the effects anymore. And they're like, well, we are testing the effects. As like, well, then what are the effects? They're like, well, we've distributed 10 million pills.

That's okay. Okay. So you and I get into this shit often. That's a fucking output. It's not a fucking outcome. Exactly. That's step one to do what you set out to do. Like, it's truly fucking wild.

How often you and I get into data on this show and the results of a given study or a clinical trial or a whatever are just, here's what we did. And then that celebrated like something that has like an impact on people's health out in the world. But Aubrey save, save your rage. I'm getting too angry too soon, Arnai. This is a three-part debunking,

and that was only part one. What? That is frankly nothing short of a personal attack. So at the same time that this reanalysis of the 2004 paper is going on, I think this is a coincidence. There's also a meta analysis of every previous study on D-Worming that's ever been done.

One thing that drives me absolutely batch it about this story and about the way that it's presented in this like effective altruism book is everyone pretends like no one thought of fucking D-Worming before. There have been D-Worming projects since the 1970s. So in 2015, there's a Cochrane review of every previous like decent study on this.

It covers 45 studies going back to the late 1970s. It includes 20 randomized control trials in Africa. It finds the review says, there is quite substantial evidence that dewarming programs do not show benefit in terms of average nutritional status, hemoglobin, cognition, school performance or mortality. Nothing. That's the good stuff right there. The world's largest range of mice control trial, or at least according to the study itself,

was performed in India in 2013 on Dewarming. It finds no effect. There's a huge randomized control trial in China, which also does not find anything. There have been 10 years of various deworming projects in Myanmar, and the rates of warm infection are now higher than they were 10 years ago. What? Which I don't blame that I don't think that has anything

to do with the studies itself. I think, again, it's like, I think there's something else going on there. Sure. What drives me nuts about this is like, why would we still be talking about this 2004 study in Kenya when there's like... 50 other fucking tests of this by now? Yeah.

People have tested this, and it has been found wanting. Time and time and time and time again. It doesn't have any of the effects that people said that it had. And that's on any measure, right? We're no longer in school attendance world. I mean, they mention school attendance. This is actually like one of the main things of like the worm wars, like the big methodological debate that comes out is because Michael Kramer and others say that like you can't test

whether it affects school attendance and like a meta analysis because there's not enough studies that do school attendance. It's only based on this one and they they all, like, they throw accusations back and forth, but, but, but, but, but, but, but it's like, there are other studies that look at school attendance and it doesn't seem to have any impact on them. And also, even if it did,

it's like, why are we talking about school attendance when, like, it doesn't affect school performance and it doesn't affect health. Right, then what? Then what? Then what? And also, there's also some interesting sort of second order effects that oftentimes when school attendance rates go up in the developing world,

the teachers are then overcrowded and like lots of the kids end up doing worse on tests because they're getting less attention from their teachers. Not to say that like school attendance is bad or whatever, but it's like, anytime you're doing these development projects, there's always these weird second order effects. And so it's like, it's not, there are actually places where even if it did increase school attendance, if it increases it significantly,

and they're not able to hire teachers, it might not actually have the long-term effects that you're going for. Yeah. Every study on this seems to find that it doesn't really do what they set out to do. In a lot of these studies, you can tell they don't want to admit that it didn't have

any effect. You and me are like trained to say, this now you'll read the abstract and they'll be like, we distributed 50 million pills and then like, we surveyed a bunch of kids and the kids had heard of the program. Yeah. And you're like, wait, that's not like no one would look at a dewarming program be like, have the kids heard of this or not. Yeah. It's like, you the kids heard of this or not. Yeah, it's like you're clearly picking something you can publish. We know that calorie labeling works because some people saw a billboard about it one time.

Exactly. I was sub-tweeting this when we were recording that episode. I was sub-tweeting the deworming people. I have to say, I really appreciate how often in the researching of an episode you or I end up like cryptically tweeting about something so that we don't spoil it only for the other person. The thing is I was gonna block you on Twitter because I was like about to do something talking about the warming and I was like I don't know. I had a whole plan. I was gonna be like,

Aubrey, I just want you to know I blocked you on Twitter. It's not you. I tweeted about D-Worming. But then I was like, oh, I don't want her to know. So then I just, I walked away from my computer, ultimately. Here's the hilarious thing. As you know, I have been very off Twitter for the last, like, I don't know, four months or so. So the thing that is hilarious to me is, I think the rest of the internet would be more likely to figure that out

than I would. Yeah, exactly. I know. And then did you know Mike blocked you on Twitter? Yeah, I do. Just like Mike blocked on being on Twitter. Trouble and paradise. So okay, final stage of the deep munking.

Okay. Okay, we've talked about the 2004 paper problems. We've talked about the field of deworming programs in general meta-analyses problem. We're now talking about the problems of upscaling. Because this is another thing that as a former development guy, I know that what happens in 75 schools in Kenya doesn't happen the same way.

Once you start rolling this out, on like 10s and millions of kids in like in entire countries. Sure. So this is something that happens in 2008 in a region of Tanzania called Moragoro. Tanzania adopted like a national deworming strategy and was rolling this out across the country. And so the idea was that they were gonna give kids a bunch of deworming pills and they had a town hall meeting a couple days before this happened to like tell the parents about it. But according to the parents later, they didn't like really get the message that this was happening or like they just had other things happening that day like agricultural

duties, things that night that they had to do. So very few parents ended up attending this like one town hall meeting being like we're going gonna give your kids deworming treatments. And according to the parents who actually went, the information that they gave the parents there was not even very good because they didn't really explain to the parents that they were doing prophylactic deworming. Deworming pills are like pretty widely available

in sub-Saharan Africa. And like when you get symptoms of a worm infection, like you go to the pharmacy and if you can afford it, you buy the pills and you take the pills and the deworms are gone. These are known, these are like known drug treatments, but the idea of we're gonna give every kid deworming pills like just in case

that hadn't really been done before. It's a little bit like some teacher coming to you and saying like well, you know, your kid takes Tylenol for headaches, right? Well, we're just gonna give every kid Tylenol at 3 p.m. every day. You'd be like, what? It just doesn't really fit with your understanding of how that particular medication works. Yeah. So, after this town hall meeting that very few people attend and doesn't give very useful information anyway, they start doing this deworming with the kids.

But another change that they make is that the deworming pills that people in Tanzania are familiar with, I think it's called albendazole. It's like a little tiny pill, and apparently it tastes kind of sweet. And the kids are used to taking it. You know what, it's like kind of difficulty at kids to take pills in the first place.

Sure, absolutely. But like, because it's small, because it's easy to swallow, the kids are kind of used to it, so it's not that big of a deal to get kids to take this. But there's different species of intestinal worms. So they're also having these kids take this other pill that is like big and bitter,

and they have to test the kids' height before they give it to them to get the dose. And so some of the kids are taking three of this, some of the kids are taking two of this. It's just like big and like really unpleasant to swallow. And the other thing, I cannot fucking believe this, but this second pill that they're giving people, you should take it on a full stomach.

Okay. Otherwise, it starts to give you these side effects. But like, we are in a remote, very poor region of Africa. Like most of these kids are only eating one meal a day. So it's like, they're not taking it on a full stomach. Like whether you told the parents that are not, they're not taking it on a full stomach. What are the side effects?

It's like us, it's like stomachache, nausea. It's nothing serious. But the kids start coming home from school and they say, oh, they gave us all these pills today. They gave us this normal pill, but they gave us this new pill and they measured our height and Tanzanians are used to their weight being used to determine doses of various anti-malarials and stuff. So measuring our height instead of our weight, they gave us this new pill instead of this old pill,

and also I have a stomach ache and maybe I've been vomiting, maybe I have a headache, I had these weird symptoms. So the parents obviously are like, what the fuck, like a lot of them have not heard of this at all. Yeah, totally. And another thing that happens is fairly shortly thereafter, a little girl dies. She dies of malaria, she dies in the hospital, she was not given the pills, but it starts to look to people like these two things are connected, right?

All of a sudden, these kids are taking these weird symptoms. Oh, they're getting weird symptoms and then this girl dies. people like these two things are connected, right? All of a sudden, these kids are taking these weird pills in school. They're getting weird symptoms, and then this girl dies. And so people on the local talk radio stations start talking about, you know, three or four kids have died because of the pills. Yeah. And so what starts happening is the parents

form these kind of mobs and start showing up at the schools. So this is an excerpt from a PhD dissertation that's written about this. It says, in some schools the situation turned violent as parents and guardians argued with teachers who had been administering drugs. In one primary school in Chimuino a classroom wall was pulled down, school property was destroyed, the national flag removed, rocks were thrown at the teachers and one was severely beaten. Holy shit!

The list of injuries suffered by teachers included a broken arm, a broken finger, and a serious head injury. In one school, a health worker had the drug stuff down her throat, causing her to choke and vomit. Holy hell! I know. The riot police were mobilized and dispersed angry crowds at a number of schools using tear gas.

Cheese. The Ministry of Health and Social Welfare officially suspended the program in Morricoro and throughout mainland Tanzania at the same day. So like, I wanna be very clear that like, there's a lot of really fucking ugly tropes about like, savage Africans and like Africans blunt like, there's nothing unique to African parents here.

We are talking about a program that is giving a medical treatment, a novel medical treatment to children without the knowledge or consent of their parents. People anywhere would have, I think, very rightfully freaked out about this. Right, and also white people in the US are losing their shit about their kids have to talk about race sometimes. Well, yeah, exactly. It is to me a deeply understandable reaction that like you find out from your kid that they've been given a new pill at school.

You don't know what that was. You don't feel like you were notified about it. And now your kid is like getting sick as a result of that pill, right? I don't know, man. This is every episode of Lawn Order SVU. If anybody ever tried that with my kid, they've had a hell to pay. You know what I mean?

All of that kind of shit, right? This is a way that we talk about parenting and it's a way that a lot of parents feel, which is do not fuck with my kids. Also, as soon as we start talking about whether a development intervention works or doesn't work, we're already in the wrong place. Something can work and not be rolled out well.

Yeah, it can work in one place and not work in another place based on like, really, really obvious, like social and political conditions. Yes. So like, one of the things that happens after this in Tanzania in 2008 is the ruling party starts blaming the opposition party and says oh there are the ones that started the rumors and then they use it as an excuse for a crackdown Yeah, so it's like you might want public health not to take place in a political environment

But it does it just does that's right. This is obviously like an extreme case I mean it's this is by far the worst case of like deworming being rolled out too fast that I was able to find in the world. But also like these kinds of problems happen pretty frequently with deworming programs because they're being rolled out so quickly. So one of the things that they don't really mention

in the follow-up literature to this 2004 paper in Kenya is that in a lot of schools, the teachers aren't distributing the pills. So you can do these nasty warming programs, you're just like, ah, 80 million kids or whatever. You give a thousand pills to the teachers, and they just don't give them to the kids. Because the teachers don't know why the fuck they're doing this.

They're not getting paid extra to do it. They're not giving the context for it. And then you can go back to your donors and be like, we distribute a thousand pills in this rural village, but it's like, did they get to the kids? There's various studies that that measure like the uptake, and some of them it's like 20%. Jesus. And like even in this, this drives me nuts. Even in the original 2004 study in Kenya, again, in the paper itself, the first they gave kids a couple rounds of pills. Like it was like every six months that gave them deworming.

The first time they gave the kids deworming, 70% of the kids, they were eligible, took it. By the fourth time, they're giving kids the pills, only 44% of the kids are taking them. And all of the stuff about costs, you know, the pills are only like 15 cents. So like you hear different numbers. I've heard two cents, I've heard 45 cents.

The original 2004 paper says it cost about a buck 50 per student. Once you add up that like we had to give it to them every six months and then all this distributional stuff, the low cost of this intervention is based almost entirely on like pharmaceutical companies giving it away for free. A lot of like volunteer labor to roll the sound. Teachers doing this with with no sensitization. All of the low cost justification for these programs depends on the fast, cheap, universal

rollout that caused the problems in Tanzania. As soon as you start slowing down to actually see if this is the right thing to do, the cost savings often disappear. Well, and also, like this feels like very sort of like life hack kind of adjacent, right? Yes. It feels like doing that, but with like entrenched public policy and public health issues. Right.

Like if someone came along and said that shit about like Brexit, you're like, oh, we got this one cool thing, you do it and then Brexit, we just done did it. Right. It's just all fixed. People would rightly be like, that doesn't, we got this one cool thing, you do it, and then Brexit, we just done did it. It's just all fixed. People would rightly be like, that doesn't seem right. Okay, so we've done the bunking, and we've done the debunking. And now we have to do a little bit of re-bunking.

What's getting re-bunked? So, okay, so that guardian article that you read that was like, deworming has been completely debunked and like, it's gone, right? Is kind of famous in the sort of worm war spheres for being a little overstated. Journalists only know how to tell two stories about international development.

It's either this is going to solve everything or this is a debunked fucking lie. We went through this with microcredit. Remember, it was like cover of Newsweek, microcredit, it's gonna solve poverty. And then they expanded it again too quickly. They expanded it to areas where it didn't work as well. And then it was like microcredits, a fucking scam.

Yeah. It's like, no, neither one of those are true. Neither one of those are true. And also, you only get those two stories if you keep buying into the first one that's like, this is gonna fix everything. Exactly. And also, I mean, my takeaway from this original 2004 paper

that has been dissected so much is that after all the sort of statistical reanalysis, the schools that got the deworming treatments have a roughly 6% higher attendance rate than the schools that didn't get dewarming. To me, that is a promising result. It's not bullshit, and it's not gonna save the world. It's promising.

It feels like a grounded result, too. Right. If you think about your life and sort of things that have made differences in your life, my guess is that it's a constellation of a lot of 6% things. Exactly. It's a kind of study to me that it's like, okay, it seemed to quote unquote work in 75 schools. Let's try it in another 75 schools. Yeah. It seemed to be fine in this region of Kenya. Let's try it next door in Uganda. Yeah. It never, been expanded to 140 million

kids. The thing that I wish that I could impress upon people in development aid is that literally nothing is going to be so good and so proven that you can go from 75 schools to fucking three million schools on another continent. Yeah. You're never gonna find it. I'll say in public policy advocacy world,

which is the world that I come from, there is such a deep scarcity mindset, right? That you start from a place of constraints, rather than starting from a place of like, okay, if we really wanted to get to the bottom of school attendance, what would that take? Instead, we start from a place of like, we found this one thing and that does it,

and we're just gonna get a shit ton of it out there at the end. And just hope for the best, right? That we get to the passage or non-passage of a policy, and we think that's the end of the story, and actually that's exactly when the rubber hits the road. Like, you have to pay just as much attention to implementation, the least exciting sounding thing.

I know, but like, that's where the promise of the policy comes to life, right? And in this one, it feels like folks just completely skipped over program evaluation and assessment stuff. Right, well, I mean, there are places where this works. There's a big study in Uganda that finds that it does increase kids' heights and weights,

and it does increase their test scores. Although it doesn't affect their school attendance, which I thought was really funny. So it's like the mirror image of the Kenya study, but it's like, okay, so it works on these metrics in this place. The, the, by far the most interesting study I read on this was, there's a meta-analysis of 153 D-Worming projects in Kenya. And the whole paper is about like, okay, what makes this work? Because every development intervention works in some places and not others. And it's not always obvious where it's going to work. Like you said, implementation, this is the hard part.

And what they find in Kenya is that the specifics of the implementation really matter. So basically every D-Worming project that only gives kids one dose of D-Worming pills doesn't work. Because the D-Worming pills, they kill all the worms that are inside of you, which is like napalms your entire digestive system, but it doesn't prevent you from getting reinfected. So if you only go there and give kids one round of pills, and you come back in six months and they all have worms again. And it really only works in areas

where there's like really high infection levels, because again, if you have like a light or moderate worm infection, like it sounds really gross to think about, but like it doesn't, it often doesn't cause any symptoms. Like the worm just like lives in your body for a year, and then it dies and you poop it out, and that's it.

So unless you're dealing with an area where there's like large numbers of heavy infections, like in the original study, it was like a 22% heavy infection rate. Like that's exactly the kind of place where you should be doing deworming. Like that's really debilitating for kids. But if it's in a place where like, you know,

10% of people have worm infections, only maybe 1% of people have heavy worm infections. You don't wanna give pills to every single kid for that, right? It's like giving every single kid a Tylenol for one kid having a headache. Yeah. Another thing they find in Kenya is that it works much better when it's coupled with other interventions.

So when you're giving kids like these deworming pills, even if they do have heavy infections, the worms are gone, but then they need extra food to kind of get, like, catch up growth, because oftentimes they've had, like, stunted growth and stunted weight. It's like they need to eat more for, like, a couple months after that,

like, catch up to where they should be. But if these are really poor kids, and they're in a really poor area, and they don't have access to food. Right. Good luck. Yeah, like, it's kind of like, there's no point in de-warming them,

because whatever benefits they would get from de-warming, they want to, they want to, because there's not enough food around for them to eat. Just like you also have to do these nutritional interventions. And in general, deworming works a lot better when it's coupled with this much more holistic program about like sanitation, like getting people running water.

Also some education programs are part of this. I'm skeptical of those in general, but apparently they work here in some circumstances. So it's like dewormming as part of a larger program of like we're gonna work on improved sanitation in this area, then like, yeah, there's a really essential component of that because you need to get the fucking worms out of the kids. But then, you're back to an expensive project. And you're back to a project where like you have to actually know the local area and you have to talk to people beforehand. And you have to build relationships.

And donors don't like that. And like the Ted Talk industrial complex Davos, they don't like that either. I mean, I think part of the reason that that stuff feels so frustrating to folks is that it feels so slow compared to, I got a million pills out. What I would argue is that building something slowly and right and making sure that it works actually probably takes less time than going on five of these wild goose chase scenarios. Jesus Christ. For everybody thinks we've found the silver bullet for whatever.

It's the like public policy and international aid equivalent of a get rich quick scheme. Oh, it is. I mean, it literally is for aid equivalent of a get rich quick scheme. Oh, it is. I mean, it literally is for the developing world, like get rich quick. Yes. Yes. Yes. The most chilling thing to me about this is the way that this generation of development aid has replicated exactly the mistakes of the previous generation. Boo! Say more about that. So when Michael Kramer got into development during this statistical dark ages, right? Everything was kind of like gut-based, vibe-based, right? And they've now created a situation

where they're basically saying like, we're just gonna ignore all of the data. The World Health Organization is still doing mass-deworming. The World Bank still stands behind this. The Copenhagen consensus. the Givwell charity, which is like a group that like I really respect, has written these weird defenses of the warming where they're like, no, no, no, no, ignore all the meta-analyses.

What we're really interested in is like the 10-year follow-up study. We know that the warming works because of of the 10 year follow-up study, but it's now, there's no plausible path to them having better lives. Like it's completely deranged to say that a program of deworming in which like up to half the kids didn't even participate, like didn't even get the deworming pills

and had no effect on health, no effect on test scores and an extremely modest effect on attendance would be affecting people's earnings 10 years later. I'm sorry. It isn't plausible to me, and also, it's very weird for this entire field of development. The whole thing was like the randomistas, that's what they called this generation of mostly economists

who came in to develop and were like, let's do this in a more rigorous way, that now these same brand of me says these like rigorous people are now telling me to ignore meta-analyses on the basis of a single study. Like that is so chilling to me. It's like, what was the point of all this? Yeah, I mean, imagine being like,

definitely don't pay attention to a fucking cock-run review. Right. This is, okay, this is, this is what Michael Kramer, like during the whole kind of warm war everybody fighting on the internet period in 2015. This is in a sort of response to the response, to the response to the response, like this whole thing is like, fucking tumblr comments in like 2014. He's not like defending D-Worming. He says, the estimated value of benefits exceeds the cost by more than a hundredfold. While the results of any one study should not be taken isolation and effects may differ across environments, even if policymakers believe there's a one-in-one hundred chance

of experiencing effects of this magnitude, the expected benefits of D-Worming would exceed the costs. They're playing our song, Mike. He's basically saying, like, even if this doesn't work, look how cheap it is. Look how cost-effective this is. This is like the asbestos argument. Give-well says the same thing.

Give-well says, because Massey Worming is so cheap, there's a good case for donating to support D-Worming, even when in substantial doubt about the evidence. This is very similar to some of the responses to the calorie labeling stuff. Oh yeah. Yeah, absolutely. That people are like,

well, it doesn't work for everybody, but look how cheap and effective. And you're like, effective measured by what? Right, effective how? We're doing something, we're not doing nothing, but also the something that you're doing has the effect of nothing. And also plus maybe negatives.

Like, what? I also, I forgot to mention, Michael Kramer won the Nobel Prize in 2019. Come on, Michael! It's not just for this, it's for like other work, and it's for like this whole kind of the systematization of statistics within international development. And he shared it with some other economists, and like, I actually actually think that using more randomized

control trials in development is a good thing. I think statistical rigor is a good thing, but also that has come with drawbacks and side effects and a lot of weird and bad cul-de-sacs. I also think to sort of defend the original 2004 paper, there's not going to be a perfect methodology. Sure. You can find things to quibble with in anything,

in like a meta-analysis and any individual study, and ultimately, every single thing that we do in public health is going to be a judgment call. In principle, that's not necessarily a bad thing. I think the idea of a judgment-free science is always going to be a myth. You're always going to have to have values. Yeah, and if you don't get explicit about articulating and committing to your own set of values, then values will show up in your work.

Yeah, exactly. Yeah, that's a good way to put it. Without your intending them, exactly. Yeah, that's a good way to put it. Without your intending them to, right? So if we're thinking about, for example, pretty much all of the weight loss research that we have out in the world starts from a place of assuming that fat people's experiences are immaterial. Right.

If I ask some of those researchers, like, hey, do you think that like the life experiences of people who are trying the things that you're testing out matter, I imagine quite a few of them would be like, well, yes, right? But they haven't designed a thing that reflects that. Right. And like, that's part of what we're getting into here is this is research that was not designed in a way that reflected an understanding that there was knowledge on the ground that they needed to know about. There are a bunch of things that again,

if you don't articulate those values, and then take the next step of going, and here's how we're gonna operationalize them. Other values will speak on your behalf. I think the parallel to fact people is really good because it's driven by the same thing. Like this is for their own good and like I'm helping. It's driven by paternalism. Yeah. And I think people give themselves permission to like not think things through. To not

think things through and to not ask for feedback. Right. That's the like toxic good guy narrative that worries me. Yeah me so much in international development and in public health and in lots and lots and lots of places. So that was, that was you warming. Is that what we're gonna end it? Just me being like, everything's fucked and we're all done and we all lost it and you're like, there we go.

I feel like, okay, so you, this is kind of a digression. You posted on the maintenance phase Instagram, the other day, it was like a meme about how every millennial podcast is just one person talking excitedly and the other person going, that's fucked up every five minutes. I was like, wow, they've really got our number. I think we both have the same reaction of like,

owned, fair enough. When you're right, you're right. I was trying to structure something around like, how can I get Aubrey to say like, that's fucked up. Fucked up, it's a millennial podcast, that's fucked up. We did it, we got that eh? ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ‍ʻ� you