The a16z Show - a16z Podcast: A Guide to Making Data-Based Decisions in Health, Parenting... and Life

Episode Date: May 11, 2019

with Emily Oster (@ProfEmilyOster) and Hanne Tidnam (@omnivorousread) Are chia seeds actually that good for you? Will Vitamin E keep you healthy? Will breastfeeding babies make them smarter? There’s maybe no other arena where understanding what the evidence truly tells us is harder than in health… and parenting. And yet we make decisions based on what we hear about in studies like the ones listed above every day. In this episode, Brown University economics professor Emily Oster, author of Expecting Better and the recently released book Cribsheet: A Data-driven Guide to Better, More Relaxed Parenting, from Birth to Preschool, in conversation with Hanne Tidnam, dives into what lies beneath those studies... and how to make smarter decisions based on them (or not). Oster walks us through the science and the data behind the studies we hear about -- especially those hot-button parenting issues that are murkiest of all, from screen time to sleep training. How we can tell what’s real and what’s not? Oster shows us the research about how these guidelines and advice that we are "supposed" to follow get formalized and accepted inside and outside of healthcare settings -- from obstetrics practices to pediatrics to diet and lifestyle; how they can (or can’t) be changed; and finally, how the course of science itself can be influenced by how these studies are done. Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Hi and welcome to the A16Z podcast. I'm Hannah. Good data, bad data. There's maybe no other area where understanding what the evidence actually tells us is harder than in health and parenting. In this episode, economics professor Emily Oster, author of Expecting Better and the recently released crib sheet, a data-driven guide to better, more relaxed parenting, does just that, looking at the science and the data behind the studies we hear about and make decisions based on in those worlds, from whether to breastfeed your child to screen time to sleep training. We talk about what it means to make data-based decisions in these settings, in diet and in health and in life, like whether chia seeds are actually good for you and how we can tell what's real and what's not. We also talk about how

guidelines and advice like this gets formalized and accepted for better or for worse, and how they can or can't be changed. And finally, how the course of science itself can be changed by how these studies are done. You describe yourself as teasing out causality in health economics. Can you give us a little primer on what exactly that means and how you start going about doing that? So there are a lot of settings in health. And in all of those settings, we have to figure out what does the evidence say? And I think about some of them in this context of parenting, but you can think about even questions like, you know, is it a good idea to eat eggs or is it a good idea to take vitamins, other kinds of health decisions? And you can sort of think about there being kind of two types of data you could bring to that. One

be randomized data so you could run a randomized trial in which half of the people got eggs and half of the people didn't and you followed them for 50 years and you saw which of them died. And that would be very compelling and convincing. And when we have data like that, it's really great. I mean, I kind of think of that as being the default, no? Is that not at all the standard? That is the gold standard. Yeah. But it is not the default. So many of the kinds of recommendations that I look at in parenting, but that you look at in general and health are based on observational data, which the other kind, where we compare people who do one thing to people who do another thing and we look at their outcomes.

And one of the ways in which the people differ is on the thing that you're studying, but of course there are other ways that they may differ also. A million other ways. A million other ways. Yes. And data like that is really subject to these kind of biases that the kind of people who make one choice are different from the kind of people who make another choice. One of the things that's very frustrating in a lot of the health literature is that there isn't

always that much effort to improve the conclusions that we draw from those kind of data. And we're drawing, we're using that kind of approach because of the inability to have long, longitudinal studies, or does it tend to be a shortcut? So I think it is, it is both things. So it is much easier, faster to write papers to produce research about that. And it can be really useful for developing hypotheses. So it's like a scratch pad almost. In the best case scenario, be like a scratch pad. Like, let's just look in, let's look in the data and see what kinds of things are associated with good health or associated with good outcomes for kids. And then we could imagine a next step where you would analyze more rigorous gold standard. And sometimes that happens. So there's

one really nice example of the book where this happens exactly like you would hope, which is in studying the impact of peanut exposure on peanut allergies. Right. So the first paper on that is written by a guy. And what he did was he just compared Jewish kids in the U.S. or sorry, Jewish kids in the U.K. to Jewish kids in Israel. And he saw that the kids in Israel were less likely to be allergic to peanuts. And he said that's because they eat this peanut snack when they're being. Bombas, right? Yeah. The bomba. And so, so then that's, that's like the hypothesis generation. And then he went and did the thing you would really like would just say, okay, let's run a randomized trial and let's randomly give some kids early peanuts and some kids not. And indeed, like,

he found that he was, he was right. So that's like a great example of like how you would hope that literature would evolve. But in many of the kinds of health settings we're interested in that you can't do that or it is much harder to do that because the outcomes would take a long time to realize or it's expensive or it's hard to manipulate what people are doing. And then we often end up relying on these more biased sources of data to draw our conclusions, not just as a scratch pad. And I think that's where we encounter problems. That's where it gets murky and we never know whether we should eat eggs or not. Yeah. And that's exactly the area that you tend to focus on. Yeah, Exactly. I try to first see, are there good pieces of data that we can, that we can use? And then if we're

stuck with the data that isn't good, trying to figure out which of the murky studies are better than others. And, you know, what would you mean by better? Well, you know, how it's roughly like how good is this study at controlling or adjusting for the differences across people. So you talk about kind of breaking it down into both into the relationship between data and preference. How do you factor in that in the healthcare system where it's so diverse, where preference has such an incredible effect and puts you into so many different possibilities. I think this is why in these spaces decision making should be so personal. We often run up in health and also in parenting in all of these spaces into a place where we're telling people like there's a right thing. There's a right

thing to do. And I think that that can be problematic because it doesn't recognize this, this difference and preferences across people. You have to basically accept the variety in the system and then give a space for preference in the decision making. Right. Yeah, but I think it's exactly these preferences that, of course, make it hard to learn about these relationships and the data. Because once you recognize that a lot of the reason that some people choose to eat eggs

and some people choose to eat Cocoa Krispies is that some people really like Cocoa Krispies, some people really like eggs, how can you ever learn about the impact of eggs? Because we know there must be differences across people. And I think that that becomes even more extreme when we think about really important decisions that people are making, like the kinds of choices they make in parenting, or also in their diets. So can you walk us through one example like that of where it was a really kind of murky, gray area and how you pull out the causality? The best example of this in the data in the parenting space is probably in breastfeeding. Let's say you want to know the impact of breastfeeding on

obesity in kids. That's a thing which you hear a lot. Breastfeeding is a way to make your kid skinny and so on. And so the basic way you might analyze that is to compare kids who are breastfed to kids who are not and look at their obesity when they're, say, seven or eight. And indeed, if you do that, you will find that the kids who are breastfed are less likely to be obese than the kids who are not. But you will also find that there's all kinds of relationships between obesity and income and obesity, mothers income and mother's education and other things about the family. And those things correlate with breastfeeding and they also correlate with obesity.

So you can't really pull apart this web. So it's hard to pull apart the web. So I would say this is an example where like the data is suggested. Like it would certainly be consistent with an effect of breastfeeding on obesity, but I think it doesn't prove an effect. And then you can sort of take the next step and say, okay, well, do we have any data that's better? And in that example, you know, we do have one kind of randomized data. But again, we run up against the limits of all kinds of evidence. So the randomized data on this question is from a randomized trial that was run in Belarus in the 1990s.

Oh. They randomly encourage some moms to breastfeed and some moms not. And so there's a lot of good things that we can learn from that. But such a specific place in time. Exactly. It's so specific. And you said, like, well, you know, how do I take that result to, you know, the Bay Area in 2019? Right.

It's a challenge. Okay. Well, was there any other within this space of, you know, not randomized data is or anything that's better? And in that case, there is, you know, there are some studies that, like, compare siblings. Oh, okay. Where you look at two kids, you know, born to the same siblings, born to the same mom. One of whom was breastfed and one of whom was not.

and then look at their obesity rates and when you do that, you find there's basically no impact. So then you're kind of holding constant, like who's the mom? So if you're a worry was that there are differences across parents in their choices to breastfeed, well, now you're looking at the same parent.

Right. You're normalizing. And so you may think, oh, that's great. Perfect. I'm totally done. But of course, you're not, this isn't perfect because why did the mom choose to breastfeed one kid and not the other?

People are not choosing randomly. You had a C-section one time. You didn't another time. If that were the reason, that would be great. Right. If the reason we're just like kind of worked one time, it didn't work the other time, you know, if there were something that was effectively a little bit random, then that would be exactly which that would be exactly the kind of variation you'd want to use. Okay. But the thing you worry about is like one kid is not doing well as unhealthy. So the mom chooses not to breastfeed or chooses to breastfeed to try to make them healthier. Those are the kind of things where there's some other reason that they're choosing differences in breastfeeding, which has its own effect on on the kid's outcomes. So you kind of like, Some of what I try to do in the book is sort of like put all of these pieces together and kind of like look at them and think about them all as a as a sort of totality of evidence and just think like how compelling is this altogether. It sounds almost like sifting like using a sifter. You take all this very murky data, very variable from all sorts of different contexts and like put it through the sifter of like this kind of data, this kind of data and then match it all up and say, okay, what do we have left?

And then therefore and then hand that over and say it. you make the decision based on this. Based on this. Right. Here's kind of what we can be more or less or less sure about. You talk a little bit about the idea of constrained optimization as being very important. Can you explain what that means and how that plays out? In economics, we think about people optimizing their utility function.

The idea is that you have a bunch of things that make you happy. That's your utility. They produce your utility. And you want to make the choices that are going to optimize your utility, that you're going to give you the most amount of happiness point. Udels, called youdles. It's a very warm and fuzzy discipline. I'm going to go home and use that with my kids.

Absolutely. You gave me some utils today. But we also recognize that people have constraints. In the absence of constraints, like having money to buy things or time to do them, people would just have an infinite amount of stuff. That's the thing that would make them the most happy. And so, but when you're actually making choices, you're constrained by either money or time. And in the book, I talk a lot about this in the context of time that, you know, you're,

you're as a parent, you're making your choices and you have some preferences and some things you would like to do, but you are also facing some constraints. But is there, is information flow kind of what, and the data itself, a constraint in that regard? Is that a, because it's so piecemeal the information you get. That feels almost totally random. Like some media story picks up on something, you tend, you know, some tidbit you hear unless you're like systemically studying a graduate seminar on parenting, which none of us do,

you know, then it is random. Yeah. Yeah. And I think we wouldn't necessarily think of that as in constraints because, of course, in our models, people are fully informed about everything all the time. Right. Right. That's one of the great things about the models.

But in real life? But in real life. Yeah. I think people face constraints associated with just not having all the information. And, you know, also the fact that this, these kind of information like whipsaws over time. that, you know, you get one piece and then and then you kind of, the next day, there's a, there's a different piece of information and we have a tendency to kind of glom on to whatever is the most recent thing that we have seen about this as opposed to what is the whole literature

over this whole period of time say. Right. You say you have a great quote where you say in confronting the questions here, we also have to confront the limits of the data and the limits of all data. There's no perfect studies, so there will always be some uncertainty about conclusions. the only data we have will be problematic. There will be a single not very good study, and all we can say is that this study doesn't support a relationship.

So it feels kind of hopeless. I loved when you talked about the first three days of when you brought Penelope home, and it really brought that back for me as I was just this dark room that you're kind of alone making these decisions. How do you even begin to see this data, you know, as a decision-making practice?

Like, how does that translate? there are pieces where it's where it's easier where the data is better and makes it is clearer about what you need to do or what the choices are. You will be making many choices without the benefit of evidence or data or very good data. I think part of what makes some of this parenting so hard is that for those of us who like, you know, evidence and facts and it's hard to accept. I'm just going to have to make this decision basically based on what I think is a good idea. Based on my gut. Based on my gut. And, you know, maybe based on my mom.

Yeah. And, you know, and I... Which is a sample size of one. Sample size of one. And, you know, maybe if you have like a mother-in-law, father-in-law, it's like a sample size of two. But that's kind of it. And I think that that's really scary,

especially when the choices seem so important. Yeah. I mean, but it feels like, you know, that's kind of at heart what you're trying to do, right? Is like to translate and to give tools in this decision-making place. So how would you begin to... to systematize that? I mean, is there a way to bridge that gap better in the system? I think that it would be helpful if more information was shared. So I think a lot of these things, there is a lot of

information that is contained in people's experiences that we are not using in our evidence production. So in the book, I talk about like the sleep schedule. Right. So you're sort of told as a parent, like, oh, you know, this is kind of roughly like around six weeks. Your kid will start sleeping like longer at night. But there's no, the information that's sort of typically conveyed to people is, is not, is not a range. It's just like around six, six weeks-ish. Yeah, it'll start to happen. But the truth is like, yeah, that's, that's kind of right. But it's, it's, if you look at data on when that actually happens, it's pretty, it's a pretty wide range. And I think part of what is so stressful about this early, these, like, the

early parts of parenting are that it's very hard to understand whether what you are experiencing is like normal. And I think if you could understand like, yeah, most kids don't do this thing at this time or most parents have this experience or the way the graph plots kind of a little more broadly. Exactly. I think that would be that would be super helpful. And that's a place where I can imagine, you know, data collection helping, right? That, you know, we have much more of an ability at this point to to like get information about what is happening with our kids, what's happening with, you know, with our health. There is a sense in which that could be helpful in in just setting some norms for the normal, the standard variation across people. So looking at the variation and

providing that as like a piece of the information. Here's also the variation on that. Yeah. Yeah. Yeah. And I think that's that is kind of part of like generating the uncertainty and sort of showing people like what are the limits of the data. Yeah. That how sure are you that this should happen? at this time. Not just how sure, but like what are some of the like other ends of the curve? I mean, that's just information you just don't get. Yeah. Right. So let's zoom out a little bit as somebody who lives in the world deeply of data in the health system. We're in a time of enormous shift, right, for data. Does the improvement, does our kind of the sea of data and like better data, cleaner data, more granular data, all that help this at all this question? Yeah. And, you know,

I think we are collecting a. so much data on people, both sort of individual people are collecting a lot of data about themselves. Health systems are collecting a lot of data about people. This data is like underutilized, I think. We're amassing pools of it, but not in ways that are especially helpful. So, you know, in my, when I go to conferences and people who work on healthcare, like there's a tremendous amount of data that's being used on health claims. Right. So if you sort of think about like what are some kinds of data that we have, we have like health claims data, like payments, everything that where there's an individual payment for it,

we'll like see, we'll see it. There's almost no work with medical records. Even though every hospital, everybody's using Epic, you would think that that would make it straightforward to have that data in a usable form. But it's not. And but, you know, at the same time,

the potential for sort of going beyond, like, here is all the tests that you ordered into actually like what happened with those tests and then what happened to this person later, like that data is not being mined. in the way that we could to try to look at some, you know, at some of the kinds of outcomes that are, that are a result. The causality that you would pull out afterwards.

Yeah, absolutely. You know, how can we improve our causality? More data is helpful. More information about people is helpful. Being able to look at, you know, the timing relationship between some treatment and some outcome. Those are all the kinds of things that, you know, having better data would help us, would help us do. Are there other areas where you start, you are starting to see the data coalescing? in a way where you're able to pull meaningful insights from it?

So I think, yes, when we have better data, we can use better tools even if we don't have randomization. A classic example in health is looking at the impacts of like a really advanced neonatal care. Like how cost effective is it to have like, you know, kids in sort of getting like really extensive NICU care. Right. Like how, like how effective is that in terms of improving survival and how much does it cost? Oh, such a basic question.

Such a basic question and super hard to imagine analyzing because, of course, you know, babies that are very small and are sick cost more but also have worse outcomes. Yeah. And so if you sort of looked at that, you would be like, well, actually spending more, we're not getting anything because those babies are more likely to die than babies that are spending less. We define very low birth weight babies as less than 1,500 grams. Okay. Which means that the treatment that you get if you're a baby at 1,503 grams, is very different than the treatment that you get as a baby at 14.000. 1,197 grams, which is completely arbitrary. Yeah.

I mean, the choice of 1,500 grams has nothing to do with science. It's like this line in the sand. That's not a good way to set policy. However, having set the policy like that, you can then say, okay, well, now we have some babies that are almost exactly the same, but the babies that are a little bit lighter that are like 1,497 grams, get all kinds of additional interventions relative to the babies that are 1,503 grams. And when people have done that, they see actually the babies at 1,497 grams do better. So the line actually is beneficial in that way because you're defining these two groups very closely.

So the line that's setting this line in this arbitrary way lets you get at some causality. Even though not good for the babies. So having done it, good for good for research. Good for information. Yeah. Interesting. What are some of the other tools? Are there others in that list?

So that's that is an example called regression discontinuity that there's some discontinuous change in policy on either side of a cutoff. And that has become a sort of part of a. big toolkit of things people are using more of the other is to look at sort of sharp changes in policies at a time at like a moment in time. Oh, so the same thing. The same thing, yes. Then there's that. And then there's, you know, looking across when different policies change differently for

different groups. So all of these things have become easier with more data and become more possible with more data. And I think that that has improved our inference in some, in some of these settings. I love that you talked a little bit about the experience. of doing your own data collection kind of in the wild with this spreadsheet after Penelope was born, which made me laugh so much because it was so much like my spreadsheet. It was just so sad to think of like all these moms alone in their bedrooms at night.

I know. I like Google Drive. I mean, I think there's been a lot more apps since like since that. We had that help. But still, how, I love that you said it gives the illusion of control, not control. And in that particular, in these kinds of like data vacuums, like if we're not good at statistical analysis or like pulling out causality from these murky areas, like if we're not Emily Oster basically, how do you like, or even if you are, how do you kind of stay on that line of like the illusion of control versus like actual knowledge that like impacts real decision making? No, I think it's super hard because the thing is the illusion of control is a very powerful illusion. Very.

And both empowering and dangerous in health, in health context. Exactly. Like we would, you know, you would sort of, we'd like people to feel like they're in, they're in control. Some of the message of this book, I think people have taken, not quite right, but to say, like, well, it doesn't really matter what choices you make. Like, all choices are good choices. Wow.

I think there's, that's not quite the right. It's not quite the message. I'm surprised that that's the message that people take from that. Occasionally. There are a lot of different good choices that you could make about parenting. And so I think that there, there, there, is a piece that like we should we maybe don't need to be so like obsessive about all of these about one of those about one of those any one of those of those choices point about range it's like well let's give let's

educate a little bit more about like the spectrum of the spectrum of good yeah of good choices yeah another area I feel like where every other day there's a new study that says something different and it feels like there's a plethora of studies is screen time I'm just I'm going to put that out there right now. I'm sorry. Everybody, we're going to touch that third rail. So can you walk us through, like, can you help guide us through some of that maze? So when I looked into screen time, I had always thought about like screen time is like bad. Like the question is, is it bad or not? Right. But actually there's like a whole other side of this, which is some people like screen time is the way to make your kid smart. Like you can, like your baby can learn from them. Okay. So

point number one is what does screen time actually mean? Right. Which is a bunch of different stuff. Yeah. And I think that's part of the, like, that's part of the problem with this is like when you say screen time, like, what do you mean? Yeah. Do you mean like, you know, educational apps? Yeah. Do you mean, uh, do you mean timing with grandma? Do you mean jumping the shower? Yeah. Right. Exactly. And or yeah, and that point like, do you, like, while you jump in the shower. Yeah. Like, what is the other thing you're going to be doing with your time? Yeah. I think this is where all of these recommendations seem to assume that the alternative use of your, of, like, if your kid wasn't watching Sesame Street, you would be like on the floor, like playing puzzles with them. And. And. And. And. And. And. And. And. And. And. And. Like, like, and super engaged with them, which, like, maybe is true. Taking them to the zoo and, like, having them touch different textures of animal skins or whatever, like, sensory development, yeah. Yeah, which, like, is great stuff that you should definitely do with your kid. But some of the time when, you know, when our kids are watching TV, it's because, like, it's, it's, that maybe isn't the thing that you would otherwise be doing.

Yeah. Well, you could be, like, purring healthy vegetables to, like, feed them well. Yeah, exactly. I'm sure that's what we're all doing. It's definitely for sure. Or maybe watching. a little reality TV for five minutes while you fold laundry. A little bit of call the midwife, you know, got a little bit of, yeah.

The problem with screen time is that the evidence is very poor. Can you just break up like why the evidence is so poor? Because this does seem like an area where there should have been time for that kind of gold standard randomized study to develop. No. What is the evidence problem? So I think the evidence problem is twofold. One, it's actually not a super easy thing to run a randomized trial on because these are choices that people are, thinking a lot about. And, you know, think about something like an iPad, like, do you want to be

involved in a randomized trial of whether you're a kid? Oh, there's too much intention, too much at stake. Too much lifestyle. Exactly. Too much like lifestyle stuff. Some people have been able to use, like, the introduction of TV, which was sort of had some random features to like look at the impacts of TV. And that evidence is sort of reassuring and suggested TV is okay. But of course, it's very old. It's like from the 50s. A whole different way of consuming. Exactly. Everything. Yeah. And I think the other thing is the other problem with the sort of current answering the current questions people want like what about iPads

what about apps you know is that they just haven't been around long enough and so a lot of the kinds of outcomes you would want to know that even things like short run like test scores you know I got the first iPad when my daughter was born like that was like one and I remember getting it

being like this is never going to catch on this is why I'm not in deck I was like who would use this meanwhile your daughter's like swiping meanwhile you're like okay but you know now she's she's in second grade, like that's kind of the earliest that you could kind of imagine getting some kind of what we'd measure test scores or something like that. But even, you know, she didn't

use the iPad anywhere near as near as like Fasel away as my four-year-old, right? This is evolving so quickly that any kind of even slightly longer term outcomes are really hard to imagine measuring, let alone, you know, absent a randomized trial, the, if you, like if you weren't able to randomize this, which I think you won't be able to, the, the, the, the, the, the, the, you, amount of time kids spend on these screens is really wrapped up with other features of their of their household yeah okay so you have the definitions you have the time and the speed at which things are changing and then you have the willingness for people to actually like engage in change or doing things differently and then so all of that leads to what kind of so what the

what do the studies actually tend to look like in this space that we draw conclusions from so actually there's almost nothing about ipads or phones That seems so contrary to like what the media is saying every five minutes. Yeah. So there's tons of studies on TV, which compare kids who watch more and less TV. And, you know, you can, but most of that, again, is sort of studies that are like based on data where before people were watching TV on these screens. Yeah.

Maybe TV is TV and, you know, you can imagine that that would be kind of similar. But things like these absolutely, just like no studies. You know, or there'll be, there's like, I think there's one, like, abstract from a conference. This is not a paper. There's like an abjury in comments where it's just like we have some kids and we like compare the kids who like spend like the babies who spend more time watching their parents' phones and then they like do worse. They're like,

but it's like, it's not. It's pathetic. It's sad. This is a terrible piece of evidence. So is this an area in which you just go with your gut? I mean, I try to generate a fancy version of go with your gut, which is called Bayesian updating.

And so I basically try to say, look, you know, I mean, we want to step back and think about what are the places of uncertainty. Logic would tell you, you know, your kid is awake for what it is like 13 hours a day, 12 hours a day. If your two-year-old is spending seven of those 12 hours playing on the iPad, then there's a lot of things that they are not doing. Right. That's probably not good. Right. On the other hand, you know, if your kid is spending 20 minutes every three days, it's very hard to imagine how that could be bad. So just thinking about it purely in times of like time allotted to any one activity, basically. And then I think once you do that, then you're sort of like, okay, but, you know, there are

things that we're uncertain about, you know, what if my kid watches an hour of TV every day or spends an hour on a screen every day? Like, is that too much? Is the limit? If we sort of accept, like, five minutes a day is fine. Seven hours a day is too much. Like, is the limit at an hour, is a limit at two hours, you know? And I think the truth is what we will find if we end up in doing any studies like this is that it depends a lot what other things they would be doing Yeah. With their time. Wouldn't it also depend so much on the child?

Some children need, you know, learn in a kind of way that lends itself to this technology. Some children need other kinds of learning, you know, that it's highly individual. Yeah. I mean, I think this gets into the problem with studying older kids in general that just like there's so much, there's so many differences across kids. It's hard to even think about how you would structure a study to learn about them, never mind actually using evidence it exists. It's really interesting because the last time we went to take. my daughter for her annual checkup or maybe it was my son. I can't even remember it. It's so different from the first days of those early spread. She were not like, um, did I even get her? Which one is that?

Yeah, exactly. Anyways, the doctor said very concretely, two hours, two hours max within any day. But it was really interesting to me that it was such a specific line in the sand. And now I'm thinking about how that information would even get into that to percolate down to that level of like the system and get kind of fossilized into the system so that that that recommendation is being passed on to parents. Like, how does that happen with these studies? How do they translate to that level of advice? Yeah.

I mean, I think what happens, the, what happens is, like, organizations like the American Academy of Pediatrics, they bring people together to basically talk about the conversation we just had, which was like, well, it's fine, okay, let's agree, sort of like, we don't know that much about this. Like, five minutes seems fine. Seven hours is too much. These are, like, smart people who see kids a lot, who presumably are using some knowledge that they have about kids to pick some number.

But the answer is like you could pick a lot of different numbers. We sort of say this and then it becomes like this rule. Yeah. And people have some impression that it comes from some piece of evidence as opposed to sort of like, you know, a some gold standard study. Some gold standard study. Of expert opinion or something. Yeah. Which is really what it's what it's from.

You also work specifically on certain health recommendations. So how they change over time and how we stick to them. you wrote a paper on behavioral feedback. And then you talk about how those individual choices might in fact be changing the science itself. Can you talk about what that means and how that might be happening? I was thinking about exactly this issue of like, okay, we just make some recommendation and sometimes those recommendations are kind of arbitrary. But then they go out in the world and like people take on lives of their own. Exactly. And so like a sort of a good example. Vitamin E supplements like in the early 90s, there were a couple of studies which suggested

that like they are good for your health that like prevent cancer. Okay. And so then there was like a recommendation like people should take vitamin E. And you know, then we have to ask a question like what like who takes vitamin E after that. And one of the concerns is that the kind of people who would adopt these new recommendations, like who, who listens to their doctor. It is people who are, you know, probably maybe they're more educated, maybe they're richer. But but like above all they are interested in in their health.

right? So they are taking vitamin E so they avoid cancer, but they're also exercising so they avoid cancer. And they're eating vegetables so they avoid cancer. We just call them selected. Okay. These people are like positively selected on other health things. And so indeed, you can see in the data that the people who start taking vitamin E after this recommendation changes are kind of also exercising and not smoking and doing all kinds of other stuff. Why is that like, you know, interesting or problematic? Well, later we're going to go back to the data. because that's like the way science works. But now the people who take vitamin E are even more different than they were before.

Right. So now these people are like... So you've added another layer. You've added another layer. So in fact, you can see that in the data. You can see that basically before these recommendations changed, there was sort of a small relationship between taking vitamin E and like subsequent mortality rates. Okay.

But after the recommendations change, you see like a very large, very large relationship between vitamin E and mortality rates. And so it looks like, basically ends up looking like vitamin E like is really great for you. Has this big impact. But of course that's because at least it seems like it must be, at least in part, because the people who adopt vitamin E are the people who are also doing these other things. So what does that mean then? It feels like such a loss.

Like how does one ever, yes. How would one ever develop like a recommendation based on what we think we know? I know. And untangle it from like, so this paper is very, it's just very destructive in some it's sort of like, other than saying like it probably doesn't matter if you take vitamin E. So that's like that's like news you can use. You can take that home with you. But I mean, I think it does, it does, yeah, more or less just highlight some of the the inherent and very deep limitations with our ability to learn about some of these, some of these effects, particularly when they're small. Is this basically part of the sort of crisis of reproducibility?

I think it's, I think it's not unrelated. Yeah. So I often think about this idea of P hacking, which refers to the idea that you like keep, you can like, you keep running your studies until you like get a significant result. There are a bunch of, like, there's a bunch of people interested in this process of like how science, how like science evolves. And the ways in which the like the evolution of science influences the science itself. Yeah. incentives for research influence how how science works. And I think it's particularly hard to draw conclusions in these spaces like diet or these health behaviors where the honest truth is probably a lot of these effects are very small. So if you ask the question, like, what is the effect of like chia seeds on your health? My dad is like really into chia seeds. Yeah. That was a thing. There was a moment. Yeah. Well, he's still in that moment. He's still in there. And, you know, like, what is the effect of those on your health. The actual effect is probably about zero. Maybe it's not exactly zero, but it's almost

certainly about zero. But are there sometimes secret sleeper, like, whoa, there actually might, you know, the only way to find out is to do these things. Yeah. Yeah. And so maybe there are some secret sleeper. Like maybe Kail really is magic. Maybe it is, but it's probably not. Okay. You know, I don't, I, you know, I spent a lot of time with these diet data and, you know, there's this sort of like dietary patterns, like the Mediterranean diet, which do seem to have some sort of vague, you know, support in the in the evidence. But I would be extremely surprised if we ever turn up like a one single thing. One magical food. So the point is it's the pattern. It's the pattern. And it's all the other things that you're doing. If you smoke three packs a day and you never exercise,

but you eat some kale, that's not going to help you. Yeah. Yeah. The kale's not going to help. What about when you really do need to affect change? What are the ways in which these guidelines can shift over time with kind of new sources of information or data and statistics. Like, what's the positive? How does that actually play out in the right manner? Yeah. So I think there are times in which the change in evidence is so big and so, like, compelling that we can get changes, best practices in obstetrics.

Like, how do you deliver reach baby? As an example, they change, like those changed over time because there was like one very big, a well-recognized study that everybody agreed, like, this is now the state of the art. And it happens fast at that point. And then it happens pretty fast. It doesn't happen immediately. Like, you might have thought that those kind of changes could be, like, immediately affected. And I think that they're not.

But they do happen over time. Those examples really rely on there being, like, a cohort of sort of, like, experts who are all reading the guidelines and sort of seeing that they changed. And then themselves are sort of doing that. this all the time. I think part of what's hard in the in the health, in the broader health behavior space, where it's people who need to make the choices, not, you know, not physicians. Yes. When it's in the home, in those dark bedrooms. Right. It's like that, you know, that's much, it's much harder to get people to change their behavior in those spaces. It's not these pediatric

guidelines. Those are not effective. Yeah, I do not think those are effective. Or I think we don't see any evidence in the data that those are effective at moving these, at least these in these kind of spaces. So what's the answer? How do we positively affect change and gather these insights? And and have smart people making good recommendations. So, I mean, I think one answer is media attention. They kind of few times when we see very large spikes and changes, they actually seem to correspond with some media coverage. On the flip side, like media can often be very bad.

Some of these big changes in these expert things were kind of resulted from media coverage, which was really like sensationalist and like totally inappropriate. Yeah. And, you know, it wasn't like a very nice like New York Times story about some study. it was like a like a sensationalist 2020. About what? This was, this is about vacuum extraction,

which is a way of pulling the baby out and has gone down a lot over time. And it was like this sort of sensationalist like John Stossel 2020 episode about how it could hurt your baby, which caused like big reductions. Interesting. Yeah. But you know that like,

but the science was there. The science. Yeah, was there. I mean, he overstated the science. Yeah. But it was, it was probably there. So it's almost like a random confluence of like when the science.

science is there and the media hits it the right way and then we see change? Yeah. Which is something to hope for it. Yeah. That doesn't feel like we can plan so much for that. No. You also study when we are resistant to change. You looked specifically at diabetes, people I think who had been diagnosed with diabetes and then whether or not their behavior changed even given a certain amount of information. So what do you see there about our resistance to change even with the right kinds of information? Yeah. I mean, I think one of the big, challenges in the in the health space at the moment is that like so much of the so many of the health problems that we have in in the u.s are like problems associated with behavior just the fundamental

fact that like people do not eat great and we have a lot of morbidity and expense associated with with that and and i think there is often a lot of emphasis on the idea like if we just get the information out if people just understood vegetables were good for them he would eat vegetables doesn't happen that's not true yeah i think uh and so you know this paper is about sort of looking at something where kind of a pretty extreme thing happens to people. Like they are diagnosed with diabetes and we can see what happens to their diet. And the answer is, you know, it improves a tiny amount. Even with a real come to Jesus moment.

Exactly. And a lot of new information. Right. Like, you know, and monitoring, follow up, right? I mean, you're diagnosed with diabetes. Like, you know, you have to take medicine every day. You've got to go to the doctor. I get to test your, you know, test your insulin, at least for some period of time.

So this isn't like something where you can just forget that it happened. And even then the changes. and diet, you know, they're there, but they're really small. They're like, you know, one, like one less soda a week or something. Like, really, like really small. And how are you noticing these? We're inferring information on diagnosis from people's purchases of testing products and then following their grocery purchases. So this is like an example of using, you know, a different kind of data. Yeah. So not health data in this case. It's actually like like Nielsen data. So Nielsen data on what people buy. But then, you know, using some like, like,

machine learning techniques to try to figure out from the kinds of things people buy, when were they diagnosed with diabetes, and then looking at their diets over time. So is the answer that there has to be some sensational story that talks about? I mean, I think there I'm not even sure that would help. I think part of the problem is people really like the diets that they're comfortable with. Like diet is like such a habit formation thing. And, you know, people are willing to make important health sacrifices to maintain the diet that they, that they like. we get into some of these questions of preferences.

Like, and, you know, if people, if that is the choice that people want to make, like, should we be trying to intervene with policy? Like, let's say everybody had all the information. They knew that they shouldn't drink so much soda and that they should lose weight, but they still chose not to. Like, do we want to develop policies that affect that? I'm not sure. Yeah.

Or maybe that's just free will. Yeah. Maybe that's just free will. Yeah. And it comes up in the parenting stuff too. Like, you know, how much do we want to be externally controlling the choices people make with their kids? even if we don't think that they're the right choices.

But I do think there's a segment of people who want to make the change but the gravity, you know, because of the information, but the gravity of the habit is so much that it's hard to know where to go about it. I guess I would say, where do you see this data going? Like if you had your fantasy for where you want the kind of data and the way that we see this data evolving

and the way that you see that kind of percolating out to the public, I mean, in terms of being sort of a translator and providing people the tools, like what do you want to see? in terms of the way the system responds to or integrates this data in the future. Yeah. I mean, I think the big message of the book is in some sense that, like, you should use the data to make yourself, like, confident and happy in your choices.

I think so much of what is hard about parenting is that in the moment you are not often confident in your choices. And then when somebody asks you, like, why did you do that? Then you feel bad. Yeah. Right? And I think that there's, there's a sense in which sort of looking at the data, but then confronting, like, well, we don't know. But you'd be like, okay, I made this choice. You know, I decided to let my kids watch an hour of TV every day.

Yeah. Because, like, I thought about it and I thought there wasn't any data. I'm like, that's the choice that I made. That sort of that confidence is, like, important for being happy. And if we could sort of, like, move in that direction, I think that would be, that would be good. It reminds me a lot of what one of my good friends, Branny said to me when I was in the trenches of, like, babyhood and having a lot of anxieties. around all these hot button issues, breastfeeding, sleep time, like all of it. She had been through it. Her kids were in college and she was like, let me give you a piece of advice. Be wrong, but be wrong with confidence. Yes. Just be wrong with confidence. That's all that matters. Yeah. No, exactly. Be wrong with confidence. I love that. Yes, I am wrong with confidence so frequently. And actually turns out to be right. The truth is there's a lot of good options. A lot of good options.

Yeah. Thank you so much for joining us on the A16D podcast. Thank you for having me.