Short Wave - Coronavirus Models Aren't "Wrong." That's Not How They Work.

Episode Date: April 20, 2020

Scientific models of disease don't predict the future. They're just one tool to help us all prepare for it. NPR global health correspondent Nurith Aizenman explains how scientific models of disease are built and how they're used by public health experts. We also look at one influential model forecasting when individual states might begin to reopen. Email the show at [e-mail]. See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

You're listening to Shortwave from NPR. Maddie Safaya here with NPR Global Health Correspondent, Narit Eisenman. Hi, Noree. Hi, Maddie. So we're going to start today at the White House. Thank you very much, everyone. Back on March 31st, early on a Tuesday evening. Our country is in the midst of a great national trial.

Right. It feels like months ago, it was actually just a few weeks ago. President Trump had just extended federal social distancing guidelines, urging people to stay home from work and school until April 30th. A number of states had already taken more drastic steps on their own, and this press conference was supposed to be where they explained why. Why they made that decision to tell us all to stay home. It had to do with computer simulations that try and predict what could happen during an outbreak. They're called models. I just want to thank the five or six international.

and domestic modelers. There was a modeler out of the University of Washington, from Harvard, from Columbia, from Northeastern, from Imperial, who helped us tremendously. Dr. Deborah Birx, the response coordinator of the White House Coronavirus Task Force, stood in front of these big charts and summarized what one model said would happen without social distancing. Between 1.5 million and 2.2 million people in the United States

succumbing to this virus without mitigation. And how that changed. with social distancing. 100 to 200,000 deaths, which is still way too much. Yeah, I remember the headlines after this press conference, that we would basically need to drastically social distance because the models were telling us that it would limit deaths in this still awful best-case scenario of at least 100,000 people dead in the U.S.

And I don't know about you, Noree, but to me, this was the moment that, you know, this somewhat obscure corner of science that most people didn't even know existed kind of blew up. And models have been front and center ever since that moment. Here we are weeks down the road. And the good news is that at least one prominent projection is that 69,000 people will die in this current wave of infections. Still a staggering number, but way less than the 200,000 that the White House was talking about. And now states are trying to figure out when they can lift social distancing. and once again, they're turning to models to help them figure that out.

These models, if they keep getting things wrong, changing their predictions, why are we relying on them? Well, actually, that's sort of the wrong way to think about it. A model is not something that either turns out to be true or doesn't. A model is a forecast. And just like a weather forecast, it gets harder to make the further into the future you go. So it's complicated. And it's especially complicated when we're talking about modeling.

a never-before-seen viral pandemic. The modeling that Dr. Berg showed predicts that number that you saw. We don't accept that number that's what's going to be. And this was a point Dr. Anthony Fauci tried to drive home back in March that models don't predict the future. They're just one really important tool to help us prepare for the future. So, you know, I don't want it to be a mixed message. This is the thing that we need to anticipate.

But that doesn't mean that that's what we're going to accept. We want to do much, much better than that. So today on the show, how scientists build models of disease, how officials factor those models into their decisions, and how the rest of us should think about models as we make our way through the pandemic. So, Noree, obviously there are a lot of models out there. Today, we're going to focus on just one. Yeah, it's important to stress that there are a lot of important. models being done, federal and state officials should and are using quite a number of them to inform their decisions. But the one we've picked to talk about today, we're looking at it because

it's one that's widely cited. And it's also one the White House seemed to be really leaning on back in March. Interestingly, Dr. Burks has never really explained the task force's internal modeling that got them to that 100,000 to 200,000 deaths figure. Instead, she's just said that it's very similar to a model that she kept pointing to at the press conference, and it's the model we're going to talk about right now. It's from the University of Washington. Yeah, I'm Chris Murray. I'm here in Seattle. We're actually in shoreline, Washington. So Chris is the director of the Institute for Health Metrics and Evaluation, or IHME at the University of Washington, which is why his model has become known as the IHME model for short. And his work started pretty early on. Remember how Washington

State was one of the first hotspots in the country. You know, from my family and I, we have literally almost not left the house except to shop for food since March 10th. Wow. We're pretty extreme on the social distancing. So that was the early stages of the outbreak in the Seattle region. And University of Washington hospitals were trying to come up with a plan for the surge of patients that were coming. So they asked Chris and his team to help them model how many hospital beds they'd need, how many ICU beds, how many ventilators. stuff like that.

And so we did. And then other hospitals heard about this, and they asked us to help them. And then we got flooded with so many requests. We decided we would just do it for every state. And suddenly lots of people became interested in our forecast. And some of those people worked in the White House. You know, we put the numbers out for states, I think on a Wednesday. It's all a blur now.

And then by Saturday morning, we, We were in multiple rounds of calls with people in the White House on the task force explaining what our model was. And as we've said, this was not the only model the task force members were interested in, but it stood out. Why? Like, what made it different? So the traditional way scientists model disease outbreaks kind of goes like this. You make a bunch of assumptions about all these different variables. You know, how many days are you infectious and how many days are you asymptomatic and et cetera, et cetera, et cetera. And so you come up with all these assumptions, rules kind of, for how you think this particular disease will go.

And then you hit the play button on that sort of model of the universe and see what happens over time in a given population. But to do that, you're saying you have to make assumptions about how the virus works. But this is kind of a brand new virus. We don't know that much about it. So a lot of those assumptions are going to be wrong or not completely accurate. Exactly. And so what Chris and the team at IHME did that was different was they thought, okay, well, maybe we don't have to know the answers to those questions about how many days are you infectious, how many days are you symptomatic, et cetera. Maybe we can just look to places where the pandemic hit earlier.

At first this was Wuhan, China, and look at those places that are further along to help us model how it's likely to go in the U.S. We're trying to find statistical predictors in each country. Yeah. And why? Because advantage of, you know, things that we can see in data is that they actually happen somewhere and it's not entirely theoretical. So he's saying that even though the virus is new, it did hit some countries earlier than ours. And he can look at those places for information that can inform the model. Yes.

And specifically, after cases started rising, those places in terms of, reduced varying levels of social distancing. And then their rate of new infections and new deaths and new hospitalizations slowed. So IHME's approach was to say, okay, from that, we can come up with essentially a formula for if state health officials impose these various measures, how will it change their curve? You know, when do they reach their peak number of daily deaths and hospitalizations? We try to relate the time from introducing social distancing. Yeah. to figure out time to peak. And then once you know the time to peak,

the rest of the model fitting is looking at how fast the epidemic is going up, and that'll tell you how high you're going to get on the peak day. Now, we should note that not everyone is a fan of this approach. Other prominent epidemiologists who favor the traditional method have critiqued it. And Chris Murray of IHME, the guy behind this model, also says it has some limitations. He notes that a challenge of the approach that IHME uses is that early on in an outbreak, when there still aren't that many cases to base your trend line on, your forecast is going to be less accurate.

There wasn't really a way to know that New York was going to have this extraordinary epidemic. And, you know, Louisiana, a pretty substantial one. But Georgia that looked like they were going to have a big epidemic in the first four or five days hasn't. A second challenge? Remember, the model looks to other places for what the impact of social distancing is. Right. Social distancing in Wuhan doesn't necessarily look like social distancing in Wyoming. Precisely. So it probably won't have the same effect. Still, they can at least keep updating their data. And as the outbreak has played out in additional places, including several locations in Italy and Spain, IHME has been able to refine that formula for projecting the effect of social distancing.

In fact, based on those refinements, they then concluded that social distancing in the U.S. was going to bend the curve even faster and more effectively than the early version of their model had projected. The data change. We get new knowledge every day. We want the numbers to change. We are learning as we go. You don't want a model that is driven by your assumptions. You want a model that is driven by the data. All of which is to say a model like this is never going to be perfect. There is constantly new data to feed into it. So basically we should expect these models to keep changing their projections. Yes. There's a quote about modeling generally attributed to an eminent statistician named George Box, who wrote, all models are wrong, but some are useful.

Honestly, sounds like PhD students. You know what I mean? It's probably a good one for reporters, too. Anyway, yeah, go ahead. And now modeling like this is being. used to tell states when they can consider opening up. Right. And actually, IHME has released a new version of their model, which is basically looking at, you know, how low does your caseload need to get before you could consider opening up? And they've projected for each state when they'll hit that date. So people can look this up.

As they do that, is that going to give people kind of a false sense of security, maybe even make them, you know, like not as vigilant about social distancing? Yeah. People can see those charts and not necessarily read them the same way an infectious disease expert would. I actually asked Chris Murray about this too. I think the big challenge for the country will be in May, if we're right, when numbers are coming down and there will be a strong temptation for individuals to get back to their lives. And there'll be a temptation for state governments. And so there's just a real risk that we prematurely take off the brakes on transmission. Because, of course, if and when a state quashes the current wave of infections,

the vast majority of people will still not have been infected. They are still going to be vulnerable to this virus. Coronavirus could flare up all over again. It just underscores that the behavioral element of this pandemic, the psychological toll of being cooped up at home for weeks and weeks, when you combine that with maybe some mixed messaging from the state and federal level, that's a really hard thing to model. Just like some of the models overestimate because they don't take into account the behavioral response,

you know, once things start to look bad of individuals, we may also, you know, in the flip side, we also need to take into account the reverse form of behavioral response when things are looking better. In other words, he's saying initially their model might have been too pessimistic, about how much Americans would social distance before state officials ordered them to. But now he's worried that their model might actually be too optimistic about how long Americans will keep up social distancing once they see cases going down. And so, again, that gets at the point we made before. Models are forecasts. They are not our fate.

And our behavior can change the forecast in a big way. Okay. I feel like I know substantially more about pandemic modeling right now. So I appreciate you. Thank you, Noree. You're welcome, Maddie. This episode was produced by Brett Bachman, edited by Andrea Kisick, and fact-checked by Emily Vaughn. I'm Maddie Safaya, back tomorrow with more shortwave from NPR.