Instant Genius - COVID numbers, with Professor Sir David Spiegelhalter

Episode Date: June 5, 2022

What story do the statistics tell about the pandemic? Sir David Spiegelhalter, the non-executive director the UK Statistics Authority, explores what lessons we’ve learned over the last two years. Once you’ve mastered the basics with Instant Genius, dive deeper with Instant Genius Extra, where you’ll find longer, richer discussions about the most exciting ideas in the world of science and technology. Only available on Apple Podcasts. Produced by the team behind BBC Science Focus Magazine. Visit our website: sciencefocus.com Hosted on Acast. See acast.com/privacy for more information. Learn more about your ad choices. Visit podcastchoices.com/adchoices

You said this place was steps from the water. We just haven't found the steps yet. How much did we save? Enough. Enough to get lost. Or you could book a stay with Hilton. Welcome to your ocean front room. Just steps from the water.

The Hilton sale is on now. Book on Hilton.com or the Hilton app and save up to 20% to get the stay you expected. When you want savings, not surprises. It matters where you stay. Hilton, for the stay. It's peak pollination season, and my business is scaling fast. To keep the nectar flowing, I need a phone plan with top priority data speeds.

That's why I chose GoogleFi Wireless. My connections stay strong even when the hive is buzzing. Plus, unlimited plans started $35 a month. Now, that's a deal that doesn't stay. Explore Google Fi Wireless plans today. Plus taxes and government fees. Google Fi Wireless is not subject to data traffic deprioritization during times of high network usage. No one goes to Hank's for his spreadsheets.

They go for a darn good pizza. Lately, though, the shop's been quiet. So Hank decides to bring back the $1 slice. He asks Copilot in Microsoft Excel to look at his sales and costs. Help him see if he can afford it. Copilot shows Hank where the money's going and which little extras make the dollar slice work. Now, Hanks has a line out the door. Hank makes the pizza.

Copilot handles the spreadsheets. Learn more at M365 copilot.com slash work. This podcast is sponsored by name, audio and focal. Streaming has made music more accessible than ever, but true listening is about more than ease. It's about quality. British audio experts name audio, alongside French acoustic specialist focal,

combine handcrafted tradition with cutting-edge innovation and high-end materials, delivering digital precision with analogue warmth. So you can experience exceptional sound at home. Music just as the artist intended. Visit name audio.com to learn more. From BBC, science focus, this is instant genius. A bite-sized master class in podcast form. I'm Daniel Bennett at the magazine's editor,

and today we're taking a look at what lessons we can learn from the pandemic. And I'm joined by Professor Sir David Spiegelhalter, the non-executive director of the UK Statistics Authority and the chair of the Winton Centre for Risk and Evidence Communication at Cambridge University. His new book, written together with Dr. Anthony Masters, who's an ambassador for the Royal Statistical Society, is called COVID-Bin Numbers,

and it's a succinct but comprehensive guide to the statistics that tell the story of the pandemic. I just wondered if you could give us a sense of the picture that you were getting as the pandemic unfolded. You know, there's a lot of, experts in the media were kind of throwing out a lot of different predictions and stats and numbers and sometimes it looked all quite confusing and I wondered from your perspective as a statistician trying to take a broad view of what was happening was it sometimes overwhelming the amount of

data you had or was it brilliant you could dive in and pick out details of what was happening just wanted to get a sense of what it was like what the kind of quality and I suppose resolution of the picture that you were able to build as it happened? Well, things changed so much during the pandemic. I mean, really at the beginning, back in March and April 2020, is a very, very confused picture. It only just started in this country. And we weren't testing. So we'd no idea how many people actually had the virus. We're, you know, counting a few deaths, but that's a hugely lagged indicator of what's going on. We now know, of course, that the virus was, you know, erupting all over the country simultaneously

by mid-March, essentially. And we now know that. We didn't know it at the time. And so the dribs and drabs of numbers that came out were really unsatisfactory, very difficult to do anything with. There are also numbers coming out, which are, of course, the epidemic model projections. Now, these are not statistics.

These are, you know, sort of simulations of possible futures under, you know, very strong assumptions. And they got huge amounts of attention. And in a way, I think as a warning, it was quite reasonable. They got too much attention. But I think it's very important to distinguish numbers that are things that are actually measuring what is going on in the world around us and numbers which are possible projections for what might happen in the future under certain circumstances.

So I think we need to distinguish those very carefully. Well, that's, I mean, that's probably a good thing to just dive into because there was a lot of talk in the media and sort of confusion, I guess, around projections and what they actually meant. I'm going to probably get this wrong, but there was, what's it called, sort of the worst case scenario? You had an abbreviation for that. Okay. So it's traditional in-government modelling that analysts are asked to do a projection based on what's called a reasonable worst-case scenario. And this has been around for years in government planning. So this is something that's plausible, but is, you know, at very much the serious end in a reasonable worst case. And so it's a

deeply pessimistic scenario that is, and that's what models are asked to do. That's the, that's the instructions they are given to do that. It's quite clear. It's not just their choice. They're told to do this. And so, for example, Imperial in their very influential paper from March the 16th, You know, their reasonable worst case scenario they looked at it was as if the pandemic went wild and we didn't do anything whatsoever against it. And then they projected, oh, there'll be half a million, half a million deaths and whatever. And of course, that's quite plausible. But of course, if we had done nothing about it, we're not going to, you know, this is completely an implausible in a sense future because we're never going to sit around and watch half a million people die. We'd have done something about it.

But as a, and so this was, you know, has been called by people who want to run down the modelling, a prediction. And because this is nonsense, it's never a prediction. It's a sort of upper bound if we did absolutely nothing. And, and then models will produce all sorts of other things under more plausible scenarios of reacting to it. But I'd say, you know, throughout this whole pandemic, we might as well deal with this now. I think the models have received, over much attention by everybody because they are just projections under particular assumptions. These assumptions are actually dictated by the government to say, well, you know, the models are told what to do to a large extent. They only cover certain outcomes and they can't. The crucial thing is, of course, we don't know is human behavior. And the whole pandemic is driven by human behavior. And so the assumptions made about human behavior were massively influential, and yet are the ones that are the least data on. And we know who know? We don't know how people will react. We just don't know.

So my feeling is that these models, which are based on what technically we call highly non-linear sequence of events where things can explode upwards very rapidly or not, you know, these are not nice, smooth models. small little tweaks in assumptions can send them off in very different directions. It's almost chaotic. So I think that the modelers were perhaps taking a little bit too much and certainly too much has been made of what they've done. Are you optimistic then that will come out of these two years

with, as a country, better literacy around statistics and numbers? You know, it's long been your mission to help people understand data, risk and statistics. I am hopeful. I am hopeful. Certainly the public interest in data and stats is unprecedented. We just know that. The willingness of the media to engage with statistics seriously

and to use graphics and to explain the numbers is unprecedented, extraordinary efforts. Right across the range. I mean, I've worked with every bit, I think. And even, you know, some of the more, perhaps unlikely ones have done very well. in putting graphs up and explaining what's going on. So I'm quite optimistic. There's still a real problem within the media

of when things get away from the journalists who actually know what they're talking about. The health journalists and science journalists are I think have been very good indeed. They haven't fallen through for all the misinformation that's been floating around. They've been, I think, very reliable

and some brilliant explanations of what's going on. The problem is, is when it gets the hands of the general journalists than the political journalists, who frankly can be utterly clueless. And it just becomes embarrassing. Okay, here's a classic one. You know, people have known for more than two years that the number of cases or deaths being reported each day, particularly deaths, depends crucially on the day of the week. They're very low on Sundays and Mondays because deaths don't, the records don't come in, reports don't come in. And very high on

Tuesdays and Wednesdays, when they catch up Tuesdays in particular, always have a massive number, usually double what it is on Mondays, for example. So Tuesday's data is completely unreliable. Three times the evening standard has had a headline, COVID death saw to record numbers or something like that, and three times, using the same words. And the only thing that's in common between these headlines is they're all on Tuesdays. And you know, you just think, for heaven's sake, please, please, just understand that these are deeply unreliable numbers. And, you know, actually, it's not just journalists. Other people have used those numbers to make, you know, when they want to say,

oh, it's all awful, it's all terrible, what's going on. And so there's been a lot of misuse of numbers, sometimes even knowingly. But often, I think it's just through people not understanding where the numbers come from. And you can say the same with COVID deaths. I mean, there's at least three, four definitions of a COVID deaths that one can use. And so it's very complicated. I can understand people having trouble, but the good journalists have come to really understand this stuff.

Did you get a bit, and you know, I did see you on Twitter quite often fighting. Ranting away. I think you were very, very polite, actually. I tried to be. I tried. Yeah. Did you feel that was a kind of a public service almost? Twitter's a strange thing. I mean, my following built up hugely. I'm quite sparing how I use it now.

For me, it's been invaluable. It really has been. It's been extraordinary. There's such good people on Twitter. The real insight for me, which is surprising, is the role of what you might call the independent analysts. First of all, conflict of interest. I'm a executive, non-executive director of the UK Statistics Authority, which oversees the work of the Office for National Statistics, the COVID infection survey, the census, etc. So I would say they're a wonderful group of people doing fantastic work. They are a wonderful group of people, but that is my conflict of interest. So the analysis by them and by the UK Health Security Agency, what was Public Health England,

has been extraordinarily good, really fantastic sort of way we might call it institutional analysis. But what's been essential is all the independence. Now, some of those are organisations, the actuaries, have done a, brilliant job. The people, Ignar, who look at intensive care data, wonderful reports. But then there's January, and then there's academics, you know, Oliver Johnson and others, and I suppose I've done some. And then there are real independence, people who've got other jobs and who just have been putting out, sort of James Ward, Paul Mainwood, and others on Twitter who have done brilliant modelling and analysis and real insights in their spare time. And I don't know how they got

spare time, really. And for me, those people, it's been completely invaluable and journalists have picked up on these people and using them, and not just on more or less and other programs, but on, you know, from other sources. So I think the, in praise of the independent analyst, I think this is really showing valuable. They can, as a check on what the big boys are doing. I, there were a note, you obviously have to be careful who you follow and who you listen to, but you're right, you're absolutely right. There were a lot of hobbyist statisticians doing brilliant work. And there are also some awful hobbyist statisticians.

It's been awful work. You know, really dire, manipulative misinformation has come out. And there's no immediate way of telling from someone's Twitter profile whether they're trustworthy or not. You just have to learn. You can learn quite quickly by their tone. I mean, the crucial thing, you know, my indication always is, you know, is somebody trying to genuinely inform you?

Are they trying to persuade you of their point of view? And you can usually, I'd pick that out pretty damn quick. Well, that's a good point to jump onto one of the really common myths that persisted on Twitter, which is, this is just like seasonal flu. Why aren't we treating this like flu? This is like flu. I wonder if you could just speak to that, because you did look at the data here and you've got some start statistics on that. How different is it to seasonal flu?

So, it was rather ironically, at about the time we're speaking, it's probably not that different. Now, it certainly wasn't two years ago. There was nothing like seasonal flu. It was far worse. It was really punishing the infection mortality. Fatality rates, probably 10 times seasonal flu. It's far more infectious, so much worse than seasonal flu.

Obviously, it was at the beginning. Things are different now. So, you know, it would be dreadful if, because now it's becoming much more like seasonal flu, and I think that's likely what it would be like in the future. If people say, oh, well, why do we need all that stuff two years ago? Yeah, because it wasn't like that then. 99% of us have got antibodies. Come on.

At the start, zero percent of us had antibodies. You know, there is a huge difference now between them. So I think that at the beginning it clearly wasn't and the people who were saying that and it's just a seasonal thing, clearly diluted and dangerous in their misinformation. So I wonder, you know, you've obviously had a great career studying risk and how people understand them. What surprised you the most over the last two years, I wonder? Oh, from a risk point of view, it's age.

I have to say by March 2020 I was on more or less banging on about the unbelievable importance of age and that the risk of dying from COVID doubled for every about six, seven years you were older and that this meant that someone in their 90s had maybe 10,000 the risk of someone who was nine years old. So, you know, this unbelievable effect of age, which of course is when normal mortality has that, you know, link with age as well. And COVID risk follows normal risk. Study after study has shown that your risk of dying from COVID is proportional to your normal risk of dying each year. Very, very closely, some factors are slightly different, but amazingly closely. What COVID does is take any vulnerability you've got and exaggerate it. And the biggest vulnerability

is just being old. So we know that your risk of dying each year, you know, increases massively. you get older. We know that scientifically. I don't think people do really do realize this, you know, actuarial risk. They're unaware of just how steeply risk increases as you get older. Everyone knows it happens, but they haven't got a feeling for it. They haven't got a natural feeling. And that's partly because there's real difficulty of grasping exponential growth. And this is exponential growth in risk with age, with, as I said, doubling every six to seven years. So that's a really, really tricky thing to grasp. And I don't think anyone's ever grasped it.

There's young people going around, you know, really worried about all getting COVID. Okay, yeah, some young people have died. Very few, really. Kids, you know, essentially, you know, nearly all primary school kids have had COVID. They're not unvaccinated. They've all had COVID. Just about not, not. Yeah, nearly all.

And so actually, without much, I don't think really much harm. There are, you know, there are obviously long COVID cases. I can't deny that that happens, of course. But in terms of really of fatalities and things, no, it's unbelievable. And it's incredibly dangerous for old people. So I think it's that aspect which never has really got through to people. And the government has been, Chris Witty went on about this right from the beginning. I think they've been worried about saying it that the average age that which people die from COVID is 82, which is the same average age that people die normally.

And I think we've been worried about compaging it because it might give the idea, oh, well, COVID's only for old people anyway. So we don't have it. So I think that's been, it may have been, you know, I think that's why there's been a reluctance to, to emphasize the massive risk gradient among people in order to almost to keep everybody, to some extent, to keep everybody. Well, we could talk about this. You know, I think it may have been, you know, there has been some attempt to keep people anxious. And I think it's now we're getting the payback of it, of a lot of anxious people. And then one thing I just want to put to bed because it's probably the two, there were the two biggest things was on the TV every day.

We saw the total number of cases and the total number of deaths. And again, there was a lot of statisticians popping up on Twitter. When I say that, I mean, you know, the new ones who suddenly found themselves. trying to make sense of the data. And there was a lot of querying of these figures. What do we know now about the total numbers of cases and deaths in terms of... It depends what you mean by case and what you mean by death. Cases, you know, it's confirmed cases with PCR tests or whatever.

And now, of course, that bears no resemblance to the actual number of people with infections. If you look at the number of people who have been infected with COVID, I don't know, 40 million, you know, something, you know, it's just unbelievable vast numbers, almost pointless to record it. As I said, 99% have got antibodies, but that includes, of course, people have been vaccinated,

but huge numbers of people also had it after the vaccination, like I did, you know, vast, vast numbers. So it's almost pointless to talk about the number of people infected because you might as well just say, well, almost everyone. Not quite, there will be some people who haven't had it, but not. And yet, if you just use confirmed cases, then that just depends on testing regimes. We know that. As soon as tests stop being free, suddenly cases dropped.

Isn't that amazing? So then deaths also, you know, what definition do you use? The daily deaths, which is the rapid report one, just who died within 28 days of a positive COVID test. Actually, for some time, tracked fairly well the real number or better number, which is what's on record on the death certificate, where they lost track. Recently, they sort of lost track of each other more. For a while, it wasn't too bad. The daily number wasn't too bad.

But then if you just look at what is generally considered the gold standard, which is presence on the death certificate, then you've got three different things there. You've got people who died due to COVID, where it was the underlying cause of the death. You've got people who died with COVID as a contributory factor. It wasn't their main cause, but it contributed to the death.

And then you've got people who died generally with COVID. In other words, they had COVID then died of something else, in which case, COVID will not be on the death certificate or should not be on the death certificate. Death certificate data itself is pretty complicated. So, you know, there's no, you can't say how many people died of COVID. Oh, for a start, in the first wave, huge numbers of people died from COVID

who weren't recorded as dying from COVID. It was under-diagnosed. So many people died in care homes, and all people died. home, died of COVID, doctors didn't see them, didn't want to put it on the death certificate because they hadn't seen them. And so in the first wave, there was a real large, you know, thousands of excess non-COVID deaths. Well, these were COVID deaths, really. You know, even the gold standard death certificate data is not perfect at all. So we don't know. I mean,

for a start, we don't know because what does it mean to die from COVID? What does that mean? You can think of all sorts of different definitions. You know, we get very philosophical. You know, what is cause? You know, these are not well-defined countable numbers. We can't go out there and identify these cases perfectly. So that's why people look at excess deaths. But then that's all, you know, just count the bodies and see how many more there are than you'd normally expect.

But then you have to decide, well, what would we normally expect? And that requires assumptions and modelling and things like that. So essentially none of these things are absolutely hard and fast, cold hard numbers. They're always constructed on one basis or another. And all we can say is that, yeah, there's been a lot of people who wouldn't have died so soon. Yeah, I mean, what are we talking about, $150,000 or something like that, which is a lot, whose lives have been shortened by COVID. There's a really interesting study and kind of an anecdote in the book that I just want to be. wanted to highlight and bring to the conversation, which is, we talked about projections earlier,

but then in the book you talk about how when the sort of models are really hard to make, because there's so much minutia, then we can turn to experts and see what they think. And I think it was you carried out the study where you asked a bunch of experts what they predicted the number of cases would be. And it turns out we're all quite optimistic. Is that right? Yeah, yeah. I was, I've got, I'm hopelessly optimistic.

I wrote down my predictions in March 2020 with probabilities on how many people would die. Hopelessly optimistic. Really, really, no, hopeless. And I would completely admit that. I'm glad I've never been at any responsibility at all because I'm hopelessly optimistic and deluded. And I wrote down my projections, yeah. And people were, on the whole. They just didn't envisage that something like this would be so bad,

partly because you're taking the lesson from things like SARS and MERS as they occurred in, and the whole government response to emerging diseases, you know, it was based on the assumption that it would be like SARS and MERS in the Far East, where these have happened before or into Canada as well, where they, because they almost, they're so infectious that, and, and, but they're infectious when people have got symptoms, that you can actually isolate it quite quickly. And so the government projections had, for emerging diseases, had, you know, a couple hundred deaths as the reasonable worst case scenario.

And they had a, you know, big flu pandemic modeled, and they had a big, they had an emerging, you know, a disease one. What they hadn't taken into account was an emerging disease in which there was asymptomatic spread, in which you couldn't just isolate people and stop it spreading. Is that something we can account for in our thinking in future when it comes to, I don't know, does that come into modelling? The whole point about, you know, identifying reasonable worst case scenarios is to protect them against people like me. You don't want people like me doing disaster plan because I'm, oh, it'll be fine, it'll be fine. So, no, you don't want people like me at all. So that's why they go for a reasonable worst case scenarios, quite rightly. But they do, because of that, you do tend to, very much.

much focus on specific scenarios. And almost certainly that isn't what's going to happen. And it looks like in our, you know, the planning for this, that what actually happened fell between the gaps. That was Professor Sir David Spiegelhalter there, talking about how modelling works. If you'd like to dig a little deeper into the statistics of COVID and find out what David might tell his future self if there was another pandemic, check out instant genius. extra, a bonus podcast available via subscription on Apple's podcast app. Alternatively, do pick up a copy of David's new book, COVID By Numbers, which is written with Dr Anthony Masters.

It goes on sale later this year and it's published by Pelican, an imprint of Penguin books. Thank you for listening. The Instant Genius podcast is brought to you by the team behind BBC Science Focus magazine, which you can find on sale now in supermarkets and newsagents. well as on your preferred app store. Alternatively, you can come find us online at sciencefocus.com. This podcast is sponsored by Name, Audio and Focal.

The texture and emotional depth of music can be lost through digital sources or poor signal. Name Audio believes you can have digital precision with analog warmth. Alongside French acoustic specialist Focal, name creates high-end audio systems, combining innovation with craftsmanship, So you can listen to music, just as the artist intended. Discover more at name audio.com. Ambition comes in all shapes and sizes. At First Citizens Bank, we roll with your goals

because we're built for what you're building. Fit for your ambition for Citizens Bank. At Santa Monica College, a world-class education is affordable. Just $46 a unit with free tuition for eligible first-time California students. Plus flexible online and on-campus. classes close to home. Summer starts June 22nd, SMC.edu.