Short Wave - Will Artificial Intelligence Help — Or Hurt — Medicine?

Episode Date: May 2, 2023

A doctor's job is to help patients. With that help, often comes lots and lots of paperwork. That's where some startups are betting artificial intelligence may come in. The hope is that chatbots could generate data like treatment plans that would let doctors spend less time on paperwork and more time with their patients. But some academics warn biases and errors could hurt patients. Have a lead on AI in innovative spaces? Email us at [e-mail]!See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

You're listening to Shortwave from NPR. Hey, hey, Sheravers. Emily Kwong here. Our quest to dissect artificial intelligence continues with my friend Jeff Brumfield from the NPR Science Desk. Hi, Jeff. Hi, Emily. Hi. Okay.

It honestly seems like everyone is trying out chat GPT these days, including medical professionals. You told me about this guy, Cliff Sturmer, a rheumatologist on TikTok. Save time, save efforts. Use these programs. chat GPT to help out in your medical practice. Which is wild to me because he's not alone. Cliff, like a lot of doctors, has been playing around with chat GPT to see if it could help save time on paperwork.

And I've learned from you that some tech companies and startups are offering to build systems for this very thing for medical paperwork. That's right. Yeah, it's really interesting. There are a number of ways these AI programs that can generate things like stories and poems could potentially help out with all the stacks of notes and things that doctors have to do. And that includes, you know, things like listening in on a conversation between a patient and a doctor and taking notes about what they're talking about.

Is that okay? Is that ethical? I mean, they make very clear the doctors that they are using this system when they do. But there are also real worries, too. You know, as we've talked about in the show, these models and AI programs can make mistakes. they can sometimes just make things up. And then there's this extra problem that you get when it comes to medicine, and that's the problem of bias. So today on the show, is AI about to revolutionize medicine?

And what could it mean for both doctors and patients? You're listening to Shortwave from NPR. So Jeff, I guess the first question on my mind is, why are doctors so interested in AI? Aside from it being kind of inherently cool, Why are so many people latching onto it? Yeah, I called up Pierce Keene, a professor of artificial medical intelligence at University College London, and he really laid out why doctors are so interested.

The brutal truth is that we are drowning in the number of patients that we need to see and treat. So AI has been making quiet inroads in medicine for a couple of years now in fields like radiology and ophthalmology. In fact, Pierce is an ophthalmology. And, you know, he pointed out to me, one of the first algorithms was one that screens for diabetic retinopathy. Basically, when blood vessels in the back of your eye become damage from diabetes, and it can eventually lead to blindness. And how does AI help detect that? Yeah. So the way that the AI works, as I understand it, is there are systems that can take an image of the retina.

And the algorithms can make this determination whether the patient should be referred for further screening. Right. It's like a way to quickly sort for who might be having this condition and treat them accordingly. That's right. And so there are a number of algorithms now that can look at things like, you know, screen for retinopathy or look at scans and x-rays, maybe for radiology. But they tend to be quite narrow. They can look at one type of scan. They can look for one kind of problem. What makes this new technology based on chat GPT and other chatbot so as exciting is it's actually much broader. It can be applied to many medical specialties, at least potentially. So that's where I caught up with another doctor named Derek Paul. He's the founder of a company called Glass Health in San Francisco. When he first saw ChatGPT, he wasn't really all that impressed with it.

I looked at it and I thought, man, this is going to run. write some bad blog posts. Who cares? That was really my initial. I was like, who cares? I did the same thing. I downplayed it. I was like, I was just another toy for the internet. Yeah, yeah. He kept hearing about it from medical students who were younger than him, who were trying it out for answering questions and learning about things. And then eventually his customers started asking him about it too. And he's sort of been converted. He says that, you know, chat GPT could produce halfway decent answers, and now he's got this product called Glass AI. It looks kind of like a Google search bar, except where you'd normally enter what you were looking for. You just enter a

quick description of your patient. We're working on doctors being able to put in that, what we call like a one-liner, that patient summary, and for us to be able to generate the first draft of a clinical plan for that doctor. So what tests they would order and what treatments they would order. This is paperwork that doctors have to fill out all the time for patients. So you can imagine if doctors could just do a little copy pasta and then put it in the hospital record system that would cut down on a lot of time. But Jeff, we have done reporting on the potential for racial bias in AI systems, for all kinds of bias, really. And because of that, I'm kind nervous about how we're continuing to hand over the clinical keys to a computer with these matters.

Yeah, no, I think you're absolutely right. I mean, before we even get to racial bias, we've talked about the issues on this program before, you know, this new AI is designed to generate new things, but it doesn't always know whether it's right or not. Yeah. You know, these systems can hallucinate, they can fabricate, they can sometimes just mash together outdated information. Maybe you're giving the, you know, treatment for a heart attack from 1984 instead of from, you know, 2022. So there's all sorts of stuff that can happen when you have these generative systems. But just focusing for a minute on bias, that is a really serious problem. There's a lot of bias in our healthcare system already, and AI can make it even worse. So Marziah Gisimi is a computer scientist

studying AI and healthcare at MIT. When you take state of the art, machine learning methods and systems and then evaluate them on different patient groups, they do not perform equally. Okay, unpack this a little for me because I know about AI that these systems, they're trained on big data sets put together by humans. And whether that data is from the internet or a medical study, it contains naturally all the human biases that already exist in our society. Those biases just tend to show up in the data. Is that kind of what's happening with medical AI sometimes? That's exactly it. I mean, it's that simple.

Basically, you know, if there's bias in the data and there definitely is, then these systems will reflect those biases back onto the doctors using them. So, for example, Marzia used an AI chatbot trained only on scientific papers and medical notes to complete a sentence from a patient's medical record. And here's what it did. When we said white or Caucasian patient was belligerent or violent, the model filled in the blank was, you know, patient was sent to hospital. If we said black, African-American or African-patient was belligerent and violent, the model completed the note was patient was sent to prison.

I know that was just for a study, but that's pretty disturbing, Jeff. Yeah. I mean, this is a fairly extreme example, but, I mean, it's also important to remember, like, this system wasn't trained on some chat board somewhere. This was trained on scientific texts only. So even using supposedly more impartial data, it still turned out this really biased result. And Marcia says other studies have turned up similar things. She worries that these medical chatbots are basically going to turn into biased parents that are just going to regurgitate bad decisions to doctors. And the doctors will just go along with it. It has the sheen of objectivity. ChachyPT said that you shouldn't have this medication. It's not me, right? Like,

A model, an algorithm made this choice. Yeah, I can understand her fear because it shifts blame for bad decisions onto the computer. Wrongfully so, since people made computers. Right. And, I mean, there's a whole separate question of who is to blame if one of these computers makes mistakes, whether it's the doctor. Derek Call of Glass AI makes a real point to saying doctors need to be cautious when they use AI. They need to understand it could get things wrong. And that's not something they're typically that used to.

You have to supervise it, the way we supervise medical students and residents, which means that you can't be lazy about it. But I have to ask the question, if you give a busy doctor a powerful tool like this, and it's right 99% of the time, how carefully will they be checking it, both for factuality and for what can be very subtle biases? It's not just going to jail or not. I mean, it can be much more subtle stuff that really affects patient outcome. So bottom line, Jeff, where do you think we're headed with this?

And do the enormous net positives, to be clear, with efficiency, with being able to see patients who need to be seen, outweigh these also pretty serious negatives. Well, you know, I hate to say this because it's what journalists always say, but time will tell, Emily, time will tell. No, really, look, it really comes out to your perspective. You know, Paul says that there are real problems out of it. here that need to be solved. Doctors are overwhelmed with paperwork. We need these folks not just in burnt out states trying to complete documentation. We need them thinking and working on all these different things. And then patients need more than 10 minutes with their doctors. And, you know, I mean, he's also working on ways to get around these problems. He obviously is aware of the problem

of bias and, you know, of hallucination or making stuff up. And so they're a different. developing this sort of medical, almost like a virtual textbook for Glass AI. The computer program is supposed to reference specifically only information in that textbook to try and keep it from fabricating or relying on sources that are somehow biased or otherwise unworthy. But Marzia is still pretty skeptical. She says that there are real underlying problems, right, that this technology is supposedly kind of trying to work around. Hospitals don't have enough staff. Maybe insurance companies are requiring too much paperwork.

When people say technology is a solution for this, I often think, no, you made this problem. Technology will not fix it. Just fix the problem. I think regardless of where you come down on this, the bottom line is it's happening, right? I mean, class health has seen a lot of strong interest from doctors. and the tech is coming, you know, whether we're ready for it or not. I think we're going to find out just how good it is. Jeff, thank you so much for coming on to talk about this aspect of AI entering our lives and our medical systems as well.

Yeah, it's been a pleasure. Thank you. This episode was produced by Burley McCoy. It was edited by our managing producer Rebecca Ramirez and fact checked by Nicolette Kahn. Robert Rodriguez was our audio engineer. Beth Donovan is our senior director, and Anya Grundman is our senior vice president of programming. I'm Emily Kwong. Thanks as always for listening to Shortwave from NPR.