Science Friday - AI is everywhere in healthcare now. Doctors are conflicted

Episode Date: August 28, 2026

AI is popping up in many corners of society, but how are doctors using it in their practice? We asked our physician listeners to call in, and many of you did. What we heard mirrored broad trends: that more and more doctors are using AI tools for diagnosis, paperwork, and getting up to speed on patients. But they also have concerns. So how is AI changing medical care? And how is it affecting patients? Joining Host Flora Lichtman to sort fact from hallucination is physician-computer scientist Jonathan Chen, who’s studied the accuracy of these models and how healthcare workers use them. Guest: Dr. Jonathan Chen is an associate professor of medicine and director for Medical Education in Artificial Intelligence at Stanford University. Transcript will be available after the show airs on sciencefriday.com. Subscribe to this podcast. Follow our show on Instagram, TikTok, Facebook, and Bluesky @scifri and sign up for our newsletters. Got a science question that’s keeping you up at night? Call us: 877-472-4374 Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Hey, it's Flora and you're listening to Science Friday. I am a endocrinologist and professor of medicine. I'm a psychiatrist. I am a hospital medicine physician. I am a medical student. We're headed to the doctor's office and we're asking how are doctors using AI in their practice? We asked our physician listeners to call in with answers and many of you did. I use artificial intelligence every day in my work.

I've been using it for less than half a year, and I find it to be extremely helpful. One of the ways I use it is when a patient asks me a question directly in their electronic chart, and it automatically generates a reply that I could potentially send back to the patient. Sometimes the AI version of the answer is much more polite than my version would be of the test. What we heard from doctors mirrored broad trends that more and more doctors are using AI tools for diagnosis, paperwork, and getting up to speed on patients. So how is AI changing medical care and how is it affecting patients? Here to sort fact from hallucination is Dr. Jonathan Chen, director for medical education in artificial intelligence at Stanford University. He studied the accuracy of these models and how health care workers use them.

Jonathan, welcome to the show. Thanks so much for having me, Flores. Looking forward to a very dynamic conversation that's changing hands under our feet. Me too. I mean, can you give me a sense of how dynamic this is? I mean, how much the daily workflow has changed for doctors in the past year or two? I mean, I liken it to. It's like the Internet got invented three years ago.

We're rapidly trying to adapt to it. I worked in the space, AI and medicine 10 years ago, right? Before it was cool. And six years ago, students were going to this, the stuff was unusable. You wouldn't even bother talking about it. three years ago, Chachyipati blows up on the scenes like, whoa, this can kind of barely pass a medical exam, but that's barely passing. That's the thing's not smarter than me. And now it's like, no, it's basically as good or better than most doctors be answering medical questions.

And we're still figuring out how to integrate that in a responsible and effective way while managing very predictable harms at the same time. How many doctors in the U.S. are using AI for their work, do we know? I mean, I bet a lot of them are using without even realize. it, right? If you just go into Google search, it is starting to integrate AI into those responses, whether you even intended to for not. So more and more, I think the majority are starting to do it. Two years ago when we did a study, maybe a third had never touched it before, a third used it once or twice, maybe a third of docs had used it two or years ago, but now it's clearly the majority are starting to figure out, at least have touched it,

but that doesn't mean they know how to use it effectively. That's still an ongoing battle to figure out. What are the tools or platforms that are most often used? by doctors. But the reality is just the generic chat platforms, right? Your chat chavit, your clods, your geninized. Within medical-specific ones, there are several have started to blown up. How do you combine this great chat capability but with actual medical-specific knowledge references? So now we just had a study recently.

There's, you know, there's Ambos came out. There's this German company, Glass Health, on there, but also Doxymity has a product and open evidence. It's showing the company that kind of blew up out of nowhere in the past several years. It became a multi-billion-dollar company. trying to create this interface in the form that doctors are used to. And it's, it is, it is pervasive all of the place when three years ago you've never even heard of these things before. What are doctors using this chat feature for? What kinds of questions are they asking?

And are there other uses? I mean, for one, there are also these ambient scribe tools. Like, let's just listen to our conversations. Your doctor's not spending their time writing into their note. They could just be talking to you or look into your face. So there's that kind of thing. A lot of these ones, I think are more compelling because it's really like answering questions. They're really almost like a second consultation. We used to, as doctors, look up an article in a medical encyclopedia and try to figure out if applies to you. I see a lot of doctors and trainees and I work with, they don't want to look at an article and then read it and figure out if applies to. You just want an answer now, right?

Which means you ask an AI system, it'll read the article for you and say, I read the article, this looks like the answer. And that's a different dynamic and a different interface. I think why people are really gravitating sure. and really kind of getting hooked on it. I don't really see how we're going back. What are the problems that these tools are solving for doctors? Oh, gosh. Solve is actually a higher bar. It's certainly assisting with a lot of things, but making sense of and organizing medical information, keeping track of things is actually very powerful. A very common human behavior, doctor behavior, it's like, we're all very

smart, but there's unlimited medical knowledge. So we'll often run into our colleague, like, hey, what's the latest cholesterol guidelines? I literally had a patient ask me, if a patient has a hip fracture and there's some blood around the joint, does that need surgery? Like, I'm not a surgeon. I don't really know. But could I run into one?

Or a curbside console is a classic phrase where I run into one of the curbside, can I ask you a quick question? That's the kind of thing. The reality is pretty freaking good at this kind of thing. And is it perfect? No, but it's also way more accessible in ways that just wouldn't be practical for doctors or patients. How is, so if a doctor was looking for help with a curbside consultation and they opened open evidence, how is that query or that process different from my experience in chat GPT?

The reality is it's grounded in the same underlying technology, the large language model technology that's reading your phrase and you digest in the internet, auto complete and steroids, guess is next word. What a lot of these medical ones are focusing on is, I mean, if a chapter. How do you prescribe morphine? I mean, who knows where it got that from the internet? You don't really want to trust that. But here they combine it with rag, retrieval augmented generation. What does that mean? They'll find an article, a medical guideline. Ah, this seems to be talking about your question, and it will read sections of that article and tell you what the answer is, which is nice because then you can trust but verify. We feel a lot better if we can point back to a source and we can say, ah, I can go back and look at this directly. What about hallucination? I mean, is that a problem you have to worry about? We're making up citations. It is a very real issue. I won't say that's a solved problem, but it's a lot better than it was two or three years ago. Two or three years ago is a huge issue. These chapters would confidently make up citations. And that's worse than if it's wrong. Because it's so disarming. It's so has the appearance of credibility and hallucinations and confabulations can very easily lead the wrong way. And there have been studies to show people are sometimes worse off using AI because they are confidently following. a wrong path. It looks so believable. I want to say it's a solved problem, but these rag-based systems, the one that look up articles for you and really can. You can click on the leak and go to the source. It helps a lot. I would say now you have different kinds of issues.

What if the question doesn't have an answer in the literature? Many different kinds of studies are higher quality than others. What if it cites something and doesn't really interpret in the right way or interprets a low-quality study? These are very real tensions. It's not an AI problem. It's a fundamental medical knowledge and reasoning problem that this is. This is a is a danger because it looks so good. Oh my gosh, it looks so good. People treat these things like they're the Oracle, like they're God. And I had to tell my trainees, this thing is not God. It is very powerful. It's really cool. But just because it says something does not mean you can blindly trust it. I want to get back to trainees in a second. But we asked our listeners,

you know, as patients, how do they feel about their doctors using AI? And we got a variety of responses. They ask before they use it. Is it okay? And I think that's pretty important. I kind of feel a lot of times some big skepticism when I see things, hey, I will fix this. But maybe this is a good thing. Let's wait and see. I don't think it's ready for the medical system. Okay, so here's my question, Jonathan.

Have we done the research to know if these tools, which more and more doctors, you say the majority of doctors are using, are helping patients, are doing more good than harm? Ooh, that's a, that's a deep question. You know, there are some very interesting studies, not mine, others who study regular people using chatch-P-T or other things to answer medical questions. And the AI made patients actually worse. In our studies, it sometimes helps doctors, sometimes doesn't help them enough. But in patient cases, sometimes it made them worse because it's not just having the knowledge, right? With a great podcast interview, it's how do you elicit the information, frame the question, interpret the answer the right way?

If you don't do that well, it'll go way off the rails. Yeah, well, I wanted to ask you about this. I mean, we know prompts matter so much in what you get out of AI. Are doctors getting trained to use these tools? At the high level, no, you know, when we submitted one of our studies a couple years ago, showing what happens with doctors use GPT in this case, a peer review said, that's not realistic. No one would unleash some computer system for doctors you without properly training them first.

I'm like, what do you talk about? That is exactly what they do. That happens all the time. It's happening right now. So most clinicians, patients are not getting any formal training in how to use these things. One of my jobs now is a newly created role director for medical education, AI, Stanford, but specifically because we need more of that here and broadly. And it is letting the trainees and the doctors know these real tools,

you understand a little bit how they work because they're clearly going to become a part of your life and your work. Know what the caveats are so you can use them safely. effectively and responsibly. What about procedures? I mean, we know hospitals and medical systems have tons of procedures often to cover their butts in terms of, you know, liability. Are hospitals or medical centers actively writing new AI usage policies? They are, and it's a tough thing, let alone scientific peer review usage policies.

They're lagging, right? The technology is moving so fast. It's crazy, right? It's moving every month. It's like, shoot, another update. And your policy you wrote six months ago is, already out of date. A couple years ago, a lot of people are trying to ban the technology.

We don't know how to deal with it. It's kind of scary. So just ban it. Nobody's allowed to use it. That's completely impractical, even if it were a good idea, which I don't think it is. That's my opinion. And there's not consensus. You cannot effectively ban it because someone can just put out their phone right and work

around you. And now they're probably using a non-secure, non-privacy compliant tool to do what they would have done anyway. So more of these policies, more of these frameworks are coming out. but it's happening so haphazard, so distributed because the technology has just moved so fast that human institutions cannot keep up pace with how fast it's moving. After the break, I want to talk to you about medical students and trainees and how they should be using these tools.

You down to stay with us? Absolutely. Don't go away. You know, one thing that was interesting in the calls we got from doctors is that even though, you know, almost everybody who called us said they were using it, they also were concerned. conflicted about it for kind of interesting reasons. It is definitely medical school because it's something that we'll be using in our futures as physicians. I do worry that learners will become too reliant on generative AI.

And as a result, they may lose some of their clinical reasoning skills. And I also worry about biases that are inherent in the models that AI is trained on. Are you worried about this, that med students, are going to be overly reliant. They won't be able to vet the information they're getting from these tools. It is a real issue. It's a real issue. I say there's a very huge dilemma, and we have absolutely not consensus on this. We recently had a poll, like, 60% of favor, 40% against. We had a debate, and then went to 55% of favor. And yet you still need a policy in what to do. Like, if this brief anecdotes really punctuate the point in our medical reasoning class at Stanford, you know,

here's a case, what do you think the diagnosis is, what do you think the treatment of management should be? on the homework, on the homework two years ago, the students are killing it. They're killing it. But when it came time for the closed book exams, like twice as many students failed that exam compared to usual. Like, what happened? It's obvious what happened.

They're using the AI to do their homework, and then they miss the point. No, homework, you're supposed to struggle through homework so that you actually then actually learn and are ready for the closed book exam or the live patient interaction where you can't have AI do it for you. So that's definitely a real danger.

On the other hand, when we cautioned them last year, this is a very powerful tool. The best tutor you've ever had, if you use it right, but like a chainsaw, it's a powerful tool that you could really hurt yourself with if you don't use it properly. This time, the students actually did better on the closed book exam, and they learned to use it in practice when they're understanding how to use it with the right guard wheels. We've been talking a lot about chatbots and sort of querying for medical questions, but what about AI for analyzing radiology? Where are we with that kind of use of AI in medicine? Sure, computer vision actually, I would say, is much more mature. That was a very hot thing, five, six, seven years. It was, here, look through this x-ray, you know, tell me if I have a lung infection. There's a nice study where doctors using a thing to help it find polyps on a colonoscopy. It made them better.

And when they turned off the AI computer vision tool, the doctors were worse than before they started. They got used to. They started to depend on the technology. So very powerful capabilities. But the famous thing here is Jeffrey Hinton, Nobel Prize winner, kind of father of a lot of neural networks. He famously said about seven, eight years ago, it's so obvious all radiologists should like just stop working, right? They're all going to be replaced by computers within five years. He said this like eight-ish years ago.

Clearly, he's wrong because now there's more radiologists needed than ever before. But it was a shifting and understanding of what the jobs and needs are. Spotting things on an x-ray, maybe that's not the task anymore. It's like synthesizing, organizing that, and manage that for increasing demand for these services, too. Hmm. Do you worry about privacy concerns? Like, are these tools, hipaccompliant? Do we know how the patient data is getting used?

So it's a huge issue. And that's actually why, you know, one of the key trainings we give our people, doctors and trainees. FYI, you cannot put in real patient information into chat chit or Claude or Gemini or whatever. You have totally just uploaded private patient information to public survey when you do that. And some tech company now knows all of that. it's very easy for that to be invisible and people do not notice that. So that's been a lot of the guardrails in place to manage this. But the companies also have to be responsible for where you go wrong.

Right now, there's too thin of really fake disclaimer. FYI, you should not use this for actual medical advice. Even though I know you're actually doing that. And literally companies will put out press releases. Look at this cancer patient where it's saving their life. They're really talking out of both sides of their mouth, right? It's like, okay, if you want to provide the benefit, You've got to take the responsibility.

But if something goes bad, literally somebody can sue me for medical malpractice. Are you liable or is the tech company liable? Like if you use one of these tools, like open evidence or something else, you rely on it for, you know, medical advice, you pass it along, something goes really wrong. Who gets sued? Yeah, it's a very thorny issue. Well, if you're a malpractice, what you do is you sue everybody and you see where it sticks. Right now, in theory, sticks to the doctor. all these tools, all this, at the end of the day, I signed the order.

And so in theory, I'm the one who gets sued. If like, I use 20 different tools, I don't even know how all of them work. This magical reference library, this AI, so it seemed to have reasonable information. So that actually is an awkward thing that they have very limited responsibility. The FDA, health and human services, many agents are trying to figure out how to wrap their heads and their brains around this, realizing it's a very difficult problem. And we have to figure out how to adapt to it. Well, speaking of a magical reference library, I read that you're also a magician. Do you see any relationship between this work and that work?

It is. That's somewhat what was the coincidental hobby that span out of control, but is really aligned. And being a good magician is actually about having good empathy. You have to understand what another person is thinking so that you can trick them with the other thing and get them to think another way. And so much of AI is like, that's a really, it's so believable. but it can't be real. This thing looks like it expresses emotion. It's right. The caller said it's more polite.

It's not polite. It's not kind. It doesn't think. It's a computer. It doesn't do anything. But man, is that illusion really convincing? And if you want to operate in the world effectively, you have to be able to distinguish. You still need your judgment, not your knowledge.

You need your judgment to distinguish what is real and what is not so that you can do some good without hurting yourself or others. Dr. Jonathan Chen is an associate professor of medicine. at Stanford University. Thank you for taking the time to talk to us today. Great to talk to you. Look forward to catching up. This episode was produced by D. Peter Schmidt. Speaking of questions, we're working on a segment about perimenopause. And if like me, this topic is dominating your group chat, I want to hear a story about how you feel like perimenopause is affecting you. Brain fog, rage cleaning, a desire to run away from your family and make a new life in a motel room. Whatever it is, I want to hear the story. We'll have an expert to sort.

out the science behind the symptoms. We hear so much about, at least on social media. 8774 SciFRI is our number. Thanks for listening. We'll catch you next time. I'm Floor Lichtman.