AI is everywhere in healthcare now. Doctors are conflicted

How doctors use AI tools

0:00Hey, it's Flora, and you're listening to Science Friday. I am an endocrinologist and professor of medicine. I'm a psychiatrist. I am a hospital medicine physician. I am a medical student. We're headed to the doctor's office, and we're asking, how are doctors using AI in their practice? We asked our physician listeners to call in with answers, and many of you did. I use artificial intelligence every day in my work.

0:32I've been using it for less than half a year, and I find it to be extremely helpful. One of the ways I use it is when a patient asks me a question directly in their electronic chart, and it automatically generates a reply that I could potentially send back to the patient. Sometimes the AI version of the answer is much more polite than my version would be of the test. What we heard from doctors mirrored broad trends that more and more doctors are using AI tools for diagnosis,

1:10paperwork, and getting up to speed on patients. So how is AI changing medical care, and how is it affecting patients? Here to sort fact from hallucination is Dr. Jonathan Chen, Director for Medical Education in Artificial Intelligence at Stanford University. He's studied the accuracy of these models and how healthcare workers use them. Jonathan, welcome to the show. Thanks so much for having me, Flora. Looking forward to a very dynamic conversation that's changing under our feet. Me too. I mean, can you give me a sense of how dynamic this is?

1:43I mean, how much the daily workflow has changed for doctors in the past year or two? I mean, I liken it to, it's like the internet got invented three years ago. We're rapidly trying to adapt to it. I worked in this space, AI and medicine, 10 years ago, right? Before it was cool. And six years ago, I had students working on this. This stuff was unusable. You wouldn't even bother talking about it. Three years ago, Chak Chepetit blows up on the scene. It's like, whoa, this can kind of barely pass a medical exam. But that's barely passing. That thing's not smarter than me. And now it's like, no, it's basically as good or better than most doctors would be answering medical questions.

2:17And we're still figuring out how to integrate that in a responsible and effective way while managing very predictable harms at the same time. How many doctors in the U.S. are using AI for their work? Do we know? I mean, I bet a lot of them are using it without even realizing it, right? If you just go into Google search, it is starting to integrate AI into those responses, whether even intended to or not. So more and more, I think the majority are starting to do it. Two years ago, when we did a study, maybe a third had never touched it before. A third used it once or twice. Maybe a third of docs had used it two years ago.

2:49But now it's clear that the majority are starting to figure out, at least have touched it. But that doesn't mean they know how to use it effectively. That's still an ongoing battle to figure out. What are the tools or platforms that are most often used by doctors? The reality is just the generic chat platforms, right? Your Chachapiti, your Clodge, your Gemini's. Within medical-specific ones, there are several that have started to blown up. How do you combine this great chat capability but with actual medical-specific knowledge references? So now we just had a study recently.

3:20There's, you know, there's Amboss came out. There's this German company, Glass Health, on there. But also Doximity has a product, and Open Evidence is showing the company that kind of blew up out of nowhere in the past several years. It became a multibillion-dollar company trying to create this interface in the form that doctors are used to. And it is pervasive all over the place when three years ago you've never even heard of these things before. What are doctors using this chat feature for? What kinds of questions are they asking, and are there other uses? I mean, for one, there are also these ambient scribe tools.

3:51Like, let's just listen to our conversations. Your doctor's not spending their time writing into their note. They could just be talking to you and looking to your face. So there's that kind of thing. A lot of these ones I think are more compelling because it's really, like, answering questions. They're really almost like a second consultation. We used to, as doctors, look up an article in a, you know, a medical encyclopedia and try to figure out if it applies to you. I see a lot of doctors and trainees I work with, they don't want to look up an article and then read it and figure out if it applies to you. They just, you just want an answer now, right? Which means you ask an AI system, it'll read the article for you and say, I read the article, this looks like the answer.

4:26And that's a different dynamic and a different interface. And I think why people are really gravitating toward it and really kind of getting hooked on it. I don't really see how we're going back. What are the problems that these tools are solving for doctors? Oh, gosh. Solve is actually a higher bar. It's certainly assisting with a lot of things. But making sense of and organizing medical information, keeping track of things is actually very powerful. A very common human behavior, doctor behavior, it's like we're all very smart, but there's unlimited medical knowledge.

4:59So we'll often run into our colleague, hey, what's the latest cholesterol guidelines? I literally had a patient ask me, if a patient has a hip fracture and there's some blood around the joint, does that need surgery? Like, I'm not a surgeon. I don't really know. But could I run into someone? Or a curbside console is a classic phrase where I run into you on the curbside, can I ask you a quick question? That's the kind of thing. The reality is AI is pretty freaking good at this kind of thing. And is it perfect? No. But it's also way more accessible in ways that just wouldn't be practical for doctors or patients.

5:29How is – so if a doctor was looking for help with a curbside consultation and they opened open evidence, how is that query or that process different from my experience in chat GPT? The reality is it's grounded in the same underlying technology, the large language model technology that's reading your phrase and digesting the internet, autocomplete and steroids, guess is next word. What a lot of these medical ones are focusing on is – I mean, if a chatbot says, how do you prescribe morphine?

6:01I mean, who knows where it got that from the internet? You don't really want to trust that. But here they combine it with RAG, retrieval augmented generation. What does that mean? And they'll find an article, a medical guideline, ah, this seems to be talking about your question, and it will read sections of that article and tell you what the answer is, which is nice because then you can trust but verify. We feel a lot better if we can point back to a source and we can say, ah, I can go back and look at this directly.

Hallucinations and clinical risks

6:26What about hallucination? I mean, is that a problem you have to worry about? We're making up citations. It is a very real issue. I won't say that's a solved problem, but it's a lot better than it was two or three years ago. Two or three years ago, it was a huge issue. These chapters would confidently make up citations, and that's worse than if it's wrong because it's so disarming. It so – has the appearance of credibility, and hallucinations and confabulations can very easily lead you the wrong way. And there have been studies that show people are sometimes worse off using AI because they are confidently following a wrong path. This mail looks so believable.

6:57I won't say it's a solved problem, but these RAG-based systems, the one that look up articles for you and really can – you can click on the link and go to the source. It helps a lot. But I would say now you have different kinds of issues. What if the question doesn't have an answer in the literature? Many different kinds of studies are higher quality than others. What if it cites something and doesn't really interpret it in the right way or interprets a low-quality study? These are very real tensions. It's not an AI problem. It's just a fundamental medical knowledge and reasoning problem that this is a danger because it looks so good. Oh, my gosh, it looks so good.

7:28People treat these things like they're the oracle, like they're God. And I had to tell my trainees, this thing is not God. It is very powerful. It's really cool. But just because it says something does not mean you can blindly trust it. I want to get back to trainees in a second. But we asked our listeners, you know, as patients, how do they feel about their doctors using AI? And we got a variety of responses. They ask before they use it, is it OK? And I think that's pretty important. I kind of feel a lot of times some big skepticism when I see things, hey, I will fix this.

8:01But maybe this is a good thing. Let's wait and see. I don't think it's ready for the medical system. OK, so here's my question, Jonathan. Have we done the research to know if these tools, which more and more doctors, you say the majority of doctors are using, are helping patients, are doing more good than harm? Ooh, that's a deep question. You know, there are some very interesting studies, not mine, others who study regular people using Chachi PT or other things to answer medical questions.

8:34And the AI made the patients actually worse. In our studies, it sometimes helps doctors, sometimes it doesn't help them enough. But in patient cases, sometimes it made them worse because it's not just having the knowledge, right? With a great podcast interview, it's how do you elicit the information, frame the question, interpret the answer the right way? If you don't do that well, it'll go way off the rails. Yeah, I wanted to ask you about this. I mean, we know prompts matter so much in what you get out of AI. Are doctors getting trained to use these tools? At the high level, no.

9:06You know, when we submitted one of our studies a couple years ago showing what happens with doctors who use Chachi PT in this case, a peer review said, that's not realistic. No one would unleash some computer system for doctors who use it without properly training them first. I'm like, what are you talking about? That is exactly what they do. That happens all the time. It's happening right now. So most clinicians, patients are not getting any formal training on how to use these things. One of my jobs now is this newly created role, Director for Medical Education at AI Stanford, specifically because we need more of that here and broadly.

9:38And it is letting them, the trainees and the doctors know these real tools, you need to understand a little bit how they work because they're clearly going to become a part of your life and your work. Know what the caveats are so you can use them safely, effectively and responsibly. What about procedures? I mean, we know hospitals and medical systems have tons of procedures often to cover their butts in terms of, you know, liability. Are hospitals or medical centers actively writing new AI usage policies? They are.

10:09And it's a tough thing, let alone scientific peer review usage policies. They're lagging, right? The technology is moving so fast. It's crazy, right? It's moving every month. It's like, shoot, another update. And your policy you wrote six months ago is already out of date. A couple of years ago, a lot of people are trying to ban the technology. We don't know how to deal with it. It's kind of scary. So just ban it. Nobody's allowed to use it. That's completely impractical, even if it were a good idea, which I don't think it is. That's my opinion. And there's not consensus. You cannot effectively ban it because someone can just pull out their phone, right, and work around you.

10:42And now they're probably using a non-secure, non-privacy compliant tool to do what they would have done anyway. So more of these policies, more of these frameworks are coming out. But it's happening so haphazard, so distributed because the technology has just moved so fast that human institutions cannot keep up pace with how fast it's moving. After the break, I want to talk to you about medical students and trainees and how they should be using these tools. You down to stay with us? Absolutely. Don't go away. You know, one thing that was interesting in the calls we got from doctors is that even though, you know,

Medical student reliance and training

11:30almost everybody who called us said they were using it, they also were conflicted about it for kind of interesting reasons. It is definitely something that we are learning about in medical school because it's something that we will be using in our futures as physicians. I do worry that learners will become too reliant on generative AI, and as a result, they may lose some of their clinical reasoning skills. And I also worry about biases that are inherent in the models that AI is trained on.

12:04Are you worried about this, that med students are going to be overly reliant? They won't be able to vet the information they're getting from these tools? It is a real issue. It's a real issue. I say there's a very huge dilemma, and we have absolutely not consensus on this. We recently had a poll, like, 60% of favor, 40% against. We had a debate, and then it went to 55% of favor. And yet you still need a policy on what to do. I'll give this brief anecdotes to really punctuate the point. In our medical reasoning class at Stanford, you know, here's a case. What do you think the diagnosis is? What do you think the treatment and management should be?

12:35On the homework, on the homework two years ago, the students are killing it. They're killing it. But when it came time for the closed book exams, like, twice as many students failed that exam compared to usual. Like, what happened? It's obvious what happened. They're using the AI to do their homework, and then they missed the point. No, homework, you're supposed to struggle through homework so that you actually then actually learn and are ready for the closed book exam or the live patient interaction where you can't have AI do it for you. So that's definitely a real danger. On the other hand, when we cautioned them last year, this is a very powerful tool, the best tutor you've ever had if you use it right.

13:09But like a chainsaw, it's a powerful tool that you could really hurt yourself with if you don't use it properly. This time, the students actually did better on the closed book exam, and they're learning to use it in practice when they're understanding how to use it with the right guardrails. You know, we've been talking a lot about chatbots and sort of querying for medical questions. But what about AI for analyzing radiology? Where are we with that kind of use of AI in medicine? Sure. Computer vision, actually, I would say is much more mature. That was a very hot thing five, six, seven years. It was, here, look through this x-ray.

13:41You know, tell me if I have a lung infection. There's a nice study where doctors using a thing to help it find polyps on a colonoscopy, it made them better. And when they turned off the AI computer vision tool, the doctors were worse than before they started, right? They got used to it. They started to depend on the technology. So very powerful capabilities. But the famous thing here is Jeffrey Hinton, Nobel Prize winner, kind of a father of a lot of neural networks. He famously said about seven, eight years ago, it's so obvious all radiologists should, like, just stop working, right? They're all going to be replaced by computers within five years. He said this, like, eight-ish years ago.

14:12So clearly he's wrong because now there's more radiologists needed than ever before, but it was a shifting and understanding of what the jobs and needs are. Spotting things on x-ray, maybe that's not the task anymore. It's, like, it's synthesizing, organizing that, and manage that for increasing demand for these services, too. Hmm.

Privacy liability and medical illusion

14:32Do you worry about privacy concerns? Like, are these tools HIPAA compliant? Do we know how the patient data is getting used? So it's a huge issue. And that's actually why, you know, one of the key trainings we give our people, doctors and trainees, FYI, you cannot put in real patient information into ChatGPT or Claude or Gemini or whatever. You have totally just uploaded private patient information to a public server when you do that. And some tech company now knows all of that. It's very easy for that to be invisible and people to not notice that.

15:03So that's been a lot of the guardrails in place to manage this. But the companies also have to be responsible for where it can go wrong. Right now, there's too thin a really fake disclaimer. FYI, you should not use this for actual medical advice, even though I know you're actually doing that. And literally, companies will put out press releases. Look at this cancer patient. We're, like, saving their life. Like, they're really talking out of both sides of their mouth, right? It's like, okay, if you want to provide the benefit, you've got to take the responsibility. But if something goes bad, literally somebody can sue me for medical malpractice.

15:33Are you liable or is the tech company liable? Like, if you use one of these tools, like open evidence or something else, you rely on it for medical advice, you pass it along, something goes really wrong, who gets sued? Yeah, it's a very thorny issue. Well, if you're a malpractice, what you do is you sue everybody and you see where it sticks. Right now, it, in theory, sticks to the doctor. All these tools, all this, at the end of the day, I signed the order. And so, in theory, I'm the one who gets sued.

16:03If, like, I use 20 different tools, I don't even know how all of them work. This magical reference library, this AI, it seemed to have reasonable information. So, that actually is an awkward thing that they have very limited responsibility. The FDA, Health and Human Services, many agents are trying to figure out how to wrap their heads and their brains around this, realizing it's a very difficult problem and we have to figure out how to adapt to it. Speaking of a magical reference library, I read that you're also a magician.

16:34Do you see any relationship between this work and that work?

16:39It is. That's somewhat of a coincidental hobby that spanned out of control, but it's really aligned and being a good magician is actually about having good empathy. You have to understand what another person is thinking so that you can trick them with the other thing and get them to think another way. And so much of AI is like, that's a really, it's so believable, but it can't be real. This thing looks like it expresses emotion. It's, right, the caller said it's more polite. It's not polite. It's not kind. It doesn't think. It's a computer. It doesn't do anything. But, man, is that illusion really convincing.

17:11And if you want to operate in the world effectively, you have to be able to distinguish. You still need your judgment, not your knowledge. You need your judgment to distinguish what is real and what is not so that you can do some good without hurting yourself or others. Dr. Jonathan Chen is an associate professor of medicine at Stanford University. Thank you for taking the time to talk to us today. Great to talk to you. I look forward to catching up. This episode was produced by D. Peter Schmidt. Speaking of questions, we're working on a segment about perimenopause. And if, like me, this topic is dominating your group chat, I want to hear a story about how you feel like perimenopause is affecting you.

17:48Brain fog, rage cleaning, a desire to run away from your family and make a new life in a motel room. But whatever it is, I want to hear the story. We'll have an expert to sort out the science behind the symptoms we hear so much about, at least on social media. 877-4SY-FRI is our number. Thanks for listening. We'll catch you next time. I'm Flora Lichtman. We'll catch you next time.