ACM ByteCast - Haider Warraich - Episode 90

Episode Date: September 17, 2026

In this episode, part of a special collaboration between ACM ByteCast and the American Medical Informatics Association (AMIA)’s For Your Informatics podcast, Sabrina Hsueh and Li Zhou host Haider Warraich, a Program Manager at the Advanced Research Projects Agency for Health (ARPA-H) and a practicing cardiologist at Boston Medical Center. At ARPA-H he leads the ADVOCATE program that supports the development, regulation, evaluation and deployment of Cinical Agentic AI to revolutionize care delivery for patients with chronic disease. Previously, he served at the U.S. Food and Drug Administration as a Senior Clinical Advisor for Chronic Disease to the FDA Commissioner, where he helped shape policy around drugs, devices, nutrition, digital health, and AI. Before that, Warraich was a faculty member at Harvard Medical School and Brigham and Women’s Hospital and Director of the Heart Failure Program, at the Veterans Affairs Health System in Boston. He is the author of more than 180 peer reviewed papers and several books. His writing has also appeared in The New York Times, The Guardian and The Atlantic. The episode explores how agentic AI can address critical gaps in chronic disease management, focusing on heart failure and cardiovascular health. Dr. Warraich shares insights from his transition through clinical practice, academic research, the FDA, and his current role at ARPA-H, notably the newly launched ADVOCATE program, an ARPA-H initiative designed to build, evaluate, and deploy FDA-ready, patient-facing agentic AI technologies to serve as clinician extenders. The discussion covers geographic access gaps in care, autonomous agents and “copilots,” the current state of AI evaluation, and extending the ADVOCATE program from heart failure to all chronic disease management.

This episode is part of the special collaboration between the ACM podcast and the AMIA for your informatics podcast series. ACM, the Association for Computing Machinery, is the world's largest educational and scientific computing society. Amia, the American Medical Informatics Association, is the world's largest medical informatics community. In this series, we feature leaders in AI in medicine. including researchers, practitioners, and innovators who are shaping the future of healthcare at the intersection of computing, artificial intelligence, informatics, and the life sciences.

Our guests share their experiences in their interdisciplinary career paths, the lessons learned on AI in medicine, and their own visions for the future of computing. So welcome to the joining series, from ACM and BICAS and AMAs for your Informatics podcast. I'm Dr. Sabrina Shea from Pfizer. Join with me today is Dr. Lee Jo, with Harvard Medical School, where it's exploring the intersection of AI and healthcare

with the builders and stakeholders defining the field. We are at across the roads, where 40 million people daily use CHAPT for healthcare-related questions. Now that a single agentic AI has received FDA, relation, meanwhile nearly half of U.S. counties like a cardiologist, and only a fraction of patients receive skyline recommended care. So our guest, Dr. Heider, Warrich, is tackling these gaps at Upper Edge, and a practicing cardiologist and former senior clinical advisor to the FDA Commissioner. He now leads the efforts of Advocates Program. His mission is,

to build an FDA-ready, a GENTII Clination is tender for cardiovascular disease designed with a rigorous supervisory layer for safety from day one. Hi there, welcome to the South. It's great to be here. Thank you so much for the invitation. Right. So today we want to try something new, which you'll be the first guest to try this with us. We're going to do some rapid firearms to start with our audience here to know. hold you a little bit better in a very short period of time. Ready to go?

Let's do it. So first question. Advocate, can you describe in one sentence? Sure. So Advocate is a program launched by ARPA-H, whose goal is to support the development of Agentic AI technologies that are patient facing that can essentially do everything a clinician can over the phone

and really build out a regulatory evaluation an implementation pathway for clinical agentic AI broadly. Argentic AI in healthcare, hype or inflection points? I would like to think we're at the inflection point, but we'll have to see. But I think so far there has been a lot of hype. Some of it is understandable. Some of it is overblown. But I would like to think that we are either at or approaching the inflection point.

So will the first FDA authorized clinical agents arrive before 2013? Yes or not? Yes, for sure. And we're going to be leading that effort. One word for the current state of our clinical AI evaluation? I would say stuck in place. I know that's not one word, but I think that some of the evaluation is lagging in part because the types of technologies that we really need to evaluate in real time have not reached the clinical space. But that's part of what we're trying to do through advocate is to really unlock this field, not just. that using AI for care delivery, but using AI also to supervise the agents that provide care delivery. How about biggest myths about agenti AI in medicine?

I would say that the biggest myth about agentic AI in medicine is, I think a lot of people are assuming that it's going to be easier than it is actually going to be. I think this path is much harder than, I think, how a lot of people imagine this to be, in part because the data environment in healthcare and real healthcare is so much messier, and the quality of these technologies are only as good as the data that they receive. So I think this road is going to be harder than I think how a lot of people imagine this to be, but I think that's exactly why we need all hands-on-deck effort to bring this to fruition. Chatbot, call palette, or autonomous agent, which one might be?

most of our chronic diseases? I am biased, but I would say that I think to be able to manage chronic disease at scale, you need a technology that is autonomous. You know, the reason we're excited about this technology is because no matter how, no matter what we do, there's going to be a big gap in the amount of number of clinicians and how involved or engage those clinicians can be with patients. So the opportunity space for, and the more dependent these technologies are on clinicians, and the more limited their capabilities are the less the impact they'll have.

So I really feel like, again, we have to build the road and we have to evaluate it. We have to make sure it works. But in the end, I think the technology that's really going to make a difference is going to be autonomous. Yeah, so thank you so much. And welcome again to the podcast. Your career has certainly has been amazing when we read your resume. You're having a cardiologist for more than 150,000 people enrolled in Roe's. North Carolina, you have been director of the Hart Valley Program at VA, Boston,

and you have been the senior clinical advisor at FDA and now program manager at Upper Edge, and you certainly write a lot and has been doing the best selling books. So for our audience here, could you also have a diverse background, could you just give us some ideas of some of the inflection points in your journey that shaped how you think about medicine and AI. today and maybe even for your own career past under reflection. How did you think that brings you today? Absolutely. So, you know, I think the first inflection point in my career was, so I went to medical school in Pakistan and I came to the United States after medical school

and to do residency training and research. And I thought I was going to come to, you know, the richest country in the world and so much of what the issues in health and that we're going to have a utopian healthcare system, that everything's going to be great, everyone's going to be able to afford excellent care, and that's so much of what I'd struggled with, or so much of what I'd seen we could do better in a place like Pakistan, I'd never have to think about again. And I was really shocked to see that so many of the things that we're talking about are, you know, around poor access to care, variation in quality of care, how expensive medical care is, all those, I was having the same conversations here. So that was really eye-opening for me, and that was

something that certainly I could not have anticipated when I first moved here. I would say that the second big inflection point in my career was when I joined the FDA. So I was asked to join the FDA as a senior advisor to the FDA commissioner and my focus was on chronic disease. But the problem that I saw in that role was that, first of all, it was very hard to get people to get really care about chronic disease at that time, even though seven of the ten most common causes of death were from chronic disease. And the problem in chronic disease isn't that, oh, we don't have enough good treatments for chronic disease. You know, if you take a look at, for example, I'm a cardiologist. So for most of the things that we deal, most of cardiovascular disease, I think it can

be argued, is preventable. With existing therapies, most of them were cheap, most of them are widely available, but there's this real implementation gap between what we know works for patients and what they actually get. And at the same time, this was around the time that chat GPT was released, and I think most people who are thinking about medicine and look at this technology, it doesn't take much to figure out that this could be a great way to scale clinical knowledge broadly to patients across geographies. But the fact was that even though there had been so much progress in AI and AI technologies in clinical medicine, there's not a single technology that uses generative AI or Gentic AI that was able to perform essentially clinical functions directly with the patient.

So that was the other inflection point in my career. And I figured that one way to potentially solve that would be to move to an organization like ARPA-H and to convince them that we really need to make a major initiative to be able to unlock this field. Thanks, Heather. So you mentioned that you are now program manager at APAH to promote the adoption of Agentsiai to improve health care, improve. improve chronic diseases. Upper H is modeled after Depper, the agency behind

since like the internet, GPS, and the test technology. Now you are inside the agent that was built to take big swings in healthcare, high risk, high rewarded, that might not fit anywhere else. What does that feel like from the inside? And how does the upper H model change what you are able to try compared to with working in academic environment or at the FDA? Well, I mean, I'll be, I mean, I think I had a fairly traditional sort of academic career before I joined ARPA-H.

I was a sort of clinical researcher doing outcomes research, really, I think, thinking in the mold of how I think most clinicians and researchers think. But ARPA-H required, I think, me, and I think that's really the opportunity, it required me to really think differently and be bolder, but I think the sort of ideas that we can support and the sort of things that we could potentially do, why the agency can do that is because their mission allows them to take more risks, like things that may be hard to do in a way that there's really no other agency, research agency that's focused on health care can. And so that required me to also think differently about the sort of problems that I

should be thinking about, the sort of timelines I should be thinking about, and to really think in ways that I just wouldn't be able to do if this agency didn't exist. I mean, the question I pose is, you know, if you mentioned that, you know, ARPA age is modeled after DARPA, and DARPA is a defense advanced research projects agency. So even though they're focused on defense, so much of what they've been able to do has changed the, I would say, the course of human history, right? They helped invent the internet, GPS, stealth technology, driverless cars, MRNA vaccines, etc. etc. And so the question is, what would the world look like if we had an ARPA age for the past 50 years? What types of progress in human health would we have seen if an agency like this had existed before?

It's a really interesting place. And I think that we are only just starting to figure out how we can have the sort of impact that we've been really given this mandate to do. Yeah, sounds like a very, very exciting opportunity. I'm hoping that we would see more impact coming off on Alpha Edge. for sure. And especially in the area you're already leading, right, you have noticed the problem of any solution that relies on clinician is bound to fail. This is quoted by you. And given that, there are only 2%, 2.5% of American patients are on life-saving, recommended medications, right? And only 46% of U.S. counties, and still 36% of the U.S. County don't have a single cardiologist. This is really soaring for us to hear. Could you work us through a little bit more about how these lead you to advocates? Sure, yeah. So I'm a heart failure cardiologist, and which means that we take care of patients who have really crippling heart disease. And there are so many things that we can do as heart failure cardiologists. We can do, you know, we can do heart transplants. You can do LVAS, which are these mechanical pumps,

that essentially replace the heart, and patients can sort of live with them for several years. But the fact is that, you know, by the time a patient has heart failure, you've kind of already lost, right? I mean, there's so much you could have done before they even had heart failure that could have even prevented them from being that sick to begin with. One of the things that drew me to heart failure was the fact that this is a field that we have a lot of clinical evidence. We have a lot of treatments.

So we have, you know, so most patients with heart failure, there are four types of drugs or medications that are all shown to not just improve their quality of life, but to help them live longer. And if patients were on these medications, they would on average live for, you know, up to like 10 years longer than if they weren't on those medications. So this is a field where you think that, oh, this is a really bad disease. We have really good treatments. Most of them are pretty cheap, accessible. That we would be making a huge difference in these patients' lives with all these new innovations. and the fact is that even as we've had so much scientific progress, population-based outcomes for patients with heart failure are either unchanged or maybe at times even getting worse.

So why is that? And if you look at... And so a lot of my research before coming to RPA-H was focused on how are we actually failing patients with this chronic disease. And one of the things that we've found is that life-saving medications that patients should be on, only about 3% of patients, in fact, less than 3% are on, evidence-based treatments. And so that to me tells me that the entire system doesn't work,

right? It is no individual's fault. It is no one person's fault. There's no one thing that you can do to fix it, right? The whole system. And you could do this, you could say the same thing about essentially most other chronic diseases. You could say that about high blood pressure, for example, which is the leading cause of preventable death in the entire world, and yet 50% of patient with high blood pressure don't have their blood pressure under control. So really, we need a reimagination of how care is delivered for patients with chronic disease. And I think that the greatest opportunity is not when a patient is in the hospital, not even when a patient is in the clinic in front of me, but really when they are at home, which is where they spend most of their

life. In 99% of the time that a patient with chronic disease spends is at home. And so how can we provide care to those patients between visits, when they're not face to face for the clinician. And you mentioned some of the work that we've done previously to show that almost half the counties in this country don't have a single cardiologist. But also, this is not just true of those places. I live in Boston. I practice in Boston. So we have more cardiologists in this area than pretty much any other place in the entire world. And yet it takes people five to six months to see a cardiologist. Even when they see a cardiologist, I can see them maybe what, like, once every three months, once every six months, once every year. And so the whole model is just not able to meet

the needs of patients with really any chronic disease. Heart failure is the one that I care about more because that's what my training is in. But really the idea is that how do you create a new model of care that in which you can develop a new template for patients with chronic disease using heart failure as an example because it is such a crippling disease, but developing a template that can be used across other conditions. So there's a need for leverage advanced technology like AI to help our patient access the care that is evidence-based and can deliver this care in time and to the home. That's your vocate program that you're currently managed at upper age. It's aim to build

the first FDA-ready clinical AI agents that can do such work. So, for listeners who haven't yet see the advocate innovative solution opening, can you give us the program in your own words? What does a successful advocate clinician's tender actually do for a patient with advanced heart failure that currently the model cannot, the health system can't? Absolutely. So just to catch folks up, so ARPAH is a research funding, agency. We don't do research ourselves. We provide money to researchers, companies, startups, etc.

to perform the research. What we do do is that, and we don't provide grants. We give awards that are very metric and milestone-based. And our by-h program is essentially we put out a sort of vision of the future of what we want the future to look like. What are the key? And usually that is built around some type of technology. And then really have the best teams in the world apply and tell us how they would achieve or how they would build that solution or how they would solve that problem. And then we end up, you know, selecting, you know, oftentimes several teams to be able to overcome those problems so that we are looking at multiple different approaches at the same time to see which one is going to be best suited to achieve our goal.

So the Advocate Program is basically centered around three main areas. The first area is that we want teams to develop a patient-facing agentic AI technology that can provide clinical care directly to a patient. Our focus is on patients with heart failure because if you're building a medical device, and again, the medical device is the term that is used, you know, that's like FDA speak for any technology that can actually provide clinical care, whether that's AI-based or software-based as well. We want teams to build a solution in which they can say, how will they build an agendic-a-technology? It could be either voice-based, it can be an app. That will be able to essentially do everything a clinician like myself can over the phone. So this includes being able to make a diagnosis. So, for example, if a patient is feeling short of breath, traditionally, they would try and call the clinic.

Or maybe they call 911. In this case, they might engage with this technology, and they might say, hey, this is what I'm feeling, what should I do? So this technology should be able to a-perform triage. It needs to be able to figure out, okay, does the patient stay here? Do they need to go to the clinic? Can they wait until tomorrow? do they call 911, et cetera, all the things that someone like me would have to do if a patient had called me with those symptoms.

The second thing is that you cannot provide, you cannot actually perform triage if you don't have a diagnosis, right? So you can't actually, how can you tell a patient what to do, even where to be, if you don't know what's going on? And so this technology, she needs to be able to make a diagnosis. In this case, that is within scope for the patient with heart failure. And the third thing that it needs to be able to do is. is it needs to be able to close the loop, right?

It needs to be able to close the loop and actually provide the treatment that the patient needs that is within the scope of that technology. For example, that might mean that maybe they need to make an adjustment to one of their medications. So for example, patients with heart failure, they oftentimes need to have adjustments made to their diuretic medications to help them get rid of extra fluid that might have built up. Or they might just need to start a new medication that, will have a more long-term effect in helping improve their care. This is one of those or evidence-based treatments that patients, only 3% of patients are prescribed. And we think that

this is really important. The reason is, A, that our mission is to, in fact, support technologies that are transformative. I think if a technology is only just routing things to a clinician to sign off on, it's not the scope of technology or the scope of capability that is a fit for us to be supporting. And what we're saying is that these technologies have to be built with FD authorization in mind, and that teams need to have a pathway, a two-year pathway, develop, evaluate, and test these technologies to create a package that they can take to FDA to be able to support their authorization. We'll be working closely with the FDA through this process to be able to facilitate that. So that's one piece. So if you think about it,

that's sort of your clinician, that's a patient AI interface. The second thing is the supervisory agent. And what we wanted to do was instead of making a technology that's spans, the patient, the AI, and the health system, we wanted to disambiguate those two. And what we wanted was that to help scale these technologies, to help health systems be comfortable with deploying these technologies, we believe that you will need an extra layer of supervision or that is real time. I don't think every AI technology needs real-time evaluation or monitoring. But if you have a high-risk technology that can prescribe your grandmother some medication, right, or take away a medication or miss sort of life-threatening,

you would, you probably want something that is in addition that is helping you determine how to optimize human oversight. And what that means is that if you have a technology, if you have a patient-facing a Gentic-KI technology that is going to the clinician and asking for their approval for every decision, that's not a scalable solution, right? Because we've already taught the reason we're excited about this technology is because we just don't have enough clinicians. But if you have a technology that never goes to the clinician and is just working autonomously, there's a huge risk that things might happen and that health systems may not want a technology like this. There's a huge, and so really what you need is a technology that is

operating. Again, this is hard to do because you may not have ground truth in a real clinical environment, right? This is not a predictive AI tool. This is an agentic AI, generative AI tools, and we're really interested in those outputs. It can be often very hard to know what is the right or wrong thing to do. Even many clinicians will disagree with how to act or respond to specific clinical scenarios, but that is really the challenge that we've posed to our teams. What we've asked teams to do to reply, who are actually developing this supervisory agent, is to really build out that clinician AI interface that can really optimize the clinical resources

to the outputs that need the most attention based on surrogate measures of accuracy. So sometimes, you know, maybe they will catch something that's egregiously inaccurate. But maybe they won't. So how do you then divert, how do you focus attention? And that will have to be through surrogate measures, such as uncertainty, risk, acuity, etc. And then the third thing that we are really focused, that we're supporting through this program is we believe that implementation of AI is going to be one of the biggest challenges that we face. In part because, you know, it's one thing to say that there's a new drug. that will replace the last one.

So it doesn't actually change the workflow of how medicine is practiced. But if you're going to have a technology that is going to fundamentally change how care is delivered, who delivers care that may create some friction, you want the health systems that are going to deploy this to have a seat at the table from day one. And you want them really focused on thinking about what is it that they can do to optimize development of these technologies as well as the deployment of these technologies in clinical practice settings. So the third technical area that we're supporting as part of advocate are recruiting health systems

that are going to first co-develop this technology, really set the rules of engagement for how this technology, what this technology should be able to do, and how should be positioned within the clinical practice environment. And they're also going to provide data that can be used to train and sort of fine-tune these technologies. And then lastly, they will then, at the end of those two years, they will deploy the best performing heart failure genetic technology and supervisory layer in a randomized clinical trial in their health care system at scale. So that is the program. Those are sort of the three arms. Our applications have closed. We are evaluating proposals. And we're just happy to say that, you know, our main goal was to make sure can we get the very

best teams to want to apply? Can we show them that we are a trusted partner, that we can help deliver on this vision, and I will say that the answer to that question is a resounding yes. We're very excited about the teams that have applied. But our goal is broader. Our goal is to elevate this entire field. Our goal is to make sure that whatever we learn through this program, we share with everyone else, everyone including folks who are listening to this podcast, and that we overcome the blockers that are going to be universally applicable for anyone who is trying to develop clinical AI tools so that our impact is not just restricted to the teams that are in the program, but really help uplift the entire field. Absolutely. Cateovascular disease is just a

starting point. There are so many others. There are therapeutic areas that will need this. And if you get the FDA approval process, figure out everybody wants to follow your first step here. Yeah, but you answer actually a lot of our question first. But I want to type in a little a bit deeper here for those, since an advocate program is awarding these three arms across the three technical areas, a patient-facing agents, real-time supervisory agents, and having to make sure they have health system partner for design deployment in real-world scenarios. Are there any award criteria you will build in your evaluation process for them, for safety, for accuracy, So a couple of things.

So, you know, I think for, so each of the technical areas has a different need. And so we've documented all of that within our solicitation. And, you know, so for example, for building the medical device, you know, I think really thinking around execution and is a really important piece there, being able to build a medical product, right, which is and building that patients are going to use that is going to work. and then thinking about what type of unique strength that a team has that can help them be successful quickly, because we have very little time that we've given our teams. So those are some of the things. And then how strong their technical approach was, how strong their regulatory approach was. I think for the supervisory agent really thinking around, is the evaluation system or monitoring system that the teams have provided,

is it going to be fit for purpose? do they have the right capabilities, technical expertise, background to be able to execute on this, knowing that plan A may not work, right? Knowing that whatever plan that they have, it's possible that it may not work in this specific setting, you know, what type of depth do they have? Are they really thinking about, you know, so for example, the medical device piece, the TA1, the heart failure agent, they're the user as a patient. But from a supervisory agent perspective, the user is now the clinician or the health system.

And so really designing tools that are fit for the user is really, really important and what is the strategy there? And then from a health system perspective, we're really thinking about what types of resources are the health system going to provide to the team so that they can build really, really strong solutions? What type of experience they have in implementing these technologies at scale? Are they able to get us information that is generalizable? Right. I mean, in the end, you know, I think that we were looking for, and these are all in our solicitation. So, but we wanted health systems where if the technology succeeded there, it would give a lot of confidence to many other places. Yes, we can do this too because this system looks like who we are, as opposed to a system that has such capabilities that they're simply are not very generalizable.

So those are, and then what we do then afterwards is, again, this is how also we are a bit different from other funders is that we are. very metric and milestone-based. So we give very clear milestones that teams have to meet. We give very clear metrics that they have to do. We often will get a partner, which we call an independent verification validation partner that will really, in this case, perform comparative effectiveness of these agenetic technologies, which again is a really interesting research question, right? I mean, how do you, in fact, compare two different agentic AI technologies, both using clinical data but also using synthetic data and, and, determinations of accuracy and safety even before they'd ever touch a patient.

So I think, and again, because, and our goal is going to be to perform those and then to actually transparentlyly share them with the rest of the ecosystem because I think that we have the opportunity to generate knowledge that can really move the entire field forward. So those are, and all of those are listed in our solicitation. We make it very, way clear what those metrics are and when those are measured. We will definitely share that with our audience. And thanks for sharing your insight about how the authorities or institutions build employment to their solutions. We have some questions about the pathway for FDA authorization. So nowadays there's

no clinical agency A.I has received FDA authorization yet. I think you mentioned that it need like at least two years for go through the path. So there's a So there's a lot of uncertainty, maybe there's profound technical limitations and implementation hurdles. Drawing on your previous FDA experience, reversing chronic disease, medications, devices, digital health, and AI. What does the path to FDA authorization actually look like for a clinical AI agent? What's the difference from authorizing software as a medical device?

already kind of understand. Just to provide, you know, context. I don't think that the FDA is a blocker here. Like, I think that they are, you know, I know the FDA folks very well, especially those who are working on the AI and software side. They want to help. Like, they want to help unlock this field. They want to be able to understand how best to study these technologies and

understand them both before they're deployed, but also after they're deployed. I think that a successful pathway is one in that looks that is as familiar as possible. I think this idea that we should treat AI very differently from any technology before that, you know, I just don't, I think that actually creates more risk for, you know, this idea of sort of AI exceptionalism, that we need sort of exceptional rules for AI. I don't think so. I think that there is enough, I think, the existing regulatory pathways, provide a path, provide a route for these technologies to read patients. There's some things that are

important there. For example, I mean, one of the things that FDA really sort of focuses on is intended use and the intended use population. For example, for this type of technology, I mean, what I like to say is that, you know, FDA cannot authorize an AI cardiologist, but they can authorize a technology that improves outcomes for patients with heart failure. They've been doing that for a long time. Part of the reason this program exists is to, in fact, be able to get, is to use existing regulatory pathways and to facilitate teams taking those pathways to get to market authorization. And I do think that there's a path there. I think it's not going to be very, very different from any other technology where initially you do performance testing of the technology, you do

pre-clinical testing, often using simulated data, synthetic data, and you do small studies. In this case, maybe studies that are done in shadow mode, where, for example, there is sort of human oversight of every action, and then over time, you increase the level of autonomy these technologies have until they reach a certain threshold where you can then let them run autonomously and study them and compare them to regular care, routine clinical care, which is your control arm. I also want to say, I think that there's a second order question, and the second order question is that, is this in fact the right pathway for these technologies? Is this, in fact, fit for purpose for these technologies, or is there other way that we should be valuing these technologies

before they reach patients? That is a question. So I think we are setting up a platform to answer that question. And I do think that at some point, maybe soon, we need to think about another mechanism, perhaps one that uses simulation, to be able to study these technologies and get them to patients. But I think when we think about the first generation of these technologies, first generation of autonomous patient-facing clinical agentic AI, I think they need to follow. as much of the familiar processes as possible so that patients are comfortable, clinicians are comfortable, health systems are comfortable, regulators are comfortable, knowing that these technologies are performing at a standard and are using familiar pathways rather than we are creating an exceptional

pathway for them that will only create more suspicion, that will only raise more questions. And I think that this technology is good enough that it can go through the regulatory system and it can succeed. your informatics are available on Apple Podcasts, Google Podcasts, Spotify, Stitcher, and other major podcast platforms. If you are enjoying this episode, please subscribe and leave us a review on your favorite platform. Absolutely. Just yesterday I was in another discussion and people started to talk about we need to have

AI to take the license exam to get license. But certainly that's a traditional past. It will take a few years for people really to absorb that idea. So starting from the traditional past to find out how to do it will be certainly the first past that will help us to get there sooner for now. And if for that reason that, I think we come back to this question that we have a lot earlier, we have this cardiovascular disease right now as our first use case for this clinical agentic AI program you're working on. Can you tell us a little bit more about how we expect

this to generalize other chronic conditions? There are three parts of the program, right? There's a medical device part, which is focused on heart failure, then there's a supervisory agent, and then there's the implementation piece, which is we're working with health systems. So implementing these technologies is not specific to heart failure. You know, how you implement this technology and how you build it, how you build sort of the standards or the orchestration layer or the clinical practice adjustments to actually bring this technology into healthcare system, it's not specific. It's going to be the same whether we have to do it for diabetes or hypertension or hyperlipidemia

chronic kidney disease. So the implementation in the program is disease agnostic. The supervisory agent, we specifically want it to be disease agnostic because we don't want that for every single medical technology that you need a new type of supervised system, again, the core of that technology is going to be the same. The medical device piece, yes, it is specific to patients with heart failure, and it should be because if you don't have a specific population, how are you going to study it? And how are you going to make it best for that use case? But there's so much else about how these technologies are going to be architected, how are they going to be governed,

what is going to be in scope for them, that is not going to be restricted to heart failure. So I think that, yes, you do have to pick a intended use, I think an intended population. We believe that heart failure is actually one of the reasons we picked heart failure was because we felt like, well, if you do something hard, if you're able to succeed in that, then everything below that, everyone else will have confidence that is going to work for whatever their problem they're working on too. So for example, like if it works for heart failure, now if you want to build a technology for hypertension or high cholesterol, you have confidence that, yes, if you have confidence that, yes,

If it worked for heart failure, of course, we can do something like this for our use case. But if you do it the other way around where we start with a low-risk condition, it doesn't give you the same type of confidence that it's going to work for a much higher risk condition afterwards. But the other reason is also that if you look at how traditionally medical devices or medical technologies come to the market, they usually start with the high-risk population. They start with a population that, for example, has no other treatment options. right? So a chemotherapy for patients who have failed other chemotherapies or a new procedure for patients who are not candidates for surgery, for example. You don't start in the lower-risk population. You actually start with the higher-risk population because you can only make things better for them. Because otherwise, and heart failure is the condition in which we know that it is the leading cause of preventable hospitalization in this country. We know the standard of care for them is very poor. Within a very few months, you can actually show things like mortality benefit just by optimizing treatment for them. them. We have excellent evidence for treatments, right? There's very little subjectivity in making

treatment decisions. I mean, I personally think that a condition like, I think mental health is much harder than heart failure. That's really the goal of the program. The goal of the program is we have to be smart about problems we pick where we think we can succeed. There are many other reasons why I can sort of convince you that heart failure is a very strong place to start. In fact, because it is, I believe, is the surest shot for us to actually succeed. But then generate knowledge and generate technologies that are widely applicable, that are not, in fact, disease-specific. Very true, very true.

So now we are entered to the second round of rectifier questions. Are you ready? Let's do it. Let's do it. It's about build and buyer and governance. So, how people build their own clinical agent, brief, or reckless? I think it is brave.

I don't think it is reckless. I think it is brave because I think one of the nice things about this technology is that it's building, it's bringing our worlds together. It's bringing the world of computational science and clinical care. It's coming it together because it is, because these tools are increasingly easier to build. They're more accessible to people who are not,

I mean, again, you want to have experts as part of your team. but I think that hospitals understand the problems that well. I think they know where the opportunity might be. I do think that they need to approach this with a fair degree of caution because these technologies can act in unpredictable ways. They need to have the right team. But I think that they are well-posite.

They understand the problems. They understand the need. And I think at least many of the hospitals that I've seen are approaching this in a fairly responsible way. And I would encourage them to keep doing it, again, with responsibility. Open source foundation models for clinical use. Foolish or bearish? I am super bullish on open source models.

I think that we have, I think even over the past few years, sometimes the gap in performance between open source and proprietary models. Sometimes it's wide, sometimes it's narrow. It kind of goes up and down. But in the end, when I look at it from a regulatory perspective, it's much more easy to audit an open source model. Right? it's very easy to get more information about an open source model.

I think it's much more easier to sort of fine-tune these models. I think it's much more easier for them for you to have really sort of rigorous reinforcement learning for these models. So again, you know, I think that I think there's always going to be a gap. And I don't know, you know, and that gap is going to vary from time to time. But I remain very bullish. And that's one of the things that you're trying to promote through our program is really is at least have some of the teams actually use open source models,

whether that's for the medical device piece or whether that's for the supervisory agent. To again, be able to, this is a hype, this is a testable question. And I think that that's one of the things that you'd like to look at. What are the most underrated category for clinical AI agents? One of the things that we've been thinking about, I've spoken to this already, is the intended use. Right now you have general purpose models that are being used for general health care issues. I think as you move more towards more specific tools that are for specific conditions, then I think the idea of, is this technology being used for what it is designed for

and evaluated for and authorized for is going to become a more important guardrail for these technologies? Human in the root, human on the roof, human out of the route. What is that for cardiology? I think we need a risk-based approach. I think that that's what the smart teams will figure out. I don't think that you need a human in the loop for everything, right? I think that, but I think that you need a system that can help figure out when does a human being need to be engaged and really optimize that, right? If there's a technology that's overpinging the human, then it's never going to, the clinicians are not going to use that technology, right?

They already have enough pop-ups, but the opposite is also true. So I think that the best solutions are going to be risk-based and they're going to, engage humans when needed, but let the technology perform autonomously when not. Once they would recommend that every health care system, chief medical information officer, should be doing this quarter? I think one of the things that every, you know, CMIO should be doing is really finding out who within their organization is actually excited about using these technologies. What is there, what is their local talent pool?

Or is there a physician who's been, you know, sort of vibe coding on cloud code? or is there a resident who is using this for to develop sort of teaching aids for themselves? I mean, I think that what we're seeing is that this technology is diffusing in sort of atypical strange ways to people. I think really just understanding who are going to be the champions of this technology that are within your health system already. I know we talked about inventories of like AI models that are out there, but you really just having an inventory of the staff who are going to be part of the solution and really encouraging them, giving them platforms, making sure that they have the resources needed to be able to build solutions that can help advance the missions and goals of the organization. I think talent is the most scarce resource at this moment. And being able to identify who within your organization is going to be as local champion is, I think, probably really important thing to do.

Last one in this rerampereau. Pre-market evaluation or post-market surveillance, which matters more for you. I think it's post-market. I'll tell you why. First of all, we don't know how people are. going to use these technologies, right? And they might change and vary based on your local context, et cetera, and the underlying technology is going to keep evolving too. But even the best pre-market data, right, the very bad, let's say you do a double-blind randomized trial, it might be out of

date by the time you're done with the trial, right? So I think post-market is very important. The other reason is very important is that then it also unburdens a pre-market evaluation. If you don't have a good post-market system in place, then you're taking a real risk in the pre-market space. You'll probably need a lot more data in the pre-market space. Then if you felt like you had a good safety harness in the post-market. I'm really bullish on post-market as a way to both help deploy these technologies, but also unburden the pre-market evaluation.

That might be our next program. Yeah, now we move to another topic, very important topic, evaluation, guttrials, and observability. So this is a recurring thread in our podcast series. When we interviewed El-Tal Porter, he mentioned like governance paradox that briefs led you drive faster.

So we would love your own views as someone now option-nolizing this inside our federal program. So we mentioned that. that in our podcast that good AI governance is not just compliance, compliance exercises. It's actually how you move faster safely. So for clinical AI agents, that is making real recommendations to real patients, what does good governance look like to you?

Is it something you check off at the end or does it need to be built into the work from day one? I would say even before day one. I really think that I, so I totally agree. I think if teams know what they can and cannot do, they can move quicker. They can move faster, right? As opposed to building solutions, building tools, and then finding out midway or half way or at the end state that, oh, we can't do this, we can't do that, or we didn't

think about this. I really think that, I do think that there is something called overgovernance too. I think that that can be a problem too. I think we can, sometimes I feel like there are too many AI bureaucrats, not enough builders. And so I do think that you can go in the wrong direction as well. But I personally think, I mean, I've given this example a few times. But when cars were first developed, one of the big problems was that a lot of people were getting run over by cars. And so people stopped buying cars because, and they resolve this negative connotation. So the car industry essentially passed laws to

essentially invent the crosswalk and criminalize things like jaywalking so that they could make the technology, they could protect their field, we create a false choice between innovation and safety. And I think that you can do both if you prioritize both, right? I think that there's a synergy between them. If you look in how we've designed this program, that's one of the reasons why the supervisory agent exists, right? So we feel like you have to build a supervisory monitoring technology at the same pace and given the same resources that you provide to the technology that actually are going to provide the clinical care because they're synergistic. And if one succeeds, the other succeeds. The thing about these technologies is the averages are never going to matter.

For example, like if there's a technology that's, oh, you're 99% safe. Oh, tell me about that 1%. What's happening in that 1%? Right. If it is a child who's committing suicide, it doesn't matter what the average is. Right. And so I do think that there are never events that can happen and that this technology will be judged to a very high bar, much higher than the bar that, for example, a human clinician would have to pass. So I think, therefore, it is in the interest of this industry, it is in the interest of the entire field to be able to make sure that, A, be clear about the fact that there are going to be harms, obviously, right? No perfect technology of any sort. But we need to be focused as much on developing synergy between safety and innovation. And really, I think

advocate is an example of, you know, the government making that type of investment. So now we will just type in a little bit more on the evaluation. piece of this. I guess earlier we talked about safety and performance level. In AMIA, we have hosted the AI evaluation showcase community ourselves to see if we can bring

consensus on the best practice to do this. Right. So now, from what you can see in upper edge, how did you see the meaningful evaluation of clinical agents should look like? Are there any gap? Are there any observability

when to further building before we can even start doing this. Yeah, I mean, I think, you know, I mentioned earlier, I think that I think evaluation also has to be use case specific in the sense that it needs to be risk-based. So if you have a low-risk technology, probably needs a different paradigm. But if you have a high-risk technology,

which this would be a high-risk technology, then you need a different approach. And if I think about, you know, what is the hardest part of this program? Is it implementing AI? Is it developing the clinical technology or is it evaluating them? I will tell you evaluating them. evaluating them is the hardest part.

That is the most frontier part of this program. It doesn't mean it can't be done. And we do think that we will have teams that are able to take a real shot at this. But when I think about what is the role that academia can play, I really think focused on evaluation. I think focusing on evaluation as being one of the primary functions of AI research within academia should be that. But also, I think that there is an industry building around this.

part of my hope is that we can really create a market for technologies that evaluate and ensure safety of AI products for clinical care, create an expectation just like we have for medical devices, right, that they have been certified or we have, you know, many medical devices, we have registries that the industry pays for themselves, that we can use to be able to figure out if there is a safety signal that we're seeing. So I think that organization like Amy and your members can really, I think, create that market, make that an essential part, make that an essential piece that is needed for these technologies to be accepted and deployed. I think that that will be really important.

And I don't think it'll be easy, but I do think that it is doable. You all and your members are very well positioned for that. Yeah, evaluation is so important and also prevent to those errors. I think that is also very important because the patient safety is a bit, a bit, a bit of Obviously, no negotiable, as you mentioned. But with the AI agents, we are dealing with systems that can act, hallucinate, or even being manipulated. You mentioned that the supervisory system that can monitor or guide and catch the failure that when they build the system, they may not anticipate.

So can you give more insight and suggestions how when people develop this such an agent, how we handle those? how we're privates it, how we supervise it? I mean, so one of the things that we wanted to make sure was that the technical approach to actually developing the medical technology, the clinical technology is very different from the approach used for supervision. I think that, so the first thing I'll say is that we are really leaning into the AI judging AI paradigm. You know, I personally feel like this idea that human beings, So I think human beings are essentially incapable of evaluating AI.

That's just most of the research shows that even when clinicians know that this is an AI they're looking for hallucinations, they fail to pick up most hallucinations. That is what the research shows. And especially when you scale that up, right, when you scale up the outputs, when you're getting reams and reams of output. And these technologies are very convincing. So first of all, I will say that we are leaning into the idea that the best technology that can evaluate AI is, in fact, an AI.

approach. And I think within that, I think teams have looked at, you know, many different ways to be able to do it. I do think that part of, I think, what we're trying to do as part of this program is to make sure that the clinically AI technologies are auditable, that they are providing the information needed, not just the inputs and outputs, but more details about reasoning traces, metadata, et cetera, for supervisory agents so that they have the context needed to be able to make appropriate effective decisions about whether something should be blocked or not. So part of it, again, I think that so much of what we're doing is trying to, you know, establish what a best practice is. I think the more black box a system is, a clinical system

is, the harder it will be to value it. We're creating a precedent through this program that a black box is just not going to work. How you provide outputs needs to be standardized in specific ways rather than everyone is doing it in a different way so that we can build systems in the standard approach. I think formats that we really like for that is medlog. So we have we're asked, we've asked teams to be to consider that. I think how teams are really building out this orchestration layer is also something we've spoken to in our solicitation. So we really have asked teams to consider MCP as the standard way that they use to communicate with tools, et cetera, just so that we can, the more I think we can standardize the better.

But we're going to learn so much more. Again, I think that right now we're just getting started. every team has given us their first shot. They said, okay, this is how we're going to do it. And it's going to be interesting to see if that works. And if it doesn't, are we ready to pivot? Because the other thing that has to be thought about in this, one of the other questions that we've been thinking about from a supervisory perspective is question on latency, right?

Like if you build a system that is, you know, patients are used to having fast responses. And you want a system that is efficient because, you know, token use can balloon pretty quickly if you scale these technologies. And so we're thinking about all those different, I think all those different factors are going to be important when it comes to developing a supervisory system that you can scale. You know, different settings may have different resources. You know, in some places, deep reasoning may be important,

but in others, you may want something that is much faster, you know, that can serve as a first pass. If you're using deep reasoning for every output, your system is going to collapse. It's not, you just can't scale. It's my view, again, that can change. And I think teams are being very thoughtful. And those are sort of metrics that we provided.

Those are some of the challenges that we provided, hey, how are you going to build this and still make sure that the responses that the patients are getting are coming back on time. Those are very, very important point. You mentioned a lot, like transparency, audits, and latency, you know, all those are so important. Great to hear. We're a little bit sure on times.

I'm going to speed that combining question. do it. Right. So just a last question is about the standards and leadership and regulation, right? As we know, AI is still evolving. So many players out here, how does Appraach think about this and how did you position yourself within the USAI Action Funds for the sector-based standards? Sure, of course.

I mean, what I will say is that, you know, RPAH is not a policymaker, but I think we can be a policymaker by proxy. We can be a convener. and we can use investments in technology to be able to stress test the system. I came up with this program when I was still at FDA, and I work very closely with my colleagues at FTA. We have spoken to colleagues at NIST. We've spoken a bunch with folks at CMS, CMMI. I think that's really our responsibility in government to make sure that we are making coordinated collaborative efforts. And there's been a lot of emphasis on that, not just through the course of

this program, but really all AI efforts across HHS. There's been a real effort to make sure that We're all in lockstep, that all our strategies make sense. We also have some questions for our AIMAN audience. Most our email listeners work inside health care delivery systems, work with electronic harsh record systems or vendor companies. So when at WorldKit, funded technology is ready for the market for the work, what does an informatics leader at a mid-sized health system need to have in place to safely deploy that kind of clinical AI agent.

What infrastructure, governance, and clinical training are needed to be ready? So great question. So I think part of what we've asked our TA3 health systems to do is that once they deploy these technologies, that one of the things that they have to do is that they have to publish an implementation blueprint. And when we are looking at health systems that we want to select,

part of it is that we want diversity with regards to their operational capacity. There are some health systems in this country that have amazing resources. It would actually be much less riskier for us to work with them. But then the worry is always that no one else has those same types of resources. And so that creates a huge sort of scaling gap. Our goal is to make sure that the health systems that we end up selecting are ones that look like America, that look like the sort of health systems across this country so that everyone else has confidence in being able to deploy those solutions. And that's also something we're looking at with our performers. We're asking our performers to tell us, tell us solutions that are going

to be generalizable. So far, people have been very thoughtful. Yeah, I'm also asking another question for our ECM audience. So we are worried about the descaling these days. And it seems that the agents get more capable. People might start losing the capability to do those things. As you think about this, how did you interpret that for your clean-insurance standards? I think it's a real question, but especially more so, for clinician-facing-facing tools. In my view, because we're primarily a patient-facing technology, I'm thinking that can we actually up-skill patients, right?

I mean, I think that we will see some regression to the mean with regards to both. My hope is that these technologies, yes, there's a risk of deskilling with clinicians, but what if we can, my focus is on how can we up-skill patients? How can we make them more involved and engaged in their own healthcare? How can we educate them better? That's my focus, in part because this is primarily a patient-facing technology that we're helping develop. For when a patient is actually not even in front of a clinician.

We also have some questions regarding collaborations, field building. So ACM and Amy both sit at the same between computing and medicine. You are at Upper H and they're the place combining practitioners on both sides. What should our professional societies be doing more of and where is the field and felt? Yeah, I mean, I think I think about them from my own team's perspective. You know, I want teams, I want members in my team with complementary skill sets. If all of us are doing the same thing, then we're not actually, then that's not a good team. And I think that that is similar when it comes to organizations.

So I mean, when I think about an organization, so I really think that there's a real need for professional societies to be able to provide guidance to the community at large, whether that's the computational community, whether informatics, whether that's clinicians, et cetera, about how best to approach this. And right now I see a real gap, right? I see a real gap that people just don't know, so everyone's coming up with their own thing. R by H is one organization,

and I do want to lean more into convening and communication. But I think there is a real opportunity for an organization, like organizations like ACM, AMA, you guys are already working together, which is great. But, you know, how do you pull in organizations that are more focused on clinicians? And how do you build a diverse coalition, which has complementary audiences,

complementary skillsets, and differing perspectives, and produce communications, documents, conferences, guidelines, whatever, that are going to be broadly applied and implemented. So I think that your organizations are very well positioned

to find good partners that can really American College of Physicians, I mean, AMA, there are so many that are focused from clinicians. I think being able to bring together, just like we, we need a team. Like this is, you know, if they're only clinicians making this technology, it's never going to work.

And I would say the same thing is true if there was only just, you know, engineers and no one with a clinical perspective. So I think that's really where the opportunity lies. And looking five to ten years out here, right? So how is your defined advocate's success and how should our communities help you with that? I mean, I think successful advocate is that a patient can access high quality care regardless of where they are, what time it is, at any point, across. this country and around the world. But there's no guarantee that that's going to happen.

Right. There's no guarantee that's just going to happen. So there's a huge opportunity, I think, for us to make sure that we really don't waste this opportunity, right, to be able to transform healthcare. So I think that there's a, I feel a lot of pressure to want to do this well because the stakes are very high. And regardless of where we are today, all of us are going to be patients at some point, right? Wouldn't you want a technology like this? I mean, I would. So the hope is that we can really build a solution where clinical care is abundant and it is not resource constrained, and we do it in a way that is safe and effective. Very, very well said.

Now we are in our final round of Rappifier. I think we'll code it a day ready for the last round. Just do it. Yeah. Yeah. Decided. So now, the book that most shape how you think about medicine, but it's not one of your own. Yeah, I was thinking about this.

The book that came to mind was Attul Gwanda wrote this book called Complications. There's a book of short stories or short article that I thought were really powerful about, you know, his own experiences as he trained to be a clinician and surgeon. That was the one that came to mind. But I honestly tried to avoid reading books on medicine in part because I always worry that I always worry that. I'm going to imitate what other people are writing. So I try and not read about medicine at all in part because I always worry that I'm going to be impacted or affected by what someone else writes. And so oftentimes I read sort of fiction or nonfiction.

And there's a lot we can learn and apply to what we do through those books. Your most surprising thing you have learned since joined ARPA-H. The most surprising thing I've learned joining ARPA-H is, I think it has been how profoundly differently I think about the world and how much I think about risk and how much I feel like I had been constrained in how I thought about problems until I came to our page, in part because this place didn't exist. And so it's really surprising. I went from someone who was really conservative in how I thought about taking risk and about technology to where now I feel we're not doing enough. And we need to think about how we can actually come together to be

be able to really make the most of this amazing opportunity we have. And if advocates succeeds, what's the next disease you will tackle? Well, I mean, I think that, I mean, certainly advocate is focused on the disease. I would like to think that I think about problems in a disease agnostic way. I'm really interested in, I mentioned this earlier, but I'm really interested in thinking about how we can do clinical trials in a different way. And I think that, and that is really what my sort of focus now on is we live in the golden age of predictive modeling, I think, how do you get to a model that can get causal inference?

That's, I think, the big opportunity that we have that I'm really interested in. And then how do you use models like these to develop platforms for simulation for pre-market evaluation of agentic AI technologies? Because I do think that even though I said that we want to use familiar processes to bring advocate technologies to market, I also don't think that these are scalable and that I really don't think that they're fit for purpose as the technology itself evolves at such a rapid pace. One piece of advice you would give your rural North Carolina cardiologist. Well, I would tell them that they're making a big difference. You know, sometimes when you're kind of on your own and you're practicing medicine, you can always, it can feel like you're just

kind of stuck. Especially when you're treating chronic disease, you never really cure someone, right? I mean, you just, like, you just, and you can feel like you're just running from one point to another. But I think what I see on the other end, that doctors make such a huge difference in people's lives. And sometimes it can be hard to know. And certainly, doctors need all the help they can get. So that's what I would say to them. Yeah, I think we should end this call with one last, last question. So many of our listeners are early and mid-career.

What advice would you give someone who want to do this meaningful work in the, section or AI medicine? Well, I'm still very bullish on medicine. You know, I think that a lot of people feel that, oh, the good old days of medicine are gone. I couldn't disagree more. I think that our best days are ahead of us. I think that there are so many problems that we are.

One example I give is that everyone complains about the EHR. But the EHR was like we discovered oil, but we didn't have a car. If we didn't have a tool that could actually use this information and meaningful way that actually worked. And now we don't just have a car. We have a jet plane that's called AI, right? And I think that we have this real opportunity to really put all this data to work and get the sort of insights that we just never had access to.

So I'm really excited about this profession. I'm really excited about the intersection of AI and healthcare. If you look at the people who I feel are best at communicating about the potential of AI. So if you hear someone like Demis Havis talk, he always talks about health care. He always centers his conversation about AI and healthcare because there's so much distraction. I think being able to transform health using AI is one of the greatest opportunities we've ever had. So it's such an exciting time. It's not going anywhere.

This is here to stay. Yeah, so I'm pretty excited to be on this path. There are so many amazing young people who have committed their careers to this. That's super exciting. Yeah. And we are all here already. What a wonderful opportunity for all of us. So thank you so much, Dr. Werich.

We will put all your books, all your citations, with our own. recording when we share this with our audience. Thank you so much. We learned so much about not just this program, but also about you. Amazing. Thank you so much. It was a pleasure. ACM Bycast is a production of ACM's practitioners board, and AMIA's for your informatics is a production of women in AMIA. To learn more about ACM, visit ACM.org. And to learn more about AMIA, visit amia. Visit amia.com. and visit a community of women in Asia.