In The Arena by TechArena - Innovaccer’s Tapan Shah on Scaling Healthcare AI Safely

Episode Date: April 25, 2026

AI architect Tapan Shah joins the Data Insights podcast to discuss scaling AI in healthcare, governance, AI agents, and how technology can improve patient outcomes and reduce provider burnout.

Welcome to Tech Arena, featuring authentic discussions between tech's leading innovators and our host, Alison Klein. Now, let's step into the arena. Welcome in the arena. My name is Allison Klein, and today is another Data Insights episode, which means I'm with Janice Naurowski. Welcome back to the program, Janice. Hi, Allison. Thank you very much. It's great to be back, always. So, Janice, why don't you tell us about? what topic we've got cued up today and who we've brought along as a guest. We are, as everyone is, always, always, all day focused on AI.

And more and more folks out there want to understand how is AI impacting health care and how is health care being impacted by the innovations of AI. And so today, we actually have Topan Shah with us, who is an AI architect for Innovacer. So welcome to the program. It's a pun. And it's really nice to have you. Thanks for having my Janice and Alyssa. Really happy to be here and sharing some of my experiences.

Now, Tappan, you've got an amazing title. Architect and a Gen-Tic AI work group lead at, I think it's Chi, the Coalition for Health AI. Can you give me a little bit about your role and how that fits in with the topic of today in terms of the evolution of AI into enterprise? Absolutely, absolutely. So I'm currently the AI architect at Innoveser and what my role entails is creating AI systems and like more recently AI agents that work into actual healthcare environments. Not just in a finite setting, in actual enterprise systems wherein they get task done, they get work done and create well and affect patient and provider of course.

So that is my fundamental role there. it's actually very interesting to think about. And I would like to start with an example because that actually just highlights the challenges which we face. So for example, let's say you are creating a clinical load or we have an AI generating a clinical load for a cardiologic practice. Now, that works great. It works a student pilot. We have deployed that. And now when we move to, let's say, oncology or orthopedic, the same system probably does not work as well.

right? Or even if you go to a different practice, a different EHS systems or a different clinical practice which might have their own requirements and expectations, it doesn't work. And that highlights the challenges of how scaling AI into enterprise healthcare and healthcare systems. And my role here at Inovacer is fundamentally enabling so that we can go and use AI systems from a pilot phase to more at scale and production. systems. So that's my fundamental role. And what I've also learned over the past one and a half two years where I have been working in healthcare is AI is not, especially AI in healthcare, is not about what model are you using or the model architectures or what inference engines are you using.

The hard problems, the real hard problems are how do you ensure that the agent which we are developing, they have the right axis, right? they are creating safe decisions. They are able to understand what the limitations are, an answer accordingly, right? Or how does it create sufficiently transparent and explainable generations or recommendations so that even a skeptical clinician would accept? So what we realize is that scaling AI over enterprise system

is less of an AI problem and more of a system design problem. Interesting. And with that, background, Can you tell us a little bit about the types of AI initiatives, health care organizations are prioritizing right now? And what's changed in how those initiatives are being validated compared to, say, a few years ago? That's actually a very interesting question.

So the big shift over the past many years, like over, let's say, 10 years, especially since the advent of large language models, the big shift is from building models to building agents. and that has been, I would say, a seismic ship which has really revolutionized the industrial subsets. Historically, AI would be used to build creative models, for example, for human detection or for detecting oncology, onset, etc.

That was pretty easy to validate. You would build your models, you would have an holdout set, measure your accuracy on that set, deploy, and you are good to go. Now, agents are different. In agents, they pull information from different data sources. They use different tools or executable softwares. And they use and they combine this to create or perform a particular task. Now, in this case, that is not often a single source of proof.

So it becomes very difficult to validate because under the same scenario, using the same tools and same data, clinicians can have very different interpretations for that. And we have seen examples for that. There can be cases of missing data. There can be cases where certain users are not allowed to use certain tools or certain softwares. So in this, the challenge becomes, how do you ensure that the agents which you are building are safe, which is when it encounters something which is new or novel, it handles the scenario gracefully? And so you are not only computing an aggregate accuracy or a signal number.

you are computing or measuring the performance along a rubric of dementia. And essentially the goal here is to make sure that the agent behaves safely and threatened me even in a novel scenario. And that has been the fundamental ship when it comes to over the past few years of how we have moved from building models to building agents and evaluation and validation of that agents has fundamentally been trying to get. that robust and predictable performance in novel citizens. Yeah, Toppa, we've been talking to a lot of AI practitioners across verticals,

and I think healthcare isn't unique in the challenge that we've had a lot of successful pilots and enterprise, but very few examples can point to scaled outcomes. So from when you're talking to different health care providers, what are you seeing around the primary friction points once AI moves beyond a single department or use kits. So let me reframe your question, listen, and I would try to answer it slightly differently. I will recreate the example, which I do initially.

So you have a clinical node generator, and then you use the AI node generator to generate clinical nodes for, let's say, a particular practice for cardiology. The same AI, when we want to move it to a different practice or a different specialty, What we have often seen, and that's true across the board, is there are some gaps, right? And that scaling piece, it typically works very well in a very content environment to scale it across multiple HRs, to scale it across multiple specialities.

To scale it across multiple personas is where the real challenge comes in. And that, I think, has been the primary restriction point when we talk about scale. bring AI from pilots to enterprise systems. Another example, when we are building a pilot and we are building an AI agent to perform a certain task, in a pilot phase, we can provide a wide variety of access to different data sources at the pilot stage. But when we go into production, there has to be a better access control which has to be built in.

It cannot be bolted in. It has to be built in into this system. For example, if let's say we have a cardiology node generator, that cardiology node generator should not have access to any psychiatric or behavioral psychiatric notes for the patient. So that nuance and that level of access is very important that we building and not voting. And that is the fundamental challenge when we talk about scaling DL is how do we do that in a responsible and a governed. manner so as to make sure we are not crossing lines. So just to summarize that, some of the technical decisions which we make doing a pilot, if something doesn't work, we can make some changes and we can make it work.

When it goes with your enterprise, it should have that ability to address new scenarios or scenarios which are not very clear in that scenario. If the agent is able to work, then we can confidently say that, look, now we have moved from a pilot face, enterprise solution is, and that's where we create the real value. So let's talk a little bit about governance, right? So when health care leaders hear governance, you know, they often think of like controls, constraints, but in practice, what does effective AI governance look like day to day for an organization to really successfully deploy AI at scale?

That's a very valid point you bring in Jennings. And that's a mindset ship, which. the leaders will also need to go over that journey, that AI governance is not constrain. In fact, it's enablement. And the way I think of AI governance or governance in general, it's a small example, but probably it shares my hypothesis here. It's like a constitution, right? You can use your constitution to either be a constraint and a binding document,

or you can use that constitution to be the building blocks. for doing a lot of useful things. So that is what the governance actually looks like. Now, to answer our question, what does a day-to-day activity in a governance look like? Multiple levels. For example, let's see if you have an authorization agent with adjudicates on prior authorization,

we see over a period of time that we get around 70% auto approval rate. Suddenly one day we observed that the approval rate has increased to 90%. And we observe that trend over a pair of. So day-to-day governance actually means is to detect this change, right, have a mechanism where we can identify what triggered that change. And for that, we need to understand the complete decisioning process. And that's another thing which is historically, AI needs to be all about the prediction, your recommendation.

You go to a Netflix. This is the recommendation or the prediction I'm making for this particular problem. When it comes to the agentic AI world, you are not interested in the final recommendation or the adjudication. You are interested in the entire decision graph which was taken. What data was used, what reasoning was used, why was this particular toll use? Why was this particular decision made? That entire decision graph.

So the fact that we lock the entire decision graph, the fact that we can detect a deviation, from normal and the fact we can detect that deviation to a change in the decision graph is what and actually governance would look like. Now, in this case, the possibility or the probability would have been, there had been some update in the peer guidelines or some update in the pair rule which was communicated verbally or using email for which we did not have the right insight. And in that case, the fact that we could figure out the dishing process, the fact that they could detect the deviation

and then the fact that we could identify the road cause and make appropriate modification is what a detailed governance. Other example of data governance is cost. Because the fact of the matter is that AI is like right. The way it is variable is very token-based which means it is expensive. Each time you call an AI model

you spend some sense and you spend some dollars. right? And if your AI agents are not configured correctly and if you don't have the right guardrails, the cost can literally glow up. So that is another part of governance, making sure that you have the right structures in place, the right guardrails in place to avoid any escalation in costs which are unheard of or just unknown. So that is another very day-to-day governance type of example which I can think of. Another example again is always monitoring the outputs generated by the agent, having a mechanism to continuously monitor there and flag any anomalies.

So all these structures, right, in technical terms, we call it observability, tracing, and monitor, as well as the corresponding policy and the cost structure, together they are fundamental parts of a day-to-day AI governance, which is in this. Now, you know, I think that the world has been captured at how quickly it, AI systems are becoming more autonomous and headlines have been filled with that. I don't want to make the entire podcast about that topic, but I do have a question. As we see the evolution of AI in real time, how are health care organizations evolving their approach for accountability, especially when decisions involve both clinical and administrative impact?

That's a very valid, but a very top question, Alison. I think it's one that we're grappling with collectively, right? And I'll tell you, so let me first explain why it is stuck, right? Why it is important. And then I'll probably go into what current mechanisms which we have, right? But to answer our questions, upfront, we don't have a perfect solution of how will that accountability be there when it comes to agent taking decisions. There is a lot of ongoing research happening on that space.

There is also research not only from academic and scientific circles, there is also research happening from the legal circles. What does an agent mean? Will the same laws apply to an agent? Will there be a lot? I'm not an expert in that friend, so I'll probably not comment there. But I'll tell you how we address the problem as of now. Because it's an important problem, right? The way we address that problem is whenever a decision is taken, that decision, that decision.

needs to be locked. There has to be a traceability on every decision which AI takes. That is the fundamental, I would say, bad rock on which all the accountability structures will be built on. Second, all the agents should have the right access control, the right permissions, right? They should only be accessing certain things which it is allowed to do. And that also has to be built in. So combine with the fact that we have a proper access control and combining that with the fact that all decisions are well articulated and long, those become the backdrop of how accountability will be there.

Now, the third component which is very important is human oversight. So as of now, for all of clinical use cases where AI agents are things, And many of the administrative use cases also. There is always a human oversight. The human can always override the decision. They will always be able to override particular decision with the AI takes. And that will be sort of law. And again, I don't have the perfect answer for your question on that of the accountability.

But what we do have is as in when the legalities of it and the governance of it, it evolves, the backdrop of it, right? Access control, logging of the data, and humorite. These three will be the bedrocks, which will then make sure that any change in the cardamination structures, those are handled and those are managed by those baton tracks systems. You know, a lot of AI success in healthcare, right? A lot of success stories really focus on how much time did I save or how much cost did I reduce. What metrics would you say health care organizations used to quantify kind of long-term strategic value beyond just kind of a short-term efficiency? Yes, I think right now a lot of focus is on the short-term efficiency case.

But in terms of the long-term strategy, I think the two main outcomes which we are looking for, which are the holy grade, are improved patient outcomes and improved patient outcomes. and improve provider outcomes. And if this is a path towards achieving that, I think we are probably on the right path and this is what we should focus on. So when it comes to patient outcome, it's a combination of cost of care, quality of care and the speed of care.

Like, I'll give an example, para authorizations. There are cases where it can take offers of four to three weeks for a prior authorization for a procedure, right? Can we get that down? And that actually needs to delay in the. care, right? So can we bring that down to, let's say, a day, less than a day, a few minutes, right? And that actually impacts patient outcomes. Cost of care. How do we reduce a lot of

administrative burden, which can then reduce the cost of quality of care? Are we identifying the right quality gaps, the right care gaps, and are we addressing those? So it's a combination of quality of care, speed of care and cost. When it comes to the provider world, it's possible to quantify then. And again, there is some debate on what's the best way to quantify there. The way I see it is, okay, can we help our providers to spend more time on things which they are actually good? Which is taking care of patients, which is understanding the patient condition and providing

the best possible care, rather than spending an in-obbitant amount of time on administrative tasks. Can we reduce their stress levels and burnout, which is an actual problem out there and there is significant research out there which indicates that provider burnout and provide stress levels is a big problem. So can we reduce? So essentially, if we can have this AI systems which can make providers engage and interact more with the patients and at the same type, reduce their fatigue levels, reduce their stress levels. I think those are the final outcomes which we should be looking for. And again, it will be a step-by-step process, right? It will be phase or a period of time. But unless when you talk about long-term goals, these are the two

goals which I believe AI can play a set in the data already. So, so I think that, you know, one thing that I think about is healthcare organizations are pretty complex. You're touching with an AI initiative, potentially multiple audiences, including clinicians, IT operators, business operations, and compliance managers, all that have maybe some competing and overlapping priorities. How do organizations align these groups around shared ownership for AI so that AI can actually become enterprise capability holistically? You actually bring a very good point, Alison there. And I'll share an example, right?

So even in a typical system, you will have a security, who will want a very high degree of access control and networking. You'll have a compliance team which would want proof for almost all access being made. You would have an operations team which would want a very fast response time, right? And you would probably want a clinician or a care manager who would want your solution to work across multiple E. Yes. Okay, these are diverse requirements. Many of them competing, many of them alignments are diverse requirements. So the key goal of an architect is to make sure that when you design your TI system, when you design your TI agents, all these requirements are thought of and baked it into the system.

Because if we try and bolt it over it becomes a challenge and then we have observed that something fails, something does not fail. And then, we sort of end up creating friction points and creating even bigger silos. So at the time of engineering the systems, at the time of architect in these systems, if you can make sure that all these requirements are well architected, then you sort of create that AI system, the AI layer, which has reduced friction. I cannot directly eliminating friction, but it reduced friction. And I'll give you an example.

So as part of governors, and I will again think that those three observations, observability, tracing and monitoring. If we do that well, if all our agents are, we have great observability, we are all the perfect tracing, we have great monitoring, we can help each and every persona well cater. We can have the security team. They can do what it's whenever they require. We can have the compliance team who can make sure that we are complying with the right legal

print. We can have the operations team who can get observability into what is a our latency, how much is our response time when we are responding to a particular task. We can have clinicians and the same framework would also provide understanding to the clinician that, okay, this agent works well on this scenario

this agent does not work well on this scenario. So again, going back to my point on observability, tracing, and monitoring, that is one of the key example of how well architected systems will ensure that all this competing requirements or even multiple requirements in sign-note organizations, they are all in care, and they are all on board when this agent goes into production. So, Thubon, have you kind of seen examples where governance exists on paper but doesn't translate into outcomes?

And how do you bridge the gap between the governance theater and, a structure that actually scales? That genus is actually a great question and something which is very close to my heart, right? And actually my heart is close to my manager's heart and is close to our organization. Just having policy frameworks and governance frameworks on pages and papers and slides and dogs,

it's great, it's important. But having said that, that has limited impact, right? what really has the impact is how do you integrate in your day-to-day systems? And again, folks with background and software engineering, there is a term called CICD, right, this continuous improvement, continuous development. How do you integrate this governance concepts as part of your CICT, your Agile systems, your CICD systems, which essentially means is that it is not an overhead. It is not something which is fit in from top.

It is part of an existing work. So it's a day-to-day activity. Now, let me give you an example for, let's say an agent made a decision last Friday. If I am in a position to within a few minutes, query a database, query a system, and get all the related aspects for that agent decision, what was the data use, what was the model use, what was the reasoning use, how confident was the most. model when it made the decision. If I can get everything within a few minutes, then I can safely say that I have that needs good working system.

But if it takes me a week to get that information, then I'm sorry, that is not a working governance system. It's probably that in the paper, but it's not there. And that's what we strive towards. That as a company, as an organization, we have been striving towards a scenario where any decision which AI makes, it should be available to us within a few,

clicks and sequence, right? Similarly, let's look at another example. And again, there can be documents and documents of pages on what access policies and access control should look like. But if I cannot within a few minutes get what were the tools accessed by this particular PI agent, then again, that's not very hard. Or if the answer is that we will review it in a quarterly meet, that's not the answer. It should be a. app. So when I talk about actual governance or actual governance in the system and not just governance theater, it's about whenever I want to know about any decision rate, any access of a database, any access of a tool which an AI has made. If I can get that within two clicks,

within few queries, then I can say that we have architected a well-designed or well-thought-out AI system. Now, looking ahead, how should healthcare organizations design durable governance models like what you're talking about, that won't need to be reinvented as AI capabilities evolve. I love what you've said about integration into CICD, but as the landscape changes, does that just evolve and push forward the CID CD pipeline? Or how do you see that evolving? Great question again, Alison. And the way I think about the governance is it cannot be tied into an implementation. Like, if there is a governor's policy which says that you should always use a rag retrieval augmented generation or your confidence score should always be computed using this,

then that's a recipe for tech tech, right? Because things will evolve and what implementations which we are using today, they will no longer be useful. Right. So the key aspect here is to understand tied up the governance with the interface rather than with the implementation. I'll give me an example. So probably three years ago, I would control data access at the database level. Today, I control it via API level. So the technology has evolved, but the fundamental principle of access control steel limits. So as long as we can get the right principles and make sure those principles are well highlighted, very documented.

and not tie up our governance on implementations, I think we should be good there. And again, many of these principles are not AI specific. So observability, decisioning, tracing, monitoring, these are ageal concepts, good software engineering concept, which have always been there, right? When you mean towards an AI or an agentic AI world, the interpretations become slightly distant,

but these are basic system design concepts. And that's the reason in the initial part, I think I remember I mentioned. When you scale AI agents into an enterprise world, it is less of an AI problem. It is more of a system design problem. And as long as those fundamental aspects of system design, it comes to robustness. For example, let's say you call a model API, the model API fails. That model API you can today use open. you can today use Claude, you can use your own model.

But the fact that you should have a robust retry mechanism, a robust fallback mechanism, right? Those are fundamental system design problems, and those still written valid. So as long as your governance implement are based upon this solid system design principles, and you don't tie your governance with specific implementations, I think we should be good there. This has been an incredibly rich conversation

Tapan, we appreciate it so much. Where can our listeners go to learn more information about everything that was discussed today? Absolutely. So for some of the compliance fundamentals and AI governance stuff, there is a special publication by NIST, right? It's 8006. It's a great resource. Even HHS has published newer guidelines for HIPAA for the AI world. Again, that's a good starting point. Chai, Coalition of Health AI, for which I am the agentic AI lead there. It has some very good resources on agentic AI and agentic AI governance. It's a great resource that. So I would also want to highlight so. In a way, sir, we last year released an AI platform called Gravity. And if you go to some of the blogs we have released as part of that, that

highnecks how deeply we have thought about governance, security, compliance and we have very late bottoms up by design because many a time what we see around is governance and aspects of governance

become afterthoughts and they are bolted egg but we have fundamentally slipped the egg and we have thought about those aspects first before even we put out the first stage because we know how critical healthcare and what levels of compliance and security are expected when it comes to health care. And we want to make sure that our platform enables that.

So that's another great resource we should be comfortable. Thank you so much for being on the show today. It was so fascinating to hear your journey thus far in the healthcare AI arena and where we're going. I love that you shared your expertise. I'm sure Fetz will find it useful as well. We'd love to have you back on the show sometime. Thank you so much. It was great talking to Alison Janus.

Thank you. Thanks so much. And with that, that's another episode of Data Insights. Thanks for joining Tech Arena. Subscribe and engage at our website, Techorina. All content is copyright by TechRena.