How I AI - How this PM streamlines 60k-page FDA submissions and saves millions with Claude, Streamlit, and clever AI workflows | Prerna Kaul

Episode Date: July 14, 2025

Prerna Kaul is a product and platform leader who has spent over 14 years turning machine-learning research into consumer and B2B products at Amazon Alexa, AGI, Moderna, and now Panasonic Well. In today’s episode, she explains how she’s using AI to slash some of the most time-consuming, expensive tasks in life sciences—from generating 60,000-page FDA submissions to crafting communication frameworks that help product managers navigate complex stakeholder dynamics. Her innovations are saving millions of dollars and helping lifesaving treatments reach the market faster.What you’ll learn:How Prerna built an AI system that automates the creation of 60,000-page regulatory documents for the FDA—reducing a process that took 4 to 6 months and 20 specialists to just minutesA step-by-step system for detecting and redacting PHI (protected health information) in clinical trial data using ClaudeHow to build user-friendly interfaces for non-technical colleagues using Streamlit to democratize AI toolsHow to use Claude’s prompt generator to create powerful communication frameworks that help PMs navigate complex stakeholder situationsWhy transparency about AI costs is crucial for gaining organizational buy-in and tracking ROIA practical framework for approaching AI safety and ethics in highly regulated industries—Brought to you by:CodeRabbit—Cut code review time and bugs in half. Instantly: https://lovable.dev/Lovable—Build apps by simply chatting with AI: https://lovable.dev/—Where to find Prerna Kaul:LinkedIn: https://www.linkedin.com/in/prernakkaul/—Where to find Claire Vo:ChatPRD: https://www.chatprd.ai/Website: https://clairevo.com/LinkedIn: https://www.linkedin.com/in/clairevo/X: https://x.com/clairevo—In this episode, we cover:(00:00) Introduction to Prerna(03:01) The FDA submission challenge: 60,000 pages, months of work, millions in costs(05:20) Getting started in Claude: from prompt to production-ready prototype(10:13) How Claude selected the right models for medical entity recognition(12:04) Using Streamlit to create accessible UIs for non-technical users(16:04) Detecting and redacting PHI in unstructured clinical notes(18:44) Generating the Common Technical Document (CTD) for FDA submission(21:54) Tracking and displaying AI operation costs for stakeholder buy-in(24:38) Real-world impact on vaccine development timelines and costs(26:12) Creating an AI communication coach for product managers(30:22) Training Claude on classic literature and persuasion techniques(31:53) Analyzing a complex stakeholder scenario with multiple competing priorities(34:40) Getting personalized communication strategies inspired by tech leaders(35:40) Summarizing strategic approaches(38:26) Conclusion and final thoughts—Tools referenced:• Claude: https://claude.ai/• Streamlit: https://streamlit.io/• Anthropic Console: https://console.anthropic.com/• Claude Sonnet 4: https://www.anthropic.com/claude/sonnet—Other references:• Claude project chat (AI Product Management Stakeholder Challenges): https://claude.ai/share/caba4ab0-b28a-480c-8633-71920b12999e• XML: ⁠https://www.w3.org/XML/⁠• Python: ⁠https://www.python.org/⁠• RegEx: ⁠https://regex101.com/• Moderna: https://www.modernatx.com/• FDA: https://www.fda.gov/• Project Gutenberg: https://www.gutenberg.org/• FDA Biologics License Application: https://www.fda.gov/vaccines-blood-biologics/development-approval-process-cber/biologics-license-applications-bla-process-cber• Protected health information (PHI): https://www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [e-mail].

So you are working in the life sciences on really health impacting vaccines and treatments. And with all that amazing investment and scientific work also comes a lot of paperwork. We had to develop a nearly 60,000 page document. It would have taken about four to six months of effort and nearly 20 specialists, not to mention the millions of dollars spent. Where did you start? I gave Claude the problem statement, the pitch and a demo. The thinking I had in mind is that Claude is a software engineer and I'm talking to them

and trying to tell them why it matters like any good PM would and trying to tell them what end product we want to produce as a result. And not only did it fully understand what I was asking it to develop, it created a little set up instructions markdown file for me and it gives me all of the set of instructions and all of the set of instructions and all of the tasks that it does and its capabilities, which is super handy. Were you really able to shift the cost there? Did you see an impact here? We did. We did.

And any kind of cost savings you can generate have a direct impact to the bottom line. And in addition, if you're saving on time, you're bringing life-saving vaccines in the hands of people who actually need it. Welcome back to How I AI. I'm Claire Vaux, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today we have Prairna Kahl, who's taken years of machine learning,

experience and help develop some of the products you know and rely on every day at companies like Amazon Alexa, Moderna, and Panasonic. She's going to show us how she uses AI to accelerate drug discovery and vaccine approval, something that takes tens of thousands of pages of documents, dozens of people, and months and months and months of work. She'll also show us how you can use Jane Austen, Dale Carnegie, and some AI to manage your stakeholders just a little bit easier. let's get to it. This episode is brought to you by CodeRabbit, the AI code review platform, transforming how engineering teams ship faster with AI without sacrificing code quality. Quality code reviews are critical, but time consuming. CodeRabbit acts as your AI co-pilot,

providing instant code review comments and potential impacts of every poll request, beyond just flagging issues. CodeRabbit provides one-click fix suggestions and lets you to find custom code quality rules using AST grut patterns, catching subtle issues that traditional static analysis tools might miss. CodeRabbit brings AI-powered code reviews directly into VS code, cursor, and windsor. CodeRabit has so far reviewed more than 10 million PRs, been installed on 1 million repositories, and has been used by 70,000 open source projects. Get CodeRabbit free for an entire year at Codderabbit.com. AI and use the code how I AI.

Prairna, it's so nice to have you. I'm really excited about your workflows. Thank you so much for having me. Very excited to be here. So you are working in the life sciences on really health impacting vaccines and treatments. And with all that amazing investment and scientific work also comes a lot of paper work. If I am correct. And so in addition to doing all this amazing research, you also have to document and submit a lot of stuff to the FDA and other regulating industries.

And I am sure that is a total pain. So tell me a little bit about the problem that led you to building some of these AI workflows. Well, yeah, firstly, thank you so much for having me. And you're totally right. With life sciences, I think the challenge across all parts of the organization is a lot of paperwork, a lot of regulations. and going back and forth with these regulated industries could take months of time, not to mention all of the people resources involved and delays in getting, in some cases, life-saving drugs and vaccines to market that people need.

For example, when I was working at Moderna, we were developing the INT cancer vaccine, which is necessary for folks with skin cancer. It was sort of a new niche concept that had come out. and we had to develop a nearly 60,000 page document, which is known as the, it's a technical jargon, but a biological license agreement, which took around four to six, it would have taken about four to six months of effort

and nearly 20 specialists, not to mention the millions of dollars spent across the organization and getting this document together and providing it to regulated authorities across the world. Very labor-intense, very time. time-consuming. And that's where I kind of came in. I was a Gen. AI product manager there and really came into think through how can we reimagine this using Generative AI? This was back in 2023 where I came up with the solution, so a couple of years back. And I kind of evolved it over time to kind of specialize in certain areas as well. I'd love to show what we came up with.

So it sounds like you created a really high-impact tool in 60,000 pages is pretty intense. What I love about this workflow is you're going to show us actually how you use Gen AI to build this Gen AI solution. So like all good product people, you started with requirements. Can you just show us your process from you've identified this problem, which is we've got to create this massive document. And it's going to take millions and millions of dollars and tons of people to actually having this production thing that your team and colleagues could use. So where did you start? Like every good PM started with the requirements. And I actually didn't, hadn't fully formed all of the ideas initially because I think that with the complexity in the domain knowledge required, it took a few iterations too. But I'd love to showcase sort of where we started with very simple pitch and then go through how we actually evolved the idea as well.

So I gave Claude the problem statement, the pitch and a demo. The thinking I had in mind is that Claude is a software engineer, and I'm talking to them and trying to tell them why it matters like any good PM would, and trying to tell them what end product we want to produce as a result. So I've included statements such as, you know, this is Stefan Barselyl, who is Baudenna, he was a passion project, he's very excited about it, here is the cost savings you can get, here's why it matters and the impact. and then also spoken about the demo that I want to showcase and how we could sort of iterate and build on it. And Claude came back to me with not only did it fully understand what I was asking it to develop. It created like a little set up instructions markdown file for me. And it gives me all of the set of instructions and all of the tasks that it does and its capabilities, which is super handy.

not only did it do that, it also gave me all of the code and all of the different aspects to the code as well. So I kind of started here and started thinking through what do I really need to do to make this a production ready system. And there were two things that were extremely important in the process. First is because the BLA is a structured document, it is XML based, it has certain formats, I needed Claude to successfully, in a very strict and structured way, take data and structure it into a certain format. So that was a very strict requirement and that's not super easy to get with some of these models as well. The second was that because it is clinical data, it's highly regulated and being able to detect PHI, redacted information was extremely important so that it could correctly redact that and then not have that included in the actual.

BLA that we would be sending to any authority because it's patient data essentially. So the reason why we choose Anthropic Clod is because of the reputation around safety and alignment with human values as well. And we formed this partnership with Clod and really focused on some of the PHA and the safety aspects to make sure that not just from a compliance standpoint, but also as we think about alignment, we were taking care of those use cases. So this is what it came up with. It did a really good job, and I was able to get some code running as well. And I'd love to show you what it came up with.

If we can go back to the requirements really quickly, there's just a couple of things I want to call out for folks that I think is really interesting. If you go to the step-by-step instructions, what I love about it is if you go to the top, it not only gives you sort of how to technically write the code, but something I wanted to call out at the bottom is it actually gives you, you a pitch narrative to how to run your demo. So what I think is so interesting about this specific prompt is you gave it kind of pretty loose requirements. It's only a couple paragraphs up top. And it gave you a whole set of things. I'm sure many of which you weren't anticipating. I wouldn't

have anticipated getting a demo script from from Claude when I'm asking it to build Python. But it gave you that. I thought that was really interesting. And then if you scroll down in the left side, something else I'd like to call out for folks that I think. think is really interesting is it gave you these kind of like component names of different technical pieces of the product. So like coming up with mock data, doing PHI reduction, all that kind of stuff. You know, you as a software engineer and a product person, how long do you think it would have taken you to come up with this full set of just this first one shot prompt requirements before? At least a couple of days, at least about a week.

to really think through everything. And then in addition to knowing you wanted to work with the anthropic models because of safety and alignment, were there other libraries or other models that you knew you wanted to use? Did it pick the right ones? Like how close to the technical implementation did it get? In particular, with the redaction piece, it would have taken a couple of data scientists to figure out the right solution and work with us. So data scientists as well as software engineers to work with us and figure out what

what's the right solve for this? How do we implement it? How do we test it to get a certain synthetic data? So I think where it really excelled is in picking the right models. And I have some amount of machine learning knowledge, but it would have taken me a couple of days of study to figure out, okay, this is the right model for medical named entity recognition, and this is how you go about it.

But it did such a great job with very little information up front on the domain itself. It was able to absolutely pick the right models and give me the code and then implement it and show me the confidence code across that as well. Yeah. And if we look at the code, I just have to laugh because, you know, many, many years ago, I think I have lost hours and hours and days and days of my life to writing Rudge X. Like, I used to spend all my time writing Rudge X. Terrible. That didn't work. It was just awful. And so one of the kind of miracles of AI I see is it reduces this toil in technical tools you know you need to use that have documented patterns,

but they're just so hard to research and get perfectly right out of the gate. So this was my favorite part of seeing your code up, is seeing all that code just pre-populated for you. Yeah, that's such a common and shared pinpoint. Regix has been a, yeah, it's a bean of any software engineer's existence. Yeah. So, okay, so it created this Python script for you. And you said you scrolled through it and it looked, you know, pretty correct.

And then you decided to deploy this on Streamlet. Can you tell us a little bit about that choice, why you pick that? Yeah. So I love using AI to solve for some and automating my own work and solve for my own sort of day-to-day problems, either be it related to work or something I'm trying to work in my side project so personally. But I think the real unlock, at least I've found, is that if I can create something, of value for other people. And then especially as they are non-technical folks,

making a UI and making something generally available and hosting it locally has proven to be a really, really big unlock. Both career-wise and also working with different colleagues, they find that to be really valuable because they might have something in mind and I have all these skills and I can sort of offer that. So that's actually something I've been doing in my current organization as well

where I recently recently, created a Google add-on to convert PRDs into Jira format. And that was a lot of fun. And the product managers especially were very excited about it because they don't necessarily like having to write requirements twice, one in a PRD and then also in Jira. And we found a good way to circumvent that

and do it in a matter of minutes. So I did that. And then the streamlit idea, I also did in a couple of my previous roles and my current role as well, give non-technical users access to all of these tools. Yeah, and just for folks that don't know, Streamlit is a really easy way to put a pretty simple UI on front of like a Python kind of file and set of APIs.

Exactly. So Claude created your PRD. It gave you code. You were pretty happy with the code. How do you get from us looking at this code right now to a product that your team could actually use to create these documents? I'm going to run for you the actual Streamlet app. So I'll take you to Terminal.

I essentially run it from the command line and it opens up a web app. It takes me here. It has preloaded an API key from Claude, which I've provided in the code. So I've hidden some of those properties. And I'm going to generate synthetic data for us. And I'll talk through, and I'll go through the step as well because it takes a bit of time. And I'll talk through what it's done. So as I mentioned, there's this format called a common technical doctor.

as part of the BLE. There are several models, about five or six different modules, and it has a structure around it. So there is a study report that has a tabular listing. It's supposed to have very specific points across the data from the clinical trials, and then different literature references as well. Did Claude know about this format kind of from just general knowledge, or did you have to provide the format to it? So it searched up the common technical document format for me, and then it figured out the right module to apply to my use case. Great. That's awesome. Yeah. It's doing pretty well.

So I'm running here what is, I'm going through the different, we have different participants here, and there are some clinical notes per participant. This is all clinical trial data, very, very sort of common that you would see. in any trial that runs. And by the way, the trials itself, they could run in the order of billions. So redoing a clinical trial is very, very expensive. And so as much as possible, we want to give very accurate information to the FDA so they don't ask us to redo anything. And what I've done here is I've de-identified some of this patient data to remove any sort

of names, date of birth, any information that could be associated with a person. And one of the challenges I'd love to call out for folks that maybe aren't seeing this is you generated some fake data so you can validate it works. And it's not only this tabular data that has patient ID, ethnicity, whether they were in the control or otherwise, but you also have these free form clinical notes where there may be embedded PHI written in by a clinician or a study administrator. So it's not as simple as saying, just find all the data birth columns and redact that. you actually have to go through this unstructured data and find potential sources of PHA. Is that part of the challenge? That's exactly it. It's a complex problem usually, you know, machine learning doesn't always do perfect job there, but Claude immediately identified the right model to do this.

Great. So, okay, so you have this clinical data preview and then you're going to show us how you de-identify it. So you're not sharing that with folks that don't have access to that help data. Exactly. So with this synthetic data, I clicked on this button to detect and redact PHA information. And it scanned through all of the rows. And it found for me all of the PHAs. So here's what it came up with. So see in the clinical notes, it found that there was some dates, names associated.

There was some birth date information as well. And the remaining, it was able to keep sort of in a de-identified way so that I can include it in my reports. Great. And then how would this have been done in the past? I think you said there are software engineers that would have built these models to redact the data. Was there any human in the loop at any point in this as well? Like how was this done before? For the past couple of years, we've been, most pharma companies have been using machine learning for this. But these, these projects, they end up, because it is PHI and there's very high bar, they end up taking months of effort to get right. And especially when clinical trial data might not look the same

for every drug. As a company like Modena was trying to scale, we were trying to launch nearly 50 drugs, bring 50 drugs to market in clinical trials and then 15 to market in the next three to five years. That problem was exacerbated. So thinking about coming up with new machine learning, to solve different problems across each of these vaccines. It's a matter of scale and time, really, and we really needed to find a better way to do this. Great. Okay, so we've got redacted health data.

You have all your great billions of patient data points. What's next in this flow? So the next thing is to generate this common technical document. And I'll show you a small preview, just for the purpose of this demo. So what it did is it took all of that and it gave me a summary of the different data. It was able to give me a synopsis, which is what the FDA looks for, the study participants, the counts, the age, all of the statistical data points as well as some of the specific medical terminology that they're looking for.

So it kind of summarized everything. It gave me this. And for context for folks, this is typically done by a medical writer. and their whole, I mean, I've worked very closely with them. It's so time-consuming because they not only, they can do a pretty good job of generating like a first draft summary, but they have to go back and forth with their stakeholders for weeks sometimes on a single module to come up with the right set of criteria

and getting everyone to align, all of their stakeholders to align, is very, very time-consuming. 20 specialists, as I mentioned, are working on it at any given point in time. And as part of what some of these, the chat GPTs of the world and Anthropic can provide, there are also common projects and common environments where all stakeholders can kind of collaborate and work together on these summaries, which is so much easier and so much better than working, let's say, in a sort of a shared document and just providing comments. Yep, great. Okay, so it generated from that data this very strong, I mean, it looks official to me.

I'm not, I'm not part of the FDA, but, you know, it's a summary of statistics, it's summary of the methodologies, it's summary of the impact. It's pretty long and detailed. It looks like you can download it both to a TXT form, like a doc form, as well as this, as you mentioned, proprietary XML format that's required. And I'm presuming the Claude Code just gave you two buttons to make that happen. That's exactly it. And I know it's a little bit proprietary, but I cannot tell you how excited I was to see this button. This episode is brought to you by Lovable. If you've ever had an idea for an app but didn't know where to start, Lovable is for you. Loveable lets you build working apps and websites by simply chatting with AI. Then you can customize it, add automations, and deploy it to a live domain. It's perfect for marketers spinning up tools, product managers prototyping new ideas, or

or founders launching their next business. Unlike no code tools, Loveable isn't about static pages. It builds full apps with real functionality. And it's fast. What used to take weeks, months, or even years, you can now do over the weekend. So if you've been sitting on an idea,

now's the time to bring it to life. Get started for free at lovable.dev. That's lovable. Dot dev. Okay, so if we could just take a halfway point of the podcast. The two things you were most excited about and I was most excited about, I was excited about the Rejects, you were excited about the XML. This is a very nerdy and very specific how I AI podcast. Okay, so you've generated the doc and then I love this last part. So let's talk about

what's this last piece of the product. What's super interesting is I have worked in a couple of different organizations where as we are trying to scale different AI use cases, cost comes to be a factor because not just token costs, but also acquiring licenses for these models, and then running it as the number of consumers or the users of the actual use case scale, the dollar cost also scales proportionally. So one of the ways in which we could potentially continue to launch and continue to build products around this is by actually being transparent about the cost associated with the operations.

and, you know, generally, one of the stakeholder arguments that I've previously received pushback is this is going to be high cost, can we find a simpler way that's not AI-based? And how I've tried to solve this is actually being transparent about the cost associated with it, how long it would take, and as you can see, this trace and cost analysis is super useful because it says cost per operation, it gives a duration against it. actually breaks down by operation what was the token cost and the total cost. So extremely useful, you can continue to track and monitor this in production as well. Yeah.

So then you can say, for example, you know, PHA redaction per patient costs 15 cents. And you can decide as an organization is that more or less expensive than alternative models. That's, I mean, I actually haven't heard anybody give this, this suggestion. So I want everybody to hear it, which is if you're. getting internal resistance to cost. In fact, I feel this even as a, even as a solo founder who's vibe coding things and I see Claude code give me like, this cost you $5.13. And I send that to people, they're like, that's so much money. I'm like, it's $5. And so I love this idea bringing true transparency. And I'm sure these numbers are much smaller than your production

numbers on your, you know, million person studies. But it does bring a true sense of ROI and investment here that I think can get you through internal hurdles. Yeah, it really has helped. And I really love streamnote for this because it's tracking everything by per operation. So we can even plug and play very easily. This is awesome. And so you built this and are getting it in the hands of folks that need to submit this. And what do you think the actual, you know, you were telling me millions of dollars, months and months and months, tons of people. Were you really able to shift the cost there? Did you see an impact here?

We did. We did. And we actually applied some of this thinking to our cancer vaccine, the I&T vaccine that's coming out. And our RSV, COVID-R-SV combined vaccine as well. Both are past the clinical trial. And I think we're looking at commercialization there. So we found that cost savings was definitely one thing. but the stakeholders themselves were super engaged in the process, and they built a system that scales pretty well as the company scales as well.

So that was the most useful thing is we're saving on repeated work, where if one thinks about, I'll just give some numbers out there. So billions of dollars in costs per trial, three clinical trials per vaccine, want to bring 15 vaccines to market, that's probably in the order of, of a couple of 100 billion. I mean, that's a pretty significant number. And usually most companies don't have that sort of cash. And any kind of cost savings you can generate have a direct impact to the bottom line.

And in addition, if you're saving on time, you're bringing life-saving vaccines in the hands of people who actually need it. So I was really motivated to kind of help here. Well, this is such a great use case. I love that you showed going from quad to streamlit app. and then from Streenlet app to real impact on something that I think, you know, as a mom, RSV and COVID vaccines, I thank you. So the sooner you can bring those to market, the happier I'm going to be. So this is amazing.

I know you have one more workflow for us on the product management side, and you tease this like PRD to Jira thing. So I know you're thinking a lot about just product management generally and AI. So you want to show us to your next workflow? Absolutely. So the problem statement here. is that with product managers, we in our day-to-day rules, end up working with a lot of different stakeholders

who have various priorities when it comes to the product. And I personally, I really want to make sure my stakeholders have heard that we're working very, very collaboratively. And while we do that, at the same time, I want to make sure we're making the right decisions and also kind of bringing everyone along

and moving things forward. And with influence and communication, which is a super important skill as a PM, it's sort of custom where it's hard to say which communication framework to apply when and what type of communication would land best with a certain stakeholder because it's so personal and you really have to empathize and understand folks. So a very important skill to work on. I've been starting to lean in on Claude a little. bit and training it to kind of give me ideas and use it as sort of a brainstorming buddy in the

process. And this has actually really helped me save time in meetings, ahead of meetings in preparation. So I'm going to go through a prompt that I created. I want to show you this really, really cool thing that Claude developed. It's called a prom generator. So if you go to the console, console. dot Anthropic.com, you'd be able to create a net new prompts or generate prompts as well. And I created one for influence and communication, where essentially I didn't create this, by the way, generated it for me. I gave it some instructions. And then it gave me very, very precise information on what the instructions are. And then it gave me structured values that I can include as part of the prompt and how it would break down the problem, how it would. How

it would think and reason through it and then what outputs it would give me in return as well. So a couple of things I see here is one, you use this tool that Claude has called prompt it generated the prompt. For those that aren't looking at the screen, the other thing I want, it's XML is the name, name of the game today. So it has these kind of like code block sections for the situation, your knowledge base, how you do strategic thinking. Like all good prompts, it starts with the URA blank and gives it,

a role. And then if we scroll down a little bit, you can see that it also lets you put in different knowledge. And then it tells you to provide your response in a very specific format that has a combination of analysis, your communication strategy, play by play, and then like an anticipated question. So this is a very structured prompt. Have you started to reuse this sort of like kind of XML style prompt or, you know, have you played with different things? what's your approach? Do you just use what Cloud gives you? Yeah, so I usually start with generating a prompt,

and then you see this button here, use Cloud to optimize your prompt. This is extremely useful because what it does, you can give it feedback and in real time it will improve your prompt and then add it back in here. So now let's put this prompt in the actual Cloud project. So here we go into projects. and this is our influence and communication coach.

I'm going to update this. You can see here that you can set project instructions. So I'm going to update this prompt here into Claude and save these instructions. And I've also given it a project knowledge. So I've actually trained it on free books from Project Gutenberg that consists of all sorts of things from literature like Dale Carnegie and tactics on persuasion. and how to win friends and influence people with all of these cool books that are all free and in the public domain.

So I was not expecting Jane Austen to be in your product manager strategic communication, a quad-anthropic project, but I am excited to see it. So you not only gave its prompt, but you kind of loaded it up with classic literature, classic writing on influence, persuasion, communication. Very, very interesting. Yeah. Jane Auster is a favorite of mine. I'm not sure. Omnip folks would use that for persuasion, but I found it very enlightening. So now I'm going to walk you through a prompt that I have been working through, and this is just a test case, but we can try something live.

How about we try that, Claire? Let's do that. Amazing. So do you have a situation in mind, or I can ask it to generate a couple of situations? Oh, let's ask it to generate a couple of situations. let's ask it to generate a couple situations. I'm curious what it will come up with. So I'm using Claude Sonnet 4, which is one of the reasoning models. And as a reminder, the goal of this particular project is to essentially take a curveball problem at work and then

come up with sort of a structured strategy. So here when it's done, it's made the assumption that I'm Alex Kim. I'm an AI product manager working at a healthcare AI startup. And these are all of my stakeholders, very, very, you know, sort of experts in the domain and have very strong opinions about various things. And it has, it's broken down a problem statement that I'm trying to solve where two weeks before a major presentation, these are all of the concerns that my stakeholders mentioned. So it's actually created that for me. And it has also described of the core challenge here, navigating ethics, compliance, competitive pressure, stakeholder expectations. And it's also kind of personalize it, which is cool.

So for people that aren't seeing what it generated, which is a very realistic, unfortunately, a challenge, which is it generated this scenario, which we're just going to pretend as a scenario we're facing at work, which is I'm a PM and my head of clinical data just identified a bunch of issues, both privacy and accuracy data two weeks before we have a major presentation with a huge, huge prospect. And then it lists out everyone's different opinions. And I think PMs can really empathize with maybe not this particular situation, but a situation like this. So you've created an AI project that says, give me a sticky situation. And I will help you navigate, how to navigate the situation given all this all the stakeholders and so it's listed to the stakeholders it's

given you the core challenge so what do we do what do we get next right so I'm going to ask it to use this and proceed next steps and then this is what it's going to use your strategy framework that you came up with in the prompt to sort of break down the problem and give you a approach you might be use. Exactly. So it's now sort of analyzing based on the project information that I had shared with it. And as we can see, it's picked out the most relevant sections from the knowledge as well. So I had given it around 10, 15 different books and documents, but it found the top two and then picked out the right sections from those. So all of this, it was able to search that information and point to that and analyze the whole workflow. So it gave me sort of a situation analysis, which I found particularly helpful because it's sort of validating that it understands the core challenge and the organizational context and some of the cultural factors as well.

And here's my favorite part of the prompt where it actually goes in and it looks at some of the influential tech leaders that we have all heard and know of. And it sort of inspires the user to say, okay, this is how Satyr Nadella or Andy Jassy or Tim Cook or Jensen-Mong would actually approach the problem. This is amazing. And so you're giving really real life examples or quotes that someone can say, okay, if I were to approach it like this other person, which I see, I hear as a very common AI prompting technique, act like this person or act like you work at this place,

can really bring a set of unstated principles to a problem. that I think is really effective. Okay. And so it's come up with document examples, tech leader examples, communication principles. We all love how to make friends and influence people. So some Dale Carnegie principles. And then it's giving you some strategies you might be able to take here. Yeah.

So out of all of that vast array of information that I gave it, it picked out the three strategies that would apply. So it talks about the ethics of it, which I personally, agree with quite a bit. So it's talked about positioning as privacy first as a competitive advantage. And then it's talked about compliance and then now it's going to give me sort of a collaboration strategy as well. So after analyzing the situation, it kind of summarized for me here are the three things that you should do in terms of strategically approaching the situation. First, make sure you're acknowledging and validating each stakeholders' expertise. Second,

collaborate to discover an approach that demonstrates responsible AI leadership and then talk about the ethics aspect of it so it's really kind of honed in there so it's very helpful as of course like a good PM product leader in the space would have this info but it's it's helpful to see that as a sort of a thought partner as you're brainstorming over this as well and then it's actually broken down my pre-work for me on a day-to-day basis so what are the individual conversations I can have with each of my stakeholders and what question to focus on and ask in that meeting. And then it's given me a minute-by-minute structure for the final leadership meeting as

well. And here is my favorite part. Not only the play-by-play in the meeting, it has given me back pocket questions that could be thrown as curveballs that I should probably have an answer prepared for. My mind is blown right now because I can just think of so many PMs that stress out so much when they're asked to take on a big problem that has executive kind of like leadership eyes on it. And so many just don't know how to approach those meetings. Like, who do I talk to? Who do I not need to talk to? What do I show up in this meeting and do?

And how do I show up to this meeting really, really well prepared, even when it's a challenging problem and you've created this kind of like project that just says here's my problem here are all the people and it's given you a schedule and an agenda to follow in these conversations which is super helpful and would help pms not only save a lot of time and prep for these meetings but actually show up just more prepared and more confident i agree completely it has been extremely helpful and yeah i i'm actually also even happy to share this publicly but it's helpful to folks people are definitely going to want it. So post in the comments if you would like this publicly shared so you too can prepare for your meeting where all your stakeholders disagree. Well, this has been

so amazing. I love that you both showed us building something very technical that solved a really complex and expensive problem, but also something really practical that brings a lot of like heart and experience and strategy to a human problem that feels like it doesn't have a structured solution to it, but in a lot of times it does have a structured solution to it. So we're going to hit you up with some lightning round questions and then I will get you back to whatever you are building next or your really tough meetings that you're very prepared for. So I think the first thing I would say is so many of our listeners are in software and technology, but what you're showing is that the life sciences, biotech, hard sciences have so much to benefit from embracing AI. So

So what would you say to your peers in life sciences who have maybe been doing something a very specific way for a long time about how they can adopt AI and how it can transform the impact that can have on the world? The thing that I have been doing quite a bit with my stakeholders and I have been encouraging folks as well, my peers to work on is figuring out where they spend majority of their time and which things, which tasks do they actually want to focus on and which tasks do they actually want to dedicate? So what are energy sort of maximizing tasks and what are actually a drain on their energy and time? And now you have access to AI and you can take a plot project and

create one for any of the use cases and tasks that you don't really want to spend time on. So I think the best thing about this is it's not just for individuals. But you can make GPDs and projects available to all of your stakeholders, and all of them can benefit from it and really enjoy the perks of your great thinking. So I would encourage people in life sciences as well as any folks who are interested in to play around with these tools and not be afraid of actually trying to optimize their own work because they might find it's a common thing for all of their peers in the process. So speaking of being a little bit afraid, I mean, you work in a highly regulated industry with, you know, patient data. You have to have the highest ethics. You really, the burden of accuracy is so important. The burden of privacy is so important. How do you think about safety, privacy, alignment? It's not a problem that I think a lot of the kind of like vibe coders or PMs in software companies really have to grapple with in their day to day, but you do. And we just curious. I'm curious how you approach it, what's your framework, where does it stack rank in your priorities? Absolutely. I think with most regulated industries I've worked in finance before and in healthcare, I also currently work with family wellness where there are teenagers and kids involved.

Compliance and privacy comes first, but it's a bit more than privacy and compliance. It's the ethics overall that one must prioritize. prioritize. And as it relates to LLMs, I think looking at the benchmark data of which LLMs are performing higher on ethics and what edge cases have been observed in terms of safety is sort of the priority zero of a product leader in the process. Because of course, technical leaders, engineers can make decisions on which models to use when. But I think the product leaders can provide that lens of, hey, this model has shown for these kind of populations, has shown not so promising results in terms of ethics and alignment with human values.

How might we shift the process and how might we avoid using some solutions that might not benefit the end consumer? The other aspect of this is also, in particular with safety and alignment, it's important to identify how you would evaluate your solutions both offline as you're developing it but also in production. So the online performance monitoring of it. And that's also an area where product leaders are particularly helpful because they can provide both and share like here. Here are my like hundred things I want to know that the model does well on when it comes to safety. If it's not doing well, let's do a rollback. Let's find, you know, quick solution. And oftentimes as, as you know, in tech,

things move extremely fast. We're also rolling very quickly and we're operating one-click faster than usually we do. So I think just like taking that strategic view is extremely important. And those are some practical ways that I would suggest perhaps folks are using and limbs could apply. Well, you're setting the bar and I'm really glad that you are one of the product leaders out there that are building these kinds of products because it's clear you take it seriously. Okay, last question. I have to ask everybody, when AI is not listening, when it gives you terrible strategic communication advice. What's your tactic? How do you get the LLM to listen? Many emojis. It does really well with emojis. And it also seems to adjust when you put in emotion. I literally tell Claude, I'm sad. I have a bug. fixes everything.

I do love the reasoning models that expose their reasoning because I just love reading, like, in the sidebar of cursor, like, user is extremely frustrated and disappointed in my response. I understand why they're so sad. So, okay, emojis, emotions, and XML. That's going to be the tagline of this episode. Prerna, it was so great to talk to you. Where can we find you? How can we be helpful?

You can find me on LinkedIn, and I'm happy to connect with. folks. I'm also happy to share some of the tools I've developed and how you can help me just stay in touch and continue to engage and talk about all the cool things you're building. Amazing. Well, thank you so much. Thank you. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at how IAIIPod.com. See you next time.