ACM ByteCast - Mor Peleg - Episode 41

Episode Date: July 26, 2023

In this episode, part of a special collaboration between ACM ByteCast and the American Medical Informatics Association (AMIA)’s For Your Informatics podcast, hosts Sabrina Hsueh and Adela Grando welcome Mor Peleg, Professor of Information Systems at the University of Haifa and Founding Director and Head of its Data Science Research Center. She is Editor in Chief of the Journal of Biomedical Informatics and an international fellow of the American College of Medical Informatics (ACMI). She received AMIA's New Investigator Award for work on the GLIF3 guideline modeling language. Mor is a renowned researcher in clinical guideline-based decision support. Initially fascinated by biomedical engineering, Mor shares how she arrived at the intersection of information systems and medicine, after working in IT and completing her postdoctoral research at Stanford. She mentions her recent project, MobiGuide, which aims to narrow the gap between clinical guidance and patient needs by providing 24/7 decision support to patients and providers. Its current focus is on improving the mental wellbeing of cancer patients through evidence-based practices such as exercise, yoga, and positive psychology. Mor also shares advice for people (especially women) looking to work in interdisciplinary fields. She emphasizes the importance of health equity and how AI can be employed in the service of detecting unfairness.

This episode is part of a special collaboration between ACM ByteCast and AMIA For Your Informatics Podcast, a joint podcast series for the Association of Computing Machinery, the world's largest educational and scientific computing society, and the American Medical Informatics Association, the world's largest medical informatics community. In this new series, we talk to women leaders, researchers, practitioners, and innovators who are at the intersection of computing research and practice to apply AI to healthcare and life science. They share their experiences in their interdisciplinary career paths, the lessons learned for health equity, and their own visions for the future

of computing. Hello and welcome to the ACN-AMIA joint podcast series. This joint podcast series aims to explore the interdisciplinary field of medical informatics, where both the practitioners of AI ML solution builders and the stakeholders in the healthcare ecosystem take interest. I am the co-host here, Dr. Sabrina Xie, with the Association of Computing Machinery, by CASS series. With me today is Dr. Adela Grano

from IGMEA for Your Informatics Podcast. We have the pleasure of speaking with our new series guest today, Dr. Mor Pelek. Thank you so much for joining this podcast. And today we have a very special guest, Dr. Mor Pelek. Thank you so much, Mor, for joining us today. Thank you, Adela. Thank you, Sabrina. It's wonderful to have you here. And Maureen is a full professor of information systems, and she's also the head of the University of Haifa's Data Science Research Center. She's the editor-in-chief of the Journal of Biomedical Informatics. She's also an international fellow of ACMI, the American College of Medical Informatics. She's also a fellow of the International

Academy of Health Sciences Informatics. Moore's research focused on clinical guidelines-based decision support systems for both patients and physicians. She was the lead developer of the computer-interpretable guideline language GLIF3, and she was the recipient of the Computer Interpretable Guideline Language, GLIF3, and she was the recipient of the 2005 AMIA New Investigator Award. Her current projects focus on supporting disease management for patients with multimorbidity and on applying psychobehavioral theories to engage patients in their health. So, very impressive, Sylvie Moore. And I was just trying to recall when we met, and that was in 2008. You were at Stanford. That was quite a while ago.

And you were a visiting scholar. And I was doing my postdoc. and I was just starting to learn about computer interpretable guidelines. And you were already a well-renowned expert on that. I have just finished my PhD in computer science. I honestly knew very, very little about clinical informatics. And you were just such a wonderful mentor, so patient with me, so generous with your knowledge and expertise. And you truly play a very key role in helping me to apply computer science to medicine. I will always be grateful to you for

that. Thank you. And I do believe that you have a very similar professional journey. So you train in the field of information systems, and then you were very successful merging it with medicine. So we wanted to do this podcast so you could share with our audience how was that journey, and why did you decide to work in the intersection of information systems and medicine? Yeah, so thank you so much, Adela, for your very kind introduction. I always enjoy working with you. So actually, I think my journey even started before that time, you know, a very long time ago. Even when I was in high school, I had to choose what would be my focus in high school. And I was always in between math and biology, and I couldn't really decide.

I just went my best friend. She really liked biology, so I chose biology. But I did a lot of extracurriculums that had to do with programming, PL1 and Pascal and, you know, different languages when I was 12, 13 years old and throughout high school. It's always, I liked both things. And then I actually started learning first electrical engineering in the hope that I would one day go into graduate school to study biomedical engineering. It's something, a concept I knew about since I was 17 years old and really fascinated me. And I knew the road there was very long and you have to start with electrical engineering. Only I didn't like it. And the studies were difficult. I was the only woman in class with hundreds of guys.

And it was really hard for me on my own. And I decided to move to biology. So I don't know if you knew that, but I have two, my undergraduate and my master's are in biology. I didn't know that. Yes. And so, you know, after like a couple of semesters, I moved to biology. I was really, really good at that. I even got a national award from the president of Israel at that time.

Wow, that's impressive. Yeah, but I started my master's and after two months, I knew that lab life is not for me. I was working with radioactive materials. I started to get phobias and I survived the master's, but I knew that I had to do, you know, another transition. So I thought, what should I study? Computer science. And it was back then in 1994. And bioinformatics was not really an established field at that time. The human genome project was ongoing. It really fascinated me. And I wanted to find a way to go into that field. And I started to go and interview in the computer science and in information systems department.

And I thought that because I'm really good at organizing things, I would be better in information systems and going into databases rather than the algorithm side. So I started the PhD there. And when I was about to graduate, I actually heard a talk by Yuval Shachal, who is another very important Israeli medical informatics scientist, and he was at Stanford. And that's why I came to know that there is a program for medical informatics at Stanford. And I applied to Mark Mewson for a postdoc, and I ended up working with Ted Shortliffe and Samson Tu on the Glyph project that you know. So it was kind of a long and winding road of things that interested me plus chance.

That's a great story. Thank you for sharing. Sure. Yeah, it's highly impressive to hear the story always for our audience here in the intersection of AMIA and ACM, right? And to benefit them a little bit more, let's also understand a bit more what leads you to your current roles at University of Haifa and also as the chief editor of the JBI. Are there any specific pressing issues you are facing now, you feel, our audience? Yeah, so I was born in Israel and in the city called Haifa, and I always wanted to go back.

My postdoc was for four years, and my family was my immediate family. My husband and my kids were with me when I did the postdoc, but I always wanted to go back. It were my parents, my cousins, my uncles. So I went back to Israel. And I got a position at the Department of Information Systems, which was very new. I was, I think, the second person who joined the faculty. So that was a good opportunity for me to be in a leadership position right away at my department, which was also something that I fancied. And the postdoc with Ted Shortliffe, he was one of the founders of the medical informatics field. It made a very big impact

at my career. Ted was the editor-in-chief for the Journal of Biomedical Influenics for 20 years. And he invited me first to join the editorial board and later become an associate editor, deputy editor. And finally, I became the editor-in-chief that replaced Ted. And I'm so happy to follow his leadership. You cannot really replace Ted because he has done a lot for our field, but you can try to follow his footsteps. And I'm very, very grateful to him, to many other people like Samson Tu and Mark Newsom, Vimla Patel, Bob Greenis, that really helped me shape my career in a way that it can be very fulfilling and have an impact on other people's lives as well. So what is the pressing issues that I face now with JBI?

Some of it is really to educate the researchers about what we're seeking in our journal. So the Journal of Biomedical Informatics has a big emphasis on research methods, the novel methodologies that are specific to biomedical informatics and are also translational to medicine. So it's not just applying existing machine learning algorithms to cool data sets. It's starting with a question that clinicians face or biologists face and trying to see what methods are already there and why they're not working. And can we come up with a new method that is generalizable and could be applied in more than one case and then explain it very well and very clearly to the readers when you write a paper and also suggest how it will fit into the clinical workflow and

have a clinical collaborator that would second this opinion that you have. So you have to validate it in some way. So it's hard to reach the audience. I actually have to desk reject many papers because they don't follow this lead. And I try to explain it in every time I make a decision. And also with other colleagues, we co-authored in a tutorial about what we're seeking in machine learning based methods paper for JBI. And just to explain what is there that we were seeking and what doesn't make it for, for our journal.

So I'm, I'm inviting all of you to, to look at, look it up. Another thing that I, I focus on in, in JBI is I try to think what are the, the new topics that our community should be targeting.

And I try to invite people to write methodological reviews, to write new research papers, and also to have new special issue call for papers. And I will tell you more about it later on in the interview. I will tell you about one of the upcoming special issues that I would really love to get submissions for. Wonderful. Thank you so much, Maura. And it brings me memories. I remember our first paper together was submitted to JBI

and I remember you just explained to me exactly what you explained to the audience now. So it brings me memories. So let's continue a little more talking about your journey and how you combine information systems and medicine. So what would you recommend to those who want to start to work in an interdisciplinary field? Were there some early career moves that you found helpful and you would like to share? And especially if you have any advice for women. Yes, I think that's a great question. I think there are many, many routes to achieve what you want to do, and you don't know right away what you want to do. So you need to be

very open-minded and very trustful of your own abilities to say to yourself, I can do it. And there is some pathway I will find to do something that will be rewarding and fulfilling. For me, it was different degrees. So, you know, I told you my journey. I started with electrical engineering. I went to biology. Then I went to information systems. Then I heard that there was something called medical informatics. And that was like, really the holy grail for me, but I didn't know about

it right away. I was stuck with this vision of biomedical engineering, which I never went in that way. So, you know, don't be afraid is one advice and, you know, work on something that you really, really find engaging and important, and where you have a chance to be really good at. And you don't know until you try. And I think everything you do in your pathway makes you a unique individual with a unique experience that you can leverage for making interesting and novel research. And when you step into a multidisciplinary field, you can do it at different stages of your career.

So if you do it very early, you can have another degree. But later on, you don't really go back after you're a professor and you say, okay, now I'm going to do another undergraduate degree. Not many people actually don't know anyone who did that. So what you can do is work with collaborators who know the field, the other field that you don't know, and they are experts in the other field. And then you start to do multidisciplinary research with those people. You educate yourself at the very basics, you know, by reading, by hearing talks, and you engage with these other professionals and ask a lot of questions and learn to speak their language and come up with very interesting questions that are very important to answer. And you work together and you have to

spend a lot of time to reach a good communication. So the thing for me was to find people that I like to work with. So it's first of all people, it's not researchers. It's people that you somehow find a good connection with, a good fit. You like them as persons. And then you explore together and it takes a lot of time. You cannot just say, I will work one month and it will be over. It's something that you will probably do for a long period of time. And I found that I really, really like to collaborate. I collaborated with you, Adela, on several papers. And I do,

you know, in other ways too, you know, conferences, and it doesn't have to be just research. It can also be management, you know, to co-organize an event or to work together on an editorial board. And all of these things make research and life very interesting. You asked me about women. Do I think it's different? I think it is because I think that one of the things that are necessary, although, you know, we have Zoom and we have teleconferencing, it's really, really important to meet people in person in physical space. And for that, we have conferences or we have visits like when you came to Stanford for a visit. I think that can make a connection last for a very long time. And I think

women, sometimes it's difficult for them to travel. They sometimes have children and they have to have a very supportive family to supportive husbands, supportive parents. That's how it's been for me. So, you know, I always make a list of what everyone has to do to replace me and they cannot do without the list, but I find ways and I am replaceable. So I think, you know, women would benefit from going abroad and I encourage them to do so. Yeah, we should definitely have another workshop on this kind of 50-50 responsibility sharing for women. Yeah. And some of our audience here are already in their mid-career, especially in medical informatics field. A lot of us are pursuing this interdisciplinary career in the middle of medicine and information systems. So did you have more advice to share for those mid-career? Yeah, so I think mid-career is often challenging because sometimes you kind of

think that what you did for the past 20 years is no longer in the very focus. You started when it was very hype and now it's not hype anymore, and there's a different hype. Do you really want to join the celebration? And sometimes you're not able to do it by your own. I mean, you get some exposure, but you know you will not be a leader in a new field very easily. So some of the things, these management opportunities are really great. You can get

involved in committees. They can be university-wide committees or national committees or even international committees. AMIA has a lot of these committees and ACMI does too. And you get a lot of exposure to new topics, to new people, and you hear a lot of points of view. And then you can go and implement some of these ideas in your local environment. I had the opportunity, I think it's a good story, I will tell it. So I was on sabbatical, I think six years ago, again at Stanford, and the dean asked me, he called me on WhatsApp and he said, can I talk to you? I know you're on sabbatical, but we want to set up a new data science undergraduate program. And it will be multidisciplinary between the three departments of computer science, information systems and statistics.

And no one can get the departments to talk to each other and to agree. Can you do it? And I was up for the challenge. And I started really researching, you know, what are other programs that are in different places in the world? What do they teach? What are the essentials of data science? And what is available in the different departments? And I worked on that. And then I was known in the university to be an expert, although my training is not really in machine learning. I took a course on neural networks when I was a PhD student,

but that's about it. And I collaborated with experts, but I'm not an expert myself. I am part of, let's say, the life cycle of data science. I'm certainly part of, and I'm expert on some of those parts, but not in the middle part that's really the big hype. But having had the reputation that at the university that I set up this program, then I started to get opportunities also on national level. I was invited to different committees and that's how I ended up founding the Center for Research on Data Science for the entire university. So this is all the departments in the university and all the faculties. And it's a center with 55 faculty

members. And it's about research, but it's not my research. It's the research of these people. And my challenges there is how to help them form collaborations and how to make them capable of forming the collaborations and how to motivate them to work on multidisciplinary projects. And I think it's fascinating because, you know, you have to invent methods of how to get people to work together. So it's not exactly informatics, but it's still very interesting

and novel, at least to me, because it's not something I did before. Well, thank you so much for sharing that story, Maureen. It was really interesting. And talking about expertise, you are a world renowned expert in knowledge-based clinical decision support system, which is an area of AI applied to medicine. And, you know, unfortunately, recently there has been a lot of discussion about bias in AI. So health equity, it's a very relevant topic now. So we wanted to know from you, what is health equity to you? And what is the most pressing issue?

Yes, that's a great question. And remember that I told you that I'm going to tell you about a new call for papers, a new special issue that we just opened the call for JBI. And the deadline is in October 2023. So health equity is very important. And at first, I thought it only had technical issues because, you know, I'm a technical person, I'm in medical informatics, in the methods. And I thought there is the fair data sharing. So, you know, how to have the, first of all,

the data being representative of all populations. And then when you get the data set, how can you share it with other people in the way that you actually explain the semantics of what is in the data set, how it was collected. And so it has to be accessible and findable and reusable and interoperable. So this is the acronym for FAIR. All of these things are very technical. Then there is another side about you want the models to be explainable. So when the public or a decision maker wants to knowledge that the model knows and how can we

say it and articulate it in a way that people can understand it and can really capture the main attributes of the logic of the model. But that's another technical aspect. So quite recently, and that was the motivation for the special issue, I got an email from a reader. And the reader of one of the papers that just came out, it was a paper about natural language processing and a technical paper as well. And this person, his name is Professor Jonathan Herring, and he's from the UK. And he's actually a professor of law.

But he deals with equity and fairness in the medical field. And he was upset with the paper. He wrote to me and he said, the paper uses offensive language. It talks about mental retardation. And this is not the term that we should be using. And then I thought, well, the medical records have this. I looked it up immediately in SNOMED CT, and I saw that this is the official term. So this is the term that will be found in the medical databases in the electronic health records. So you cannot get away from using the term. But then I thought, you know, he pointed me to one of his papers about people giving names to genes. And in the genes, there will also be the name of the disease and how this can be discriminating for a person who wants to get insurance.

And I started thinking, well, he has a point. And I looked at the paper again and I looked at some of the examples that were there. And I also consulted with the authors and also with reviewers of the paper. And no one did anything that was offensive. That was a given. I know all of these people are very good people. But I looked again and I saw that their example was about a young woman with mental retardation. I thought, as a woman, I was a little bit upset.

Why did the example have to be a woman? And maybe this is, in fact, a sentence that appeared in one of the clinical notes, but it kind of still upset me. And I want to have a special issue in which we address not just the technical issues, but these psychological, sociological, soft issues that maybe we can educate our readers and our scientists who write papers on how also to use examples in writing. You know, what are the case studies that we present as an example? And what language do we use? And that's a part of the equity and part of thinking about ourselves as people, as individuals, as being fair and as being equal and not discriminate.

Even if you don't intend to discriminate and for sure no one intended to do no harm there, you have to be more aware and more sensitive. So this is the part that really engaged me and together with Chiam Weissman and with Yuan Liu, I asked him to join me and to help me in co-organize this special issue for JBI. And I really hope that the audience and other people as well will submit papers to that special issue. As someone who has been thinking about AI essays every day now, I can vouch for the timeliness and importance of this issue. This is really, AI is the new kid on the block, right? But no one knows how to trust it until you know better about all these guardrails that should be put around it. And how should we increase the health equity with all these issues considered?

ACM Bycast and AMIA FYI Podcast are available on Apple Podcasts, Google Podcasts, Spotify, Stitcher, and other services. If you're enjoying this episode, please subscribe and leave us a review on your favorite platform. That leads us to the discussion about also in your current role as the chief editor of JBI, you have done a lot in promoting equity through increasing the diversity in the representation of your board, right? Can you talk a little bit about that? And also more in general, in this field, between medicine and computer science information systems, what did you see as essential for us to do together? Yes, thank you for the question.

So I really wanted to increase equity in many respects. I think many times people think about men and women, and I do too. And this was one of the parameters, but it was only one of the parameters that I wanted to increase. And right now, actually, we're 50-50, 50% women, 50% men. If you look at the editorial board plus associate editors, I'm really happy about that. I could do it fairly quickly within two years by trying to recruit each time more women than men and also

increasing the size of the editorial board. So that allowed me to recruit more women in total. But it's not only just diversity in women, it's also diversity, for example, in the areas of expertise, which is very natural. So you have to have on the editorial board people who are experts in all the spectrum of topics that you want your journal to cover. Expertise from different countries. So, you know, different countries. I don't want everyone there to be just from one country or even just from one or two continents. I want to have diversity of countries. I also thought that if we have so many papers of machine learnings that are coming from China and India, we should have people on our editorial board and also associate editors that are from these countries because they are also sensitive to some of the

things that are written there and they can give better advice to the writers even when they reject or when they follow a paper through several rounds of revision of improving that paper. Sometimes they need to explain things that are culturally sensitive. So I want to have diversity also in the countries. Another aspect is industry versus universities versus hospitals, for example. I want to have people from all of these sectors. And also seniority, so not just full professors, but also younger scientists that are very good. All of the people on our editorial board have published in the journal several papers and have acted as reviewers for several papers. So they're knowledgeable and they understand the policies and the aims and the scope.

But we don't just take very seniors. So we have equity and diversity in many, many different respects. Your second question about how can information systems or computer science work together with health and also increase the fairness? I think that, you know, AI is a great tool to detect areas when there is no fairness. So when there is bias, bias in data sets, bias in the language in which we explain the results. So we need the results to be explainable not only to professionals, but to the public, to people with different backgrounds and different health literacy. Thank you so much, Mor, for the work that you are doing for JBI.

Amazing work. And one of the things I really admire about you is how versatile you are. You're able to collaborate with people from US, but also in Europe. You're equally active in both spaces, which is very rare. I wanted to talk to you about your latest work funded by the European Union, and I'm referring to your past project, MoviGuy, but the recent one, Capable. And both projects have a very strong focus on narrowing the gaps between clinical guidelines and patient needs. So are there any improvements in personalizing clinical guidelines to patient needs that

you have facilitated making or new changes that you would like to happen? Yeah, thank you for this question. So Mobiguide, I was the coordinator of this large collaboration. It was like 13 different organizations and maybe 60 people worked together on this project. It went for four years and the idea was that we wanted to provide decision support anytime, everywhere, to patients and to their care providers. So before Mobiguide, to me, the focus was always on the caregivers, the care providers, sorry. And now the focus started to become since 2011 for me on patients and the people that need to help the patients. So clearly the healthcare professionals,

also caregivers. Of course, it's not one size fits all. I wanted to personalize the decision support. So in MobiGuide, the personalization was related to the patient's context. I had a student, Adi Fuchs, and he was working with me and Pnina Sofer, one of the professors in our department. And he said the following thing, that patients, they don't want to really think of themselves as patients. They want to lead their normal daily lives. And sometimes all these medications and all these tests, they're really hurting their ability to lead normal lives. So maybe they can negotiate with a doctor, and maybe they can be more lenient with what they're asking them. For example, if you go on

vacation, maybe you don't need to monitor your blood glucose every day, four times a day, or your blood pressure every day. Maybe you can do it once every two days. So with that, he went to interview patients and doctors, and he read the literature, and we looked at all these different aspects that define your personal context and which could influence decision support. The patients thought like Adi, like the student, but the doctors actually, when they said, oh, a person on vacation, we have to be extra careful because he actually can be compromising his health. So instead of having him, the patient, measure blood pressure every day, he needs to measure twice a day. So it didn't work exactly in the best way we hoped. But there were cases in which they said,

yes, we can be more lenient if we see that the patient is monitoring themselves very well. And also, they are within the bounds. So the different results of the monitoring show that they're in good health. Then we can be lenient. And then we can say, okay, it doesn't have to be every day. It can be maybe three times a week. So that was in MobiGuide. We also personalized there the timing of the reminders and little things like that. Then my collaborator, Silvana Quaglini from University of Pavia, she was one of the key partners, as was Yuval Shacha in Mobigad. So Silvana proposed the CAPABLE project, and she's the coordinator, and I'm one of the PIs in that project. And we're now starting the fourth and final year already with patients. There,

the personalization is different. We are focusing on cancer patients and we want to improve their well-being. So it's not only pharmaceutical drugs that we're giving to patients. It's also their mental well-being. And to influence their mental well-being, like anxiety, depression, stress, many effective evidence-based therapies are non-pharmacological. They involve things like mindfulness, exercise-based yoga, tai chi, positive psychology. They're still evidence-based, but they're not drugs. And the difficult part about it is how can you form these exercises into a habit, something you never did like deep breathing or Tai Chi, and you have to now do it every day.

So there are behavioral theories, and we're using behavioral theory of B.J. Fogg. He says that in order to develop habits, three things need to co-occur. One is that you need to be motivated enough to do the new habit. And the second thing is you have to have the capability. So if we want you to do Tai Chi for 40 minutes, maybe you cannot do it, but maybe five minutes you can do. And the third thing is a trigger that will remind you to do the habit over and over, over again. And, you know, maybe at a specific time window that co-occurs, let's say 7 p.m. every day, or right after you wake up, or right before you

go to bed, or after you brush your teeth, or when you drink your first coffee. And what we're trying to do is use machine learning to try to personalize the trigger to the individual preferences to patients. Sometimes we can ask the patients about preferences, and sometimes we can learn them from data that is generated by their wearables, like the smartwatch or the smartphone. So I've been working with Shimon Wilk and Aneta Lisovska from University of Poznan, and we published a lot in that area in the past two years. The other part of your question is also about challenges. So, what are the new challenges that I think I would like to address or like other people to address? So, I think forming these good health habits by people like myself, it's not that difficult.

And people like myself, they have health literacy, they have high socioeconomic sectors. But what about the people who are really suffering more? They're on less educated, low socioeconomic status. Maybe they are in a culture that doesn't understand much about what constitutes good health. And maybe it clashes with other culture elements, like you're supposed to eat a lot of cakes in parties and always to drink sweet drinks or things that are not healthy for them. What can I do to help them? And not just me. I mean, many people need to help there.

So these are big, big challenges. I really wish a lot of people would work on. Yeah, these are great examples that give us some hope that in the future, with all this learning health system in place, then this continuous feedback loop can help us to get to the next level of healthcare, which we need to move the needle from healthcare post-disease diagnosis to prevention. Yes, that's right. Yeah, but for some of our audience who might not be familiar with the idea of learning health system,

certainly this is the time when the cloud computing and EHR and other technical protocol like FHIR standards being available. So it seems that there is a growing trend here for health system to start growing their learning health system within their organization and then it's finally the time to do it or start doing this in a more mature way so so for our audience here can you introduce a little bit more about that here and are there anything that you think our audience can learn about about new application area to help to realize its potential yeah thanks sabrina i would use this opportunity to explain so if you have a process or or a system and it can be open looped, you know, not a loop, an open process that you just start with input, do the process, get an output. So suppose you have even a clinical guideline that you want

to implement because it is based on evidence-based medicine. And you say, okay, I'm going to my healthcare system. I'm going to implement this guideline by implementing some rules that will check parts of the guideline. Let's say there is, if the person when someone reaches 50, he will get a rule that fires for him and he will get a reminder in his email. You know, go do the colonoscopy. So this is kind of an open process. There's no loop. To close the loop, you need to see if it's effective. So how many people received this reminder and how

many people acted upon it? And you do a controlled, you look at this, maybe you look at people who did not get the reminder, people did look and you see, you compare and you see if it's effective or not. And when you see that something is not working, you have a chance to improve it. So you close the loop. You have a hypothesis about why is it not working. Is it because it's not a good recommendation? Is it because you didn't motivate the people well enough or you didn't motivate it according to their beliefs of what is important to them? Or maybe you demanded something that was too difficult,

and maybe you can say, okay, you're afraid of colonoscopy, but you can tell if someone has, in the excretions, if they have blood, it is also a marker that someone can have colon cancer. So you can do a different test. So you can figure out kind of what is wrong in the process by analyzing data. So if you collect data from about the input, about the output of the process, comply or 50% of the people to comply? What do you wish for? And what should this standard be? And all these different parts you can change. You can learn how to change part of your system so that it operates better. And you learn how to do it by analyzing the data that you gather about the process. And you do it for all the processes that you have in your healthcare institution. But the important thing is that you need to have the data in a standard way.

And the data needs to contain a lot of metadata, I will explain what I mean, and a lot of semantics. So suppose that you get a number and this number says glucose 100. You need to know a lot of things more than the glucose 100. To which person does it belong? Is it an average over one week? Or is it a fasting blood glucose? Or is it just, was it collected from, you know, poking the finger? Or or is it in a different way? And what exactly is this blood glucose? And in what date was it collected?

A lot of metadata that you need to store. And you need to store it in a way that when you share the data to a different organization, because you want to do statistics on a much larger population that comes from several hospitals, everyone understands what the data is all about. And for that, standards organizations like Health Level 7 developed standards of EHR data. So standards like FHIR, F-H-I-R, and there are other standards from OpenHR, which is a standard that comes from Europe, Australia. It's not an American organization.

They're different standards, but all of them share the fact that they have different classes of healthcare data, like observations about the patients, medications given to the patients, and procedures that are done to the patients. Or for each of these classes, they have the different attributes, like the time step, the vocabulary code that stands for the topic that the data item stands for, the patient who data is about, and things like that. The result, you know, what is the result and the units of measure. So, of course, we have to have standards and they really help in the learning healthcare system because you can analyze the data, not just by looking at it as numbers, but as objects that have meaning and semantics. And there are many standards that also help. There is a standard like tripod standard and

data federation and sharing standards and standards for medical images that exist. So by using them and having different devices and different software that can all share the same kind of data, they all know how to read and how to visualize the same and to process the same kind of data, we can have this learning health system be very wide so that it can be about the national gathered data or international gathered data. So in COVID, we learned to collaborate between governments and to collect and share data and insights in order to, before we had the immunization, there were other measures that we could do to help not spread the

disease and to know when you can release a little bit and when you have to tell people to stay home, tell children not to go to school. All of this is part of this learning health system and standards are an important part of having the data being meaningful to different organizations. So that's wonderful learning about health system. How about evaluation? What did you think about health AI evaluation in those systems? Are there any good examples you can share with us? Yes, I think a good example is a paper that is by Ken Kawamoto and co-authors.

He's the senior author. It was published in the Journal of Biomedical Informatics in 2022 and entitled Evaluation and Lifecycle of Information Technology Framework, so Elicit Framework, supporting the innovation life cycle from business case assessment to summative evaluation. So while it does not specifically have to do with AI, I think this is a very good paper to read. And the ideas behind there can be applied to any kind of research that have to do with information technology. So I really recommend it. I also recommend our editorial, the editorial that we wrote in JBI on machine learning. So I will provide the link. And you asked for a good example. So there's a paper by Godino, BJ Fogg, and others that was published in

Lancet Diabetes Endocrinology in 2016. And it is about Project SMART, Social and Mobile Tools for Weight Loss. I think they did a beautiful study where they implemented a mobile health system to try to help patients lose weight. And they evaluated for over a period of 18 months to see if the impact is lasting. And it's really beautifully done and beautifully written, so I highly recommend that. And maybe one last thing is that TRIPOD, it stands for Transparent Reporting of a Multivariable Prediction Model

for Individual Prognosis. So I also recommend reading about that standard. Great. Yeah, that has what we are recommending for our AI evaluation showcase for this year's AMIA as well. We will provide the link with others, the link you just mentioned. Well, we have arrived to the end of the podcast. So you have made it so entertaining more.

So thank you so much. A lot of wonderful stories that you share. And most important, you know, all the details about your journey that I think the audience will enjoy and we connect with those. So we want to thank you for spending this time with us. And we wanted to give you the opportunity if you have any final words or thoughts that you want to share with our audience. Yes. So thank you very much for allowing me to have this opportunity.

I think don't be afraid to do things that you think are important and that motivate you and make you feel needed and important and impactful. And find the way to do it through collaborations, through trying out new things and hopefully, you know, publish it in, in JBI, but also in, in anything you, any good journal that can reach the audience and, and make an influence on people in our world. Such a lovely story. Thank you. Yes, it was awesome. on people in our world. Such a lovely story. Thank you so much.

Yes, it was awesome, Mar. As always. Thank you for listening to today's episode. ACM Bycast is a production of the Association for Computing Machinery's Practitioner Board. And AMIA's For Your Informatics is a production of Women in AMIA. To learn more about ACM, visit acm.org. And to learn more about AMIA, visit amia.org. For more information about this and other episodes, please visit learning.acm.org. slash B-Y-T-E-C-A-S-T. And for AMIA's For Your Informatics podcast,

visit the news tab on AMIA.org.