ACM ByteCast - Cynthia Rudin - Episode 86

Episode Date: May 28, 2026

In this episode of ACM ByteCast, Rashmi Mohan hosts 2025 ACM Fellow Cynthia Rudin, the Gilbert, Louis, and Edward Lehrman Distinguished Professor of Computer Science, Electrical and Computer Engineering, Statistical Science, Mathematics, and Biostatistics and Bioinformatics at Duke University, where she leads the Interpretable Machine Learning Lab. Her lab, which seeks to design predictive ML models that people can understand, focuses on areas including healthcare, criminal justice, and energy reliability. Among her honors, she has received the Squirrel Award for Artificial Intelligence from the Association for the Advancement of Artificial Intelligence (AAAI), as well as the IJCAI John McCarthy Award. Rudin was recently named an ACM Fellow for contributions to and leadership in interpretable machine learning and societal applications. In the interview, Cynthia clarifies the crucial distinction between "interpretable" and “explainable" AI and makes the argument that true interpretability is foundational to trustworthy, ethical AI. She shares her extensive field experience collaborating with Con Edison engineers on power grid maintenance, neurologists on medical diagnostics, and the Cambridge Police Department on crime series detection, countering the widespread industry myth that AI performance must be sacrificed for transparency. She describes an innovative paradigm her lab developed to solve the "interaction bottleneck" between data scientists and domain experts, leveraging "Rashomon sets" to generate millions of equally accurate models simultaneously, using human-computer interaction (HCI) tools to create visual, encyclopedia-like interfaces.

This is ACM Bythcast, a podcast series from the Association for Computing Machinery, the world's largest educational and scientific computing society. We talk to researchers, practitioners, and innovators who are at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their own visions for the future of computing. I am your host, Rashmi Mohan. Artificial intelligence is rapidly becoming one of the most powerful forces shaping modern life, influencing who gets hired, who receives medical care, who gets loans, and even just how justice is delivered. But what happens when the systems making these decisions can't explain themselves?

Today's guest has spent her career asking that question and challenging one of the biggest assumptions in modern AI. that accuracy must come at the cost of transparency. Cynthia Rudin is the Gilbert Lewis and Edward Lerman Distinguished Professor of Computer Science at Duke University. She is a pioneer in interpretable machine learning and the director of the Interpretable Machine Learning Lab at Duke. Her path-making work confronts the industry's long-held belief that transparency and performance cannot coexist.

Among her many accolades, she is the recipient of the esteemed 2022 Squirrel AI Award for Artificial Intelligence for the Benefit to Humanity. Her work has won her countless awards and celebrated papers, making her a very coveted speaker and a distinguished member at most conferences and committees. It is our honor to welcome you, Cynthia, to ACM Bytecast. Thanks for inviting me. Of course, Cynthia, I'd love to hear from you a question that I'm. I ask all my guests, which is if you could please introduce yourself and talk about what you currently do as well as give us some insight into what drew you into computer science in the first place.

So my name is Cynthia Rudin. I work in interpretable machine learning. There's the introduction. Good. So I'm a professor at Duke University. And I've been working in interpretable modeling for a really long time. I started working in this area when I was working on a project with Con Edison in New York City. I was employed at Columbia.

there was a group there that wanted to completely change the way that power grids are maintained. And they wanted to do it with machine learning. And so I said, okay, I've got all these tools. I learned them in graduate school. I know how to do machine learning. I'm going to take all this data. I'm going to throw it into these machine learning models, and it's going to tell me which manholes to inspect to figure out where there are problems in the power grid.

Right. We were trying to predict fires and explosions and smoking manholes and things like that. The machine learning algorithms did not help. They were just a disaster mass. They were, you know, telling us to go look at manholes where there was like almost nothing wrong with them just because they were near other manholes that had problems. So while we were troubleshooting those, I realized that if we could actually show the machine learning models to the power engineers, they could help us troubleshoot them. And they did. They helped us troubleshoot the data and the algorithms, the models.

From there, we were able to get much more accuracy than we could ever get with the black boxes, right? The interpretability made all the difference in the world in that case. And that's kind of how I realize, like, hey, you know, there's a different path for success in machine learning than just throwing black boxes and everything because it doesn't work. That's super accurate. I mean, I think the one thing that I understand from what you said also, Cynthia, is in these domains where you're working with people who are not sort of, I mean, they're experts in their own areas, but not necessarily computer scientists.

It's so critical for you to be able to sort of explain how you're making these decisions and for them to be able to provide you input to sort of refer. find your model. Yeah, especially when you're working with really noisy data, like data that's not perfect, right? Like a car, a car is a giant black box, right? We don't understand exactly how everything works, but we don't need to because we know that there's physics behind it, right? We know that this thing, every time we press this button, it does this, right? But with data, if you're building a model from data, well, that gets really noisy because you can't trust a lot of these databases, right? They're not trustworthy. And so when you're making decisions based on them,

especially when you're involving domain experts, it's really helpful to be able to understand what's going on. What's in the data? How is it translating into a decision? That makes a lot of sense. It was a similar process that, Eddie, I know I was doing a little bit of research on your previous work. The building of series finder. I know the work around crime series detection. I was a faculty member at MIT before I was at Duke. And I was sitting in a room and there were a bunch of police officers from Cambridge Police Department that had invited themselves to meet with us. It was actually a really funny story. So they walked in. there with their uniforms and their guns and here's all the professors kind of sitting at the table

kind of like cowering and they said look we need your help with something and then one by one they talked to the professors around the table and the professors were like well I don't want to like actually like I'll supervise but I don't want to actually like collaborate and I was like I'll collaborate with you guys and I was like you know definitely the youngest one there and they actually had a really interesting problem which which was they wanted to figure out which crimes were committed by the same individuals, right? They wanted to figure out crime series. So a crime series is a set of crimes that are committed by the same individual

or a set of individuals acting in concert. They were thinking of things like house breaks in Cambridge, where you have these groups of people who go and rob houses in Cambridge, and they do it while people aren't home and they break into houses the same way. They have like a modus operandi, right? They pick a certain neighborhood and have a certain way of doing things. And if you can figure out the modus operandi, then you can figure out which set of crimes are committed by the same people. And so it turns out to be like a really interesting subspace clustering problem because the modus operandi is actually the subspace and the subspace.

And the clusters are the crime series. And so if you could figure out kind of at the same time what's the modus operandi and what's the crime series, like what's the subspace and what's the clusters, then you can help solve crime because they can't do anything about a crime series unless they know that it exists. So me and my student, Tong Wong, and this is with detectives, Rich Savary, and Dan Wagner was the one leading the whole thing. He was the detective who kind of came up with the idea. We worked on this piece of code, and we made it public, which is the series finder algorithm. And then we found out years later that the NYPD, they had been asking us questions. They had a new, like, data analytics team, and the person there was asking us questions about how you implement this thing. And we were like, okay, sure, we'll help you.

And then it turns out they had implemented it at the NYPD since 2016. And so it figures out whether a new crime is related to past crimes so they can find series much more easily. That's amazing. I mean, these are such impactful projects. Do these seek you out or do you actually go looking for them? But some of both. I mean, the crime series one just fell into, you know, fell into my lap. Like I said, I was in a room and police officers were like, can we work with you? And I was like, yes. But, you know, it doesn't always work like that.

For example, I have a long-term collaboration with some neurologists at Massachusetts General Hospital, so in particular, Brandon Westover. And that started because my student, Burke Ustyn, was looking for a collaborator. And so he was like, there's this guy at MGA. She's teaching this class. It looks like right up our alley. We want to, you know, maybe we could contact them. And so we sent them an email.

I'm still working with them, and it's over a decade later. So, yeah, sometimes it happens that way. I have this other problem that sometimes I form a relationship with someone and then I can't get out of it because they still want to keep working together. And then I'm like, I have too many collaborations. So it's really hard to end things. So I'm not really careful about what I start because then I know I can't end it. But I've been loving working with Brandon West over all these years. It's been great.

That's incredible. Thank you for sharing that. I know we dove straight into your projects, Cynthia, but I'd love to sort of go back even further. Because oftentimes, I mean, the audience that we have are folks that are. early in their career, could be considering computer science, folks that are in college. So I'm curious as to what drove you into computer science in the first place. Were you exposed to it early on? I was an applied mathematician. I wanted to do applied math modeling. I think my earliest goal was to try to run one of these giant weather models. I'm so glad I didn't do that because that's

definitely not the right thing for me. I found out about machine learning because Gary Flake, he was a really important influence on my life. He was a researcher. at NEC Labs and he handed me a book and I read the book and I was like, oh, you can predict the future with data, you know? So that was, yeah, that was kind of my first introduction to kind of statistics and computer science was this book. That's how I found out about machine learning. So I read this book and I was like, that's what I want to do. I want to predict the future with data. So yeah, it wasn't hot at the time. It was not like it is now. It's hard to believe kind of the difference between what it was then versus what it is now. Like now you go to a big conference, like a machine learning conference,

and there's 16,000 people before it was like maybe 200 people. And you kind of knew the people. So it was really different. It was like, it was just like a nothing field. I mean, nobody knew about it. My husband was in biology and everybody was doing like genetics and biology. That was the cool thing at the time. That was everybody was like, oh, this is the wave of a future. They're going to change the whole world. and now machine learning's kind of taken over as being like the really important thing like AI. So I guess it just goes to show you that you don't have to go into the thing that's the most popular. Go into the thing that you really like to do, right?

Think for yourself and don't just follow the crowd. Yeah, I mean, I've never followed the crowd, right? Like working in interpretability when everybody thought black boxes were the way of the future, that's totally different than what other people were doing. Yeah, I think you hit the nail on the head, which is really, I mean, I can tell from just the way you're describing. your work, how passionate you are about it. And to your point, I think pursuing what you're most interested in, because it's so hard to predict how these fields transform over a period of time

and what becomes relevant and critical and important. Can you describe to our audience, what exactly is an interpretable ML model and how do you contrast it to a black box model? Let me elaborate a little bit on the previous question before I do that. I don't want the young people who are listening to think that all of this was easy and that I just made up my mind that I was going to do this when I was like really young. That's not true at all. I went through many years where I was like, this field just doesn't make any sense. I don't know why I'm in this field. Like I was pretty depressed and it was pretty upsetting because I was like, these black boxes are not working for me. So it's definitely hard. It just takes a lot of work and a lot of thinking to get to

the other end of it. I just can't give up. Okay. Back to things I actually really love, which is interpretable models. An interpretable model is a model that a person can understand. And interpretable machine learning is actually a pretty big field. Like you can actually work in a lot of different areas of interpretable machine learning. The field was named at least as far back as 2001 in the Breiman II Cultures paper because he used the word interpretability in that paper. A lot of people have co-opted that word to mean explaining black boxes. That is not interpretability.

That is explainability. That is explaining black boxes. And the terminology there is really important because you do not want people to get confused between a model that's actually interpretable that can be used in high-stakes decisions and a black box, which can't, and that you're just explaining it, right? You're just poking at it to see what it might have in it.

So it's really important to keep that terminology straight, and no matter how many times I say that, the explainability people ignore me. They want to use this term that's used for important high-stakes decisions. Now, interpretable models can be a lot of different things for tabular data, so data that can fit on a spreadsheet.

I like to say that the models can be fit on an index card. You can just write them down like a little simple formula that can fit on an index card or a PowerPoint slide or maybe a piece of paper. So these can be like medical scoring systems that you might have calculated for you in a hospital. Like you give them a little bit of your medical history and they give you two points for this, three points for that, four points for this. And medical scoring systems are something I've worked extensively on. and the collaboration I was mentioning earlier with Bergustin and Brandon Westover,

and also another neurologist Aaron Strzok, led to a scoring system that's widely used in ICU's and intensive care units now for predicting seizures and critically ill patients. And the model we designed is called the two helps to be score. And it has just a few things that the neurologist look at in the patient's EEG and they calculate the score. And that score determines the likelihood that the person is going to have a seizure. and they use that information to try to figure out how to treat the patients and whether to move EEG equipment around and so on. These are like tiny little models that you could memorize potentially, like a little formula.

If you're working on different types of data than tabular data, like you could be working on, say, images, then my lab works heavily on interpretable neural networks for computer vision. And so these models, for the most part, they use case-based reasoning. So case-based reasoning is a kind of like this looks like that type logic. When you're analyzing a new image, you would compare parts of it to other images that you've seen before because they're in the training set. So you could say, well, okay, I got to classify this image of a bird. What kind of bird is this? Well, the head of this bird looks like the head of that bird. I know what that bird is, right? That's a clay-colored sparrow.

The belly of that bird looks like the belly of this other bird, right? The same texture on the belly, you know, things like that. So these models are called ProtoP Nets. They've become really a very popular type of interpretable neural network for computer vision. Interpretable machine learning is really broad, so you can work on all different model classes. You can work on. We work on GAM, so generalized additive models. We also work on decision trees.

We work very extensively on sparse decision trees, which have if-then rule-based logic. We also work a lot on visualizing high dimensional data using dimension reduction. So taking a high dimensional data set and projecting it down to two dimensions so that you can get a kind of a bird's eye view of what's in it. You can see all the clusters and you can see all the manifolds and how they connect to each other. And so that's another area that we work in to try to understand high dimensional data in an interpretable way in a nice way. Yeah, so that's just some examples of what the field is. We've written some review papers that kind of talk about it. You know, it's basically models that are constrained,

like that actually have constraints so that humans can better understand what they're doing. Thank you for that explanation. That was very detailed and it really helped. One question, are there a certain set of applications or particular scenarios where interpretable models are more useful? Are they sort of broadly applicable across most use cases that you'd imagine sort of a layperson using? I know you mentioned high stakes earlier, so I was wondering if you could qualify that a little bit. Yeah, so interpretable models are really useful when you need to troubleshoot.

If you don't need to troubleshoot, then maybe they're not that useful. Like if the model is 100% accurate, then maybe you don't need to troubleshoot it. So you don't really need it to be interpretable, right? It depends on the situation there. What my lab is found is that when the data are somewhat noisy, meaning that there's a nondeterministic relationship between X and Y, then interpretable models tend to be very competitive with the black boxes in terms of accuracy. In other words, things like recidivism prediction, like criminal recidivism prediction.

You're trying to predict whether somebody is going to be arrested for a crime of a certain type within a few years of when they're released from prison. That's something you can't predict very well in advance because you don't know what's going to happen to that person in the next three years. And so for those kinds of problems, black boxes tend to do just about as well as interpretable models. So those are cases where there's no reason to use a black box because it's, you know, these are high stakes decisions, like they're decisions about people's freedom. And so you don't really want to leave those to a black box anyway. So, so I would say, you know, non-deterministic cases, high stakes cases like that. Interpretable models are really good because you can,

you can troublesheet them and they're just as accurate as black box models. Got it. Okay. And in general, do you get a lot of questions? Like, what is the biggest myth you get in terms of tradeoffs between these two? The biggest myth is the one you just mentioned. So there are a lot of people who are still wedded to the idea that when you add more complexity, you get more accuracy. And for a lot of these problems, the datasets just don't admit more accurate solutions when you add more complexity. They just overfit. And so people, they really don't like that idea because a lot of machine learning is kind of, I mean, this is my sort of theory,

that a lot of machine learning is kind of founded off the idea that you're working with, like, super clean data sets. like, for instance, if you're doing image recognition, you don't really have as much noise in the data for some of these problems. Like, if you have 10 identical images, then either they all have a chair in them or they don't. Whereas if you have 10 medical patients, they have the same medical record, then it's possible that, you know, half of them can have a stroke next year and the other half might not, right? So these are like very different problems. And so people try to use the mentality of these very clean data sets. on sort of realistic data sets, and it just doesn't work. Like, it's just not, it just doesn't apply.

And so you have to think about the statistical considerations when you're talking about interpretability. And a lot of people just can't do that. They just say, no, black boxes are more accurate, no matter what. That's where it's really hard to change people's minds. Is it fair to say that when you work with interpretable machine learning models, the domain experts that you're working with, one have a better way of providing feedback?

is that necessarily only to sort of computer scientists that are working with that model, or is that passed on all the way to what I would call the end consumer or customer? So the domain experts I work with are neurologists and radiologists and power engineers and police officers. So they're the ones who need to understand the model. Yeah. And if they can use that model, if they can do better, it definitely gets passed on. You know, yeah. Yeah, no, fair enough. That makes it a lot. of sense. How should we think about sort of accountability in these situations, Cynthia, when an algorithm makes a decision, these are critical decisions that are being made, as you explained. Yeah, so I'm

working on decisions that are high stakes and they're generally made by people, right? You don't really want to outsource a lot of these high stakes decisions to AI unless there's like serious time pressure and like a human couldn't do nearly as well in that situation. But the cases I'm working in are cases where you're assisting a human, like you're assisting a radiologist to analyze an image or you're assisting a neurologist to make a decision about a patient. So these are high-stakes decisions and they're AIDS for humans. And so the accountability rests with the human and we're just trying to help them. Understood. Yeah. Are these models harder to train? Are they more cumbersome or is it more expensive to train them? I'm not sure because it's hard to weigh the different costs of training these

things. So for us, there's a lot of troubleshooting of data, right? There's a lot of like algorithmic development. There's a lot of talking to domain experts, right? Those are things that a lot of Black Box developers don't have to deal with, but they have other challenges, right? They have to obtain very large data sets. Ideally, they should obtain them legally. They have to deal with, like, you know, a lot of hardware. I mean, we have to deal with hardware stuff too, but not nearly what they have to deal with. So I think the challenges are just different. Like, for example, you really can't compare us building a radiology model to OpenAI building JadGBT or something like that. They're which is very different problems, different goals, different data.

I know in some of your other work, you also talk about allowing users to sort of interact with models and be engaged in picking the right models for their use cases. Could you tell us more about why that is significant? Yeah, so this is actually something I'm really excited about. When we first developed the two Helps 2B Score with Brandon Westover and Burke Ustyn and Erick-Earnstruck, Burke printed out like 100 models that were all about equally good. for the data. He just literally handed Brandon and Aaron this package of models. And they had to look through those models and figure out which one of these is going to be the one that we're going to use on the patients, right? This involved like not just the data because the data was limited,

but it also involved their domain expertise, right? So what they knew about the important variables and stuff. Because there were so many equally good models, we could give them like a big choice, like what to choose from. But like giving them a giant packet of paper is not, that's one way to do it, but it's not really ideal. And to be honest, before what people normally do, they don't even hand the domain experts a packet of paper. They just hand them one model because that's what machine learning algorithms return, right? They just return one model at a time. That's not great. That is really, like if you hand a domain expert one interpretable model, they will find problems with it. They're not going to want to use it. They say, oh, this model is not good. And you say, why not? And then they try to

describe to you what it is. And you're like, okay. And so you try to reformulate the problem. And this is just a giant mess. And so we call this the interaction bottleneck. So it's the bottlenecks that's the interaction with users. And so to try to avoid that, we developed a new paradigm for machine learning that doesn't just return one model at a time. It just returns all the good models, all at once. And so you got maybe several million models or something like that that you're storing. And then you have to provide that to the domain experts. But you've got to provide it in a way that they can look through those models so that it eases the interaction bottleneck. And so we worked with human computer interaction experts to develop interfaces to these models.

And those interfaces are what the domain experts can use to kind of search through all of these good models to find one that doesn't just agree with the data. but also agrees with their domain expertise. And so we have these beautiful visual tools that kind of index the Roshamun set, the set of these good models so that they can pick models from there. So that's what we've been trying to do. It's a lot of fun. It's really rewarding. And I think it's going to totally change the way that domain experts interact with machine learning,

scientists and algorithms and data. ACM Bytecast is available on Apple Podcasts, Google Podcasts, Google Podcast, Podbean, Spotify, Stitcher, and Tune-in. If you're enjoying this episode, please subscribe and leave us a review on your favorite platform. Yeah, that sounds, I mean, it's an incredibly powerful way

to one, you know, give more agency to the domain experts to be able to pick the model. I'm sure that like to your point earlier, that rather than sort of poking holes at one model, they now have the ability to sort of decide between multiple and determine which works best, When you were talking about sort of visualized, I mean, what do each of them come up with their own metrics for how they decide, which is sort of the most applicable model and does the tool that, you know, helps them visualize these models? How does that adapt to their decision making? The answer to that question is not fully determined yet because we're still kind of like throwing this out there. We've done some like user studies with domain experts, but the answer to your question hasn't been completely resolved. And I think different people kind of work differently. But we want to kind of make it look a little bit like an encyclopedia.

so that people can kind of look up models of this kind of dig down into those models, and then look up models of a different kind of dig down into those. And so what we envision is that people will sort of go, oh, right, we really don't want models that look anything like this or this or this or this or this. And then there's a whole portion of the model space that they've excluded, and so they've narrowed it down a lot. And so that's what we're kind of hoping is that they can really help us narrow down the model space quite a lot. and then deciding between those models, well, that's either something they can do themselves

or we can help them with it. They can say, oh, you know, you need more data about this. And then we can say, oh, okay, and then we can build that. That's the ideal is that they can reduce the hypothesis space tremendously just themselves just by looking at this thing. Got it. Yeah, that makes sense. A lot of these, the domain experts that you speak of come from fields outside of computer science, but they seem to be very open.

They understand the value of using machine learning to solve some of these problems. Has there ever been a time given that you've been working in this field for a while where there was more skepticism? I think there's a lot of skepticism in general, and there should be, because you've got a ton of people selling explanations of black boxes, and those are, you should really be skeptical of those. Because, like I said, they're just like a poke at the black box, and these guys are selling these things as like actually interpretable, and you can't use those for high stakes decision. So I'm actually happy with the skepticism. I admit the people who select me is their collaborators and the people that I like to work with, they're people who know what I'm doing. Like Brandon Westover, the neurologist, he's trained his own machine learning models, right?

He's got this scheme that trains neural networks now. And my radiologist colleagues know exactly what they're doing. They also train their own models. And I work with people who are experts in heart monitoring. They also have their own teams now the trained machine learning model. So I work with a lot of domain experts who are very well educated and they know what they're doing. The power engineers that we work with even had statisticians on their team in terms of the people that I work with. It's often people who are quite educated. And the police officers that I worked with, they didn't even have, like, Dan at the time didn't have a college degree.

I mean, now he has a master's degree from Harvard, but he at the time didn't have a college degree. But he was able to try to read scientific papers. He was just an unusual guy. Like, he was just super, super smart. So I think it's part of it self-selecting, but that's okay. Because if we can develop tools that other people can use, even if they're not machine learning experts, that's fine. You see what I'm saying? Yeah, no, absolutely. And there's clearly, I mean, some folks that you've worked with her, you know, who are just exceptional in the ability to kind of spot the opportunity and then come seek you out to actually work on these problems. Thank you so much for sharing that.

And then I'll pivot a little bit to talk about ethics and AI. said I know is another area of interest for you. What does that mean to you? What does trustworthy AI mean in practice? Trustworthy AI, I think, is a huge field and it encompasses a lot of different things. And I think it does encompass interpretability because you don't really want to be using a black box for these size stakes decisions if you don't need to, right? If you can use an equally accurate interpretable model that you can troubleshoot, then you should be doing that. It's such a broad field that it's hard to sort of like trustworthy AI encompasses kind of the data and its provenance and the code and it's quite broad. I like to sit in my narrow little corner of it,

just because it's a huge thing. Actually, let me take that back. My corner of it is not narrow and tiny. My corner of it's important and people don't understand how essential interpretability is to trustworthy AI, right? This is something that I've had problems with the last 20 years. I mean, like I said, people used to, I used to give talks and people were very skeptical. And sometimes people would come up to me afterward and they would start yelling at me. Why do we need this? We don't need to see what's inside the black box, right? We just want it to work. We want it to be more accurate. And I would say, but my models are accurate. They're just as accurate as the black box models, and now I know why they work. I think people just didn't really understand kind of how essential this is.

What was the hesitance there? Was it speed? Like, what was the objection? I'm trying to understand. I mean, if you have a model that is equally accurate, but also interpretable, what would be the objection to it. Exactly, right? They just don't believe that that can possibly happen. I see. Okay. Got it. This idea that an interpretable model can always be replaced with a more accurate black box model, that pervades everything. I mean, even my own friends who work in interpretability sometimes believe this, right? I'm going to pick on my friend because I know she doesn't mind. So I have this friend who's a famous computer scientist. She's a famous AI expert, and she is one of the smartest people that ever existed on this earth. Her name is.

as Regina Barsely. She's a breast cancer survivor, and she was designing a model that predicts breast cancer five years in the future. And she also really cares about interpretability. And she told me, Cynthia, this model, you can't replace it with an interpretable model. It's a black box. Nobody knows how it works, but it can predict breast cancer five years in the future. I said to her, really, are you serious? And she said, yes. And I've built it on MGH data, tons of data. I tested it on Emory data, and it works, and she published it in radiology, and it's amazing. And so I said, you know, okay, I'll go look at it. I brought my team of radiologists and students to take a look at this model. And luckily, Regina made it public,

and so we could actually play with it. And within a short time, we figured out what was going on, and we built an interpretable model that was just as accurate. And so it turns out that her model, had been detecting very subtle asymmetries between the left breast and the right breast and the mammogram. Her algorithm was like a classic black box algorithm. It's like a bunch of convolutional layers and a transformer. And transformers like they just churn up data like it's a smoothie. I mean, there's just no way you can reverse engineer

what's going on in the transformers. But once we figured out that like these models were detecting these subtle asymmetries, we actually were able to remove the transformer altogether and just create like a symmetry detector for the mammogram. And so we got a model that was just as accurate as her model, but it's actually interpretable. And so because of that, it's actionable. Like we can pinpoint exactly where in the mammogram, it's asymmetry is between the left side and the right side. And so we can actually, what we want to do is take this model and be able to predict breast cancer in advance and tell people when they need to come back more often and tell

people when they could come back less often and so on. That's an example where, like, Somebody, like one of the smartest people in the world, thought it's got to be black box only. And it turned out that that wasn't even true. The models that you talk about, Cynthia, are they broadly deployed in the field to a lot of organizations, hospitals, or clinics? Do they adopt these models? And are they starting to be used more widely? Well, the two helps to B score that I talked to you about earlier is used in most intensive care units in this country that have EEG monitoring. This is brain monitoring.

There's brain monitoring for critically ill patients, right? This is a very common model that we published, and it's, you know, you can just look it up. You can just look up two helps to be score, and there it is. Yeah, so if you end up with the brain injury and you end up in the ICU, there's a decent chance you'll get scored with our two helps to be score. Hopefully that won't happen to you, but still, let me get the point. It is harder to adopt other methods. So the deep learning methods are harder to, because you've got to get them approved by the FDA for health care. And so a lot of those models take a while to get it approved.

A lot of models have been approved by the FDA, the black box models, and then they didn't work out. So you do want to be careful about launching things too quickly. Right. And then like I told you, our crime series detection method has been used by the NYPD since 2016 to try to figure out which crimes were part of a series. It is easier to get interpretable models used in practice generally than black box models, I would say. but, you know, we're doing our best. We're doing our best. Yeah. Yeah. I hear you, right, based on all that you've described so far, I mean, interpretable models are just, it feels like innately would be more trustworthy simply because you can see what's going on or you can understand what it is and you can provide feedback and tweak it.

That sounds like something that, you know, irrespective of what your field of expertise is, I think you'd want to be involved, especially in these high-stakes situations. One big challenge in getting a lot of things implemented is the lack of data for high-stakes decisions. So, for example, what I've been hoping is that the U.S. government starts producing data sets because they, like, the government already has proven itself to be really good at producing data sets and creating challenges. So they have, every year, they have this like facial recognition challenge. And they evaluate facial recognition algorithms from all over the world. I feel like they've, like, NIST has been a National Institute for Standards and Technology. Like, they've been a real driving force behind the quality of facial recognition methods

because they created a data set. And I don't see why they can't do the same for health monitoring. Like, I don't see why they can't revolutionize heart monitoring by providing a giant data set of, like, ECG or PPG signals, which are the signals that come out of smartwatches, right? heart monitoring, for instance, is it could make a huge difference for a ton of people if we can do heart monitoring better. But right now, the only places you can get really big data sets are if you work at like Apple Watch or something. And since I don't work at those places, I can't access those big data sets. And the public data sets are terrible for heart monitoring. And so the lack of

data has really been a major challenge for designing any kind of models, Black Box or interpretable. And is collaborating with industry on some of these initiatives? What's been your experience there? Like if you went to Apple with a proposal, Apple just being one example. But I'm just curious if you've had those experiences as well. I'm not really interested in working for a company and taking their proprietary data and producing a proprietary model. I'm interested in designing models that the public can use,

that's owned by the public, like models you can publish. So I can't imagine that a company would want to hire me to design a model that I then release. Collecting that data is expensive, right? That's their secret sauce. They're not just going to release it. Yeah. Yeah.

Fair enough. I would love to also understand from you, Cynthia, given that we have a lot of sort of young professionals who could greatly benefit from advice on how to navigate their careers and maybe an additional lens of, like, say, women in computing. What has been the single or a few incidents that have shaped your approach towards sort of problem solving

or the choices that you've made with regards to your career. I had some really good role models. So, you know, Ingrid Dobshi and Rob Shapiri were my PhD advisors. I could not have asked for better influences on my life than those two people. They're truly amazing. Ingrid, obviously, a free spirit. She does not care what anybody thinks. She has her own notion of what's beautiful, and she's going to pursue that.

Rob Shapiro is the same way. But Ingrid, she's out there. And she involved me in the women in math program at the Institute for Advanced Study. That program was a major influence on me, I guess, being around all these women mathematicians. And you don't feel like you're being judged. There's a lot of confidence issues that come with being in a field and not being the majority group. There's a lot of like imposter syndrome and things like that that people experience. Like, you really think you're the dumbest person in the room all the time.

it's important to kind of get over that at some point and just like fake it until you make it. I don't know how else to kind of say that. You know, I also don't think I'm maybe the greatest role model in the sense that I did get pretty down on machine learning for many years before I discovered the area that I really cared about, which is interpretability. And that came from actually working on a real problem with domain experts and realizing that what was in the field just wasn't doing it for me for solving this problem. problem. It took me a long time to figure that out. So I didn't have like a direct route to getting where I am. And it took me many years to get there. So not sure I would recommend people follow that. Oh, but that's real, right? I mean, I think a lot of us do have that journey. It's probably more regular or more likely to happen than not. And so, you know, thank you for sharing

that. I think that's very helpful and encouraging. And it's also nice to hear that you had these role models. And I mean, do you have any advice on how to seek out? role models. How do you find that person who'd be open to sort of investing time and energy working with you? I'm not really sure exactly how you find the right people. I just know that the first few people I found, well, I mean, I went through at least two PhD advisors before I found Gary Flake, and he was fantastic, but then he moved and he introduced me to Rob Shapiri. So that was just a coincidence. There's something to be said for choosing carefully and then realizing when it's not going to work out and switching to something else.

Yeah, the two people that I originally chose, they just weren't going to work out for various reasons. Their fields weren't right or their personalities weren't right. Yeah, I'm glad I didn't end up working with them because I wouldn't have found the people I found. And my goal kind of as an advisor, like I think about this from the advisor side, right? My goal is to make sure my students don't go through what I went through. I'm really proud of my students who, a lot of them are professors now. Like they went straight through grad school to being a professor, whereas I didn't, right? It was many years between when I graduated and when I became a faculty member. And I'm really proud of what I've been able to accomplish with my students. They're amazing. Like, these people are so smart and I'm so glad to have gotten a chance to work with that.

This is I'm talking about my students, my former students. Yeah. Your current students, right? Just honored to be able to work with them. That's wonderful. Thank you for sharing that. That's a very, very positive way to reflect on that problem or that situation. For our final bite, Cynthia, what is it that you're most excited?

about in this field of interpretable machine learning, say over the next five years? What I'm most excited about. Okay, so I'm really excited about Roshamun sets right now. So I was telling you, Roshamun Sett is a set of equally good models, right? And because there are a lot of equally good models, there are a lot of simple models. And so finding as many of them as we can is what I've been trying to do the last few years. So I'm really excited about Roshamons sets and what they can do for machine learning. I think a big question that people are asking right now, it's kind of an obvious question, is how do you make a large language model interpretable? And nobody knows the answer to that.

There are entire fields of people who are poking at the insides of these models trying to figure out what they're doing. But that's not the same as actually building a full model that's actually interpretable, like a model with interpretability constraints. And it doesn't help that you can't train LLMs, right? That's really something that you can only do when you're at certain companies and you have certain resources. and the time to do these experiments, right, these are very time-consuming experiments. People don't know the answer to that question. And it took us years to even get to interpretable computer vision models, right? So from 2012, when AlexNet came out to maybe 2019, when Protopinac came out, right?

We didn't know how to do it. I think that's an outstanding question that people are asking right now. We've been trying to build eugenic models or LLMs that use tools. And the tools have to be, if they're not. equally good solutions, we want to have the one that has the most interpretable or reliable tools. So we wrote a paper about that. And so we've been trying to kind of get at it from that direction, but I think there's a kind of more crucial role for interpretability that it hasn't yet played. Wonderful. Since you, it's been absolute pleasure to host you on our show. Thank you for

taking the time to speak with us at ACM Bytecast. My pleasure. ACM Bipecast is a production of the Association for Computing Machinery's Practitioners Board. To learn more about ACM and its activities, visit acm.org. For more information about this and other episodes, please visit our website at learning.acm.org slash bikecast. That's learning.acm.org slash B-Y-T-E-C-A-S-T.