Radiolab - 40,000 Recipes for Murder

Episode Date: September 9, 2022

Two scientists realize that the very same AI technology they have developed to discover medicines for rare diseases can also discover the most potent chemical weapons known to humankind. Inadvertently opening the Pandora’s Box of WMDs. What should they do now? Special thanks to, Xander Davies, Timnit Gebru, Jessica Fjeld, Bert Gambini and Charlotte HsuEpisode Credits: Reported by Latif NasserProduced by Matt KieltyOriginal music and sound design contributed by Matt KieltyMixing help from Arianne WackFact-checking by Emily KriegerCITATIONS:Articles:Read the Sean and Fabio’s paper here. Get Yan Liu’s book Healing with Poisons: Potent Medicines in Medieval China here. Yan is now Assistant Professor of History at the University at Buffalo.Our newsletter comes out every Wednesday. It includes short essays, recommendations, and details about other ways to interact with the show. Sign up (https://radiolab.org/newsletter)!Radiolab is supported by listeners like you. Support Radiolab by becoming a member of The Lab (https://members.radiolab.org/) today.Follow our show on Instagram, Twitter and Facebook @radiolab, and share your thoughts with us by emailing [e-mail].   Leadership support for Radiolab’s science programming is provided by the Gordon and Betty Moore Foundation, Science Sandbox, a Simons Foundation Initiative, and the John Templeton Foundation. Foundational support for Radiolab was provided by the Alfred P. Sloan Foundation.

Hey, everyone. We have a brand new story for you. Cannot wait to tell it and for you to hear it, but I'm actually going to make us both wait for just a second while I say two things. First, thank you. Thank you for listening, especially sustaining members of the lab. We are listener supported. It's how and why we do what we do That was a thing the first. Okay thing the second is

If you want to join the lab now is a really good time to do it Too sweet in the pot if you join this month you get a free radio lab t-shirt It'll make you look great, but also approachable Anyway to join and get that shirt go to radiolab.org slash join. That's radiolab.org slash join onto the show. Oh, wait, you're listening to radio lab. Radio lab.

From WNYC. Dordless listening to radio lab radio from W and Y Okay, so I'm I am this is unusual because I usually prep so much and I have not really prepped that much at all All right, you're flying fast and loose I'm flying very loose here with secret Intel about the government and poison? No, something. Is this about poison? Maybe. Okay, let's just do the thing.

All right, hey, I'm Louis Miller. I'm Lotte DeFnasser. This is Radio Lab. Okay, basically, I was just like nosing around and I found this article in a journal called Nature Machine Intelligence. Nature Machine Intelligence. Okay.

And this is a few months ago. It was in March. I don't even know what I was looking for when I was, and I just found this paper. And I had this weird, kind of boring title that I didn't understand, but I like started reading it. And the tone of it, there was something about the tone of it

that was sort of breathless. And like, oh my God, we just discovered this thing and it's kind kind of scary and we're not the only ones who are able to find this thing We're not the only ones who are actually looking for this thing and it felt like calamitous And by the time I finished it I was breathless like I was like oh my god like is this what I think it is because if it is This thing is terrifying. Okay, what and what's the thing? Okay, so let me just start from the beginning. I was just trying to manufacture out of nothing,

a kind of open cheese. No, I mean, I'm in. Okay, so here's what happened. So our scene begins with, la la la la la la la la la. You're muted. Oh, I'm muted. This guy named Sean.

Hey. I'm muted. Okay, named Sean. Hey. I'm muted. Okay, cool. All right. Okay. So Sean, I just, like, I gotta say, when I've read your paper, basically my jaw dropped and I wanted to hear you tell the story. Yeah, I'm happy to tell you how it came about.

It took quite a long time to get out. Well, before we get there, if you don't mind my interrupting, let's actually rewind a little bit. Like, who are you? What is your company? What do you do? So my name is Sean E. Kins, I'm the CEO of Collaborations Pharmaceuticals, and this is a company based in Raleigh, North Carolina.

I founded it 2015, and we work on using computational approaches for doing drug discovery. Which basically means what they do is they use AI to discover new medicines. You're a medicine hunter and you do it through computers. Exactly, yes. So essentially what they do is they've built like using a lot of like open source technology, a lot of like open technology, a lot of open source databases of drugs.

They basically created this computer program that's kind of like a search engine. And so they call it Megasyn. Or Megasyn and Megasyn. Yeah, Megasyn. So it was a really quick name for Megasynthesis. Oh, so, and sort of synthesizing drugs that exist with receptors and brains and kind of it's a little complicated It's one of those strange things where like I don't use it I have one of my employees that basically codes it and puts it all together. So that employee

Sorry, let me take my mask off here. He seems Fabio Okay, Fabio Urbina. Okay Sean and Fabio. Sean and Fabio Urbina. Okay, Sean and Fabio. Sean and Fabio. I love it. So what they do is they typically work with rare diseases, which aren't considered profitable. Big pharma has ignored them, there's no drugs for them. So what they'll do is they'll take one of these diseases that usually only a few hundred people have. And they'll be like, okay, we need a drug that will do a very specific thing in the body

to stop this disease, to stop the person from getting sick. Right. So they'll tell Megasyn, we need a drug that can do this very specific thing in the body to stop this disease, to stop the person from getting sick. So they'll tell Megasyn, we need a drug that can do this very specific thing. And then they'll hit search, and Megasyn will comb through all the available drugs that have been discovered, all the drugs that have been even evaluated, like this giant network of basically every drug

that's ever been created. And if from that, Sean and Fabio can't find a good match. Well, we're kind of out of luck. That's the end of that. Except it's not. Because Megasyn can do this other thing. It can put together a drug, basically. So how this works is drugs are basically just made up of molecules. And the thing about molecules that work as drugs is they have a certain molecular weight range. They have certain properties They're distinct. You can look at them and say okay, that's a drug or oh that looks nothing drug-like

And so using all these public databases and just inputting all this information in a megacent about chemistry Molecular engineering in a sense we've we've tried to train it to make things that a chemist would make And so with that knowledge what they do is they take the rare disease and they say, okay, Megasyn, in the infinity of molecules that could be drugs that don't even exist yet, can you make something? That might be active against our disease of interest. That could work here.

So Fabio will enter all this stuff in a Megasyn, hit run, and within minutes. It'll spit out. This brand new, never before seen molecules. Molecules that look like drugs. And then Sean and Fabio can go through these molecules and say, okay, this is the one we want to do this thing we need to do. Huh, huh. To disrupt this disease that humankind doesn't have a cure.

Oh my gosh, the speed of that just needs a moment. Yeah, yeah. It's kind of amazing. It's incredible. Yeah. Yeah. Um, okay, so this is, I mean, this is where things start to get dark. So it's 2021.

And these guys, they are just doing their thing and they get invited to this conference called the Spees or Spees, Convergence Conference. Kind of international security conference. And the goal of it really is to understand how technologies can be misapplied. Part of the theme is like this idea of like dual use. So okay, for example, dual use is like a nuclear bomb and nuclear power plants come out of the same technology, like the so-called like double-edged sword.

And so we got this invite, and of course we're thinking, Oh, well that's interesting. Why, why, why us? How can we misuse wonderful drug discovery tools? It's never even occurred to them. So the location was good. It was in Switzerland. They're kind of excited. I was like, oh, the pictures look really good. About going on a free trip to Switzerland as anybody would be.

Jumped at the chance. In the end, it was a Zoom conference, which was a bit, you know, not as exciting. But they were like, okay, let's think of something. Like, we could think of a way that we could, you know, we got this invite. Like, surely there's something we could do. So there's sort of like brainstorming like, okay,

if we were like really evil. What would we do? How would we miss you is what we know. They called it like their doctor evil plan. What would doctor evil do here? So yeah, it was very weird feeling. So they're like imagining and thinking it up, while we were running out of time,

when Sean has this idea, I hadn't given it a lot of thought, it was pretty quick. So, one of the things about Megacen is if we are trying to generate a new drug, we want to make sure it's not toxic. Fambio basically program this filter in Megacen. So that if the side effects of the medicine are going to be worse than the medicine itself, like, or than the disease itself, like, don't- Not interested. Don't bother. Because, of course, it doesn't matter if your drug hears all cancer, if it stops your heart from beating, it's going to not be a good drug.

It'll save my life, but it'll also kill me. So, I'll pass. Yeah, exactly. And so, Sean, he thought, what if instead of going right, we went left? What if we flipped the filter? What if we did the exact opposite, like photo negative of that filter of... Just spit out the deadliest?

Yeah, exactly. And so... I was using a 2015 Mac. That night. Just did a couple of copy and paste changes. Type the one where there was a zero and a zero where there was a one. It was that simple. It was literally that simple.

He hit run on Megasyn. And then went home for the evening. Next day, I did some work in the morning and then I think it was around noon, I just sort of opened up this file. And they check what they have. Fabio said, and I've got a list here, it's like overwhelming, tens of thousands of molecules. And so they skimmed like the top crop. These top toxic molecules and I looked at them. And so what they did is they would pick out a Megasyn molecule, put it into a public database. There's like a database search to see if these molecules already existed.

So they're going through this database to see if Megasyn had created anything terrible. Looking looking when all of a sudden they come across a match with a super hideous molecule called VX. So what exactly is VX? It's the thing that- From news today, a toxic substance was indeed the murder weapon. Do you remember that Kim Jong-un poisoned his half-brother in an airport?

Two women now in custody. That's VX. Wipeed his face with the toxic substance. It's a nerve agent. Developed in the United Kingdom in the 1950s. And by the United Nations and classified as a weapon of mass destruction, it is considered one of the most lethal chemical substances ever made. Ever created.

I mean, one way to think about it is if you think about salt. A few of those crystals of salt, if it was the X, would be enough to kill you. And if you did get exposed, your muscles would start to twitch, your pupils would dilate, you'd start sweating, then you'd start to vomit, after that your muscles would go completely slack. You'd be paralyzed, including your diaphragm, which would stop working. So you'd start to suffocate, and within a few minutes of being exposed, you would die. Oh, that's horrific. Yeah, I mean, it's awful. It's very awful. And Megazinn basically independently created it with the push of a button.

What did you think was going to happen? I mean, really, maybe it was just what did happen. Yeah. Well, I think what did happen was just the ease of it. I thought we would maybe get a few things that looked like VX. You did? Yeah. And we found a few, you know, in the literature and publications. A few horrible things humans had already created, and they figured that'd be it. But what we got was... Thousands of different molecules that look like VX. Thousands of brand new, never before seen molecules, they were actually predicted to be more potent than V ex-massively

more potent. Like old as a magnitude. This is just bad, right? Yeah, it's bad. It was like the alarm bells started ringing at that point. Because, according to Sean, if any chemist got their hands on this and wanted to make some of these molecules into weapons. If they did, because no one knows they exist, these weapons would be untraceable, undiagnosedable, incurable. A lot of this is so scary. It's really, really scary. I didn't sleep.

I didn't sleep. I did not sleep. There was that, gnawing away, we shouldn't have done this. And then I wish it just stopped now. Like just stop. But in a minute, we're gonna keep Stay with us. Lulu. Lutth. Radialab. So Sean and Fabio have opened Pandora's box of chemical weapons. Yeah, now we have this sort of phylinar computer and all of a sudden holds all these warfare agents. Was there a part of you that was just like delete, delete, delete, and just pretend this

never happened? Just unthink the thought experiment. That definitely crossed our minds. Yeah, we definitely had that sort of reaction to this. Like, I don't want to know any more. I don't want to know any more. But then they figured, wait a second, other people could do this. You do not need a PhD to do this.

You just need some basic coding knowledge, a basic laptop, and then all the data is available online for free. This could be something people are already doing or have already done. You know, these tools are in the hands of people that there is no control of. You know, anyone could do this anywhere in the world. And so the two of them in this moment were faced with this dilemma of now that you know what you know, what do you do? Do you tell people? dilemma of now that you know what you know, what do you do?

Do you tell people? Well, we have to make people aware of these potentially uses of the technology and show people that yes, these technologies can be misused. So maybe people could prepare for it, try to prevent it. Exactly. But at the same time, if you tell people... We could inspire instead of prevents. Then maybe people that would want to do this.

See how far they could push it. What do you do? What do you do? What do you do? Ah! It's exactly the double-edged sword. So what happened? So it's okay. So they make this decision together where they're like,

we gotta go to this international security conference and we gotta say this out loud, but we're not gonna show anybody the specifics and we're gonna tell them just enough that they know that this is a really serious problem. Yeah. And we got to flag it and hopefully someone smarter than us can figure out a solution to

this. And so they go to the conference, they present it to conference, they then publish a few months later, a comment in this journal, which is basically effectively what they say at the conference. And it blows up, well, maybe that's the wrong word to use, but it goes everywhere. Wired, scientific American. There was a thing on it in the Washington Post. There was a thing on it in the economist.

There was a thing on it in the news thing. And a lot of people like me stumble upon this thing. There's a lot of active discussion about it from not just chemical weapons people, but also AI people and people in the pharmaceutical industry and philosophers and you know, weapons people and like all kinds of different people are like weighing in on it and thinking about this and like what do you do here?

So it's kind of, I think they got what they want but they also like every night they're going to bed thinking like tomorrow I could wake up and some horrible thing could happen and it could have been because of me. Um, Lulu, Lulu, are you there? Hello, hello, oh my gosh, sorry, sorry, sorry, I think my mic just cocked out. I might need to just use my com computer mic right now.

Can you hear this? Yeah, yeah, yeah. If you can hear me, let's just keep going with this. No problem. That's fine. Just in one sentence, can you just introduce yourself for me? Okay. Okay.

So in the midst of reporting this story, I ended up talking to this expert. So my name is Sonia Benweigram Gormley. She teaches at George Mason University. The BioDefense Program. And I study weapons of my distraction, particularly biological weapons. And I found weapons of my distraction, particularly biological weapons. And I found Sonia because, you know, I just, I needed somebody else to talk to about this. I

needed some kind of outside perspective because when I read Sean and Fabio's paper, I legitimately thought this was very, very terrifying. Yeah, that's, that's the impression it gives. And very terrifying. Yeah, that's the impression it gives. And my point is that the thought experiment is just a thought experiment. It just shows that it is possible to identify new molecules. But there's a long way between the idea and the production of an actual drug or an actual weapon. Like, if you're a chemist who's going to make something that just exists on paper, the production of an actual drug or an actual weapon. Like, if you're a chemist who's gonna make something that just exists on paper, like that takes a long time, a lot of investment,

a lot of work, a lot of thought. And it's already hard enough to do it with chemicals that already exist. She said there's plenty of examples of scientists who try to transform them, who try to tweak them just a little bit to make them more harmful and very often they failed. Because it's just super, super difficult.

That's the point. And I told her it was still kind of hard to have wrapped my head around it because the way that I was thinking about it, I was actually thinking of like the anarchist cookbook, the molecules, it's almost like recipes or something, right? It's like this, the molecules is almost like recipes or something, right? It's like this thing spat out 40,000 recipes or whatever, right? And then you just go to the store and get some fertilizer and know and make it and it's the exact proportion. Exactly, so I was worried about that.

Right, I'm picturing that too. Is that the right analogy or would you use like a different analogy? Um. And she was like, no, you're actually that's the same metaphor I use when I'm teaching my students. But think about it this way. If we take the analogy of a cake, making a cake, these molecules are like really fancy cakes. And based on what I read, right? Based on

the article, what they have is a list of different ingredients. And to your mind, it would take a five star Michelin chef with a whole kitchen full of, you know, sous chefs to kind of figure that out and to ultimately make any of those cakes. Exactly, exactly. It's like a level of craft. It's a level of craft and expertise. It's like, yeah, you could make the David in theory

if you had marble and a chisel, but like... Right, but it's like, I mean, to take a step back, I feel like your takeaway about this paper is that you, you have not lost any sleep over this paper. No, no, I think I found it a little bit too alarmist. Huh. So, Sonia is assuming us. Yeah. Are you going to set this whole section to like a nice little abide bedtime? Like it's okay. It's okay. It's okay. No.

Because there's still a problem, which is that, I mean, Michelin Star chefs do exist. And, you know, I don't know if this could go on air just yet, but I don't think it's private or anything, but we were just contacted by the White House this morning. Oh my God. Really? Yeah, we weren't expecting that. So what did they say? They want us to brief them on the paper, basically.

Holy crap. Apparently, it's causing a lot of buzz in the White House right now. Whoa. And so I actually followed up with Sean after our first interview about this very thing. So we ended up doing a Zoom call with folks from the Office of Science and Technology Policy and the National Security Council, which was very surreal. So this was just in March 2022, Sean, Fabio and some of their team got on the Zoom call with these folks in their office in the White House, with the White House crest in the background.

So they do this presentation for the White House folks. I think they were worried that we were, you know, kind of crazy people and we were just gonna let all of this information out there. But at the end of their presentation, there's like a Q&A and the White House folks are asking them questions, is this information, you know, sort of locked away somewhere? And it is. So, yeah, you know, one of the first things I do is put in a file and encrypt it really heavily.

And it's on a computer that is not connected to the internet. It's air gap. Fabio has it on his machine, locked away and encrypted. But as the White House staff kept asking questions, Sean was like, oh, I was just waiting for the question. He's like anticipating it. It was just, when are they going to ask the question, the elephant in the room? And then finally they asked the question.

Can we have the data? And it's like another one of these moments where Sean has to decide, you know, maybe the government should know about this. So they can anticipate it, try to regulate it. But on the other hand, now you're actually handing over the list to one of the most powerful governments in the world. Like if anybody has access to Michelin Star Shafs, here they are.

And Sean, it's his call, right? Because he's the CEO of the company. And Sean says, you know, no. No. And to me, that was like, I was like, oh my God, how would you even? And how did they respond? Well, the reaction was basically, I think it was like, okay. I mean, it was pretty, all right, well, you just have to accept that. And what's your rationale for saying, why not share it with them? I just didn't feel that I wanted to hand it over to them. not share it with them. I just didn't feel that I wanted to hand it over to them. We had other scientists reach out to us as well asking exactly the same question. Now, they told the White House that I didn't feel like we should do that. We should give it over to them. I just didn't feel that it was right.

You were basically like, no to you White House and no to everybody. We're not sharing this with anybody. And you've told everybody about the existence of the list, but you're not sharing the list. What's the impetus there? I mean, it's bad enough to put the article out there and tell people how to do it and watch it sort of spread out across the globe.

But if I gave them the list of molecules, it's almost like a red flag to a bull, right? The chemist out there are just gonna try to figure out, all right, which ones are the easiest ones to make. And then just make them. And I don't wanna be the person that has to say, yeah, I'm responsible for that. And so for now, the list just sits on Fabio's computer,

kind of waiting to be used in the case of an emergency in case someone was to recreate it somewhere else, and actually start making some of these things. Reporting this story, I talk to a lot of people, AI experts, weapons experts, that kind of thing. Also, for my own expertise studying the history of science, this is a thing that just happens where something gets created and someone finds a way to use it for the opposite purpose. That's the story of the nuclear bomb and nuclear power. It's the story, and this is what I've actually done on the show before, of the nitrogen-fixing

process that can give you fertilizer to feed billions or gunpowder and explosives. This kind of thing just happens over and over and over in history. But I think this thing that struck me about what Sean and Fabio were doing is how when it came to change this machine from making drugs that would help people, it's just how easy it was. I just did a couple of copy and paste changes. To flip it. Type the one where there was a zero and a zero where there was a one. It was that simple, was literally that simple.

Like, the line between this thing doing good and this thing doing bad felt so thin. And it was in thinking about this that I reached out to an old friend of mine, a guy named Yan Liu. He studies ancient Chinese medicine. He now works at the University of Buffalo. And I called him because he always would tell me this thing that when it comes to drugs, there's no clear boundary between what is poison and what is medicine. That an individual drug can be good or bad for you, we could go either way.

It is the matter of a dose that make a difference. That it's all about the dosage, or also about the intent of the person giving it to you, or about the body of the person who's taking it. It's more about the context than it is about the drug itself. Exactly. And I showed him Sean and Fabio's paper, and he was like, oh, this feels like a toxicology paper. This separation between pharmacology and toxicology,

you know, we have a long tradition, especially in the West, it's a sense of separation between what is good and what is bad. But he's like pharmacologists and toxicologists are studying the same molecules. But it's like, we and toxicologists are studying the same molecules But it's like we just want them to be separate exactly, but every substance Has the potential to either heal or to cure And that's what I keep thinking about about the story is that there's no real line here

Everything is capable of everything and it's just a matter of how we choose to use it. Yeah, it's like the idea that there is a line or that you could ever stay on one side of the line is, I don't know what the myth of blind spot. Yeah, it's a bitter pill to swallow. But it is also apparently a sweet pill. Right, a spoonful of sugar to help the medicine go down. No, like in the office, they're the side of the bitter sweet. It's a bitter pill, but it's sweet pill at the right dosage or something. But maybe the fact that most of us are not Michelangelo's or great chefs,

that that alone, it's the sadness in terms of like curing diseases, but maybe it's the comfort. Yeah, I don't know if that's my silver lining takeaway. It's like, no, we're too dumb to make chemical weapons. That's kind of my, I mean, that's actually just sifting through all of this is like, I think I will be able to sleep a little better at night

by thinking about the fact that most of us are not Michelangelo's. The Michelangelo's of chemical weapons, yeah. Yeah. But even sculpting, like there are a lot of people who are like woodworkers or carpenters or the rest of you. Yeah, but there's like no one who's Michelangelo. That's what I like.

But then you just need a 3D printer to just, you know. I mean, I think it's like why I'm settling on Michelangelo's like, because I kind of still don't think a 3D printer could do it. Could print the David? Yeah. And because I'm like, aren't there like, if you go to any museum gift shop, like you'll find out?

Yeah, but they're not, yeah, but they're not. They're not like, How different are they though? They're teaching David. They are different. Yeah, like they are different. I don't think they're that different though like I Think it's really different. I don't think it's that different. Well, you don't get them like how the lights But the average person The In March 2022, the U.S. under its obligations for the International Chemical Weapons Convention

destroyed its last stockpile of VX. It is now destroyed over 95% of its chemical weapons and is slated to destroy them all by September of 2023. So, Mark here, calendars, I guess. ["Mathieu Kilti"] This episode was reported by me, Lattif Nasser. It was produced by the ever-non-toxic, Matthew Kilti, with production help from actual pastry chef Rachel Kusik.

Original music and sound design contributed by Matthew Kilti, with mixing help from Arian Wack. And given Lulu's mic situation, I'm going to sign off for both of us. I'm Lulu Miller. I'm Lattifnasser. Thank you for listening. Ha ha. Radio Lab was created by Jada Bimrod and is edited by Soren Wheeler.

Lulu Miller and Lattifnasser are our co-hosts. Susie Lektinberg is our executive producer. Dylan Keef is our director of sound design. Our staff includes Simon Adler, Jeremy Bloom, Becca Brestler, Rachel Kusik, Aketty Foster Keys, W. Harry Fortuna, David Gable, Maria Paz Gutierrez, Sindu Nyanasambandhan, Matt Kielte, Annie McEwan, Alex Niesen, Sara Carrey, honor rescue at BOSS, Sarah Sandback, Aryan Wack, Pat Walters, and Molly Webster. Our fact checkers are Diane Kelly, Emily Krieger, and Natalie Milton.

Hi, this is Susanna Culling from Washington, DC. Leadership support for Radio Lab Science Programming is provided by the Gordon and Betty Moore Foundation, Science and Box, Assignment Foundation Initiative by the Gordon and Betty Moore Foundation. Science and Box, Assignment Foundation Initiative, and the John Templeton Foundation. Foundation for Support for Radio Lab was provided by the Alfred P. Sloan Foundation. you