School of War - Ep 112: Paul Scharre on AI 101

Episode Date: February 27, 2024

Paul Scharre, Executive Vice President and Director of Studies at CNAS and author of Four Battlegrounds: Power in the Age of Artificial Intelligence, joins the show to talk about how AI will change the battlefield. ▪️  Times      •      01:38 Introduction      •      01:54 Becoming a Ranger     •      03:48 A defining moment      •      07:25 A historical parallel for AI     •      11:16 Hardware      •      14:10 “Taiwan is the Saudi Arabia of chips”      •      16:20 Military applications     •      19:37 Battle damage assessment and AI tracking       •      22:50 Autonomous weapons     •      27:50 Legal, ethical, and control issues     •      30:08 Battlefield applications     •      32:43 Operational ability      •      36:51 WMDs     •      40:09 Countermeasures      •      43:53 Transportable?     •      46:40 AI and nuclear weapons Follow along  on Instagram Find a transcript of today’s episode on our School of War Substack Buy the book here - Four Battlegrounds: Power in the Age of Artificial Intelligence

Artificial intelligence seems inescapable in 2024, and only a fool would assert that it's not going to be a significant factor in war fighting and national security more broadly. But how? What is artificial intelligence in the first place? Are there historical analogies that can guide our thinking about it? What are its military applications? How are they emerging? How do we expect them to further evolve? And what will the net effect of all of this on the battlefield be? Today, we are going to do A, 101. It is a prescription for war, this Iraqi invasion of Kuwait. December 7,1941, a date which will live in infamy. The bloody experience of Vietnam is to end in a stale. We continue to face a grave situation in Iran. We shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets.

We shall never surrender. For maps, videos, and images, follow us on Instagram, and also feel free to follow me on Twitter at Aaron B. McLean. Hi, I'm Aaron McLean. Thanks for joining School of War. I'm delighted to welcome to the show today, Paul Shari. He is the executive vice president and director of studies at the Center for New American Security. He is the author most recently of Four Battlegrounds, Power in the Age of Artificial Intelligence. We'll get into his background more here in a second, but he's also a veteran in the figurative sense. as a civilian in the Department of Defense, but he is a veteran as well of the Ranger Regiment and the U.S. Army. Paul, thank you so much for joining the show. Thank you. Thanks for having me.

So before we get into our subject today, which is your work on AI and its strategic impacts, tell us a bit about yourself. How did you grow up? How'd you get interested in service? How'd you end up in the Rangers? Yeah. You know, I think I went through like a lot of folks kind of, you know, crisis, decision point in college, what am I to do with my life? And, At the time, I guess this will figure out, you can kind of bracket my age here. This was during the Kosovo Air Campaign. And, you know, I just had this strong feeling that look at these things that are happening in the world.

And I think this is truly at any point in time, you could look out at things happening in the world today or back then and say, wow, these things seem significant. They're important. And I wanted to be engaged in U.S. foreign policy and national security issues. I was young. I had really nothing, you know, no skills, no building. It's coming fresh out of college. And I heard about, you know, sort of options for going into the military and special operations

that seemed pretty exciting, you know, to me, a chance to jump on a plane and crawl around in the mud and do all that fun stuff. And so I signed up with a Ranger contract. I joined in June of 2001. So it was actually in basic training when 9-11 happened. And then, you know, it was a busy, busy time that first couple of years. How many years did you stay in? I was in for four years on my initial tour, and then I got out.

And then it was called back in when we were doing that for a period of time. I did another year-long tour in Iraq. Got it. Was it all Iraq for you? Did you make it to Afghanistan at all? Yeah, some of both. Yeah. So I did three tours to Afghanistan with the Rangers and then later did a tour a year in Iraq

as a civil affairs specialist. In the Army's infinite wisdom, when they called me back, they'd made me the opposite, I guess, of an Army Ranger had reclassified me to Civil Affairs, and so did that. And so you go out on the other side of things. You come to D.C. How is it that this, the question of AI specifically, but you are, you've written extensively about sort of questions of emerging technologies more generally in the strategic impacts. How did you come to that as a subject?

You know, there was this like singular defining moment, actually for me. When I was in Iraq, when I thought about robotics in particular and their value and work. And we're driving down the road. we came across a roadside bomb and improvised an exclusive device. Now, we saw it first, which is the preferred method of finding them, right? Rather than just running into it. And we called up the bomb disposal team.

And so I was expecting, you know, the bomb tech to come out in that big suit they have and get up there and slip the wires. And instead they sound like this little robot. And the light bulb went on in my head. I was like, yeah, like have the robot to fuse the bomb. You don't want your face up there or just, you know, sniping the wires on the bomb. And then the more I thought about it,

And when I thought, well, like, there's a lot of things we're doing that are dangerous, that you could have robots doing. Why don't we have a robot doing this job? And so when I left the Army and I ended up working in the Pentagon as a civilian policy analyst, that was one of the things that I worked on was how do we get more robotics, more drones, more autonomous systems out in the field? And I just, you know, personally saw a lot of potential there to help protect our service members. Yeah.

Your experience of the robot versus the guy in the suit squares a bit with my experience, which was neither actually. It was the first time I saw an actual EOD tech work on an actual bomb. And for many times thereafter, the gunnery sergeant in question walked up to where we thought the IEDA was, pulled out his bayonet, began poking the ground, found the pressure plate, and then proceeded to diffuse the bomb, which was the accepted practice, circa 2009, 2010, where I was. Probably, I'm going to guess because of a mismatch between the number of IEDs and the availability of robots and other technologies you might use to find them.

It's a hairy job. I mean, and the bomb techs were really, I mean, just right on the front lines of these last wars. Oh, yeah. Actually, one of my favorite stories from that deployment involves the same guy who we, my platoon, found. What at that point was, at least so we were told, the largest improvised explosive device, really series of devices yet found in Helmand Province. It was a daisy-chained set of military-grade explosives that went down an alleyway where we were.

And we saw sort of the wire and indications that there was something down that ally that we needed to check out. Yeah. And the EOD team leader went over to the Marine Engineer Paltoon that was there and asked for the bulldozer to drive, you know, basically dig him a path down the middle of the lane so he could have a safe place to walk up and down as he was figuring out what it was. And the bulldozer driver said, and this, for anyone with children listening, the Falling will evolve. The Bulldozer said, you know, Gunny, this, this bulldozer is only going to be a little bit of all language. The bulldozer said, you know, Gunny, this bulldozer is only going to take one hit to which the gunnery sergeant responded. Motherfucker, I'm only going to take one hit. Yeah.

The bulldozer was persuaded to do the job. So sorry, we have an important topic here, and I want to get to it. And I'm grateful for you making the time today because the question of AI is something that I think is universally, people universally agree it's important. and then if not universally, perhaps approaching universally, don't really understand it. Maybe I'm making myself feel better when I say that because I don't feel like I understand it. No one understands it. It's not even. So maybe that can be my first proper question to you, which is how should we think about AI strategic impacts?

And more specifically, what is the best historical experience that warfare, whether in the American experience or just more broadly, that humanity has gone through in terms of technological evolution that we can kind of use to track or control our thinking about AI. Now, that's a great question, because sort of this search for what's the historical parallel matters quite a bit, and it can take you in different directions. If you're like, oh, AI is like nuclear weapons,

that might take you in one direction. You know, I think that actually a very common comparison, I think a good one, is to the first and second industrial revolution. And I'll tell you why, Because AI, like other technologies at the time, the internal combustion engine, electricity, is a general purpose technology that is a whole wide range of applications and is likely to be applied

across society in widespread ways, a wide range of industries, the same way that those technologies were. And we saw, I think, some really key geopolitical effects from those technologies. One is that we saw nations rise and fall on the global stage based on how rapidly they industrialize. And some nations like Great Britain and Germany

raced ahead in economic and military power by virtue of seizing a whole of these technologies and integrating them into their societies faster than others. We also saw the key metrics of power change. So things like coal and steel production became key inputs of national power. Oil becomes this geostrategic resource that countries are willing to go to war over.

And so I think AI is likely to similarly change, not just the balance of power, but even the key metrics of power, And then, of course, the Industrial Revolution transformed war in incredible ways, incredibly destructive ways that you saw in World War I and World War II that not just required militaries to develop new tactics and respond accordingly, but really ended up mobilizing entire societies for industrial production and increased the scale, the physical scale, and destructiveness in war in a way that was really unprecedented up until that moment.

Is there any way we can, so I take your point about the industrial revolution or waves of industrial revolutions, and it's obviously right that we should think in terms that are broad like that because it's not just a question of an application. You know, it's not the introduction of a bomber. It's broader than that. But we talk about industrial revolution is so broad. Is there something, you know, like, I don't know, electricity, the introduction of electricity. Is there something that is a kind of a concrete example that you've played around with that would make sense here?

I mean, I think there's a couple, right? So Kevin Kelly, the sort of tech guru and thinker, has made this comparison to electricity. I'm going to paraphrase here. But he basically said that everything that we have formally electrified, we will now cognize. This idea that, you know, today we live in this world where we have physical devices that have been imbued with power via electricity. and increasingly networked also as well, and that we're going to be layering on top of that stack of powered devices, network, which is we're layering around artificial intelligence now, and that these devices are going to sort of come alive and have the ability to process information

and think to some extent, right, and engage with us in ways that will make them not just be powered, but also intelligent. Right. Okay, so I'm going to punt on the harder question here, and we'll come to it in a few minutes, which is, what does that, what does that mean? What does it mean that they think and what are the applications and how do we deal with them? But just to stick with something that at least I find easier to comprehend with the hardware side of it. There's just stuff you need to have AI, to develop AI, and that stuff is made in certain places and not made in other places and different people control it. And that just introduces a sort of a normal kind of geopolitical dimension to the question that applies across any kinds of other number of energies, the obvious one. issues that we all think about a lot of the time and sort of understand.

Help us understand the hardware dimension of AI and how it impacts strategic thinking or ought to impact strategic thinking. Yeah. So hardware is a really essential element of this geopolitical competition for AI for a couple reasons. At the technical level, there's three key technical inputs to current machine learning systems. Data, algorithms, and hardware, the chips that are used. Machine learning algorithms, there are algorithms that are trained on data using computer hardware.

So there's an algorithm, you feed in a bunch of data, sometimes these massive data sets. Some of these large language models are trained on trillions of words, huge data sets, and they're trained using very large amounts of computing hardware. So the most advanced AI models are using tens of thousands of the most advanced chips. When you think about these three inputs of data, algorithms, algorithms and hardware. Hardware is unique in that it's a physical asset. It is, you know, easier to control, very hard to say control the spread of algorithms. It is a rival asset in the sense that if I have a chip, you cannot have the same chip. You cannot use the same chip at the same time. And it has a unique position and that the United States has a really unique advantage

in the supply chain. Now, it's not actually that these chips are made in the U.S. In fact, none of the most advanced chips in the world are made in the United States. They're made in Taiwan. Now, on the face of it, this is terrible because, of course, Taiwan is an island 100 miles off the coast of China that the Chinese Communist Party has pledged to absorb by force if necessary. That's not good for the U.S. But all of the technology to manufacture the chips relies on U.S. technology and U.S. software. And there's actually three countries in the world, the United States, the Netherlands and Japan, that control the market for the manufacturing. manufacture equipment for the most advanced chips.

And we've seen the US use this position to lock China out of the most advanced chips by placing export controls that are extra territorial, they're outside of the United States, on advanced chips going to China. So even if there's a Chinese chip design firm that designs a chip, they send it designed to Taiwan to make the chip, and it's going to be shipped back to China, nowhere comes to come to the United States, U.S. export controls say you can't do that because that FAB in Taiwan relies on U.S. technology and U.S. government said you can't do that. Help us understand what this all means, though. So is it just as simple as the fossil fuels

analogy, whereas, you know, to a degree that's still quite significant, even in 2024, some people have oil, some don't, some people have gas, some don't, and those places that have it are significant as a consequence and need to be factored into strategic decision-making. Do I just apply the same thinking to chips? Is it more complicated than that? Well, yeah, in a lot of ways, it is more complicated. There are some analogies, right? So I think it's, you know, not a bad comparison to think about Taiwan's position

at sort of this geopolitical fulcrum from hardware as maybe roughly analogous to the role that Saudi Arabia plays in the oil industry, right? So I say something like, well, Taiwan is the Saudi Arabia of chips. I think it's like a useful maybe shorthand for thinking about how important Taiwan is. Of course, one of the differences is oils produced in a lot of places. The oil market is very globalized and oil is a fungible asset, whereas the chips literally only come from Taiwan. Now, a little bit unclear how effective U.S. export controls are going to be in terms of, you know, there's ways around controls. China is working to indigenous in their own chip making. They've got

a steep hill to climb here, but they're working on it. And there are opportunities for smaller AI models that don't rely on the largest, most advanced, you know, sense of chips. And then right now, there's a very vibrant open source community in the United States where we actually just release these models open source, which is sort of like this big, maybe here's another historical analogy here. It makes these expert controls a bit like a imagine no line here, where we're sort of blocking China from getting access to the chips. But if they get access to a fully trained model open source, they don't need the chips in the first place. And that's right now where the U.S. AI ecosystem is. Right. Okay. So let's let's let's

shift into the hard stuff then these algorithms, the models that they help to produce when they're, when, when, when the data is run through them. I guess by the way, actually data itself, by the way, is another another kind of semi-physical consideration in the sense that some people have more access to data than others, but we can we can talk about that maybe in a minute. But, but to get to the thing itself, there is this technology, this tool, this, this, this, this platform out of which many applications can be made that is the end result or the near end result of this process. What are what what is it? Let me just start with that and then what are the most important applications in military terms we ought to start thinking about or be thinking

about. Sure. So what happens is you start with this data set, a very large data set. It could be images, for example, or text or sometimes combinations of both. And then the, algorithm is trained on this data set. And what the algorithm does is it learns to create a model that represents the types of things inside the data. It was to generalize somewhat from that data set. So if you were, for example, trading an image classifier. An image classifier is a model that's used to identify objects. So what you could do is from a military standpoint, right? You could have satellite imageries of different objects and they're all labeled. And you feed it into this AI model and you say, look, this is an image of an airfield.

This is an airplane. This is a tank. This is a radar installation. Now, you need millions of images, thousands of each different class of images. You need a lot of data. And worth pointing out more than you would need for a person. But if you give an imagery analyst, you might need to give them a couple examples of a thing. You don't need to give them thousands of examples.

And for a while they'd be like, I got it. The AI actually needs a lot of examples. But the end result is a trained model that then you could feed in new data that's never seen before. And if it was in the class of things that was trained on, it could identify it. If it's something novel, it's never seen before. The AI is not going to be able to do that. Can't generalize very well. And these systems are, that's the kind of thing that the Defense Department is already using.

If you go back to Project Maven, the DoD's first big AI project coming out of the Deep Learning Revolution back in 2017, that was what, DoD was doing use image classifiers for drone video feeds. And that is just one application that's very powerful because you can imagine in this front, DOD, the Defense Department is collecting and the intelligence community way more information than they could ever process with people, whether it's signals intelligence, imagery intelligence, other things. And so AI tools could be a good way to just sift through that information and sort it out and kind of help humans makes sense of it. Yeah, well, this this brings to mind the 2021 Gaza War that the Israelis branded as the quote unquote first AI war. And I think this is the process you just described is essentially

what they were talking about, that they have a lot of sensors in Gaza, ultimately an incomplete complex, as later years would reveal. But nevertheless, a lot of sensors in Gaza collecting a lot of data. And so they applied this technology to target more efficiently. In fact, my understanding is still with humans in the loop, but rather than human analysts, as it were, deciding whether or not, you know, I'm going to make all this up, but we might imagine this or that phone call was relevant and suggested that one of the people was on the line was significant or not. Like, you could actually task some of that to a computer initially, such that by the time the human encountered the stuff, the data, there was a degree of analysis that had already

occurred, the speeding up the cycle, speeding up the loop and making the campaign more lethal. So that's basically like the sort of application number one that we all, well, if we don't all understand, but is sort of out there. Is that fair? Is that a fair characterization? Yeah. I'll give you two like actual real examples that the Defense Department did. That when I was researching my book for Battlegrounds, the DoD gave me incredible access to

the then the joint AI center. They've now morphed into the CDAO, this new organization. But General Jack Shanahan, who was running the AI Center at the time, you know, gave me just incredible access to their people. And so two things that they were working on. One was looking at battle damage assessment. They were doing it for disaster relief. And so what they did was they built a model

that could look at flooded areas during like a hurricane and then could help create a map for people to first responders to make it navigate through flooded areas. To say like based on this imagery we're looking at, the satellite imagery, this road is flooded, you need to take a new way to do it. So you can imagine for the military,

that kind of tool would be very valuable in a war zone in terms of doing real-time updating, mapping, and navigation. And they had a parallel version that was looking at structures damaged from, like, wind damage, for example, in a hurricane, and licking that to FEMA categories of destruction, and then using that to create sort of a first cut for first responders to say,

Here's where things are damage. Again, valuable military applications. When we think about doing battle damage assessment. So those are like some of the things that we saw people working on. Another example that they were working on was the ability to use image classifiers in drone video feeds to say, watch a building,

track people coming and going from a building or vehicles, and then through some of these wide area surveillance tools like Gorgon Stair, that are looking over like an entire city, to use that to rewind the video recording over time to then track people. So let's say that there's a car bomb. We don't know where the car bomb's going to go off. It happens.

Now we have video footage of that city for the last 24 hours. You could in principle have a human analyst, you know, watch this video in reverse and see where the car came from. It's just time consuming. Having an AI do that a lot faster, the AI can zip through this, find a building, and then you can say,

okay, I want you to identify the timestamp of every time a person came in or out of this building in the 96 hours prior. And using AI tools to speed up a lot of the things that humans might be doing. So what you just described seems like a sort of, from a purely military perspective, sensible and really welcome, actually,

kind of way of dealing with the sophistication in the present day of sensor strike complexes that any modern military has. We have so many sensors, we have so much information, thank goodness, that we now have technology that can actually help us be efficient about that information. We start to run into more complicated considerations, really interesting considerations that you've written about a fair amount. When we start talking about autonomy and removing humans or reducing the role of humans in the loop, because I think everything we've discussed till now is basically analysis that tease up human decision making.

Talk about autonomous weapons, not quite the same thing. as AI, but it relates to AI, just help us understand what that means. Yeah, so if you look at maybe what we're seeing on the battlefield right now in Ukraine, there might be a good starting point. We're seeing obviously a lot of small drones being used to target people and vehicles. And one of the things that we're seeing incorporated into these drones is more autonomous terminal guidance. So right now, the drones are still piloted by a person, and a human is sort of choosing the target.

But there is a lot of counter-drone innovation going on in the war in Ukraine, on both sides, electronic warfare systems, jamming, and that communications link to a human is a point of vulnerability. If you jam the communications link and the drone is remotely controlled, it's no longer valuable. It's not going to do anything. We've also seen electronic warfare tools to find the location of the drone operator who can then be targeted. If you kill the drone operator, that's also a good way of making the drone ineffective. And so all of that creates pressures towards more autonomy. And we've started to see companies incorporate this into drones in Ukraine.

One Ukrainian company claims last fall that they had fueled it and was used in combat a fully autonomous weapon. Now that's not independently verified. That's their claim. But we certainly are seeing, you know, image classifiers, things that could identify objects. This is a tank. This is a vehicle. Have been used in Ukraine.

They're on drones now. I'm not, I think, maybe widely spread, but they are used. used. And so adding more autonomy where you can get to the point where either a person chooses a target and then can kind of go hands off and the drone can do terminal guidance. That's valuable. We're starting to see that. And I think over time, if the war continues to drag on, we will in all likelihood see autonomous weapons where someone is deploying a drone or sent of drones into an area where they know there are enemy targets, people or vehicles. And then the drones are choosing their own targets and attacking them. And humans are still involved, but they're just a little bit further removed.

Yeah. So, well, I guess that your last comment points to my question, which is how different is this really, except perhaps somewhat more sophisticated than, say, very at this point, traditional guidance technology from, you know, heat seeking missiles. The missile goes off into the world. Its job is to find something hot and blow itself up next to it. If anything, this is maybe just a little more precise, a little more sophisticated, but basically the same thing. I put a question mark at the end of that sentence. Yeah. Please.

I think there is a big conceptual difference between the idea of humans making these decisions about whom to kill a battlefield and humans delegating that to machines. The question is when you start to look at it closer, there's not like just one decision. There's lots of small decisions that get made as part of this targeting cycle. Am I, you know, where are the enemy forces? You know, what are the things I'm targeting? Are they here in this point in time? I think one way to think about this is right now, a lot of the systems that we have,

homing munitions, torpedoes, many of them are fire and forget. Once it's launched, it's not coming back. You don't want it to come back, in fact. And many of them have some seeker on board that can sense an enemy target. But today, humans are launching them at some known or suspected enemy target. They have some indication that there is a valid enemy target at some point in space and time. And shifting to a world where the human is instead saying, I'm going to launch it into this area to some killbox and I don't know exactly where the enemy is. It's sort of a

qualitatively different thing. It doesn't mean that it's necessarily immoral or legal. It depends actually quite a bit on the context for the use. But it is a different role in human decision making. And then you could imagine that kind of killbox begins to expand over time. And instead of being one small area, it turns into a bigger area and a bigger set of targets. There is very much, I think, a risk that we make these incremental moves towards autonomy and then look back and go like,

oh, well, we actually moved quite a bit. And we didn't realize it. At some point in time, we did cross a meaningful line, but it maybe wasn't as obvious at the time. Well, wouldn't, I mean, don't you think it's fair to say that whatever considerations we may now have? And I'm curious, you've worked on this,

you should tell us what US government DoD policy is on this question, but isn't the reality that as soon as we find ourselves in a shooting war, say in the Western Pacific, that this will accelerate, that is to say the degree of autonomy will accelerate, and our good intentions about what seems ethical will be not the most important consideration. The most important consideration will be battlefield effectiveness as the other guy accelerates. I have a sense. You've thought more about this than me, that the PLA planning cells are probably less invested in the thought of what's good for humanity and probably more invested in the thought of what's going to work. But you tell me. No, I think that's right.

And I think that the operational pressures, you know, in peacetime, it's one thing. And that there's a lot of debate to me. If you look at how the Defense Department right now is talking about, for example, the China threat, there's a lot of talk about worries about the PLA modernization and Xi Jinping saying they need to be ready to invade Taiwan by 2027. In practice, if we look what the Defense Department's doing, it's not on a wartime footing. It's not remotely at the sort of urgency that you would want to have. If you actually thought that was true,

it might be only a few years away from a conflict with China. And that's across the board. So I think there is this huge difference between sort of the peacetime way, the sense of urgency versus in wartime. And there's no question that on the Chinese side, the sort of legal and ethical considerations don't get the same amount of play. I've had really the incredible opportunity to be in a lot of conversations with Chinese counterparts who work on military AI issues through track two dialogues between the U.S. and China and academic to academic

exchanges. They're illuminating to better understand how they're thinking about these issues. And, you know, they just, they don't, there's not as much of a focus on legal and ethical issues their way there is in kind of U.S. discussions. Now, they are worried about control, and they are very concerned about the unreliability of AI systems, and they're very worried about political control and ensuring that their leadership all the way to the top is very tight control over what the military is doing. It's just sort of, it's coming from a somewhat different perspective and that their objections are more about keeping control than they are about ethical issues. Yeah, common consideration of totalitarian or near totalitarian societies is your military

It's the ultimate heat-seeking missile coming back on you is losing control of your own weapons. Okay, so let's stick with battlefield applications for a minute. So we've talked about targeting. We've talked about, I guess in a way all we've talked about is targeting, but targeting in terms of data analysis and then targeting in terms of autonomous systems, what else on the battlefield is something we're thinking about with AI? Because they're obviously off-battlefield applications that are also very relevant. cyber, electronic warfare.

Talk us through what should be in front of mind. So I was in a conversation recently where someone said, you know, all that AI could do is like improve decision making, which to me seems like actually a good way to sum it up and seems incredibly valuable, right? So I think like one, maybe to go back to the industrial revolution analogy, the industrial revolution transformed physical aspects of warfare. AI is likely to do the same to cognitive aspects of warfare.

So data analysis, information processing, decision making. So if you think about really every stage of the targeting cycle, the actual missile on target is the simplest component of that in many ways. It's finding where is the enemy, queuing intelligence resources to gather information, processing that, getting that information to the right people at the right point in time. All of those things could be spent. better with AI, anywhere from using AI to process imagery or other forms of sensors more accurately

to make sure that the right information is going to the right people, to having communication networks. They're able to flex to demands at the time, right? So we're able to say, okay, we're getting increased demand here in the communications network. We need to maybe shift assets accordingly. we can automate some of that process and do some ways of reliable and good that could be helpful to things like as we're presenting information to people, being able to present that information

in a way that makes it easier of people to make high-quality decisions faster is all going to be, I think, incredibly valuable. Is it the case, and by the way, I should just confess as I asked this question, I'm about asking a question about how AI and cyber work together, which is a little bit like for me,

asking a question about how like, you know, multi, you know, variable calculus and Indian cuisine work together. Like, I'm just very out of my depth. But is there is there a way in which, you know, you think about cyber penetrations of systems, is there a way in which AI there, the way in which it helps is beyond decision making, it could make the weapon more effective just because presumably, if one is a cyber warrior of some sort, one is working in a world where there are things that need decryption and there just sort of walls to be burst through.

There's a lot of data and algorithms that work in this process. And something about AI just makes your weapon more effective. It makes your operational ability sharper. Is that fair? Yes. Yes and yes. So there's a lot of really interesting things that kind of the cyber AI nexus here. So one is that a lot of cyber activities, both on the offensive, defensive side, can be automated.

So certain types of attacks, what we know can be automated. This was demonstrated way back in 2016 in the DARPA Cyber Grand Challenge, as well as finding vulnerabilities and patching them at a defensive side can largely be automated, which can help shore up networks and help secure, you know, kind of unsecured devices. That doesn't even get into things like machine learning. So there's a couple ways that AI can be helpful on the cyber side. On the defensive side, one of the ideas is that you could use AI systems to train them on data for malware, to look for indications of malware or indications of suspicious activity operating within your network. So one of the things that AI systems can do is they can both, if you sort of have patterns of activity that you know are suspicious, whether it's cyber activity, it could be financial activity like wire fraud.

you can code that in, or if it's too complicated to write the rules down, but you have good data sets for it, you can train an algorithm on that data, and they can learn to identify patterns. Just like it's identifying patterns, you know, people's faces to do facial recognition. AI system can also identify just anomalous activity. So you can train algorithms to say, this is not normal. I don't know what this is, but it's not normal. And so AI tools are used for things like spam filters, for example. to detect, you know, spam emails or fishing attacks.

On the offensive side, one of the things that we're right now seeing programmers doing is using AI text generation tools to help accelerate programming. And so you can go into tools like chat GPT is kind of more general purpose, so it's not super great at this, although it can do this, but there are more special purpose kind of coding tools where you can tell it, hey, I need to write a script for XYZ, write it in this language for me, and it can write you a first cut. Or you can say, hey, I have this computer code, and it's not working, and I'm getting this error message. Can you help me debug this?

These tools are actually incredibly helpful to programmers. They would also be presumably helpful to cyber attackers, trying to write kind of malware and conduct cyber attacks. A lot of these tools are right now, not at the level where they're necessary. enabling entirely new forms of attacks. There are things like lowering the bar for the skill you need to carry out an attack or to maybe do something good as well as a computer program or accelerating the productivity

of people that are experts in this field. But I suspect that over time, we'll see the models get better and they will enable just new things. That humans, you're like, well, I never thought of that, but that's actually pretty clever and I discovered that. That's common too. Yeah.

So this is starting to make. move us away from the battlefield, but the way you just outlined that makes me want to ask about the ways in which AI sort of lowers the bar, lowers the barrier for entry into various kinds of activities, some of which are really dangerous and distressing in which AI could obviously play a kind of destabilizing role. So the manufacture of chemical or biological agents is something you've written about and something that's obviously of great concern. The manipulation of DNA I know is a great concern to very smart people who follow these things. And here we are, I guess we're into a subject matter where we're talking about,

I think for the most part up till now in our conversation, we've discussed the way in which AI is going to be used, probably and for the most part by states and militaries and large organizations. Here we're moving into a, I guess maybe cyber is the exception of that there at the end. Here we're moving into a space where actually its potential is to be destabilizing in a generator of violence. for very small, not very well-funded entities, like people, individuals, please.

Well, this intersection of AI and chemical and biological weapons is probably one of the more concerning areas, certainly one of the areas where there, if this really panned out in a dangerous way, the risk should be massive. We all just lived through a global pandemic. I think we've seen, and as global pandemics go, that was a very mild one compared to prior historical examples.

And so, you know, the potential scale of destruction, for example, from biological weapons is just really quite, quite massive and very troubling. Now, there's also, I should say, a huge amount of uncertainty here. So I will maybe lay out the case for why people are concerned, but I do want to kind of say that this is an area where there's a lot of debate. I sort of fall, I'll preview my view, in the camp of, well, let's not wait until some terrorist group or some apocalyptic cope has made some terrible biological weapon and then go, oops, too bad, let's err on the side of being cautious here. And I think some caution is warranted.

But, you know, there are a couple of concerns. One is lowering this barrier to entry. And the concern effectively is that these general purpose models, like ChatGPT or other large language models, can do a whole host of things. They can write a poem. They can create a screenplay. They can write a pitch for a podcast. They could help aid in conducting scientific experiments.

And they can also aid in the development of cancer. chemical and biological weapons. The people have demonstrated the ability with these models, the uncensored versions. If you go to it right now and you ask these models, like a censored version is going to say, I'm not going to help you carry out this terrorist attack. But the uncensored versions can help somewhat for a non-expert. They're not going to help an expert who already has some knowledge. But if you're a non-expert, not help.

I think it's very unclear whether they're more helpful than just what you can find out on Google right now. but the models are getting better because what they do in this is, I think, really important to understand this. They're not just capturing information. A lot of this information exists on the web. They're capturing it, and they're synthesizing it

into a model that then has knowledge. So it's more like talking to a person or a specialized assistant who has some knowledge, and then it can help you do things like debug scientific experiments. So you can say, I'm trying to conduct this experiment.

It's not working for me. Here's what's happening. Can you help me out? Now, the models are like okay at this right now, but they're going to get better. And so a world where anyone can access the equivalent of like a postdoc in biology or chemistry at their fingertips who has specialized knowledge like this, that's going to make some of these tools more accessible. That's one concern. Another concern is that more specialized biological tools just for biology could help aid in the development of even more capable biological weapons. those are going to be things that, and then only experts can use.

Your average person is not going to know how to use that. But those are going to probably expand the horizon of sort of the letality, the transmissibility of what might be possible. And then the scarier thing is the intersection of these two. It's someone who doesn't have the expertise using a general purpose model to help plan things and a general purpose model itself being capable enough to use more specialized biological design tools. to then develop more dangerous pathogens.

And we're not there yet, but it's a definite possibility when we look into the years ahead. I think it's something we need to take very seriously. I'm glad to hear you say that we're not there yet, both for the obvious reasons, but also because my only interactions with AI technology or large language models directly

has been attempts to use chat GPT as a research assistant. And inevitably it fails me. I'll have some specific question, like, you know, when did Joe in Lai make his famous joke to Henry Kissinger about, you know, how do you, how, what's your view of the French Revolution. It's too soon to say, like, tell me when that happened. When's the date and when's the transcript?

This actually happened. And it couldn't. It just kept coming back to me with various versions of gobbled to gook. I got into an argument with it. It was ugly. But as you point out, but if I go back a year from now, maybe it'll be able to, maybe it'll be able to do it. Well, okay, so say more, if you will, if you will,

about the countermeasures to these threats, whether regulatory, technological, like what, say more about what is, what might be done if we were inclined to do something. Well, there's a couple of areas of policy discussion that are very active conversations right now in Washington. And some of these were included in the recent executive order out of the White House on artificial intelligence. Some of them were sort of signaled in the executive order. So what involves red teaming these models, basically doing safety testing on them, for models above a certain threshold. The executive order sets a threshold, which was of computing power used to train the models,

which is sort of a bit of a crude proxy for things at the frontier of AI development. Some new, really big, very capable models that are being developed. And we don't have good safety standards yet, but it tasks the National Institute of Standards and Technology to develop a set of safety standards that then companies would have to evaluate their models against. So we could say things like, well, does this model help someone create biological weapons? We need to have a mechanism of actually testing that first. And then we can say, well, if it does by how much,

and then debate, is it safe to release? So I think that's one big area of conversation. Another one surrounds open source. A lot of the leading AI companies, Google, OpenAI, Anthropic, have moved towards restricting release of their models. You can interface with chat GPT, but you do it through this interface where the model is withheld by OpenAI. You're just sending queries to the model and you're getting responses.

You don't have access to the underlying model itself. Now, META has taken a different approach, and they're releasing their models open source. Now, the problem with this is once it's open source, anybody can fine-tune the model to get rid of these safeguards. So the model that META is releasing itself has been fine-tuned to put in these safeguards. So if you ask the model to do something bad, carry out a terrorist attack, it's not going to help you do that. But at very low cost and very easily people can get rid of those safeguards. And we see within a day of these models being released, there's an uncensored version online because it's the first people doing the internet.

They're like, well, let's get rid of these safeguards. I don't want to deal with this. And so that's another very active area of policy discussion of do we need to kind of have maybe more some stringent guardrails in place? And what's liability for these companies look like if these models are used for harm? Can I ask a really dumb question? And how transportable are the models? That is to say, okay, so a company has a model

and it can make the code available. How big is the thing? Can I have it on my laptop? Do I need specialized equipment? Like once this thing is out in the wild, can any Joe essentially transport it around? Yeah, I mean, this is actually a fabulous question because these technical details matter a ton.

If you look at something like nuclear weapons, what has made nuclear nonproliferation largely successful, not perfect, but large successful, is it's really hard to make a nuclear weapon. And even if you were to get a hold of one, you can't just copy it to get more nuclear weapons. So there's this big asymmetry with AI right now. We're training a large, very capable model is very computationally intensive. So it requires tens of thousands of these most advanced chips in the world, which only a couple people have access to that number of chips, There's only a handful of the big AI labs.

They need to be run for months at a time. It's very costly to do that. The companies don't release their cost numbers for these models, but the best independent estimates are that it costs tens of millions of dollars to train these very capable models. So it's really, from a sort of proliferation standpoint, there's only so many people that can do this. And the engineer requirements as well are also very challenging

to just make all this work. Now, once the model's been tried, trained, that's entirely different. The trained model itself is just a piece of software. And so folks may have heard about this term model weights. The model itself is basically represented by a series of numbers that are called these model weights that represent the weights of the connections in this neural network. Basically, it's a giant data set of numbers. If you have the model weights, you have the model. That is the thing. And if you know how to use it, then you can employee and you can adjust the model and fine tune it for your own purposes.

The train model does not need a lot of hardware. So the biggest, most advanced models can't necessarily run on like a run-of-the-mill laptop, but they certainly could run on hardware that's pretty accessible. And oftentimes, the model itself can be distilled into a smaller version that is not quite as capable, it's still pretty good, that can run on a laptop. So we see very quickly this proliferation trend where only a handful of people can train the model requires massive amounts of computing hardware

but then once it's trained, that trained model is a piece of software. It's online, it's open source, anyone can use it. And very quickly people distill that into even smaller versions that are still pretty good that those versions actually anyone can run on a laptop and then the cat's out of the bag in terms of controlling the technology.

Yeah. A little bit of a change in some. subject, but I want to get to this before we conclude, which is we've talked about nukes and we've talked about AI. Let's talk about AI and nukes. You wrote about this recently for War on the Rocks. How do these things work together and what should we be concerned about here? So what's fascinating here is the United States has a very clear policy of this that came out

in the 2020-22 nuclear posture review where the Defense Department said that there will always be a human in the loop for any decisions that are critical to informing or executing a decision by the president to use nuclear weapons. That's a very clear policy statement that doesn't exist in other aspects of the duty thinking about AI and autonomy. So they have notably not said that about, say, conventional weapons. They have not said there will always be a human in the loop. Sometimes military, like, officers will say that statement off the cuff.

They'll say in a press release, like, well, you know, a press conference, well, we'll always have a human in the loop. The official policy. That's just their opinion. And the policy is a little bit more lax. but not on nuclear weapons. And so I think that's certainly important. I think, you know, this is the most dangerous military technology,

the most critical military mission that we need to get right. We need an intense surety over nuclear weapons in the sense that you both never want, what has been characterized as always never dilemma. You never want them use when they're not authorized, but you always want them to be used if they are authorized to be used. And in fact, deterrence hinges upon your enemy knowing that if the president says we're going to use it, it's going to happen. And it's not quite as simple what the article that I wrote with my colleague Michael Depp in War on the Rocks kind of talked about was that it's not as simple as saying, well, there's just going to be no AI in nuclear operations.

When I talked about earlier here, things like AI image classifiers, help it to identify enemy objects and satellite imagery or drone vehicles. video fees. Well, that might be the kind of thing that you actually want AI being used for to help identify early warnings of some kind of attack. You know, could we use AI to process satellite imagery so that we know if North Korea is going to do a nuclear test, you're going to launch a missile? We want to know that. AI might help people to speed that up, get that information to our decision makers. AI might be able to help with early warning. But this is an area we want it to be very reliable. We don't want a false alarm. And we want to make sure that, you know, that there's always humans in whatever the critical components are.

And so one other things, open questions is, what are those critical components? Kind of what is the next step beyond that policy guidance? How do you put that into practice? Paul Shari, the Center for a New American Security, author most recently of Four Battlegrounds, Power in the Age of Artificial Intelligence, a book which I hope our conversation has now persuaded you. You really ought to check out. I really appreciate you making the time.

This is a fascinating conversation. I will sleep less soundly tonight. Well, I'm sorry for that, but hopefully some food for thought for you and listeners, and thanks for having me on. Really appreciate it. This is a nebulous media production. Find us wherever you get your podcasts.