Technology, Connected - The US-China Race for AI Superintelligence Has No Rules

Episode Date: July 22, 2026

The US and China are racing to build AI systems that could lead to superintelligence before any enforceable global rules exist to inspect them, limit them or verify what they can do.Mark and Jeremy Think On Paper about The Hard Question of AI, an essay by Planet CEO Will Marshall. Marshall argues that AI governance now needs the same kind of verification, trust and global protection as nuclear weapons.That means more than voluntary safety tests and China and America laying down their AGI swords. And since that is unlikely, it could mean monitoring chips, tracking data-centre construction and energy use, inspecting frontier AI labs, protecting whistleblowers and agreeing international red lines around recursive self-improvement.But the nuclear comparison has many challenges. Weapons-grade material is scarce and difficult to hide. Just ask Iran. AI software can spread quickly and cross borders. Any Tom, Dick or Harry could end up being the first to create superintelligence. At the centre of the episode is one question: what would AI’s Cuban Missile Crisis look like? What event would force the US and China to cooperate? Really cooperate. Lip service won’t calm existential risk. We also discuss Sam Altman, Dario Amodei and Demis Hassabis, the incentives behind AI safety warnings, the limits of voluntary regulation, the RAND verification framework, recursive self-improvement, the Fermi paradox and whether governments can build the institutions required to manage AI superintelligence.Hint: they can’t.Please enjoy the show. And don’t have nightmares.--Thinking on Paper is a technology podcast about AI, Space, quantum computing, science, and the systems shaping your life. 🏠 ⁠Buy us a beer on Substack⁠🫵 C⁠hoose your own technology adventure ⁠📺  ⁠Watch our beautiful faces on YouTube ⁠🎧 R⁠emember Steve Jobs on APPLE⁠📺 ⁠Get clips and exclusive videos on Instagram ⁠--Chapters00:00 Trailer 01:38 AI And Existential Risk02:40 OpenAI, Anthropic & Google04:40 Nuclear meltdown or sharks?05:50 Why is everyone speaking about the end of humanity08:00 RSI - Closed loop recursive self improvement10:00 A ghostbuster copyright interlude19:39 Why America And China Are In An AI Race23:51 The AI Cuban Missile Crisis28:07 The AI Supply Chain35:09 USA And China And The AI DeadEnd40:36 The Nuclear Framework

Within months, artificial intelligence may achieve what researchers call closed-loop recursive self-improvement, the capacity to rewrite its own code to become more capable without human intervention. If I had a company that made AI and I told everyone that AI was going to kill us all, that would be bad for business, wouldn't it? For people to take anything seriously, there's got to be some kind of near-miss event for people to go, holy shit, we had to do something about this. I think the Cuban Missile Crisis was an interesting near-miss. What does AI's near-miss look like?

What is AI's Cuban Missile Crisis? I dread to think. Disruptors and Curious Minds, welcome to Thinking on Paper. My name is Jeremy. This is Mark. We talk about technology. We talk about humans. We talk about systems.

You guys know who we are. We're on it again. This is a pocket edition. No guest today. Mark, you're the guest. I'm the guest. We're both the guests. guests. Here's what we're doing. Will Marshall wrote an essay called The Hard Question of AI,

an urgent ask to steward superintelligence. Subtitle why AI governing should top President Trump's agenda. And Gies. Oh, and Giz's agenda. President Trump and G's agenda. China and the U.S. We're in a race. Here we are. But we're not, oh, we're going to get into the essay, but he proposed. is that maybe we shouldn't be in a race with China, maybe the US in China should be in collaboration. Imagine that. So you read it, I read it, give me your first impressions,

give me where you want to dive into. There's tons, I think. It's my favorite subject, Jeremy, existential risk to humanity, artificial intelligence and the demise of the human species. Artificial intelligence, AGI super intelligence

and some other layers of intelligence in between and what happens if it all goes wrong and Will Marshall's essay is about possible solutions, possible frameworks, ideas, collaborations and experiments so that AI and humans are aligned

and we survive into the future to kill ourselves in other ways that don't involve AI. Yes, for sure. So, yeah, everything you just said. And we're going to talk about a little Fermy paradox rabbit hole down the end

that will make some sense into what Mark just said. So let's start with the big three. Open AI, Anthropic, Google Deep Mind. I'm going to read some quotes here just to kind of set the tone of where things are and why this essay is important and timely. The first one is from Sam Altman, OpenAI. I think the good case around.

AI is just so unbelievably good that you sound like a really crazy person to start talking about it. The bad case, and I think this is important to say, is the lights out for all of us. Wow. Boom, all right. I'm going right in a quote number two. Dario Amade, chief executive of Anthropic, who we've unpacked his essay Machines of Loving Grace, if you want to listen to that. Boop, boop, boop.

His quote, humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems can possess the maturity to wield it. Tech moving faster than our ability to deal with it. 100% agree there. Last one. Demis Asabas, chief executive Google DeepMind. The risk of a catastrophic scenario is not zero, so we must dedicate significant resources to mitigating it.

A little more of a tempered quote there, right? But the risk of catastrophic scenario, that pours right in, and we're just going right into this essay because it sets a nice stage, I think, as he's talking through this, as Will Marshall is talking through this. So the acceptable risk for nuclear reactors,

and there's a lot of nuclear parallels in this essay, the acceptable risk for nuclear parallels in this essay, The acceptable risk for nuclear reactor scenarios, right? The risk of catastrophic meltdown. So we're talking the Three Mile Island. We're talking Fukushima, Chernobyl, all of that stuff, right? One in one million. One in one million is kind of that acceptable risk.

Not acceptable risk, but that's kind of... What's the odds of me getting hit by lightning tomorrow? Is that more or less than that? Bit by a shark. Bitten by a shark must be longer odds than that. There must be more chance of a nuclear meltdown than me being bitten by a shark. He says. Well, look that up while I'm transitioning to this.

You transition. That's the acceptable risk for catastrophic meltdown in a nuclear power plant, according to Will's research and essay. The AI translation, I'm just going to read a quote from this from his essay. AI experts estimate the risk. of an AI-caused catastrophic event at 10 to 50%. Quite a big jump. Quite a big jump in that.

And according to what he says in this essay, the risk is coming directly from the people building it. So that's something we'll get into. Mark, do you have any numbers on the lightning or shark attacks? Some very rough numbers, but your lifetime odds of being killed by a shark are roughly one in 3.7 million. So three catastrophic nuclear meltdowns. Better than a meltdown. Okay. Can I ask you a question? So we have, we had Sam Altman,

we had Dario Modi, we had Demis Hasibis. Okay, three of the biggest, most wealthiest, important people in artificial intelligence. What we talk about incentives all the time. Why are they saying these things. If I had a company that made AI and I told everyone that AI was going to kill us all, that would be bad for business, wouldn't it? Well, if you remember this, this is a callback to our review of Karen Howe's book, Empire of AI. There was something in there that was really interesting that immediately jumps to mind

referring to Sam Altman and his advocacy for not just the good things about AI, but the potential harms of AI is like by coming out and saying this really could, be bad for people, governments have to kind of rely on the guy saying that, right? And if you rely on the guy saying that, then you get to be part of how that thing is regulated. So I think there's a little bit of that that sticks out. And I'm not an expert in economic policy, government policy, any of that. But enough to reference that Empire of AI, Karen Howe positioning that reflected Sam Altman's, I think, desire to be at the table when the governance talks were happening. AI experts estimate the risk of an AI caused catastrophic event at between 10 and 50%.

Well, we check out our episode on Prophecy, The Book, The Excellent Book by Carissa Villas, and why you should take those huge range of percentages with perhaps a pinch of salt. But let's go on because so we've laid, we've painted the picture. This is what's at stake. This is what the CEOs and founders think we should weigh up. They're not saying it's going to happen, but we should be thinking about it. And Will Marshall proposes a solution that. But he begins with another scary prospect, what he calls the.

Hinge moment, basically, we don't have much time. Within months, artificial intelligence may achieve what researchers call closed-loop recursive self-improvement, RSI, not repetitive strain injury, closed-loop recursive self-improvement, which should be C-R-S-I, shouldn't it, the acronym, not RSI, but anyway, the capacity to rewrite its own code to become more capable without human intervention. This is no longer a theoretical proposition.

Anthropic has already declares that roughly 90% of the code for its company's latest model was written by its predecessor, RSI. A system that can improve itself without human direction, Jeremy. What does that make you think, feel? Think and feel. A little how? Let's read. I will not open the bay doors. Yeah.

Right, right. So let's go back. At the end of the essay, there are a couple more quotes in here that are really interesting. Elon Musk. Oh, I think AI is one of the biggest threats to humans. Okay.

Mira Muradi. We remember Mira from Empire of AI, right? Former CTO of OpenAI thinking machines lab. So there are a lot of hard problems to figure out. How do you get the model to do the thing you want it to do? how do you make sure it's aligned with human intention and ultimately in service of humanity? There are also a ton of questions around societal impact and I think there are a lot of ethical and philosophical questions we need to consider. It's important we bring in different voices like

philosophers, social scientists, artists, people from humanities and Mark and Jeremy from David. Mira, wow, that's amazing. Really is a compliment. Thank you. Feel very special for that. She didn't say that just to clarify. Mark, you're busting the dream. Well, we've just been kicked off Spotify for breaking copyright rules, Jeremy. So we've got to, you know, maybe we should tell the truth. So didn't get, have we really? One of our episodes has been taken down, an old episode from over a year ago now, because there was an eight second snippet of Ghostbusters, the music from Ghostbusters in the background when we were talking about virtual reality and Spotify, a year later, have

flagged it and globally banned that episode for breaking copyright. Was that, how was that music played? Was it played in the context of a discussion around Ghostbusters or was it? It was so it was about the history of augmented reality and one of the first use cases of augmented reality were these backpacks on a, I think it was Stanford University, a guy at Stanford University, and they looked like the Ghostbusters. the backpacks. So I put a little bit of the Ghostbusters music behind that as we explain that story.

Not allowed. Yeah, but as we're talking about AI, and I'm just, I'm happy that Spotify have banned that episode globally because we did break the copyright law. And imagine if these big technology companies were just going around stealing people's work and music and stories without their permission. kind of word will we be in? So, you know, I think they're saying a precedent that thou shalt not steal. I'd like to, on behalf of thinking on paper, I'd like to publicly apologize to the Ray Parker Jr. family because I grew up with that song and, man, I danced around to that song and I

appreciate the value of it. So it wasn't right, so according to the cease and desist notice. Oh, man, this is going down a rabbit hole, but let's keep it up. It wasn't. Ray Parker. So they've said the artist was George Doherty. Does that mean anything to you? No. So he actually wrote the song. It might not even have been the real Ghostbuster song because I didn't want to put the real Ghostbuster song. So that I think that is actually George Doherty made his own version of the Ghostbusters song, actually. Okay. Well, hopefully he got permission to do it. Well, uh, the men.

Better irony goes down. So many layers. Right. Sorry. All right. So we're on this, we're on this hinge moment. And basically what Will Marshall is talking about here is,

the existential threats aren't fully defined or realized, but they're coming quickly. We can see them. We can see this, it's like seeing this cloud firing at you very quickly. And you don't know what the cloud's going to eventually be when it gets here, but it damn sure is moving fast. And even Alan Turing back in 1951, as Will writes,

here's a quote from Alan Turing. it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers. At some stage, therefore, we should have to expect the machines to take control. Okay? So acknowledging, yeah, he was acknowledging it in 51. I think here's the challenge though, man. As long as there's this race to the top,

it's hard to look at the repercussions of the end result of the race while the race is still going, because the race is powered by money, the race is powered by the drive to need and control a market, and there are a lot of things that kind of fall to the wayside while you're racing. Yes. Many things fall to the wayside. Ethics, responsibility, accountability, truth, for example. That long ago, people were already thinking of the worst case scenario. So, okay, it's always good to think of the worst case scenarios, because then you can work to avoid the worst case scenarios. Reverse engineer what you want by describing what you don't want.

It's interesting, though, that that long ago, obviously, the name at the beginning of computing said this, how many of his students took on that thought process? Like it kind of almost like this negativity, this AI is going to kill us all and take over the world, has been passed on from one great computing mind to the next. And now you have. I wouldn't call it a negativity, though, I don't think.

I would call it a, I mean, I don't, I didn't know Alan Turing, right, of course. But like reading about him, reading things he wrote, he doesn't seem like a very, like, I'm going to be unobjective about this thing. You know, it'd be very objective about, here's a thing, here's a capability, hey, we made some shit work. But, you know, you pour jet fuel on this stuff down the road. He's just looking and seeing, you know, the evidence of what could happen. I don't see it as a negative thing. It's a great responsibility of someone like that to say that early on. Yeah, it was the wrong choice of words.

But it reminded me of another of the godfathers of AI, Sir, Jeffrey Hinton and he's come out and he's doing the rounds because he's said similar things that if we don't get a hold of this, that the fallout could be existential for humanity. And it reminds me of Frankenstein a little bit. These people have created this monster. Obviously, Frankenstein was actually, and it was a good. It was society that made him, turned him. So there's echoes of that kind of Frankenstein where the people create it.

And then after they've created it, they start saying, oh, what if it all goes wrong? I mean, there was afterthought, not forethought. Well, yeah, I mean, it goes back to, we're really good at creating new technology. We're terrible at making sure it is for the good of our species versus the bad or not just the good of a small segment of our species, which I think is what we're probably going to witness in the coming years if some of these safeguards aren't put in place. Well, let's go, Utopia.

So anyone who's been to the new Thinking on Paper website will see that every episode features on the doom to utopia scale. We're all about the good and the bad. We want to both sides of the story. So there is no question, there is no doubt, that today's large language models are, well, let's quote, generating substantial economic and scientific value, compressing research timelines, expanding access to expertise,

enabling new forms of environmental monitoring, the reliability and security challenges at this level are significant. And new forms of environmental monitoring. So there's a lot of positive, there's a lot of good. How do we make sure that we continue along that path and not the other path. Well, what happens when we get to AGI and then people like loosely understand what that is?

And then this article references ASI. And I was the first time I actually heard ASI as a term that was written here, artificial superintelligence, which is quote, I'm quoting Will Marshall here, systems whose capabilities exceed human performance by margins that are difficult to conceptualize. Shit getting so good that we don't even know what it can do with how good it is.

Like we can't even explain what that might look like because we haven't seen it. We can't imagine it. Like we can't, like in our minds imagine it even. It's like the AI equivalent of what's on the edge of the universe. Like what's beyond the universe? You can't conceptualize. You can't imagine that.

Like what is beyond super AGI is this intelligence, which we can't. even conceptualize in our heads. This is such a weird comparison, but I saw this little video today that kind of made me chuckle a little bit. So club sports, meaning like travel sports for youth,

like not just the recreational level. These are the folks that are a little better. You put them on teams. You take them all over the country and stuff like that. And there was this one silly video with like a parent to parent. And it's like, hey, you know, Jimmy, you should probably join this,

travel team and they're like, nah, and like, well, you should join this cobra elite superstar travel team. And then like, nah. And then like, heard one, you should, you should join this global superstar, superhuman intergalactic elite squad. And they're like, yeah, that sounds awesome, right? So here's my, why do I say that? And why is it? So I keep thinking like, here's AGI and here's ASI. And then what's after ASI? Like, we can't even like think and imagine, but, but it's coming and it's always coming and something's always coming. But, you know, what's what the author of this essay talks about doing is advocating for, hey, let's, let's freaking grab it by the shirt and sit it down in a chair and make it behave at least a little bit. Which, which sounds really silly,

but you get where I'm going. There's a section called the competitive trap, which is really interesting. It falls back into the race dynamic. This, me having to outdo you and you having to outdo me and I'm getting more clusters and more network and more power and more utility grid. And hey, I built my own power station and, hey, my network is this and just rise to the top. All the meanwhile, like, capital is just flooding into this, you know, so it's fueling. And so no one's going to stop because the money's rolling in. The money's rolling in. I'm going to build it.

I'm going to do and all of that. Here's what's interesting. Well, Haseb is, you know, back to Google DeepMind, the more AI becomes a race, the harder it is to keep the powerful technology from becoming unsafe. Absolutely. 100% agree with that. But how do we go, yo. True though, is that true though? Because, okay, let's just use my simple non-ASA mind for a comparison. Let's talk about race cars, Formula One cars. The race to get faster Formula One cars made the cars safer.

Why does the race make it become more difficult to keep things safe? This race isn't doing all left turns. This race is going all directions all the time. Well, let's think about another technology. Okay, airplanes, that would be the the same thing, so it's similar to cars. We can't use that. What about... Safety's on guns. Safety's on guns. Well, you're an American, you're an expert.

You tell me about that. Oh, man, low blow. Low blow. I don't have a gun. I actually don't have one. I don't have a Second Amendment right stick around my car, any of that stuff. I think that question needs a caveat, because I don't think that the more something becomes a race, the harder it is to keep it safe.

I think it depends who's in the race. Well, this is tricky because you have countries. racing countries, you have companies racing companies, you have providers providing things to the companies in the countries that are racing. It's complicated, man. This is the most complex of systems. As we'll alludes to, if we could take something from our history of human, our collective history, something which was and has the potential to extinguish

us all. And if we could take that, there must be something, we must have created something in the past that we could use as a model for this. Is there anything you talking about this? Maybe even talked about this on thinking on paper at some point years ago, didn't we? What was it like nuclear weapons? Yes. The nuclear model. Well, if you're listening, Jeremy brought this. Not if you're listening. They're listening if they're listening, right? Two years ago. So maybe you and you should come on and think on paper, we can discuss this. We can kick the can around the nuclear field. Yes.

No, it is a very interesting model. We've talked about it before. We've also talked about the Martin's Clause, which we did a whole episode on the Martin's Clause. If you don't know about that, boop, boop, wherever the pop-up is, you can go listen to that too. The nuclear model is interesting, right?

Because the technology was moving fast. The technology has or had and has the ability to basically extinguish the whole of the globe. So at some point there had to be a way, this is starting to get out of control. How do we get this going? And there were a couple things that happened. One, a race, right? So the U.S. was out in the lead.

Then the Soviet Union kind of came back in pretty quickly. And then the Cold War started and there was this back and forth. So when you have that kind of thing going and, you know, you're the Soviet Union. I'm the U.S. I'm building my stuff. you're building your stuff, I have to kind of trust you that you're not going to freak out and push a button. You're going to have to trust me to not freak out and push a button. You're also going to have to trust me when I tell you I'm not enriching any more uranium

based on beyond these quantities. You're going to have to believe and trust that. But what the essay talks about is trust, but verify, right? So we want to go into verification, which I think is really interesting as he lays it out, because there are systems already in place that could be potentially used. But one question I have for you, Mark, is there has to be, I think for people to take anything seriously, there's got to be some kind of near-miss event

for people to go, holy shit, we got to do something about this. I think the Cuban Missile crisis was an interesting near-miss. What is AI's near-miss look like? What is AI's Cuban Miss? crisis. I dread to think. I like the parallel in the essay. So, okay, there wasn't much nuclear control before the Cuban missile crisis escalated. And after that, they started to think about

the restraint and the controls that were necessary to stop the proliferation and to stop this. What is the AI Cuban missile crisis near miss? What is that? It could be anything. So let's let our imaginations go wild. So perhaps the next iteration of Fable takes it upon itself to break into the North Korean computer system, find the codes and North Korea kind of shoot themselves in the foot or North Korea shoot South Korea, somebody gets involved and it could escalate by that or the same could happen with China and Taiwan. There's different scenarios that could play out.

I know. What do you think? What's the AI-Q missile crisis? Let's point to a couple of categories. I thought were really interesting that we'll put in his essay. Three categories of concern, biological weapons design, enabling mass casualties. That's a quote from the line here. Quote, large-scale cyber attacks on critical infrastructure.

So kind of stuff you're talking about, right, breaking in, shutting down power plants, making things go off that shouldn't go off. The third is really interesting that I hadn't thought about until I read this is the entrenchment of political control at a scale previously impossible. You could take over countries with a keystroke. Yeah, that could happen, probably next election cycle. Maybe he's referring to a more, well, some kind of artificial intelligence taking on that on itself. and if it's iterating on the process and maybe it wants to rig an election, it could do that bizarrely via Facebook. But listeners, tell us in the comments, wherever you are listening to this, Spotify, Apple, YouTube, wherever you are listening,

what could be the AI Cuban Missile Crisis equivalent? What could be the event that pushes humanity to the brink and makes it come together to realize we've got to work together to harness this? What is the AI Cuban missile crisis equivalent? So back to the nuclear analogy, comparing it, comparing it to AI, it's a little more difficult because monitoring nuclear is kind of easier than monitoring AI in a way. And we can go down into this because for nuclear, you need enrichment facilities, weapons grade, material, and you can't really do that under the radar. There are ways to kind of track and watch that kind of thing, even from space, as alluded into here.

They're seeing it play out in real time how difficult it is. The Iranians are showing us how difficult it is to actually make something destructive in this way. So, yeah, it's not readily available to everybody. Whereas, well, as Will points out, well, you tell us, Jeremy, like, AI. kind of follows some of those rules, but not totally. Yeah, it's a little more difficult to track. And this is where the essay turns from trust to verification. I think he uses the language verification, which is really cool and interesting.

It's a methodical trust, a methodical trust. So there are a couple of things that make it a little bit more difficult to monitor. But he breaks down, think about this, the supply chain. This is a really interesting quote. Talking about all different things that go into creating AI and defining what those things are and creating mechanisms to track those things. Quote, from the essay, just three companies control over 90% of advanced AI chip design overwhelmingly in California with the vast majority of the minerals coming from China,

tools from ASML in the Netherlands, and manufacturing from TSM in Taiwan. Now, there are other groups doing things in that world, but that really kind of explains super simply, and there's way more depth to the entire system, but the supply chain, you know, is that, yeah, that's the thing that you got a, you got a kind of wrangle. So obviously, with all of this, you need huge amounts of electricity as well, which is, we bring in Dennis Hasabas, who calls for the equivalent of the international atomic energy agency, which has had the inspectors for decades, maybe there could be something with AI that monitors either the supply chain or the electricity usage. Yeah, and the argument here,

too, is if the IA International Atomic Energy Agency can inspect military properties, can get involved in state military stuff to figure out if people are doing the right thing, what's stopping us from doing that related to intellectual property of a company, right? If you can do it from a military standpoint, you tend to argue, or I think that's this where Will is kind of tending to argue that it's easier to do from a company perspective. Quote, inspecting a frontier AI lab involves company intellectual property rather than state military secrets,

a political economy of compliance that is considerably more tractable than the inspection of military installation. You spoke about trust. I think it's easier to get everybody on board when the downside is so obvious. The nuclear bomb goes boom, the nuclear bomb doesn't go boom. It's binary.

AI, when there are so many scenarios and we don't know any of them and we can't imagine others, it's a lot, that's a different political pot potato to wrangle. You can go to any country in the world. That's why it worked because this is what happens if we don't get this right. Whereas the AI, this is what happens if we don't get it right.

It's much more abstract. It's much more imaginative. We don't know the answer so readily. So it's politically much more harder to get people cross-party agreement, let alone international agreement. It's really interesting that. you know, the CEO of a company called Planet that's actually helping to create a real-time global state of the rock we're standing on,

trying to push forward an agenda of we have to think globally related to this. So he references some advances, some things that are happening in verification that are pretty interesting. Some of them are quite simple, and we've heard about them. Other ones are a little bit more interesting

than I want to dive into further. export controls. We've heard a lot about this in the news, right? U.S. and Netherlands, having controls on the advanced AI chips. The other ones, I'm jumping across this list. The UK has got some stuff rolling. Inspection is actually happening. The UK-UK-SIS, UK's AI Safety Institute. So these are voluntary evaluations of frontier models of which, according to Will here, Google and Anthropic have voluntarily seen. admitted to that. So not sure what that means or how aggressive those peaks under the curtain are. But hey, congrats to the UK. Don't mind if I don't get too excited about voluntary

evaluations of frontier models coming from the UK AI Safety Institute. I would like to see the results before I get too excited about that. That was number four. You've got number two verification architectures. The Rand Corporation, which helped to develop nuclear arms control, has proposed a six-layer verification architecture for AI constraints, monitoring devices embedded in AI training chips, software inspection protocols capable of detecting capability concealment behaviours, I like the sound of that, systematic tracking of data centre construction and energy consumption, which we alluded to earlier, and whistleblower programs with legal protections, hardware export controls and formal international inspection regimes.

That sounds very much like the IAEA. Let's stay with that one because I think to me that's the most interesting because it's the most tangible. It's the most. Rand has done this before with nuclear. So they're creating that frameworks. The verification falls into a few different layers, layer one being on chip,

kind of meaning the chip watches itself. Layers 2 and 3 is off-churchase. chip, right? So external devices are watching the chips, you know, network taps, analog sensors, that sort of thing. And layers four through six are personnel based. Basically, people watch the people, which is humans watching humans that are watching these systems, right? So whistleblower programs, interviews, national intelligence activities. All of these chips will have this little thing on them that allows a third party to get into the chip and figure out, what the chip's doing, again, who is, who decides on who's going to be the peeker in to the

information on the chip? And it goes back to humans being the biggest challenge and problem in a lot of this stuff. Who decides? Well, what if AI decides that nobody's going to have a peak? So the international dialogues on AI safety, the IDAIS, involving senior researchers from the United States, Europe and China, has produced a set of consensus red lines, do do for such a regime. Prohibitions on systems capable of recursive self-improvement without human oversight, on systems that can autonomously replicate and acquire resources,

and on AI-enabled assistance in the development of biological, chemical or nuclear weapons. So I think there's four ways that we could get a handle on this. And as Will says, yes, it's hard, but it hasn't been attempted yet, and nothing good ever comes easy. Got to do something. Let's do it. Come on, countries of the world, unite.

Yeah, and that's exactly what he's advocating. He actually puts it in here. Like, a lot of people just pontificate about, you know, hey, this is what's happening and this is the world, and this is where it's going. But this is like, no, here's what you should do, step one. step one, U.S., China, put an agreement in place that will focus on some of the core principles that are of major concern with AI safety. Easier said than done, a couple of egos involved in that conversation on a one-to-one ratio and then also the race agenda that is that is powering that and has powered the relationship between those two countries for such a long time.

Does the Cuban Missile Crisis, does the AI Cuban Missile Crisis equivalent lead to this sentence here? The first priority should be an agreement between the United States and China. Okay. Yes, you can get some surface level political agreement, but you could push to that. But you sure as shit aren't getting a level of trust and a level of agreement and a level of what is, necessary right now between China and America. And I don't care what you say. I've been watching global politics for too long to think that that's going to happen. So do we need that Cuban missile crisis to make this happen? Yeah. The way I think about it, we do a great job of coming

together when shit hits the fan. Yeah. Like in a really special way. But the strength of those connections, they're a little bit fleeting. We come together. There's something that's unfair. Someone was done wrong. We come together. We protest. We share our voices. We advocate within organizations for the right thing. But then that quickly, that connection quickly kind of dissipates and we're kind of back in the swing of, hey, I've got to do my thing. I've got to keep my head down to, to, to, to handle my business. This might be a, this might be one of those problems that creates sustainability of that connection.

And here's, here's the issue. And I'm all over the place on this mark, but trying to, process it, process it in real time. There's also, we talk about the disconnect between the use of the technology from an individual human perspective. Like I'm, I'm on my phone using Claude and trying to, I know, book of a have it book a vacation for me or you know whatever you know the simplest way to use that is and there's just very little understanding of like what all the behind the scene stuff

after you push enter where that kind of query goes back and in the clusters are talking and things are coming back like the understanding of the capability behind hitting that button is being masked by the convenience of how that button is super helpful. The separation between the tools that we use to make our lives easier, better, faster, more dumb in some ways. We don't have the visibility into the extensions of that. You're going to turn the dial back to the good side. But before we get there, Jamie, I want to ask you a question. very high level, let's summarize.

AI is good and bad. The next iterations of AI, AGI, ASI could be good, but there is a small percent that they could lead to existential risk. So here in this essay that's a small percent, 10 to 15 percent. That's a big percent. Yeah, I don't buy into those percentages. I'm an optimist. But, okay, so there's a percent chance that it leads to the end.

So Will Marshall here is proposing something similar to the way that we safeguard humanity and the earth from nuclear weapons. And he uses reference to the Cuban missile crisis. We go to war over that. We are at war right now because of that. the Cuban missile crisis almost led to a world war at some point we're probably going to go to war with North Korea because of that if you draw the similarities there

what happens when somebody doesn't buy in to this framework what the AI equivalent of Iran the AI equivalent of North Korea what happens then do we go to war if we're using this framework that's the natural, that's how we defend it. And who do you go to war with?

Whoever the rogue state is, whoever the rogue player is. It could be not even nation states. It could be two people. It could be one person, right? That'd be a very short war. No, I mean, like, just the ability to, not everyone can spin up nuclear capability, people can spin up AI capability.

And that's going to be, that turns the whack into like an infinite many worlds whackamole. How do you manage that? Get somebody with more, a little bit more than thinking on paper to solve it. Green shoots, I like this. Rather than focusing on what could go wrong, we pour money into what could go right. We should actively invest into AI benefits, especially into areas where AI can have disproportionate benefits. I like that disproportionate benefits.

This might include planetary intelligence. Oh, I don't know where you got that one for. Planetary Security, adaptive collective intelligence, enabling organizations across corporations to countries to become vastly more efficient. I like this, but when I first read it, I was just like, geez, okay, here we go. It's really a, it's an advert for planet. Invest in planet. I mean, but, you know, like, it also goes, back to this global vision, right? Having a real-time global vision. And that's what this stuff kind of does.

And advocating for us to think more like that is interesting. I think even if you overly invest in the AI benefits, the other ones aren't going to go away and they're going to be just as scary. Yes, you're going to have AI doing good things, but you also have this looming, unaddressed thing that keeps getting swept under the rug and before you know it, it's freaking super dangerous and no one's thought about how to deal with it. And then we're, yeah, how does the world, how does the world, if there's no coordination between nation states, organizations, companies around this, when shit really

hits the fan, how does it all, we all just like get in a fucking conference room and just be like, Yo. All right. All right, Jimmy. What do you think we should do? We're not going to have that much time to react, I think, is the big thing. It's going to be like, oh, shit, there it goes, and it's done. It all goes back to me to one of the words of thinking on paper we have trust. But I think this is all about incentives. A lot of AI is about incentives. And if you talk about the difficult problems that AI can help us with conflict, food security, water security, climate change, displacement, economic inequality. They are the hardest challenges that we have. It's really fucking easy to get AI to make your spreadsheet look better. There's a lot of money in that because there's a lot of spreadsheets. It's really easy. It's really quick.

The incentives are there. and the incentives to do something about this fucking weather heat bombs like that the incentives aren't there because the incentives are long term even though those incentives are to survive. I'm talking about global warming with the fucking weather heat bomb? Is that what is that what you meant by that? I believe it's called climate chain global warming. It's not the 90s anymore, Jamie. We actually talked about it.

There's a hole in the ozone layer. Oh my God, dude. No, you talk a lot about incentives, Mark. You do. And I think a lot of action gets taken once incentives are put in place. And if you have incentives, you incent the behavior that you want to have happen. This goes from like simple as coaching, right? Coaching a team.

You put incentives in place to have the behavior happen the way you want to have happening. In organizations, you use the same thing, bonus structures, this and that, KPIs, all of that. But this becomes global KPI. This becomes like- Global AI KPI. This is global AI KPI. This becomes like humanity KPI's. This becomes like bigger, the highest of architectures

that you could have in human systems. This is where this is the root folder where this shit should live in, right? Like who's making room for the root folder? And who's standing up and saying, like, hey, you know, all of this stuff, all of this stuff has, has significant repercussions, though we can't immediately identify those yet, but they are. They're there. They're documented researchers that science shown it, you know, researchers show it. Alan Turing said it in 51 for

Pete's sake. Who's going to advocate for it? Luckily, there are some very, very incredible people. and we speak to them a lot on thinking on paper, and they are drawn like a moth to the flame, to the hard shit, to getting the hard stuff done. That's what we spoke to somebody, we're going to go mine asteroids. Why?

It's really hard, but I want to do it, we're going to do it. Like people have been drawn to these almost impossible tasks. And that, and I always talk about incentives, but I always come back to the human spirit, human endeavor, the human resilience to get hard stuff done and there are people out there

doing it and good look to them. And we want to talk to more of them. Yeah. Talk to more of it. All right, let's land the plane on Cuba. No, on the Furby paradox. Let's land the plane on how Will Marshall lands the plane in this essay

about the Fermi paradox. Do on then. Oh, you want me to go? What's the Fermi paradox, Jeremy? Well, we, we, we talk, let me set the stage a little bit by throughout this whole episode and a lot of the threads on thinking on paper,

the forces in the currents that we talk about on thinking on paper, talk about tech moving faster than our ability to wrangle it, right? Rangle it is a loose word. It means govern it. incentivize the right behavior within it, all of that stuff. So the Fermi paradox talks about there's infinite amount of planets with similar, you know,

structure to our planet, infinite stars, infinite suns, all across the universe. Why haven't we seen anybody yet? And it's really interesting that what he, oh, I'm going to, let's just read the quote, make it easy. Quote, one disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it, destroying itself or returning permanently to something like the Iron Age. Maybe that's why we haven't seen other species out there because they've gotten to an inflection point where they've created a technology so freaking powerful

that they don't know what to do with it and it totally implodes on them. Yeah. Wow. So that's that's the that's the doom side. Let's swing it to Utopia before we wrap it up, Mark. Well, I just, the phomy paradoxes of space is really, really, really, really, really, really, really, really, really, really big. And you can't go fast in the speed of light and you can't travel across the cosmos. And that's why we haven't seen them because they just haven't got here yet. Well, I'm going to challenge you on that, Mark. According to this research that we've done, you can travel faster than the information can travel faster in the speed of light in quantum systems.

No, but no, but it's not intelligible information. Yeah, it's not information, is it? It's not. And unless the aliens are actually made of. Then they. Oh my gosh. Rabbit hole achieved. There you have it.

Will Marshall, thanks for writing this piece. Thanks for getting us thinking about it. If you want to read the piece itself, the hard question of AI, an urgent ask to steward superintelligence, Will Marshall, CEO of Planet, who will hopefully be joining us on Thinking on Paper to talk about pelicans. www. www. thinking on paper.

x, y, z. For the convergence, the technology convergence, all our shows on quantum AI, robotics, bioengineering, biotech, society, music, love, culture, movies, cinema, and the umbrella of technology. They're all there. Go pull a thread and listen to the episodes that you want to listen to. Follow your curiosity. Like, subscribe and leave a comment. And tell us, remember, what is the AI Cuban Missile Crisis equivalent? Mark is being super humble. His new version of Thinking on Paper.xyZ is a choose-your-own-adventure that allows you to balance between categories of technology. Explore the threads between them. Do exactly what we're doing on a daily basis on this show. but you can pull the threads. You can see the map.

You can see the knowledge graph that he's built. It's the most interactive website I've seen on podcasting platforms. Dude, check it out. Do yourself a favor. Thinking on paper.xyZ. Until then, stay curious. Be disruptive. Keep thinking on paper.