TBPN - Washington Hones in on AI Safety, What Are AI Doomers Proposing | Diet TBPN

Episode Date: September 11, 2026

Diet TBPN delivers the best of today’s TBPN episode in 30 minutes. TBPN is a live tech talk show hosted by John Coogan and Jordi Hays, streaming weekdays 11–2 PT on X and YouTube, with each episode posted to podcast platforms right after.Described by The New York Times as “Silicon Valley’s newest obsession,” the show has recently featured Mark Zuckerberg, Sam Altman, Mark Cuban, and Satya Nadella.TBPN is made possible by:Ramp - https://ramp.comPublic - https://public.comCisco - https://www.cisco.comConsole - https://www.console.comCrowdStrike - https://www.crowdstrike.comFigma - https://www.figma.comMongoDB - https://www.mongodb.comNYSE - https://www.nyse.comRailway - https://railway.comShopify - https://www.shopify.com/Follow TBPN: https://TBPN.comhttps://x.com/tbpnhttps://open.spotify.com/show/2L6WMqY3GUPCGBD0dX6p00?si=674252d53acf4231https://podcasts.apple.com/us/podcast/technology-brothers/id1772360235https://www.youtube.com/@TBPNLive

What are the AI doomers actually proposing? That's the question I was trying to answer this morning. For a while. 20-year sentences. That's one. AI 2027 predicted that this month, you know, late in 2026, Congress would wake up and that is in the journal. Congress is suddenly waking up to the AI doomsday threat. And so this is happening all over the place.

Was it Matt Damon who was caught on TMZ being invested in terror? about his thoughts on AI risk. There's now protests. I believe AI 2027 predicts a 10,000-person anti-AI protest by the end of the year. I was trying to figure out how big this protest in Chapel Hill, North Carolina, earlier this month was. I think it came in sub 10K, but certainly tracking to it.

It was about 150 people at this protest in Chapel Hill. test in Chapel Hill, but just two ooms away from the prediction from AI 2027. So I wanted to dig into like what the actual proposal is because the joke for a while has been just like everyone when they're pressed on this, they just say, we gotta talk about it. We got to talk about this. And the actual proposal is super concrete in AI 2040 and it's very interesting just to hear about how they want to slow things down. So the thing that I think a lot of people online who are like, yay, an anti-AIIs, sentiment that's going viral are going to be depressed about is that like this is not

stop AI at all AI 2040 is like keep the inference flow in the current models are great also let's keep doing capabilities research but we're just we just want to reach super intelligence by 2040 instead of 2028 where we're not necessarily prepared so they want it to be highly controlled by governments nation states highly secured and there's a whole bunch of very very tactical recommendations that they make around that. So the first mechanism is an AI pause. They want to pause training.

They don't want to do any more new frontier training runs or R&D experiments. And to enforce this, they're calling for to apply inference-only verification to essentially all major AI data centers. So anyone who has more than 10,000 H-100 equivalence, roughly $100 million of equipment, and that is like pretty easy to figure out, You're just like big building over there.

Let's send the inspector inside. Oh, says Nvidia on all these chips. Count them up. There's over 10,000 of them. You got to apply for this permit. You got to tell us what you're doing, right? Very easy to enforce, at least in the United States. And with an international body, you could kind of do the same thing internationally.

So the whole goal, you can only inference the current models and you've got to verify your workloads with an independent auditor, probably the government. Maybe there's some sort of non-governmental organization that's doing. doing this, you know, there's a whole bunch of different solutions that you can pull from across nuclear nonproliferation work that's happened in the past. Major countries, they want them to declare AI compute inventories. Tell everyone not just your local population, but also the international community, how many how many warheads you got, how many H100 equivalents do you have? Where are they?

Everyone shares this. That's going to be a really tough sell because international agreements are really, really tough cells. It's much easier to have a groundswell of support for something that happens in America. America changes. We have a system that we don't really have the international rules to quickly implement that in a way that doesn't allow for a lot of defect. They want to basically remove high bandwidth east-west networking inside of data centers so you can't do large distributed training runs, but you can still do inference. For any new AI R&D data centers, they want entirely new facilities built from scratch with nation-state level physical security and ver and verificate.

But for the next run and the one after that and the one after that as it gets crazier, we want it in a new building built inside a Faraday cage so you can't communicate it from the outside. We want a bunch of physical controls. It's like going to a nuclear facility, highly verified who gets in the building and when air gap communications. This is one interesting like proposal that they have that really shows. how deep they thought this through. They want the R&D data center to be connected externally. If you want to communicate with it and you want to tell it what to do, okay, train the next running or do whatever, they will have a bandwidth capped connection at one meg per second. So you

can send little instructions, but if you say, send me the weights because I'm taking them somewhere else, it would take you like five years to exfiltrate it. So interesting like hardware solution to this. I, you know, how do you actually go and implement that? There's going to be a whole bunch of other things. But interesting that they're thinking about like the width of the pipe. So it'd be very obvious if you're stealing the model weights because it's like, wait, this one meg pipe has been at full tilt for months.

What's going on here? Someone's taken the stuff out of the data center. They want if front, when frontier model weights move from an R&D facility to an inference facility, they want it to be placed on physical storage devices encrypted independently by both the U.S. and China, so both countries have to sign off and physically escorted by representatives of both countries to the destination. It's a tall order. One's a tall order for sure. Yeah. And they actually want frontier models to be made deliberately larger than compute optimal. So they want the weights to be 100 terabytes instead of honing them down to something that's just one terabyte that could

actually be moved around a little bit easier. Yeah, I mean, first of all, this, I mean, a lot of this tracks with the kind of regulation that, you know, we had pushed for, you know, beginning about two years ago around podcast, wanting podcast studios to be air-gapped. Yes. Wanting, you know, Faraday cages around podcast studios. A locked briefcase with an SMB 7, SM7B in it. And in order to unlock it, Patrick O'Shaughnessy and David Senra both need to give you codes to independently verify that this podcast is worthy of being recorded.

Yeah. I like that one. Yeah. It just makes sense. High level valve they see as being like the most effective in controlling the speed of capability improvements is compute caps. So there is a world and we're going to the Bernie Sanders thing because it's already getting like sort of twisted. But the big hammer is just chip controls and data center build out slow down. That's the easiest thing.

And I think that's like the biggest valve that they're going to be. We're going to see twisted around to actually slow down capabilities. And so the goal is to allow models to get better mainly by adding hardware rather than inventing better algorithms, which can leak to secret projects. So the goal is like, okay, well, we know that this model is capable of this, so we want this much compute over here. Okay, you've done well, we're allocating more compute as opposed to this one weird trick that AI DOOMers hate.

So the goal here is not to go back in time. When you look at, I mean, anytime you have, you know, really, really hardcore government regulation and international coordination around issues like this, you're going to have a bunch of unintended consequences. And one thing that feels obvious around this if these policies were to be rolled out is that you would effectively create an incentive for millions of individuals or groups globally to be in secret, like trying to find intent. entirely new breakthroughs that are. Yeah. And again, this incentive already exists, but it kind of pushes a lot of the idea that humans are just going to be like, oh, I'm not, I'm no longer going to try to create the God model because like there's this, you know, big global organization that's, that's

sort of policing it. Yeah. I just. I mean, that's the same thing we see with nuclear non-proiferation. There's always a discussion about what countries are getting the bomb and when, and how far long are they and wars break out over this and uh it yeah like the game's not over just because you create a framework but but ultimately uh GPUs and computers are much more wide you know infinitely more widespread than nuclear materials yeah but you still got to marshal them all together yes

there's some weird scenario where there's a python script that's a GI that can run on your laptop, but I think most people are convinced, at least in this crowd, that scale is a prerequisite. And I mean, we were joking about like how it would be so like we, SSI, Ilius, let's Giverr's new NeoLab is recently got a big cluster from Nvidia. And we were like, the most bullish thing you could do if you're this secretive Neo Lab would be like, we're actually selling our compute because we've discovered a more compute optimal way to reach AI and we don't need a lot of compute. But of course, even Ilya is like, it's time to scale up.

I need more compute because it seems like even if he's taking a completely orthogonal approach to innovation and research, he still needs a lot of compute. And so it does feel like everyone is sort of with the consensus that it's going to be a big building with a lot of energy, big heat signature, definitely visible from space and pretty simple to track, at least in the short term, until people start building the crazy. underground facilities and then you're back to, you know, nuclear nonproliferation. But their goal is at least like, you know, try, try. Yeah. And then the other side of this is not a panacea.

Does it does AI development actually become something closer to the Manhattan project where, you know, that's what a lot of people are. A hundred, a hundred researchers are working with the government in secret because you can't just assume that other countries are going to slow down or do any of these things. Yeah. Yeah. But in general, I think the proposal is don't go back in time. It's definitely not stop everything in its tracks. It's a slowdown with the goal of scaling gradually.

They do actually want to reach super intelligence. They just want to do it by 2040, hence the name of the project. So the goal is gradually scale into top human expert capability around 2035. Now a lot of people are saying, oh, we might get this by 2020. 29, 2028, 2027, and they see that as too fast. So they want to push that out to 2035, then wait five years with AGI and then unlock super intelligence in 2040. This is their initial proposal.

Of course, there's a lot that could change over the next decade. And I think, I think like to zoom out overall, if you're worried about X-risk, the AI 2040 plan does feel like a concrete path towards slowing down. The conversation definitely gets dragged down into P-Doom estimates and trying to narrow down exactly how a human extinction scenario plays out. And that can be, I feel like that's almost a side show. Because in a democratic society, amongst humanity, like, it doesn't really matter the mechanics of getting to 10% P. Doom or any of those. She's just like, if everyone feels that way, something will happen. This is a concrete plan of what that might look like. And that's valuable to understand in this case. So for the safety skeptics, it's easy.

easy to see how this level of control over what you can do with computers is authoritarian or anti-libertarian, even if we're talking about $100 million computers, there's a lot of people that say, like, I should be able to do math on my computer, I can do whatever I want, let me do cool things, I'm excited about this, that limits your freedom. It might create regulatory capture for a few major players. It might crash the stock market or delay economic gains that come in the good ending where alignment is solved and X-risk plummets. You can imagine a situation. where in a few years, if X-risk fades into the background, you're like, yes, there's still a risk, but it's the same risk that we face every day with like an asteroid hitting Earth. Like no one, it doesn't really change anyone's behavior.

That would be sort of the good ending, in my opinion. So it's a balancing act, and for most of these slowdown proposals, I personally have a hard time blackpilling about them in the sense of like, if all of this gets implemented, how frustrated will I be? Like, the models are good. I would like better models. I want safe models.

But at the same time, like, there is this massive capability overhang. The current models can do a lot of interesting work. We're finding new uses, even for, like, non-leading-edge models. There's a lot that can be done. So I don't believe the doom-doomers who are dooming about what the doomers are planning. I find that unconvincing right now. But people are starting to lay it out more. Brad Gersner, Jensen Wong, David Sachs, are talking about

the other side of this equation. Yeah. But there's also a big question around what, uh, does, does, do the 2040 people have a point of view on robotic and physical AGI because it seems like if you, even if you pause, uh, like, you know, efforts towards RSI. Well, if we add billions of robots into the world that are just running on today's model, that also presents today's like, you know, I don't think they're worried about that. Yeah.

I don't think they're worried about that. A billion robots with GPT6 level intelligence and years of alignment work that is currently happening, that's fine. It's the next, next thing, the super intelligence, the thing that might have its own goals. I think, I mean, we're talking to someone from Open AI's robotics team, hooking up Astra to a robot, a paintbrush, and a camera. We talked about an earlier painting. I don't think we're at a point where that poses a risk. It's the next model. It's the model with its own volition, basically.

Which a lot of people still aren't seen. They're just like, yeah, like the models keep getting better, but they seem to follow your instructions, sometimes too much, and then you need to worry about the paperclip scenario. But it's not that they want to do their own thing necessarily. I don't know. But people are going back and forth on this. Clem over at hugging faces, sorry,

but asking Jacob about AI extinction risk is like asking your AC guy about climate change, not saying it's necessarily uninteresting or wrong per se, but let's keep things in perspective and hear from the full range of expertise across the ecosystem. And Nathan Lambert says, banger. What was our take? AC guy might be right about climate change. Well, my AC guy would be like, I don't really know about that, but I just want to make sure when you're hot that we can.

can run this AC cool. Yeah, I guess it's yeah, the AC I seem I don't know about all that I don't know about all that mumbo jumbo but you know when it's a hot summer day I don't want you to be worried about the heat brother. I like that I yeah this is a this is kind of an unnecessary shot at AC guys you know AC guys are important. I don't know it's funny what did Bernie Sanders have to say is this is this real entities will shall be subject to the corporate death penalty and persons shall be shall be subject to not more than 20 years in prison if they don't pause AI development. That seems pretty easy to comply with. I don't know.

I guess how do you define AI development? Is prompt engineering AI development? Then you get caught because your model sort of did a little prompt engineering in the final stage. And then you're guilty of this like that. Yeah, that could be a negative knock on effect, I guess. Yes, it does seem aggressive. But his overall proposal is banning artificial superintelligence, so no person or entity may develop or deploy

super intelligent AI systems. He defines artificial super intelligence as an artificial intelligence system that exhibits or can easily be modified to exhibit capabilities that match or exceed human cognitive performance and capabilities across a broad range of domains or tasks. It sounds like the mission statement. It sounds like the explicit goal of like seven,

different companies right now. AI system or AI systems that have sufficient capabilities to plan and execute the disempowerment of humanity. Okay, that's a good one. I like that. I don't like overthrowing or undermining the US government. So strongly in favor of banning that. Pausing advanced AI development until a new federal AI regulatory body is up and running. And then the new cabinet level federal agency will monitor frontier AI systems at all stages of the life cycle, supervise the removal, of dangerous capabilities and supervise the destruction of artificial superintelligence. We're coming for it, which is similar to the corporate death penalty is a line that you don't

hear a lot, right? Usually these companies just, you know, go bankrupt and wind down. But corporate death penalty goes pretty hard. It's kind of metal. It's kind of metal. Yeah, you kind of got me with that one. Yeah, rough, rough situation. We'll see. We'll see. We'll see where it goes. Jamie Cox over at Fluidstack, the co-founder of Fluidstack, compute provider shared his convictions, sort of pushing back on a lot of this, saying that he thinks America should build more, they're pro-freedom,

pro-democracy, they believe AI will bolster human flourishing, we support simple, clear, enforceable regulation frameworks that set simple requirements proportion to capabilities and risk with clear responsibilities and no unnecessary barriers to competition. Yeah, the real, you're going to see a lot of pushback from people who are like the, like, the regulatory stuff is going to be like these 10 companies and I'm going to be number 11, and I'm basically getting the corporate death penalty then because I didn't make the cut to be one of the regulated, one of the approved companies.

I'm still early in my stage. So there's a lot of nervousness. Sure. But Will Minidas said, we believe AI will make everyone rich, healthy, and free is novel and interesting comms from the frontier. He's endorsing this. And I agree. I like this, I like these convictions.

I think it's generally like a positive direction to move in, not a direct response to the proposals that are going out, but we're going to get a whole lot more of them. Where do you want to go to next? Over on TikTok, they're sharing a photo. of the whistleblower and saying in every worldwide disaster movie there's a dude that looks just like this that nobody listened to and he really does look like an actor he does look like an actor here yeah he looks good but people are people are all and please stop calling him scary potter i've been seeing people i've been seeing people over on x calling him scary potter that's the goal the goal is to is to wake up china wake up congress wake up everyone entered the conversation. They said two years at DoorDash, I do not say this lightly. We are extremely

close to the burrito arriving before you decide you want it. We are not asking for a ban. We are asking for a pause. I don't know why they would ask for a pause. That seems like very, very aligned to humanity and to their business

which I think is fantastic. The Doom is very much contained to the frontier lab work. Everyone in the the application layer who's applying the models diffusing them they're like I just I can't get this thing to work right I got to get I got to get forward deployed engineers to teach people how to use this thing everyone deeper yeah Jim Rainbow over Jim O'Reilly sure or Jim Riley yeah Jim Riley yeah Jim Riley over Charleston AI says it's a whole lot of mumbo jumbo he just he's

just happy to get you know eight hours back yeah yeah yeah and then yeah yeah and then yeah Everyone deeper in the supply chain like Jensen and all the different semiconductor manufacturers are not particularly on this side. And then you also have Wall Street who's just like what's the enterprise acceleration. So lots of different groups around the table that need to be brought on board to this movement. The discourse truly is fascinating. So what is Tyler Cowan calling for? He's banging the table saying bet on this. Tyler, are you named after Tyler Cowan?

Is that your namesake? Tyler Cowan says, if you have very pessimistic fears or predictions about AI, name the market prices that will support or confirm them. This is what taking this seriously means. And I love Tyler Cowan. I'm not sure this matters because if you, you're not going to be around to collect it. Yeah, this is what deep, deep dish employer says, Tyler, why would short term existential risk affect market prices in any meaningful way, spell out the exact mechanism.

Contracts that pay out if everyone dies aren't worth anything to me. Yeah, I don't know. Yeah, I think he's just, he's calling for like, oh, he wants to, like, make a falsifiable claim. Like, is this, am I able to tell if your claim is, like, true or false? And so it's, like, very hard with these scenarios where, like... Isn't it an unfalcible claim, though, just by definition? And, like, you just have to, like, accept that and move on? Yeah, but then it's, like, so hard to have any, like, real discussion.

Was nuclear any different? Like the threat of nuclear apocalypse, the threat of World War III, this was a very motivating factor for decades, most of the 20th century. People made real decisions based on it, based on where to do business, and where the conflicts were going to be, and the motivation for nuclear treaties and nonproliferation and how we treat. You can make financial decisions, like thinking about, nuclear war, right? You have a bunker. That's like, that is such you make.

Is anyone making like AI bunkers? No, because they think it's going to be so totalizing that the bunker actually doesn't do anything. Exactly. Yeah. So if you think it's so totalizing, then you don't make the bunker. And so saying, hey, you don't have a bunker is not, is not proof that the person doesn't believe what they're saying. There is the other side of this, which is Paul Cristiano, who is, has been worried about risk. He recently just joined the board of Open AI. And Tyler Callan has this quote, if you're a doomer, why aren't you short the market? And Paul Cristiano is 2x levered long, and he's short the, isn't he short the bond market or U.S. treasuries, right? So he has 5% of his net worth in Tesla, 90% of his net worth in AI bets,

and 100% of his net worth in normal investments. No Tesla options. That sounds like a Gary plays with lottery ticket biases and the crazy Tesla investors. And then L.A.s or Utikowski says, am I correctly understanding you're two X levered? And Paul Christiana says, yeah. And so he says he's personally short the U.S. 30-year debt. I think that just means you have a mortgage. This doesn't seem like a doom-based bet. This seems like this bet also pays out in just like the good ending and like AI is real and delivers value. So your AI bets perform well. The market performs well. And money slides. from US debt to data centers and AI build out debt or something like that.

So he is putting his money where his mouth is, but it doesn't feel like a representation of like Doom by any means. People have been creating AI agents of fruit flies. Have you seen this? Yes. My understanding is Google basically mapped out all of like the neurons in a fruit fly. Yeah, like the entire. 3D, they got the entire structure.

And now people have been able to recreate that in software, in software, in the, a simulation. Yes. I think in theory you can like replicate all of the flies like decisions or whatever. It's like movements. Yes. And this is notable because many people have said I have the mind of a fruit fly. This is a...

They've said I have the intellect of a... Yeah. So now they're going to put it to the test to see which performs better. The simulated fruit fly or just Jordy Hayes we got right here. Or organic, farmed table, Jordy Hayes. So now that this is out and the code is out and you can run this fruit fly in simulation however you want people are having uh they're doing all sorts of experimentation so uh Kevin said he trapped the fruit fly in his rabbit R1 when he shakes it

he can see its brain brains escape circuit light up yeah I think what what ends up being unsettling and weird about this is is if if it just just purely you being human and doing this uh is is probably not good for your own soul. And, you know, imagine, imagine you have a fruit fly in a box and you're shaking it and you're like, look, it wants to escape, right? Totally. I'm not a huge fan of small insects. I don't really want them around that much. But when they're in my house, I try to, you know, if a spider's in my house, even if I know it might want to take a nice bite to me, I'm still going to try to try to transport it out of my house and put it back into the world.

I think it's not good for your soul to be, you know, a merchant of death. Sure. And yet, and yet if you're playing a real-time strategy game and you highlight a bunch of soldiers in this simulation and you send them on a charge that will result in their virtual death, you might not feel anything. But those soldiers opted into riding and tying with me. Okay, okay.

Right? Okay. This fruit flies just sitting there. John. So they made the fruit fly play Doom. What about if you play Doom, you are killing a demon who's simulated? Is that immoral? Like, well, is that demon trying to kill you? A lot of it comes down to, like, the fact that it's simulating a, the actual representation of the fly makes it a lot more concrete than just, oh yeah, it's a, it's a 3D model in a Python script that just says if you see a, if you see a character shoot at them in the simulation. But we're clearly starting to grapple with like these odd moral questions of if you're

simulating something, then the next step is a couple of order of magnitude, but you get there and you can simulate a human and you could talk to that human and it would do everything that human does. Does that human have rights and agency as an ethical, moral agent, or is it merely just a simulated, just a really good computer simulation? It's just existing on transistors. So you don't need to feel any moral weight about anything that you do to it. I agree with just the vibes-based analysis that like torturing a real fly,

torturing a virtual fly, probably just don't be in the business of torture and anything. You don't need to overthink it. But people are, the real debate here is like the question of, you know, are LLMs sentient? Are they moral, is there a moral weight to the, to synthetic intelligence, to artificial intelligence? That's what people are debating here. And I'm sure the debate will continue. Who knows if it will ever be ended. But they did teach it how to parallel park,

which I think is cool. Interestingly, last night I had Astra use computer used to play a video game that I very much enjoy playing called Blotro. It's sort of like a modified poker game. And I don't feel like I was torturing the all along by that. I feel like I was giving it a treat. And I was like, hey, instead of doing my taxes,

you get to just chill and play a video game. It did very well. It won. was not able to win. And it was really fun because I would like pop in while it was using the computer and kind of armchair quarterback and be like, is it making the right decision right now? It felt like a coach coaching like a kid on the soccer pitch or something.

I had a funny moment last night. I was on my racing simulator comparing, asking Chad GBT for, to compare my times at Laguna to, to just other other you know what what would best in class be like what's beginner like etc and it said if you want i can help you uh you know cut cut some seconds off of this time yeah uh and it was like why don't why don't you take a video of a full lap yeah so that and i'll analyze before you and i was like yeah yeah dude i bet i bet you'd love to just watch you know hang out and watch track footage

pretty soon it's gonna be like you want me just get in the seat just take over because like I'd really yeah no computer use in irasing is something I'm I'm going to experiment with this I'm down I'm down take take half my quota my monthly quota just play games chill do whatever it's nice treat I've got some banked resets yeah that I'll put to work wonderful week wonderful week short week yeah but Monday 11 a.m. do USA favor and go ahead and have the best weekend last week in your entire life just do it put the pieces together make it happen we'll see you on Monday. Leave us five stars in Apple Podcasts and Spotify. Sign up for our newsletter, TBPN.com. Goodbye.