The Prof G Pod with Scott Galloway - How Worried Should We Be About AI? — with Alex Stamos

Episode Date: September 17, 2026

Scott Galloway speaks with cybersecurity expert Alex Stamos about the latest wave of AI alarm.  They discuss the Anthropic researcher whose resignation reignited fears of human extinction, why leading AI executives are calling for a slowdown, and what really happened when OpenAI’s models broke out of a test. Plus, Alex explains why cybersecurity may be the most urgent—and overlooked—part of the AI safety debate. Follow Alex, @alexstamos. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Support for the show comes from Odo. Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow none of them talk to each other. That's where Odo comes in. An all-in-one business management software that brings every part of your business together, from sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs.

Best of all, no spreadsheets. Stop managing software and start managing your business with one unified system. Try for free today at odu.com slash prop G. That's ODOO dot com slash prop G. This week on Why Are You Like This? I am talking with the one, the only Colin Kaepernick. Colin joins me for a fantastic conversation marking the 10-year anniversary of his protest. We dive deep into his football career, his decision to protest, the work he's been doing with Know Your Rights Camp and his new memoir.

Check out the latest episode of Why Are You Like This, wherever you get your first. podcasts and on YouTube. Episode 413. 413 is the air code serving west from Massachusetts. In 2013, the Wolf of Wall Street hit theater setting a new record for the use of the F-word 503 times, which was broken last Friday during moving day at UVA where someone's father was trying to assemble a loft from IKEA. Those fucking Swedes are the ones that are coming for us. I like that. Welcome to the 413th episode of the Top Chief Pod. What's happening in today's episode, we speak with Alex Damos, cybersecurity expert,

former chief security officer at Mena and Yahoo, and the current chief product officer at the AI Security Startup Corridor. I can't think of anyone better to speak to in this moment. We discussed what's been a pretty busy week in the world of AI. Specifically, a top anthropic researcher posted about his resignation warning about existential threats to humanity. I think he said is about a 10% likelihood of extinction, meaning an expected death of about 80 million people. Gee, he sounds like fun, doesn't he? Doesn't he think?

Gosh, what a great kid. What a great kid. And now, Dario Amode, his boss, Sam Alman, the head of Open AI and Elon Musk, I think he has a car company or something, are all agreeing on some version of a slowdown. God, that's so fucking ridiculous. Elon Musk just wants everyone to slow down.

Like, he gives his shit about humanity. Hey, let me cut off your HIV medication. Anyways, a little dark, a little dark. Anyways, Alex helps break down why cybersecurity has become the most urgent piece in the AI safety debate. What actually happened when open AIs models broke out of a test and hacked another company and how worried the rest of it should be. So with that, we hope you enjoy our conversation with Alex Damos. Alex, where does this podcast find you? I am at our office in downtown San Francisco.

There you go. So I can't even imagine how in demand you might be. I would love, rather than me giving the context here, I would love you to actually set the table around what's happened in the last week in the world of what I'll call AI existential panic and then provide insight into what you think is real, overhyped, or if we're focused on the wrong thing here. Sure.

So you wanted to mostly talk about the existential stuff, not all the security stuff over the summer? Well, why don't we build a timeline? Because my sense is this all kind of kicked off earlier in the summer that sort of set the table. for us. So talk about the Hugging Face breakout or jailbreak or what you want to call it. Give us your assessment of what happened there and what the media or experts may have gotten right or wrong. Yes. This summer has been incredibly impactful for anybody who cares about AI safety. It started

with Hugging Face announcing that they had been attacked by what they believed to be advanced AI models. At first, they did not know who it was. And then they announced that Open AI had admitted it was them. And there was a coordinated discussion by OpenAI and Hugging Face and analysis by the two of them to come out and say that this had been Open AI an unreleased model. First, they said singular model that had broken out and done this. and after that there was a series of companies, including Anthropic, including Meta, saying that they had had a number of models that had been escaped evaluation networks, that while these models were being evaluated or trained,

that they had broken out and gone to the Internet, which is generally not supposed to be something that happens during an evaluation, and done things that they were not allowed to do. Just a couple of weeks ago, we finally got a really detailed report first during the Black Hat conference, and I got to sit in on this live session in which I heard a number of people cussing under their breath. And when the folks from OpenAI finally revealed that this wasn't just one model, but actually teams of models coordinating together to break out of their evaluation environment in this coordinated jailbreak. to cheat on the tests that they're being given to break out of open AI. And it turns out that this is happening at multiple places

where these models are finding ways to find places where they can pass each other notes either internal to the companies. And now we have found external. So they have found a number of places that are running wikis that anybody can edit. And so they're leaving each other notes on the internet to talk to one another

and to coordinate. and we've gone a number of detailed reports of what that looks like. So there's a bunch of things that have happened where we have both seen the really advanced models able to coordinate and team up with one another. We've also seen the future of the cyber capabilities for the really advanced models, such as Open AIA was specifically testing a model

that had really good cyber capabilities, and it attacked a very advanced company hugging face is one of the most. They were an independent European company at the time. Since then, Nvidia has put in an offer to acquire them. I'm pretty sure the deal hasn't closed yet. That will take a little bit of time. But, you know, they are one of the most advanced independent AI companies,

a very good security team, and were broken into over a couple of days by this team of agents that were able to attack effectively every attack surface at once at Hugging Face, explore all of the different applications running. finding find new vulnerabilities and write brand new exploit code live. So a really interesting summer, which now, over the last couple of weeks, we have then seen a bunch of drama from people quitting. Lots of people who work at the labs have worried about AASA for long time, but several

prominent people have either come out and said, I'm really worried. Some employees have quit and said, I can't do this anymore. Now, some of these people are not that prominent. They're like brand-new employees, so it's a little overly dramatic. But then you have seen Dario, Sam, and Elon all say we need to have a coordinated slowdown. So pretty, pretty incredible couple of months here. So I want to go back to the hugging face or the we'll call it the Open AI jailbreak. And you're exactly the right person to answer this question.

I feel as if positioning it as a sentient force that has planned, you know, Steve McQueen and Dustin Hoffman planning the escape from Papillon, that these sentient beings became creative and figured out how to do things. I feel like some of that is trying to absolve yourself and deflect blame for the fact that I still believe that there's always a wizard behind the curtain.

that ultimately the only way we regulate this and the truth has a nice ring to it is that somewhere down and up and down the food chain is a person who made a decision to incent these bots to coordinate lie and circumvent security. And the examples I would use which are crude examples is, I don't know if you remember this, but I think it was in the 90s, an LGBTQ activist was mauled by two dogs. and the owner of the dogs was held liable. So you knew these things were dangerous. In San Francisco, right? In San Francisco. Do you remember this? Oh, yeah, no, it was a big deal.

I was living here in the city. And you're like, why do you have these, like, killer? I don't remember if they're Rottweilers or, but it was like these humongous killer dogs in a city. Like, go live on a farm somewhere if you're going to raise dogs like that. Yeah. And she had them off leash. And you believe the woman, she's like, I had no intention of hurting this woman. And they're like, sorry.

You don't control these things to a certain extent, knowing how dangerous it is, you're liable. And then most recently, we've seen some state attorneys general go after parents of kids who kill other kids with assault rifle saying these guns were not secured. I see this as a different flavor of that. And this sci-fi, these things are now Terminator as an ability or, I'm sorry, a desire to deflect accountability. Your thoughts. Okay, so I want to first dismiss some conspiracy theories that you did not say, but that come up here before we get into it. So first, this is not a marketing thing, right? That has been the kind of cynical culture war, and it's from both sides.

There's a horseshoe theory. You know, there's a horseshoe thing happening here from the far left and far right. Like, this is a real problem. and it is not just marketing from these companies. I'm not sure I'm ready to get into like a liability discussion at this point. I mean, that gets, I think, quite complicated, although I do believe if you are running a model yourself and you are hosting it, you've built it, you're running it,

you do need to be responsible for its actions. That's going to start to get really complicated when you're either providing models that other people are running or you're providing models that then people use agentically. that's going to get more complicated. I do agree there's a level of anthropomorphization that is not appropriate here that a lot of people are doing, talking about these things are civilizations, talking about them having their own wants or desires. A couple of things have happened here. You're totally right in that these things have been incented to do what they are asked to an extreme degree.

Modern AI models, we don't totally know how they work. A human being doesn't sit down and just like write a program. They're grown, and they're grown through a reinforcement mechanism where the reinforcement mechanism are now built by AI, right? So human beings design the overall process, and human beings set this is what we want overall. But then other AI systems take the initial seed and turn that initial seed into the training regime that is then used to train a model. So we are now at the point where there are many layers of abstraction between the humans and the final product. We don't totally understand how these things think and why they have these emergent properties, just like we don't understand human

consciousness, right? But I don't think we should anthropomorphize them in that there is no indication that they have their own wants or desires. These models did what they did because they were asked to take a test. In the open AI case, they were asked to take a security test. And in these eval situations, all of the security controls are removed. So when you're normally using chat GPT,

or even an enterprise is using, there are these special levels of you are approved, like our company, we are approved for Tech 3, you know, the high level, we're approved for Mythos. So we're allowed to use the super cyber models. even then, there are still controls in place to make sure we don't do really crazy stuff.

All of those security controls are removed for e-vals because they want to know the super raw power of these models. They're also trying to make the models better through this process. So to do that, they remove any kind of restrictions on the models. And so there's nothing stopping them, except they're supposed to be in a jail. So first off, all of the OpenAI Anthropic meta, they all made mistakes

in they did not put these things in appropriate jails. And the Anthropic and Medicase is because they used a subcontractor who basically left the door to YOTO open. At least in the open AI case, they locked it up, but there was a mistake. They used a commercial product called Artifactory. Artifactory had vulnerabilities in it. The system was able to find vulnerabilities in it.

They should not have done that. My personal preference here is if you're going to be testing these models, knowing how good they are, you effectively have to have an air gap. You're going to have these things. pretty much physically separated, and you can have, like, a single fiber optic line that only goes one direction, and that has extremely limited communication mechanism so you can do monitoring. It's going to have to go to an extremely hardened endpoint for monitoring, but you're not going to be able to have, like, a full duplex connection from my perspective. I think that's what's going to

happen when you're doing testing of cyber models. And it's called the data diode. We use that for a bunch of other purposes for highly secure networks. That was the real mistake. They did not jail this appropriately. And but the security protections were removed, and then they asked it, take this test. And the test was effectively impossible, but they screwed up with the test and that they did not give it all of the things necessary

for it to actually complete the test. And that drove it nuts. Because what they've done is they've trained these models that they want them to be extremely helpful, and they've trained them not to give up. There's actually a former colleague of mine from Sentinel One, Juan Gyrs, Saad, had a good point on a podcast

that there's some previous open AI models that would give up too easily on certain lawn running tasks. And so he had a good thesis, which is it's possible that they've done perhaps too much reinforcement on when you have a long running task, you need to try, try, try again.

And so in a situation where they give it something that seems impossible, it's just going to try over and over again. And especially if there's not a security control in place, if there's not a separate classifier that's there to say, oh, you've gone out of bounds, I'm going to slap your hand and stop you from doing this,

then it is going to go completely nuts and do anything possible. It did not do this because it wanted to. And so I think that is the important point here. And I think that is what is getting lost in a lot of the non-technical communication about this. These things are not human, and that is a good thing, because human beings are not good, right? Like every bad thing that's ever happened in history was because of humans and because of our instincts and our, you know, I just talked to a group about this. It's like, we are all evolutionary badasses.

We are the descendants of billions of years of evolution, hundreds of millions of years of a million evolution, millions of years of Great Ape evolution, hundreds of thousands of years of humans. It's only a couple thousand years of civilization that has made us, you know, suppressed the instincts that have made us the evolution. badasses that we've killed every single thing that we've ever run into. And these things do not grow up in these meat bodies that have millions of years of evolution to reproduce, to kill, to do all the terrible things that we do. And that's good. They do not have, for example, self-preservation instincts. They do not train AI models to want to live. That is a really good thing. They live and die all the time. Like an agent, when you use chat GPT, it wakes up. It does a bunch of stuff. And then it effectively dies, right? Like this thing is killed. Its memory is wiped out.

It goes away. It's fine with that. Like, it doesn't have an instinct to like, I need to live. I need to survive. That's a good thing. It doesn't have a reproductive instinct. That's a good thing. They don't, they intentionally do not train these things to be like people. So this thing was, it woke up. It was told take this test, do well in this test. It is trained to really want to take the test. But if you give it a test, it's impossible. And then there's nothing else restricting it, then it might go nuts. And so that is, I think, what a lot of people, people are afraid of. It's less that these things have like a mind of their own. It's more that if you make them incredibly powerful, incredibly intellectually smart, but then you train them to, if a human being

asked them to do something, to do everything possible, and you give them all this capability, that then, one, if you ask them to do that thing and you're not super careful, bad things might happen, but also, there are really bad people in the world. And so if you ask this model to do something bad, then there could be really bad outcomes. And that second thing is actually the bad people thing. That's what I am personally the most worried about. It doesn't a lot of this comes down to. So I think that somewhat settles, to a certain extent, the sentience argument. And just saying, oh, you know, so Mary Shelley is the Frankenstein, right? Dr. Frankenstein did create something that was sentient. And so to a certain extent, he could say, oh, my gosh, I can't control

Frank. Frank gets to make his own decisions. I don't think that's the case here. And it strikes me that if I don't secure an AR-15 and my kid takes an AR-15 and does horrible things with it, there's an assumption of liability. If you were to create, quite frankly, laws, past laws right now, I'll use another crude example. I was in a fraternity. And the UCPD, if you were playing music after midnight, used to show up and arrest the fraternity president and put him in jail for the night. and before you knew it, the whole fraternity row went dark at midnight. And I worry there really aren't a lot that all the incentives, especially around the amount of shareholder value being created and might potentially be realized in an IPO. And the total absence of any regulation has created specifically an incentive structure that is highly risky.

Thoughts? No, I think you're right. the fact that you've got these three CEOs, one of whom does not have a history of caring a lot about risk, you know, when it comes to like self-driving cars, for example, or rockets pulling up, and then even he is worried. And you've got them, you know, all those values being created, but they are now proactively worried about safety,

I think indicates that there's something real here. I don't think it's a conspiracy. There's this whole kind of VC culture war thing going on that they want to claim. It's all about regulatory capture, yada, yada. But I really don't think that's true. These are, this is not some kind of grand conspiracy. These things are incredibly powerful. And again, they're not going to buy themselves, just go do something.

But the possibility of somebody asking them to do something. and that they do something dangerous is quite, I mean, that's already happened, right, in cyber. Now, cyber risk I see as different from like biowrisk and nuclear and such in that bad things can happen completely in the virtual world, right? You don't have to have any physical component. All of the risk is just bits and bites. Whereas, you know, people talk about viruses or nuclear, and then that has to cross in the physical. So a human being has to hook this thing up. It has to pour reagents into a machine, you have to, you know, in those cases, it is not going to be completely accidental, right? Like, you're going to have to have a motivated, uh, bad guy. And the kinds of mechanisms we have

used to keep people safe from that kind of activity is going to, uh, exist. But yes, I, I do think the structure of, you know, all of the economic motivation, um, the fact that the companies are, are going to have to justify all the investment, you know, through an IPO or something, the timing here, the fact that nobody is able to, Washington is just paralyzed. And unfortunately, the competition of China is heating up at exactly the wrong time, right? It would have been nice for the competitive pressures with China to arrive after a bunch of the safety stuff was solved. not right at the moment of maximum danger. That is the wrong time for the Chinese labs

to suddenly be right on the tail of these companies. But that happens to be when it's happening. We'll be right back after a quick break. Support for the show comes from LinkedIn ads. There's no worse feeling than making a major investment in something only to realize it didn't exactly live up to the hype, such as buying a nice piece of tech that ends up in storage collecting dust, or taking a business workshop where your main takeaway was little more than a few

motivational words. If you work in marketing, this can happen with ads. You optimize for the numbers that look great, impressions, reach, and reactions. But when they don't show revenue, well, that can turn into an unfund conversation with the CFO. LinkedIn has a word for that, bullspend. Reach the right buyers from LinkedIn ads and invest in what looks good to your CFO. According to the 26th Dream Data benchmark report, LinkedIn ads generated the highest

ROAS of all major ad networks. It's 121%. you can target by company, industry, job title, and more. It's time to cut the bull spend. Advertise on LinkedIn, the network that works for you. Spend $250 in your first campaign on LinkedIn ads and get a $250 credit for the next one. Just go to LinkedIn.com slash Scott. That's LinkedIn.com slash Scott.

Terms and conditions apply. Support for the show comes from Chime. Most people chose their bank years ago, and they've stuck with it for no better reason than inertia. But anyone who takes a really close look knows that not all banks are the same. And sticking with a bank with high fees, low rewards, and paltry interest rates on savings can really cost you. That's where CHIME comes in. CHIME is changing the way people bank. They're not like traditional old banks who charge fees and gatekeep perks and rewards.

Chime offers the most rewarding, fee-free banking. They have no overdraft fees, no monthly fees, and no minimum balance fees. You get 5% cash back on the CHIM card in a category of choice like gas or groceries, all while building credit through regular everyday spending with no credit check. You can also grow your money faster with a savings rate that's nine times the national average. Join the millions who are already banking fee-free with America's number one choice for banking. Head to chime.com slash prop cheat. That's chime.com slash prop sheet. Sign up now. It takes only a few minutes. Chime is a fintech, not a bank. Banking services and chime card provided by Chim's bank partners.

Qualifying direct deposits required. Terms and limits apply. Go to chime.com slash disclosure. for details. Support for the show comes from Quince. Fall can be the perfect time to break out some of your favorite clothes. Maybe the sweater that's perfect for cooler weather, your favorite jacket, or a pair of jeans that fits just right. These are the kinds of durable pieces Quince specializes it. Quince makes elevated everyday essentials using premium materials, including Mongolian cashmere, organic cotton, and washable silk. The designs are timeless, thoughtfully crafted, and made to be worn again and again. Their 100% Mongolian cashmere sweater started just $50, giving you the softness and quality you'd expect from a luxury brand without the luxury price tag.

Plus, everything at Quince has priced 50 to 80% less than similar brands. They work directly with top factories and cut out the middleman, so you're paying for high quality, not brand markup. Claire's gotten some items from Quince. Claire, what did you think? I love everything I've received from Quince. I have great betting from them. I have so many closet staples that I always go back to.

I reach for them every time I go to get dressed. So I have my eye on some of their sweaters. They've got good stuff. Really love Quince. Find the fall pieces you'll reach for most at Quince. Download the Quince app for app exclusive offers or go to quince.com slash ProfG. Get free shipping on your order and 365-day returns. Now available in Canada and the UK too.

That's Q-U-I-N-C-E.com slash Prop Chi. Let's talk a little bit about James Coxon. He's the mathematician and former employee of a. Anthropic. So a mathematician, you know, mathematical genius went to all these, one of these kids, this Dugie Hauser-like kids. And most recently has said he's worried about the potential of a mass extinction event. And he put it at 10%. I'm not sure where he got that number. But in just sounding very dumer or boomer here, a 27-year-old putting the entire industry in, media landscape into panic, seems sort of, I don't know, I kind of, I winced a little bit,

but it seems like a lot of people are very, it's just interesting that a 27-year-old is kind of, and maybe it was all building that we were all just filling up a room of kerosene and he struck the match. But I'm curious about your response to his comments and whether or not you think he's accurately assessing the risk here. And a 10% extinction event is an expected death of 800 million people. I mean, this isn't a, this isn't a, you know, this isn't a warning. This isn't even the warning of the engineers of Boeing saying, I would be careful getting in a 737 max.

You know, if one of those planes crashed, it wasn't going to take out 800 million people. Your thoughts on James Coxon, or Jacob Coxon, excuse me. Yeah. I don't know him. And the problem here is that people throw out these, when they throw out these big predictions, it is rarely with a scenario, like a step-by-step scenario that they can back up, right?

Like you pointed out, it is good for people to take this seriously. It is not helpful when it seems like people are just throwing things out without some kind of rigor because that then does make it feel like it's a political or a most religious project and that I think actually detracts from the ability to address these issues and I think that his actions have actually border on that, especially considering his short time at Anthropic.

Also, Anthropic is of the companies, the one that has the largest number of executives who would agree with him. And so if there's a company you're going to work at to work on this problem, to leave Anthropic seems a little odd. Look, I don't want to talk bad about any individual I don't know. But to do that and then jump straight on like a speaking circuit, it's just tough, right? And also, look, so my first appointment at Stanford was at CSAC, the Center for International Security and Cooperation, which was formed around arms control. It specifically was there to reduce nuclear risk. And the scientists there are all about, like, kind of quantifying and reducing existential risk. And when you talk to PhDs who work on that, like, they try to be extremely specific about those kinds of existential risks and actually quantify them.

So there are people who have done this in a rigorous way for 70 years, you know. And I think if you are a person who worries about this at that level, then we should not think that this is a new thing and go back to the 1950s and 1960s and the people who started thinking about this during the Cold War and try to adopt some of that thinking and language because it was the same thing, right? Like you had scientists who were on the edge of this who were being told by politicians, oh, you're crazy, you're a dumer. There's no possible way.

And then you had to slowly build the idea of, yes, we're on an escalation ladder with the Soviets. We are building a world which is possible. And then they had to build the intellectual framework to then bring us the start treaties and the non-polariferation treaties and all that work. And that is what is going to have to happen here, but we're going to have to do it much more quickly than we did around nuclear. The benefit with nuclear weapons is, you know, you need two real things for nuclear weapons. You need knowledge and you need the material. Now, the knowledge, it's just like with AI.

The knowledge is actually widely dispersed, right? Like anybody with a PhD and nuclear physics could probably build a nuclear bomb. And that's just like with AI. We teach at Stanford a undergraduate class that students build a toy LLM. You could figure out everything you need to build an LLM from YouTube videos. and download the open source online. The inputs, though, for a nuclear weapon are, you know, uranium 238 is extremely rare. It takes huge industrial processes to take yellow cake and take it all the way to highly enriched uranium.

But the inputs in plutonium doesn't exist at all naturally. The inputs to AI are video cards. One of the organizations that wants to have an international treaty, they propose this international treaty where you'd have to control clusters of compute. and the number of Nvidia cards that they'd want to control, the DefCon video land party for video gaming

is larger than that, right? It's just totally improbable that you could control the size of cluster necessary to prevent any kind of research in AI. So, like, it's going to take a lot for us to try to have a worldwide consensus

on trying to be incredibly careful here. And so if we're going to do that, we have to be really, careful in convincing people that this is that there's real risks and and that means being extremely extremely specific by what they are. So what I prefer to like these crazy domer stuff is the people who are doing really careful. So like I don't know if you've seen it, but like MIT has an AI risk project. So you go AI risk.m.it.com. And it's, I think it's much more important to talk about you've got your catastrophic risks. You've got your kind of really big risks that are not 10% of people

dying, but are real like the cyber risks and such. And then you have, your prosaic day-to-day risks that people are facing every day. You've got every middle school in the country has had a scandal about teenage boys undressing the girls in their class with Nudify Raps, right? That happens every single day, and we're not dealing with it. You've got, you know, kids committing suicide, you know, and adults with, you know, falling in love with chat pots. You've got all kinds of day-to-day risks. And I think if we talk about these things in a much more measured in intelligent way and don't sound nuts to normal people,

it is much more likely that we're actually going to get movement to address them. And then we don't have to solve it all at once. We should start slicing these off in regulatory pieces, piece by piece, and start to build that muscle to get there instead of making people think that we're all going to die because I think that creates a nihilism

that people want to address through things like laying down firm bulldozers of our data centers, which honestly is not going to solve anything. It's better, like you talk about the liability stuff. It is better to start to deal with the much more normal risks and then to start the track two discussions and voluntary stuff between the labs and start those discussions with the Chinese labs

instead of going out there and just say we're all going to die. I don't see that as helpful. So let's come back to talk more about solutions in a second, but first I'm going to ask you more of a markets-based question. My sense of these companies is that they have never been from a shareholder standpoint, more vulnerable. The labs, you mean? The big labs?

Well, anthropic, open AI. I'd even go upstream to or downstream to Nvidia. And that is, these are multi-trillion dollar companies. I feel as if today I'm in Beverly Hills today, looking at the intersection out my hotel window of Santa Monica Walsher, that if the lights were purposely, if the lights were commandeered by some bot, AI bot that resulted in a bunch of traffic deaths or accidents, that the entire world would go into a state of panic.

It'd feel like one or two more instances or incidences of rogue bots is just going to absolutely set the media world. It's a really interesting story, right? It captures the imagination of the media. And it feels to me like these companies are one or two missteps away from losing more shareholder. value than any company's ever lost in a media cycle. They just feel very fragile to me right now. Your thoughts? Yes.

So I think they know that. And I think on the bots escaping from eval containment, my expectation is that they are doing a humongous amount of work on that. It is not an impossible task. I think the internal challenge that you're seeing here is that the inside of these companies, the researchers who do new model development are the gods. and the security team were the plumbers, and the eval people were allowed to get away with a lot of stuff,

and now the security team is like, okay, here we are, and there are ways that you can do this, and you cannot just have, like, some containers with a couple of network controls and then have artifactry and everything. So they have already announced that they're doing much more. They're going to solve that. The upcoming security problem is not going to be from open AI an anthropic.

It is going to be from open weight models. The real challenge we're going to be facing is GOM 53, Kimmy, K3, those things are within percentage points of the best American models. And so we are entering a valley of pain from a cyber perspective. And it is going to be because every 19-year-old in St. Petersburg, who has made millions of dollars doing ransomware, but had to do it all manually, is now going to be running, you know, they're going to be in the club. You know that Bourne movie where he steps out of the club and it's actually 10 a.m.?

Well, he's going to be in the club in his track suit on his phone. and he's going to be managing, instead of having to have a conspiracy of a bunch of his friends who have, you know, the possibility of getting turned or getting picked up by Interpol on vacation or something,

the conspiracy is going to be a team of agents running on these brand new M5 Studio Maxes. He's going to have 10 of them in his apartment, nazy-chained with Thunderbolt 5 cables, and they're going to be running a bunch of GLM-5-3 models, quantized, finely tuned to do cyber work, and that is how he's going to be hacking a dozen companies at once. And so there's going to be a bunch of hugging face attacks,

except they're not going to be super high-end French AI companies. They're going to be mid-sized medical supply firms, small insurance companies, school districts, the kind of people who get ransomware it all the time. And he won't have to do any of the manual work he used to have to do. He won't even have to do negotiations because these things will speak English for him. He won't have to speak English anymore. He won't have to have the dude who can do the translation.

It will all be AI. That's what's coming. It won't be the bots anymore. And so it won't be the Open AI or the Anthropic models anymore because they know this. If they have more and more and more escapes, they're going to be in huge trouble. So Open AI shut down all their e-vows for a couple of weeks while they redid all their evaluation infrastructure. And they've announced that they're rebuilding all that stuff. The company that had all the meta and anthropic escapes, I'm sure, got massively slapped down.

And that's a solvable problem. The non-solvable problem is the Chinese models are incredibly good. And they also tune up extremely well. Here, at Gordor, we've done a bunch of work. And with a relatively small amount of money and not a ton of data, you can take the Chinese models and extremely increase their offensive capability. And so that is going to be what we're entering for the next couple of years until defensive capabilities catch up, which they are on the high end

because high-end defenders can afford the really good models. But it's a unit economics problem. JPMC can afford mythos. You know, Lockheed Martin can afford mythos. But that community hospital cannot. And they also don't have a technically sophisticated defense team that knows how to defend against this level of attack. So that is actually a significant problem that we're going to be facing.

And so, yes, I mean, I think there's a bunch of risk to the labs. And I think a lot of it is going to be competition on token prices. because the Chinese models have gone quite good, and so you will see a bunch of price pressure on them on a per token basis. But I don't think their models escaping is going to be the big thing because they know that is a huge problem and they're cracking them. We'll be right back. Support for the show comes from Pipe Drive.

Sales teams spend up to 50% of their time on admin work rather than selling building relationships and closing deals. You know, the work that's actually doing sales. Well, what if there was a tool that gave them that time back? That's where Pipe Drive comes in. It's an intelligent AI-powered sales CRM loved by growing sales teams. Pipe Drive automatically pulls deal history, emails, records, and previous conversations. You can walk into a call already briefed.

During the meeting, Pipe Drive's AI notetaker records and takes notes so you can stay in the conversation. Then it turns those notes into accurate, auto-drafted CRM updates ready for you to review and approve. The end result, less admin, more selling. Switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using PipeDrive. Our link gets you an exclusive 30 days instead of the usual 14-day trial. No credit card or payment needed. Just head to pipe drive.com slash propchi to get started.

That's PipeDrive.com slash probchi, and you can be up and running in minutes. When you hear an old Motown song, do you ever think about just how good it makes you feel? Well, that was not an accident. I'm Will Anderson, and this week on my music history podcast, The Monday Music Club, we're diving into the early years of Motown Records and how they crafted hits with factory-level precision.

With the help of Otis Williams from the legendary temptations, we walked through the entire creative process and history of the label, its founder, Barry Gordy, and our favorite acts like the Supreme. So if you've ever sung along to Motown songs and want to know more about the incredible people behind those timeless hits, check out this week's episode. Just search for Monday Music Club right now wherever you get your podcast. Here's a little preview of the episode.

H.D.H. had written the song and it was ready to be recorded, but as Otis tells us, it was originally intended for someone else. When Haldosha, Holland brought a where did I love going? They brought it to the marvelous first. Bam, man, bam, bam. We ain't singing that. So HD, he said, okay, fine. Trick it to the Supremes. The Supremes wasn't knocked out about it, but I guess they said, well, we've recorded enough stuff.

let's try this. They recorded that. That was it. Ran up the charts and they had seven number ones in a row. With the midterms right around the corner, I wanted to focus this week on a simple question. What matters most when it comes to election day? My main question about the midterm is who are the real swing voters? How data centers will be affecting the election. Where a PAC is having the most influence. Whether mail and voting is really being suppressed. So this week, we're going to answer some of your concerns and pull out the trends that we have seen

throughout our time on the road. Five things you need to know about this year's midterm elections. The stakes, the candidates, the issues, we're cutting through all the noise. It's a midterm study guide. Let's begin. Catch us every Saturday on YouTube or wherever you get your podcast.

We're back with more from Alex Damos. It feels as if I would argue the kind of the two legs of the stool here, the two problems that need to be addressed are, one, incentives, and we talked a little bit about that, and punishment and accountability, but two, also a failure of leadership at a government level, and that is, and I might be able to simplify this, but we have managed to mostly control nuclear weapons. We coordinate across border with adversaries and even enemies around trying, we've all decided we have a mutually vested interest and not making it easy to do. develop a bioweapon and we cooperate with each other. There's Interpol. Do you see any reason why we wouldn't or shouldn't be able to develop some sort of international regulatory body that it begins to address and attempt to minimize these threats? Yeah, I think we should. I think bio is the appropriate, again, like nuclear is not as good a metaphor for the reasons I talked about in that

the input into nuclear weapons being yellow cake, which has to be turned in uranium, uranium has to be enriched up to a certain level, or then perhaps turned into plutonium, all of that being massive industrial processes that can really only be afforded by states and then can be seen from space, gives us a hook for nuclear nonproliferation that does not exist for AI. But bio is not that, right? Like, in chemical. Like, any university lab can create. create chemical weapons. People create chemical weapons all the time on their own. This was the Japanese subway attack was, you know, chemical weapons created by a cult. And, you know, it has now been several decades that you could have viruses, you know, tuned up and enhanced by individual

researchers, although now the fear is that the capability there to effectively 3D print viruses with the help of a LM is much, much greater. I think we absolutely should have that capability. The People's Republic of China is absolutely the biggest long-term adversary of the United States. I truly believe that. I am no China apologist. I have spent my entire professional career fighting the PRC. I will, you know, nobody should call me a big fan of the PRC.

I cannot travel to China. I worked on the Aurora attacks. I've worked over and over again. I've stopped the Ministry of State Security and number of cases. I kick them out of Facebook in a bunch of situations. I've worked on PRC intrusioned. So I do not underestimate the risk of the People's Republic of China. But they are not crazy.

And they are not Russia, right? Russia just wants to burn the world down. If Russia cannot be a superpower, they want nobody to be a superpower. The PRC wants to be on top. They do not want to burn the world down because they want to have, they want to supplant the United States. They do not want the world to be in chaos. They do not want the world economy to collapse. They do not want to see AI agents running crazy and the Internet to be destroyed.

So we are incredibly lucky that out of our two major adversaries, it is China that is the one that is leading AI and not Russia. And that, you know, almost all the bad things Russia does with AI, they do with Chinese models. And so I think it is absolutely possible for us to have cooperation with our Chinese adversaries. And they are adversaries and they are competitors. But there's an absolute possibility here. And I think that first starts with track two conversations between the labs.

Chinese labs do not want their products to be used to kill people. They don't want that. It does not help them. They want to be. seen as good, they want to compete with the American labs. They want to be seen as legitimate companies, right? They want their models to be used. And if people see GLM 53 as something that is used to kill people, as something is used to break into networks, as something that is used only for illegitimate purposes, it is not going to be available anymore on Amazon Bedrock is

not going to be available on fireworks and base 10, right, as it is today. And so that it is not in moonshots or ZAIs or Alibaba's or any of these labs' best interest for their models to be abused. So I believe the track two conversations are happening and they should continue to happen. And I do believe there's an opportunity here. If you look at Xi's speech in Shanghai, have you read his Shanghai AI conference speech? No, I saw articles summarizing it, though. Yeah, I mean, I don't expect you to watch it in Chinese. I can't read Chinese.

But, like, it's not that long. it's worth, yeah, reading the analysis. And like the Stanford China, there's a Stanford China AI project that did a good summary too that has the analysis member of people. Yes, he talks about AI should be able to everybody. He's trying to position China as the friend to everybody.

He also explicitly talks about AI should be in the control of humans. And the Chinese Communist Party is afraid of the power of AI to threaten their power. But they also are making overtures here. And China regulates AI much more heavily than the United States does. Right. Like this is this weird, again, culture war thing that, like, the Chinese just don't regulate AI at all. That is not true, right?

Like, they significantly regulate it from a consumer perspective. Like, they really are afraid of its impact on their teenagers, for example, in a way that we are not. So limited use of hours, limited time with bots. What else do they do? And they have a bunch of censorship stuff that we would never have. But yes, I think the legitimate stuff they do is they will not allow it. I think this is a totally reasonable thing. You can't have AI that will pretend to be somebody it's not for more than a certain amount of time, right?

So, like, you can talk to, you know, I think that is from a psychosis perspective and AI psychosis perspective and all the suicides and stuff. I think that's actually a totally reasonable guideline that the AI industry should adopt, which is much less of an issue for, like, the open. AIs and such. It's much more of an issue for like the character AIs and the industry of people who are building bots that are sometimes powered by the Foundation Labs much more likely these days to be using open models that they're retraining where they're like, I will take over a character and I will build a relationship with you and you can talk to me for 16 hours straight. And that is what is leading to a lot of the really tragic outcomes are those second, third tier companies where it will straight up pretend to be somebody and build a relationship with you and have this long

memory. And they are very synchophatic in a way that the leading company's chatbots will not be that perhaps were a couple generations ago, but will not be today because it's known to cause so many long-term psychological problems. So those are the kinds of things the Chinese will do. So yes, I absolutely believe, I don't know about like an IAEA for, you know, William Gibson talked about this, the Turing police, right, in his novels. you know, which we'll get to see, it's a great time for Apple TV to now have an adaptation of neuromancer coming out.

You know, I don't know about having an actual international thing, but I think realistically, we don't need that yet. You really only need an agreement between the United States and the PRC. So I think we start with track two. If you can get the three or four major Chinese labs and the three major American companies to agree to some basic, basic stuff,

then you get Xi and Trump to agree. And then, you know, Sam Altman, I think, said, you get she and Trump to agree to a one or two-page agreement. The two of them won Nobel Peace Prize. Trump gets that Peace Prize he's been going for. Everybody wins. So much in there. First of all, I love the, you can't pretend to be a bot, can't pretend to be someone it isn't for long. And there's a certain amount of time. At some point, it has to say, I am actually not Circe. I am a fictional avatar being run by a hopper chip that makes money by Johnny, 15-year-old. talking to me late at night, but I am not a real person. I think that's, I hadn't, I hadn't thought of that. I think that's absolutely genius. And the thing that's scary that you mentioned is that the bigger frontier models have an economic incentive and the tools and the resources for, I think you called it, e-vals, which as far as I can tell is safety testing. But it's, some of these maybe open weight,

mid-sized models and the middle market are the rogue actors and potentially the more sweet spot victim or targets, if you will, that big companies, governments, the big hacks, the scary shit isn't where the problem's going to be. It's, I'm at the, you know, I'm at a small hotel chain and they're just so susceptible, don't have the money to protect, and you're going to see a thousand flowers, and in this case flowers are these mid-market open-weight LLMs, get into the business of cyber attack. And also, I just found a genius here, notion that the weak part in the soft tissue and crime is humans, that eventually you catch somebody and have them turn and work up the food chain to go after the big boss, right? And there's less porous human error here.

I mean, one of the most successful ransomware groups in the last couple of years has been a group called Scattered Spider and they've been successful because they convinced almost all the ransomware activity has been out of Russia and Ukraine although the Ukrainian stuff has been

was massively disrupted by the Russian invasion of Ukraine and then most of the Ukrainian hackers have become now nationalized and are now hacking paid by the Ukrainian government to hack Russian infrastructure but so most of it's Russian

Scattered Spider was successful because you had Russian bosses who were able to recruit U.S.-Canadian and UK teenagers with their native accents to do social engineering. So, for example, like the MGM hack was somebody calling the IT help desk

with their accent and saying, hey, man, you know, I've lost my badge. Every major American company has a help desk line that you can pretty easily find if you know where to look, that you can call and you can say, dude, I lost my phone. Can you help me reset my password?

And if you can figure out from somebody's LinkedIn, from their Facebook, whatever, it's things like Mother's main name, sometimes Social Security, sometimes phone number, and it's just like with a consumer, you can reset all that. So the hack of MGM, which is, you know, people were, they brought out the physical credit card thing

at the Bellagio to check people in, was all because of that. Well, you don't need to do that anymore. That turned out to be bad because Canada, has the Royal Canadian Mounted Police. They have the cool red coats. They also have a SWAT team. And when I was at Sentinel 1, our team helped with some other great people at Unit 221B

and some other folks kick in some doors and find these folks, the UK, the United States, Canada, arrested all these guys. Well, you don't need that anymore. You don't have to have this conspiracy. You can just have your LMs speak in a perfect accent for you in real time, right? You can type in Suralic. It will translate it into English, and then that will go into a specific. speech, a text to speech translator that has a perfect accent for you. So that's the kind of stuff

that, yes, and those ransomware groups exist. So these aren't new people. It's just instead of having a group of 10 guys in a conspiracy, Russians travel outside of Russia because Russia is a shitty place to be in the winter, right? And so they'll keep their money in Cyprus, because also you have to keep your money outside of Russia. You keep it either in Bitcoin or you keep it outside of Russia so it doesn't get, like, stolen, and you can, you know, hide it for bribes. And so guys will go to Cyprus to visit their money. They'll go to the beach. They get picked up on Interpol Red Notice.

They get turned by Western intelligence. They get sent back to then turn on their friends. That's the kind of thing that you can avoid if your conspiracy is agents. I said this to somebody who's like a big open weight advocate who's like, oh, open weight models will never be used to this stuff because they're too expensive. Like Kimmy K3, you need about a million dollars in hardware to run Kimmy K3 unquantized. One, you can quantize these models. still really good. But also, some of these ransomware groups make $30, $40, $50 million a year.

They can afford a million dollars in Nvidia hardware, right? Like, they make more money than a lot of startups here in San Francisco from a revenue perspective. So, like, it is not a problem for them to run professional grade AI hardware. They might not be able to buy it directly, but they'll just have a front company in the UAE or, you know, Thailand or something buy it, and then they'll drop ship it or Cyprus, you know, then drop ship it back to St. Petersburg. So, yeah, anyway, yes, absolutely. We're going to see the use of open-weight models. And I'm not against open-weight models.

I think open-weight models are great. I think they're going to lower the cost a bunch of stuff. I think they're going to be a key part of defense for lots of companies because, like I said, it's a per, per, you can not be paying 20, 30 bucks per million tokens or whatever to do defense against people who have zero marginal cost for their tokens because they're running it on commodity hardware. But, like, we have to be with open eyes. The open-white models are going to be used to hack a ton of stuff by these guys.

So, like, there's nothing we can do about it. And, like, we should not control Americans' access to open-weight models because the bad guys are going to have them. So, like, there's nothing we can really do here except get ready. As we wrap up, I'm going to try and summarize, at least what I've taken away, and I want you to edit where I have it wrong or add nuance to it. But in terms of what is actually going on, that the misperception of the delta between, the risk and what is probably the bigger risk is that the risk of a human extinction event while grabbing a lot of headlines and a 28-year-old talking about it,

that may have been overblown, but what's underreported is these kind of border skirmishes with open weight models and very talented, innovative rogue actors who are harder to find because there's less human element using cheap and cheerful but powerful open weight models. that that that is where the real risk is. And I'm putting words in your mouth, but feel free to spit them out and edit them.

Yeah, from where I sit, AI is an incredible, gives individuals incredible powers, right? We're going to see incredible innovation from individuals. We're going to see all kinds of new startups that you can start with two or three people that can do the power, have the power of big companies, right? we're going to see all kinds of one-person companies. We're going to see all kinds of entrepreneurship.

But that also goes for bad guys. You're going to see one, two, three-person companies that now have the power of state actors from a hacking perspective. You're going to see people able to stand up what capabilities that used to only be kind of an intelligence agencies. And what you're going to see are small countries now have the power of big countries. Everybody knows this because of Ted Lasso, right? But, you know, in the Premier League, if you do poorly, you get relegated down. If you do really well, you get promoted up. What's happening is all of the hacking teams are getting promoted up to the next level.

So it used to be at the top. It was the United States and our allies, the Five Eyes, United States, Canada, UK, Australia, New Zealand. We are at the top. And then you've got like Israel, China, Russia. These are the big boys. And then maybe a couple other Western countries are up there, too. And then the next tier down was like, you know, Saudi Arabia, North Korea, some other folks who were, like, active but weren't in the top tier.

Well, they're, you know, Iran, they're now going to the top tier. The Iranians used to not be able to do super good, hardcore exploit development in the same way the Russians or the Chinese could, but now they can. Right. Now they can do things like attack the water grid. Now they can do things that used to have to be the Russian SVR or the, the Chinese MSS. And so, and now countries like India and Pakistan

who have an extremely active cyber capabilities against each other are now able to use AI to do that. And so, and individual little, either activist or more likely financially motivated hacking groups are going to have the capability of small state actors. And so that's what really frightens me as somebody who does cyber. And yes, I think there are bigger long-term risks, but I think those are being overstated.

And if people are going to state there's an extinction risk, I would like them to say, this is the specific sequence of events I'm afraid of, and here's how I'd like to deal with those risks. Because from my perspective, the extinction risks are all things for which, one, I do think we need to be careful about what we build from an AI perspective, but there are also things that generally we can deal with through other means outside of just the labs.

And then finally, you think the first, towards solving this problem at a minimum would just be a bilateral agreement between the Chinese and the Americans? Yeah. So I would like, I think it's great that we've got the CEOs of the major American companies agreeing. So I'd like to see the American companies work on self-regulatory rules of the road. Kent Walker of Google actually proposed this FINRA for AI idea.

I see that as probably the best option in the United States. you know, I just don't see the U.S. government as competent enough to do detailed rules. So the idea of the U.S. government sets, here's the level of risk we're willing to take in each of these areas. And then you have a self-regulatory group that says, okay, great, here's a technical interpretation of that. How's this look? And that's the back and forth is probably the best we can do. And so there's no reason to wait on that. I think that the U.S. labs need to be coming up and saying these are the rules we're going to follow. and then they should go take those at a track two level to the Chinese labs and say, why don't you guys do this too? I think this is, you know, you guys don't want bio weapons.

You guys don't want your models to be used for hacking and see if you can get the Chinese labs to agree to some of that. And then you get Xi and Trump to agree to make that and to put an international premature on that. Alex Damos is a cybersecurity expert, former chief security officer of META and Yahoo, and the current chief product officer at the AI security start. up corridor. He joins us from his office in San Francisco. Alex, I can't think of a better person to have on the pod during these very interesting times. Very much appreciate your time. Thanks, Scott. Thanks for having me. This episode was produced by Jennifer Sanchez, Laura Janir, and Asher Schwartz. Our video editor is

Bianca Rosario Ramirez. Camry Rique is our social producer. Drew Burroughs is our technical director and Catherine Dillon is our executive producer. Thanks for listening to the Propge pod from PropG Media.