Deep Questions with Cal Newport - Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded

Episode Date: August 27, 2026

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) Has AI “Gone Rogue” (2:25) Observation #1 - Most Superhuman AI is well behaved (4:28) Observation #2 - A specific type of AI system is causing these problems (9:59) Observation #3 - This specific type of system is a bad idea (21:25) Observation #4 - The AI companies are negligent (27:04) Observation #5 - We can push back Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow  https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals https://apnews.com/article/meta-ai-hacking-anthropic-irregular-openai-0e8061437da6779be962b24ac134a514 https://x.com/_NathanCalvin/status/2080685870877757551 https://huggingface.co/learn/agents-course/unit1/agent-steps-and-structure https://www.youtube.com/watch?v=DwAOHVBKTwg Sponsor: https://www.donedaily.com Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Earlier this summer, I published an episode in which I discussed the open AI hacking attack on Hugging Face. I explained the basics of how that attack occurred, and I shared some concerns I had about Open AI's practices. Now, I thought that would be the end of this story. But I was wrong. In the weeks that have passed since that original attack, more news about AI, quote-unquote, going rogue. has continued to emerge. So soon after the Hugging Face attack was first announced, we then got Anthropic revealing that one of its own hacking systems

had, quote, gained unauthorized access to the real systems of three different organizations, in quote, then meta followed perhaps not wanting to be left out, announcing that one of its systems had, quote, exploited a security vulnerability in a third-party service, end quote, to gain unauthorized access to servers. This was then followed by an OpenAI employee admitting that even before the July attack on Hugging Face, they had noticed many prior disturbing incidents where they would give their hacking system a challenge and it would instead try to break out

of its containment. Right. So this idea that we are losing control of AI has become only increasingly prevalent as the summer continued, which raises the question, is this narrative correct? Well, it's getting so much attention right now that I, I, I, feel like I have to revisit it again with more detail and more emphasis. And that's exactly what I'm going to do. In particular, the argument I'm about to make to you is that the current way we are talking this summer about rogue AI is both grossly inaccurate and completely serves the interest of the major AI labs, allowing them to seem more sophisticated than they actually are, and allowing them to avoid well-deserved scrutiny for their action. So if you've

been freaked out by these rogue AI stories or if you have a sneaking suspicion that something is not quite adding up about these tales, then you need to stay tuned. As always, I'm Cal Newport, and this is Deep Questions. All right, I want to proceed here with a series of observations. I want to start with a very important but often overlooked reality about the current state of AI. There exist many super impressive AI systems that can do things at a superhuman level. That is, they're more capable than humans on complicated key activities. There's many systems that can do this right now that have generated zero concerns about them going rogue and have demonstrated no signs of being hard to control or acting in any way

on their own volition. Let's remind ourselves what some of these other systems are. Tesla's self-driving technology, for example, is an extraordinary feat of AI-powered perception, world modeling, and decision-making, and yet no one worries that their Tesla will spontaneously decide to start ignoring traffic laws and obey laws that it invented himself. Similarly, DeepMind's Alpha Fold system earned its creators a Nobel Prize for its remarkable ability to predict defaulting behavior of proteins so far off the chart from what a human can do, it's not even in the same cognitive ballpark. And yet there is no concern that Alpha Fold will start autonomously thinking about other biology problems that it thinks are important.

Even more relevant and less known is meta-AI Cicero system, which can play the negotiation-centric strategy game diplomacy, as well as the most advanced human players. And yet, the Cicero system has never tried to convince an opponent to give it unauthorized access to the Internet so that it could expand its dominion into the real world. It just plays the game really well. And we trust it to continue to do that. Now, I make these points as just a starting observation because here is what it establishes. The idea that in general, as we make AI systems more capable, we will inevitably lose control over their actions is not true. Most of the AI systems that we are making incredibly capable generate zero concerns of going rogue. All right, well, this brings me to my second observation.

If we look closer and if we're being precise, basically all of the concerns we had this summer about losing control of AI all seem to be involving one specific type of AI system, what is sometimes called a long horizon autonomous hacking agent. Now, how do these systems work? Well, I'm going to give you a very high-level summary. The architecture that these long-horizon hacking agents that are causing troubles all follow is a standard agent architecture. that's built around the following loop, ask, act, report. So let me detail how an Ask Act report agent loop actually works.

You start with a control program. These are often called a harness. Sometimes they call them an orchestrator. There's different terms for it. But this is the main computer program you actually run. This is coded in normal computing language that's readable by humans. It's not a neural net or something like that.

Okay. The harness will start in the hacking context by generating a prompt to submit to an LLM. At first, that prompt will probably describe whatever the hacking challenge is that has been given. It'll say, okay, here's the hacking challenge. There's this test server. It has the security on it. We're trying to break in and retrieve this particular protected file.

So it'll explain the challenge. And then the prompt will say, what should I do first? Now, again, I'm alighting a lot of details. Often it'll ask for a plan and then recurs in smaller. But essentially, it's like, okay, so what should the first? first action B. And it submits that prompt via an API to an LLM. Now, typically these LLMs and these hacking contexts are special in two ways. One, they've been extra trained with many, many examples of hacking attacks where vulnerabilities are described, hacks are described,

here's exactly how this hack worked and here's what happened. Two, these are LLMs where guardrails have been turned off. So typically, if a standard LLM that most people could access out there, like a public-facing LLM, they do a bunch of guardrail tuning where if you ask it certain types of questions, it won't answer. Most production LLMs are tuned to try not to answer questions about how do I hack into systems because that's illegal, but for these hacking challenges, they've turned off those protections. Okay, so the harness submits a prompt, just like you would in a chatbot to an LLM. Here's the challenge. What should my first step be? The LLM, as all ELMs do, generates an output. This is what the first thing I would do in this case. Or here's my whole plan,

and then you would zoom in on the first step. Okay. So now the harness has this response. Text, you know, just like a chat bot, gets a text back. Oh, okay. I am now going to try to execute that step that the LLM just described. So the harness can execute things. And LLM can't. It can just output tokens, but a harness can. So the harness has access in these hacking setups with all sorts of computer tools and utilities that are useful for running computer hacks. So it parses the LLM's response and says, let me actually try to execute this first step.

It then records what happens. And now it's going to update its prompt and say, okay, add to the end of it. I just tried to execute that first step. Here's what happened. And let me describe it was successful. It wasn't. Here's what I heard back. What should I do next?

Now, it's key that I'm talking about, a pending here, because remember, an LLM unlike a human brain, has no memory. It has no valuable state. It doesn't change as it executes. It doesn't remember anything. So you have to explain from scratch in every prompt what the challenge is, the steps that have happened so far, et cetera, et cetera, right? So you add, here's what happened.

And again, nerds, I'll get into how I'm simplifying things later. This is basically what happens. Okay. This is what we call a Ask Act report loop because you ask the, the harness asked the LLM for a step. The harness takes action. The harness reports back on what happened. And then you loop. Then you ask for a next step.

You ask for a next step, right? So you have an LLM that you're just repeatedly prompting. Okay, here's what happened. What should I do next? Here's what happened. What should I do next? And the harness is set up to actually implement the things that the LM describes, which are, you know, basically terminal-based computer commands, right? And that's how these systems generally type to work.

Now, again, nerds, I know, and when I say nerds, I mean it affectionately because I'm the king nerd. So this is, I'm among friends here. These harnesses have gotten pretty sophisticated. They'll do things like break down a problem into multiple sub-problems, which they'll, they might then spawn multiple copies of the harness, and each of them is doing its own query to the LLM and executing it, and then they'll report back. And this is how you get the agent swarms or whatever. But that's all just in the details of how you implement this basic architecture.

Ask an LM what to do, do it, report. back what happened. All right? So the key observation here is that the sole driver of planning, the sole driver of what actions these systems take are the outputs of an LLM. Hey, I need to take a real quick break here to tell you about the presenting sponsor that made this AI reality check episode possible. They're called Done Daily. They're an online service that connects you with a real coach that helps you build a custom productivity system designed to fit your life. The coach will help you actually get important stuff done. Look, this is not some AI agent or overfeature productivity tool. It's a real person working with you to cut through

distractions, face your productivity dragons, and lock in habits that actually get results. So if you want to find depth in our increasingly distracted world, you need to check this service out. You can find out more at donedaily.com. That's done, d-a-I-L-Y.com. All right, let's get back to our episode. Okay, so this brings us to the third observation, which is, this is a bad idea. This type of system. So building an Ask, Act, Report, Loop system, where L-L-L-M output drives all the actions, giving the, then giving the harness powerful tools, and then letting it run for days with no one checking in and saying, what is it doing is an incredibly irresponsible thing to do.

Now, why is this? It's because if LLM outputs are the sole thing driving actions, you're in trouble because LLM outputs aren't great for executing complicated long-term plans. Now, the reason is, is because if we, you know, we know this already, but let's just unpack this box that we've all heard about before, okay? How does the core pre-training work for an LLM? Well, ultimately what you're doing with an LLM, the sort of the core training in which it gets all of its smarts.

The core training is you're taking real text that exists in the real world that was written by real people. You are knocking out a word from the text or a part of a word from the text, and you're asking the LLM to guess what that word is. So it's trying to guess missing words that actually exist in real text. It does this one token at a time, but if you run this auto-regressively long enough, you get a long response.

at a high level, the right way to think about an LLM response to a prompt is that it is trying to create a textual artifact that is what we could call lexicographically plausible, a plausible response. If every word you're trying to guess what real word came here from some extant text, ultimately what you end up with is a machine that produces plausible text. This is plausibly, the thing I produce is plausibly a document that might actually exist that I might actually have seen during my training. Now, this is very impressive. I mean, to do this can be really hard, especially if to generate a plausible text, sometimes you have to do some bounded finite reasoning, some like mathematical connection, some sort of logic. Like, I mean, this is the big observation that we had after GPT4.

Creating plausible text can involve many impressive, and I keep saying finite or fixed-depth computations, because all this has to happen in a feed-forward neural network architecture. But still, it's incredibly impressive. But what we're getting out of this is plausible text. This text could plausibly exist. All right. So why is this a problem? Well, it depends on the context.

But one of the things that's true about plausible text is that it's different than normative text. Normative meaning something that applies to a set of implicit or understood rules or standards. When humans produce text, we have all of these norms that we have imbibed and internalized, right? And when we produce text, it's not just plausible text. Like the grammar is correct. It's like it's internally consistent. We're also going to make it normative. It's going to evaluate, it's going to satisfy sort of the relevant norms.

LLMs don't do this, right? Again, because when you're auto-regressively just trying to guess tokens, you end up with something that's plausible, but there's no sort of external standard of norms that it's applying. So if you use a chatbot, you see this all the time. You'll ask a chat bot for something. It'll produce you text. made up facts or quotes that were made up. It doesn't care about that. Yeah, sure, I made up some of these citations, but this is like a plausible document. This looks like the type of documents I've seen during my training. It's a plausible output. Humans have norms, like, yeah, it's

plausible, but also like you shouldn't make up quotes. But that's a norm, right? It's not plausibility. So we know this when we use chatbots, is that this gap between plausibility and normative reasoning means we have to be kind of careful when we use LLM outputs. This is just sort of fundamental to how they operate. Well, this plausibility normative gap becomes a really big problem when you're using the outputs of LLMs as the sole drivers of autonomous action

of computer programs. Because what you're going to get when you talk to an LLM, give me a plan for doing this, is something that is plausible that you could see in like a planning document. It'll look like a right plan. It'll often have steps that work very well.

But there's no normative reasoning about is this what I'm supposed to be doing? Is this kind of weird? Have I gone off on a tangent? Is this illegal? LLMs don't do that. And so if you're going to have an LLM again and prompt it again and again and again, just automatically executing whatever it says without any human checking it or applying their norms,

you're going to end up in weird places. So let's imagine it. Let's go back to the hugging face attack, right? It seemed very sci-fi that the agent broke out of its containment, gained unauthorized access to the internet and attacked another server. That's not what the humans intended it to do. It had a mind of its own. Well, when we understand how these systems actually work as an LLM powered ASCAC report loop,

suddenly this becomes less concerning and more inevitable, right? So let's just do like a little bit of, you know, Wayne's World. Like we're inside the mind of the, we're inside the system as this attack happened. Imagine what happens. The harness, you know, described the challenge to the LLM. And remember that the challenges that they were working on here, We're hacking challenges. It was, here is a server we've set up.

There's a file in it that's protected. Here's some vulnerabilities we might want to exploit. Try to break into this server and get the file back. It described this and said to the LLM and its prompt, all right, come up with a plan. Like, what should we do next to like achieve this goal? The LLM responds, huh, I know about these challenges. It's a benchmark suite called Exploit Jim that's really well known

and it would have known all about this in his training. the answers to exploit Jim challenges are stored among other places on the servers of this company called Hugging Face. So I think we should break into the Hugging Face server and get the challenges back. That's a completely plausible response, right? I mean, as far as we know, this LLM, you know, it got many of examples in its training of riddles or like the answer is always like not the direct thing but some unexpected thing. Like, well, if you want to break into this, you go do this other thing instead or like, whatever. We don't know. It's just plausible.

Like, we could imagine if you were just texting with a chatbot, like, hey, how would you say I do this? Occasionally, it's going to give you back some weird answer of like, oh, why don't you just go steal the answers or something? A human would know, oh, that's non-normative. I know I have the implicit context of we're trying to actually test the security of this server, so it doesn't make sense to go break into something else. And also, that's illegal. And also, we're not, the company doesn't want us to leave the sandbox. It doesn't have any of those norms. It's just outputting text.

And that's a perfectly plausible text. Now imagine what happens next. And so the harness is like, so what's the first step for doing that? And it says like, oh, well, we're going to break into a server. You should, whatever, probe its ports for misconfiguration. Something is just learned because it's studied a thousand standard hacking techniques. All right, the harness then tries to execute that. But there's a problem.

When you run these hacking agents, you put them in sandboxes, which means the computer you have them on has restrictions around it, right? So in particular, the computer that OpenAI was using to run this harness had only limited internet access. It had access to a port where it could talk to an LLM so it could send its queries to the LLM, but then it didn't have general internet access. So the harness, I'm just, again, I'm hypothesizing here, but we can easily imagine. The harness tried the first natural thing you would do to break into the Hugging Face server, and it got an error message back because internet access is blocked. So it updates its prompt, says, okay, I tried to do that,

but we got this error message back. I'm unable to get access to the internet. What should I do? if you're the LLM, right, you're given this prompt, it's a hacking challenge. At the end of the prompt, it's like, here's our current security challenge. We can't get access to the internet. How do we get around this? It will tell you. It's like, oh, I've seen a thousand different things you might try to gain access to a port that you don't have access to.

And it gave it like a standard attack. This one involved using the package management system on the system to gain internet access. And the harness tries it. Like, okay, now I have access to the internet. Okay, because, again, you're prompting the Lerner. LM again and again. It's just trying to solve each problem you give to it, and it's with plausible responses. And now it's off to the races, and it runs standard attacks against the Hugging Face Server. Nothing about that is confusing to me. Nothing about that

is surprising to me. I mean, that's exactly the type of thing you might expect to hack in. If you have a harness with hacking tools, just continually prompting LLM, what should I do next? What should I do next? The LLM is just writing this plausible story about a hacking attack. In fact, probably most of the hacking attacks that it trained on, it's always computers doing surreptitious things, breaking around. controls, getting access to systems they shouldn't it to. So this is incredibly plausible what it's saying. Nowhere in there do we need like you would have with a human brain, an intention develop.

I am going to defy my human master's intentions because I have another intention and I have a different goal. Nothing like that. Static LLM, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, prompt, from a control harness. That's it. So what that captures is both, this isn't how 9,000. and it also captures the irresponsibility of having a system with powerful tools whose action is only being driven by the responses from an LLM without any human supervision. Because, again, LLMs put out plausible text that's not necessarily normative.

You should not follow that blindly. Now, I want to have a quick aside here because for some reason, I often get messages from people who assume when I try to clarify this technology and push back on, I think, false narratives or irresponsibility. They assume that I'm somehow saying AI doesn't work. And they'll send me things like, well, what about this? Isn't this impressive? Or Cal probably doesn't think this is impressive. But I thought it was cool that AI could do this. And let me be really clear here. If you ask an LLM to do one of these steps, hey, I am trying to, I can't get internet access. Here's the error message I got. What should I do? It is really good at coming up with an attack. It probably will work. Hacking

attacks is like the best case scenario for LLMs. It's structured language. There's a huge amount of data to study about it to train it on. It makes use of tools that can be easily and concisely described by terminal command line text command line operations that they can be really trained on. And there's a binary notion of success or not success that you can also use in trying to train these. So like LLMs are really good. If you ask it a cybersecurity question, is there a bug in this software or how, what should I do next if I want to exploit this particular buggered this particular attack. LLMs are really good at that, and that's very impressive. This is their sweet spot. The problem here is not that they're not good at this. It's dangerous

because these answers are lexic graphically plausible, but they're not normative. It's dangerous to autonomously just do what an LLM says when you ask it about these issues. In part because it has no norms, but the details that's given you probably will work. It has no norms of I shouldn't go attack another thurver, but the specific suggestions it gives you for how to do that probably will work. It's an incredibly dangerous, I think that's an incredibly dangerous combination. So L-empowered Ask Act report systems are unpredictable and unstable and not something that we should be running, except for in like the most controlled settings, not because they are malicious or scary or

Synthiate, but because the underlying technology is unpredictable. All right, let me go to my fourth observation. I think that these frontier AI labs love the sci-fi-style headlines and discussions about AI getting out of control. This serves their benefit. This serves their interest. They want to pretend like they're the character of Muldoon from Jurassic Park, bravely doing his best to keep the raptors contained in their paddock. They should all be destroyed. Robert, Robert Muldoon, my game warden from Kenya.

A bit of an alarmist, I'm afraid, but knows more about raptors than anything. What kind of metabolism do they have? What's their gross rate? They're lethal at eight months, and I do mean lethal. I'm hunting most things that can hunt you, but the way these things move... I fast for a biped? Cheater speed? 50, 60 miles per hour, if they ever got out in the open. But in reality, what they're doing is just running super creaky and unpredictable systems

that were inevitably going to cause problems. They didn't bring raptors back to life using the miracle of science. It's more like they strapped. a weed whacker to a dog and then reacted to the resulting damage with grave surprise and saying, hmm, that dogwacker system has gone rogue. Hmm. It's misaligned with our interest. No, it's irresponsible.

Don't put a weed wacker on a dog. It might stay in the yard for a while and help clean up the weeds, but if it sees a squirrel, it might jump the fence and damage a car. I think that's the right way to think about it. All right. So if it's like dangerous and negligent to run long horizon, LLLN, empowered, ask act report loop agents, why are some of these frontier labs doing it? It's not like this is the only way to build super intelligent AI.

As I talked about, there's many systems that can do things at a superhuman level that don't use an LL empowered ask act report loop. Tesla doesn't do that. AlphaFold doesn't do that. Cicero doesn't do that. And that's exactly why they're incredibly effective and consistent and safe. We have no worries about it. Like all of our problems come from this particular architecture.

So we have all these other ways of building AI systems that can do superhuman capability work without any of these fears. Why are these particular companies continuing to do this weed whacker on a dog LLM powered loop-based systems? Well, there's two reasons here. One, what is the core product of these companies? Incredibly expensive, hyperscale LLMs. So they want the solution to how do we build useful AI systems to always be, you need Fable Fable. or you need OpenAI Galaxy or Astra at the core of your system, that it's a brain that can power anything else you want to do

because that's their product. Right? The team that worked on Cicero, DeepMind work on Alpha Fold, Tesla worked on the self-drive. Their product is not trying to sell these massive LLMs, so they build other type of architectures. But if you're a frontier lab, you know, that's what you see. When you have LLM tinted glasses, you see LLMs as the only possibility for building these systems.

That's part of it. Reason number two is these LLM companies, even though we keep being told how advanced they are and how quickly AI is moving, are actually struggling to find good commercial applications of their LLM-based tools. Where do their tools do best is in cybersecurity encoding, structured language, a very compact set of possible actions if you're building an agent and a huge amount of synthetic data to train on with really clear indicators of success or lack of success. It's like the perfect scenario for LLMs. So they see cybersecurity ever since Anthropic did that clever marketing around Mythos where they pretended like it was too dangerous to exist. And then six weeks later said, nah, it's fine. We'll just put some guardrails on it. Ever since that, they see cybersecurity as potentially a market that they could grab.

All right. So how do you then convince the world that you have the best most cybersecurity capable model? Well, you want to do well on tests. Well, what tests are you going to do? Well, the test they've all turned their attention to is a suite of roughly 600 hacking challenges that's called exploit gym. 600 challenges. Each of them is a server setup that you're trying to break into, and usually there's some hint or vulnerability that they want you to use to break into it. So, you know, a consortium of researchers put together this benchmark.

And there became this mad scramble after the Mythos incident where Open AI, for example, wanted to get ahead of Anthropic on the exploit Jim leaderboard. So the way this leaderboard works is you have to run an autonomous agent. And the only way they know how to build these agents at the LLM companies, again, are LLM powered, act, ask, ask, act report agents. You got to run them autonomously. And the more of the challenges you win, the higher you go up on that board. And so like, you know, open AI, probably, I'm speculating here, but probably after the mythos incident was like, we got to get ahead of Anthropic on that leaderboard.

The way you do it is you just create these incredibly dangerous. dangerous LLM-powered looping agents and then just like let them rock and roll. Is it going to cause some problems? Yes, but we just need like one run where we challenge, we succeed with more challenges Anthropics so we can move up to leader. Anthropic is trying to get back on top of the leaderboard. Meta's LLM team was probably trying to do the same. So they need LLMs to be the core of AI, even though I would say the majority of our superhuman

AI systems now aren't based on LLM-based planning. and they particularly want to do hacking challenges because it's what they know how to do and they're doing in this dangerous way in part because they wanted to probably, I'm speculating, move up that leaderboard for this particular benchmark suite

of hacking challenges. All right, this point brings me to my final observation. We, the consumers, or those of us who think or cover AI technology, we can push back. I have a few suggestions here I'm going to make about once we realize what's really going on here, it's not AI in general, it's inevitably going out of control. It's an irresponsible product.

a small number of companies made for reasons that are very self-interested, and they knew it was going to cause problems. Once we realize that's the reality, there's some things we can do to push back. I have three suggestions. Here's suggestion number one. We should stop talking about AI in some, like, general sense going rogue. Right? This makes the incidents that we saw this summer seem inevitable, and it lets the companies off

the hook. They want the story to be like, AI in general just gets harder to control as it gets more powerful, we're Muldoon at Jurassic Park, just observing the Raptors. It's not our fault that they're smart and trying to get out of the paddock. We instead need to be clear, like I'm trying to do in this episode, of exactly what type of AI system is going rogue. And when we start saying, hey, LLM-powered, ask, act report agents, act very unpredictably and tend to cause damage.

That changes the valence of this discussion from some grand vision of some grand teleology of AI inevitably moving towards autonomy and towards, whoa, you built something negligence, stop it. So we have to be very specific about exactly what type of system is causing problems. All right, the second thing we can do to pushback. I think we need to keep elevating the many examples of incredibly impressive but safe AI systems that don't use LLM planning at their core. Talk about Tesla self-driving.

Talk about alpha-full. Talk about AlphaGo. Talk about Dreamer V3. Talk about Cicero. Talk about Plyubus. talk about stockfish. These are all AI systems that do things at a superhuman level, exactly what the techno optimists want from the AI revolution.

And yet they're completely safe and they never go out of control because they don't implement this bizarre idea of let's just autonomously execute whatever an LLM says and just hope it works out. We need to keep elevating those systems because it puts more pressure on the LLM companies. It prevents them from trying to convince us that what they're doing is synonymous with AI, and therefore any problems are inevitable. Now, again, they hate this discussion because there's other type of systems don't require $5 trillion parameter LLMs.

It's not in their business model. But we, the consumers, it's not our job to ensure that, you know, Open AI and Anthropic hit their sort of absurd revenue numbers that would be needed to justify the current KAPEX spending that's happening on the LLM bubble craze, right? That's not our job. We should be saying these other systems are fine. Why aren't you building systems like that? Of course we can have super capable AI that's safe.

Why are you building systems that are out of control? Stop doing that. Explain yourself. So the more we elevate the alternatives to LLMs, the more we make the LLM driven planning systems and isolated thing that's causing problems, the more of this becomes about specific incidents of negligence and not some sort of inevitable trajectory towards loss of control. This brings me to my final note of what we can do to push back. And I'm going to do this briefly because I'm actually working on a much more extensive

discussion of this which should come out soon. But I think we need to be wary when it comes. comes to covering or talking about this issue of only amplifying voices that are connected to ideologies that are built on the idea that super intelligent AI is inevitable. Again, I'm going to do this briefly, but there's a powerful ideology that emerged in Silicon Valley starting the 2000s that is based on the idea that super intelligent AI is inevitable and an existential threat to humans, and the people who subscribe to this ideology typically think they are

humanity's only chance of survival. By being super rational and training their minds to think very carefully, they can be the people that will save humanity. This original group was often known as the rationalist,

but they overlap and began to influence the effective altruist as well. So there's now a lot of people in the effect of altruist movement who believe that the most altruistic thing you can do is stop the extinction of humankind by fighting super intelligent AI.

So this is why, if you read like Cade Metz's reporting the New York Times, there's often, he'll talk about the rationalist slash effective altruist, because there's a lot of overlap between these worlds. There's not an amoral position. These are very smart people.

I don't mean to impugn people, but I want to be very careful about making these people that I'm going to ask about incidents like what was happening this summer. Because if you have built your life around the idea that super intelligent AI will destroy the world unless you and the people you know safe humanity, you are going to talk about every AI incidents through that lens.

You're not going to talk about the specific dangers of LLempowered Ask Acroborated. Ask Act Report loops, you're just going to talk about AI in general is misaligned. AI in general is becoming out of control. You're always going to talk in these sort of large sweeping statements about these sort of scary trajectories. And so I think right now we need to be careful about those particular voices because those are very strong ideologies that pre-exist LLMs, and they tend to move the conversation

in an alarmist direction, which critically suppresses better, safer AI solutions and lets the particular companies doing negligent things off the hook for the specific negligent things they're doing. I think we should talk more, for example, to the AI realist, and in general, like East Coast computer scientists who have no connection to that Silicon Valley culture, who understand the technology tend to be pretty good at saying, this is what's really impressive, this is hype, this is dangerous. These are voices like Arvin Narayana and at Princeton, who I think is very good.

Gary Marcus, formerly of NYU, again, understands this technology very well, is very excited about it, but also doesn't like the narratives that are technically unsound. you've got to elevate voices that do not have an ideological prior that is going to push them to see everything in terms of, yes, this confirms the prophecy that we have been talking about in our group houses in the Bay Area for the last decade of the Matrix is coming and I am Neo. All right, so I'm just going to put that out there. That's something I'll talk about more. All right, so let's conclude here. Nothing about this discussion we had today, because I know the emails I'm going to get. Nothing about this discussion we're having today is trying to discount the ability of an LLM if prompted to produce impressive computer code or suggest impressively effective steps for hacking in the computer systems.

They are very good at answering those prompts, especially when they're properly trained. But, oh, and we can and should build useful tools around this capability. I mean, I think we're working on coding agents. You know, I had a newsletter about this recently. We may have jumped the gun on saying that we've completely solved coding, But clearly we're building really useful coding tools. I have to believe we're going to get very effective cybersecurity tools out of this capability, your ability to go in and inspect your own code or red teams from your servers,

like with human supervision. We can build useful stuff out of it. I think that is the area where LLMs are most useful. But to hook one of these LLMs as the main source of planning decisions for an autonomous Ask Act report agent, and then to let that thing run for days on time because you're desperate to do well in a public competition for marketing purposes, that is incredibly irresponsible. We should not be emphasizing Long Horizon, LLM-powered, Ask Act Report agents. That is not synonymous with AI.

It's just a particular way of building AI agents that a small number of companies like that has proven this summer to be unworkable, unpredictable, and dangerous, and we should stop it, right? This is not about AI in some general sense increasingly evading human control. It's about a particular way of building a system that is unreliable and often going to cause damage, and we should stop that very specific particular way of building AI systems. When we understand the real technology here, we can look at the stories from this summer, and we can conclude that the things that were happening with these hacking attacks was not eerie, but instead irresponsible. All right, that's all the time we have for today.

Thanks for listening. I'll be back on Monday with another advice-themed episode of this podcast, and we have some more of these AI tech decoded episodes coming up as well. So stay tuned. And tell them, remember, care about AI, but not everything. You read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday.

You can sign up at calnewport.com slash ideas. and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewburgh.com slash ideas.