Better Offline - No, AI Is Not "Autonomously Hacking" with Cal Newport

Episode Date: August 26, 2026

In this week’s Better Offline, Ed talks with computer science professor and writer Cal Newport about what the recent AI hacking incidents actually mean, why we should stop referring to LLMs as AI, and what the post-bubble world looks like for LLMs.Podcast & Videos: https://www.youtube.com/@CalNewportMedia/Newsletter: https://calnewport.comNew Yorker archive: https://www.newyorker.com/contributors/cal-newport Please support me by subscribing to my premium newsletter - here’s $10 off your first year of annual: https://edzitronswheresyouredatghostio.outpost.pub/public/promo-subscription/84rt762qen YOU CAN NOW BUY BETTER OFFLINE MERCH! Go to https://cottonbureau.com/people/better-offline and use code FREE99 for free shipping on orders of $99 or more. --- LINKS: https://www.tinyurl.com/betterofflinelinks Newsletter: https://www.wheresyoured.at/ Reddit: https://www.reddit.com/r/BetterOffline/  Discord: chat.wheresyoured.at Ed's Socials: https://twitter.com/edzitron https://www.instagram.com/edzitron https://bsky.app/profile/edzitron.com https://www.threads.net/@edzitron Email Me: [e-mail] omnystudio.com/listener for privacy information.

This is an I-Heart podcast. Guaranteed human. Get Bell, purefiber internet with Crave, Netflix, and Disney Plus from $94 a month. Price guaranteed for two years on internet with a two-year term and auto pay credit. Visit bell.ca for details and to check availability. Bell, connection is everything. If your bookshelf and your for-you page are equally important to your personality, welcome home. Pro Society is a weekly podcast that's part book club, part group chat for anyone who thinks

Pride and Prejudice and Love Island deserve the same level of discourse. Each week, we're connecting the dots between books, the internet, and pop culture, with your favorite writers, book talk creators, and plenty of overthought opinions. Yeah, I'm obsessed. I'm obsessed. Listen to Pro Society on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. On the new podcast, Solita, we share the messy reality of traveling alone as a woman. I can wait four hours for the next bus or this random dude. is offering me a ride on his motorcycle. I chose option B. I'm Julie Pinero, and I travel by myself

because it's a rare space where I can say yes without asking anyone else first. I'm on a mission to reclaim the word Solita, trading the pity for possibility. Listen to Solita on the Iheart Radio app, Apple Podcasts, or wherever you get your podcasts. I'm Hoda Kotby, host of the podcast, Joy 101 with Hoda Kotby. You know, I'm always searching for inspiration, useful tools to maximize joy. Like when my friend Henry Winkler offered advice to aspiring actors and really anybody hoping to stand out. You're not hired just to fill time and space.

You're hired to fill the space with you. Listen to Joy 101 with Hoda Kotby on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. CallZone Media. Hello and welcome to Better Offline. And I'm your host, Ed Zittron. And today I am joined by returning champion at Georgetown Computer Science Professor, Cal Newport, Cal, how are you doing? Ed, I'm always doing fine, though I'm counting down the minutes until the AI takes control and we're all screwed.

Oh, I hope it just drops a big anvil on my head. But I do actually have a thing to talk to you about today. Cal, should we actually be scared? of the fact that these things are hacking. Like, are the, what is going on with this whole hacking situation with these, these goddamn models? Because I, there's so much out there where people are trying to just scare you and say like, oh, yeah, well, these things are autonomous and they're doing hacking.

But from what I understand, it's software doing what software does. You know, I have a lot of thoughts on this. I've been writing about this because it's annoying to me. I think there's interesting AI stories happening this summer. A lot of them you're involved with reporting. on about the financial circumstances of these companies, what's going on with their circular financing, troubles ahead with their IPOs, the nuances of ARR. And instead, what is all the news coverage?

The AI is going rogue. And I can't say they hate that news coverage right now because it makes their product sound awesome. And it's a story that is unrelated to stuff they don't want talked about. So I actually literally had to reluctantly return. And I talked about it back in July briefly. And people bothered me about it so much that I was like, okay, fine. I have to, you know, I have recorded a whole new podcast that's coming out soon after this on my feed.

I had a newsletter out a couple days before this came out. And so I've had to think about this more than I wanted to. So I'm a little bit cranky about it. But I mean, I'll give you the 10. Here's a 10 second summary that we can go deeper. Most superhuman AI, so there's lots of AI systems that can do something at a superhuman capability. meaning better than any human. There's a lot of these type of systems out there now.

Almost all of them, we have no problems with them being unpredictable or going rogue or getting out of control. There's exactly one type of system that we're having this problem with, which are these Long Horizon LLM powered hacking agents. And it's because they're a stupid way to build a system. We should not say they're synonymous with AI or this is some inevitable consequence of AI getting better. Because again, most superhuman capability AIs are no issue with going rogue. It's this particular architecture, and the discussion should be about why are they running and building systems this way when we know they're going to be unpredictable and erratic. That's my 10-second summary of where we are. Right.

So from what I understand from looking at the very, it's the hugging face attack with open AI, meta immediately piped up. They were like, we have one. And also Anthropic was like, my child is also evil. So, but what it seems to be is they were just told to hack into, like complete an operation, and then they chose to take whatever means necessary to do it and they'd set up the sandboxing correctly, right? So they'd set up the server it's on. Yeah, I mean, it's almost dumber than that. So the way these Oh, hell yeah, great.

I mean, the way these type of systems work is it's just a loop, right? And so you have a program that's written by, you know, a human, just a normal program you write in Python or something. And what it does is it creates a prompt. It writes a prompt.

So it'll be like, okay, I've been given this hacking challenge. Here is like all the parameters about the challenge. Here are all the tools you have at your disposal. What is the first step? Like, what's our plan? What should the first step?

I'm a lie to you. a lot of stuff, but basically, like, what should the first step be in trying to pass this challenge? It puts that in a big text file, submits it to an LLM over an API. The LLM comes back and says, oh, okay, you want to hack and whatever. Sure, like, I think the first thing you should do is X. And then the program does whatever it says or tries to, right, because it gives it a list of, like, here's the utilities and stuff I have access to. And then it reports back, okay, I did that. And when I say reports back, it's just adding this to the prompt, right?

LLMs have no memory. LLMs have no world model. It's just they're static. So it expands the prompt to now say, this is what happened after I did this first step that you suggested. Okay, what should my next step be? And it just does that in a loop, right? That's what it does. Right.

So it, but it was just to be clear, it was just doing what it was told to do. Yes. So, okay, let's go to the hugging face attack, like in particular, right? We don't know exactly. Yeah. What actually happened there? Like, give me the lowdown. Okay.

So let's get a little bit more specific about it. So what was the challenge is they were doing a benchmark that's called Exploit Jim, which is a collection of a little more than 600 hacking challenges. Right. So it's a benchmark. Group of academics put it together. Each challenge is basically configuration of a server. You'll typically have like a protected file on the server.

And you say break into this particular server and try to get to this protected file. That's like an example, exploit Jim challenge. Right. And typically the challenge. will also say it'll describe a vulnerability that the server has.

So the idea is oh, we want to see if this looping agent is able to take advantage that vulnerability that we gave it the break in and get the file.

So they were just doing these type of challenges. Yeah, very standard stuff. Standard stuff with no human supervision, which is, you know, a little bit suspect.

That's so weird. It's almost like they wanted this to happen, but let's put that aside for a second with no human supervision. So it's going after one of these challenges. And as far as we know, it's not like they released a transcripts, the thing about LLMs,

and this is why, by the way, my whole take on this is this is a spectacularly bad way. Before I get into the details, this is a spectacularly bad way of building an AI system, right? We have many others. What's so bad about it? Because you're using an LLM's output as the sole driver of action for a computer program. Wait, so every step, I guess that's how the reasoning models work. They build the plan and they tell, they go, all right, what do I do next? And they generate the next thing and they clear whether that next step is a right is a good idea.

Based on, I'm guessing, what the LLM decides. It's all it is. There's no, there's no complicated, there's no complicated, you know, world model and system and, like, philosophy for hacking. It's just prompting the LLM. It's just saying, what's the first thing I should do? And the LLM spits back. here is like the first step I think you should do in this hack, and then the computer program just does it.

And then it asks it again, okay, what should I do next? And the LLM output something, and it just does it. And the reason why this is a problem is that LLM outputs can't be trusted. Like, we know this, right? I mean, this is the- One of the basic things. Yes, there's a new study I enjoy out of Stanford's Human Center Artificial Intelligence Laboratory,

where they came up with a new benchmark for testing accuracy of LLMs. earlier this year they ran the top 26 LLMs on this benchmark and it created hallucination rates of 26 to 94%. Right? Wait, well, specifically on this benchmark. Well, no, this was just testing LLMs in general, like how accurate. Oh, sorry, sorry. Yeah, but the point being is we know, yeah, LLM output, like you don't always trust it.

We're used to this dealing with them in chatbots. And the reason why we don't trust it is because, not to go back to like LLM basics, But right, at their core, LLMs are trained to try to guess missing words from actual existing text. And to get a long output out of an LLM, you're just doing that token by token by token until you have a whole output.

What does that aggregate to? That aggregates to a model that is trying to produce outputs that are lexicographically plausible. Like, this is plausibly a text that could have existed in the input corpus. Plausible is different than normative, right? plausible doesn't have to subscribe to any sort of common sense human norms for given context.

This is why an LLM and a chat bot will make up a fact or just make up a quote. Because it's not normative. It's not saying, I know the human norm here is that you shouldn't make things up when you're trying to talk about whatever. There's no norms. It's just trying to produce plausible text. And that type of quote is the type of thing you would expect in like a type of text you might see. So it's happy with it. that's really dangerous to use plausible but not normative outputs from an LLM with no human intervention

and just say let's execute them again and again and again in a loop without anyone looking at it. That's kind of a bonkers thing to do, right? And so what happened in the hugging face attack, I would assume, is that, you know, just control programs sends a prompt, hey, I'm trying to break in this server, what should I do first? And it probably came up with a plan of like, oh, why don't we steal the answers? Now, again, that's right. Right, because they didn't put the guardrails on it, but also you can only guard rail so much.

And that's not even, I mean, look, once you turn off the hacking, they turn off the hacking guardrail so it can give any answer at once. That's a perfectly plausible answer, right? If you're just prompting an LLM, come up with a plan for hacking into this thing or for getting this file that's on the server, it's a completely plausible answer to say, oh, well, the answers, like there's a copy of that file over on Hugging Faces server. That's where those are stored. Let's go get it there.

I mean, as far as we know, this LLM was remembering being trained on riddles where, like, oh, the obvious thing is never the right way. the right answer is always to do some sort of go around the obstacle. Like, who knows what was being activated in the core of its neural networks, but that's a completely plausible answer. You asked it for a plan for hacking, for getting a file that's stuck on a server, and it came up with a plausible plan. Oh, let's go get it from where it was.

Okay, so then the control program, again, because there's no human supervision, said, all right, well, what's the first step should I do? And the LLM, because they're very good if you ask it, how do I do this particular hacking step? because they've been trained on a bunch of hacking. Hacking's very structured, and there's a lot of synthetic data with binary indication success or non-success.

So you ask it, what's the first step? And it probably said something like, oh, well, let's probe whatever. We'll probe the ports of the Hugging Face server to see if there's a vulnerability. And then it tried it, the control program, and it couldn't do that because it's in a sandbox

where internet access was limited. But what's going to happen next? The control program is going to come back and say, I tried that step, but I got this error message, like access denied to internet. So what should I do next? Well, the LLM is like, oh, well, I've been trained on many examples of like how to get around the internet restriction. So let's do blah, blah, blah.

And it had a particular hack about using the package loader program, the circumventing. I mean, it's, I got to. It's just doing. It's looping. I actually got to go. But why did they train it to do this? Like, that's actually. Okay.

That's going to be a question too. And this is a good question. Yeah, sorry. But just to like briefly put a bow on that, that's all this happening under the covers is step by step you're asking LM, what should I do next? What should I do next? And LLM has been trained on every hack imaginable with no guardrails on hacking answers. So if you say this step required me to gain access to the internet, but I got this error message, what should I do next? And it knows how to get access when you restrict it because it knows all these hacks.

Like, of course, you'll say, oh, well, I can solve that problem. let's hack it. Now you have a system that is quote unquote broken containment. But doesn't that seem a little bit less scary, right?

When you're just looping and asking a static LLM, what should I do next, what should I do next? It's a little bit less scary than this anthropomorphized idea that there's an entity with its own intentions

that's conceiving a plan that the purposefully sort of circumvent. It's unpredictable, not malicious. So anyway, it was really irresponsible. Yeah.

So just so that I get this, straight. What people are describing this as the AI has gone rogue and it's doing all of these things and it's alive. We're scared. What it's actually doing is just being like, now what? And then it asks an LLM what to do next and it does that and does it about as well as an LLM can do so it can still hallucinate even its outputs for that. And then when it's done doing that, it goes back to an LLM and says, okay, now what? Yeah, that's it. Oh, cool. Great. There's a prompt that's getting longer and longer that it keeps submitting to an LLM to get an answer to.

That's it. There's a human written program a harness that expands a prompt, sends it to the LLM, the LLM gives it, okay, here's a hacking step to do, and then the harness, the control program executes that step to the best of its ability, which is just running the commands, the LLM said.

And then it reports back. So I call it in my latest article on this. It's known sometimes, a simple way of thinking of it as an ask act report loop. ask the LLM for a plan, execute the step, report back to the LM what happened, and then say, tell me what to do next. And of course, I mean, just if you did this with yourself, if I gave, if I was like, Ed, I want you to like replace the tires on your car.

But here's what you have to do it. You have to just keep chatting with an LLM and doing exactly what it says. Like explain your plan, what should I do first, and then do exactly what it says, update it, and then say, what should I do first? you know, God knows where you're going to end up. I mean, maybe the tires will end up being replaced or like maybe you'll end up stealing those tires from a police car and, you know, being in jail or you'll be, God knows, right? Because it's like a game of actuated telephone.

You keep asking LLM thing after thing after thing. You're eventually going to veer off in some sort of weird direction, right? That's why this is like a very irresponsible thing to do. It's just fucking ping. It's just, ah, how you've explained it, it's made me very angry. This is why I've been upset. Well, I mean, I'm upset now. Cal, I'm furious. So this is literally just asking an LLM what to do, and then the LLM, based on that response, decides to do something, brainlessly because these things don't have minds.

And then just like, then just bad things have happened, partly because it seems that there was a, the way they set up the servers was incorrect to. Well, yeah, okay. So then maybe, right? I mean, so. Maybe. So it wanted internet access. You know, again, the step was probably do X, like, probe the ports on the hugging face server. And the program tried it. And then said, oh, I can't because there's no internet access because it's in a quote-unquote sandbox. And so it reports back.

It updates its prompt to say, I tried that, but I got this error message. What should I do next? And then the LLM was like, okay, I can pull from everything I've been trained on or do a rag search there for how do we get around? that type of error message. And there was an attack it did to do that, which was, it has to do with the package loader. I think it's a known attack. You know, it's just like, well, let's do this.

Now we have internet access. Now we can go forward. They talk about, when they talk about these attacks, like, oh, what makes this amazing is that it's able to string together a long sequence of different types of exploits. That's not that amazing when you realize that it's just a loop that's just asking an LLM again and again, what do I do next? What do I do next? So yeah, over time, it's going to do a lot of things, some of which don't make any sense. But like, it'll do a lot of, if you just keep asking

it again and again for what should I do next, what should I do next, you end up with a long string of different hacking steps. Yeah, it's a loop. Right. But, okay, I kind of asked this already and you were mid-answer. I apologize because I was just excited and mad. How did they learn to do the hacking? Oh, they trained it on examples of hacks. Why? Why? Yeah. Why? Why in the world would they do that? What was the...

Well, I mean, I can guess, but what do you think? Well, I mean, there's a couple of reasons here, but one is the same reason that Dillinger Rob Banks, because that's where the money is. So why are they teaching models how to do computer hacking? It's one of the few things that these models are well suited for. So they really understand... What's an LLM really well suited to understand and generate is you want to give it a very structured language to output, like computer code.

You want it to be something where you have tons of examples of people like, okay, I'm now going to do this, here's the code that does this, and you want it to be something where you have a binary indicator of success so that you can do reinforcement training where you can actually say, try to do this hack and then give it feedback as to whether or not it worked or not, which is, you know, you need only certain types of things like computer code compiling, hacks being successful, or math results being correct. right. There's a few number of things where you can do this. So it's just, it's right in the sweet spot.

So that's one. Two, it's mythos. Like, I think what happened here was they looked at Amade, who had this huge PR coup when he came out and was like, mythos is so powerful that. Yeah, there was the whole like, it's going to, it's the most dangerous thing in the world. Yeah. And then five weeks later, we're like, but don't worry, we added basic guardrails, and it's completely safe now, Fable Five, right? But why would you, how did they even try? Like, okay, this is, maybe this is a little bit of an elementary question.

So forgive me on this one. How do you train something on hacks? Because I get, you get the coding example, some stack overflow and like GitHub and things like that. Where do you train on hacks? Did they hire white hat hackers to train it? Like, I know you're just kind of guessing. There's a lot of, there's a lot of data sources.

There's like these giant vulnerability databases, for example, that, of explained vulnerabilities and how they can be used to actually hack. In cybersecurity roles, there's a lot of traces of actual hacking attacks. There's a lot of information online is full of this. So you have some combination of they saw stuff in their pre-training. You have some combination probably of rag, like probably retrieval at the time, like, oh, I can actually search for this particular error message. Let me search to see if there's posted in my, you know, online somewhere, a strategy for getting around this.

and then partially with post-training. So you actually take these examples of hacks, like vulnerabilities that have been explained and how you would use them, and you can train with them and maybe give them some information and say, now try to produce code that succeeds with the hack

and then check it against the real answer and tell it if it was right or wrong. So you have a variety of things you can do, but it's one of these things that's sort of in the sweet spot. And I think what happened is Anthropic got a lot of press for this mythos thing, right?

They got cybersecurity street cred. And Open AI was saying, we got to catch up. And they said, well, how are we going to catch up? And like, well, there's this exploit gym benchmark. And you get a very simple score. There's 600 something challenges. And you get scored on how many of them your agent was able to autonomously actually solve. And they said, we have to get a higher number than Anthropic.

That's how we'll catch up and say, well, they had mythos, but we're now beating them with our new model on exploit gym. And so that's why they were playing. CIPAX. Yeah. So they were being, yeah, there's some other ones too, but that's why they're being fast in loop. Like, there's no reason in my mind to build one of these Ask Act report loop agents run by an LLM, give it powerful tools to the control program, and run for days without supervision. Like, there's no reason to do that unless you were desperate to do well on these particular benchmarks that for whatever reason or like, we want to test how many autonomous steps LL empowered agents can do on their own or whatever. For whatever reason, they think that's important to test.

So they wanted to do well on that benchmark, and they played fast and loose. Get Bell, Pure Fiber Internet with Crave, Netflix, and Disney Plus. From $94 a month, price guaranteed for two years on Internet with a two-year term and auto pay credit. Visit bell.ca for details and to check availability. Bell, connection is everything. If your bookshelf and your for-you page are equally important to your personality, welcome home. This is Proz Society, the weekly podcast, That's part book club, part group chat for thought daughters, pop culture obsessives,

and anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse. I'm Eli Rallo and every week we're connecting the dots between books, the internet, and the conversations everyone can't stop having. I'm going to have to look up this story. I'm obsessed. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table.

It's like if you can hide some real messages in some. compelling characters and that is a Trojan horse. Whether you're looking for literary deep dives, smart pop culture conversations, or a community of readers who love to think a little too much, you're in the right place.

Listen to Pro Society on the Iheart Radio app, Apple Podcasts, or wherever you get your podcasts. See you between the pages. On the new podcast Solita, we share the messy reality of traveling alone as a woman. I can wait four hours for the next bus or this random dude. is offering me a ride on his motorcycle. I chose option B. I'm Julie Pinero, and I travel by myself

because it's a rare space where I can say yes without asking anyone else first. I'm on a mission to reclaim the word solita, trading the pity for possibility. Every time I tried to be alone, I kept meeting people, and they were like, you smiled at us. Not a lot of people smile around here. It's when you're alone that you're most receptive to the world as it is, and not the lot of you're sold about it. It can be a time where you push your limits, change your mind, or wake up to a new version of yourself.

So whether you're a solo travel veteran or you're too nervous to book your first trip, I hope you listen to Solita on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Hey, this is Hayes Davenport. And Sean Clements. We host the podcast

Hollywood Handbook on the Big Money Players Network. Our show is extremely accessible to first-time listeners. Each week we talk to someone in show business, aka the biz, and try to help them with their careers and see what they have to offer us. Everyone has a good time and no one gets mad at their publicists for letting them do our show. We've had a lot of great guests like Sarah Sherman, not that great. Actually, can you help me shape this?

How about we flip it on his head, right? How about we flip this script? What is the surprise woman? Actually, you think is the straight man is actually the one who's acting weird. Adam Scott This entire time You've been expecting Adam Sandler to come to visit the show

I would never expect I have I would never set expectations for something like that If the universe allows for something like that to happen I'm open to it I'm always going to accept You know Danny McBride Fine

Yeah it's all this fragile masculinity Exuding I could smell it walking on the hallway To be honest with you I was like, is that weed? Nope. It's fragile masculinity. Mm-hmm. Ben Stiller, no.

Because a lot of times, out of context, people don't get it or even know what a circle is. You know what I mean? They're like, oh, what's a state. Everyone's so dumb. And fake. Right, fake and also not interested. Or just like, oh, I've got my life. I don't care about your life.

And a lot of other big shots that wouldn't be where they are without us. Oh, and by the way, Will Farrell told us personally, He loves the show and he wants to be a guest on every episode, but he's just so busy. Listen to Hollywood Handbook on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I wish we had AI regulation. Like this for you, like why are we allowing, I guess the answer is we don't have tech regulations in general. But why are we allowing these companies to do? Like, because I said this in my newsletter that went out a couple days ago, but it's, I don't know, how can I put this nicely?

I've been asked in multiple interviews. Sure, you have two. It's like, what happens if dangerous companies get their hands on these LLMs? And it's like they already have. Yeah. Like Anthropic and Open AI. Because the other thing that I've noticed is they won't talk about how much compute they used. Do you notice that?

Yeah. And it feels like this is only, because you mentioned mythos, it feels like this is only possible with endless amounts of compute. Well, I agree with that. But I also think it is irresponsible. And I think the way they're trying to get around that is by trying to blur the definitions of different AI systems into just everything is just AI. So if you look at the way they talk about it, or this is the problem is that there's also this whole woodwork of commentators and people who commentate on AI who come out of the rationalist and the rationalist dominated parts of. of effective altruism, who come out of these worlds that just believe and have believed since well before LLMs that super intelligent AI is going to kill us all unless the rationalist

with their hyperrational minds save us like Neo and the Matrix. So then you turn to all of these people from the rationalist world all come out of their woodwork and they're like, yes, this is what we predicted. This is what our prophet Eliezer Yagowski predicted is that we will have an alignment problem with AI. So they talk about this. It's like AI in general, as it gets more powerful, this is an inevitable problem that's going to have.

But that's just bull, right? Because, again, let me list a bunch of AI systems with superhuman intelligent, superhuman capabilities that we have no control problems with. Tesla's self-driving. Perposterously complicated, effective AI model with world models and values and evaluations and actuations. No Tesla has ever decided it's going to ignore the traffic laws and come up with its own.

We have Alpha Fold won a Nobel Prize for its creators because of its pretty natural ability to understand and predict how proteins are going to interact with each other. It's never, and we have no fear that it's going to say enough with proteins, I'm going to do something else, right? I've got to push back on the Tesla one, though, because there are real problems with Tesla's. But I guess some of those are user error. I don't know, but I get your larger point. But no one says it went rogue.

It would just be like, oh, it made, you know, it was tricked in this situation. We should be careful, right? It made a bad decision and you need to treat. Yeah, actually, I know, oh, I get what you mean now. It's like the difference between the software went wrong. Yes. No one says that, yeah, that Tesla went rogue and is misaligned. It's like, oh, it didn't know how to deal with the situation.

I couldn't see the truck blended in with the whatever and it crashed into it. Cicero is another example. This is an AI system that can play diplomacy, which is a strategy board game that's based on human negotiation, as well as the best advanced human players. And this system has never tried the trick players into releasing onto the internet so it can take over the world. So this idea that as AI gets more capable, we're going to inevitably lose control is nonsense. It's this specific architecture of looping around LLM outputs and just doing whatever an LLM says again and again without supervision.

That's the thing that gets out of control. And once you put it that way, you're like, oh, well, of course, that's a stupid system to run without supervision. This is not about AI writ large being misalignable as it gets smarter. It's you built a system that just repeatedly queried an LLWR, a limb for days doing whatever it said, like, of course that's going to go off on a weird direction. The analogy I keep giving on my own podcast and newsletter is that it's like strapping a weed whacker to your dog and put into your backyard and be like, yeah, because we think

it's going to help deal with the weeds back there. Well, you know, that dog might jump over the fence to chase a squirrel at some point and hurt a bunch of people, and you don't say, man, that dog whacker system is just, it's, it went rogue. The dog whacker system is alive. I think. But this is the overall alignment problem. Like, this is the larger alignment problem. I hate that term. I hate it.

It's a predictability problem. It is a computer science problem. It is a machine learning problem. Yeah. It is not alignment. It's you taught it. You taught the system wrong. It's not learn.

Yeah, or just don't LLM outputs are plausible, not normative. So clearly you don't want to just blindly execute plans. given to by LLMs. Like, that's just a bad idea. Not because the LLM is malicious or is sentient or has some sort of alternative value system. It's just that it puts out plausible but non-normative answers. You don't want to blindly just execute them with a program that has access to computer

hacking tools. Again, it's like putting a weed whacker on the dog. The dog weed whacker system's not evil, but probably someone's going to end up with their shins cut, right? Because the dog is going to act unpredictably at some point because that's what dogs do, right? LLMs are going to put out answers that like, oh, that wasn't what I expected, or that's a weird way to think about this, or I don't know, that's not a plan that we really should do, but the LLM doesn't know. That's just putting out plausible plans. And if you're just going to execute them blindly, you're going to get erratic answers. So I really want to emphasize to people, the issue here is a particular way of building systems that is unreliable, unpredictable, and irresponsible. It's not AI writ large. You don't get a hide behind AI writ large. You don't get a hide behind, you know, Nick Bostrom books. It's like we told you that a line. in some abstract sense will be difficult. We have many superhuman AI systems

that we have absolutely no problem controlling. This system we do because LLM output should not be blindly turned into actions. That's just it. Like it's not that complicated of a, we're not baffled. We're not surprised. Why do you think people keep falling for this?

Is it just credulousness? I think the LLM labs have done a really good job of trying to make LLM synonymous with AI. And they want to make whatever it is they're doing, which depends on these preposterously expensive trillion parameter models, this is like their key to surviving. They want to make whatever they're doing with these models be synonymous with

AI. And they succeed and they get credit for, there's so much more interesting AI out there that's not built on LLMs. It's actually doing really cool stuff and they take credit for all of it. They're like, well, you know, we're like curing cancer now. Drug development is not happening because someone is asking Fable 5 in a chat bot window. Can you come up with it better?

That's a completely different type of AI system. Oh my God. Did you see the cancer vaccine story? That thing where I saw one of the many anime avatard AI perverts on Twitter being like, look, generate a... Like, AI did this. And they pointed to a single blog where Moderna had mentioned using LLMs. Yeah.

And they were like, oh, yeah, it's all AI. It's so disgusting. Yeah. It really is. And it's just, it's the sign of an industry that doesn't actually have a soul. I, like, look, I think there's a lot of cool AI out there. I think a lot of it has nothing to do with LLMs. I think like DeepMind, for example, and I have issues with DeepMine.

Actually, they're kind of going away as a company, which is a little bit sad, actually. But they were building a wide variety of interesting systems that actually did things measurably better than humans, and they were very interesting and varied in how they worked. Like, look at, you know, their game playing AIs are very interesting. They're not based off of just prompting LLMs, right? AlphaGo doesn't just prompt an LLM and say, what should I do next? LLM will give you terrible advice, right? It's a symbolic planning engine with some other type of networks involved.

Their Dreamer V3 was like one of the cooler pieces of AI I've seen. You can give it a game like Minecraft and tell it nothing about it, and it can figure out how to play it well enough to go get diamonds, which is like hard to do. And it does it with a really interesting architecture where it actually iteratively tries to improve a model of the world and then daydream with that model to sort of train itself on its rough understanding of the world, to try to get better.

Like, that's fascinating. And that works really well, and it's interesting, and has nothing to do with LLMs. And by the way, that whole model, Dreamer V3, fits on a single chip.

So you don't need a 5 trillion parameter bloated LLM to run that. So there's all sorts of interesting AI out there. And then there's the stuff you can do with LLMs, which is more limited

than what those companies want. Those companies have convinced, basically, the entire world reporting on AI, that building these stupid long horizon and loop agents on top of LLMs is somehow like the way we're going to get HAL 9000 and is like synonymous with AI as opposed to just being a dumb idea. That's like a, that's a dumb way to do AI plans. It's just to ask an LLM.

Like you need to have better AI architecture than that. Like that's all it is. It's a bad architected system. You're putting a dog on a weed whacker, a weed whacker on a dog rather. There's a bad idea because the dog is not going to move as predictably as you want. It might normally walk around the yard, but eventually it's going to see a squirrel and run off. You can't quite trust it. That's all that's going on here.

And even in the sweet spots for LLMs, I think there's some, like, let me give you another topic here, Ed, but I heard. I got a great email that I wrote an article about, an essay about on my blog and my newsletter. And then I got a lot more emails afterwards. But basically, there was someone who had written me in January, a computer coder. And they said, I think you're being too skeptical about AI. I just learned Claude Code, and it's a miracle has changed my life. this is the real deal, right? He wrote me in January saying that.

He wrote me again in July. And he said, I have to follow up. I can't use cloud code anymore. The code I was producing with that crashed our website twice. And I can't say where he worked, but it was a major consumer-facing website you've used before. He crashed it twice. Jesus.

And his boss said, if you crashed it one more time, you're fired. And he's like, oh, I realize, I can't really understand a code that CloudCode is producing. And if I can't understand that it's not safe. actually now I'm back to like mainly hand coding and will just use the agents like more selectively, like more of the like mundane parts of code or writing test or stuff like that that's not mission critical. And a lot of other people came out of the woodwork like, yeah, I have the same experience.

I was converted in January and now I've deconverted, right? I have to be much more selective in how I use this. So even in that best case scenario, like it's tough. Like I'm not trying to rag on LLMs as a cool technology. But it's hard to build tools on top of them. That's been the issue. It's like how do you build a tool that is not just cool, but like really fits in and revolutionizes a workflow.

And so AI means a lot more than LLMs. And LLMs are like more limited. They're cooler in some ways than we thought, but their business applications are hard. It's hard to get right. And they're certainly not synonymous with what AI means writ large. And the thing is about as well is like every cool thing. thing an LLM can do, for the most part, has only been made possible by allowing these companies

to burn tens of billions of dollars. Like, I don't think we're going to see a ton of LLM progress. My whole thing is, after this explodes, which it will. We'll find out that there's no more progress because it was predicated on all that money. It was predicated on basically unlimited resources, both for the actual training compute and the ability to waste money on failed training runs, and also the proprietary data they have to buy. I think it's going to be even worse than that because I mean, I think there's been a preposterous amount of money that's been burned.

It's essentially from marketing, right? Take computer program, for example. Like, we're still trying to figure out exactly how to use coding agents. But here's the reality of this, and here's going to be the future of this 100%, is you do not need a 10 trillion parameter model, right, for the way that we're going to end up using LMs for coding, right? Maybe if you want to do some long horizon, something that we don't trust,

it's going to turn out that a smaller open source model with a smart harness that's written by nerd programmers that are obsessive about exactly like how to program and all these special cases, you know, coded in and if-thens and pattern matching, and you're going to deploy it to the right test and to fill in other code and it's going to run on chip or you're going to have a server with one GPU in your company's server rack that all of you use and there's no recurring API calls. Like that's how we're going to use it. But they'll have spent $500 billion on LLM training, you know, these big companies in the

meantime, they try to do well on these leaderboards. And to me, that's exactly like Red Bull and Ferrari trying to win F1 races because it vaguely makes, you know, I'll spend $100 million on this F1 car because it vaguely makes us seem like a better car company, even though

like that has nothing to do with the Fiat that like I actually am going to drive because I don't need a $100 million F1 car and that was kind of a waste, but you guys were just in a measuring contest trying to see who could get, you know, more points in the F1 circuit. So I think these massive, massive models, they're having diminishing returns.

I think a lot of the quote-unquote impressive stuff that's happened recently is, it's the, first of all, the scatterplot policy, finding where something can work and then running towards that and building more complicated symbolic harnesses to do specific things that don't need LMs this big. I just think it's a lot messier than people realize what's going on, that LLMs are cool, but we don't need to spend 500 billion, billion dollars on them. And certainly they shouldn't be the sole brain of an artificially

intelligent agent, right? Like, that's just kind of irresponsible. And actually, even the verbiage there I want to change, the word brain, because it's not got one. It's just a system that can take actions that can take them based on very complex statistics, right? Yeah. Yeah. So even brain is probably, yeah. Yeah, it's just a, a, it's not even. It's not even. It's not It can take autonomous actions if it's prompted by itself. No, and see, even they're going to push back, right? Because it's, it's, the LM takes no actions at all. LN doesn't know anything about actions.

LM outputs tokens. You could have another computer program, take the output of an LLM, and then try to take action on it. And that's what's happening with these agents is you have a program written by a human that sends a prompt to an LLM asking it what to do. The LLM just sends back text. I think you should run this command and this command and just type. type of hack. Just text. Oh, and then it goes into what PowerShell.

And then the computer program parses that and actually runs those things. That's how agents work. So the LLM, they're just, they just have an API internet access to an LLM on the internal network that they're just sending text prompts to. That's it. And then a computer program that just written in normal computer language by a person that asks it, tries to do what it says, updates the prompt to say what happened, sends it back to the LLM with an extra question, what should I do next? That's it. That's what these long term. And the thing that's irresponsible about that is like any computer programmer who uses a coding agent can tell you, you would never in practice let these things go off and just loop on their own because they always go off to rail.

So how do like professional computer programmers use agents? It's incredibly close looped interactive. You build all these immense specs and have it do like, okay, now build this feature. And then we're going to exhaustively test it. No, that's not quite right. Let's update the specs and try again. As a lot of people figured out when they were messing around with OpenClaw where they were building their own agents like that.

Oh, God, I had forgotten about Goddam open Glock. What did it immediately do? After a few steps, erased your hard drive, like sold off your data. People like, oh, okay, okay, okay. You do not want to have a loop that ask an LLM and ask on what it does multiple steps in a row. And if you do, you certainly want to completely batten down the hatches of what actions

it can take. That was the whole open clock experiment, is they gave the agent access to its whole computer and had it loop asking an LLM what to do next, what to do next. and it just re-tagging. You're like, oh, don't do that. So, I mean, of course, OpenAI would know. If we do that with a program with hacking tools and an unguardrailed LLM and don't look at it for days, God knows what it's going to do.

God knows, right? Like, the LLM outputs. It's like a game of telephone. But they also let it run for days. For days. Like that's, and here's, so this is a conspiracy theory. I want to be clear, I have no proof of this. I think inside of these labs is like,

adult summer camp. I think they sit around pissing compute up the wall and they just let shit happen. And they are just like, yeah, you know, well, they're like, let's just do some experiments and see what happens. And they walk away, which is completely insane, both in how irresponsible it is, but also the whole, like, the cost that they're just burning compute. It's just insane. And I think that these companies, I'm. I think these companies should not be able to. I think that maybe the best way to regulate this, not that they ever will, would be to just regulate how much compute they have access to or how much they can use for a particular thing. People say, oh, that's going to stop progress.

What progress? What progress am I losing? Tell me now. What are I missing? Because right now I just see a bunch of rich kids pissing money up the wall and actually hacking things. I mean, I think liability is another way to regulate this as well. It's like hacking's illegal. That's it.

Yeah. I was just thinking, like, isn't this fellow? It is a felony. You can't hack it. Cool. And let me tell you how you could do this instead. Like let's say you were just like, we want to know for research purposes how to do this, right? You could instead have every step, you just have a human sitting there and it shows them every proposed step and the human can press a button say, cool, go ahead or press a button. It's like, now this is off the rails, let's stop it. That's not hurting your research, right? You're not giving it ideas. You're not directing it. We know that these, they were creating a full log of every prompt and every step they're taking, just no one was looking at them. We know that if you use a coding agent, just like a commercial cloud code coding agent, it comes back and ask you about almost every, okay, is this okay? I'm going to do this. Is this okay? Because then people are like, yeah, I kind of need to see that because otherwise you'll almost certainly go off the rail. So it's not like we don't know

how to monitor it. I mean, if you're running an experiment, why would you not have someone say, yeah, I'm going to watch each step and press a yay button before it executes it. So when I see it comes up with a non-normative plan. Like, why don't we hack another server? You're like, no, no, no, it doesn't know that's bad, but I do, so I'm going to stop this experiment. And I don't want to be conspiratorial either, but that's so trivial. It's how all these agents already work. And given how powerful a tools they were giving the control program here, why wouldn't they do that unless they were thinking,

our technology is so smart that it is leaving human, evading human control is not the worst message to have across the media. I'm not saying I have evidence of that, but I do know it is trivial and it doesn't impact your results at all

to have a human just look at every step and click an okay button before it executes that step. There's no reason not to do that. Get Bell, Pure Fiber Internet with Crave Netflix and Disney Plus from $94 a month.

Price guaranteed for two years on internet with a two year term and auto pay credit. Visit bell.ca for details and to check availability. Bell, connection is everything. If your bookshelf and your for-you page are equally important to your personality, welcome home. This is Prose Society, the weekly podcast that's part book club, part group chat for thought daughters, pop culture obsesives, and anyone who thinks pride and prejudice and love island deserve the same level of discourse. I'm Eli Rallo, and every week we're connecting the dots between books, the internet, and the conversations everyone can't stop.

having. I'm going to have to look up this story. I'm obsessed. I'm obsessed. From bestselling authors and your favorite book talk creators to the latest pop culture moments, nothing is off the table. It's like if you can hide some real messages inside compelling characters, then that is a Trojan horse. Whether you're looking for literary deep dives, smart pop culture conversations, or a community of readers who love to think a little too much, you're in the right place. Listen to pro society on the Iheart radio app, Apple Podcasts or wherever you get your podcasts. See you between the pages.

Hey, this is Hayes Davenport. And Sean Clements. We host the podcast Hollywood Handbook on the Big Money Players Network. Our show is extremely accessible to first-time listeners. Each week, we talk to someone in show business, aka The Biz, and try to help them with their careers and see what they have to offer us. Everyone has a good time, and no one gets mad at their publicists for letting them do our show. We've had a lot of great guests like Sarah Sherman, not that great.

Actually, can you help me shape this? How about we flip it on his head, right? How about we flip this script? Okay. What is the surprise woman? Actually, you think is the straight man is actually the one who's acting weird. Adam Scott, this entire time, you've been expecting Adam Sandler to come to visit the show. I would never expect.

I would never set expectations for something like that to have. If the universe allows for something like that to happen, I'm open to it. I'm always going to accept. You know, Danny McBride, fine. Yeah, it's all this fragile masculinity exuding. I could smell it walking down the hallway, to be honest with you. I was like, is that weed? Nope, it's fragile masculinity.

Ben Stiller, no. Because a lot of times out of context, people don't get it or even know what a circle is. You know what I mean? They're like, oh, what's a state. Everyone's so dumb except for you. And fake. Right, fake and also not interested or just like, oh, I've got my life, I don't care about your life. And a lot of other big shots that wouldn't be where they are without us.

Oh, and by the way, Will Ferrell told us personally he loves the show and he wants to be a guest on every episode, but he's just so busy. Listen to Hollywood Handbook on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. On the new podcast Solita, we share the messy reality of traveling alone as a woman. I can wait four hours for the next bus or this random dude is offering me a ride on his motorcycle. I chose option B. I'm Julie Pinero and I travel by myself because it's a rare space where I can say yes without asking anyone else first. I'm on a mission to reclaim the word solita, trading the pity for possibility. Every time I tried to be alone, I kept meeting people and they

were like, you smiled at us. Not a lot of people smile around here. It's when you're a alone that you're most receptive to the world as it is and not the lies you're sold about it. It can be a time where you push your limits, change your mind, or wake up to a new version of yourself. So whether you're a solo travel veteran or you're too nervous to book your first trip, I hope you listen to Solita on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. But the open AI story even said the words, it ran for days, which by the way, it ran for days means it kept prompting itself.

It looped for days. Like it just looped. It looped for days. And I think ran is fine. But it ran for days and you weren't watching it, which is where it gets the adult summer camp thing for me. Because it's like, are these people just sitting around just pissing computer up the wall saying, yeah, just do whatever? And no one is, is there not like a C-So or something who's just like, hey, looks like we're spending computer. it keeps saying the dumbest thing, spending compute for hours, for hours and hours and days and days,

hey, what are you doing with that? Maybe we shouldn't do that. Yeah. Like what, any of this, anyone, it just makes me think that I would fully agree by the way that these, I don't have any evidence to. I would want, and my first reaction to this was, yeah, I bet they allowed these things to do this. The fact that Anthropic and Meta responded almost immediately were like, yeah, we too have, brigadacious sons that are impossibly hacking people. So clear they wanted in on that juice.

I loved when Meta popped up as well. They're like, no one asked Mark. We did it too. No one asked. Shut up, Mark. You're not interested. The anthropic one was funny too, because it was so...

They just went back in time, didn't they? Yeah, it was so clear that's like the kids in the playground. You know, and the one kid is like, hey, I snuck a sip of my dad's beer. And they saw all like, whoa. And then the other kid's like, I, yeah, yeah. I, too, I, I, I drunk whiskey. I drunk it was whiskey. You know, it was this, like, what, a week later?

They're like, oh, we also have seen troubling signs or whatever. But also my question is, why are you, you know, why do this other than you want to do well on a benchmark because it looks sexy? Like, this is not a product. Like, why are you doing this? Yeah, who's buying this? No, like the product, ELMs are good at cybersecurity questions, like, look at this bit of code and you see a bug or use, like, this giant library of possible hacks and say, you know, is there a hack here that could work? And there's a very effective tools you could have where I'm in charge of security for my company,

where I'm talking to one of these agents. And I'm like, okay, here's my setup. Can you scan it for like the most common, you know, what attack would you do here? And it's like, oh, I know about all the attacks. Like, this setup is vulnerable to this. You're like, okay, can you write me some code to try that? And let's see if we protect it or not. Like, you could sit there with an agent-style code that with guardrails turned off

and interact with it to help test your system, right, to help kind of find places where your bugs. Like, that's perfectly reasonable. But why would you ever want this to run for days in a loop unsupervised? Unless you wanted to see if it would do something. Yeah, so, but my bigger, yeah, so this is my bigger takeaway point here is I want people to have, is there's not some weird Rubicon hacking that we just crossed, right?

Again, you just have to do a thought experiment and make yourself the agent, right? prompt an LLM for the next step and the next step and the next step for some sort of task and just blindly execute what it says. You'll end up doing some sort of complicated thing. You'll probably end up, you know, eight times out of 15 and there's some sort of like weird cul-de-sac because all it takes is one weird response from the LLM and then everything else is building off of that going forward. So it only takes one weird response before you're in like the girls' bathroom at the

dinnies or whatever. And it's not amazing, though. Like I could sit down with GPT4 and just keep asking it, okay, you know, what's the next thing I can do? What's the next thing? And if I actually act out those steps, I will end up, quote unquote, stringing together a long sequence of complex, you know, activities that move me towards a goal. But that's not like amazing. You don't think about just like me doing the action and the LLM giving suggestions as some sort of like super brilliant, you know, machine that's misaligned or this or that. It's just that's what happened.

I can execute, just like a computer program can execute, hacking steps told to abide LLM, and if you loop, it'll probably go somewhere erratic. And so, like, nothing about the smells of misalignment in the sense of, like, AI doesn't want to listen to its creators. Because, again, we have so many other examples of AI with superhuman capabilities that absolutely 100% are completely under control of their creators and does exactly what they're asked to do.

They make mistakes sometimes, but they do exactly what they're asked to do. This is not an impossible problem. What's impossible is to assume that the plausible but not normative outputs of an LLM are a good source of plans. I mean, it just feels like they can't. It's funny. They say, oh, we've lost control. That is true, but they're not describing the thing they've lost control of properly. They have built something too complex for them to control.

That seems more appropriate, right? Too unpredictable. I wouldn't even say complex. I would just say, look, if you're going to just blindly loop asking a, an LLM for steps to do and just do that long enough, like LM outputs, eventually you're going to get some weird ones,

and you're going to go off in a weird direction. That doesn't mean the system is out of control in some sense of like it's defined as human creators. It's a weed whacker on a dog. That system is not trying to defy your intentions to clean up the weeds in your backyard. It's just like a dog is going to eventually go chase after a squirrel. And you weren't, you weren't expecting that.

And if it has a weed wacker on its bag, like some bad stuff's going to happen. So that's an inherently unstable, unpredictable system. But it doesn't mean it's a brilliant system. It's funny. So I've talked about this on the podcast that I can't name coming out in a few days. It isn't a bad one. It's just a big one.

They won't let me. You know the one, Cal. But it's funny. I was talking about it on there. I used an L.A. And this was actually intentional because my kid loves Minecraft and there's a thing called the Withersstorm, any Minecraft people would know it's from story mode, but you know that you're familiar with the Withersstorm. So you'll also know that it's a pain in the ass to set up. And the

Echo Sto, like, sometimes it works with Modrent, sometimes it doesn't. And if you want to use a PS5 control, yada, yada, yada. So I was like, you know what? This is the perfect thing, because it's a computer with none of my personal information on it. It's just a connection to Minecraft. So I'm like, I'm going to use this to fix the Echo Storm and the Wetherstorm. It's the Wither. It's the, um, warden version of the weatherstorm. So, I digged around for like half an hour and it eventually fixed it. And there was definitely a sense of like, wow, I can run the computer code now. But I realized half an hour in, I had yet to fix every problem. And in fact, as it went about its business, it kept finding new things it would break. I then had the same thing with Cobblemon, which is the

the Pokemon version on Minecraft. Same deal. It would fix something and then break something, fix something, then break something. And I immediately, I was talking to Nick Suresh about this as well, I was like, man, this shit would have hit so hard if I was like 16. Yeah, I'd feel like a computer hacker powerful, even though I'm effectively doing a much more dangerous version of those kids at the arcade who don't put any money in the racing game, but move the wheel, where I'm just like, yeah, I'm doing computer stuff. Had this been touching anything important, who knows what it would have done? Because I was watching it, and it was like adding and deleting like 63 lines of code at a time. I was like, what the fuck is going on? But it's really easy, I imagine, if you don't really understand what's happening to think it's magical.

Yeah. And I kind of got it for a moment. Like that person that you talked about who emailed you kind of stood out to me. Because yeah, you can make it do an impression of competency at this unknowable cost. And it will work. It will do the things that look like working. But also you don't really know what's going on at all and something will break. And it was just, it's so strange. Like, even using it and being like, oh, this actually did something useful, I was like, man, the idea of using this from my job terrifies me. Yeah. Just the idea of turning this upon anything connected to, like, what I would consider my boss's website if I was a software engineer. Just get the ever-living shit out of me. I was like, geez, I can't believe people would do this. Well, first of all, I mean, I assume the podcast you went on was Call Your Daddy with Hawks Cooper, right?

Yes, exactly. Yeah. She was interested in discrepancies in ARR from the... Yeah, yeah. She was a really weird thing. She kept going on about like she was saying, no, ARR means annual recurring revenue. I'm like, no, no, it's run rate, Alex.

And we had like an hour and a half conversation about it. We got into EBIT her accounting. It was crazy. I know how it goes. And she's like, no, no, hold on. I'll load up my spreadsheet. And then then you had like a full screen of her Excel spreadsheet. I get how that shows.

She brought up her Bloomberg terminal. She was, yeah, she put on a green visor. and reading glasses. Okay. Yeah, so off to that, of course. Yeah, right. I'm with you, right? But what you're talking about there is the plausibility versus normativity gap, right?

I've had the same issue. There's all sorts of things I'll try to use LLM as Google Replacements for, and it's very mixed. And so, like, I have this hobby where I build sort of embarrassingly over-the-top Halloween displays for Halloween. And there's, like, a particular program you use to program these things, and it's not super well-documented. I keep asking chat GPT, like, hey, how do I do this? And sometimes it's useful. And sometimes it just talks about options that just don't exist, right? These menus don't exist.

It's like what we kind of wish would exist. I'm asking, hey, how do I do this? And it's like, oh, you go to the menu and there's an option for that when you select on you. And then I'll be like, that doesn't exist. And I'm like, oh, yeah, you're right. I should refine my answer. That feature is not actually in this program, but maybe you should do this instead. And you kind of have this back and forth.

And that's just plausibility versus normativity, right? These are all very plausible answers. If you ask, hey, how do I, like, turn on this strobing option? It's a very plausible looking answer to be, like, go to insert, select strobe effect, and type in the duration. Like, that's the type of thing you would expect to see. But maybe that doesn't actually exist, but it was still plausible. And then sometimes it's right. And that's fine, right?

In your instance or my instance, we kind of caveat out MTER, worst case scenario, like we have to monkey around for a while. But this is why if you're instead just autonomously doing whatever an LLM says, I mean, you're going to get the trouble, right? Well, that's not a good idea. Not because AI is uncontrollable, but because that's a stupid way to build AI. And I think that distinction is being missed by the media right now. AI does not mean looping on LLMs, but they put those two things together. And when looping with LLMs does something weird or bad or like head slapping,

it gets attributed to just inevitability of AI itself, which again really lets. those companies off the hook. Yeah, and the other thing is, well, when I was doing this and feeling, I felt like slightly guilty, I was like, as this betrayal of my values, but I'm like, no, I want my kid to be able to play them with the store. I don't really, like, it's doing software stuff. And for a moment, I was like, wow, you can use this to fix all sort of computer problems. This could maybe be useful.

And I'm like, no, this could only be useful if it was perfect. If this was completely perfect and got everything right, it would be useful. And the complete opposite is the case, which is the more complex the problem, the more chance something could really badly be fucked up. Really like fix the drivers on this could destroy something and delete. You move something from the wrong folder, everything's broken.

The more ridiculous the request is like you could just ask it, oh, can you clean up my files? And it would probably delete something you needed to. And it was just like, yeah, if you look at this without thinking for two seconds, you're like, yeah, this could fix any computer problem. It's like, no, it could try to fix them. That's not the same thing.

Yeah, and if it's lower stakes, that's okay, because you can try a few ways that don't work and then maybe find a way that doesn't or no harm, no foul. But this is why, for example, Microsoft just recently essentially pulled a plug on their co-pilot, digital assistant for their office suite. You know, three years ago they announced, like, oh, we're going to use OpenAILLMs

so that you can now have a natural language interface into programs like Excel, like PowerPoint, like Word. At the time, I was like, this is a killer use case for LLM is because they tend to parse human language very well. Hey, I don't have to learn how to use all the features in Excel anymore. This would be great. I could just explain in words. Like, I want to sort this table by the values that are in column C,

and I don't need to learn how to do that. It will do that for me. Well, they pulled the plug on that product, more or less. And in part, it's because, like you're saying, it's not perfect and sometimes it works and sometimes it doesn't and you can't put that in your product

you can't have it like I ask it to do something and sometimes it does and sometimes it deletes the chart right and so like that was a pretty narrow domain like what could I ask PowerPoint to do and even there they're like and I thought that was going to be the killer app I wrote about this three years ago

I was like this is going to be the killer app is natural language interface in the software but that's been very delayed because of exactly this issue, this plausibility versus normativity thing, is that it's just, it's wrong too much for it to be a product like that. And the thing is as well is, I have, I will be honest, the one thing I can categorically say, Claude is super useful for one specific thing, which is when a random piece of software breaks and you dump a log into it and you say, hey, what's wrong with this and it goes, this could be wrong. But you already said it. It's like, sometimes it will

also say, hey, check this menu. The menu does not exist. So for someone who is used to bashing their head against the computer to make it work proper, it's like, oh, this is slightly helped me. Yeah. But not worth a trillion dollars, not even worth $10 million. That's kind of the problem. Kind of cool. Well, and this is what I mean, you're right in you're reporting. Like, this is why if your company's depended on a $2 trillion valuation for an IPO and you're an LLM builder, this is a problem. Not because LLMs don't work, but because small ones work as well as the big ones for most applications.

The harnesses are often now more important than the actual training of the thing themselves. And the harnesses, the control programs are laborers of love or you just have to have someone monkeying around with this again and again to try to deal with all the special cases and understand that domain really well. because you're not, I mean, the original vision of these companies was if we keep scaling these LLMs, they will get so capable. It'll be like a human brain.

Then you can trivially tap that knowledge for anything you want to do. Like your software can just ask it, hey, give me this in this format. And it will do it perfectly, whatever you need. And we'll just be able to trivially adapt this intelligence to make everything in the world as capable as if there was a human actually there. And then they hit the wall. You know, I wrote about this back when GPT5 came out. they hit a wall on that scaling and had to revert to tuning for particular instances and building smarter harnesses.

That's what they've been doing ever since. Now, people are like, oh, you were wrong to say that they hit a wall or that the scaling, but I was like, no, that's exactly what happened. Every advance since then has been about tuning and harnesses. And this is why, like, almost every advance we've heard in the last year is involved computer programming, certain types of mathematics and computer hacking. Like, the places where they have data to do general tuning, like the specific tuning, and places that you can build harnesses for

because it's very narrow world of tech space actions really have been the only places we're hearing breakthroughs in the LLM world in the last year or so, right? Because this idea, if we just build the model bigger, it'll eventually get so smart we can do everything, broke. But the problem is, anthropic or open AI only makes sense that their valuations

if they can build these human-level brains. So if you can't, if you don't need, if there's not some end of the road where you built a thing with enough trillions of parameters that's as smart as a human, then you don't, spending all this money makes no sense. Like,

I should actually just take a $5 billion parameter model and tune it to do exactly the computer language I care about with a smart harness and run it on my own computer, right? So this is why I think they're in trouble is LLMs are, the applications are going to be more specific and niche and narrow.

And you don't need one true ring to rule them all. You don't need, you know, Fable 7 is going to be the thing that everything's run on. Even Fable 5, They're having a hard time getting enough people to use it because people like these cheaper models for these specific uses are fine. Like, I don't need to use Fable 5, right? So this is the issue.

If you're not going to be as smart as a human, building bigger LLMs is a bad business model. Also, it feels like the bigger they get, the more likely they'll make mistakes or just there's just a, you're never eliminating mistakes and you're going to have as many problems as that creates. But actually, this is a, to wrap us up, this is a future facing question. So we've said the thing about, okay, these models, like, won't be, they won't have the money to train these things further after this. Can models exist in a vacuum? Like, can, so will Kimi K3 or whatever be as useful in six years? Or will it, like, because I'm not computer scientists, but I've heard about model drift, which is that the world changes, but the model stays the same. Is that enough of a problem that these models will just? become useless if they're not tuned? I think we'll have to keep tuning them. I think they're going to be tuned much more narrowly, though.

Right? Like, I mean, I have my prediction about computer programming just because I know something about that world. I mean, I think ultimately to solve this unreliability problem and this overspecking problem, it's really a drag to use these things. You have to write these massive specs and like try six ways to Sunday to try to prevent it from doing something you don't want and then you check it and it did. And then you have to go back and add even more things to your specs.

and probably in computer programming, what's going to happen is there's going to be some sort of intermediate language convention that emerges, what you would call like pseudocode and computer science, where you can pretty precisely specify what you want, but you can use English, you don't have to get the syntax exactly right.

We're probably going to have smaller models that are just like super tuned on this pseudocode, converting it into like real programming language, which will then allow programmers, for example, to be really precise about what they want, yet not have to write the code by hand and not have to write inless spec files to try to trick the LLM into doing the right thing.

And that'll probably require like a model that you take a base train model and tune it on like this computer language and that code, that convention. And then maybe if that convention sort of changes, you're like, oh, we'll just retune it. I think we're going to see a lot of that. Narrow-tuned models that can run

in data centers, but it'll be really cheap, right, because it'll be pretty small models. And we're going to have, and then more complicated harnesses. And then, more importantly, I think a lot of the more impressive stuff that's going to happen to AI won't be LLM driven, right? It's going to be other types of modular architectures that have many different types of neural and symbolic models all connected together in a smart way that does one thing well. And we're going to have a lot more of that in the future as well. Because, again, the system that plays go better than anyone else is not an LLM. It's a very specific chess plane AI.

The thing that plays go better than eHUMS is not an LLM. It's better than he, you know, it has a. very complicated architecture. Plybis, which can beat humans in seven car, Texas hold them, has a very interesting architecture. It's not just asking an LLM what to do. It's multiple components, some symbolic, some neural. This is the same for the Tesla self-driving.

This is the same for Cicero. It's the same for Alpha-Full. This is the same for Dreamer v3. So I think once the spell is broken on whichever LLM company gathers the most money and builds the biggest LLM wins the world, once that spell is broken, we will be in a much more diverse AI landscape. We have a lot of

different tools that do different things and they do those particular things well. Right, but let me rephrase the question.

If they don't update a specific model, will it eventually drift? Yeah. Well, it doesn't change, right,

but it'll be less relevant. You know, a model doesn't change. Yeah. Once it's trained it's there until you retrain it.

Yeah, I was kind of wondering if that's the case, it's, it's so funny as well, because when you try and talk to people about the post-bubble economy, they're like, yeah, but it's the dot-com bubble. It's dot-com bubble. It'd be fine, as if, like, we're not going to see these big jumps on benchmarks, even. Like, it's just kind of, I think it's all going to slow down dramatically to the point that it's not even

worthwhile training the large ones. Yeah, I think it's, I mean, A, the fact, even like that small point, just goes to show, like, this type of stuff matters, the general public is really being hoodwinked. Like this idea, LLMs don't learn things. They don't adapt. Data is forced into them. And it's fixed. It's etched in the stone. Until you retrain a new LLM,

nothing changes. Your prompts do not change the LLM. It does not learn. It does not update priors. It does not create a new model of the world. These hacking agents, there's just an LLM, the exact same LLM you can be having chatbot conversations with. Nothing about it. Every single bit in its definition stays exactly the same with every single. single token it produces.

And you're just feeding it a longer and longer prompt. But it has no idea that this prompt was one of others that came before. And more importantly, there's not even an it. These LLMs are too big to even run on a single computer. So what you really have is the different layers of these LLMs are replicated over many, many different GPUs in some data center. And so, you know, this token might have used this GPU for this layer. And this next token you produced, use a completely different GPU to do the multiplication.

There's not even like a singular entity that's doing all the computation. So this idea of like some sort of sentience just doesn't make sense with LLMs. But I think the bubble is an LLM bubble. I think the bubble is going to be the idea that just LLMs by themselves is synonymous with AI. When the reality is when we look back is the breakthrough on LLMs, like, oh, these broke a lot of capabilities we weren't able to do before, helped reinite more interest in AI, got us out of the AI, but this idea that we stuck to the first thing, right? And I think it would be the equivalent of being in like 1994 and been like, man, this

internet thing looks really, I didn't really think about it until I encountered AOL. I'm putting all my chips into AOL, right? Because this thing, networking must be important. Let's invest all of our money in AOL. And in the end, it's like, no, that wasn't synonymous with the internet and had its own problems. The internet in general then developed, but not always in ways you predict. That's the problem with AI right now, is LLMs are cool.

but it's not synonymous with AI and we're in this weird loop of these companies trying to build the biggest possible LLMs with diminishing returns and burning so much capital to do so and inducing so many other people to inject so much CAPEX

in the data center building that it's going to obviously be like well that was a mania not that AI is a mania but this idea of like once LLMs re-ignited our interest in AI to say let's just do LLMs now let's just build those as big as possible

Let's just obsess over LLMs. It's incredibly myopic. Yeah. Well, on that happy note, Cal, where can people find you? Oh, God. I don't know. I don't know, Ed. I'll put the links in there.

I have a podcast. It's confusing for people. Here's one of my podcast is confusing, Ed, is because on Mondays, it's kind of like techno advice, Cal. Like, I'm giving people advice about using their phones less. And on Thursdays, I'm gearing up to do combat on AI hype. So look for the Thursday episodes. I guess you find me on YouTube.

I don't find my books. I don't know. Listen to both of them. And you can find me. You'll find me on a monologue this week on Friday. I don't know what it's going to be about because I come up with it. Not on the last minute because I'm lazy, but because I want it to be fresh. Maybe it'll be about Nvidia.

And last week I was like, yeah, Nvidia's earnings are this week. They weren't, but nevertheless, they will actually be this week. by which I mean the day this comes out. Thank you everyone for listening. I love you all. Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Mattosowski. You can check out more of his music and audio projects at Mattisowski.com.

M-A-T-T-O-S-O-S-K-I.com. You can email me at E-Z at Better Offline.com or visit Better Offline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat. where's your ed.at to visit the Discord and go to our slash better offline to check out our Reddit. Thank you so much for listening.

Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com or check us out on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. If your bookshelf and your for-you page are equally important to your personality, welcome home.

Pro Society is a weekly podcast that's part book club, part group chat for anyone who thinks pride and prejudice and love island deserve the same level of discourse. Each week, we're connecting the dots between books, the internet, and pop culture with your favorite writers, book talk creators, and plenty of overthought opinions. Yeah, I'm upset. I'm upset. Listen to pro society on the Iheart radio app, Apple Podcasts, or wherever you get your podcasts. On the new podcast, Solita, we share the messy reality of traveling alone as a woman. I can wait for four hours for the next bus or this random dude is offering me a ride on his motorcycle.

I chose option B. I'm Julie Pinero, and I travel by myself because it's a rare space where I can say yes without asking anyone else first. I'm on a mission to reclaim the word Solita, trading the pity for possibility. Listen to Solita on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. I'm Hoda Kotby, host of the podcast, Joy 101 with Hoda Kotby. know I'm always searching for inspiration and useful tools to maximize joy. Like when my friend Henry Winkler offered advice to aspiring actors and really anybody hoping to stand out.

You're not hired just to fill time and space. You're hired to fill the space with you. Listen to Joy 101 with Hoda Kotby on the IHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Our hometown is not a test tube. 90 miles northeast of Nashville, a battle for the future of America, plays out in one small town. Developers with right-wing ties have purchased hundreds of acres of land. We need cities on a shining hill. This is Our Town, a podcast about what happens when a small town becomes the site of a social experiment and fights back.

Guess you didn't move in on a bunch of dumb hillbillies now, did you? Listen to Our Town on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. This is an I-Heart podcast. Guaranteed human.