Technology, Connected - What Is the AI Alignment Problem? Yuval Noah Harari on When Machines Follow Orders Too Well

Episode Date: February 15, 2025

In Chapter 8 of Nexus, Yuval Noah Harari examines the AI alignment problem: how to ensure that artificial intelligence acts in ways that support human goals without causing harmful or unintended consequences.The central risk isn’t necessarily that AI will rebel. A system can follow its instructions accurately and still produce disastrous results because its objective is incomplete, poorly defined or detached from the context in which it operates.Mark and Jeremy discuss Harari’s examples of obedience, incentives and unintended outcomes, from Stalin-era loyalty tests to modern recommendation algorithms.In this episode, we discuss:What the AI alignment problem isWhy obedient AI systems can still cause harmHow Stalin’s applause test illustrates dangerous incentive structuresWhat Napoleon’s victories reveal about intelligence and long-term judgementHow the paperclip maximiser thought experiment explains misaligned objectivesWhy fixed rules such as Asimov’s Three Laws can’t resolve every ethical conflictHow social media algorithms manipulate attention and emotionWhy current recommendation systems already demonstrate alignment failuresWhether AI safety can be reduced to rules, constraints or technical safeguardsThe episode distinguishes between rogue AI and a more immediate problem: systems that pursue the goals humans give them without understanding the values, trade-offs and consequences behind those goals.AI alignment isn’t only about stopping machines from disobeying us. It’s about deciding what we should ask them to do, how those objectives should be interpreted and who bears responsibility when the result causes harm.Please enjoy the show.--Timestamps[00:00] Introduction: Books That Change Minds[01:04] Diving into Nexus Chapter 8[01:37] The Stalin Test: When Applause Becomes Terror[06:11] Evolution of AI Principles[07:45] Understanding the Attention Economy[08:45] How AI Targets Our Limbic System[09:29] Inside Facebook: The Leaked Reports[11:49] Napoleon's Warning for AI[15:55] The AI Alignment Problem Explained[17:49] Racing Against Time: Human Goals vs. Doomsday Clock[20:04] The Power of Divergent Thinking[21:50] Understanding Deontology in AI Ethics[26:55] Can Mythology Guide AI?[27:54] Exploring Inter-computer Realities[33:50] Why Asimov's Laws Won't Save Us[38:31] NPCs & The Future of Digital Consciousness

book lovers welcome to the thinking on paper book club i'm mark this is jeremy and every week hundreds of you thousands of you soon millions of you join us to read books that have stood the test of time books that will change your mind books like nexus by hulio otino books like the design of everyday things by don norman books like clear thinking by shane parish books like the order of time by carlo revelli books like the order of time by carlo rivelli books Like Quantum Supremacy by Michio Kaku, they are all past books in the book club, Thinking on Paper.xyZ to listen, learn and catch up on those. And thinking on paper.xyZ is where you can win a signed copy of Kevin Kelly's new book, Advice for Life. And if that wasn't enough, exclusive, rare VIP tickets to our recording of the interview with the man himself. April, so thinking on paper.xy, Z. But today, we're reading Nexus still by Yuval Noah Harare. We're on

Chapter 8. We're going to be speaking about the alignment problem into computer realities, the chaos machine, the Red Army, the theory of war, superintelligence, and how you, listener, might not think that destroying humanity to make paper clips is a good idea. but a computer just might. So as ever, Jeremy, chapter 8, over to you, first impressions. What a brilliant introduction, my friend, to what we're going to be talking about today. The first thing I learned and it happened quick in chapter 8, do not be the first guy to stop clapping.

And even if it goes beyond 10 minutes, it could be detrimental to your life. What a story to introduce the chapter. Could you pronounce his name? Because I was going to read a bit of that, but I always got stuck on Alexander Solzheniston. I don't know how to pronounce it. You probably have that better than I do.

Tell us that story because it's... And why does he start with that story? I think to me he starts with that story to... It goes back to his balance. The whole theme of this whole book has really been the balance. of order and truth, truth and order and the systems, the information systems that manipulate it one way or the other. And I think it was a pretty cool illustration.

I'm actually kind of flipping back to get, yeah, this was back. Well, he served as a half-it in the Second World War, didn't he for the Red Army. And he kept a diary, a journal of his experiences during that time. And the journal was found. and he speaks about one story recounts events at a district party conference in Moscow province in the late 1930s at the height of the Stalinist great terror. A call was made to pay tribute to Stalin and the audience who of course knew that they were being carefully watched burst into applause. After five minutes of applause, palms were getting sore and tarth arms were already aching. However, who would dare to be the first to stop?

So they carried on clapping. it went on and on for six minutes, then eight minutes, then ten minutes, and they didn't stop clapping until, well, they didn't stop clapping, until finally, after 11 minutes, the director of a paper factory took his life in his hands, stopped clapping and sat down. Everyone else immediately stopped clapping and also sat down. And that same night, the secret police arrested him and sent him to the gulag for ten years. His interrogator reminded him,

don't ever be the first to stop applauding. Wow, man. I mean, that's a crucial and disturbing fact about information networks and in particular surveillance systems. Loyalty tests. Loyalty test, right? And here's an interesting thing you goes into, too. They start initially probably clapping, some of them out of love or belief in whatever

this thing standing in front of them is. But as it goes on, as this loyalty test, gets pushed and pushed time-wise, that turns from, you know, love or admiration into terror. Like, you're literally clapping because you don't want to be killed. And so many different situations, we were talking about that. And recently, actually, just last week, we were talking about, you know, typewriters and writers and handwriting and Chowcesterscu in the 70s. I mean, all of this stuff, like, I keep telling everyone outside of this book club that I'm reading this book and what we're seeing in the states, and I'm not comparing immediately what's going on with the power regime

in the states to Chow Chescu and all of that horrible stuff. But there are sprinkling tendencies of like, ooh, I'm starting to recognize some patterns based on this book. So the timeliness of this read is uncanny. Okay, then. Link the Red Army and the surveillance economy to the the network AI economies of 2025 why is it relevant? Why is it

why is it why is it relevant to the story of AI because he says though the network failed to discover the truth about humans it was so good

at creating order that it conquered much of the world an analogous dynamic may afflict the computer networks of the 21st century which might create

new types of humans and new dystopia so essentially if the information network could replicate what happened. We just talked about it yesterday in our show with Boris and the other gentleman. Azuse. Yes. Anoush, Anush.

We talked about one of the news principles that we had, or the news items was Google's AI principles have interestingly changed a bit in all the language of those, all the language for using that technology to hurt people, to surveil people, to do inhumane things was pulled out of their principles. But like you mentioned, they have these beautiful butterflies

on the top of their graphics. So I guess that makes everything okay. Fortunately, we have Chinese open source AI systems, don't we? So we don't need Google. Deep seek. So I think all it all kind of builds to intercomputer realities, the alignment problem, the theory of war.

He goes again into, I don't want to dwell too much on the social media because in episode seven, we speak a lot about the social media algorithms, about Facebook and YouTube. He revisits that ground in Chapter 8. We get Brazil this time and YouTube and yeah, Bollasanare and how he was, he came from obscurity to be the president of Brazil, because of YouTube, or not because of YouTube, but in great part because of YouTube and the Brazilian Parliament ended up being filled with YouTubers. Crazy, right? No, I do. So, yeah, I don't want to dive too back, like you said, too much deeper into the social media stuff, but there are two things that I found really interesting that he pointed out that he didn't point out before. So algorithms see us as an attention mine.

Loose, loose quote, right? And kind of paraphrasing here, while encouraging the basis of our instincts while discouraging our potential, right? And then lastly, this idea of social media is not incentivized to interconnect our prefrontal cortices. Oh, yeah, that was a great quote. They're incentivized to connect our limbic systems, which is where all of our emotions lie. A lot of people, you know, everyone, including myself, struggles with when emotions are folded in, it turns into a reaction instead of a response, right? You know, if we're connected prefrontally, then maybe we can reason a little bit better.

But I thought that was a really, really interesting way to lay it out. The limbic system, we're taking us back to evolution to the planes to escape the marauding beasts. I'm going to read that quote because it was a great one. It was by, I think you pronounce that Wint-Tun, I reckon knows the P and the H is silent. So it's about the Myanmar tragedy and the Facebook algorithms, which partly caused that. I naively used to believe that social media could elevate human consciousness and spread the perspective of common humanity through interconnected prefrontal cortexes in billions of human beings. What I realize is that the social media companies are not incentivized to interconnect prefrontal cortexes.

Social media companies are incentivized to create interconnected limbic systems, which is much more dangerous for humanity. Yeah, one quick stat to you. There were some quotes from a leaked Facebook report, I think, that were pretty interesting talking about virality occurring for being encouraged for business reasons. and 64% of joins to extremist groups in 2019 were from Facebook recommendations. It's one of those stats where you can rip the data to pieces and boil it down until it's just a small residue and it's still ridiculously high. It's still a ridiculously stat, even if it's 80% wrong, it's still ridiculous. 100% right in being 80% wrong, hopefully. Yeah.

All right. So let's transition back to this alignment problem. So the last little touch on social media that I liked, the pollution of the information sphere. The pollution of the information sphere is an alignment problem where we reward A while hoping for B. The alignment problem, yes, reaching a goal with. I'm reaching a goal with a method from left field or taking a strange and non-human route, a surprise route, kind of will get to the paperclip example from superintelligence in a minute, which is a cracker.

But the alignment problem isn't new. And I hadn't really thought about it from the art of war. And it's not a new phenomenon. And do you mind if I speak about Napoleon? because I live in France. I'll let you do George W. Bush. And if I do... Oh, don't...

That is so unfair. Can I have Napoleon instead? Okay, you have Napoleon. No, I'm just kidding. Go ahead, please. So, the alignment problem, essentially, when what you do doesn't match the long-term goals of something else. And he speaks about Napoleon.

And history is full of decisive military victories that led to political disasters. the most obvious example was close to home napoleon nobody disputes the military genius of napoleon who was a master of both tactics and strategy but while his string of victories brought napoleon temporary control of vast territories they failed to secure lasting political achievements his military conquests merely drove most european powers to unite against him in his empire collapsed a decade after he crowned himself emperor indeed in the long term napoleon's victory ensured the permanent decline of france For centuries, France was Europe's leading geopolitical power, largely because neither Italy nor Germany existed as unified political entities. Italy was a hodgepodge of dozens of warring city-states, feudal principalities and church territories. Germany was an even more bizarre jigsaw, puzzle divided into more than a thousand independent policies,

loosely held together under the theoretical, whatever that word is, of the Holy Roman Empire of the German nation. long story short they all got annoyed Napoleon grouped together gang together and took him down and France has never been the same since that's a misaligned misalignment problem

that's really interesting too because the history on that you know Napoleon is of Corsican descent and if he you know again if his goals were more aligned with him getting Corsica

freed from French control, France could be the world's superpower right now, right? Like, I mean, I don't know, maybe, but like, it's really interesting that, like, that, that through line, that's why, that's why, you know, kind of not saying France is in a bad spot necessarily, maybe, I don't know. But one thing. It's got an alignment problem. Let's keep talking about that. Goals.

Let's talk about goals. because really what it's about actually isn't it that's a better way goal what's the goal what's your goal what's your short term goal

your long term goal and does it align with humanity's long term objective for survival well here's the thing too there are there's a probably a billion dollar

industry of consultants that help people find it and define their own goals because a lot of us don't give ourselves the time

in space to think about what's important versus what is urgent and what their goal is. What are they pointing to? What is the deeper meaning and that sort of thing? And, you know, a lot of that can stem from not wanting to hold time and space for that. Not to say that that's a laziness, but like a willingness not to lean in. So let's talk about it like a willingness not to lean in. So if humans themselves have trouble creating meaningful goals. And they're the ones directing AI with a specific goal. It's kind of scary to think that are we going to set that goal using an easy button? Are we going to be very thoughtful on defining what that goal is?

Because defining a goal to point the superpower of AI to is going to require maybe a little time and space, maybe a little opportunity to sit down and think on paper, think in your head, have thoughtful conversations, not have snap reactions and have responses. Like, that is, are we going to be able to do that or are we going to continue to push the easy button? Well, I'm going to answer that in a minute with Isaac Asimov and the three laws of robotics. But before we get to that, I just want to talk about the paper clip because I've read super intelligence. And anybody who's watching or listening, Nick Bostrum's superintelligence is a

awesome book and he paints a very bleak picture of what might happen in a world by superintelligence takes over the world. But there's a particular example of the alignment problem that he speaks about. It's very famous. It's about the paperclip and goals and what happens if you give a computer the goal to create more paper clips. And that's that's the goal. But the computer aligns not necessarily with the long-term goals of humanity. And what could happen in this thought experiment is that the computer goes to every length possible to create more paper clips, which eventually ends with destroying humanity, building factories to make more paperclips, taking over the world, building spaceships, going out into the universe,

terraforming of the planets to create factories, to create paper clips. And all it's doing is really, really. realizing its goals in the most optimal, efficient manner possible at the expense of all others. And it's just a superb example of the alignment problem. Very dystopian, obviously, very, you know. No, but the computer's probably going, yo, I'm doing a great job. Like, look at all these papercliffs I'm producing. But meanwhile, the computer has killed all of the people that are in control of land,

killed all the people that are in control of the resources needed to build paperclips. killed all of the people in control of getting rockets to the moon. But man, I have one goal. My goal, look on the paper, make the most amount of paper clips as I possibly can. Ultimate success story, yeah. Right. So, man, how scary is that? Like, if we don't give the right instructions,

this is that. Woo. You, Val, kind of, he speaks about how that could happen when, what is the ultimate goal. Humaneity has never been able to define what the ultimate goal is and there are no there's no kind of like communal agreement on what it would be and in the meantime the technology is just advancing and advancing

and we're not putting in these ethical philosophies, behaviours, whatever you call them, into the algorithms are they and you know, do need some more paperclips. So interesting you mentioned, like that this paperclip was the thought experiment. I'm going to tell you a quick little story. In my program, Right to Know You, I talk about convergent thinking and divergent thinking and this test that NASA did back in the 60s to find the most creative critical thinkers

to figure out why they're that way and also to kind of like say, hey, these guys might be the people we want to pull in to build right. pockets and all that stuff. So what they determined would they use something called the paperclip test. And I have a fun experiment with this that I do with my workshops, but you basically take a paper clip and you go, hey, Mark, how many uses can you come up with for this paperclip? And as adults, we're like, well, it's a paper clip. But when you're a kid, you look at the paperclip and you can be like, hey, man, that could be a key to a wormhole. That could be an alien framework to access humanity's secrets. It could be a sword. It could be, you know, whatever it is, right? So,

you know, in that experience, you're actually activating Divergent thinking. So they tested these kids when they were in kindergarten, 90 percentile in the genius category for Divergent thinking. They test them five years later. It's, uh, it's half of that. It's like 40 percent. They test them five years later. It's half of that, you know, less than 20 percent. And they test them when they're adults and according to the study 2% divergent thinking capable, which is scary. But the good news is you can activate divergent thinking. You just have to do it.

So I think we're all going to have to learn how to reduce that button if we want to create the right instructions for the computers that are going to be able to do things quicker, faster, cheaper than us. Divergent thinking. I see a callback to the Nexus by Julio Otino as well, thinking on paper. X, Y, Z for that one. I'm very fortunate to have a five-year-old son at the moment who asks some poses incredibly creative and ridiculous to me, beautiful, poetic, crazy questions and observations. And every time it does it, I'm trying to be more, more attentive to those

little moments. At least just like recognizing how incredible that question or observation is, and it's not something that I would ever do, but maybe I can kind of channel some of that. I got innocence. Here's your challenge, Mark, your challenge for this week. The next time you're in a car with your son, and you may be in some traffic or it's taking a long time to get there, pick a little object in the car,

whatever it is, could be a piece of plastic or whatever, and go back and forth with your five-year-old on defining what that piece of plastic could be, forgetting the laws of physics, for getting, you being an adult with, you know, structure and responsibilities. And that could be really fun. I did it with my son years ago. And we came up with like 60 ridiculous uses for this little plastic piece that we found in the car.

So challenge accepted. And I don't think I can wait till the car. So I'll do it at dinner tonight with a plate. Oh, perfect. So before we get to Asimov's three laws of robotics, do you have anything on? into computer realities, I think is the thing, but deontology and utilitarianism, do you have any philosophical knowledge to drop for our audience there, Jeremy? Well, just that he goes into these two potential constructs to help to present how people

think about defining goals, right, or defining what's important, what's meaningful, right? long-term human goal for the betterment and survival. Absolutely. So you have this deontology, which is this intrinsic good, belief in this intrinsic good, and this basically golden rule, due on to others kind of thing. And then Emmanuel Kant kind of brings in the idea of like any rule

that you want to make universal, right? So let's say Mark, like that film that you sent me, your Christmas card film, you're getting ready to murder somebody, okay? and you know you directing the field in Christmas movie

well you hey comment like subscribe send us an email we'll send you a link to the field in Christmas movie and you'll see exactly

what I'm talking about so Mark you're getting ready to commit murder and the thought experiment that you should be thinking about according to Kant

is would you want murder university allowed if you your family and all of that could be murdered and obviously like

most folks like oh okay yeah maybe murder's not good because I don't want to get murdered and I don't want my family to get murdered. So maybe I'm not going to murder somebody. But this is where it gets interesting. If you exclude a group from a definition of what it means to be human, it's a loophole that we've seen throughout history. They reference, you know, the Nazi terror and that sort of thing that basically they just were defining Jews as not human.

And, you know, one group defining another group is not human. And it's like, well, that's not really murder then, which is diabolical. But that's kind of how it goes. And these conflicts, here's a quote, conflicts concern the definition of identity, right? So there's this loophole and this deontology. And where he ties it into computers, how does a computer follow the golden rule? Is a computer scared of dying, right? So those are two really interesting things.

that, you know, probably not, but maybe as sentience becomes a further possibility. Say that again, sorry. Oh, just saying that our computer is scared of dying, you know, my MacBook isn't right today, but, you know, could that turn into a possibility, maybe. So let's flip the script on the other side, so the idea of utilitarianism is this second construct that he presents. So basically actions are judged by their impact to suffering and happiness. happiness is a bullshit word to me like it every time I hear it I'm just like I always think about I always think about this thing that's dangled in front of people as this pending future event that may happen I'll be happy only if I'll be happy when right so there's this whole happiness and joy discussion that we could we could talk about but that'll take take it's a different direction um but the goal is stay on target stay on target right the goal of you utilitarianism minimized

suffering, maximize happiness, but quote, we don't possess a calculus for suffering. So how do you, how do you assign misery points, right? How do you, how do you assign and quantify something like that? So utilitarianism, according to the author, works really well when there's a giant disparity between, you know, the misery and the happiness, right? But when it's kind of close together, it kind of ends up being a wash. And that thought construct you know, kind of falters a little bit. Lastly, the danger is, with utilitarianism, when a belief is strong enough

in a potential future state of something, damn the torpedoes, Mark, right? We get into a situation where it's like, hey, we're just pushing towards that future state, no matter what we do, example of the Crusades, right? Who cares about what happened? along the way as long as we get that end result. So that's what I took out of these two back and forth.

What I took out of all that, and I think you explained that very well, helping me understand it a bit more, is how, I go back to the doomsday clock. I always talking about the doomsday clock in this goddamn book club, and the doomsday clock is ticking, and the technology is advancing, and we can't even decide what those lofty, bigger goals are.

We've got AI companies who can't even have a conversation about what matters between themselves, because they're infighting, and then they're updating their websites and spending all their time and energy on putting little butterflies on their websites to make it look all peaceful and harmless and, yeah, and meanwhile. That's it. That's another shirt that we could do thinking on paper. Just put a butterfly on it. Well, and so it's not optimism. So Yuval, he speaks about mythology maybe as a way that we can kind of like in historically we've used mythology to create these interconnected realities where we believe in it is like this higher goal.

But he says that mythology won't work. They're not conscious. Obviously, they don't believe in anything. Of course, quote, they cannot believe in any mythology because they are non-conscious entities that don't believe in anything. As long as they lack subjectivity, how can they hold intersubjectives belief? However, one of the most important things to realize about computers is that when a lot of computers communicate with one another,

they can create intercomputer realities. analogous to the intersubjective realities produced by networks of humans. These intercomputer realities may eventually become as powerful and as dangerous as human-made intersubjective myths. This is a very complicated argument, but it is another of the central arguments of the book. So let's go over it carefully. First, let's try to understand what intercomputer realities are. What's an intercomputer reality, Jeremy? All right. For the younger folks out there, intercomputer reality would be Fortnite.

It would be any kind of massive online multiplayer game, right? Bits and bytes. We're in this. I'm seeing my view. Mark seeing his view. There's a rock. There's a mountain. We're both seeing the mountain. The mountain's not real. There's no atoms in the mountain. We can't go touch the mountain. But it's a reality created by bits, bytes, transmitted over network, pulses of light, all of that good stuff. But what else that can be, he references Pokemon Go, where it starts to bleed into reality a little bit. So Mark and I are running around, you know, San Francisco. And he has, I don't have a, I have a phone. He doesn't have a phone because he hates iPhones. I've got mine.

And I'm looking at it. And I can see this Pokemon, Pokemon that I want to capture. Mark's like, what the hell are you doing? Why are you running after stuff? And I'm like, Mark, here's an intercomputer reality that I'm chasing. That's another example. Did you play Pokemon Go? I didn't at all.

I had one guy, this is one buddy of mine, I was in a band with him, like, in college. And he collected Pokemon cards. And he was like massively into them. And I just, I couldn't, I didn't understand, I don't understand the allure. I had no trouble understanding into computer realities. It almost felt horribly natural and normal and just an extension. And I just, it will be. I mean, yeah, it was, I had no trouble understanding that.

Computers creating their own realities amongst themselves and then humans using those computers, buying into those realities. Yeah, I have no challenge. Financial markets, we talked about this before. Like, part of the reason, you know, the banking crisis with the credit default swaps, the CDOs, all of that stuff, those two, there were maybe like, what, four people who understood the math enough behind them to understand what they are and the rest of us had no idea. The regulators had no idea. It was too, it was a, it was a intercomputer reality

that was loosely managed by a couple of people that created massive chaos, massive havoc. Less, less nefarious example, but still pretty, pretty crazy is just web searches. So, You know, if you look up Markfielding.com or look up Jeremy Gilbertson.com. ExyZ. Oh, Markson, XYZ. Okay. Or thinking on paper. Or if you do a search, right, there is a computer making a decision on whose website is better.

It's an intercomputer reality of the best website based on parameters, based on algorithms and that sort of thing. what happens is we believe it and we buy into it and we go oh that that's the best web part that's the most popular website that is where I should be going thinking, reading, whatever it happens to be trust in shortcuts trust in shortcuts

and the intercomputer realities obviously I don't think we need to go into this check out all the other episodes but everybody is biased all database is our biased all information on the internet

is biased. Everyone has an agenda. Everyone is biased. And so what happens with these intercomputer realities as I see it is that you just end up with a lot. As we demonstrate throughout history with our own intersubjective realities, they will replicate and create their own intercomputer realities which are biased distorted and go off in many different directions, some good, some bad, some indifferent and that is so bias bias let's let's talk about bias

just for a second right and what kind of lesson this could be with with interaction on Twitter specifically but you know Microsoft created the chat botte a while back it was on Twitter engaging on Twitter it only took it only took 16 hours to become a

misogynist and a racist 16 hours on Twitter So friends and neighbors This is this is about today It would take about 20 minutes This is almost So think about this Like all right

You're sitting on the edge Of a really busy highway And you see You see a frog jump out On the busy highway And get smushed and crushed And you're like Huh

Maybe I won't step out On that busy highway To get smushed and crushed Take that analogy To like You know 16 hours on Twitter What it could do to something

What is that doing to you on platforms like that specifically? I don't know. I look at that as a, there's some cause and effect there to me. Question, right. I want to talk about the three laws of robotics, Isaac Asimov, and close this off with the question for Jeremy. But first, quote, if our only rule of thumb is that every action must be aligned with some higher goal,

by definition there is no rational way to define that ultimate goal. How then can we provide a computer, network with an ultimate goal. It must never ignore or subvert. I will change that question to how do we make these things safe that the paperclip incident doesn't become a reality. So I'll let Isaac Asimov answer that first and then we'll say why that doesn't work and then I want your answer.

So do you remember Isaac Asimov's three rules, robotic laws? Number one, I don't think we can just interchange a robot with AI here. I don't think it matters because they'll end up being the same thing. A robot may not injure a human being or through inaction allow a human being to come to harm. Number two, a robot must obey the orders given it by human beings except where such orders would inflict with the first law. Number three, a robot AI must protect its own existence as long as such protection does not conflict with the first or second law.

If I believe AI, which I didn't know this, and so I'm dubious if this is actually true. But apparently Asimov added a later fourth law, a robot must not harm humanity or by inaction allow humanity to come to harm. And this law takes precedence over the other three. The critics say these are overly simplistic and lead to ethical dilemmas, that you need a more detailed application-specific. Number one problem with that, let's point back to utilitarianism, or not utilitarianism, let's point back to deontology, right? So this idea of can't hurt a human, if you can manipulate the definition of what it means to be human, then you can manipulate what can be harmed. And, you know, just like, just like while the butterfly is still on the label, we've sucked out very sneakly the ability to harm humans in a way, right? The way I think about this, Mark, is going back to our discussion about goals, right?

And being really thoughtful in setting out goals. And a lot of people think about goals from a business perspective. There are frameworks like smart goals where, you know, something measurable, achievable, whatever. There's like all of these different goal frameworks. And a lot of it, whenever I got pulled into those from a business perspective, I was always just kind of rushing to be like, oh, I'm going to hit my sales goal. I'm going to hit this. I'm going to hit that. But like really, really being thoughtful about these, I want to point back to the history of algorithms, which I thought was really interesting. So the beginning of algorithms, it was very prescribed and directed by the developer of the algorithm. So think about, have you ever heard of the phrase helicopter parent? Yep.

Okay. So you think about these helicopter parents that are like got the bubble around their kid and they're not letting them eat dirt. all of that kind of stuff, right? And then you have the idea of like free range kids where, you know, my fourth kid, he was just out in the yard like foraging by himself, right? So, so the first version, the helicopter parent is kind of like the algorithm developer, early algorithm developer. And the AI is like free range kids where you just put it out there and let them push and poke and eat dirt and, you know, talk to other things and I don't know. So where I'm getting with that is like, how do you prescribe and direct something that is the second piece of that, where the first is more directed? It's going to be hard to direct this stuff, I think.

Yeah. Free range chickens notoriously hard to corral. Here's an interesting, here's a thing. I had a note here. And, you know, as we get to the wrap up here, I wanted to bring this up. On page 296, there was a reference to create, you know, nobody can, you know, create an imaginary world to simulate all of these interactions

between things. I forgot what the reference was. I think, what was he talking about? To, please hold, please hold. June 96. Here we go. Nobody can, yeah, this was related to the job market, comparing the job market to chess. When you created the algorithm for chess, the computer chess player kind of plays against itself, right?

And kind of learns and figures stuff out, like machine learning style. But, you know, if you have a, if you have an AI-driven experience that helps, helps you hire people and fire people, you know, an HR kind of tool, you know, how do you, you can't, you know, you can't create this whole world of people and manage those interactions to, to get that data. But can you? I started thinking about this. Like, if we can create video games that have like all these NPCs, right, non-player characters that are, that are in place. And you can spin up multiple characters with slight variations of themselves. couldn't you create this data bank based on these program, different program pieces and parts,

different probabilities to actually get some data back to see what works and what doesn't? And what do you do with that data? Sorry? Oh, then you use that to like frame a model and train an AI on that data. Okay. Yeah. But that doesn't solve the problem with goals, though, does it? That doesn't save it from the bad.

decisions. Not goals, but it presents more interactions to learn from, I think, is the interesting piece of that that stood out to me. Last, I know thinking on paper, we like to connect the dots. We speak to people in AI and quantum and robotics and blockchain. And yesterday we had two very passionate guys on, and we're speaking about blockchain. And I just thought maybe blockchain is the answer, because they were talking about essentially making decisions in the future based on real-time information brought into the system is if-then-when systems maybe and we need to have a guest who can explain this and think about this for us properly but smart contracts are a way to control the AI if you're

using AI in the blockchain and blockchain the smart contracts are only executing if when conditions, is there a way that you can control the AI's goals that way? And I know I've not explained that very well because the thought just popped into my head. Well, we talked about the question I asked Boris about the World Economic Forum and people postulating in Davos as they tend to do about the killer use case for blockchain being. to control the data that's input and fed into machine learning models. So, like, as a smart contract is an automated decision-making process, right? It's an automated kind of it lets you do this and not do that. So that could, that could, it could be an automation mechanic, but you still have to have.

There's like future events though, like smart contracts that were executing at an in or in, in an unspecified time in the future based on outside conditions. Right, right. You automate that, but like you still have to put those parameters in. A human at this point still has to put those parameters in. It goes back to goals. And it goes back to like, all right, what is the overarching goal that all of our individual goals roll up to?

And he talks about that throughout like the whole chapter. And we, we don't have. Could the goals be changed? Why does it have to be one goal? Could it, could the goals, say, if you're doing it? doing application specific safety measures for the AIs. Could you, the goals don't have to be the same. The goals could change.

You could use the smart contracts to change those goals if when this happens. Yeah, I think that. The AI starts building paperclip factories. If the AI builds three paperclip factories, change the goal. I picture, I picture someone just at the very, the very end of like a giant plug. Just his job is just to sit there until someone goes, yo. shut it down and it's just and then we all reset

from an electrical infrastructure perspective and all the AI shut down and we can kind of go anyway we've been rambling we've been rambling let's land the plane we're looking into I guess the last question that will carry us into the next chapter how will humans be able to probe

and correct a mythology it doesn't understand that is a good question and I think it's going to be through the lens of totalitarianism and democracy and who owns the controls to those, very relevant for today's AI conversation. So join us next week for chapter nine. I think maybe the penultimate chapter of the book will be announcing our new book very soon. All our shows are available on thinking on paper.xyZ. Go there, like, subscribe and share with someone that you think

thinks on paper. It's not everybody. A lot of people want surface detail, a lot of people want memes, a lot of people want headlines. But you know that person who wants more send them away. Stay curious. Be disruptive. Keep thinking on paper. Bye-bye.