Making Sense with Sam Harris - #379 — Regulating Artificial Intelligence

Episode Date: August 12, 2024

Sam Harris speaks with Yoshua Bengio and Scott Wiener about AI risk and the new bill introduced in California intended to mitigate it. They discuss the controversy over regulating AI and the assumptions that lead people to discount the danger of an AI arms race. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe.   Learning how to train your mind is the single greatest investment you can make in life. That’s why Sam Harris created the Waking Up app. From rational mindfulness practice to lessons on some of life’s most important topics, join Sam as he demystifies the practice of meditation and explores the theory behind it.

Welcome to the Making Sense Podcast. This is Sam Harris. Just a note to say that if you're hearing this, you're not currently on our subscriber feed, and will only be hearing the first part of this conversation. In order to access full episodes of the Making Sense Podcast, you'll need to subscribe at samharris.org. There you'll also find our scholarship program, where we offer free accounts to anyone who can't afford one. We don't run ads on the

podcast, and therefore it's made possible entirely through the support of our subscribers. So if you enjoy what we're doing here, please consider becoming one. Well, I've been on the road. I just did a short retreat with my friends Joseph Goldstein and Dan Harris, where we did some meditation but also recorded some conversations. Those will eventually be available over at Waking Up. And I am still traveling, so I'll not be doing a long housekeeping here. I am resisting the tractor beam pull of politics at the moment. No doubt it will soon be all-encompassing. But today I am focused on

artificial intelligence and its attendant risks and the growing effort to regulate it, which remains controversial. Today I'm speaking with Scott Wiener and Yoshua Bengio. Scott is a member of the California State Senate, and he has introduced a bill, SB 1047, which aims to reduce the risks of the frontier models of AI, models bigger than any which currently exist. And if it passes, it will be an important piece of legislation. The bill has already passed the California Senate, and it's approaching an Assembly floor vote later this month. And joining Scott is Yoshua Bengio,

who's one of the leading lights of artificial intelligence. He's known for breakthroughs in deep learning and other relevant technology. He won the Turing Award in 2018, which has been described as the Nobel Prize for computer science. And he's a full professor at the University of Montreal. He's also a fellow of the Royal Society of London and Canada, a knight of the Legion of Honor of France, and has other distinctions too numerous to name here. One thing he is not is someone who is uninformed about the current state of the technology, as well as the prospects of making surprising progress toward artificial general intelligence. So we talk about AI risk,

artificial general intelligence. So we talk about AI risk, the strange assumptions of certain people, some of whom I've spoken with on this podcast, who seem to think there's really no serious risk to worry about, and who view any concept of regulation as premature and economically injurious. Anyway, fascinating topic, all too consequential. And now I bring you Scott Wiener and Yoshua Bengio. I am here with Scott Wiener and Yoshua Bengio. Scott, Yoshua, thanks for joining me. Pleasure. Thanks for having us. So I will have introduced you properly in my housekeeping, but perhaps you can each tell me,

I'll start with you, Scott, how have you come to focus on the issue of AI safety, which is what we're going to talk about? Sure. And again, thanks for having us today and for talking about this issue. So I have the great honor and privilege of representing the great city of San Francisco, which is really the beating heart of AI innovation, no offense to other parts of the world. And I'm really proud of that. And so I am immersed in the tech world in terms of just people who are around me in the community, ranging from high-level executives at large tech companies to startup founders to just frontline technologists, academics, investors, just the entire gamut.

And about a year and a half or so ago, folks in the community in the AI space started talking to me about the issue of the safety of large language models and started looking into it more, had a series of dinners and salons and meetings, started reaching out to a number of people and realized that this was an area that we should be addressing. And that's how it all started. Nice. Nice. Yashua? Yeah. After four decades of research in AI and contributing to the exciting advances in deep learning that have produced Gerard of AI as we see it today, I came to realize with the advent of ChatGPT that things were moving a lot faster than I and almost everyone in the field anticipated. And I started thinking about what this could mean for humans, for society, for democracy, if we continued on that trajectory towards human level or AGI. And I thought, well, society is not prepared for that. And we need to start thinking right now about mitigating the risks. We're going to talk about a bill that is in the process of finding its way toward possibly

becoming law, SB 1047. But before we jump into the details of regulating AI and why we might want to do that and how, Joshua, I thought you and I were talking offline about kind of where you stand on this continuum of concern. I mean, you're one of the leaders in this field, and I've spoken to many people, you know, both inside the field and, you know, at some part of its periphery, who have focused on this question of AI safety. And there's a wide range of attitudes here. And I would say on the far side of freaked out, you have someone like Eliezer Yudkowsky, who's been on the podcast before. Also someone like Nick Bostrom, whose book Superintelligence was very influential, and I've spoken to him as well. And then on the far side of insouciant and, to my eye, utterly in denial that there's any possible downside here,

you have people like Rodney Brooks, the roboticist, and Marc Andreessen, the venture capitalist. Rodney hasn't actually been on the podcast, but I debated him at an event, and Mark has been here. And then in the middle, you have someone like, not quite in the middle, but at a place that really has always struck me as very rational and still worried, is someone like Stuart Russell, the computer scientist at Berkeley. So I'm wondering, can you locate yourself on this continuum, or are you in some other spot in the space of all possible attitudes? So in a way, I'm looking at this scene, and no one can honestly say what scenario is going to unfold. The scientists among themselves disagree, but there are enough people who believe that the risk is potentially catastrophic, and could be just a few years, but it could also equally be decades. We don't know.

So there's enough uncertainty and enough potential level of harm that the rational thing to do is to consider all of these scenarios and then act accordingly, according to the precautionary principle. In other words, well, we need to be ready in case it happens in, so I don't know, 2030, that we get human level or worse, I mean, even superhuman. And we need to be prepared in case we don't handle it well. And companies haven't found a way to make sure AI cannot be misused by bad actors in catastrophic ways, or companies haven't figured out how to control an AI so that it doesn't turn against humans. So all of these catastrophic possibilities, right now, we don't have the answers. possibilities. Right now, we don't have the answers. And so the sort of rational thing to do here is to work to mitigate those risks. So that means understanding those risks better,

rather than denying them, which is not going to help to find solutions, and putting in place the right protection for the public in case these potentially catastrophic things are more dangerous or shorter term than many people might expect. So I'm really in the agnostic camp, but rational, meaning we have to actually do things in order to avoid bad scenarios. Right. And would you acknowledge that there are two, broadly speaking, two levels of risk here? There's the near-term risk of, really, that I think we see already, even with so-called narrow AI, where it's just the human misuse of increasingly powerful tools, whether it's just to derange our politics with misinformation or cyber warfare or any other malicious use of increasingly powerful AI. And then we tip over at some point, provided we just continue to make progress,

into what you seem to be referencing, which is more often thought of as the problem of misaligned AI, the so-called alignment problem, where we could build something that is truly general in intelligence and more powerful than ourselves cognitively, and yet we could build it in such a way, whereas it would be unaligned with our, you know, ongoing happy cohabitation with it. Yes, there are currently no defensible scientific arguments that neither of these are possible. So we, and I want to make a little correction to the risks that you described, because even if we find a way to create AI that is controllable, aligned, and so on, it could still become dangerous in the wrong hands. First of all, these safety protections,

if you control the system, you just remove those safety protections so that the AI will do bad things for you. Because you have to understand how AI works. AI is about how to achieve goals or how to respond to queries using knowledge and reasoning. But really, who decides on the goals? That normally is the user. And if we have safety protections, maybe we can filter goals that are not acceptable. But of course, the humans could still do bad things. So even if we go to superhuman AI, if it's in the wrong hands, we could end up with a world dictatorship, right? And that's very bad. Maybe not as bad as human extinction, but it's very, very bad. Clearly, we want to make sure we don't

let anything close to that happen. Joshua, one more question for you, and then I'll pivot to the bill itself. But what do you make of people who have a similar degree of knowledge to your own, right? People who are in the field, out of whom you get more or less nothing but happy talk and dismissals about these concerns. I mean, there are people like, perhaps I'm not being entirely fair to everyone here, but someone like Jan LeCun or even Jeffrey Hinton. He's had an epiphany which has caused him to be quite worried and voluble on this topic in public. But for the longest time, you know, here we have the, you know, one of the true patriarchs of the field kind of moving along and making progress and not seeming to anticipate that he might wake up one day and realize, well,

wait a minute, we're on course to build something smarter than we are. This entails the possibility of risk. How is it that there's a diversity of opinion among well-informed people that there are any significant risks at all to worry about? So it's interesting that you're talking about Jeff Hinton and Jan LeCun because the three of us are good friends. And of course, Jan and I, I mean, Jeff and I kind of agree, and Jan doesn't, about the risks. So Jeff and I independently shifted our views, like really pivoted around January or February 23, a few months after ChatGPT became available. And our views before that were that, oh, human-level intelligence is something so far into the

future that we could reap benefits of AI well before we got there. But what happened with chat and GPT is realized that, well, things are moving a lot faster than we thought. We now have machines that essentially pass what is called the Turing test. In other words, they master language as well as humans. They can pass for humans in a dialogue. That's what the Turing test is. And so our time lads suddenly shifted down to anything from a few years to a few decades, whereas previously we thought it would be like centuries or decades. So that's really the main reason we shifted. So why is it that...

So the crucial variable was the change in expectation around the time horizon. Yes, yes. And in fact, if you dig and try to understand why most scientists disagree on the risk, it's because they disagree on the timeline. But my position is we don't know what the timeline is. Okay, but you also asked me about like, why is it Jan LeCun, who's like basically at the same level of proficiency in the field as Jeff and I, why is it that he continues to think that we shouldn't worry about risks? It's an interesting

question. I think I wrote about it in my blog. By the way, my latest blog entry goes through pretty much all of the criticisms I've read about taking risks seriously and rationally trying to explain why. Because of our uncertainty, the scientific lack of knowledge, we really need to pay attention. But I think for many people, there's all kinds of psychological biases going on. Imagine you've been working on something all your life, and suddenly somebody tells you that it actually could be bad for democracy or humanity. Well, it isn't something you really want to hear. Or if you're rich because of working in a field that would eventually bring really dangerous things to society, well, maybe this is not something you want to hear either.

So you prefer to go to something more comfortable, like the belief that it's all going to be all right. For example, Jan is saying, he's agreeing actually on almost everything I'm saying. He just thinks that we'll find a solution beforehand. And so don't worry. Well, I would like that to happen, but I think we need to proactively make sure we do the right thing, we provide the right incentives to companies to do the right research, and so on. So yeah, it's complicated. Wouldn't he admit that a completely unregulated arms race among all parties is not the right system of incentives by which to find a solution?

No, he wouldn't. He thinks that it's better to let everybody have access to very powerful AI and there will be more good AIs than bad AIs. But that isn't rational either. In a conflict situation, you have attackers and defenders. And depending on the attack threat, it could be that there is an advantage to the attacker or there could be an advantage to the defender. In the case of AI, it depends on which technology the AI is used for to attack, say, democracies. An example is cyber attacks. Cyber attacks, it's hard for the defender because you have to plug all the holes, whereas the attacker just needs to find one hole. Or bioweapons, like if an attacker uses AI to design a bioweapon, the attacker can work for months to design the bioweapon, you know, the attacker can work for months to

design the bioweapon, and then they release it in many places in the world, and then people start dying, and it's going to take months at least to find a cure during which people are dying, right? So it's not symmetrical. It's not because you have more, like, good people controlling AIs than bad ones that the world is protected. It just doesn't work like this. controlling AIs and bad ones, that the world is protected. It just doesn't work like this. Yeah. Yeah. Okay. So let's dive into the possibilities of controlling the chaos. Scott, what is this bill that has various technologists worried? Yeah. So Senate Bill 1047 has a fairly basic premise that if you are training and releasing an incredibly powerful model, which we define as exceeding 10 to the 26th flop, we've also added in that you've spent at least $100 million in

training the model, and that'll go up with inflation, that if you are training and releasing a model of that scale, of that magnitude, of that power, perform reasonable safety evaluations ahead of time. If your safety evaluations show a significant risk of catastrophic harm, take reasonable steps to mitigate the risk. This is not about eliminating risk. Life is about risk. It's about trying to get ahead of the risk instead of saying, well, let's wait and see. And after something catastrophic happens, then we'll figure it out. That's sort of the human way of things at times. Let's get ahead of it. And what's interesting here is that all of the large labs have already committed to doing this testing. All of their CEOs have gone to the White

House, to Congress, most recently to Seoul, South Korea, and have sworn up and down that they either are doing or they will be doing this safety testing. And the bill doesn't micromanage what the safety testing will be. It provides flexibility. It's light touch. But now we have people coming forward and saying, oh, wait, we know we committed to it or they committed to it, but don't actually require it. And that doesn't make sense to me. And so that's the heart of the bill.

There are some other aspects that if a model is still in your possession, you have to be able to shut it down and a few things like that. And maybe I can make a connection here between that answer and the previous question. So there are people, as you said, Sam, who don't believe that there is any risk. And of course, if we only went by their choices, they wouldn't do the right thing in terms of safety. So it's not enough to have these commitments. We need to make sure that everyone

actually does it. And that is why you need laws. Voluntary commitments are great. And they're mostly committed, but we need to make sure it actually happens. Yeah. And I agree with that. And we've seen in a lot of industries that voluntary commitments only get you so far. And even if all of the current leadership of the labs are fully and deeply committed to doing this, and I take them at their word and that they're acting in good faith, we have no idea who's going to be running these labs or other labs that don't exist yet two, three, five years from now and what the pressures are going to be. The other thing I just want to add, which in this bill, we've had some critics

of the bill that have really engaged with us in good faith, and we're so appreciative of that. There are other critics who it's, there's a lot of whataboutism. And one of them is, well, what about other risks and other technology that causes risk? Okay, yes, there are other technology that could cause risk, but we're focused on this very real intangible potential risk. but we're focused on this very real intangible potential risk. And the other argument that they sometimes make, and I really find this a little bit offensive,

dismissing anyone who raises any question or concern about safety, saying you're a doomer. You're a doomer. You're part of a cult. It's a cult-like behavior, and you're a doomer. You're a doomer. You're part of a cult. It's a cult-like behavior. And you're a doomer. And you're an extremist. And by the way, none of these risks are real. It's all science fiction. It's all made up. And my response to them is, well, first of all, not everyone's a doomer just because you care about safety or want to maybe take some action around safety. But if you really believe that these risks are fabricated, made up, just pure science

fiction, then why are you concerned about the bill? Because if the risks are really fake, then if you really believe that, then you should also believe that the bill won't cover anything because none of these harms will ever happen and it's all science fiction. And so the fact that they are fighting so hard against the bill, led by A16, the fact that they're fighting so hard against it sort of really contradicts their claim that they believe that these risks are science fiction. So I can imagine that someone over at Andreessen Horowitz would say that, first of all, it's going to pose an economic and legal burden to any company in this business,

and there's going to be capital flight out of California. I mean, people will just do this work elsewhere because California has become an even more hostile place for business now. It has become an even more hostile place for business now. And so is there something about, if we were going to give a charitable gloss of their fears, what's the worst case scenario from their point of view that is actually honestly engaging with what you intend, right? So like what kind of lawsuits could bedevil a company that produces an AI that does something bad? Just what kind of liability are you trying to expose these companies to? And just how expensive is it in time or resources to do the kind of safety testing you envision, Joshua? Well, they're already doing it. I mean, at least many of them are since they committed to the Biden executive order. And they have been doing these tests since then, or even before in some cases. And it's not hugely expensive. So in terms of the liability, I think if they do sort of reasonable tests that one would expect from somebody who knows about the state of the art, then they shouldn't worry too much about liability. in advance. I understand worrying about that. But just on the safety testing, have any of these companies that have agreed to persist in testing disclosed what percent of their budget is getting

absorbed by AI safety concerns? I mean, are we talking about a 5% spend or a 40% spend or what is it? Do we know? We think it's very small, especially in the, it's not, I think it's more like the, not less than 5%. It's a few percentage points. And so we think it's about two to 3% as far as we can tell. And so it's, you know, and again, this is the large labs. It's when you're spending at least a hundred million dollars to train. This is not about startups. I understand there are startups that have concerns because they want to make sure they have access to Lama and other open source models. But in terms of who's going to have to comply with this, it's not startups. It is large labs that are spending massive amounts of money to train

these models. And they are absolutely able to do it. And they have all said that they either are doing it or are committing to do it. So it's really interesting to me that you have a large lab saying that they're committing to doing it or already doing it. And then you have some investors, most notably A16, saying, oh, it's all made up. The safety testing is not real. It's impossible. And so it's like, okay, well, which is it? And they say that they're already doing it. Okay. So let's say they do all of this good faith safety testing, and yet safety testing is not perfect. And one of these models, let's say it's Chad GPT-5, gets used to do something nefarious, you know, somebody

weaponizes it against our energy grid and it just turns out the lights in half of America, say, and when all the costs of that power outage are tallied, it's plausible that that would run to the tens of billions of dollars

and there'd be many deaths, that would run to the tens of billions of dollars and there'd be many deaths, right? I mean, what are the consequences of turning out the lights in a hospital or in every hospital in every major city in half of America for 48 hours? Somebody's going to die, right? So what are you imagining on the liability front? Does all of that trickle up to Sam Altman in his house in Napa drinking white wine on a summer afternoon? What are we picturing here? Yeah. So, well, under this bill, if they've done what the bill requires, which is to perform the safety evaluations and so forth, if they do that, then they're not liable under this bill. Again, it's not about eliminating risk.

So companies, labs can protect themselves from the very focused liability under this bill, this bill, which first of all, is not dependent on training or releasing the model in California or being physically located in California, which is why this whole claim that labs or startups are going to leave California. If you are training and releasing your model from Miami or from Omaha, Nebraska, if you are doing business in California, which they all will be, it's the fifth largest economy in the world, it's the epicenter of the technology sector, unless you're going to not do business in California, which I'm highly doubtful of, you are covered by the bill. And only the attorney general can file a lawsuit. And it's only if you

have not complied with the bill and one of these huge harms happens. One thing that the opponents of the bill continue to just refuse to acknowledge is that there is liability today much broader than what is created by SB 1047. on, you release that model, and then that model somehow contributes to a much smaller harm than what we're talking about here, burning down someone's house, doing something that harmed someone. That person can sue you today under just regular tort liability law in California, and I assume in all 50 states. They can sue you today. That liability will be disputed and litigated. And I'm sure in the coming years, the courts are going to spend a lot of time sculpting what the contours of liability is for artificial intelligence. But that liability risk, that litigation risk exists today in a much broader way than what SB 1047 provides. And that's why the reaction to the liability aspect

of this bill, I think is overstated. And then on top of that, they keep spreading misinformation that model developers are going to go to prison if your model contributes to harm, which is completely false and made up. Interesting. So why do this at the state level? As you've indicated, there's already movement at the federal level that the Biden administration has made similar noises about this. Why shouldn't this just be a federal effort? Well, in an ideal world, it would be a federal effort. And I would love for Congress to pass a strong AI safety law. I would also love for Congress to pass a federal data privacy law, which it has never done. I would also love Congress to pass a strong net neutrality law, which it has never done. And so as a result,

I authored California's net neutrality law six years ago, and we also passed in California a data privacy law. I would love for all of that to be federal, but Congress, with the exception of banning TikTok, has not passed a significant piece of technology legislation since the 1990s. That may change soon with this child protection social media law, we'll see. But Congress has what can only be described as a poor record of trying to regulate the technology sector. So yes, it would be great for Congress to do it. I'm not holding my breath. The Biden executive order, I like it. I applaud it. It's an executive order. It does not have the force of law. And the Republican platform has stated, Donald Trump's platform states, that that executive

order will be revoked on day one if Donald Trump is elected president, God forbid. Does this have any effect on open source AI, or are you just imagining targeting the largest companies that are doing closed source work? The bill does not distinguish between open source and closed source. They're both covered equally by the bill. We have made some amendments to the bill in response to feedback from the open source community. One change that we made was to make it crystal clear that if a model is no longer in your possession, you're not responsible to be able to shut it down

because that was some feedback we had received that if it's been open sourced and you no longer have it, you are not able to shut it down. So we made that change. We also made some changes around clarifying when a model, say that is open source, is no longer a derivative model. In other words, there's enough changes or fine tuning to the model that it effectively becomes a new model at a certain point, and that the original developer is no longer responsible

once someone else has changed the model sufficiently. That changer, the person fine-tuning, would then become effectively the person responsible under the law. I'd like to add something about open source. So you have to remember there's this threshold which can be adapted in the future, the cost or the size of these models. And most of the open source that is happening in academia and startups that are generated by these companies or these universities, they're much smaller because they don't have $100 million to train their system.

And so all of that open source activity can continue and not be affected by SB 1047. Yeah. Did you say it was 10 to the 23rd, 10 to the 26th? 26. That's floating point operations per second? Is that the measure? Yes. Floating or integer. So how big is that in relation to the current biggest model? So ChatGPT 4.0? It's above all of the existing ones. Okay. So everything that we currently have, the best LLMs, haven't yet met the threshold that would invoke this regulation.

Yeah. So this is only for the future models that at least we don't know about yet. And there's a good reason for that, because the models that harbor the most risks are the ones that haven't been played with, haven't been made available. So there's a lot more unknowns. So it does make sense to focus on the frontier systems when you're thinking about risks. When you're thinking about the frontier, doesn't that play both ways in the sense that critics of this regulation, I can imagine, and certainly critics of the kinds of fears you and I and others have expressed about AGI, artificial general intelligence, would say and have said that we simply don't know enough

to be rationally looking for any sort of break to pull or any, you know, safety guidelines to enshrine into law. I mean, I'm thinking of, I think it was Andrew Ng who once said, you know, worrying about artificial general intelligence is like worrying about overpopulation on Mars, right? Like it's just, it's so far, again, this invokes the timeline, which you and many other people now think is far shorter than assumed there. But it's not just a matter of time. It's just that the architecture of the coming robot overlord may be quite different from what we're currently playing with, with LLMs. Is there any charitable version of that that we could prop up? It's too soon for us to be drawing guidelines because we simply don't know enough. Okay, I have several things to say about this. First of all, if we're worried about the short-term possibilities, like say five years or something, or 2030, then it's very likely that it's going to be something very close to what we have now. If you'd like to continue listening to this conversation, you'll need to subscribe at SamHarris.org. Once you do, you'll get access to all full-length episodes of the Making Sense

podcast. The podcast is available to everyone through our scholarship program. So if you can't afford a subscription, please request a free account on the website. The Making Sense podcast is ad-free and relies entirely on listener support. And you can subscribe now at samharris.org.