Your Undivided Attention - ‘We Have to Get It Right’: Gary Marcus On Untamed AI

Episode Date: September 26, 2024

It’s a confusing moment in AI. Depending on who you ask, we’re either on the fast track to AI that’s smarter than most humans, or the technology is about to hit a wall. Gary Marcus is in the latter camp. He’s a cognitive psychologist and computer scientist who built his own successful AI start-up. But he’s also been called AI’s loudest critic.On Your Undivided Attention this week, Gary sits down with CHT Executive Director Daniel Barcay to defend his skepticism of generative AI and to discuss what we need to do as a society to get the rollout of this technology right… which is the focus of his new book, Taming Silicon Valley: How We Can Ensure That AI Works for Us.The bottom line: No matter how quickly AI progresses, Gary argues that our society is woefully unprepared for the risks that will come from the AI we already have.Your Undivided Attention is produced by the Center for Humane Technology. Follow us on Twitter: @HumaneTech_ RECOMMENDED MEDIALink to Gary’s book: Taming Silicon Valley: How We Can Ensure That AI Works for UsFurther reading on the deepfake of the CEO of India's National Stock ExchangeFurther reading on the deepfake of of an explosion near the Pentagon.The study Gary cited on AI and false memories.Footage from Gary and Sam Altman’s Senate testimony. RECOMMENDED YUA EPISODESFormer OpenAI Engineer William Saunders on Silence, Safety, and the Right to WarnTaylor Swift is Not Alone: The Deepfake Nightmare Sweeping the InternetNo One is Immune to AI Harms with Dr. Joy Buolamwini Correction: Gary mistakenly listed the reliability of GPS systems as 98%. The federal government’s standard for GPS reliability is 95%.

Hey, everyone. It's Daniel. Before we start, two announcements. Our very own Tristan Harris is going to be in conversation with Yuval Noah Harari, the evening of Thursday, October 3rd, at the Meyer Theater in Santa Clara, to discuss among other things, Yuval's new book, Nexus. If you're interested, see the link in the show notes. As a special gift to listeners, you can use the code CHT at checkout for $15 off tickets. And a quick reminder that we want you to send us your questions. Anything you've been wondering about? Please tape yourself on your phone and send us the voice memo. at Undivided at HumaneTech.com. Thanks. It's a confusing moment in AI. On the one hand, it can feel like we're on a runaway freight train. Companies are pouring billions of dollars into the technology, promising new models every few months that are vastly more powerful than the ones before.

Some experts in Silicon Valley predict that we'll have artificial general intelligence, AI that's smarter and more capable than most humans, in just a few years. The hype is massive, and the promises are huge. On the other hand, the AI that we use today is often full of hallucinations and other unexpected behavior, and those same AI companies have seen their stocks take nosedives in recent weeks. And you have equally knowledgeable experts predicting that we're about to hit a wall in AI model capabilities,

that the bubble will burst, leaving society with a half-usable technology and so many broken promises. One of those people is Gary Marcus. Gary's a cognitive psychologist and computer scientist who's become a least, leading voice in the public AI debate. He built his own successful AI startup, which he sold to Uber. And recently, he's argued on his substack and in the press that the AI race we're seeing today is going to slow down considerably by the end of the year. He says we've seen the end of exponential growth in the power of large language models.

But in his new book, Taming Silicon Valley, how we can ensure that AI works for us, Gary argues that regardless of how quickly the technology progresses, our society is woefully unprepared for the risks that will come from the AI that we already have. And here at CHT, we couldn't agree more. That's why I'm so excited to have Gary on the show today to talk about this head-spitting moment in AI. As you'll hear, though, we don't always agree on the details, especially when it comes to questions of where this technology is headed.

We'll explore those disagreements today. So Gary, welcome to your undivided attention. Thanks, and that was a really lovely introduction. So I imagine that a lot of listeners are struggling to hold two things in their head at the same time, And I struggle with this too. On one hand, you have people in Silicon Valley who are really worried about the pace of AI development, that it's going to be smarter and more capable than most humans in just a few years.

And then we have other experts, including you, talking about the limitations of the fundamental AI technology, expecting that we're about to hit a wall. Help me with this. How do we square that circle? My view is AI will someday be much smarter than the smartest human. I think that's an inevitability. But it's not inevitable that it will happen in the next two years. or next five years, and I don't think that it will.

I think it's at least a decade away and probably longer than that. I come to this from the perspective of cognitive science and how the mind works, and I look at all of the things that a human mind needs to do, and making analogies as one, doing statistical analysis as one, but reasoning is another, for example. It took us, you know, 70 years of AI research to get to the point where we can really do well

a kind of statistical inferencing kind of thing. But there are other things that we just haven't made that much progress on, like reasoning from incomplete information. And there are ways in which current systems are just really, really bad. One of them is sticking to facts, and you can't be smarter than the smartest human and fail on basic facts. We still don't understand abstract knowledge, reasoning, and things like that.

We still don't know how to put those into our AI systems, and we will need fundamental innovation. Some people look at a curve, and they think it will follow indefinitely. and jaded people who have been around for a while realize curves don't always go that way. So Moore's Law went for a while, but it's not a law of the universe, and eventually it ran out.

And in a similar way, it looks like there was genuinely exponential progress between GPT and GPT2, between GPT2 and GPT3, GPT4. You could actually fit a curve and say what the exponent was, depending on what your measure is.

But that hasn't continued. and some people just sort of want it to continue and other people say yeah but it's been two years since GPT4 we haven't really seen something that fits with that and so that looks more like an asymptote where things are starting to be a point of diminishing returns rather than continuing exponentials. And the third thing is my view is the executives

all want to project very fast progress and some of the people in some part of the AI ethics world also want to project it. But if you look at from a cognitive science perspective, It's just not happening. Let's talk a little bit about where you sit in this AI debate. I think it's sort of fair to say that you're skeptical, a lot of the claims coming out of the big AI companies and their models.

And you've sometimes been called AI as loudest critic. But in your book and in your background, and when we talk, I see a more nuanced picture. In your book, you say, quote, I work on AI every day and I quarrel with those who wish to take risky shortcuts, not because I want to end AI, but because I want it to live up to its potential. What's your reaction to being called an AI critic, and why do you think you get that label?

That's a complicated question about information warfare. The quote that you just read is something I've also said on Twitter, but people ignore it when I say it on Twitter. And probably the reputation as loud as critic primarily comes from Twitter or X. I have repeatedly said there that I actually love AI. I have sometimes pointed to pieces of AI that I think are really great. But I don't like generative AI. I genuinely don't like generative AI. And for the last five years, that's almost all that anybody has wanted to talk about.

And it's very hard in a place like Twitter to make a nuanced argument. I have certainly tried and said, you know, my target here is generative AI, not all AI. I know one example of this that just baffles me almost is that the accelerationists all hate me. They all hate me. And they don't understand. I'm actually trying to accelerate AI. but I just have a different view. Their view is the way that you accelerate AI

is to have no regulation on it and to scale things as fast as possible. My view is that that would be a catastrophe for AI and for society. So if we scale the stuff we have right now, it just makes it more and more dangerous as a weapon of misinformation and cybercrime and so forth doesn't solve the truthfulness problems.

Is it a disaster waiting to happen? And if we have no regulation, then the disaster really will happen, And then there'll be an enormous backlash against AI. And so, like, if you want AI to succeed, you have to realize that there has to be a regulatory environment for it. And it has to be done right. I mean, imagine, like, what would have happened

if we had stuck to Durigible after the Hindenburg crash because there had been a bunch of dirigible accelerationalists shouting down the other people who'd be more like me saying, you know, maybe we should try a different approach, maybe we should have some regulation, maybe we shouldn't build them this big unless we know how to put out the fires, and they'd be like, you bastards,

you know, why are you slowing us down? And in fact, that would never have worked, right? Eventually, the public would have lost heart. In fact, they did lose heart after the Hindenberg. And so we're going to wind up, I think, most likely, in fact, with something similar, which is going to lead to a pause on AI because the public gets really sick and tired

of seeing what they're seeing, and because the industry itself is not taking, I think, appropriate responsibility. There is a different scenario, I think a more utopian scenario where we look at both the moral and technical failings of current AI, of generative AI, and we build something better by improving the technical stuff, which I think requires a different approach than we're taking right now, and by having some kind of regulatory framework, you know, look at airplanes, like the reason commercial airlines are so safe is because we have multiple layers of regulation, it's very well thought through and so forth, and we're going to need something similar for AI. And if we get to that, then I think there's a chance we will get to the Utopia, where AI helps with science, medicine, agriculture, and all of those kinds of things that we have long talked about. But we have to get it right.

You could make an analogy, by the way, to driverless cars. Like, if you just said tomorrow, hey, Tesla, just put this stuff on the road, you know, let anybody use it, take humans out of the loop. There would be an enormous number of accidents, and there would be congressional investigations, and the whole thing would be shut down. But generative AI has been a little different. It's just been thrown out there to the entire world, and it is, in fact, causing problems. There is, in fact, a developing backlash.

The patience that we've seen in the driverless car industry has not been there. The hard part in talking about AI is it's one of those catch-all terms like sports, right? It means so many things to so many people. And there are so many different kinds of AI with different strengths and different weaknesses. Of course, the one that's getting all the attention now is generative AI, like ChatTPT and Gemini and Claude. So let's really start there. In your view, the tech industry is putting way too much attention in energy. on generative AI, but you point out that it has all of these unique weaknesses.

Can you talk a little bit about what those weaknesses are and how they show up? Yeah, maybe even as level-setting as they sometimes say in the business, let's just be clear. AI is incredibly useful. So, you know, nobody's getting the press that generative AI is, but generative AI is just one small part of AI. You know, we also have, for example, the GPS systems that help us navigate in cities we don't know, that work fabulously well and that I'm not complaining.

about. It's generative AI in particular that I think is quite flawed. I'll give you some examples of why and so forth, but has also somehow gotten all of the money and all of the attention. I think it's gotten all the money and all the attention because it's anthropomorphic in a way

that others aren't. So people don't really fall in love with their GPS nav systems. There was a little period a few years ago where you could use a voice and people kind of like that, but they don't really fall in love with the system. They really have many people, not all, have fallen in love with generative AI. They relate to it as a person.

And from a technical perspective, it just isn't reliable. So, you know, you can compare. Calculator is 100% reliable within its limits. GPS system is really like, you know, 98% reliable within its limits. Generative AI promises

to answer literally any question about the world. It almost never says, I don't know, or that's not in my scope. There's a few tiny circumstances where I'll do that. So basically, it promises the moon, but it can't really deliver. On almost any question that you might ask,

it might be right or it might make something up. And it sucks because you actually get different answers on different occasions, and it depends on the exact wording and the exact context. And so there's no inherent reliability in the system. It's really not very good. It's not what we dreamed of with AI. I mean, you wouldn't want the Star Trek computer

to be like 50-50 or ring your dealer to see what you get. That's an old alley-G joke. Well, I hear that. I mean, I really do hear that, right? But, you know, if there is an analogy to generative AI, it is sort of that we've developed machine intuition. And like our intuition, it's inscrutable. You can't look at it and understand it.

It sometimes gives you really wrong answers, also dependent on whether you're trained right in the area. And sometimes gorgeous answers, right? I mean, there's no doubt about that. But it is unchecked. The only way that I feel comfortable with it ever is when there's a human in the loop. because there has to be someone there to check the intuition.

It should never be trusted on its own because it cannot do sanity checks. Well, I want to push on this a little bit because some of the experts that disagree with your analysis of hitting a wall will say things like increasing the amount of tool use that this tool uses, better prompting of the LLM to do self-reflection.

So when you say something, it looks at its own results and it sees what it comes up with, different experts and mixtures of experts looking and rewriting each other's work, you fundamentally don't believe that this is going to get us to a more beautiful future either, right? Whereas I think the other experts say that will put the intuition in check. Not in its current form.

It is not going to work in its current form. So, you know, the real problem is in the translation. So what you have from a large language model is it gives you an output sentence, and then you want to feed that to your tool. But if the output sentence isn't the right output sentence, your tool is not going to do the right thing. And so, like, if you take a word problem for kids and you've, feed it in, you may get it to transform it into the right piece of mathematical stuff, but you might not.

It might just come up with the wrong thing entirely. And because it's a black box, it's just a sentence, you don't really know what's there. I actually like to use an analogy here, which is an old joke. A guy's got, says, I've got a perpetual motion machine. And the other guy says, well, you know, I mean, really? And he says, yeah, I've got the whole thing going, really? Well, okay, except for one thing. What's the one thing?

It's this part that goes back and forth and back and forth and back and forth. And so, you know, the point of the joke is, like, you can think you're really close, but if you don't have the critical element, you've got nothing. I partly agree with what you're saying, and yet I wonder, there's sort of these two biases. I'm having a hard time adjudicating how stuck I think we are. And, you know, according to you, we're actually quite stuck until we come up with a new architecture. And another part of me says you begin putting.

better memory on these models. You begin having other models checking each other's work. You begin having tool use saying, oh, run that down for me. Is that really a thing? Is it not? And then all of a sudden it gets just so much better that even though we can't prove anything about it, it's like I can't prove how Tylenol works or I can't prove that my taxi driver will get me to my destination.

But it ends up working well enough for me to really lean on. So let me see how to put this. At some level, I agree with what you just said, and you agree with whatever. We won't get into all the levels of meta agreement. But what I'm saying, is to do all of those things

where you say, well, I will just add an external memory, for example. Let's focus on that one. It turns out that each of these problems is actually a serious research problem that people have worked on for decades on and not made much progress. And so I think it can be done,

but it is deeply non-trivial to do it. There are many bodies along the path. You have a lot of problems like this. People are like, well, you just need an X, you just need a Y, you just need a Z. and those of us who have been in the field for a long time know that those X, Y, and Z are A, really hard, and B, they're really hard to integrate

with massive, uninterpretable black boxes. And so somebody is going to find a way through this forest, but it's not like we're, you know, two minutes from the light and the water, it's just not like that. Okay, and maybe just to pull us back out, like maybe whatever technical disagreements we might have about how to get there.

And I could talk to you about that for hours. But the disagreements may not matter, right? Because I think we agree that there are these real and these present harms coming from this tech now. So let's slow all this down. So can you say more about those specific dangers that generative AI has on society right now? And regardless of how fast this goes, why should we be concerned about the AI that we have right now?

So I actually have a chart in the book of a dozen risks that are immediate. There's also discussion about less immediate risk, and we should certainly talk about them. But immediate risk include things like disinformation from political purposes. That's clearly already a serious problem. This year in the U.S. elections is a problem in the German elections and so forth. There's also similar problems around stock markets and stuff. And I warned about that when I spoke to the Senate a year and a half ago, and I was like, should I mention this one?

I don't think it's actually happened yet. A week later, we saw the first evidence of that in the U.S. and now we've seen a bunch of evidence in the Indian stock markets. And here you're talking about like pump and dump disinformation within the stock market. So the first one was the picture of the Pentagon being on fire, which drove down the market. And I don't know that that was deliberate as a test pilot of all of this, but it may have been. And certainly people took note. And then, yes, there have been some pump and dump stuff in India.

Just a quick note about what we're talking about here. Anonymous bad actors have been using AI deepfakes to manipulate stock markets, like a recent audio deepfake of the CEO of India's National Stock Exchange telling people which stocks to buy, and a fake photo that circulated recently of an explosion near the Pentagon, which affected the U.S. stock market. This kind of thing is super dangerous, not just because it destabilizes stock markets,

but because it funnels money to the people who are using these tools for personal gain. That's right. It's not hard to imagine, in fact, people taking AI agents and trying to hack the infrastructure in order to cause harm in order to make money. I will be surprised, in fact, that that doesn't happen in the next five years. And one thing, this is a bit of a sidebar, but one thing that's really interesting is that bad actors often don't need AI to work as well as good actors. So a good actor who wants to use AI to solve medicine

is just going to realize that it's not really there yet, and they're also going to see that the accuracy is a huge problem for them. I mean, that doesn't mean people shouldn't try it, but it's an issue. Whereas bad actors often don't care I mean think about spam They send something out to a million users It only convinces like five of them But that doesn't matter

If they get one and 200,000 people To send them whatever $1,000 Then they're doing really really well And so bad actors often don't care Take another example Current AI is very bad at factuality You mentioned hallucinations You don't want to write a newspaper with that

Because people will start to complain You know CNET tried it And failed and got pushback But if you're a bad actor, you don't care. You run these websites. You make money off fake news, or you distort the public view of reality, which we've seen an enormous amount, I think, recently on X, and probably not their outlets, TikTok, and so forth.

And so the bad actors don't care if, like, 80% of what they put out is viewed as garbage and people ignore it, as long as 20% of it suits their purposes. For sure, because all you need to do is convince a subset of the population. You don't actually need to cure something. You don't need to find a new solution. It's a lot easier to scare a bunch of people or deceive a bunch of people. You just have to find some suckers.

Yeah. Yeah, and the bar for suckers is lower. So, okay, so there's disinformation, there's misinformation, there's defamation. You know, I'll leave it to the book for people to see all dozen. But the point is there are a lot of risks right now. I'll just mention one other, which is deepfakes, both for their misinformation potential, but also the deep fake porn, the way teens, teenagers are using it against each other as a weapon now, is just really tragic.

That's horrible. That's horrible. And speaking about your book, I was really excited to see you open the conversation talking about the parallels with social media. In fact, we started paying attention to it because we saw the same incentives that played out in social media play out in AI. Why did you start your book with social media? Because it's a really good example of how the tech industry screwed up and because

there are a lot of parallels. so I should say like I've been a gadget head all my life I've always loved technology I think social media is one of the places where we really screwed up social media has been an example where we have a powerful technology it's a dual-use technology like AI in the sense of that technical term which is it can be used for good and can be used for evil like social media can be used to organize people for reasonable protests against repressive regimes and that was one of its early uses and why I think people were positive towards it initially.

But it can also be used to addict people. I mean, it's crazy for me to even explain it on your show, since I'm sure you listeners know very well, but just to make this a little bit freestanding this episode, you know, it has caused enormous harm, presumably to teenagers, it has caused enormous polarization to society. I'm sure you have many episodes detailing all of that. So that's an example where a dual-use technology probably on balance has done more harm than good

and certainly hasn't done anywhere near as much good as one would have liked it to have done. Well, from our perspective, it wasn't the fault of the technology per se. I mean, the technology, it's just beyond the idea that the technology is a platform that can be used in different ways. It's that really the shape of the technology was driven by those incentives, by the business models, by all of it. So it wasn't so much a failure of the technology itself.

It was a failure of the societal institutions and the incentives that we wrapped around that technology? 100%. So, you know, in principle, we could have wrapped a different set of incentives around it and it might have worked better. But we got caught up in things about anonymity and rewards for, you know, news feeds and things like that.

Like, I always think newsfeed was one of the worst moments in all of this history when suddenly, you know, news feed combined with Section 230, which is a legal thing that basically gives immunity from liability to these companies, was a horrible combination because that's the point at which these companies suddenly really started shaping the news and not having responsibility for it. So coming back to AI, I see a lot of the same dynamics at work. So I see inadequate regulation. I see the government doing too little. I see the lobbyists being extremely effective and well-funded.

There are a lot of parallels at that level. There's a surveillance aspect to social media that we may see even in a deeper way, and I'd like to take a second to talk about that, with respect to AI. We see a possibility of scale causing enormous harm. Just on the surveillance point, it turns out that large language models can do crazy things like instill false beliefs in people.

And so the amount of power that the social media company had to kind of shape our thought is maybe even greater with the large language models probably is even greater. I mean, you ran over that. That's so fast, but it's a very deep point. Can we slow down on that one? The idea that interacting with a large language model can instill a false belief in some...

That's right. There was a new study that just showed this. Elizabeth Loftus, who's done most of the most famous work on false beliefs, was an author, and it was a collaboration with a group of AI researchers. I'm blanking on who it was. It just came out a couple weeks ago. And they showed that output of large language models could persuade people that certain things happen that didn't.

I mean it's as simple as that and the results is very simple but the implications are profound right I mean first of all the first layer of that is you have software that not by design

but by inherent property hallucinates so large language models hallucinate nobody knows how to solve that so first layer of this is that you have software that can hallucinate and they can then persuade people that things that didn't happen happened. That is terrifying for democracy and for the fabric of society. You have these things

at large scale. The next layer is there's no law about any of this. It's the Wild West. So if the large language model do this, it's not clearly they have any liability. And then there's another layer that's even more frightening, which is that the large language model companies could do this deliberately. So let's say somebody has a lot of money. We won't name any names and wants to build their own large language model and they have the means to distribute it and they want a certain bias in it. That bias might put false memories in people. And so the people running these companies could easily influence the structure of society. Oh, and it doesn't even need to be a mustache twirling person who wants to inject a political belief. It could be banal capitalist

reasons, right? It could be that you're being paid by a shoe company and you ask the LLM some question about how do I get a job and it starts saying well you know first of all you got to wear the shoes for the job you got to wear the shoes for the job you want to be in and like that kind of world with that kind of persuasion i think it's a really risky world to be in before i went to the senate i was told that i should modulate how many times i was a wise ass i wasn't told that in those words and still good advice i gave sam altman a really hard time when he ducked the question of what was he most afraid of. So Senator Blumenthal said, are you most afraid of jobs? And Sam Altman said, no, he wasn't really worried about that problem because there's always been jobs. And I said,

Senator Blumenthal, you might wish to get Sam Altman on the record as to what he is most worried about. And that was the point in which he acknowledged that these systems could cause great harm to humanity. I don't know if I'm allowed to do this, but I will note that Sam's worst fear, I do not think, is employment. And he never told us what his worst fear actually is. And I think it's germane to find out. Thank you. I'm going to ask Mr. Altman if he cares to respond. Yeah. Look, we have tried to be very clear about the magnitude of the risks here.

My worst fears are that we cause significant, we the field, the technology, the industry, caused significant harm to the world. So then there was a later thing where I passed on my wise-ass moment, but since I had used it up. But what I wanted to say is you probably know the famous line. when Zuckerberg was asked about his business model and he said, Senator, we sell ads.

And so Sam was asked, do we sell ads? And he said, well, we don't right now. I forget exactly his word. But I wanted to say, what he's saying is, Senator, we don't sell ads yet. And in fact, there's a much more pernicious, maybe is the word, more subtle thing, which is not just selling ads, but tilting the scale as you just answer queries,

which is kind of what you're talking about, right? It doesn't have to be because you're working for the Russian government. It could be just like the temptation to do product placement in large language models is huge. I think people are already playing around with that. I don't know if it's happening in the commercial domain, but I think there's been experimentation. So the temptation year is going to be enormous. Then you have the side note that all of these things are now essentially available open source, which means anybody can open up shop to do their version of push-polling or we, we, we,

need a new word for it like push prompting or that's not quite the right word but you know tilting the scale we're going to see a lot of that and it may be very effective and yet under the radar so people don't know that it's happening to them yeah i mean i'm i'm i'm very scared by that future in terms of the lack of transparency right if you have lack of transparency even on what's happening on the way that it's being prompted in how you're you're moving from a place where it's a query in a response to kind of a relationship. And then you're beginning to have that relationship manipulated for reasons you can't tell. And there's no transparency to any level.

So there's no transparency, first of all, on what the systems are trained on. And there's no transparency on what might go into the, for example, reinforcement with human feedback stuff, which also can shift the scale. There's no transparency to any stage. I guess maybe the question I have for you is, so how much of those harms do you think comes from this sort of semi-broken, unreliable LLM technology per se, and how much is coming from the bad incentives?

And the lack of regulation. I think these three things. So some of them are from the technical limitations. Some are from bad incentives and some are from the lack of government oversight that we could have hoped for and still should hope for. So I'll give some examples of each. I'm not sure I can put, it seems like a hard question.

A good example where the failings of the technology kind of are the problem is disinformation because the systems simply cannot handle truth. They don't know what's true. And so they cannot filter what they're saying at all. And that makes them highly vulnerable to misuse. Fishing attacks are another example like this. It's just very easy to craft them to do very deceitful things. and that's in the nature of the technology.

A third thing like that is people have talked about the risk of AI killing us all and what if the machines are misaligned with us and so forth. They certainly are misaligned with us. Even if they don't kill us all, they at least, it's just very hard to specify to a machine of the nature of the machines that we're talking about what it is you actually want them to align on. They don't understand, you know, don't do evil to human beings or don't do harm to human beings and so forth.

That is in the nature of the architecture that we have right now that they can't be very well aligned and that is a very serious deep problem. You could imagine regulation that might incentivize people to wrap all three together to build architectures

that didn't have those problems but right now we don't have the regulations, we don't have the incentives, and so people build stuff that is really very difficult to align. In your book, you have a wonderful set of interventions about how do you mitigate these incentives

that I think we agree are driving a whole suite of risks. Can you talk about those interventions you think are the most effective at mitigating these risks? I mean, this is like Sophie's choice except I have 11 children, right? There are 11 different recommendations. And the first recommendation is

don't expect one recommendation because, and it's kind of depressing. It's much easier to say, you know, here's the one thing you should do or shouldn't do, especially in my position is a guest on a podcast. But the reality is that we are now messing around with a kind of general purpose AI. It doesn't work very well in all the regards that I've talked about.

But it can be used in so many different ways that it does pose many different risks. And it's just not realistic to expect one silver bullet. The closest would be stop building AI, but that's never going to happen. And I'm not sure it should happen because I do think there are positive uses. So the next thing would be keep it in a lab and don't deploy it. understand an argument for that it's also probably not going to happen so once you move past that you have other interventions probably if i had to prioritize the number one would be anything that's going to be used at large scale should in fact have pre-deployment checking with some kind of actual teeth if it

doesn't meet the pre-deployment check so we do that for drugs right you cannot release a drug if you can't make a case that it is not going to cause massive harm and that it is going to cause good clinical trials We have a whole procedure with the FDA and you can argue about whether it's perfect and too slow and whatever, but we have a procedure there to try to keep people from being harmed by random charlatans making drugs.

And it works reasonably well, I think, if you actually kind of look at what could have filtered in and didn't and so forth. I'm not saying it's perfect, but it works pretty well. We need something like that for AI, and especially if somebody's going to deploy something for 100 million customers or something

like that. Now, I'm going to assume for the sake of argument, the GPT4 was, you know, a small net positive, let's say, to the society. We'll just, you know, make that up. It doesn't really matter because the point I want to make is, what about the next system, right? The next system that is significantly different from GPT4, whenever that comes, is going to have a new set of harms, possibly more severe harms. Who makes that decision? How does that decision get made? I think a reasonable thing to say is that somebody with some knowledge outside the company, some independent, oversight should be there

and there should be a process. So with drugs it is a process so it's not like the FDA does go like never come back it's more like come back when you fix this problem maybe to fix this problem you just need to put a warning label don't take this on an empty stomach because

it could rot your stomach and like now we're good to go and sometimes it could be worse so we do need in fact warning labels on large language models and remarkably large number of people don't understand that they lie and the little fine print at the bottom is not sufficient. But probably for the next generation,

I'm going to need more than that. So if I had one thing and only one thing, it would be pre-deployment testing. But honestly, we also need auditing. We need transparency. One of the things we need transparency around is the data that goes into the systems for all the reasons we talked about earlier.

We also need transparency into what internal testing was done. We don't want to be sort of replicating what happened with the Ford Pinto, where those guys knew that the gas tanks would explode, but they made a financial decision. There's nothing stopping Open AI or any of these companies from making financial decisions, even if they know that there are certain harms

for next generation models from releasing them anyway. And there's nothing so far obligating them even for disclosing what they've done internally. Yeah. And in your book, you talk a lot about how you think there needs to be a new center of power, a new agency to be able to handle those responsibilities. Are you modeling this off of the FDA?

So part of what an agency should do is, and let me rewind one sentence. The point of the agency is to be more nimble than Congress can be. And Congress obviously can't be very nimble at all. So that's not a high bar. But the point of an agency is you have people with the expertise and the power to move quickly, right? So we have an agency for health education and welfare, for defense and so forth. Like, you know, you could imagine a world where we don't have a defense department

and we just kind of like distribute the responsibility around. and the health education and welfare department has to kind of do the work of the defense department. Like that would be ludicrous, right? You could imagine doing it that way, right? Or you could imagine, like, having no defense department and just like the president sits around in the Oval Office and says to his or her buddies, hey, you know, what do you think?

What do we invade Poland today? How do you think that's going to go? Like, that's absurd, right? You want a defense department. You should have an AI department. And the AI department, one of the things it should do is, like, what has happened this week? You know, is there something, because things are moving very quickly, even if there's

diminishing returns on the absolute sort of competence of the models, people find new applications. There's a lot of fighting over turf about what to do. We should have somebody running point. What do you do about these various risks? There should be an agency looking at all of these things, having it all in the radar. There should be, you know, a person in the United States with the staff in order to do that. Well, I mean, I too want help sense making on some of the.

of this. And I too want a body to help collate the news, to help figure out where things are ready and where they aren't. But there's actually a lot of voices who say, you know, that will get captured. It will get either overtly corrupted or it will just grind into the dust like some of the other institutions.

What do you say to those people? I say those people are absolutely right to have that concern, right? Every agency, in fact, faces the risk of regulatory capture, which means that the companies that we're trying to regulate co-op the process. And boy, the companies that we're trying to regulate are good at that. They did a phenomenal job in social media of co-opting the regulation that we had.

Section 230, as I mentioned earlier, has been a complete disaster. So I 100% think that we need to watch out for regulatory capture. I think one of the ways to address that is to have independent scientists in the mix. I have squirmed every time I have seen a major government have a meeting in which they have three or 20, many representatives of big companies in the room and no major scientist. And I have seen that over and over and over again. I've seen it in the United States.

I've seen it in other countries. This tendency to treat the leaders of the companies as rock stars and treat scientists as an afterthought is a huge mistake. And it's also a mistake optically, I think, although people don't care enough about it. But it just looks wrong. And I think we cannot succeed

if the only people at the table are the big companies because they will co-op things. They have a long history of doing that. So I'm extremely sympathetic to that. And my only really good answer is to have independent voices. I tend to emphasize scientists because I am one, but independent ethicists and legal scholars and so forth. We have to have multiple voices at the table looking out,

saying, you know, what are the ways in which this could get co-opted or just go wrong? So, you know, it's worth getting it right. That's right. Gary, this is obviously a really deep topic that we could talk about forever and we haven't even talked about international competition

and a bunch of other things but I would really encourage people to read your book where you lay a lot of this out. Speaking of what, you end your book with a call to action and I really want to give you the chance to make that call to action

straight to the listeners of this podcast. To everyone listening right now, whether you're a policymaker or a computer scientist or someone completely outside of this world, what do you think we all need to do to make this go better? We need to speak up.

We can't leave it just to the companies to self-regulate, which is what they want. And we can't leave it to the government, especially in the United States, with citizens united and so forth. We're just not going to get the government on its own without people speaking up to take care of us.

We need the citizens to speak up. And that can be in the form of direct action, indirect action. One direct action that, you know, I don't know that everybody will do, but we should think about is actually boycotting certain products until they are ethical, until they're reliable.

So I give an example, there are generative AI programs that make images. They're super fun to play with. But they are using the work of artists and they're not compensating the artists. It's like using coffee that's not fairly sourced. We could as a society say there are so many negative externalities of these products that if you use them, you're helping build a kind of new form of capitalism that is very exploitative. So we haven't talked much today about the environmental cost, for example, but they're large and they're getting larger

because the industry's only idea about how to fix its problems, and everybody knows these problems, hallucinations and so forth, is to try to build even bigger models that are going to cause even more harm to the environment, take an even larger fraction of the energy of the world, probably drive prices up. We could say, as a society, enough. We're not saying never build AI, but why should we use your products now when you don't have an answer for who's going to pay the cost, You don't have an answer for how to make it reliable.

And if we incentivize the companies by insisting that they make their products better and not release them until they are, we could actually force them to a position where the AI would be better. I honestly think we're at a kind of like knife's edge between a dystopia here and a utopia. The utopia is we could make AI that really helps with science, really helps the many and not just the few, or we could wind up with an AI that's basically just a,

gift of intellectual property to a small number of companies that cause all of these harms, discrimination, disinformation, and so forth. And which way we wind up depends on what do we do now? Do we do anything like a boycott? Do we at least insist that as part of the election that people tell us their position on it? I mean, the thing that flabbergast me is like everyday people talk about immigration and economy and so forth, and nobody is talking about AI. There's been no discussion so far on.

policy and AI. I hope that this is outdated by the time that comes to air, but I don't know that it will be. The very least the people could do is to call their senators, their Congress people, and say, what are you doing about AI in all of these risks? The American public is in fact worried about AI, but they're not loud enough about it. Well, Gary, thanks for laying all of this out, and thank you so much for coming on your undivided attention. Well, you had my undivided attention, and I think the connection between what you think about in this show at CHT, and what I'm talking about in AI is profound. Everything that you have been worried about applies in AI and possibly even worse. And so I really

appreciate the chance to speak to your listeners. If you enjoy this conversation and you want to learn more, you can find Gary's book, Taming Silicon Valley, wherever books are sold. Your undivided attention is produced by the Center for Humane Technology, a nonprofit working to catalyze a humane future. Our senior producer is Julius Scott. Josh Lash is our researcher and producer. And our executive producer is Sasha Fegan, mixing on this episode by Jeff Sudeikin,

original music by Ryan and Hayes Holiday. And a special thanks to the whole Center for Humane Technology team for making this podcast possible. You can find show notes, transcripts, and much more at humanetech.com. And if you liked the podcast, we'd be grateful if you could rate it on Apple Podcast,

because it helps other people find the show. And if you made it all the way here, thank you for giving us your undivided attention.