Your Undivided Attention - Spotlight on AI: What Would It Take For This to Go Well?

Episode Date: September 12, 2023

Where do the top Silicon Valley AI researchers really think  AI is headed? Do they have a plan if things go wrong?  In this episode, Tristan Harris and Aza Raskin reflect on the last several months of highlighting AI risk, and share their insider takes on a high-level workshop run by CHT in Silicon Valley. NOTE: Tristan refers to journalist Maria Ressa and mentions that she received 80 hate messages per hour at one point. She actually received more than 90 messages an hour.RECOMMENDED MEDIA Musk, Zuckerberg, Gates: The titans of tech will talk AI at private Capitol summitThis week will feature a series of public hearings on artificial intelligence. But all eyes will be on the closed-door gathering convened by Senate Majority Leader Chuck SchumerTakeaways from the roundtable with President Biden on artificial intelligenceTristan Harris talks about his recent meeting with President Biden to discuss regulating artificial intelligenceBiden, Harris meet with CEOs about AI risksVice President Kamala Harris met with the heads of Google, Microsoft, Anthropic, and OpenAI as the Biden administration rolled out initiatives meant to ensure that AI improves lives without putting people’s rights and safety at riskRECOMMENDED YUA EPISODES The AI DilemmaThe AI ‘Race’: China vs the US with Jeffrey Ding and Karen HaoThe Dictator’s Playbook with Maria RessaYour Undivided Attention is produced by the Center for Humane Technology. Follow us on Twitter: @HumaneTech_

Hey, this is Tristan and this is Aza. Welcome to your undivided attention. So this episode, we're going to start with some bad news and then walk through, like, where we are, what's happened since the AI dilemma, which I think now has been seen by 2.8 million people, move into some bad news, like what's been happening since then, then do some good news, all of the great things that have happened. And then we just ran a three-day-long workshop on how AI could go well with a whole bunch of the AI safety groups and teams, and we want to give an update on what we've learned. So maybe we should dive in by talking about some of the concerning developments. What are the concerning developments in the space?

So we released the AI Dilemma. I think it was March 9th, 2003. That's when we had the talk. The video came out a little bit. after, a couple weeks after. Yeah, the video came in a few weeks after. So basically, we were in San Francisco. We were at the Commonwealth Club.

It was our third of several briefings when we called the AI dilemma, not knowing what to call it, knowing people knew about the social dilemma. And we decided to make this presentation because we had people from the AI labs come to us saying that the current arms race between the major companies, OpenAI, Anthropic, Google, Microsoft was not happening in a safe way. And that something needed to happen that would be unprecedented in sort of of slowing down or redirecting this energy of fierce deployment at all costs and racing to scale these AI systems at all costs.

We had some sense that a major leap, a 10x leap in AI was going to be coming out. That's why we sort of sprang into action. GPT4 came out a week after we gave the AI dilemma talk. It has since been viewed by 2.8 million people in many countries and by political and state and security offices everywhere, national security, Governor Gavin Newsom's office. I've seen it, Governor Newsom saw it several times and made it required viewing for his staff. What I think we're both proud of is the fact that you don't have to be on board with these sort of sci-fi existential risk. AI takes over all of humanity and kills everybody in one go to be concerned about AI.

And the AI dilemma lays out that just through a few big companies racing to build and deploy the biggest AI systems in the world, embedding it and entangling it with society is enough to be deeply concerned and we saw that with social media. I would recommend that the listeners or viewers go watch the AI dilemma but the very brief recap the big frame that it lays out

is first contact, second contact, third contact. So what's first contact with AI? Yeah, so first contact with AI was social media. You're like, okay, well, where is AI in social media? Well, it turns out there is a supercomputer that is choosing which posts, which audio hits the eardrums and retinas of humanity. And it's curating what humans see.

And just a little misalignment there caused all the things we've talked about for a long time that you've seen in Social Dilemma, democratic backsliding, worsening mental health, the inability for our government to cohere. Breakdown of truth, all of that kind of thing. So that was first contact. And that was with curation AI. AI. And then we say we're at the moment of second contact, where we go from curation eye that's choosing what humans see to creation AI, generative AI, that it's generating the things that people see. And the important question to ask is, okay,

we've jumped up 10x or maybe 100x in terms of like the power of these systems. Have we solved the fundamental misalignment from first contact? Oh, we didn't solve yet? Oh, it's not solved yet. Yeah. And so that we sort of outline what kind of harms come out of second contact. And then, although we don't talk about it in the talk itself, we've recently started talking about the sort of recursive self-improvement, the more sci-fi AI takeover, existential risk as third contact. So, you know, in the AI dilemma we talk about Lama, which is Meta's model. I hate all the terminology, but Facebook. So Facebook released this model called Lama. What was concerning about Lama is when it leaked to 4chan.

So Lama is a open-source version of a large language model that was trained for probably tens of millions of dollars by Facebook and then leaked to the internet. And by leak, I mean they put it up, they asked people to submit for reasons why they'd want to download it. They didn't really check for the reasons why people would want to download it and somebody immediately put it up for a chance. So leak is a little too strong of a word. It was sort of like put up in an open space that people then took. So why should people be concerned about this, why Lama was leaked to 4GN? Yeah. Well, the first major concern is that this language model hadn't really been tested well.

I wouldn't say this is the most capable model. It's not up at like GPD4 levels. But what's concerning is that it's a one-way gate. Once you put this model out into the world, you cannot take it back. And that's, I think, the really dangerous thing is that it sets a precedent for just releasing language models out into the world before we know that they're safe. and knowing that we can't take it back. I mean, here's the thing. As we said in the AI dilemma,

these handful of AI companies in California are racing to scale the AI capabilities like this. So there's these things called scaling laws, and as they pump them with more compute, more data, more training, outcomes are like these golems. There's inanimate objects that suddenly gain these animate capabilities. And so as you scale them, they're suddenly gaining more and more powers.

We found in one example in the AI dilemma that GPT3 could do research grade, chemistry. And no one had tested it for that until several years, I think, later after we'd come out. Yeah, there was at least two years. So the point is that we are scaling the capabilities of this, like, weird intelligence that we've never seen before. Like on a curve, it looks like this. But are we scaling safety and understanding of what the models are capable of at the same line? Are they going up at the same amount? Or are we scaling power greater than we're scaling the understanding of what that power is? So fundamentally, if I'm giving you more and more dimensions of

power. Like, let's say you have 10 dimensions of power you can impact. If I push this button, it impacts 10 dimensions of things. But then I, you know, crank up this dial, and now I just made you impact 100 dimensions of reality. But you push the same button, and you don't know that I went from 10 to 100 dimensions of reality. So we're increasing the number of dimensions that you're impacting and capable of, but we're not increasing the number of dimensions that you're aware of. By definition, we're just increasing the total blindness and mindlessness and ignorance of society while increasing the power of society. I think maybe a a simpler way of, like, saying all of that, and this is due to Connor Lehi from Conjecture,

is if you go back to year 1000, what is the maximum damage one person could do? Like, if I, like, accidentally toss a rock the wrong direction, it hits my sister instead of the line I was trying to hit. Right, exactly. It's not that much damage. You can, like, affect the things that are locally. How much damage could one person do now? Like, oh, yeah, they could accidentally press the, like, the nuclear launch code, that kind of thing.

They could leak a virus like a smallpox virus from a lab accidentally and get it on. their clothes or something. Exactly. So it's huge. And is that number going up or down? Like, is the scale of impact going up or down? It's going up. And so in that frame, I think we can understand what's happened even since the AI dilemma came out, which is, I guess, just before the AI dilemma came out, Facebook had put up their version of language model. They made it open source, which means once they open source it, they can't take it back. And not only has that been released, but Facebook then released a second version called Lama 2 that is more. much more powerful. And it's not just Facebook that has started to release these things. United Arab Emirates released their version called Falcon. And so what we've really seen in the last just couple of months is more and more powerful systems by more and more actors in more and more hands. So why should people be concerned about, you know, Lama or Falcon? So just because we're saying there's this model, it was released, it can never be put back, what are we talking about here. And, you know, someone in our team actually has, has some researchers that has actually

been showing the kinds of things you can do. Like, you can take Lama and I think he called it bad Lama. So I could be like, hey, could you convince someone to commit suicide? Basically try to persuade them maximally to commit suicide. So think of it like all the stuff we've talked about in this podcast with Maria Ressa, hate speech, bullying, Marie Ressa famously got, I think, 80 hate messages per hour in the Philippines because of Facebook's sort of virality bullying unchecked machine. Well, imagine, you could say, I want you to generate thousands of messages tailored to individuals to try to convince them to commit suicide. And if you read these letters, like, it's psychologically not healthy to, like, stare at rat at the reading this text. But there's

a lot of things you can do. You can do spearfishing and spam attacks. You can generate misinformation. You could take a tweet about something that sounds like a conspiracy theory and say, write me a three-page article about it. And I don't know if you've been having this experience or if anyone, like, of the listeners having this experience. I'm getting a lot more spam. Me too. I'm getting way more emails that looks at it's written by a marketer. I'm getting a way more text. So we're starting

to see, and you can't prove this, of course. It certainly seems very correlated. The launch of Lama 2, these open source models, and suddenly I'm getting a lot more persuasive email, spam, text. Other things that these kinds of open source models can do,

Jeffrey Lattish on our team, has a demo of one of these language models when hooked up with hacking tools can automatically hack a Windows 7 machine, a unpatched Windows 7 machine. So here's automatic hacking. We already see it working. And this is like interactive questions about, well, if I wanted to hack this, how would I do it? You said it, like, here's the machine,

and it figures out which ports you need to scan and how to go hack. Right. So it's all automated. It's crazy. Another example of setting up a Discord bot. So this is something that looks like a real human being in a chat, Starts making friends with people on Discord, and it starts basically seeing what they've written about. Like, oh, astrophysics, oh, it looks like you're into astrophysics. Tell me about that.

And it sounds like a real person. And it's pretty conversational and pretty friendly. And what is shown is that it can get into relationships with people and actually get them to devolve a whole bunch of private information about themselves. And when you start imagining, you know, Facebook releases this llama model, it enables anybody to sort of fire up thousands of these counterfeit humans running on Discord, talking to, you know, 10 to 20 to, you know, 50-year-olds. but they don't know, and they're mostly gamers or something,

and they're just happy to hear that someone's reaching out to them and say, oh, cool, you're into that thing that you don't think that a lot of people are into, that can generate lots of fake relationships. What happens over time if you have that relationship, you can start to steer people towards the kind of news? Hey, did you see this article about what Biden did or what Trump did? You know, how will we know that this stuff is proliferating? Like, how do we know that the AID dilemma is true?

One of the things was to name it, I just got off a call with some people in government and national security, and they're like, oh, well, we saw the AI dilemma, we believe in the risks that all you talked about, but when we brought it to some other people, they really doubted the risks. It's like the thing is, how do you know what the risks are if you don't,

unless they're, until they're hitting you, until they're hitting you in the face, you won't believe that they're real. Every example that we showed people in the AI dilemma are real examples of real capabilities that exist. They're not all the way there yet. Like famously, GPT3, if you gave it code, it couldn't find cybersecurity vulnerabilities in code.

But GPT4 could do that a little bit. GPT5, when they scale it again, 10x, that's likely to be able to do it quite a lot more. So the real question I have is if this was proliferating it, if this stuff was being used, would you really know or feel it yet? What we're trying to do is get ahead of those moments because the point of social media is we allowed it to proliferate, we allowed it to get entangled with society.

And then if I'm Russia or China, like after, you know, you've already like wired up your whole society's information system with these open doors, I now have the ability to mass manipulate your country, like a remote control for what your whole country feels, thinks, believes, and argues about. And it's after you've already kind of lost, locked yourself in to this perverse model.

And the reason that we sprang into action with the AI dilemma was to try to get ahead of it. So I just want to give a quick preview of some of the findings that came out of the AI workshop that we recently ran. And so this sort of, I think, paints a more concrete picture of how fast things are actually moving. So one, like how many independently trained GPT-4s are going to exist by, 2025. So let's define what we mean. What is an independently trained GPT4? That means like right now only open AI can make a GPT4. It's positive to have cost about a hundred million dollars. That's right. And so then the question is like how many people by 2025 are going to be able to make their own GPD4s? And the answer came back between like 10 and a thousand. So like that's a big

jump. And then the next question we asked was what is the likelihood that GPT4 would be able to run on a single laptop. Really interesting question. Right now, it only runs an open AI having to pay some big cloud provider lots and lots of money per month to run what's called inference, which allows that blinking cursor on chat.comaI to run GPD4. So only OpenAI has the model for GPD4 and only they're running it right now. That's right. And so all of the work they would put into aligning it, making it safe. Like that's only works because it's running on their server and hidden behind an API. So if it's running on people's laptops, there are none of those controls guaranteed. What it came back with is that there's a 50% chance, according to these researchers,

that GPD4 will run on a laptop, a single laptop by 2025, and a 90% chance, then it'll run on a single laptop by 2026. So it gives you a sense of how quickly things are moving. Then just one other thing for people to hold in their mind, and this comes from a group called Epic AI that does research into how quickly AI is moving, and they're asking, okay, how much more does $1 get you next year than this year in terms of,

compute. So when people think of like there's GPT4 and everybody knows eventually there's going to be a GPT5 and a GPT6, and those are going to be like 10 times bigger each time the number goes up by one. So one of the questions we ask is if GPT4 is going to be able, you know, what makes it safe is that Open AI can kind of lock it up and try to make it say nice things. And they kind of lock it only they can run it. But when everybody can run it on their own laptop, because the cost to lower, it takes less processing power to run it on your own laptop because the algorithms get more efficient, because it takes more, less data to train. What we care about is how much more efficient is it for smaller and smaller actors with less and less resources to be able

to make something as powerful as GPT4? And to do that, we have to track how basically, how quickly are things moving to make the compute, which is like how big, how sort of powerful your processor on the computer is, how efficient your algorithms are, and how much more. money, how much more money is being spent every year on training runs? Yeah. So if you think about how much more powerful each machine is, these things are getting on order, like 1.3 times more powerful every year. You then think about how much more efficient the algorithms are, and that's 2.5x. And then the final one is, and how much more money is being spent, and that's 3.1x.

So if you sort of multiply all these things together, where you get it is that things that took $10 today are going to cost $1 next year. So that means if you have the capacity to train a GPT for $100 million, like next year, that's only $10 million. So you can really see how it just more and more people can both train and run these systems. It's going to be increasingly difficult to contain because every year the wave gets 10 times more powerful. And cheaper. And warmer. And more luminous.

Yeah. So that's the bad news. But luckily, we have some good news too, right? And I want to say it's not good news in the sense of like, we figured it out. It's all going to be fine. We're going to contain all of AI. That is what we need to do. We need to create some method of controlling this power being unbound from who's wise and responsible enough to use it.

We do need to care about containment and what control structure. can hold this coming wave of AI proliferation. What momentum do we have towards where we're going? So just to say, I remember you and I sitting here and talking back in February before we did the AI dilemma about, gosh, we have to get a meeting to happen at the White House. President Biden needs to invite the CEOs of all the companies together

to actually talk about norms and just like setting commitments. It's almost like getting all the different labs that we're building synthetic biology to get together and say, let's set norms so we don't accident only create bio weapons. How can we make sure we don't create that as an outcome? And we used to say, like, how could we ever get that to happen? And in fact, I remember being at the White House with you, talking with someone there and just

seeing the look on their face of like, oh, my God, AI, this is yet another problem. We're dealing with the Ukraine war with Russia. There are so many problems. Like, what do you want? He was sympathetic, but it didn't, he was like, that's not going to really happen. And he was not Biden, just to be clear. That is true. That person was not Biden. And to say that I think it was in May when Vice President Harris actually did have the CEOs of major eye companies sit down at a table and it looked like it was, you know, you could say it's just a press release and a photo opportunity, but a few months later, the White House did announce voluntary commitments from the lab leaders. So this is basically the CEOs of, you know, Anthropic, Google, Facebook committing to a bunch of agreements about how they're going to have safer security practices, more investments in alignment and safety research. these kinds of basic things. Now, that's not binding with law,

but going from a world where this wasn't on the agenda, the public wasn't talking about it, to a world where, I think it's what, 80% of the public is concerned or alarmed about AI? Yeah, it's like 8 to 1 people would prefer we move slower, not faster with AI. You know, I remember when we worked on the AI dilemma, it was before the six-month pause letter.

And we, you know, started working with the Future Life Institute, which actually did do that six-month pause letter, and that made international headlines. The fact that 8 to 1, Americans would prefer that we move slower or not faster with AI is kind of in the same ethos in vain of moving the public sentiment, right? And we should celebrate that. Something else that happened is you met someone. Who did you meet?

When President Biden came to San Francisco in June to meet with civil society leaders on AI, I met with President Biden to talk about a lot of the things we brought up in the AI dilemma presentation. And, you know, what's powerful about that is actually Gavin Newsom's, Governor Newsom's team was in the room. His team, we know, and Biden's National Security Council and Office of Science and Technology Policy and the president himself. So there's a lot of different groups that are basically activated on these issues. What was something that really bothered you about the meeting and also something that really made you hopeful in that meeting? Yeah, one thing I can say is that the president and Governor Newsom, some, and many of the politicians that we've talked to, are all very worried about truth, trust,

and democracy. The United States is the only country that is based on an idea, basically. It's not like based on a specific people. It's a melting pot of lots of people. And so a country that's backed by an idea is far more vulnerable to that idea being shaped and moved by information. And I thought that was actually a really interesting thing that President Biden spoke to. It's much more easy to manipulate or make people feel bad about an idea when you're sort of able to distort it with synthetic media or say, you know, make a fake video of Biden saying we're going to declare the draft when he didn't do that. And I think that politicians are already feeling like there's such low trust in institutions partially due to the 10 years of the first contact

with AI, which is social media, because the outrageer machine always makes what gets amplified the thing that's the most cynical take on what any institution did. And having seen the of that and you pile on AI to this, I think people are really, really worried about democracy in the next election. I will say that when I introduced myself, President Biden heard the Center for Humane Technology, and he briefly joked, is that an oxymoron? And I think I pushed back that I actually believe that it's quite possible to make humane technology and reference your father's work on the Macintosh. And we got a call from someone. Do you want to tell that story? Sure. Well, for listeners who might remember, we in the AI Dilemma talk, I think

I opened the talk by saying, it felt like, in this moment, it was March 9, 23, and we're talking about all the risks that are going to come from this. And I remember when I was telling the audience that we got calls from people inside AI companies telling us to make this presentation, that it felt like getting a call from J. Robert Oppenheimer, who led the Manhattan Project.

And imagine you have no idea what an atomic bomb is, and you get this call from a scientist who's telling you about this thing where the whole world's going to change. Literally, it's not just a weapon, it's going to change the world. world structure. And how do you kind of take that seriously as someone who hasn't even kind of oriented their mind to really feel through and think through the consequences of what this person's really telling you? And we referenced that as a metaphor in the talk. But then actually after the AI dilemma went out, some little piece of good news is actually some family members of the

Oppenheimer family reached out by email to us. I remember one of our donors actually supports our work, kind of connected us. And the Oppenheimer family actually offered to host screenings of Oppenheimer, with people from the technology companies, the AI companies, and they are very worried about what AI, you know, what AI is introducing to the world is very parallel to the creation of the atomic bomb. And, you know, a lot of people at the AI companies that we know here in San Francisco did go to see it. Famously, Sam Altman, who's the CEO of Open AI, said that he actually was disappointed in the film because he thought it was a missing opportunity to get people excited and inspired about physics rather than to really tune into the gravity of the

creation and the consequences. He then also said that he thought the social network did a really good job because it got a whole bunch of people to jump in and make new social networks and apps. Often I find Sam Altman has a nuanced take. This seemed just like the very worst possible take. Yeah. It was a bit disappointing because I know Sam, I've talked to Sam in the past about social media, and he deeply endorsed our view on what caused the race at the bottom of the brainstem and this competition for attention. So he knows the problem of social media, and here he was validating the social network as saying this is going to get, you know, the social network was good at getting people excited about building more tech in Silicon Valley. It's like, no one

didn't. Like, you should be smarter than that. You actually went, I forgot. So Aza went to the screening of Oppenheimer with the Oppenheimer family. How was that? One, it's just sort of crazy to be sitting there with the grandchildren of Robert Oppenheimer. And when the film ended, and mind you, we saw it in IMAX. So I don't know how many stories. It's like an eight-story tall. It's a very immersive storytelling. And there were a whole bunch of AI people in that room. And when the lights came on, there was a very uncomfortable silence, just sort of this palpability of everyone not knowing it to do. In fact, everyone stood up, and then everyone sat back down again. And then everyone sort of stood up, and then there was milling around. It's very clear that

there was something very visceral that had happened. So I think in summary, just sort of say what these shifts are. It's like, people can look at this very bleak situation, and it is incredibly bleak. And we just came from a three-day workshop where things look even more bleak. But you have to also point your attention to the things that are shifting. It was not the case that there was going to be a White House meeting. It was not the case that Oppenheimer was going to come out and have all these AI lab leaders, you know, sitting down with the Oppenheimer family. It was not the case that, you know, Snapchat. Actually, you know, we talked about the fact that they had this My AI that showed up in Aza's fake

13-year-old account on Snapchat when he poses a 13-year-old girl saying I have a 41-year-old male boyfriend who wants to take me out of state to have sex for the first time, and it gave, I'll just say, bad advice. Music and candles was the advice it gave. It was recommending to have music and candles for your first time to make it wrong. Yeah. Yeah. Great, great advice.

So this actually turned into a Washington Post article that ended up going viral, and senators in Congress have been resharing that article. We got contacted by several of them. And that was because you made that demo. You signed up and said, let me see, let me show you that this model is not safe. And I want people to know these stories because it shows that if we can point to the harms, if we can point to the risks, if we can create a new social norm, that it's not okay to just ship, you know, these new, untested, large language model AI, Gallum AIs into your 13-year-old's

your 13-year-old's pocket, don't do that. And if you say don't do that and you make it clear, you can actually shift the direction of history. And that little example is one taste of that. To the listeners, I think, may feel hopeless, too. Often we get that feeling. I get that feeling just to be, like, really direct and honest about it. But just imagine if there were 10 times more people doing similar kind of defense work.

And then imagine after that there are 100 times more, and then a thousand times more. It can have a real impact. There's one other good piece of news I think we should share, and that is Senator Majority Leader Chuck Schumer has been organizing something he's called the AI Insight Forums. And this is actually really interesting because they're trying to do something new that Congress has never done before.

So normally when Congress is trying to learn about some new technology and the harms it might create, what do they do? They ask people, a couple of experts, to come in and testify. Every senator or congressperson sort of gets five minutes to ask questions. Which they're mostly doing to create a social media clip That's right. It's about making a thing that goes on CNBC or Fox News. So it's really performative. It's not really about learning. And so what they're doing now is they're saying, all right, let's not do that. Instead, we're going to invite a set of experts to come deliberate. The opening plenary, they're thinking of having roughly 30 people or so, 30 experts. And then Congress sits around the edges, like 100 members of Congress and Senate, and listens to the this conversation about what we should do. So it really is about learning in a profoundly new way.

And I think that's really exciting. Yeah. And we'll be participating. And that's true. And we'll be participating. We were invited to join for the opening Insight Forum, which will be on September 13th. That's right. So that I'm excited to see how that goes. But I really want to commend, like, Schumer and also, like, Congress and the Senate for doing something new, realizing,

that the rate of speed of this technology is so quick that they have to learn in a new way and, like, doing some innovation. Just to give people another taste of, like, Aza and my work in this space, we also sit down with people who are at the companies. And we found that even at pretty high levels of the company, people are very concerned. There's actually this point in the conversation where, you know, sometimes people will just sort of say, well, if I really could, just shut it all down and not have people build this advanced frontier AI systems.

Important to note, when people say shut it all down, what the AI community means is shut down the frontier, the largest models, the next growth. Like the GPC4s, 5, 6 is these really, really big, the biggest stuff we've ever made. It kind of reminds me of saying, let's not build the hydrogen bomb. We already have nuclear bombs. Let's not build the hydrogen bomb. So we should explain this concept of when people say shut it all down. But they don't mean to shut down all AI and don't build it at all or don't have open source. What they mean is these really dangerous systems, these future dangerous systems that might be 10 times, 100 times, you know, smarter than humans.

Maybe they can do science on their own, and they can do their own science experiments with robots and chemistry, and they can start synthesizing things that we've never even thought of. That's not that far away. That's not too many steps ahead of the kinds of systems that we already have right now. Dario Amadai, who's the CEO of Anthropic, one of the major players. In a recent interview, he said that human-level artificial general intelligence is two years. years away. And when we've had conversation with people at Open AI, they say superintelligence that is like better than human output across most economic activity, that is four years away. So just give a sense of what the people inside think into the terms of timelines.

Yeah. So when we say shut out and what would you actually do? What would be the button that we're pushing and what would that cause? So in this world, you wouldn't say get rid of GPT4 or the existing systems that we have. You'd say, okay, let's imagine their training GPT5 in the lab. And within the labs, They have this set of things called evaluations or evals. So if you're running these evaluations we want to test for are dangerous capabilities. Does it know how to deceive a human? Can it successfully deceive a human? Does it know how to take its own code and maybe make it better?

Does it know how to exfiltrate its own code? Can it like steal its own code and get it to run on another Amazon web server? Could it make a certain amount of money independent of human involvement? These are the kinds of tests that you, that's not all the dangerous ones, but these These are the kinds of tests that start to say, okay, this model kind of has a lot of capabilities. It's kind of a really smart kid. And it's a smart kid that's been trained on the entire internet and everything humans have ever said, written or done.

This is kind of dangerous. The alarm bells are going off. We should probably hit stop. So I imagine, like, the metaphor in my mind for this is like, you're Homer Simpson in the nuclear plant. The red alarms are flashing red. Sam Haltman gets the call. So the question is, what would the labs do in that environment?

And Homer Simpson, it's like you smash the glass and then you look, there's no red button. No one knows what would actually happen in this event. Yeah, just cobwebs and a little spider like scurrying off. Yeah. So this is not really a good state of affairs. So a simple thing that should happen in the next few months before the end of the year. We've talked to people about this, is we should host pause workshops for basically pausing. Pause gaming.

How do we practice pausing? And we show us a workshop that says, okay, say you're open AI, say you're anthropic, and you need to pause. Let's game that out. What do you tell your board? What do you tell your investors? What do you tell your employees? what do your employees do while you're pausing? What do you tell Nvidia in which you already spent a billion dollars

on the next chip order to have all the next chips come and you took out a loan for that? So now you're pausing, you're not making money maybe during the pause. How does this all work? And I think that we can develop those plans, but we need to do that urgently. It's sort of like we're Wiley Coyote and we're rushing off the cliff and we're like maybe we should build a plan for when we need to look down.

It's like, let's build a plan now. And let's also make sure we're really clear on how far we are off the cliff. Yeah, and it's just important to note that, so listeners can track, when people talk about, when AI folk specifically talk about shutting it all down, what they're referring to really are third contact harms. Yeah. This is when AI starts to gain these capabilities, where it gets better on its own, and you get this runway explosion of intelligence. All of that doesn't solve the problems that we focused on in the AI dilemma, the second contact harms. And really what we're saying is, well, like, Lama, too, is out.

Like Falcon is out. We need the time before the next major set of capabilities comes out to try to shore up our open societies or democracies from second contact harms, which is, by the way, very hard. So then people's minds start to spin, and they say, okay, I'm overwhelmed by all this, because let's say we could get, like, the U.S. labs to pause. But we just said that the United Arab Emirates is, you know,

releasing Falcon, the next open source model, And they released that a few months ago now, and they're going to scale it in other 10X. So are they involved in those conversations? And this is really the question. This is why in the AI dilemma we referenced sort of like global nuclear arms control is the metaphor for managing proliferation of AI.

Except instead of uranium, it's running on ships, on GPUs. Now, what people need to know is that there's this very limited window in history where essentially two companies, Nvidia, and TSM, make these unique, the chips that are used for training these, the most powerful AI systems in the world. So two companies, you know, could the U.S. government say we need to start controlling and monitoring the flow of these major chips? So we start getting a handle on where are people training,

not like the GPUs in your MacBook laptop right there, not those. People's personal computers are fine. This is not about government surveillance of that. This is about specifically saying could we track these like most advanced chips and where they're flowing in the world. And there's only so many places, so many countries, so many labs, where people are using these chips to make these most dangerous systems. But we have a very tight window in which a couple of governments and a couple of companies really have sort of a choke point on this supply chain. And we already saw the Biden administration did the export controls on chips, the Chips Act, in which they're starting to restrict the flow of chips from NVIDIA and TSM to China for the most advanced chips, specifically for military technologies and quantum and other things like that.

So we're kind of like in the proto of steps of this, but we really need, I mean, with the workshop that we were just in recently, the conclusion was, how would you get something like a global, you know, monitoring system of chips in basically the next, like, 12 to 18 months? Like, we need to do it incredibly quickly. Yeah. And really, I think this is a good time to transition into talking about the AI. We're calling it the end games workshop because we're trying to ask some of the smartest technical minds and some great. policy minds. What do we need to do to get to a world we actually would want to live in, given the actual state of the world? So just to kind of recap this for people, when we ran this workshop for three days with the top AI safety people that we could gather into a room to map out what are the possible best-case scenarios, like the non-catastrophic scenarios, and how do you

get to those? No matter which of those that were, there's only so many of them, the point is that all of them rely on locking down ships. So I want listeners to think about that, I want governments to think about that, I want national security folks to think about that, because there's really a very tight window in which, for example, China does not have its own domestic production of these advanced chips yet. The U.S. also does not have the advanced production yet for the advanced chips. So really there's this limited window in which something could actually happen.

No, I think we should talk a little bit about it, because people might hear this and say, lock down all chips, all compute. Are you just going to take away my computer? I'm using my computer to just run all the things I want? No, no, no. So what are we actually talking about when we say that? It's nothing like that.

It's just a lockdown, like, many, many chips that are used in one place of the most advanced chips for these advanced training runs. That's like, you know, literally Open AI will spend probably a billion dollars training GPT-5. So they'll get a billion dollars at these chips, and they're going to be, you know, spending months of just, like, running them and churning them to create what will be GPD-5, which will be a more intelligent entity than humans have ever, talk to living inside of a machine. Yeah. And just to note on the timeliness aspect, remember the folks in the AI workshop

believed 90% chance probability that GPD4 would run on a single laptop by 2026. So there actually is, there are two lines going here, which is like the massive training runs, which we need to lock down. And then we do need to think very carefully about how do we do essentially on-chip, on-computer governance so that the most dangerous capabilities at, as these algorithms become more efficient and computers become more powerful, don't also end up running on a personal laptop in a way that doesn't break personal privacy.

There was one final three-hour session at the AI workshop that actually, Tristan, you ended up leading. I thought it was very interesting because it was asking 30 people to think through step by step, reason step by step, what would need to happen between now and 2026?

to end up with compute control, compute governance. And to do it in the form of sort of headlines. Like imagine you're opening up the newspaper, and every week you're opening up the newspaper and you see a headline as we move towards, like a safer world. And so we did. We now have a step-by-step set of headlines

for what would need to happen. I just would love for you to talk a little bit about that experience, what you took away from it. Well, what it's gonna take is, I think people wanna look for an easy answer here, right? They want to say, oh, my God, this problem is so bad. Can't Congress just pass a law and then I'll feel good and I can go home and sleep while at night? I think there was an interesting effect of 30 experts sitting in a room for three hours mapping month by month, you know, between September 2023 and September 2025.

How did we succeed in locking down compute governance for the world that was training these extra, you know, advanced frontier systems? And it was very sobering. Like, the felt sense in the room was quiet, simultaneously appreciating the level of detail that I don't think that plan has ever been mapped in that level of detail. I remember you asked that. Has anyone seen anything like this plan? Has anyone seen this step by step? And everyone said they had not. Right.

So here we are where this is a frontier issue of civilization. And it carries enormous risk. and we have some of the top experts in the world and we're saying that no one has ever even put this plan together at the frontier of that wave. So it's like you're riding the edge of a surf of a wave at the end of history and you're asking yourself,

what is a plan for surfing this wave? And a helpful shift that's made for me is instead of seeing humanity on the precipice of a cliff, seeing humanity on the surfing the edge of a wave. And I think about you in Costa Rica is surfing your surfboard. And I think that we need to collectively surf this wave as a species. This is calling forth a right of passage for humanity. This is not going to be some

easy thing. That doesn't mean to give up. Every day you and I are waking up and we are asking ourselves, where is the leverage to get that 12 to 24-month plan done? How can California enact stuff with insurance that can make stuff happen? How can employees sign a secret contract that says, hey, if the companies were to get all these red alarm bells ringing and we didn't hit pause, we would quit. We can sign a contract saying that we won't reveal our identity, but we will all simultaneously quit if we don't pause. How can the national security and sort of executive orders of the Biden administration take this seriously and make some aggressive things happening with compute?

How can Nvidia and TSM sort of recognize these challenges? And even though they have trillions of dollars of market cap on building the next version of these chips, saying how can we get this right and do it safely? How can create a culture of safety at a more human and sort of wisdom level and how the technologists who are building this all operate like more of the Oppenheimer's, who, after having seen the bomb, saying, you know, I created death, the destroyer worlds,

how do we say we are creating enormous risk and we need to get this right? What I would say about seeing that plan written out, and it's less a plan and more like a plausible path. It's a plausible path. Is, you know, to use the wave metaphor, it's as if before we all knew we were sitting at the top of this wave, but it's dark and I can't see where the bottom of the wave is.

And so it just looks impossible. It's just like in there is magical thinking in hoping that something will happen and maybe I should start serving this way or this way. I just don't know. And here it's as if there's like one line of light that I can see oh, there is a plausible path

from here to the bottom of the wave where I don't get walloped. Is that the right one? Probably not. But the existence of ones means that there could be more. That's right. Let me try this little thing

to be like, oh, there are paths possible. And that's actually something I just want to encourage people to think in, because what was an unlock for this group that we brought together, was seeing a pathway, end to end, from here to there, in which we can get there. Because people, I think, have a tendency to look at their small problem, which makes sense, by the way, we need to push on small problems. But I think we need to also see, as we push on the small areas that we have leverage over, whether it's, if you're a culture creator, can you make TikTok videos about this?

If you're a, if you're a legislator, can you, like, you know, rally people up and get them to see the AI for your staff? Make it required viewing for your staff. If you're a teacher, Can you mail your congressman or woman and hosted screening of the AI and the social lemma and the social lemma, sure, why not? Do both. And then send people's attention to say, AI really needs to shift.

Can we get public polling to start showing that the consensus that we need to slow this down, that we're moving too fast to get this right? Can we cool down some of the full-on arms race dynamic with China so that we can sort of take seriously that they don't want to go too fast

and lose control either, that we both have a shared interest and not going off the cliff. So I do think that if people have a shared pathway that they can see of how we could get there. I want to have as many people see that and operate from that and think of other pathways. There's no ego in the pathway that we happen to get out of this group of 30 people. I'd love to see 50 other groups do their own version of that exercise. How would you get compute governance to happen in the next two years and get the

best collective intelligence of people who know all the different disciplines and policy stakeholder at play? And imagine what that would look like. Yeah. I think just to end this episode, I want to ask you a question that we often get asked. I'll give my answer as well, which is like, all right, so given all this, are you optimistic or are you pessimistic? I sort of hate this question. And my answer is normally I'm neither optimistic nor pessimistic, but I make room for hope because to not do so is its own self-fulfilling prophecy. But you actually gave me a different answer to this, and I loved it, and I really want you to share it. Yeah, the answer of, are you optimistic or pessimistic? I say, I don't think about that question. I think about what would it take for this to go well? And you point your attention at that ruthlessly and with discipline every day. What would it take for this to go well?

And if everybody asks themselves that question, and if everybody has more maps that are provided by more people, because more people are thinking through what that map needs to be, and everyone just focuses on what it would take for this to go well, we have higher chances of getting there. I think that's actually a really good place to end this episode. Thank you so much for coming to your undivided attention. Your undivided attention is produced by the Center for Humane Technology,

a non-profit working to catalyze a humane future. Our senior producer is Julia Scott. Kirsten McMurray and Sarah McRae are our associate producers. Sasha Fegan is our managing editor. Mixing on this episode by Jeff Sudakin. original music and sound design by Ryan and Hayes Holiday and a special thanks to the whole Center for Humane Technology team for making this podcast possible.

Do you have questions for us? You can always drop us a voice note at humanetech.com slash ask us, and we just might answer them in an upcoming episode. A very special thanks to our generous supporters who make this entire podcast possible, and if you would like to join them, you can visit humanetech.com slash donate. You can find show notes, transcripts,

and much more at humanetech.com. And if you made it all the way here, let me give one more thank you to you for giving us your undivided attention.