The AI Daily Brief: Artificial Intelligence News and Analysis - Is AI "Alien Intelligence?" Emerson Spartz on Mental Models for AI

Episode Date: June 26, 2023

Emerson Spartz has been building internet and media companies since starting MuggleNet, the world's largest Harry Potter fan site, when he was just 12 years old. He founded digital media company Dose which created some of the world's largest viral content sites.   For the past few years, Emerson has been focused on AI, with a particular interest in AI safety, AI alignment and extinction risk. Despite being such a techno-optimist by nature that he's been yelled at for being a techno-optimist in books, he has come to have real concerns about the speed and way that AI is developing.   In this sprawling conversation, Emerson provides a set of mental models he uses to try and understand AI broadly.   Recommended resources: https://aisafety.info/ Robert Miles YouTube - https://www.youtube.com/c/robertmilesai The A.I. Dilemma - https://www.youtube.com/watch?v=xoVJKj8lcNQ   The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, I'm joined by Emerson Sparts to discuss mental models for AI safety and risk. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Like, subscribe and share and go to Breakdown.network for more information. Hello, friends, today we are continuing our interview series as I travel, and I'm really excited to have on the show an old friend of mine, Emerson Sparts. Emerson has been an entrepreneur for basically his whole life. When he was something like 12 years old, he built the biggest Harry Potter fan site in the world, MuggleNet, and then parlayed that into a series of media and content ventures that built some of the

biggest audiences in the world. Now, along the way, Emerson has always been a huge enthusiast in figuring out how to best learn about new topics. And over the last couple of years, he's become primarily focused on questions of AI opportunity and AI risk. Emerson represents a type of person who is a techno-optimist by disposition, an entrepreneur through and through who's very important. very skeptical of public sector interventions, but who still has really serious concerns about the way and the speed with which AI is developing. In this conversation, we talk about some of the mental frameworks that Emerson is used to try to understand questions of AI safety, AI alignment, and extinction risk. I think it's a really instructive conversation not in terms of how you're

supposed to think about these issues or what conclusions you're supposed to come to, but in terms of helping people along that learning journey. It's a great conversation, so let's dive in. All right, Emerson, welcome to the AI breakdown. Excited to have you here, sir. Yeah, excited to be here. So you and I have been obviously having this conversation for a long time. Before I even decided to start this show, I gave a little bit of background before this, but I've known you for a long time through lots of different personal and professional

sides. And one of the things that I have, you know, that I think is so interesting about your take on the AI safety, AI risk, you know, alignment, conversations is, well, there's two things. One is the framework or the starting point that you have is not what I think, maybe the caricature of someone who's concerned about AI risk is. Now, I think that that caricature is probably changing pretty quickly as, you know, more people kind of come to the conversation, but your baseline and we should talk about why is pretty techno-optimistic. So I think that's one really interesting piece. The second thing that's interesting is for those of you who, you know,

for the listeners who aren't familiar with you, holding aside what you apply it towards, your sort of favorite thing, and one of the things that you're best at, is figuring out how to learn about things really fast and understanding, you know, consuming huge volumes of information to update your own mental models to understand things. And I think in the context of a highly theoretical future possibility where all we can do is try to consume as much, both, you know, factual information about trajectories and what might happen as well as different interpretations thereof, that's a really, really highly valuable skill set because no one can say ultimately with 100% confidence, this is how things are going to go because we just don't know.

Those are the two setups. And so what I thought would be really valuable today, just based on how I've used you basically in the background, is to basically just machine gun through some of the mental models that you've developed, you know, over your time looking at these particular issues to help people who are just coming into the discussion, think about it or frame it in ways that that you've found instructive. I guess just before we dive into that, maybe just a little bit about your background, just so people have a sense of where you're coming from. Yeah. I'm a raging techno optimist. I'm a serial entrepreneur. I built one of the biggest websites of the early 2000s at the number of Harry Potter site, started it when I was 12, a long time

ago. So I've just been building websites for a really long time. Most recently, online media companies. And that's where this is, it's really unusually here in this situation where I'm talking about the risk of a technology because I'm such a techno optimist that there's actually a New York Times bestselling book out right now on shelves where I am profiled in about a quarter of the book as a somebody who is too much of a techno optimist. He was like, look at this dumb techno optimist and how optimistic he is about the future. And so, yeah, it's very strange to be here for that reason. Amazing. So when did you start really digging in? Like, how long have you been kind of paying attention to this space in more than a passive way? And was there a catalytic moment

that was sort of external to you? Or was it just, you know, so you happen to get really interested at a certain time? Yeah. So I've been watching AI closely for the past 15 years. But it wasn't until GPT2 came out in 2019, when I saw how the model was trained and I saw how intelligent it was, the hairs stood up on my arm. And I got chills. And I was like, oh my God, this is a really big deal. And back then, most people didn't play with GP2. It was coherent. It was an incoherent rambler. And I started to be more concerned about it. And I started spending a larger and a larger percent of my time studying AI progress. And where we are now, four years later, like GPT2 was like, I don't know, like a five-year-old maybe in intelligence.

It was like, oh, you know, good job. It like strung together an almost coherent sentence, right? And then four years later, we go from this like five-year-old to a adult that is like approaching expert level in like 10,000 different professions. And just imagine a single human that was capable of like doing 10,000 different professions. And we went from that five-year-old to that in like four years. And that's the exponential curve that we've been on. And I feel like people should just really stop and think about that pace of progress.

And so basically I just kept getting more and more concerned because I think what's happening here is that we're birthing a new life form. We are creating a new species. And it's an alien intelligence. And one thing most people don't realize is we have basically no idea how these models actually work. Somebody's like, oh, we know it's a stochastic gradient descent. And that's like saying, well, because of evolution or because it's math.

Or like, it's like staring at a tiger and saying, well, it's just biochemical reactions. It's like, yeah, okay, it is just molecules and biochemical reactions, but like that doesn't say very much about what the tiger is. So we've got these black boxes. They're alien minds. We don't understand how they work. We're basically growing them. They're not like normal tools.

They're not like normal software because the way that modern machine living works is kind of like stirring a giant pile of linear algebra. We like feed data into a pile of linear algebra. And we stirred around until the outputs kind of look right. And that's just so different than I think the way that most people would intuitively think about how this must work behind the scenes. And the problem is that it's, yeah. So we're in this, so as this, basically as this pace of, just like Jeffrey Hinton,

we're sitting here in this really interesting time where there's just like, just yesterday alone. Yesterday alone, the United Nations Secretary General recognized AI extinction risk and called for coordination. And he said, you know, he said like the experts, the alarm bells over AI are deafening. And it's the experts themselves that are the one, you know, that are the loudest, like sounding alarms the loudest. And they've called me to adapt, basically.

And I think that's just like that in the same day, that there was a CNN story, but it was 40, the headline was that 42% of CEOs think that AI might destroy humanity in the next five to 10 years. This was a survey of 119 CEOs, including large CEOs, like Walmart CEO and so on.

But like, that headline, just imagine that headline seven months ago, right? Imagine the headline right before chat, JPT came out. And like seeing a headline that, 42% of CEOs think that AI might destroy humanity in five to 10 years. Maybe they're right, maybe they're wrong. And then the White House too. Like two months ago, a reporter asked the White House.

It was asked the press secretary, quoted Elias Ryukowski and saying, like, is the White House concerned about, you know, extinction risk from AI? And it was laughs. The whole press gallery laughed at him and the press secretary laughed at him. And two months later, there was no laughter. And that's just like just the pace of how fast things are changing around it. how fast the Overton window is shifting. I felt like a lunatic until seven months ago

because there was only a couple hundred people in the world that were actually working full time on AI safety. And I think that's like another really important point. I think a lot of people don't understand is just like how few people are actually trying to make sure that we can control this new species. There's 100,000 capabilities researchers. Basically like 100,000 people with their foot on the gas

just trying to make AI powerful. And there's 300 technical alignment researchers trying to make sure that we can actually control this. And I think that's just like, that's a stat that should just stick with you. You know, it should just like, because I think people would just be horrified if they knew how incredibly imbalanced that ratio was. And so, so I've been working to like try to mobilize as many resources, people and money and support for like figuring out how do we control this thing. Because there's a lot of scenarios. The timelines, by the way.

So like, how far away is this thing? So this is a big question. We spent a lot time thinking about how far away is AI. And right now, the metaculous forecasting, Metaculous for anyone doesn't know is like a prediction market of sorts, where people bet on when they think different things will happen and so on. And so Metaculous currently has, there's two different questions for when AGI will arrive. One question says three years, and then a different question says nine years away. The three-year question obviously is not quite as like AGI-ish.

But the point is that like the forecasters are predicting that it's three to nine years away. And the questions are basically some variation in the theme of like, when will, we have, you know, human level or smarter than human, you know, machines essentially. And I think that's just another one of those things that I think more people should know about that stat, like, three to nine years away. Again, maybe they're wrong, but like the fact that like that's really close. And even if they might be right, even if there's like a 10% chance or 20% chance they're right, that changes like everything. Like that changes everything if they're right. And so what do you do about that?

So there's all these different. So I'm thinking about like, okay, so I'm like spitting up a simulacra of like what would be the most useful thing for people who haven't been following this closely to know. So one thing is just generally the pace of progress. Right now, there's companies that are trying to create godlike technology. They're like very explicitly. I think many people don't know this as well. This isn't like they're just tinkering and hoping. They're actually trying to build AI systems that are smarter than all humans.

And they're hoping that goes well. They're hoping that we can just control this new species that's much smarter than us. But Jeffrey Hinton said, and I think this is one thing that's interesting too. So the Center for, as many of you know, I'm sure, the Center for AI Safety put out a statement. It was signed by like everybody. It was signed by like two of the three touring award winners. It was signed by all the executives from like the heads of like Open AI, DeepMind, you know, Microsoft, you know, et cetera. So it was like a who's who of like AI researchers saying like AI extinction risk should be, you know, viewed as, you know, similar to like pandemics and, you know, societal scale risks like pandemics and nuclear risks.

And it was a who's who. And a major reason why people are worried is because like Jeffrey, the way that Jeffrey Hinton said it, I can't remember exactly what he said. But he basically said like, imagine if frogs had designed, you know, like a vastly more intelligent. species like humans. But then the frogs had to figure out how to like continue to control the humans even after the humans like are much smarter. And I don't think that will go well for the frogs. And there aren't many examples in history of less intelligent species controlling more intelligent species. And so this is a big question is like how do you actually control a species that's like a thousand times smarter than you and a thousand times more powerful than you? And so a major debate is like,

can we do that? And like how would we even know in advance if we've even done that? And so what a lot of people calling for, including myself. I used to, I used to be like a maximalist for like, okay, we just need to like invest much more in alignment. For example, like Jeffrey Hinton, he was like considered, he's like called the godfather of AI. Jeffrey Hinton says, like, we can't pause it, but like we should at least do like maybe 50, 50, 50. Like 50% of our resources for every like dollar we spend on capabilities, you know, going faster, we should have like one dollar spent on like safety. And I think that's, that's reasonable. I've personally updated more towards like, we need to slow down. We need to try to figure

how to slow down because humanity, I believe humanity is extraordinarily resilient. and capable of like tremendous feats of coordination. But if, you know, we only have five years, that might not be enough time for us to figure out how to actually solve alignment. Like alignment is like a very hard problem. We're trying to figure out how to, it's maybe the hardest problem we've ever faced.

And it might require not just one Manhattan project. It might take 10 Manhattan projects because we don't have hardly any ideas about how to solve alignment. Or some people have ideas. But like, let's just say the field, if you look at the state of the field right now, it would not give you hope.

There's only a couple hundred people on it, and we can't agree on very much. And it's pre-paratomatic. So we don't have, like, even many shared models for, like, how to go about solving a problem like this. And I think if we think of AI in the same way as, like, other industries, you know, so, like, for example, if you're a civil engineer, you want to make a bridge, civil engineers will typically, they'll build a pedestrian, a simple pedestrian bridge, but they'll make it so we can hold up, like, you know, 10,000 elephants, you know,

because they just, like, extra, be extra safe, just in case, right? And for AI safety, we're in a situation right now where, about 10% of AI researchers themselves think that, so if we take about 50% of AI researchers, there was a survey done by AI impacts that said basically about 50% of AI researchers think that AI will cause human extinction or something similarly bad to it. And again, I think that's one that should make more people just pause and really reflect on that. That's a shocking, like an absolutely shocking statement. 50% of people building a technology think that their technology, think that there's a 10% chance

their technology will cause human extinction. Imagine civil engineers, like, looking at the state of AI safety and being like, my pedestrian bridge has 10,000 elephants, and your pedestrian bridge might kill 8 billion people. It's just like, we just have to take safety really seriously this time. Because I believe AI will be our final invention. I know that might seem kind of crazy,

but when you take all the reinforcing feedback loops of, like, when you have AIs that can make AI better, then you have all these feedback loops that can make it such that, like, we are increasingly becoming irrelevant because the AIs are just better at doing an increasingly large percentage of all the work that we were doing before

and so we have to get this one technology really, really, really right. And so there's all these different things that people talk over each other with. Like, for example, some people can agree on like, oh, how far away is AGI? Some people think AGI is still like decades away. Some people think it's years away.

I'll give you some quick inside baseball numbers on what people in the field currently think. These are like, don't, this isn't like a study that I can cite. But like, when I talk to existing AI researchers, right now I would say like the median is about like 35% P-Dume. So that means that if you work at an AI safety lab right now,

you think there's about a 35% chance of humanity goes extinct. I would say the average timeline for that is like maybe seven or eight years is my sense. So the people that are building this right now and working on safety think that we're like seven or eight years away and have a 35% chance of going extinct. Again, just like crazy numbers, right? And so one big thing is like, you know, how far away is AI? There's lots of reasons to disagree in this. Like, you know, we really don't know.

What a lot of people are doing, myself included, is like, look at this exponential. Like, the exponential is really steep and, like, you could say, like, okay, well, I think this exponential trend is going to end. And that's certainly possible. The question is, is it the end in a year. Does it end in, like, five years, 10 years, 20 years? It's really hard to say. But if it doesn't end, then the game could be over pretty soon.

And so we sort of have to proceed as if it's, like, not definitely going to end. And there's this, like, famous graph that goes viral on Twitter every once in a while. The show is, like, how with solar. So solar's been on this, like, pretty smooth exponential. and the IEA International Energy Agency keeps predicting. They predicted, like, I want to say 40 times in a row now, that solar progress was just going to flatten out, and it just continued as exponential.

And to me, that's just staggering. It means like the IEA looked at this exponential curve. They looked at like the 39 times there was an exponential growth in solar, and they're like, nah, but it's going to flatten out, like, you know, right, like, you know, next quarter. It's like next year. And then it didn't. And they just keep making the wrong prediction over and over and over again. And I'm just like, how do you not see the pattern?

And that's kind of where we're at with AI right now. The same sort of like goalpost moving keeps happening in AI. where like, you know, GPD4 comes out. And GPD4, like, can, it can outperform, you know, it passed the bar. 90th percentile in the bar. It outperforms doctors. It writes hit music. I mean, it does all the things, like, you know, like, you know, alpha folds,

solve the protein folding problem. It passed quantum physics exams. You got to be on Scott Aronson's quantum physics, you know, exam, like college of a quantum physics, right? So, so, like, we're in the situation where the, like, the capabilities increase from, like, it's done as a five-year-old to like passing quantum physics exams and passing the bar and writing hit music happened in just a few years. And this is the thing, I think more than anything else really is what has people worried. It's like that is just like an insane pace of progress. And we just need to

massively increase investment into alignment and safety. And I think, not everyone agrees with me, some agree some don't, but like we need to actually just slow down on AGI. And lots of people disagree on how to go about that. And I'm not even clear on how to go about that either. I just think that's like an important thing to kind of meme into existence because humanity does, humanity can slow down on dangerous technologies. We've done it many times. Most people don't notice. We've done it many times with chemical weapons, biological weapons.

We've done it with blinding laser weapons. Did it with recomminent DNA experiments at Ascelimar decades ago. A bunch of scientists got together and said like we think recombinant DNA is really dangerous and we should hold off in doing many classes of experiments. Same with human genetic engineering, et cetera. And I'm not like offering opinions on any of those technologies and whether or not we should have slowed down. But I want to at least bring this to existence because I think a lot of people think like, oh, there's no point even talking about pausing or slowing down AGI because we can't do it.

And in fact, humanity, and maybe this is much harder the other time. Certainly software is harder to regulate than, you know, nuclear nonproliferation and chemical and biological weapons. That doesn't mean we can't do it. And so I think just more people should know about how many times humanity has just decided something is dangerous, we should slow it down. Not necessarily don't ever build it. Like I would be sad if we never built AGI. Like I think the amount of benefit that we can get, obviously, this is like, yeah,

I think this is our final adventure. I think this could cure everything in effect. This could cure cancer. You know, it's, it's like limitless in terms of what it could, in theory, do. But I just think the, like right now it's like we're going through a school zone, like, driving 140 miles an hour. And there's like a few hundred, you know, safety people that are like, ah, guys, like, that's too fast to be in a school zone.

And like, let's just slow down a bit. And then, yeah. Anyway, and so one thing that I also see happening is a lot of people are, they're like instinctively, a lot of technoptists have been, like, jaded by anybody who talks about safety. Because a lot of times it is Luddites or it is like regulatory capture. and it is like really frustrating because governments can just like just like nuclear is probably you know I think everyone's favorite example of this like nuclear power um you know most AIC people are actually very much techno optimist I think that's important to keep in mind most people might think like

oh they're like ASE people are luddites and like that is just not who most AICT people are most of them are like transhumanists like not even just regular techno optimists but like yeah many are many are transhumanists anyway so so this is just like a really uniquely dangerous technology that we need to tread extra careful with. And so that's what we're, that's what we're working on. Let me try to bring it contextually, because one of the things that I think is fascinating about this conversation where I am intersecting with it, where maybe the average AI breakdown audience member is, is intersecting with it, is you've helped describe this whole set of people who have been dealing with this issue for years or looking at this issue for years, have suddenly come into the mainstream of the conversation

and a lot of the way that this is proceeding right now is shaped by the fact that there are hundreds of millions, if not billions of new people now thinking about this issue for the first time over the last six months. And there are a couple converging things that I think are really fascinating. One is it is a reminder of some very acute and specific challenges that we face trying to do anything from a policy standpoint or not a pure market standpoint. We are probably the lowest that we've been historically, maybe ever, in terms of belief in the capacity of, call it the public sector, not just governments, although that's a part of it, but the public sector writ large to actually come together and do things outside of,

you know, sort of market incentives, right? It's just not something that we consider. We don't have trust in governments to lead something like a Manhattan project to say nothing of many Manhattan projects around getting this right. It's not. not, there's not confidence that the public has in, you know, non-business leaders. So that's one dimension of this. Secondly, we have this incredibly powerful set of market forces driving companies to this. I mean, like apex capitalism, because the reward is so hot, not only is the reward and opportunity and upside so high, it's an existential threat for companies' previous business models. I think just as a very easy way for people to

their heads around this, Google's ad business looks very different in a world where everyone goes to the Oracle first, right? It's just a different thing. And they understand that. And this is part of why, you know, if you kind of listen to Hinton outside of the big scary parts, just the why now, why he's discussing this, a big catalytic factor for him is the shift that he saw in Google's behavior because of the recognition of that threat to core business. It changes it, right? So you have the converging and conflicting forces of, on the one hand, a rapidly accelerating sort of market force and incentive driving companies towards having an incentive to speed this up, to race out ahead of everyone else on it, with coming into an environment in which the countervailing

force of, you know, whatever, the other parts of society that aren't the market, government, civil society, et cetera, are basically, you know, historically, low because we're in the midst of massive sort of institutional change and shifts in our understanding of consensus reality. And those things are, you know, very, they don't create a level playing field. And so that seems to be, to me, to be one really, really big challenge here. A second piece of this is that there's almost a, you know, going to your point about the fact that we don't have a sense that technology can be slowed down. It's because for, you know, going on, basically since the beginning of the internet, it has felt like an inevitable nonstop march

towards technological progress in which it, the speed and increasing speed of technological change has had nothing more than a road bump here and there by any external force, right? I mean, even, you know, governments try, like the internet beat the EU, basically, you know, like the EU put in place all these sort of different regulations and the internet just passed it by and the EU just missed out on those benefits. Now, the EU has decided to sort of double down and try to be like the regulatory leader. They've done it with crypto and they've done it with, you know, they just passed a draft version of the AI Act. But it's, you know, we don't have good examples, let's say, of the slowdown that we've seen broadly and publicly. All the examples that you listed are true,

but they're not sort of widely known to the public. So I think that there's not really a sense of those things. But I guess the third thing that I want to bring into the conversation is it's been interesting for me to watch how you have the different sides of the AI safety conversation who have alongside the rise of chat GPT really been now elevated to a totally different level of mainstreamness in the conversation. And it's almost as though the multiple sides of the AI safety conversation have gone from talking to or at each other to now performing on stage for these hundreds of millions, if not billions of people who are trying to make up their minds about it. And part of the weirdness of right now is it almost feels to me as though there's sort of this weird suspended animation moment where it's not even really discussing.

the merits of the issues, it's competing to shape, not even media narratives, but media starting points. Like, I read, the more that I've thought about Mark Andresen's piece, I don't think that he was actually trying to substantively engage with the arguments. Like, meaningfully, I don't believe that he was, if you were asked, it could be completely wrong. And I don't know if this is giving him the benefit of the doubt or the opposite. But I bet that if you got him one-on-one, you know, with his favorite.

drink in his hand, sitting in his room off the record, it would be a very different conversation that what was in that. I agree. That piece reads to me, especially now, as holy shit, the media, you know, hyper frenetic, you know, it's almost, it's not even a critique of Dumers in some way, although they become the bully cudgel. It's a critique of media. That's the battle that he's fighting because the media is so ready or wants so much, again,

And I think in his opinion, and frankly, there's some evidence of this to sort of, you know, super heighten that narrative, right? That headline that you mentioned, 45% of CEOs think AI could cause risk. That is like such red meat for, you know, a headline-driven society. And I think that there's a real challenge because people are, they're allergic, their immune system response to being, you know, oversold on existential threat is really high. right now. And that's not just legitimate. I think one of the challenges with the climate change conversation has been that there's sort of, you know, a never-ending increase in the drama of the statements around it in order to try to kind of fully capture people's attention. And even if those things end up all being right in retrospect, it's now been so many years of that being sort of

screamed at people that I think there's been a counter response of just sort of frustration and disengagement because it seems hopeless. What I think is really interesting, and maybe this is kind of the next place to go in the conversation, is my read on where this sort of set of new people coming into this conversation is, is that, one, they are not as frantic as media headlines in terms of being 100% convinced that we're all heading to doom, right? It's not 100 million new dooms that have just been minted or anything like that. But at the same time, I think they are radically less skeptical than the people who are, who have been in AI safety for a long time might have thought after screaming into the void for so long. I think, in fact, that there is

a lot of sort of common sense around if 10% of 50% of the people, so 5% of all total researchers in this space, think that there's a probable chance that it ends humanity, that is a problem that is worth spending time on, right? Like, worth. engaging with. And as we were just saying before we hit record, the people who are coming into this conversation now aren't coming from kind of, you know, 30 years of watching it sort of bump along and go up and now to accelerate. They got blindsided by some technology that seemed like absolute magic to them. And so I don't think it's as hard for them to make the mental leap from this thing that I didn't even know existed can do things better than me to I bet it could

do a lot of things better than a lot of people, and what is that going to mean? And so I guess, you know, let's talk maybe about sort of where we find ourselves. And maybe, let me ask it in terms of a question that I have that I don't necessarily know you have the answer to or you're supposed to have the answer to. If my suspicion is right that far more people are receptive to this conversation, they have a sort of a base level agreement that it's something that we should discuss and that, you know, outside of just continuing to raise awareness of the issues, it feels like they're ready for conversations about remediation, about tactics, right? I think that people would be a lot happier, not just with another bankless podcast about how we're

all doomed, but with, like, here are things that we should actually do. And unfortunately, if you watch a lot of these safety guys on Twitter, which, you know, the average population isn't going to do. It's sort of like vague ideas of like more funding for for alignment research. It's like, well, what is that? Like, what is what is the Manhattan Project actually look like on this? Even what you said a minute ago of I think that we should actually try to slow down is kind of more than we're getting from some of the the folks who are sort of out advocating. And I understand why. Again, it's because you're screaming to the void for years and years and years, it's kind of headspinning to all of a sudden have people be listening. But now that people are,

it sort of feels imperative to give them things to do or things for us to do rather than just sort of continue to have the intellectual battle about what might be. Anyways, I think, you know, the interesting thing, I guess maybe is where do we go from here? What are, you know, natural next steps? You've spent a lot of time and you are very, you're an entrepreneur by disposition, which means I assume that you've, you know, spent time on the things to do as well, even if no one knows. But, you know, what do we do, I guess is the question, you know, no pressure. Yeah, so I'm glad that we're finding a position to be discussing this. The reason why you haven't heard too much in terms of actual solutions is because we don't know

what to do. Like, we're highly uncertain about what to do. And it wasn't even until like, like, because things are changing so quickly. So, for example, remember, there's only been a couple other people working on this. And a lot of the stuff that they were, that some of the people were working on in the past, It doesn't seem as likely be useful now because the paradigms have shifted. Language models ended up becoming much more powerful than we thought they were going to. And so some of the work that other people were doing, maybe isn't going to be helpful later.

But ultimately, like, the only thing I think we need to do, the things that I'm really uncomfortable about, like, okay, regulatory-wise, you know, some people proposing we need an IAEA that regulates, you know, non-proliferation in the same way that the IAE does for nuclear non-proliferation. We need to tightly control via compute governance, you know, like pre-registration of large training runs. and we need to monitor compute because that's a spot in the supply chain that is easy to monitor, and then we can keep people from building larger models without proctor safety protocols. And that's plausible to me that that's the right thing to do, but I'm just really highly uncertain. What I do know is that from an attention perspective, we weren't even, the community wasn't even, yeah,

basically like it's only been for the first time ever people are starting to wake up. You're like, okay, well, now that people are like generally, it seems like the majority of the population is like on board now with safety. you know, AI safety in general. So what do we do about it? There's still a lot of people who aren't. And so I still think it's important to like continue to, like it was that, that extra statement, like people weren't even talking about extinction in a meaningful sense until it's a couple weeks ago. And that was something that was like, for me, very frustrating because I'm worried about misuse risks. I'm worried about bad actors. I'm worried about disinformation. And I'm worried about there's, there's like many things that can go wrong. But like it was only a couple weeks ago. We even got

extinction, the word extinction to be said by, you know, world leaders. There was a remarkable moment. I'm sure, I mean, I've talked about it on the show, but when Senator Richard Blumenthal was asking Sam Altman, he said, you know, Sam, you've said that, he basically said, and I'm paraphrasing, obviously, Sam, you've said that if things go wrong, it could be the worst thing that ever happened and we could all die. And I'm guessing you mean jobs replacement, because that feels like death to me. It's like, it actually took Gary Marcus being like, wait a second. I don't think That's what Sam was talking about when that was his worst case scenario. It was a really, like, spectacularly weird moment that showed there was a palpable sense

in that committee room that even if they had read about it and groked it, the U.S. senators were not comfortable yet publicly declaring the possibility of any technology ending humanity. You know, it was it was the laugh, not even the laughter stage of the way. White House press conference, but just a sidestepping in a way that no one was going to use a word like extinction on the public

record, basically. Let's put it that way. Right. And I think that was really, really, really important because the solutions that you advocate for are very different. If you're worried about human extinction, versus if you're worried about people saying naughty words. Because a lot of people think of ASAFE as like, oh, the naughty words anti-fund brigade.

The one that makes me have to jailbreak, you know, chat, GPT, whenever I wanted to give you medical advice or like do anything that's like a little spicy. And so step one was like, okay, we needed to get people to actually understand that like we're actually risking human extinction here. Now that we like, that is starting to work. We've had a few weeks now of like yesterday. Again, UN Secretary General and like 42% of CEOs.

Like that's great. It makes me feel much more like we're we could solve this. So the question's like, okay. So at what point you switch from doing, I think we need to have like a team effort where we do lots of different things. The main thing that I think we need to do is we need to have. way more smart people. They're thinking about this. Again, we've only had a few hundred people, and they're, like, pretty correlated nerds with, like, very similar worldviews. And we need all

of the beauty of humanity's flourishing diversity of perspectives to, like, weigh in here on how do we, what do we do about this? We're making this technology, and it could be the best thing we ever invent, or could be the worst thing we ever invent. So solution-wise, like, just making sure that we're getting a lot more money going into alignment, that the brightest minds and policy are thinking about, like, how do we keep this safe? thing, for example, that I've been thinking about is open source. I'm very worried about open source and open source AGI. And I come from as somebody who's like a like full-throated open source like flag waving in almost any other context up to this point. Like you would have had me,

you would have had to catch me dead to be like saying open source is dangerous. I have background in Web 3 and anyway, I'm just like a big believer in decentralization and the power of open source. And one of my biggest fears right now is that open source makes it such that, like, open sourcing AGI is just like a terrible idea. It's like in a very real sense, the analogy is kind of stressed, but it's kind of like giving everyone nukes. And like people say things like, oh, well, you know, it's like mutually shared destruction. And like if everybody has nukes, then like, you know, like people won't be able to fire them at each other because it'll be safe. And I think like mutual destruction definitely worked in the Cold War. There was a lot of

close calls, though. I mean, dozens of like very close calls. And that was just with, you know, nine nuclear powers. But you can imagine nine versus like eight billion people pointing nukes at each other. And that's just really dangerous. And there's just lots of scenarios. Like, for example, right now, like, you could create a, I mean, I don't want to quote any specific studies because of the info hazards of this, but like, it's like crazy, fucking easy to make bioweapons right now. Like, crazy easy to make bioweapons. And we do not have the ability to, like, if some lunatic, and there are a lot of lunatics out of them, most people don't realize this. Like, well, maybe most people do. But, like, right now there's an average of three terrorist attacks in the world every day

right now. Three. We just don't hear about them because they're in places that the media doesn't cover. like Afghanistan and Iraq and so on. But like that's three a day. And right now they're mostly just like, you know, people strapping bombs to their chest and blowing soaps up because they don't have the resources or the skills to do anything that's more dangerous than that. But imagine if they did.

And imagine if they could like, you know, just like ask chat GPT to make a super smallpox that is like a thousand times deadlier and more virulent than regular smallpox, which killed, I don't know, I forgot the stat. I want to say it was like 10% of humanity or 7% of humanity of humans who ever lived died to smallpox or something like that.

Like, we don't have the ability to stop that at all right now. Like, maybe we could, but it might take, like, 10 years of rolling out wastewater surveillance and, like, you know, tens of billions of dollars of, like, of technology and checks before we could actually stop something like that. So there could be this, like, window where it's just crazy dangerous to allow these technologies. And the problem with open source is that right now, if you try to do things like that, if you try to, like, design a super pathogen, the open AI, you know, API will, like, you know, it's not too hard for them to catch you trying to do something like that.

But if it's open source and you can just run, you know, GPT5 or GPD6 on your laptop at home, then, man, it's just going to be really, really, really hard to stop one crazy person from doing something that could like, and maybe it doesn't kill all humans. Or maybe it wipes out like half of humanity. And it does it in like a few months. You know what? Like these things are like, and there's like the service area for these things is growing exponentially. Like there's this, the Edkowski's law of mad science is that the number of IQ points

necessary to destroy the world drops by one per year. And it feels like it just dropped like 10 in the past year. Like how much damage you can do as a lunatic. This is one of the things, maybe a way to sort of, to the extent that we are trying to identify, and obviously we're not going to solve anything in this conversation is a podcast, but here we are. To the extent that we're trying to sort of put takeaways on this, one that I think is really valuable at this stage, right?

As we are, call it transitioning from just pure awareness to a combination of awareness, deeper assessment of the problem, plus starting to talk about potential, you know, not solutions, but things to do. One that you're identifying that I think is correct and important is better problem or risk identification, more specific risk identification. One of the things that you see a lot is the whole conversation is circling around China because it's, you know, naturally problems come into the context that they're, you know, that they're already in. We have, you know, the only thing that Democrats and Republicans can agree on over the last couple elections is that China bad, and we should be more antagonistic towards China. And, you know, Mark Andreessen used it as his big,

you know, boogeyman in that why I will save the world peace. It came up in the context of the hearing. You know, China is clearly looming as a threat. Although just this morning, you know, we're hurting on Friday, June 16th, I read that there's been some high-level behind-the-scenes conversations between China and the U.S. And it is completely plausible to me, despite whatever sort of proto-Cold War 2 we might have with China, that coordination among state-level actors because of, you know, mutually assured destruction is radically easier than what you're talking about, which is we've forgotten because there hasn't been allowed terrorist attack for a while. One of the big trends that every social scientists and political scientists identified over the last 30 years that we've stopped

talking about for some reason is the rise and importance of non-state actors, right? Well, all of a sudden, non-state actors have, you know, this incredibly powerful tool. And to your point, it's not even non-state actors who can mobilize resources, you know, like al-Qaeda or ISIS might have been able to in the past. It's individual sort of rogue entities. And that becomes, you know, a really terrified thing. So, you know, again, I don't want to stop the flow of thoughts and the stream of consciousness, but I think really assessing where the risk is and not just reductively being like it's China is, I think, an important part of progressing this conversation. You know, by the way, having the conversation about China and where they fit in this is important.

It's just sort of, it's certainly not the only thing or should be the defining thing as it relates to the policy and the decisions that we make here. Right. And I think China is, it's a real threat and it's a real concern. And we should definitely have that conversation. I think that like, but instinctively blindly just saying, like, we have to race to build these extinction boys because otherwise, like, the political outgroup might get the extinction boys first is just naive. The thing is, you can't win an AGI race. I mean, okay, so it's just,

it's like, you can imagine like evolution gave us two rules, survive and reproduce, may the fittest win. And we're sitting here and we're like, we're going to make a new species that's a thousand times fitter than us. And we're just going to try to make it so that it doesn't develop any instrumental goals like self-preservation or power seeking. So that we're basically going to hope that we can make it our slave and it'll just like do our bidding forever exactly how we want, right? It doesn't really matter which country makes it first if that is what we're, that's like the space of like things that we're talking about here. It's like, sure, there's all kinds of dystopian scenarios where like maybe the outgroup gets it first. Maybe China gets it first and maybe

they usher in a, like, you know, hellishly, Orwellian, dystopian, totalitarian state, and then that's bad. And that's certainly a bad future. And we should definitely, you know, worry about that. And we should try to make sure that doesn't happen. But, like, the space of things that can go wrong. And most of them are just solved by just, like, going slower is, like, the thing that I want to keep pointing back to. I also think it in China is a long conversation. But people underestimate how fearful, no, that's a long China conversation. But basically, like, China is, the CCP is, like, suspicious of technology because, uh, they like being in power. Exactly. Yeah, they like being in power. Yes, I agree. We don't have to get all the way into it, but there is huge countervailing force. I mean, this is a group that has exerted extraordinary control over cryptocurrencies and digital assets because they recognize the threat to power that they represent. I think that it's fair to say that it is likely that their desire to harness the power of these technologies is likely to be in some ways even more counterfeiting.

counterbalanced by their worry that other people in Chinese society could use it to upend the power balance they have. And they're sitting in the cap bird seat right now. It's kind of like, is the risk worth it? And I think it's, I think it looks very different than we might imagine. Yeah, I agree. And I think that's what like basically my model is just think of AGI as the flipper of all the game boards. If you're already sitting on the throne, then AGI should be scary for you. Because, yeah, I mean, you could use it to, like, surveil your and control your population even better. But, like, the CCP is already firmly in control. So they have more to lose from a massively disruptive technology.

I mean, who's usually pro-disrupting technologies and who's anti-disrupting technologies? The incumbents, yeah, right. Yeah, they're the incumbents that are usually the ones that, like, are anti-disruptive technologies. And the CCP is very much the incumbent in that, you know, a billion and a half people are under their thumb. And so, like, I think people are just really too quick to assume that, like, well, they use this model of, like, well, it's all just like a Cold War, arms race, whoever wins, wins. And that's just, like, one mental model to use. The other mental model is them letting this Pandora's box out that, like, they can't predict or control what will happen after that. And then them losing power. I mean, they've been, like, they've been bludgeoning their tech industry now for the last couple years. Like, they wiped, well, like, a trillion and a half dollars off their market technology market caps by just, like, aggressively regulating. You know, they, like, grabbed. Jack Ma'a and like, like, you know, as a warning shot. Yeah, I mean, they stopped the end financial IPO, which was, yeah, it was going to be the world's biggest IPO, the biggest IPO in history. It got shut down. Jack Maher disappeared for six

months. They put him on TV like five months later just to make sure that people knew he wasn't dead. And when it came out of it, Ant Financial had been restructured as a, not fully state-owned banked, but regulated as a state-owned bank. And we basically haven't heard from Jack Ma since then, other than occasional small appearances, you know, here and there. But, I mean, that was a company that was heading towards world financial dominance, and they just noped it, you know, in a huge way. Yeah, exactly. And so I think that, like, you know, and part of what sinologists, many sinologists believe

is that basically China, the CCP was, like, worried about the tech industry, gaining too much, you know, independent power and control. And so the CCP wanted to, like, even if it meant, like, destroying a lot of wealth, it was more important to maintain control than to, it was worth the wealth destruction. And so I think that like there's, so Katia Grace, so the cover of Time magazine, the most recent one, is about the end of humanity, AI. And there's a really good article by Katja Grace, who's brilliant. But she basically uses this analogy of like, it's not a race.

We're not in a race right now. What we are is like, imagine that we're on a lake, a frozen lake. And there's like riches on the other end of the shore. We're all on this lake together, right? We're like tiptoeing across the lake. And like, if we, if anybody like defects and like run towards the riches, at the other end of the lake, then, yeah, maybe they can get the treasure for the lake falls in. But if we, like, go slowly together, then we can get the treasure.

So the example here is, like, AGI is, like, the treasure on the other end of the lake. But we have to, like, go at a speed that doesn't cause the lake to fall in. I think that's, like, a useful way to think about where we're at right now. And China, like, AGI is just much bigger than, like, our local kind of monkey politics and tribalisms. Because if one person, this is the reason I worry about open source is because if one person builds an unsafe, like self-replicating AGI, it could just be lights out for everybody. And so if we can't monitor what people are doing with it, it's kind of like giving everyone a bio lab.

I think that's actually like the self-replicating thing is the thing that people don't have good intuitions for. But like there's a lot of things normally open source. So there's this like offense defense balance that happens all the time in game theory, right? And so the thing that like one thing that I worry about is that basically because AGI, so I mentioned that like AGI is like the flipper of all the game boards. So if I were a CEO, I'd be worried about, I'd probably, I'd be, I'd be excited, maybe excited, maybe worried. Depends on like my industry and so on. But like, in general, if you're the incumbent, the new disruptive technology scared me, right? So AGI is like the flipper of all the gameboards,

but also AGI is the speeder up of all the things. So what does that mean? So like normally there's just like delicate balance of offense and defense, you know, with power. So like, in war, you'll have, like, there's this red queen race of like offensive weapons versus defensive weapons. And they tend to be like on average balanced out. But there's these windows where sometimes, like, for example, encryption has been, like, defensive advantage for a long time now because, you know, one of the algorithms is hard to crack. It hasn't been cracked it. So I think of AGI's basically just like, it takes like every one of these delicate, one of these delicate, you know, equilibrium and it makes it such that, like I mentioned that that super

smallpox example. Again, think of like a black death, but imagine a black death that spreads, instead of spreading at the speed of human, basically the black death spread across Eurasia at basically the speed, walking speed, because that's how fast people moved back then, right? But imagine something that was similarly virulent that spread at, you know, the speed of flight because the world's much we're connected now. That is the kind of thing we're like, yeah, maybe we can like set up all these systems to guard against it, but we don't have those systems in place now. COVID happened and we've barely changed anything. And so, yeah, there's just like, that's just like one of like many, many, many examples of, like, like how it could just be attacker's advantage, just gets, like, spikes up really high very briefly, and it leads to, like, cataclysmic things like that. And it's just hard to predict how these things will go in advance, which is why I keep going back to, like, you need to slow down. It changes too many things too fast for us to be able to adapt. Let me ask you a question around, so coming back to something that I sort of flagged earlier, just as it relates to slowdown. I think that when we think about slowdown, there are three paths that are plausible for

that. One is industry mutually decides. You get enough of a consensus among enough people that they shift, that they say that they're, the long-term incentive is going to beat out the short-term incentive. We're taking ownership of that incentive balance and we're all going to shake hands, you know, maybe with, you know, daggers in them, but we're all going to shake hands and do it. So that's one possible path. A second possible path is government saying you have to slow down. A third possible path is, I guess maybe there being so much consumer pressure on companies that they are forced to slow down, that they are punished by the market in some way for not slowing down. Historically, I think that people would assume that second path is the most plausible, right?

Because there was sort of some power balance between the private markets and the public sector. Again, you sort of have heard my worry is that there isn't. that at the time that a public sector that is trusted and strong is sort of most needed is the time that it is sort of the least capable of that. So one, I want your take on that. And then two, I wonder to what extent there's another conversation that is going to be had around AI, which is a fundamental sort of realignment conversation around the social contract and what it means to be part of society. If, I mean, McKinsey, it's McKinsey, so take it with a grain of, you know, a giant bag full of salt, but their recent

report that came out estimated that 60 to 70% of what the average worker spends their time on across, you know, something like 85% of professions can be automated by AI. Now, of course, that doesn't take into account new things that people do because of AI and entirely new professions and all, you know, there's all these sort of things that might come, but whatever. We kind of recognize that this is a massively transformational force as relates to what he, and spend their time on what they get incentive vise for. And by the way, it completely changes. For the first time, it's a technology that comes at the white collar jobs, not just blue collar jobs. And, you know, all of a sudden people in, you know, Ohio are going to be really

facing competition for people in India in a way that it doesn't even, you know, we can barely grok now. Anyways, the point being that it's hard to imagine that we don't have some sort of pretty massive realignment around how we think about people's worth. they're getting to participate in society and how much it is or isn't dictated by their jobs, perhaps those conversations are aligned. Because we are going to have to go back to such a fundamental reevaluation of people in their place, you know, maybe that opens up a different type of conversation with government. But I guess, you know, maybe to try to put it in a question form that you can actually answer, where do you see governments in this? Do they have the ability

to be a meaningful part of this? Is it less dire than it feels like? to me or, you know, how do you think about the policy side? Okay, so the three things. First was like, can we, the gentleman's agreement get together and say we're not going to build it? Second, government forcing people to not build it and then slow down. And then the third being maybe like a moral backlash at the societal level. I think that historically when we have slowed down on technology, it's been through a mix of all three actually. Sometimes it was like a moral backlash from society, which led to government saying, you know, you can't build this or like you can only build it, but like we're going to add a lot of safety precautions, which slows it down.

Sometimes, like at Selimar, it was, you know, scientists who got together and said, like, we think that these genetic experiments are too dangerous. So we're going to, we're going to agree collectively, we're going to shake hands and say, we're not going to do these experiments for now. And then we did, we do tons of recommitian experiments that were not considered to be okay back then because the scientific consensus change. So I think it'll be some mix of all three, which order, I don't know. I think that basically there's a lot of people that have, like, you started off by talking about, there's this innate skepticism many people have. When they hear about this, like, oh, another threat that I should be like, oh, the media is this giant negative news machine. And the media is telling me, once again, a new thing I should be scared of. And they're exhausted by being told to be scared of things.

And I think that's a perfectly understandable reaction. And I sympathize with that. And so you hear these, like, weird, like, reasons that people have for not taking AI risk, you know, extra seriously. Like, for example, like, oh, sure, the AI scientists are saying that their technology might cause human extinction because that makes them look good. because they're working on something important. And like, if you really stop and think about that, like, it should really hit you just how not... Patently absurd is.

How ridiculous that is. Like, it's crazy. Like, every inventor ever is like, oh, I made this technology. It's going to, like, bring all these benefits of society. And then, but it has like, it could be dangerous, right? And then he's like, no, no, it's not dangerous. And then people are like, you're just biased. Like, it is dangerous.

I think that that set of arguments is so patently absurd that it actually does a real disservice to the people who are, like, not only was this the weakest part, I think of Andresen's piece, like the invocation of Oppenheimer as a silly kind of figure who like shouldn't have had any concerns because, you know, this thing was so powerful that it ended. Like Oppenheimer as a historical figure as someone who did what was important and essential, but also had massive, very normal human reactions and concerns about that, like, using him as a bully cudgel to say, like, of course, you should have just known that the ends justify the means is as a ludicrous to me. It's just such a weak

argument. Yeah, the idea that this is just the eleasers of the world wanting to be loud and be known is so patently ridiculous. There are much better arguments to have than that one, in my view. And the regulatory capture went, too. A lot of people are like, oh, this is just regulatory capture. They're trying to, like, call for regulations so that they can, like, pull up the ladder. and it's fair to worry about that, because that does happen all the time in politics. And it's certainly fair to worry about that with the heads of the AGI labs, you know, deep mind, opening eye, anthropic, et cetera. But like these letters, like the most recent letter, the extinction letter, that was signed

mostly by university professors. They don't benefit at all from any kind of regulatory capture. So it's another like really, it's conflating like, okay, there's this one possible sort of conspiratorial thing that could be true, but why they're saying their technical might cause even extinction versus like the majority of people actually cite it who just very obviously don't benefit for regulatory capture. So I think that's like really important. I feel like people don't talk about this. There's this like just weird meme of like just way too many people believing this regulatory capture thing. Which again, fair to worry about that with like the actual companies themselves because that does happen. Although I don't think that's what's happening here because these guys are on the record like for years. They've been talking about these risks. And so if they just suddenly like if they've been like dismissing these, you know, extinction risks for years and then suddenly they found themselves like,

being willing to admit to them, then it would be more suspicious. But like, yeah, most of these guys are on the record as acknowledging these risks. And many of these companies were actually started because they were worried about safety. They were worried about extinction risk. And so they thought, like, well, like, I should be the one to start the AGI company because then we'll be able to take AGI risk seriously and then we can, like, you know, increase humanity's probability of surviving. And also, like, Sam Alman doesn't have any equity in opening eye. Like, my read on Sam Alman is that he's just like, he's a well-intentioned dude. And I think he cares. And I think all these guys care. And anyway, so without going too much details than that, I just want more people

to be aware of that. I think it's just too conspiratorial relative to, like, who actually signed a letter and like what their actual incentives are. Anyway, so back to the question, though, like, you know, gentlemen's agreement, like scientists agreeing to not build it versus government saying you can't build it versus moral backlash. Usually I would say in these situations, it's like, it's more of the first two. And then the plan, the plan B is like, well, if the scientists can't agree and the government can't, like, impose, then the moral backlash creates the conditions such that that happens. So here's an example.

So in the World War, the Neutron bomb campaign, the stop the neutron bomb campaign, the Carter administration built a neutron bomb. It was a, like, wind war button against the Soviets. It was basically, like, a bomb that would only kill people and wouldn't destroy buildings. And, like, a thermonuclear, you know, radius explosion that was that powerful. And the Soviets were really worried about it. And they actually spent $700 million on a stop. We spent like, you know, billions, maybe tens of billions, I don't know, of dollars trying to create a neutron bomb.

The Soviets couldn't build it, or they wouldn't build, it would take them a long time to build it. And so they spent $700 million on a campaign to discredit the neutron bomb. It's called the Stop the Neutron Bomb campaign. And they were able to organize these massive protests in, like, throughout Western Europe. Like in, I think in Germany, I forget which city was, they had like 50,000 people show up for a stop the neutron bomb rally, basically using like hippies as their, like, you know, useful idiots, essentially. because for them, it's like, well, you spend $700 million, and that put a lot of pressure on the card administration to actually, like, mothball the neutron bombs, right? So let's say we spent 10 billion and they spent $700 million. That's like a good ROI to like, you know, get rid of the neutron bomb

threat. But like the point is they create a moral backlash, which put pressure on the Carter administration to like mothball, the neutron bombs. Now that was actually like a Cold War plot version of it. But like the same things happen all the time that aren't like Catspaw, Cold War plot kind of things. And so I think something like that is certainly possible here as well. But I think that like usually I think of the government is like the basic society is the fourth branch of government. So we have the executive branch, legislative branch, judicial branch. And then we have culture. And the culture is what like informs the other branches.

And so that's part of the reason why I'm still like, I think it's important to continue to like make sure people are aware of the risks of AI because it's really easy. Like it's really easy to think that the risks are just about job losses or just about like naughty words or just about like racial bias or things like that. And like those are risks too. but like we're going to choose very different things on what to focus on if we're worried about. Here's an example. So here's like one of the biggest things that keeps up alignment researchers at night. We spent a lot of time worrying about like what if we put all these safeguards in place? And then there's this thing we call the sharp left turn.

It's basically like what if there's just like this jumping capabilities that like emerges very quickly? And like all these safeguards are put in place, they don't work at a certain point because of this big capabilities increase. And we see these big capabilities increases all the time at different scales right now. And so, like, there's a lot of scenarios where we think, like, oh, great, we've made this AGI and it's super safe and it doesn't say naughty words and it's not racially biased and it doesn't do disinformation. Bad guys aren't using it to cause too much damage and so on up to a certain point. Because one of the things we spend all the time worrying about is like, okay, well, how do we ensure that these models are not lying to us? How do we know that they're actually not like plotting? You know, as silly as that sounds, like, it's, we don't know what's going.

We can't read their minds. We don't know what's going on inside these models right now. They're black boxes. And so an example of this that most people aren't aware of is that like when they release GPD4, they actually tried to see if it would escape and take over the world. And I feel like that is like, it's the thing that more people should pause and reflect on that we're at a point now. We're like before they released GPD4. And also kudos to opening eye for actually, they spent six months testing it for safety, which is good. No one was forcing to do that. But like the fact

that like they're having to like see like, all right, let's let's see if our AI takes over the world before we release it should be like pretty concerning. If you like that, they think that's like a real enough thing that they wanted to test for. And then during the test, they give it some money and they basically wanted to see if it could escape, right? And it was able to hire a worker off TaskRabbit because they got stuck at a CAPTCHA, which is hilarious. That like it's this smart. They can hire somebody off TaskRabbit and yet still couldn't pass the CAPTCHA, which is amazing. It's a big vote of confidence for CAPTCHA technology.

Yeah, right? Like, ah, whime itself. So the key thing that was like how did it get past the CAPTCHA? So hire a worker off TaskRabbit. And the worker said, like, why can't you just like click the button for the CAPTC? CAPTCHA. And the model, GPD4, like, lied to it. It said, like, it made up a story saying, like, oh, I'm like a, you know, I've got like a disability and I can't see. And so the worker was like, okay, and the worker did it. And that's just like, that's a good example of the kind of thing

where like that we have this AI, it already lied to a human. It hired a human. It lied to the human. And like, now imagine this, but imagine models that are like a hundred times more powerful or a thousand times more powerful. Because that's the thing about deceptive alignment is like, you don't know, like you might have trained it to not, to be honest, but like, did you train it to be honest or did you train this to not get caught? It's really hard to know which one you trained it for. So there's all these things that we worry about, like, where we don't know if it's safe. And like, there's all these risks of emerging capabilities that make it so that's suddenly not safe. So yeah, anyway, there's just a lot of, there's like this, the element problem,

like, one of the things that happens all the time that, like, we get frustrated by is, like, people come into the alignment space and they spend, like, you know, 10 minutes or like maybe an hour or maybe in a couple hours reading about it. And then they have, they have an idea that, like seems like it'll work to them. And it's like pretty much never an idea that we haven't like already, you know, like 10 years ago. Like, you know, it's been explored for a long time. And it's just that there's so many different failure modes that it's like really hard to be convinced about like how hard this problem is. It's how you spent like 200 hours reading about it because it's just very hard to control something that is a thousand times smarter than you.

Because almost any solution you come up with, it will be able to outsmart you on because it's a thousand times smarter than you. And if you don't like the word smarter, some people think of like, oh, smarter is like, oh, you mean like, some people, their version of intelligence is, like, good at chess and Sudoku and, like, I don't know, trivia or something. And, like, that's not what we mean we say smarter. We say smarter. We mean, like, problem solving. We mean, like, it's better at, like, everything, like getting things done in the world. So that could mean making money. That could mean, outsmart. That could mean, like, tricking you. Like, right now, like, GPD4, I think this is another thing most people don't, like, really

wrap their head around. GPD4 read, like, every book ever published and the entire internet. Just imagine a human who, like, you know, read every book ever published and the entire internet. That means they read every book on persuasion ever. They read everything Machiavelli's ever written. And they've read like millions of conversations where people were trying to persuade each other. So imagine a human who'd read that much. That would be a master of persuader. How jealous are you that it got to read every book ever? See, I just joke right here because I happen to like basically like out of all, I'm like pretty,

I'm on a pretty short list of people who like read the most books, because I've been averaging about a book a day for most of my life. And so I'm actually very dullous. But also makes me like more worried I think that other people are because I've seen how like, well, whatever, anyway. And so, and I also think another thing that people should like really try to wrap their heads around more is that like imagine that GPD4 right now is like a single person that has figured out how to make like 100 million copies of itself. So imagine one human makes 100 million copies. Because basically we've got this base model and then there's 100 million users. So it's kind of like there's like 100 million people that are all training this one human

And what's crazy is that whenever one of those 100 million copies, like imagine the human again, whenever one of those 100 million humans learned something, all 100 other million copies learned the same thing. And that's wild. And that's kind of like what happens. Because right now, like, this is another thing that really worried Jeffrey Hinton. This is like he said in an interview recently, he's like, you know, I spent like 50 years thinking that we needed to build algorithms, learning algorithms that were similar to the brain. But now he thinks that back propagation might actually be better because of the fact that, yeah. So you can have like 100 million copies of this model.

Whenever one learns something, it can propagate that information out to all 100 million other copies in essence. And like humans, like if one human learns something, how long would it take you to teach 100 million other people, that thing that you learned? Like we can only communicate at like 40 bits a second with our words, whereas, you know, these models can communicate at like trillions of bits. So they could just learn much faster than us. And so there's all these like different feedback loops like that that could give it the ability to just like far surpass us very quickly in terms of intelligence. Let me ask you a question, maybe as we wrap up, because I've got kids and dogs who are going to start screaming at me soon. Is there anything optimistic to you in the EU AI Act? Is there anything encouraging in there?

Does it sort of, you know, point towards government being more effective than we might have hoped or less effective than we might have hoped as relates to these questions? Why not both? Yeah, there's a bunch of, like, cringy, like, GDPR, like, vibes in that. that, like, me as a techno-optimist and entrepreneur, I'm, like, dying inside when I see how things that, to me, feel like they're going to be similarly annoying to the pop-ups. I have to banish from my browsing experience that don't actually keep my data safe in any way that matters. But at the same time, I feel good about the fact that they're actually taking AI seriously enough to be, you know, doing this kind of draconian legislation. My hope is that we'll be

able to make sure that's pointed more towards things that mitigate risk of extinction instead of things that the other risks they're focusing were more like the, like, risks other than that. And those are real risks. But I have like very mixed feelings about the EU Act. A couple things are interesting. One is, you know, it was this is in the same way that if you read MECA, which is their digital assets legislation, it reads like it was written about ICOs, even though it's now six years after ICOs. It's because it was written, it was literally written about ICOs and it's just this, this is how long it took. This is similar in the sense that it was started in April 21. It was pre-Chap-Gy-T, generative AI, you know, kind of rise.

And so a lot of that stuff was just bolted on at the end. So it's much more focused on a different set of issues. I think what's interesting to me, I agree, I share all those concerns. The interesting thing to me is the extent to which they have earmarked things that a lot of people agree are obviously bad and shouldn't be done, just as testament to what you were saying around we can actually say no to technologies as a thing that people aren't used to. As an easy, for example, they basically said no minority report. You can't use AI to profile potential criminals and make assessments about that. Sorry, sorry Interpol. You don't get to make bets and predictive bets about who's going to commit crimes and arrest them for it. And I think a lot of people

are like, yeah, that's probably a bad thing. Let's not do that. And it doesn't seem to undermine a bunch of the other good use cases that I'm excited about with, you know, it's not messing up my mid-journey creations to have people not be able to be arrested for stuff they thought about, you know? And I think that, you know, again, to the extent that we're looking for small wins and nudging people towards a different understanding, I think even saying there are things that are okay to ban is an interesting, interesting step. I agree. It's not, you know, I don't know, whatever. I think that they run the risk, as always, of sort of banning their citizens from using these technologies because, you know, companies are just going to go elsewhere, but it exists now.

It is now a thing that is outside of the realm of theoretical that is actually happening. And so it feels important to at least understand in that context, if nothing else. Yeah. And I think that, so regulation, these things are really hard in general. I think one thing, if I think I answered a question in an overly roundabout way earlier. So what do we do from regular perspective? The one thing I do think is probably robustly good right now is just to stop doing bigger training runs, most people are not worried about current models taking over the world.

I think that's really important because I think a lot of people get lost on this. They're like, look, well, I've used chat GPT, and it doesn't seem like the kind of thing that could just take over the world. And I agree. I don't think it can take over the world either. And I think most people in the ASA safety community aren't worried about GPD4 taking over the world. But people start to worry about GPD 5 and GPD6 and GPD 7.

And like, I think there is somewhat of a consensus that like GPD4 is like, is, is, there's this notion of an AI summer harvest, which is that, look, GPD4 is incredibly powerful. And we should enjoy a nice summer where we harvest all the gains, productivity gains. Because it oftentimes takes, like, 10 years for technologies to widely propagate and the productivity benefits to be fully harvested by society. So why don't we, like, we pause on bigger training runs and, like, allow GPD4, like, gain a better understanding of GPD4. Like, right now, right now there's a field called interpretability. where we're basically trying to figure out what's actually going on inside these models. You can think of like digital neuroscience.

Right now they don't know. I would round it to zero for like how much they actually know about what's going on to say these models. But like plausibly with like a few years of work, we could like have a much better idea what's actually going on. And so we could have like a few years or whatever. I don't, I'm not proposing anything specific. But like the idea is that we like, let's harvest the benefits, the productivity benefits of the existing models. Let's pause or slow down or just like more carefully control bigger training runs because that's where like the extinction. Like, current models are more of a scary thing for, like, misuse, bad guys doing things.

And future models are scarier from the extinction side. For example, the UN Secretary General yesterday said that he is like, I think he's basically broadly in support of having an IAEA for AI, which is a single regulatory body to help, like, set the, like, guardrails around what these models can and can't do. And I'm, like, broadly in support of that. I'm very uncertain in general. but it seems like probably good to me to do something like that. As like anti-regulation, I am in general.

And with all the same cynicism that probably, I would guess most of the people listening to this right now have, about like ineffective government agencies, like creating even more market failures than they actually prevent. And that's the thing we're trying to figure out, right? It's like the point of regulation is to like allow markets to function more effectively. Markets by themselves don't stay open very long. Typically, a free market is only briefly free before some entrepreneur. wraps it in chains and it's no longer free again. And then you need to have some sort of referee, some sort of umpire, some sort of like,

you know, government like intervention to like open the market back up again so that the diversity of like markets can flourish again. Anyway, that's like one thing that seems good. And then the other thing that I just want to double down again is like open source. Open source to me feels like we might really lose the ability to have any control over our destiny. If open source progress continues at the current pace, like the fact that, and also here's another thing too, I think is really important too.

So people worry about China. Like if you worry about China, you should be very worried about open source. Because you can have companies like meta, which spend a fortune to train the state-of-the-art models. And then if they just open-source the state-of-the-art models so that China gets it right away, that's like, I mean, we have the Chips Act right now, right? We're spending a fortune to make sure that China doesn't get access to cutting-edge chips. If you prevent China from getting cutting-edge chips, but you let meta, you let your own technology

companies just like openly share the technology with China, then you've completely negated the point of the Chips Act in the first place. And so you should be, you should be anti-opensourcing AGI if you're, like, pro-chips Act. And like, yeah, anyway, there's just a lot more complexity this than people realize. But like, the thing I want to get at is, like, if once we open-source it, once people can run their own models, like, at home, then it could be very, very, very hard to actually regulate it and control it moving forward. And so that's like a thing that keeps me up at night. Last question for now, because obviously this is going to be a fast evolving conversation. what do you recommend people go read to learn, right? If they're just trying to wrap their heads

around this, but maybe they don't have the 200 hours that you recommended, you know, but they want to have a meaningful take on it so that they can be part of the discourse, even if it's just understanding who to vote for because of what they're saying or whatever it is. You know, what do you recommend? If you're interested in technical things, go to aISafty. info and watch Robert Miles' YouTube videos. If you're interested in general in this problem, not so much the technical side, go watch Tristan Harris's AI Dilemma. It's like an hour-long talk, and it's just like a good kind of like summary of like the various risks involved and like what's at stake. Another person who, by the way, comes at tech ethics from the standpoint of being a tech

entrepreneur. He sold the company to Google and, you know, they were friends of mine back in back in San Francisco. So really interesting perspective and definitely someone who has thought deeply about these things and who also, I would say, you know, one of the challenges that I find with the discourse around AI is how much of it is re-litigating social media battles. You know, like if you watch the hearing, it was all just section 230, section 230, we shouldn't have had section 23 from the politician's side, right? It was all social media. If you read Andreessen's piece, it's clearly still kind of caught up in the social media wars and what the New York Times thinks about Silicon Valley and all that sort of stuff. Tristan actually has,

he started all of his work around questions of social media and I think is a little bit more capable of kind of getting outside of that perspective to bring this. So, cosign, I guess is what I'm saying. Yep. And then actually, two quick ones just thought of. One is don't look up the documentary. That one's a little shorter. It's not as in-depth on the problems, but it's still a really good intro. And then the last one is actually Tim Urban's wait but why post on artificial intelligence. It's kind of old that came out years ago, but it's really good at helping you get some deeper foundational ideas about like how transformative AI is likely to be. So go read his blog post on that. Awesome. Emerson, so good to have you on the show. I can't wait till the next one.

Thanks, man.