Your Undivided Attention - AGI Beyond the Buzz: What Is It, and Are We Ready?

Episode Date: April 30, 2025

What does it really mean to ‘feel the AGI?’ Silicon Valley is racing toward AI systems that could soon match or surpass human intelligence. The implications for jobs, democracy, and our way of life are enormous.In this episode, Aza Raskin and Randy Fernando dive deep into what ‘feeling the AGI’ really means. They unpack why the surface-level debates about definitions of intelligence and capability timelines distract us from urgently needed conversations around governance, accountability, and societal readiness. Whether it's climate change, social polarization and loneliness, or toxic forever chemicals, humanity keeps creating outcomes that nobody wants because we haven't yet built the tools or incentives needed to steer powerful technologies.As the AGI wave draws closer, it's critical we upgrade our governance and shift our incentives now, before it crashes on shore. Are we capable of aligning powerful AI systems with human values? Can we overcome geopolitical competition and corporate incentives that prioritize speed over safety?Join Aza and Randy as they explore the urgent questions and choices facing humanity in the age of AGI, and discuss what we must do today to secure a future we actually want.Your Undivided Attention is produced by the Center for Humane Technology. Follow us on X: @HumaneTech_ and subscribe to our Substack.RECOMMENDED MEDIADaniel Kokotajlo et al’s “AI 2027” paperA demo of Omni Human One, referenced by RandyA paper from Redwood Research and Anthropic that found an AI was willing to lie to preserve it’s valuesA paper from Palisades Research that found an AI would cheat in order to winThe treaty that banned blinding laser weaponsFurther reading on the moratorium on germline editing RECOMMENDED YUA EPISODESThe Self-Preserving Machine: Why AI Learns to DeceiveBehind the DeepSeek Hype, AI is Learning to ReasonThe Tech-God Complex: Why We Need to be SkepticsThis Moment in AI: How We Got Here and Where We’re GoingHow to Think About AI Consciousness with Anil SethFormer OpenAI Engineer William Saunders on Silence, Safety, and the Right to WarnClarification: When Randy referenced a “$110 trillion game” as the target for AI companies, he was referring to the entire global economy. 

Hey, everyone. This is Azaraskin, and welcome to your undivided attention. There's a question that you'll hear a lot around Silicon Valley these days. Can you feel the AGI? AGI is, of course, artificial general intelligence, and while there are many different definitions, and people actually fight a lot over the definition, because it turns out there's a lot at stake, you can still broadly understand AGI as the ability for an AI to replace human beings behind a screen. That is, if the economy can't tell that we swapped out a human with an AI, well, that's what AGI is. But what does it mean to feel the AGI? It means to feel the weight of the massive wave coming over the horizon and heading towards us.

It's to take seriously the idea that AGI or something even more powerful is coming and soon. Now, timelines vary. Sam Altman has said something like AGI could be built this year. Anthropics Dario Amadei says next year, Demis Sizabas of Google Deepvine gives it five to ten years. And in a recent blog post, the former OpenAI researcher and now whistleblower Daniel Kokatello predicts that we will have superhuman intelligence by 2027. My guess is 2026, 2027. Now, this can all feel like science fiction,

but we're living at a time when things that felt like science fiction just become real. You know, I'm the co-founder of Earth Species Project, and we are using Frontier AI to decode animal communication, animal language, and we believe that'll happen before 2030. So we have to take this all seriously. But it's also critical that we can distinguish hype from the raw facts from knee-jerk skepticism. So to do that, I've invited back CHT co-founder Randy Fernando,

who spent years at NVIDIA and has been tracking this space deeply from a technical perspective. By the way, we recorded this conversation when I was in the tropic, so you'll also hear some very non-human participants chiming in with their thoughts. And with that, let's get into it. Randy, thank you so much for joining us again on your undivided attention. Glad to be here, Reza. I don't actually mean to make light of this topic, but are you feeling the AGI? I think I do. I do. And one of the things I say is that if you actually try the models a lot, you try the technology, it's hard not to feel at least some of that, right?

If you use a new model like OpenEIs O3 or Gemini Pro 2.5 or Cloud 2.7, they're all pretty smart. And if you look at demos like OmniHuman 1, which brings images to life and pairs audio and does lip sync, and it all looks fantastic. Or you look at the latest voice generation or music or video. When you see those, it's hard not to feel the AGI. And I think a lot of people who may not be feeling it just haven't seen, like, what is possible. One of the things that I think is really important here is the way most people experience AI is a chat bond. And they expect, or the way they understand it's smartness is they ask questions and they sort of evaluate how smart the answers are back. But what that misses is when, say, 01 or 03, which our Open AI's reasoning model,

gain new capabilities where before 01 last December, if you asked one of the models to solve a PhD-level physics question where the answers aren't on the internet, the model would fail miserably, then 01 comes out, and suddenly it can answer 70% of them. Most people that never experienced that because they're not asking PhD-level questions, and so they don't see the exponential rate of progress. That's right. That's right.

And I think, you know, then getting to this AGI question, right, it doesn't take a fully general AI to already have a massive economic and societal impact. And when we say feel the AGI, one of the things we want is for people to feel the AGI not just in terms of the technology, but in terms of how it would land in society, right? Feel into the world that you want for yourself and for the people, you care about. Like, you can have these really powerful technologies, but you don't want AI that's going to trick you in a cyber scam

or have deepfakes happening of your kid at school or to be competing against AI agents when you're, let's say, buying a car or buying a house, or being manipulated when you're trying to vote. Or having AIs that train on your work and then come back and compete against you in the marketplace that you work. Or being automated out, right? And that includes things like Uber and Lyft and DoorDash, right, with respect to autonomous vehicles.

Those kinds of things we have to feel into as well. And we want people in companies and governments to feel into that too. So we can make the right decisions going forward. Now, before we get into definitions, I think we need to have a disclaimer about definitions. Because definitions are used often as a way of delay. or adding doubt. There are a tool that people deploy. So, you know, a perfect example before we, you know, sort of try to define AGI, is in social media, is there's been a lot of firepower put behind.

Well, what exactly do you mean by is social media addictive? Like, let's define addiction before we say it's addictive or not. And meanwhile, people are staring into their phones and kids are laying in bed. scrolling for hours on end. You don't need to get the definition perfect for, as you're saying, there to be real-world impact. And so industry often will try to make this a conversation, well, we have to know exactly what a harm is or exactly what a definition is before we do anything about it.

And meanwhile, they're just rolling out bulldozing society. So I think that's really important to say. And, you know, there are already hundreds of billions to trillions of dollars at stake because in the deal between Microsoft and OpenAI, Microsoft gets access to OpenAI's technology until they reach AGI. So obviously, now there's going to be a huge push to try to define AGI at something for open AI

that happens a little sooner so that they get access to all the economic benefits. And that's just one example of the kinds of weird incentives you're going to see around what AGI is. there's another incentive which is to when you're talking to a customer or to an investor you are going to represent your technology as being more advanced and so the definition of EGI gets a little looser if you want to extend the timeline and say oh don't worry right you're talking to the public and you're saying hey don't worry we are still really far from

aGI now you make the definition very stringent you say it's like level five autonomous driving it's got to be perfect so of course like it's going to take a lot of time to get there. And so you can see how adjusting the definition adjusts the timelines. There's this false dichotomy between near-term problems and like the really super-smart AI that goes off the rails, right? And sometimes people put these in this big tension. But I want to make the point that solving near-term problems will also help with the longer-term problems, almost all the time. So here are three examples, right?

One is with alignment and reliability. How can you be confident that an AI system will accurately do what you ask every time? Imagine this becomes really important if an AI agent has your credit card, right? Now you care a lot about it. You don't have a lot of tolerance for error. And it also applies to AI agents that are operating in our systems, like our financial system, for example. So that's what, alignment and reliability. The second one is interpretability.

Do we understand how the models work? Do we understand how they are reasoning and sort of coming to conclusions and taking actions? We have a long way to go, even on the systems we have today. And the last one, there's just examples, but the last one is an example of the impacts on jobs and meaning, right? When we have automation at scale, how do we handle that tsunami? And how many resources are we allocating to that? problem and all of these other problems these are much simpler problems in 2025 this is the simplest version of these problems we are going to have right in the coming years and if we can't

even solve them and if we're not even dedicating resources to them like sufficient right commensurate to the resources we are putting into advancing the technology how are we going to handle aGI and superintelligence so we've got to get these prerequisites in place yeah what are your I think pointing at here is that trying to define what artificial general intelligence is and what we've crossed it or not sets up our minds to look for the bright line and that

harms will only happen after that bright line. And of course intelligence is a multivariate smear. It's not clear when you posit and as we automate intelligence they're going to be increasing changes and risks to society and we need to be tracking

those along the way and if we don't then we're setting ourselves up to fundamental fail. And just note that debates about the definition of where that line goes really is about not taking accountability for the harms that happen along the way. I think that's critical to understand. Let's quickly try to do, even though we just said we shouldn't be wary of people that try to define AGI. I think it's really good to talk a little bit about what that means. And Randy, I think you've been putting a lot of thought into it. So give us your best shot. So I tend to lean towards the more practical definitions of AGI because it brings the timeline of carrying in so we can think more about the consequences.

I would say AGI is AI that's able to match human performance in the cognitive realm. I think Aza said also it would replace a human, right? A reasonable human. Replace a human at a computer. At a computer, that's right, on cognitive tasks and computer-type tasks. So that includes language, solving problems, explaining ideas, but also. also art, music, video. It requires the ability to complete long sequences of tasks reliably, right?

So like tens or hundreds of steps reliably happening. And it has the consequence of being able to largely or fully automate hundreds of millions of cognitive jobs, generate significant economic value, and accelerate scientific advancement, which leads to compounding effects. And just note, what are the incentive of the companies? The incentive of the companies are, well, they need to beat the other companies to making the most powerful version of AI. And if you can have your AI code for you, then you can accelerate your own rate of progress.

And that, of course, puts us in the most dangerous world, which is AI working to make AI faster, everyone racing, needing to make that go fastest. And so their AIs are starting to be able to model how AI researchers work in collaboration with other AI researchers, which means you can make an agent which accelerates the work. They can do sort of the work of interns as of last year, and they're getting better and better and better. So that's sort of like where things are going. And again, no, you don't need AGI anywhere in there to define it to know that this just accelerates the rate of progress. And if you want to feel it just as a listener, right, if you try something like deep research, you can get. a feel for this, right? You say, hey, do some research on a complex topic and it will go away

and do a bunch of thinking. So you can get the feel for like what's happening to research and this level of automation. And that is just a tiny flavor, a tiny taste of what it's like inside the companies. Now, I just want to name one distinction because we haven't got there yet. Some people talk about AGI, other people talk about ASI. And this is artificial general intelligence versus artificial super-intelligence. And just, again, this may all feel like science fiction. Why are we having this conversation when there are real problems in the world,

there's geopolitical instability, and we're having what feels like a conversation about something that is like artificial super-intelligence? What is that? But the distinction is artificial general intelligence is sort of roughly at human level intelligence. artificial super intelligence, well, that's intelligence beyond the human level. Some people call A-S-I, not just smarter than humans, but smarter than all of the cognitive

output of humanity combined. And so there's some distinction there, but both of those are things that people think, some experts that, you know, 2030, 2035, we might reach that. I would just add two quick things there. One is in terms of intelligence and human intelligence, again this point about patterns so much of what we consider to be intelligence is pattern recognition and extrapolation

so it's hard to say exactly how much but it really is a very large amount and these things are very good at that these transformers are very good at that the other thing with ASI is that it will also include the ability for AIs to collaborate very efficiently at scale

so you can think of specialized versions that are now talking to each other you can imagine a massive compounding effect. And a lot of this, again, is not science fiction now, right? You can start to see it as we see more of these demos of agents working and higher performance models. Your brain can sort of, you can extrapolate to that more easily. And the last thing I think is worth mentioning is that a lot of times people interchange

AGI and ASI. I think we sometimes do that too. just as a note like you'll hear those terms AGI really is the very capable but weaker one in ASI is the really strong massively capable one

So I think we should construct sort of an argument and Randy I'm going to lean on you a little bit for this and then I'll interject when I need to let's start with constructing the argument that AGI is possible

like what trends are we seeing why should we believe that we can get there? So here's what people would say, right? People who believe strongly, they would say things like, look, we've had these scaling laws, right? We take compute data and model size. We keep growing those.

And it's worked. It's brought us really amazing results. It's brought us emergent capabilities as the models were growing. We've got new GPUs coming all the time, larger data centers that we're investing in. So, like, that's going to continue to go, even if the rate's changing a little, like, that's driving innovation.

We've got transformers that are working really well. There's other people looking at new architectures. So that's all very promising. Recently, we had reasoning and reinforcement learning working together. There's a lot of headroom there. We found a big jump in performance, right? Like the performance graphs have changed in slope when we added reasoning. new benchmarks are being beaten regularly

hallucinations are dropping consistently they're not zero but they're dropping pretty fast and in terms of data reasoning models can generate quality data so we don't need to always rely on human data which we do tend to run out of and new models can use tools really well so now the models are smart enough

to rely on external tools and this is important because the external tools are usually very capable and they don't make mistakes. So for example, the calculator doesn't make mistakes. Python doesn't make mistakes. If you write the code right and you run it, it will run

the same way every time. So all these are reasons why we should take AGI very seriously. And now, Isa, maybe you can take the skeptic side and walk us through what are the things that skeptics say that give them pause. Yeah. Well,

let me give a run-through of some of the kinds of arguments that skeptics say. And just to name my own internal bias here, which is up until the end of last year, I was much more hesitant. I wasn't sure. I could see arguments both ways that were convincing. And so I was sitting in a place of maybe.

At the end of last year, after the reasoning model starting to be deployed, that really shifted my stance to, I think it is much more likely than not that before 2030, probably by 2028, or 2027 will have hit whatever some functional definition of age I is. So I just want to name my bias for everyone. So first big skeptical argument is that this is just motivated reasoning from the lapse, right?

It is in their interest to hype the capabilities of their models because that's what gets them investment, that's what gets them better employees so they can publish more, so they can get the next big bump in valuation, so they can raise more money and gain economic dominance and market dominance. another one is that it's just going to be too expensive that yes the models continue improving but there is but one internet as Ilya the co-founder of OpenAI would say

therefore we will run out of data and the models will stop getting better and indeed it sort of looked like that was the case right we were sort of stuck at GPD 4 for a long time and now we're at GPD 4.5 what's going on there well that That's because the models were learning the patterns of the world via data on the internet. We ran out of data, so we stopped learning better models of the world,

what machine learners would call representations. And along then came at the end of last year reasoning models. DeepSeek does this, 01, 03 does this. A lot of the companies now all have these different sort of thinking modes. And what that does is that it uses the base model. model, it's a kind of intuition, and then it uses the intuition to reason, to have chains of thought, trees of thought, to find the very best answers by thinking through many different pathways. And what OpenAI found is that you could get, you know, a much, much better answer

by having the computer think for, say, a hundred times longer. So the longer it thinks, the better the answers, the better the answers, the better data you have now for, training a better base model intuition, and that thing can go recursive. And so a lot of the critiques that people had around, well, we're going to hit a data wall is what they called it. So we will never get to AGI. Those fell away at the end of last year. And actually, just so people know, my belief about how fast we're going to get to general intelligence changed. Before I'm like, well, I'm not sure. Maybe if we keep scaling up, but we don't yet have a good solution to the end of data. After 01 and 03 came out, that was a proof positive.

We were sort of waiting for that. We didn't know if it was technically possible, but everyone knew that that's what the labs were working towards. After the releases of those models, the question about data, in my mind, went away, and now it feels like there is a straight shot. Another argument that people make for why we might not reach AGI is that the models are trained to pass the test. That is to say they're very good at solving benchmarks, but maybe they're not as good at solving open-ended, ill-defined, long-term tasks. And so we will get machines that are very intelligent in a narrow way, although narrow means anything that can be tested for, that means AI will be very good at any subject that has theoretical in front of its name, math,

theoretical physics, theoretical chemistry, A.O. will be very good at that, but maybe those smushy things that human beings are very good at, like AI will not be good at. Another one is that, you know, this is not real intelligence, that AI doesn't really understand the world. They don't really reason.

They don't do it the way humans do. Look, humans learn on so much less data than the AIs do. And so they're just memorizing and, you know, speaking to the test. They're not really doing things. and then the final one is geopolitical risk that the world is heating up, there's going to be bottlenecks in supply chains,

and so there just aren't going to be enough chips. So I think that's sort of like the sum total of all the best arguments that I've found. But one more, which is reliability, right? Like, they're not reliable for large, longer sequences of steps that you can do. That's increasing every month. But when you say, hey, can you do three steps, it works.

When you do nine steps, 20 steps, it starts to fail. And those probabilities compound very fast. So as soon as you can't do something for like five steps, it starts to really fall on its face for longer sequences. So that's another reason to say, hey, gosh, we're a long way from that, you know. Yeah. And that sort of, if you put these together, you get a story or narrative for why we might not reach AGI by 2027, 2028, or even 2030.

It's the models are succeeding, but only for, for specific benchmarks in real-world tasks. We're trying to do real-world software engineering. They keep failing. They can't do long-time horizons, so they're good for toy problems. Because they're only good for toy problems, eventually that catches up with the labs. The labs can't raise money because they're not economically valuable enough. And so even though maybe it would be technically possible to build models if you can get enough

investment, you can't get enough investment, so we go through another winter. That's sort of the argument for the best argument I know how to make for why we might not reach. AGI. But it's hard for me to make that argument because what we're seeing empirically is that every seven months, AI can do tasks that are twice as long as they could before. So if they could do a task for one minute, if it would fail at two-minute tasks, just wait seven months and now they can do two-minute tasks, you wait another seven months, they can do four-minute tasks. They're already up to an hour-long task. So now we're going to be, you know, seven months, it's two hours, then four hours, and you see it doesn't take that long before you

can do day-long or week-long task. And once you do week-long, now you're into month-long, now you're into year-long, and this is the power of exponential. And those are human-equivalent times, right? Those are human-equivalent times. Like when AIS is a week-long, it means what a human would typically take a week to do. The model does it much faster. I'm going to make a point about when people say that AIs aren't doing real reasoning or they don't have a real understanding of the world, is that this is a real distraction. And the reason why is that they're trying to draw some moat around what makes us special. And the point is that when a simulation or simulacra gets good enough, it doesn't matter

whether AIs are doing real empathy or real reasoning or real planning, if they can simulate it well enough that the economy can't figure out whether it's a human or an AI, then it will be real impact on society. That's right. That's right. And it's not that the argument isn't fascinating. It really is. It's a fascinating conversation, but it completely bypasses. it diverse energy from where the energy should be, which is the practical implications of what

we even have now, which is already doing a lot of these, you can see the levels of automation that are already there and the implications of that. And we just can't get distracted. Our threshold is like, okay, where are the impacts, where are real things happening that we need to address right now? And so that's why we tend to move that part of the conversation aside and say, look, let's look at the impacts that are happening right now. That's right. And whether you believe AGI is coming or not, there are tens of trillions of dollars going into improving the capabilities

as quickly as possible to race towards replacing human beings behind computers with AIs because that's what's economically valuable. It's a $110 trillion game, right? That game is a $110 trillion game, and that is the actual game that these companies are in. People sometimes forget that because they, I think it's like, we're in the chatbot game, right? Or we're in the Gen A.I.

And the whole thing, the big pie, is the one that everyone's looking at. Okay. So we've been walking through the arguments for and against, at a technical level, why general intelligence is a thing that will be discovered or invented in the next couple of years. But we haven't really talked yet about the stakes of what is it to birth a new intelligence. if you will, that is at the level or smarter than humans. So, Randy, I think it'll get a little philosophical,

but let's talk about the implications and the stakes. So there's a few viewpoints on this, and maybe I'll give a few thoughts just to ground, like where I come from in these conversations. I kind of get back to what is happiness, what is the purpose of our lives, and I get back to the basics of, like, I would like everyone to have food, clothing, shelter, medicine, education, right?

These things matter a lot. And millions and actually billions of people, right, don't have healthy access to these things. So this is kind of where I come from, like the beginning of when I enter into conversations about AI and alignment and, you know, how fast should we run and all of these things. That's my basis. So with that said, I'm sure you've got some thoughts on these, and there's a bunch of avenues to explore here. Well, I think it's important to start.

The founder of or co-founder of Deep Mind, it's now part of Google, famously said as their mission statement, that first solve intelligence, then use that to solve everything else. Strong AI and owning, intelligence is the one ring of our time, right? The Tolkien one ring. Whoever owns that owns technical and scientific progress, owns persuasive and cultural dominance. Owns sort of the whole

thing, right? You own all of the cognitive labor, all the thinking of the world. That is a very, very powerful thing, and that means it sets up the greatest incentive to race for it, regardless of the collateral damage along the way because this is a winner-take-all war. And I just want to like set that up because this is how you get to, you know, Elon Musk saying things like it increasingly appears that humanity is a biological bootloader for digital superintelligence. And anyone hearing that would say like, well, then don't build it. We shouldn't replace ourselves. But then the next thing people will say is, well, we can't not build it because if we don't build it, then we'll lose to the people or the company or the country that does.

And then you end up, like, when you actually talk to these kinds of accelerationists that are excited about this, they'll say things like, well, even if, you know, we lose control, which is sort of a funny thing to say because we actually haven't yet figured out how to control these systems, and they are starting to exhibit deception, self-preservation tendencies, because it's trained on humans and human beings do those things. they say even if it kills us all, it'll still be worth it because we created a god or it's still worth it because at least it was the U.S. that created it, so it'll be U.S. values that continues to live on.

It's these kinds of things that people say. And I really want people to hear that this is not some fringe philosophy. So what is it is described might sound outlandish, but these are real things. These are real philosophies. And it is hard for me personally to relate to because I'm much more interested in what happens to, you know, the humans and the animals and the environment around us. Like we have to take care of those things. There's something that just goes back to food, clothing, shelter, medicine, education, like the things we need to take care of for people to be, to not be suffering, right? To be reasonably happy.

That we have some debt. I almost feel like it's a debt that you owe if you discover these kinds of. technologies that you have to share them and you have to make sure they are distributed in a way that takes care of people. And actually, a lot of the AGI leaders are saying that too. They don't disagree with that. But when it comes to the details, it's always like, oh, yeah, that's pretty complicated. And we're going to focus more of our energy on, like, how do we keep advancing the technology? This is a position that I think leaders are backed into. to because they don't want to lose the race, because they don't want to lose the race,

they are forced to take the position that, well, maybe that nihilistic, we're just a bootloader is the right position to take, because if you take that position, it confers you power now. I think that's really important for people to understand. It's very convenient. It's very convenient. And it's not everyone's belief, but it is the belief of some people that have a massive amount of capital and power for them to enact their worldview. So I think that's just really important for people to understand.

And also part of that worldview is saying, hey, don't worry. Like when we go fast, yes, some bad things will happen. But things that are illegal are illegal. People just won't do them or we'll make sure they don't do them. And okay, so if we're going to say that, then what efforts are we putting into? what resources are we actually putting into making sure those bad things are actually illegal like you actually can't do them

and what happens a lot of the time is the rhetoric is there but the allocation of actual resources actual money actual headcount to doing that research figuring out those problems is not happening one other consideration is that as AI systems become more powerful

they become harder to control because they have more degrees of freedom in the world, right? So whatever rules you thought you had said that were sufficient, they will find cracks in those rules. They will find ways to work around. Just like they're good at solving a problem you give them, they will naturally tend to accrue resources or power or keep themselves on. These are sort of natural things that you would do to be successful in the world.

And they will find ways to do that, and they will find ones that you have not thought of. As we integrate more and more of these technologies across society, right, they start to work well and we start to rely on them. And then we increasingly don't really understand where all the decisions are being made and yet we've given up more and more power to these systems, right? The technical term for this is called gradual disempowerment. So we actually build a situation where we as humans become highly disempowered, right,

in the world that we actually recreated and we live in. And so there, when AIs go off the rails, for any reason, right, it could be just something, an inaccurate judgment that it makes, or it could be something more malicious or deceptive, right? For some reason, it decided to do something that we really don't want it to do. We're kind of screwed in that scenario because we don't even understand how the system's working at that point. This isn't academic or theoretical anymore. Anthropic released a paper where it showed that AI started to scheme when it learned that

the anthropic programmers were going to retrain it to have a different set of values. It started to try to figure out how to copy itself to another server and also lie about what answers should be so that Anthropics researchers would think that it was being successfully retrained. On top of that, there's a point that even the chains of thought this is another recent research example even the chains of thought

that the models generate they look really good right when you look at them you're like wow that sounds like exactly what it's thinking and they're not they are often not even largely true sometimes it's less than 50% accurate

in terms of what they're actually saying so that's another example where already we are in a situation where there's a lot of opakness to how the model work and a very rudimentary understanding of what is actually going on, even by some of the best researchers in the world who built these very products. So I want to then just name, there's the alignment problem, which is can we get AIs to do what

we want them to do? Then there's the polyalignment problem, which, you know, it's sort of coining a term here, but it's the ability to align the sum total of all AIs to do what's good for humanity and the rest of the beings on this planet. The joke goes, like we're all talking about whether AI is conscious or not when it's not even clear that humanity is, which is to say that we as humanity, we keep getting results that nobody wants. No one really wants growing climate instability, and yet the nature of our political, geopolitical system means that if I don't burn the oil and you do, I get the industrialized nation, you don't, therefore I have to, and so we end up with climate instability. Same thing with forever chemicals polluting the world

and giving us all cancer, things like this. We keep getting things we don't want. So if we just increase the power running through that system, because human beings haven't yet shown they're actually in control so we can steer the world the way we want, then that's another different way of saying we have lost control or lost the ability to decide. And again, if we can't get simple versions of this to work now in 2025, when all of these problems are the simplest they're ever going to be, that does it bode well for the future, right? And so shifting attention to that and saying, how do we wrap our hands around these technologies right now? It's just crucial. And this is why the rhetoric

that we must beat our foreign rivals to AI is actually sort of missing the point. The competition can't just be to race towards making something that we can't control because there's a built-in implicit assumption that just like with guns and with airplanes, the more powerful you make it that just as much in control we are. With AI, it's not like that. That the race needs to be for making. a strengthened version of your society. And whoever does that better wins,

and we are not setting ourselves up right now to do the strengthening of our society versus just the generating power, which is uncontrollable. And there's a worthwhile principle here in these examples that Isa gave, right? Which is, the more general purpose a technology is, the harder it is, the harder it is.

is to disentangle its benefits from its harms. That is why this generation of technology, whether it's automated, you know, cognition or physical, right, you know, the AI stuff, the robotic stuff, all of that becomes very coupled, right, in terms of benefits and harms, because they're so flexible. And that is why we have to do the societal upgrade that is as talking about. There's no other way to kind of responsibly wield these technologies. And the difference, of course, between AI and every other technology is that if you make technology that makes, let's say, rocketry better, that doesn't also make medical research

better and mathematical advances better. But if you make advances in AI, because AI is fundamentally intelligence, it means you get advances in rocketry and biomedical advances and in mathematics. You get them all. And so the rate of change that society is going to have to deal with is going to be immense greater than we have ever faced. And then it's not like it'll stop. It'll just keep going at a faster and faster rate. This is why it makes it the hardest problem that humanity has ever had to face and likely ever will. And I think to do it, there is going to have to be some kind of international cooperation,

which, I'm just going to name it, right now, feels pretty much impossible. And we have some historical analogies for this. And, you know, Randy, you like to point out that there are no good historical analogies. This is unlike anything we've dealt with. Listen, each one has some flaw. I would say that. Well, the obvious example, and with the caveat that none of these examples are going to be perfect analogies, the obvious one is, of course, nuclear weapons.

Another place to look for hope here is blinding laser weapons. There was a international treaty signed in 1995 that banned blinding laser weapons in war. And the other one that goes along with that is germline editing, the changing of the human genome in a way that continues forward, that propagates, we as a species have successfully not walked down that technological road. The reason why we bring this all up is because it can often seem like if technology wants to bring humanity in some direction, technology wins. Humanity doesn't get to choose.

But that's not always the case. And the times when it isn't the case is when that thing which is valuable beyond which words can express about ourselves, if that is threatened in a visceral enough way, we can choose and we have chosen in the past, different path. Don't think of this, though, as, like, hope washing. It's not like, and therefore we can do this. That's not what I'm saying.

But it's just trying to, like, point at places where we can find non-naive hope, but we're going to have to collectively work very hard to get there. And I think there are some things we can put into place now. There are some really practical things. So these are things that I would love to see more energy on, especially from tech leaders. There are reasonable shared values we can build around, right? And don't kill, don't lie, don't steal. These are basic things that are shared across almost the entire human population.

It's coming back to having this standard of for ourselves and for the products we produce, that they espouse the values we would teach our children to be good citizens in the world. So that's one important thing. Then even more practically, right, get the incentives right, think about what are the incentives driving when you do analysis, right? Think about that. Get price to fold in harms, right? Our economic system is built around this magic of price where price is this one number that coordinates a lot of different resources and it reflects information and it reflects harms and it reflects this intersection of supply and demand. All that magic works reasonably when price reflects all of the major costs.

So if there's some damage being done, price needs to fold that in. And then the system can kind of make the right decisions. So make sure we get harms back into price. Harms have to show up on company balance sheets. So that's a really important principle. I think if we can't get price to fold in harms, we have a big problem. we tend to look a lot at GDP as the ultimate measure. But as power and wealth concentrate,

GDP is going to be increasingly a bad measure of success because GDP going up will not correlate well with most people's actual experience. So we need to put a lot of attention on that and kind of figure out how are we going to solve those problems. And then there's all these practical questions about what happens, as people get automated out to different degrees, this process is already beginning.

How do people get food on the table? How does that work? There's lots of different creative solutions people have come up with, but we need to really center those conversations. And I think the tech leaders have to see this as part of their responsibility. When they create these technologies that are of a really valid, vastly different scale than any technologies before. These are general automation technologies.

There are really big questions to answer, and we just can't shrut those off any longer. And while it seems very, very challenging to impossible, it's very important to notice the gap that if every human being just stopped what they're doing just sat down, we would never get AGI, we would never get ASI. And so it's not like the laws of physics are pushing the bits and atoms out into the world

that makes a uncontrollable superintelligence or some total of all AIs that push humanity in the wrong direction. So it's not physically impossible. And I think that's so important to hold because now the gap between not impossible and merely the excruciatingly difficult is a very important gap to hold because there is some kind of possibility in here

and the goal now is maximum clarity so we can all start to, in our own spheres of agency, move in the right direction. So building on that, as we think about, like at the highest level, when you kind of zoom out and say, okay, as a listener, what should I hold in my mind for like a framework

for how we escape the AI dilemma? Here's one way I like to think of it. There's five pieces. So one is we have to have a shared view of the problem and the path forward. At CHT, we spend a lot of time on this because it is the prerequisite for a lot of the other pieces to happen.

So that's the first one. Shared view of the problem and path forward. The second one is incentives and penalties, right? So when you do the right thing, you get rewarded. And when you do the wrong thing, you get penalized. This is back to that harms on balance sheets principle. Paired with that is a kind of monitoring and enforcement. There has to be a way to know

did you do the right thing or not and some certain appropriate levels of transparency that pair with that. Then there's governance that can keep up with the pace of technology, right? Technology products shift. They're being updated all the time. Sometimes it's week by week.

A lot of the updates are invisible. But there's major releases like at least every month. Is our governance keeping up with that? Like how do we do that? We have to have systems that can get feedback from citizens where we can make decisions integrate large amounts of information and respond quickly. And then the last thing is coordination at different levels. So that goes from the local to state level to country level to global coordination.

And these are all pieces that we are going to need to kind of escape the dilemma. But if you want a simple framework, I think that's a good one to keep in mind. And the only thing I'd add to that is whenever people, invoke the competition frame for why we have to race, the question that we need to ask, and you can ask back to whoever it brings up, like, but we have to win, is to ask the very simple question, but win at what? Are we winning at the pure power game for something that we don't yet know how to control? Or are we winning at strengthening our society so that our values win? If we don't get that clear, then the rest of the conversation will get

stalled in A, but we have to win. Okay, so we started this conversation by talking about, you know, the question that I got asked sitting down at dinner with one of the leading sort of alignment safety researchers, can you feel the AGI? And I think for most people, they're not really feeling the AGI yet. The future isn't equally distributed. When, Randy, do you think people are going to start to feel it in their lives? What is that going to look like?

I think we should just briefly walk through that before we end. Yeah, I mean, honestly, I think in my experience in presenting to people, it's just that they haven't been in direct contact with where the technology is today. It's not even about being in contact with an imaginary, like, future super-capable AGI. It's just seeing today's state of the art. And when they see that, they can see. They see the implications very quickly for their lives, for their children, for their parents, for their grandparents. All of that stuff just comes crashing down very, very easily.

And I think it's just a matter of being curious and spending some time looking at the demos. I think we'll try to link some of them in the notes for this podcast so you can actually check out a few links and learn and feel that experience for yourself. you know Randy you and I are thinking and looking at this kind of stuff all the time and it can be really challenging you know right now I am down in Costa Rica and my neighbor's son is 17 is Costa Rican and he was asking me yesterday what should he study he's like I you know he really wants to study engineering. And it was hard for me to answer that question because I wanted to say, yeah, study engineering. Actually, you should study AI, so you set yourself up. But it's actually, it was a very hard question to answer because, you know, he's 17 or 18

by the time he gets through college. I actually don't think that'll have been the right thing for him to study. And this is, of course, a microcosm of the overall problem, which there isn't a good answer to that question right. now and whatever age you are yeah right exactly it's hard i can sort of see the way to my own obsolescence in an economic sense and i just want to be there with everyone this can be very challenging and like to ask what is the solution to AI is like asking what species is a forest. It's a malform question. It's going to be a messy, emergent ecosystems of

things that let us steer. And we have to be okay with the not knowingness, while also not letting that be an excuse for not doing anything. I think public pressure is going to play a huge role in this next cycle. Like how this all plays out, really, depends on public pressure, perhaps more than any other lever. If you forced me to pick one, I think that's the one I would pick at this time. And I think there's a really practical thing you can do. If you're looking for like, okay, what can I do? It is re-centering, recalibrating conversations, pushing them back to the right questions again and

again and again. And when I think about the massive audience that listens to this podcast, if everyone just does that, right? On whatever social platform, whatever small groups, whatever private conversations, whatever company meetings you are at, if you ask those questions and just keep redirecting attention, we want to get in front of these problems, right? They get really scary. Right now, like, it's not all unfolded. So let's just act on it. Let's redirect those conversations. Like, have the right conversations consistently, which then translates into allocating the right resources and attention on the right problems consistently, I think that's how we give ourselves

the best chance. Well, I think that's a wonderful place to end for today. Randy, thank you so much for coming back. It's always so much fun to have these conversations, even if the topic itself isn't particularly fun, although it is fascinating, and that's the confusing part. And thank everyone, too, for listening and giving us your undivided attention. Thanks, everybody. Your undivided attention is produced by the Center for Humane Technology, a non-profit working to catalyze a humane future. Our senior producer is Julia Scott, Josh Lash is our researcher and producer,

and our executive producer is Sasha Fegan, mixing on this episode by Jeff Sudakin. Original music by Ryan and Hayes Holiday, and a special thanks to the whole Center for Humane Technology team for making this podcast possible. You can find transcripts for our interviews and bonus content on our substack and much more at humanetech.com. You can also watch all of our episodes on our YouTube channel. Just search for Center for Humane Technology. And if you liked this episode,

we'd of course be grateful if you could rate it on Apple Podcasts and on Spotify. It really does make a difference in helping others join this movement. And if you've made it all the way here, let me give you one more thank you for you giving us your undivided attention.