The AI Daily Brief: Artificial Intelligence News and Analysis - TIME Asks: Is AI the End of Humanity?

Episode Date: June 3, 2023

TIME Magazine's current cover story is titled "The End of Humanity: How Real Is The Risk?" It's a marker of how much the AI risk and safety conversation has gone mainstream. On this episode, NLW reads two pieces:   AI Is Not an Arms Race - Katja Grace The Darwinian Argument for Worrying About AI - Dan Hendrycks The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI Breakdown, we're reading selections from Time Magazine's new cover story, The End of Humanity. The AI Breakdown is a daily video and podcast about the most important news and discussions in AI. Like, subscribe and share, and go to Breakdown.network for more information. Welcome back to the AI breakdown. Last Sunday, I shared with you guys what I told you was an experiment. It was a reading of a recent article that I thought was illuminating or at least worthy of discussion. And the feedback has been really good.

This doesn't necessarily surprise me as Long Read Sunday has been a long-beloved feature of the other breakdown show. But I think when it comes to AI, there is such a frenzy around new news that having a chance to actually reflect on an essay that someone has taken the time to construct can be really valuable. Now, Time magazine has a forthcoming cover story called The End of Humanity, subtitle, How Real Is the Risk? And this, of course, captures the cultural moment we're in where people are really taking AI risk seriously in a way that we weren't just a couple of months. months ago. So for this week's reading, I'm going to actually read two selections from that issue of time. The first is called AI is not an arms race, and the second is the Darwinian argument for worrying about AI. Now, for those of you hoping for the regular Saturday weekly recap, fear not, it will be coming tomorrow instead. The first selection, AI is not an arms race, was written

by Katia Grace. Grace is the lead researcher at AI Impacts, which is an AI safety project at the nonprofit machine intelligence research institute. Katya writes, the window of what AI can't do seems to be contracting week by week. Machines can now write elegant prose and useful code, ace exams, conjure exquisite art, and predict how proteins will fold. Experts are scared. Last summer, I surveyed more than 550 AI researchers and nearly half of them thought that, if built, high-level machine intelligence would lead to impacts that had at least a 10% chance of being, quote, extremely bad, e.g. human extinction. On May 30th, hundreds of AI scientists, along with the top CEOs of, along with the CEOs of top AI labs like OpenAI, Deep Mind, and

Anthropic, signed a statement urging caution on AI. Quote, mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks, such as pandemics and nuclear war. Why think that? The simplest argument is that progress, in AI could lead to the creation of superhumanly smart artificial people, with goals that conflict with humanity's interests and the ability to pursue them autonomously. Think of a species that is to Homo sapiens what Homo sapiens is to chimps. Yet while many fear that AI could mean the end of humanity, some worry that if we, usually used to mean researchers in the West or even researchers in a particular lab or company,

don't sprint forward, someone less responsible will. If a safer lab pauses, our future might be in the hands of a more reckless lab. For example, one in China that doesn't try to avoid substantial risks. This argument analogizes the AI situation to a classic arms race. Let's say I want to beat you in a war. We both spend money to build more weapons, but without anyone gaining a relative advantage. In the end, we've spent a lot of money and gotten nowhere. It might seem crazy, but if one of us doesn't race, we lose.

We're trapped. But the AI situation is different in crucial ways. Notably, in the classic arms race, a party could always theoretically get ahead and win. but with AI, the winner may be advanced AI itself. This can make rushing the losing move. Other game-changing factors in AI include how much one-party's safety investments reduce the risk for everyone, whether coming second means a small loss or major disaster,

how much the danger rises if additional parties pick up their speed, and how others respond. The real game is more complex than simple models can suggest. In particular, if individual uncoordinated incentives lead to the sort of perverse situation described by an arms race, the winning move where possible is to leave the game. And in the real world, we can coordinate our way out of such traps. We can talk to each other.

We can make commitments and observe their adherents. We can lobby governments to regulate and make agreements. With AI, the payoffs for a given player can be different from the payoffs for society as a whole. For most of us, it may not matter much if meta beats Microsoft. But researchers and investors chasing fame and fortune might care much more, Talking about AI as an arms race strengthens the narrative that they need to pursue their interests. The rest of us should be wary of letting them be the ones to decide.

A better analogy for AI than an arms race might be in a crowd standing on thin ice, with abundant riches on the far shore. They could all reach them if they step carefully, but one person thinks, If I sprint, then the ice may break and we'd all fall in. But I bet I can sprint more carefully than Bob, and he might go for it. On AI, we could be in the exact opposite of a race. The best individual action could be to move slowly and cautiously, and collectively, we shouldn't let people throw the world away in a perverse race to destruction, especially when routes to coordinating our escape have scarcely been explored. All right, back to NLW here.

The quick thing that I want to note after this piece is that the main way that I'm seeing this arms race analogy happen is less between us or U.S. companies and another global political power and more a capitalist competition between, Google, Microsoft, and meta. This, for example, it seems like, is what Jeffrey Hinton has been talking about as a catalyst for him leaving Google, that when there was an incredible market pressure for Google to move fast, they were being more responsible. Once ChatGPT started to threaten their core search business, they started moving in a much more aggressive way. In other words, this particular arms race isn't like the Cold War. It's viewed as a zero-sum market competition, in which the next decades of business are going to be decided in the next few months or years. There are massively powerful financial incentives aligned around that, and so that might be the point at which

there needs to be some sort of intervention. Second, we read the Darwinian argument for worrying about AI. This piece was written by Dan Hendricks, who is the director of the Center for AI Safety. Now, it was, of course, the Center for AI Safety that coordinated that one sentence statement signed by so many that was shared earlier this week. Dan writes, A broad coalition of AI experts recently released a brief public statement, warning of the risk of extinction from AI. There are many different ways in which AIs might become serious dangers

to humanity, and the exact nature of the risks is still debated. But imagine a CEO who acquires an AI assistant. They begin by giving it simple, low-level assignments, like drafting emails and suggesting purchases. As the AI improves over time, it progressively becomes much better at these things than their employees. So the AI gets promoted. Rather than drafting emails, it now has full control of the inbox. Rather than suggesting purchases, it's eventually allowed to access bank accounts and buy things automatically. At first, the CEO carefully monitors the work, but as months go by without error, the AI receives less oversight and more autonomy in the name of efficiency. It occurs to the CEO that since the AI is so good at these tasks, it should take

on a wider range of more open-ended goals. Design the next model in a product line, plan a new marketing campaign, or exploit security flaws in a competitor's computer systems. The CEO observes how businesses with more restricted use of AIs are falling behind, and is further incentivized to hand over more power to the AI with less oversight. Companies that resist these trends don't stand a chance. Eventually, even the CEO's role is largely nominal. The economy is run by autonomous AI corporations, and humanity realizes too late that we've lost control. These same competitive dynamics will apply, not just to companies, but also to nations.

As the autonomy of AIs increases, so will their control over the key decisions that influence society. If this happens, our future will be highly dependent on the nature of these AI agents. The good news is that we have a say in shaping what they will be like. The bad news is that Darwin's laws do too. Though we think of natural selection as a biological phenomena, its principles guide much more, from economies to technologies. The evolutionary biologist Richard Luwanton proposed that natural selection will take hold in any environment where three conditions are present.

One, there are differences between individuals. Two, characteristics are passed on to future generations. and three, the fittest variants propagate more successfully. Consider the content recommendation algorithms used by social media platforms and streaming services. When particularly addictive algorithms hook users, they result in higher engagement in screen time. These more effective algorithms are consequently selected and further fine-tuned, while algorithms that fail to capture attention are discontinued. This fosters the survival of the most addictive dynamic.

Platforms that refuse to use addictive methods are simply outcompeted by platforms that do, leading to a race to the bottom among competitors that has already caused massive harm to society. In the biological realm, evolution is a slow process. For humans, it takes nine months to create the next generation, and around 20 years of schooling and parenting to produce fully functional adults. But scientists have observed meaningful evolutionary changes in species with rapid reproduction rates like fruit flies in fewer than 10 generations. Unconstrained by biology, AIs could adapt and therefore evolve,

even faster than fruit flies do. There are three reasons this should worry us. The first is that the selection effects make AIs difficult to control. Whereas AI researchers once spoke of designing AIs, they now speak of steering them. And even our ability to steer is slipping out of our grasp as we let AIs teach themselves and increasingly act in ways that even their creators do not fully understand. In advanced artificial neural networks, we understand the inputs that go into the system, but the output emerges from a black box with a decision-making process largely indecipherable to humans.

Second, evolution tends to produce selfish behavior. Amoral competition among AIs may select for undesirable traits. AIs that successfully gain influence and provide economic value will predominate, replacing AIs that act in a more narrow and constrained manner, even if this comes at the cost of lowering guardrails and safety measures. As an example, most businesses follow laws, but in situations where stealing trade secrets or deceiving regulators is highly lucrative and difficult to detect, a business that engages in such selfish behavior will most likely outperform its more principled

competitors. Selfishness doesn't require malice or even sentience. When an AI automates a task and leaves a human jobless, this is selfish behavior without any intent. If competitive pressures continue to drive AI development, we shouldn't be surprised if they act selfishly too. The third reason is that evolutionary pressure will likely ingrain AIs with behaviors that promote self-preservation. Skeptics of AI risks often ask, couldn't we just turn the AI off? There are a variety of practical challenges here. The AI could be under the control of a different nation or a bad actor. Or AIs could be integrated into vital infrastructure, like power grids or the internet. When embedded into these critical systems, the cost of disabling them may prove

too high for us to accept, since we would become dependent on them. AIs could become embedded in our world in ways that we can't easily reverse. But natural selection poses a more fundamental barrier. We will select against AIs that are easy to turn off, and we will come to depend on AIs that we are less likely to turn off. These strong economic and strategic pressures to adopt the systems that are most effective means that humans are incentivized to seed more and more power to AI systems that cannot be reliably controlled, putting us on a pathway towards being supplanted as the Earth's dominant species. There are no easy, surefire solutions to our predicament. A possible starting point would be to address the remarkable lack of regulation of the AI

industry, which currently operates with little oversight, much of the research taking place in the dark. Regulation needs to be done proactively rather than reactively. If something goes wrong in this domain, we may not get the chance to fix. it. The problem, however, is that competition within and between nations pushes against any common sense safety measures. AI is big business. In 2015, total corporate investment in AI was $12.7 billion. In 2021, this figure had grown to $93.5 billion. As the race towards power of systems quickens, corporations and governments are increasingly incentivized to reach the finish line first. We need research on AI safety to progress as quickly as research on improving

AI capabilities. There aren't many market incentives for this, so government should offer robust funding as soon as possible. The future of humanity is closely intertwined with the progression of AI. It is therefore a disturbing realization that natural selection may have more sway over it than we do. But as of now, we are still in command. It is time to take this threat seriously. Once we hand over control, we won't get it back. All right, so back to NLW here again for a quick wrap-up. Folks who deal with AI safety and AI risk, people who have designed their careers or their lives around those issues, have been screaming into a void for a very, very long time. The interesting place, however, that we find ourselves now is that alongside this emergence of

CHATGBT in particular, but AI in general, as a force in broad consumer consciousness, a mainstream force that is already impacting people's jobs and lives. The discussion of AI risks has moved from the fringes into the mainstream. Time putting the end of humanity, how real is the risk on the cover of their issue this week, is example of exactly that. What's more, we had a really interesting moment earlier this week. A blog reported from a military conference that a colonel in the U.S. Air Force had said that they had run an AI simulation in which an AI drone killed its own operator because they viewed that operator as getting in the way of its mission. This seeming to be exactly the type of scenario that AI risk people

had worried about, it was like lightning and went all over the internet extremely quickly. Publications around the world, and in the very, very mainstream of America, published this as though it were real. Fast forward to the next morning, the US Air Force denies that any such simulation ever took place. And finally, the colonel comes back up and says, actually, it was theoretical, but that still shouldn't change the fact that that's the type of thing that could happen. Now, if you listen to my episode about that, you heard that I was pretty frustrated with the colonel, and basically argued that this is exactly the type of thing that likely to diminish and make people take the AI safety conversation less seriously, because they

are reasonably allergic to, and have their guards up against, being manipulated and lied to. However, I had a few comments on that YouTube video that I thought were really insightful. And basically, they said, sure, it sucks that this guy was kind of fibbing, even if you don't want to call him a full liar. Let's call it over-dramatizing a theoretical, and maybe not doing so much to clarify that it was just in fact a theoretical. But what had happened was effectively an immune response from humanity. The fact that people cared as much as they did was, these comments argued, a really good sign. I think that's a pretty fair point. My belief continues to be that the hundreds of millions of new people who are paying attention to AI now that weren't six

months ago really are still trying to wrap their heads around questions of AI safety and AI risk. Those questions sit alongside much more prosaic and personal questions of how AI is going to impact their lives. It strikes me that given the risk and safety conversation is no longer on the fringes, it may be time to start to transition the core of that conversation away from things like why we should care, because there seems to be lots of evidence that people do care. And instead, we should probably start to talk about specifics. You see this happening a little bit already. People weren't cane on the idea that open source researchers would have to get licenses for their projects in a way that could just reinforce regulatory capture for big companies like meta and OpenAI and Microsoft. But maybe they had less of a problem with the idea that things above the power level of GPT4 had to go through some sort of licensing process.

That's an example of getting into specifics. I think even thornier are going to be questions of how to address the market mechanism, which create such incredible incentives, for people to wire every system with AI as fast as they possibly can. It strikes me that it's going to bring up maybe the biggest conversation we've had about the nature of free markets and capitalism in a very, very long time. I certainly don't know what the right answers are, but it does feel to me like we've turned a corner where we can begin to have a different and much more practical conversation. Anyways, guys, that was it for this week's Long Reads Saturday, I guess, instead of Longreed Sunday. Again, that Time magazine cover story is called The End of Humanity, How Real is the Risk? And this has been the AI breakdown.

If you found this useful, please like, subscribe, and share, hit the notification button so you don't miss any episodes. Go check out the podcast or the newsletter version. And thanks for taking the time to listen. Until next time, peace.