The AI Daily Brief: Artificial Intelligence News and Analysis - AI Alignment and AGI - How Worried Should We Actually Be?

Episode Date: May 28, 2023

A reading of Max Tegmark's essay "The 'Don't Look Up' Thinking That Could Doom Us With AI" The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, a reading of a recent essay by Max Tegmark. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Like, subscribe and share and find out more at Breakdown.network. Hello, friends, happy Memorial Day weekend for those of you who are in the United States. Today, we are taking a page out of the main breakdown feed and doing a Long Read Sunday for the AI breakdown. Now, for those of you who don't listen to the main breakdown show, Long Read Sunday, is basically exactly what it sounds like. In lieu of a normal episode, I read a recent essay that I think has some sort of importance. I won't always agree with the essay, but I always think that it has

something to offer as we think about important issues in this space. Now, I'm not sure if this is going to be an every week thing for the AI breakdown or a sometimes thing for the AI breakdown or a not very often thing for the AI breakdown. But given that it's a holiday weekend and news is a little bit slower. I thought this would be a good way to keep those of you who are still paying attention flush with some additional content. So here's the important context for this piece. There is, of course, a growing interest in artificial intelligence. The experience of using chat GPT or mid-jurney or one of these other tools has created an inflection point moment in society where people by and large who have tried these tools understand that there is a before and an after and that we are clearly in the after.

Now, that has produced a situation where there is a lot of racing to catch up. Individuals are racing to catch up to understand how this technology is going to impact their jobs, their careers, their prospects. Industries are racing to catch up to understand if they'll even exist in a few years. And on a broad societal level, the people who have been thinking about the risks of artificial intelligence are racing to give all of these new participants in the space the information they need to make informed decisions about how they think certain issues should be handled. It's inescapable if you pay any attention to the AI news how much of a regulatory discussion is happening. But underneath that regulatory conversation, which tends to focus on things like deep fakes and election manipulation and scams and all very important things, there is an even headier conversation about ex-risk or existential risk.

This is the risk, of course, that unleashing a superintelligence, what some call an alien intelligence, could have incredibly deleterious impacts on society, even existential. potential impacts. I've said it before and I'll say it again, I think far too often for people who are just coming into this discussion, the choices are presented in a pretty stark binary. It's either on the one end of the spectrum, you think for sure that AI is going to eliminate humanity, or on the other hand, that you find all of those arguments absurd and you're effectively an accelerationist. My belief interacting with and creating content around a lot of these new market participants is that the vast majority of people actually fall somewhere in between, or maybe putting it even

more clearly, the vast majority of people don't really have their mind made up at all yet because they're just coming into this conversation. What I try to look for then when it comes to this conversation is people who are having it in a way that is accessible, doesn't require a lot of prior knowledge, and create space and humility for a general not knowingness. So that's the setup to this piece by Max Tegmark that appeared in time last month at the end of April called the don't look up thinking that could doom us with AI. Tagmark is an academic at MIT, whose background is in cosmology, but who is very well known for his AI research as well. Max writes, suppose a large inbound asteroid were discovered, and we learned that half of all astronomers

gave it at least 10% chance of causing human extinction, just as a similar asteroid exterminated the dinosaurs about 66 million years ago. Since we have such a long history of thinking about this threat and what to do about it, from scientific conferences to Hollywood blockbusters, you might expect humanity to shift into high gear with a deflection mission to steer it in a safer direction. Sadly, I now feel that we're living the movie don't look up for another existential threat, unaligned superintelligence. We may soon have to share our planet with more intelligent minds, quote-unquote, that care less about us than we cared about mammoths. A recent survey showed that half of AI researchers

give AI at least a 10% chance of causing human extinction. Since we have such a long history of thinking about this threat and what to do about it, from scientific conferences to Hollywood blockbusters, you might expect that humanity would shift into high gear with a mission to steer AI in a safer direction than out-of-control superintelligence. Think again. Instead, the most influential responses have been a combination of denial, mockery, and resignation so darkly comical that it's deserving of an Oscar. When Don't Lookout came out in late 2021, it became popular on Netflix, their second most watch movie ever. It became even more popular among my science colleagues, many of whom hailed it as their favorite film ever, offering cathartic comic relief for years of pent-up exasperation

over their scientific concerns and policy suggestions being ignored. depicts how, although scientists have a workable plan for deflecting the aforementioned asteroid before it destroys humanity, their plan fails to compete with celebrity gossip for media attention, and is no match for lobbyists, political expediency, and asteroid denial. Although the film was intended as a satire of humanity's lackadaisical response to climate change, it's unfortunately an even better parody of humanity's reaction to the rise of AI. Below is my annotated summary of the most popular response to the rise of AI. There is no asteroid.

Many companies are working to build AGI, artificial general intelligence, defined as AI that can learn and perform most intellectual tasks that human beings can, including AI development. Below will discuss why this may rapidly lead to superintelligence, defined as general intelligence far beyond human level. I'm often told that AGI and superintelligence won't happen because it's impossible. Human level intelligence is something mysterious that can only exist in brains. Such carbon chauvinism ignores a core insight from the AI revolution, that intelligence is all about infrared, information processing, and it doesn't matter whether the information is processed by carbon atoms and brains or by silicon atoms in computers. AI has been relentlessly overtaking humans on task after task, and I invite carbon chauvinists to stop moving the goalposts and publicly predict

which tasks AI will never be able to do. It won't hit us for a long time. In 2016, Andrew Ng famously quipped that worrying about AI today is like worrying about overpopulation about Mars. Until fairly recently, about half of all researchers expected AGI to be at least decades away. AI godfather Jeff Hinton told CBS that, quote, until quite recently, I thought it was going to be 20 to 50 years before we have general purpose AI, and now I think it may be 20 years or less, with even five being a possibility. He's not alone. A recent Microsoft paper argues that GPT4 already shows, quote, sparks of AGI, and Hinton's

fellow deep learning pioneer, Joshua Benjillo, argues that GPT4 basically passes the Turing test that was once viewed as a test for AGI. And the time from AGI to superintelligence may not be very long. According to a reputable prediction market, it will probably take less than a year. Superintelligence isn't a long-term issue. It's even more short-term than EG climate change and most people's retirement planning. Mentioning the asteroid distracts for more pressing problems. Before superintelligence and its human extinction threat, AI can have many other side effects worthy of concern,

ranging from bias and discrimination to privacy loss, mass surveillance, job displacement, growing inequality, cyber attacks, lethal autonomous weapon proliferation, humans getting hacked, human enfeebleman and loss of meaning, non-transparency, mental health problems from harassment, social media addiction, social isolation, dehumanization of social interactions, and threats to democracy from polarization, misinformation, and power concentration. I support more focus on all of them. But saying that we therefore shouldn't talk about the existential threat from superintelligence because it distracts from these challenges is like saying we shouldn't talk about a literal

inbound asteroid because it distracts from climate change. If unaligned superintelligence causes human extinction in coming decades, all other risks will stop mattering. The asteroid will stop before hitting us. Most people who take AGI seriously appear to be so scared and or excited about it that they talk only about those other risks, not about the elephant in the room, superintelligence. Most media, politicians, and AI researchers hardly mention it at all, as if tech development will somehow stagnate at the AGI level for a long time. It's as if they've forgotten Irving J. Good's simple counter argument because it was made so long ago.

Quote, let an ultra-intelligent machine, what we now call AGI, be defined as a machine that can far surpass all the intellectual activities of any man, however, clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines. There would then unquestionably be an intelligence explosion, and the intelligence of man would be left far behind. The basic idea of recursive self-improvement is, of course, nothing new. The use of today's technology to build next year's technology

explains many examples of exponential tech growth, including Moore's Law. The novelty is that progress towards AGI allows even fewer humans in the loop, culminating in none. This may dramatically shorten the timescale for repeated doubling, from typical human R&D timescales of years to machine timescales of weeks or hours. The ultimate limit on such exponential growth is set not by human ingenuity, but by the laws of physics. The limit how much computing a clump of matter can do

is about a quadrillion quintillion times more than today's state of the art. The asteroid will almost stop. Remarkably, superintelligence denial is prevalent not only among non-technical folks, but also among experts working on AI and AI safety. A cynic might put this down to Upton Sinclair's analysis that, it is difficult to get a man to understand something

when his salary depends upon his not understanding it. Although it's unfortunately true that most AI researchers, including safety and ethics researchers, get funding from big tech, either directly or indirectly via grants from nonprofits funded by tech philanthropists, I believe that there are also more innocent explanations for their superintelligence denial, such as well-studied cognitive biases. It's hard for us to fear what we've never experienced, e.g. radical climate change from fossil fuels or nuclear winter. Availability bias makes it hard to see past the immediate threat to the greater

one that follows. For example, I often hear the argument that large language models are unlikely to recursively self-improve rapidly. But IJ. Good's above-mentioned intelligence explosion argument didn't assume that AI's architecture stayed the same as it self-improved. When humans attained general intelligence, we didn't achieve our subsequent exponential growth in information processing capacity by growing bigger brains, but by inventing printing, universities, computers, and tech companies. Similarly, although neural networks and LLMs are now the rage, it's naive to assume that the fastest path from AGI to superintelligence involve simply training even larger LLMs with even more data.

There are obviously much smarter AI architectures since Einstein's brain outperform GBT4 on physics by training on much less data using only 12 watts of power. Once AGI is tasked with discovering these better architectures, AI progress will be made much faster than now, with no human needed in the loop. And IJ Good's intelligence explosion has begun. And some people will task it with that if they can, just as people have already tasked GBT4 with making self-improving AI for various purposes, including destroying humanity. We'll be fine, even if the asteroid hits us. If superintelligence drives humanity extinct, it probably won't be because it turned evil or conscious, but because it turned competent,

with goals misaligned with ours. We humans drove the West African black rhino extinct not because we were rhino haters, but because we were smarter than them and had different goals for how to use their habitats and horns. In the same way, superintelligence with almost any open-ended goal would want to preserve itself and amass resources to accomplish that goal better.

Perhaps it removes the oxygen from the atmosphere to reduce metallic corrosion. Much more likely, we get extinct as a banal side effect that we can't predict any more than those rhinos, or the other 83% of wild mammals we've killed off so far, could predict what would befall them. Some will be fine arguments are downright comical. If you're chased by an AI-powered heat-seeking missile, would you be reassured by someone telling you that AI can't be conscious and AI can't have goals?

If you're an orangutan in a rainforest being clear-cut, would you be reassured by someone telling you that more intelligent life forms are automatically more kind and compassionate, or that they are just a tool you can control. Should we really consider it technological progress if we lose control over our human destiny, like factory-farmed cows and that destitute orangutang? I'm part of a growing AI safety research community that's working hard to figure out how to make superintelligence

aligned even before it exists, so that its goals will be aligned with human flourishing, or we can somehow control it. So far, we've failed to develop a trustworthy plan, and the power of AI is growing faster than regulations, strategies, and know-how for aligning it. We need more time. We've already taken all necessary precautions.

If you'd summarize the conventional past wisdom on how to avoid an intelligence explosion in a don't-do list for powerful AI, it might start like this. Don't teach it to code. This facilitates recursive self-improvement. Don't connect it to the internet. Let it learn only the minimum needed to help us, not how to manipulate us or gain power. Don't give it a public API. Prevent nefarious actors from using it within their code.

Don't start an arms race. This incentivizes everyone to prioritize. prioritize development speed over safety. Industry has collectively proven itself incapable to self-regulate by violating all of these rules. I truly believe that AGI company leaders have the best intentions, and many should be commended for expressing concern publicly. OpenAI's Sam Altman recently described the worst-case scenario as lights out for all of us, and Deep Mines Demis Hasabas said, I would advocate not moving fast and breaking things. However, the aforementioned race is making it hard for them to resist commercial and geopolitical pressure to continue full

steam ahead, and neither has agreed to the recent proposed six-month pause on training larger than GPT4 models. No player can pause alone. Don't deflect the asteroid because it's valuable. Yes, this too happens and don't look up. Even though half of all AI researchers give it at least a 10% chance of causing human extinction, many oppose efforts to prevent the arrival of superintelligence by arguing that it can bring great value if it doesn't destroy us. Even before superintelligence, AGI can, of course, bring enormous wealth and power to select individuals, companies, and governments. It's true that superintelligence can have huge upside if it's aligned. Everything I love about civilization is the product of human intelligence.

So superintelligence might solve disease, poverty, and sustainability, and help humanity flourish like never before. Not only for the next election cycle, but for billions of years, and not merely on Earth, but throughout much of our beautiful cosmos. I.J. Good put it more succinctly. Thus, the first ultra-intelligent machine is the last, invention that man ever need make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes

worthwhile to take science fiction seriously. Let's make the asteroid hit the US first. The purpose of the proposed pause is to allow safety standards and plans to be put in place, so that humanity can win the race between the growing power of the technology and the wisdom with which we manage it. The pause objection I hear most loudly is, but China, as if a six-month pause would flip the outcome of the geopolitical race. As if losing control to Chinese minds were scarier than losing control to alien digital minds that don't care about humans. As if the race to superintelligence were an arms race that would be won by us or them, when it's probably a suicide race whose only winner is it. Don't talk about the asteroid. A key reason we hear so little about superintelligence risk,

as opposed to jobs, bias, etc., is a reluctance to talk about it. It's logical for tech companies to fear regulation and for AI researchers to fear funding cuts. example, a star-studded roster of present and past presidents of the largest AI society recently published a statement endorsing work on a long list of AI risks, where superintelligence was conspicuously absent. With rare exceptions, mainstream media also shies away from the elephant in the room. This is unfortunate because the first step towards deflecting the asteroid is starting a broad conversation about how to best go about it. We deserve getting hit by an asteroid. Although everyone is entitled to their own misanthropic views, this doesn't entitle them to doom everyone

else. Asteroids are the natural next stage of cosmic life. Although sci-fi is replete with conscious human-like AI that shares human values, it's clear by now that the space of possible alien minds is vastly larger than that. So if we stumble into an intelligence explosion rather than steer carefully, it's likely that the resulting superintelligence will not only replace us, but also lack anything resembling human consciousness, compassion or morality. Something will view less as our worthy descendants than as an unstoppable plague.

It's inevitable, so let's not try to avoid it. There's no better guarantee of failure than not even trying. Although humanity is racing towards a cliff, we're not there yet, and there's still time for us to slow down, change course, and avoid falling off. And instead enjoying the amazing benefits that safe, aligned AI has to offer. This requires agreeing that the cliff actually exists, and falling off of it benefits nobody. Just look up. All right, back to NLW here.

I think this piece is a much better starting point than a lot of what's out there for people who are trying to understand AGI risks or superintelligence risks or just generally this set of risks that people are thinking about day and day out. My goal in reading this is not to endorse every piece of it, or at least necessarily Max's conclusions. My goal is to give the 100 million plus people who've come to start thinking about this world in the last six months after using chat GPT for the first time, the best tools possible to actually start engaging in a meaningful way with a conversation. It is to look up, as Max puts it, even if you disagree with what he's seeing when you do. The other thing I'll note, and I think this is encouraging, is that there really is right now

a massive, massive shift into the regulatory conversation. CEOs are getting hosted at the White House. British Prime Minister Rishi Sunnick is hosting companies in the UK, and the European Parliament is steaming forth with the AI Act. No, I don't think that most people are going to love the first set of solutions that politicians come up with. It's almost guaranteed that they'll end up looking like solutions to last generation's problems. That's just the nature of politics and the nature of change. Indeed, if you watch the AI hearing that happened a couple weeks ago in the Senate, it was so, so influenced by what they had learned from social media, even though this is something that's so fundamentally different than those challenges. But the conversation is happening.

People are not just open to it. They understand how important it is. I hope that the people who are most engaged and who have spent years screaming into the void recognize that the opportunity now is to shift strategy and actually begin to offer practical tangible starting points in the sort of bullet point form that politicians or companies might be enabled to enact. The six-month pause idea wasn't a bad start, but it didn't work. So what do you got next? Anyways, guys, that is it for this long read Sunday on the AI breakdown. If you're enjoying the AI breakdown, please like, subscribe and share. If you're listening to it as a podcast, consider leaving a five-star rating or a review. check out the newsletter which you can find at the AIbreakdown.bigh.com.

And for all of this, you can find the information you need at breakdown.network. Until next time, peace.