The AI Daily Brief: Artificial Intelligence News and Analysis - AI Researchers Put Risk of Human Extinction from AI at 5%

Episode Date: January 4, 2024

A new survey asked 2,700+ AI/ML researchers about their fears concerning AI. This is how they responded. Today's Sponsors: Listen to the chart-topping podcast 'web3 with a16z crypto' wherever you get your podcasts or here: https://link.chtbl.com/xz5kFVEK?sid=AIBreakdown  ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, we're looking at a new extensive survey of AI researchers on whether they think we should be concerned about AI's threat to humanity. Before that on the brief, with an updated keyboard for the first time in 30 years, is Microsoft kicking off the AIPC era? The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown. Network for more information about our Discord, our YouTube, and our newsletter. Welcome back to the AI breakdown brief. all the AI headline news you need in around five minutes. Today we start with a narrative that whether true or not is definitely going to be marketed and sold to you, and that is the idea that we are heading to a new era of the AI PC.

We've seen this sort of rhetoric from a number of different companies. Intel, when they introduced new chips at the end of last year, talked about how we were heading towards this AIPC era. And now just today, we've gotten news that Microsoft is introducing a co-pilot key on its keyboards. The copilot key will replace the right control key, and as many are pointing out, is the first major change to Windows-based keyboards in around 30 years. Basically, when you're in any program that has a co-pilot AI assistant, you'll be able to press that button to get right to that feature, and in areas that don't have any sort of copilot assistance, the key will just bring up

Windows search. Now, part of the timing of the announcement comes, of course, with the Vegas-based CES starting as the first laptops and keyboards to feature this key are starting to roll out. Now, on the one hand, this is a small thing. It's just a shortcut to a feature that'll exist in a bunch of different programs, right? But on the other hand, having that button front and center is likely going to push some meaningful portion of people to press that and use those features more often than they might otherwise have. To the extent that one sees things like Microsoft's co-pilot as intro or gateway experiences for people exploring an AI-powered computing era, I think this could be a meaningful update.

Now, beyond that, when it comes to this question of the AI PC year in general, I certainly think that one of the major themes that you are going to see is generative AI models sizing down to be able to run on different types of hardware, and more specifically more personal types of hardware. It's quite clear that Apple is absolutely racing to that, and many of the open source competitors, such as Mistral, are clearly innovating in the area of smaller models, which might be able to run locally on device. The types of things that that will enable could be significant. And so what we have here, I think, is one of those marketing memes that actually has some

truth in reality. Moving on today to fundraising news, the big story is that perplexity has confirmed their latest funding round. The information reported that this was in the works last year, but the company has now confirmed it. They've raised around 74 million new buckos from a variety of investors, which include notably Jeff Bezos. The investment values the company at $520 million post money.

Now, for those of you who haven't used perplexity before, it's basically an alternative approach to search. You can ask it something like, what is the breakdown network? And it answers not only with an AI-generated summary, but an AI-generated summary that actually cites its sources. It also gives you a set of related topics that can lead you down different research rabbit holes. The company reports that perplexity is being used by around 10 million people each month right now, which is obviously a far cry from the number of people who use the dominant search engines like Google, but on an anecdotal and personal level, I know many people in the AI space who have largely shifted their behavior from Google to perplexity and think it has a real shot at disrupting

the search experience of the future. Right now, the company makes $5 to $10 million in annual revenue from subscriptions, but is not yet profitable. Moving over to the regulatory sphere for a moment, OpenAI is seeking to shore up regulatory risk in the European Union by consolidating its European operations under its Ireland subsidiary OpenAI, Ireland Limited. In an email sent to users on December 28th. The company wrote, We have changed the OpenAI entity that provides services such as ChatGAPT to EEA and Swiss residents to our Irish entity, OpenAI, Ireland, Limited. Now, on top of that, their new privacy policy for Europe was also updated. Basically, OpenAI is trying to get out

ahead of GDPR privacy compliance issues, and to me the story is less that specific set of actions, and more the fact that I think that in 2024, you're going to see a lot more of this type of scrambling to get into compliance rather than fighting the global legal tide. Now, speaking of legal battles, Mid Journey is, of course, one of the image generators that is locked in a number of different lawsuits around its approach to training its models. Recently, a list of artists that were allegedly used to train Mid Journey has been leaked to much chagrin. The so-called Mid Journey Stylist spreadsheet has more than 16,000 artist names, as well as listing specific time periods, artistic movements, medium genres, and video game softwares that people can use as part of their prompts. Court documents claim that the list was first published to Discord in February 2022 by Mid Journey CEO David Holt,

who added the message, I think you're all going to get mind-blown by the style feature. I should be clear it's not just genres. It's also artist names. Of course, artists who are against unlicensed AI training on their works are none too happy about this list. In the U.S., as election season ramps up, progress on any sort of comprehensive AI legislation is very slow, but that doesn't mean that we're not seeing bills. They're just tending to happen at the state level, as every jurisdiction tries to figure out what AI is going to mean for its citizens. In California, a senator has filed a bill prohibiting state agencies from working with unethical AI companies. Basically, the bill requires California's Department of Technology to develop a set of new standards around

to develop a new set of standards around services that use AI. In addition, it prohibits the state of California from contracting with any services unless the provider of those services meets those established standards. Another bill introduced simultaneously would establish a California AI research hub designed to provide additional compute resources and other data for academics. Finally, in a move that will be sure to generate extreme emotions in one direction or another, a company in London is set to create a holographic AI version of Elvis Presley for a new stage show called Elvis Evolution. The company layered reality writes,

A life-sized digital Elvis will share his most iconic songs and moves for the very first time on a UK stage. Elvis fans can look forward to a memory-making experience like no other. Through AI and groundbreaking tech, you'll be able to witness iconic Elvis Presley performances as if you were really there and celebrate defining moments in Elvis Presley. Presley's extraordinary life and career. Like it or not, friends, this is the new reality. However, that is going to do it for today's AI breakdown brief. Next up, the main AI breakdown. Quickly a brief word from today's sponsor. As a listener of this show, I suspect you like to stay

up to date on all things AI and tech, which is why you have to check out the chart-topping podcast Web3 with A16Z crypto. Produced by venture firm Andresen Horowitz, Web3 with A16Z is the perfect companion podcast to the AI breakdown. Web3, with A16Z Crypto is your definitive resource for the future of the internet. Whether you're interested in the convergence of AI and crypto or simply curious about what's next. If you need a place to start, they recently released an excellent episode with Stanford Cryptography Professor Dan Bonae and former Google X engineer Aliyaa in conversation with host Sonal Choxi about the intersection of AI and crypto. From fighting deepfakes and proving humanity to large language models

like ChatGBT, they cover it all. I highly recommend checking it out, especially if you'd like to learn more about how AI and crypto will impact our everyday lives. Beyond crypto and AI, the show is for creators seeking more ways to truly own their work, for business leaders trying to prepare for the future today, and for innovators exploring trending tech topics. Don't miss out. Follow Web3 with A16Z crypto on Apple Podcasts, Spotify, or your favorite listening app. AI safety discourse starts this year at a very different place than it started last year. Last year at this time, people were barely coming to realize that there was this new thing called ChatGBT, GBT, and it was part of a broader set of technologies that people were calling generative AI, and that for the first time,

something that they had seen in popular science fiction was coming to reality in a major way. Now, of course, there were many people who had been beating the drum of concern around advanced artificial intelligence for years, and as the world got acquainted with Chat ChbT and MidGermy and other tools like that, that group got consequently louder, trying to let people know that they believe that there was a great concern that people should be paying more attention to. That discourse began, of course, the six-month pause letter, and then Jeffrey Hinton's famous defection from Google and the media tour that followed, and by the middle of the year, questions of X-risk or extinction risk were everywhere in mainstream media outlets.

Surveys throughout the second half of last year have suggested that people are extremely concerned about the negative potential implications of artificial intelligence, and indeed that concern in and of itself in the media surrounding it has inspired a louder counter-react. in the form of the EACCs or the effective accelerationists. We head into 2024 then in the midst of full-fledged narrative warfare between the folks on the one hand who think AGI will unleash unmeasurable prosperity and opportunity, and the group who, on the other hand, thinks that it could lead to the very end of humanity.

Now, that is the context into which a new research paper surveying more than 2,700 AI and ML researchers comes into play. The paper was called thousands of AI authors on the future of AI, and was led by a team at AI Impacts from Berkeley, California. Now, we're going to dig into many of the findings from this survey, but the first thing that I want to note is how much the big banner headline statistic is serving as a Rorschach test for how people think about AI X risk in general. So, for example, the new scientist writes,

there's a 5% chance of AI causing humans to go extinct, say scientists. In the largest survey yet of AI researchers, a majority say there is a non-trivial risk of human extinction due to the possible development of superhuman AI. Basically, the statistic that they're referencing is that around 58% of the researchers that they surveyed said that there was a 5% chance of human extinction or another extremely bad AI-related outcome. Now, it's clear that the survey's authors think that this means we should take X-risk more

seriously, said Katia Grace, who's one of the authors of the paper, it's an important signal that most AI researchers don't find it strongly implausible that advanced AI destroys humanity. I think this general belief in a non-miniscule risk is much more telling than the exact percentage risk. In other words, what the authors are saying is that a group of researchers saying that there's a 5% chance that their field causes human extinction is a really worrying note. But then you have Business Insider whose headline reads, There's only a 5% chance of AI making humans extinct a study featuring 2,700 AI researchers has found. Now, of course, the implication of that only is that it's something that we shouldn't be concerned about, that this number feels lower than one might have expected.

The lead of the piece seems to reinforce that perspective. Quote, over the last year, we've heard a lot about the risk of AI destroying humanity. Industry leaders in AI heavyweight said the rapid development of the technology could have catastrophic consequences for the world. But while most AI researchers recognize the possibility of existential threats, they don't think that dramatic outcomes are very likely. Now, I think that if you go to the actual paper itself, what it shows is a meaningful amount of concern, but also a lot of ambivalence in not knowing.

gets the sense that people are trying to rapidly update their mental models because things are changing so quickly. For example, one of the things that they asked was, if science continues undisrupted, what are the chances that unaided machines outperform humans in every possible task? The chance of that was estimated to be 10% by 2027 and 50% by 2047. That 2047 estimate was 13 years earlier than the estimate that was reached in a similar survey that was conducted just a year earlier. In other words, a single year of progress in AI. has had this group of AI researchers pull forward their estimate of when machines could outperform humans in every possible task by 13 years. The chance of all human occupations becoming fully

automatable was forecast to reach 50% by 2116, which was up from 2164 in the 2022 survey, a 38-year increase. Now again, to my mind, what matters is not the specific year, but more of the fact that the evidence in front of their eyes is forcing people to grapple with AI moving much more quickly than they had previously imagined. Now, in terms of the ambivalence, the way that the paper frames it is, most respondents expressed substantial uncertainty about the long-term value of AI progress, while 68.3% thought good outcomes from superhuman AI are more likely than bad, of these net optimists, 48% gave at least a 5% chance of extremely bad outcomes, such as human extinction, and 59% of net pessimists gave 5% or more to extremely good outcomes. Between 37.8% and 51.4%

of respondents gave at least a 10% chance of advanced AI leading to outcomes as bad as human extinction. More than half suggested that substantial or extreme concern is warranted about six different AI-related scenarios, including spread of false information, authoritarian population control, and worsened inequality. There was disagreement about whether faster or slower AI progress would be better for the future of humanity. However, and this is the upshot of the whole paper, there was broad agreement that research aimed at minimizing potential risks from AI systems ought to be prioritized more. Now, the AI safety memes account on Twitter pulled out a number of different statistics that suggests that the business insider glib interpretation may not be exactly what the

paper really conveys. For example, the paper found over 95% of the researchers were concerned about dangerous groups using AI to make powerful tools like engineered viruses. Over 95% are concerned about AI being used to manipulate large-scale public opinion. Over 95% are concerned about AI making it easier to spread false information. Over 90% are concerned about authoritarian rulers using AI to control their populations. Over 90% are concerned about AI's worsening economic inequality. Over 90% are concerned about bias, such as AI's discriminating by gender or race. Over 80% are concerned about a powerful AI having its goals not set right, causing a catastrophe. For example, developing and using powerful weapons. Over 80% are concerned about people interacting with

other humans less because they're spending more time with AI's. Over 80% are concerned about near full automation of labor, leaving most people economically powerless. Relatedly, over 70% are concerned about near full automation of labor making people struggle to find meaning in their lives, and only 20% say it's likely that we'll be able to understand what AIs are really thinking by 28. Now again, what I argued is that the upshot of the research is that these researchers by and large want to focus on AI safety research in a bigger way. Statistics that validate that. 86% say the AI alignment problem is important versus only 14% that say it's unimportant, which is a 7 to 1 ratio, and 70% want to prioritize AI safety research more versus just 7%

who want to prioritize it less. In other words, by 10 to 1, these researchers think that AI safety research should be prioritized more. Now, this account also pointed out that although that headline reported statistic of 58% of researchers finding a 5% chance of extinction, that the mean across all the researchers for their P-Doom on the question, what probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species, was actually 16.2%, which, as they point out, is basically the same odds as Russian So where should this study leave us? Well, I think what I've thought for basically the entirety of last year as well, which is that this is a classic type of modern discussion that is dominated

by extreme voices on either side. That domination by extreme voices is aided and abetted by both the media infrastructure as well as social media algorithms, which promote more extreme views versus more nuanced and moderate takes. But I think that to the extent that one is looking for optimism, there is clearly a lot of common sense space in the middle. I think by and large that people feel that technology should work for us. It should be on behalf of us. And to the extent that there are risks that technology is not in the service of humanity, that's something worth really considering. Now, where it gets tricky, of course, is that the market incentives to race ever faster forward are not slowing down anytime soon. And so the question becomes, if markets aren't creating enough of an

incentive to prioritize safety research or just appropriate speed of development, what are the countervailing forces? There are a lot of people who are uncomfortable in general with state intervention who feel like the state is the only solution in this case, and that's going to be a great big discussion throughout 2024 and beyond. But no matter what, even if you disagree with these results, even if people come up with problems of the methodology, in general, I am for having more information for more different types of people, and especially those with experience in the field.

So I welcome these numbers as one more piece of evidence in a very important discussion. However, that is going to do it for today's AI breakdown. Let me know what you think about all this. Hit me in the comments. Come join us on the AI Breakdown Discord, bit.ly slash AI breakdown. And until next time, peace.