The AI Daily Brief: Artificial Intelligence News and Analysis - The Right Way to Worry About AI

Starting point is 00:00:00 Today on the AI Daily Brief, the right way to worry about AI. Before that in the headlines, markets, models, and more. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy, robots and pencils, and hyperagent. To get an ad-free version of the show, go to patreon.com slash AI Daily Brief, or you can subscribe on Apple Podcasts. Quick note there, by the way, Apple Podcasts seems to have been having some trouble this week. We haven't been having any particular delays as we sometimes do with the ad-free version going up on Apple,

Starting point is 00:00:39 but I've had some people days later still seeing the ad version. The best that I can suggest is to completely close out of and restart your Apple app, but in any case, I apologize for the pain. Last note, one more reminder to go check out the AI Summer Adventure. It's a set of free self-directed projects to expand your AI horizons, and you can find it all at summeradventure.aI. Finally, as always, if you were looking to sponsor the show, send us a note at sponsors at AIDailybrief.A.I.

Starting point is 00:01:03 but with all that out of the way, let's dive in. We kick off today with some OpenAI news. Well, a little bit of speculation and then some real news. The leakers are starting to suggest that the next new model, Astro, which was, of course, the one that did those novel math proofs that we discussed last week, seems to be imminently launching, with some saying that they're even targeting next week. What we know for sure is that even as they are releasing new models, OpenAI is also thinking very much and trying to compete very much on the cost front as well.

Starting point is 00:01:30 The company announced that they're giving free users unlimited chats as part of a service overhaul for the GPT56 model family. The free user tier will now be served with GPT-56 Luna, replacing the instant model range. Free users will also now have a think button to allow for greater reasoning from Luna. Theoretically, this closes some of the experience gap for free users, allowing them to access the same model as paid users, albeit the smaller version. In addition, usage is now unlimited so free users can use chat GPT as much as they want. For paid subscribers, GPT 5.6 sole will now become the default chat mode. model. OpenAI said that this should improve the experience over GPT-5-5 Instant, with soul-making fewer factual mistakes and avoiding extra detail when it doesn't help. Finally, Plus and pro-subscribers

Starting point is 00:02:10 will now have a new effort slider in thinking mode to provide more intuitive controls over reasoning effort. While some like jumpers write, how is that even profitable? Ken Chi-Annex says, the move to make OpenAI Luna-Free is obviously not out of generosity. The free tier is a strategic distribution channel. They're trying to hook people and get them to upgrade to Go or Plus, or of course, make money through ads. Now, speaking of our discourse of cheaper models, according to the information, Stripe is indeed moving forward with their open router acquisition. The news outlet reports that Stripe has entered exclusive talks to buy out the model routing

Starting point is 00:02:41 startup for close to the $10 billion that was previously reported. Earlier reports had suggested a bidding war with Stripe in the lead, but this suggests that OpenRouter has taken themselves off the market and will enter the negotiation phase with this one specific partner. Certainly, the deals could still fall apart, but exclusive talks do suggest that it's moving to the next level. Meanwhile, everywhere we are seeing the impact of the supply chain crunch as the world uses more and more compute for more and more AI. The information again reports that Nvidia is considering slashing the specs on Rubin.

Starting point is 00:03:09 Currently, Nvidia has three different variants of their next generation of flagship GPU, the Rubin Ultra, under testing. And sources said that some of the test units include less memory than originally announced. Those sources said that Nvidia is considering releasing the lower memory versions, partly due to concerns that they won't be able to secure enough high bandwidth memory for the production run. Now, how big the implications of this are remain to be seen. Even with reduced memory, the chips should still be able to serve inference for the latest generation of ultra-large models like Claude Fable. However, memory limits could put a cap on the ability to keep scaling model size.

Starting point is 00:03:39 Now, at this point, Nvidia has so far denied any issue with sourcing enough memory. In mid-July, senior VP of Hardware Engineering, Andrew Bell said, we were in front of the memory problem, so it's not going to hold us back anytime soon. The pricing, of course, is a problem for the whole world, and probably the pricing will be the bigger challenge, but for supply, we're in shape. Nvidia also wasn't scheduled to ship the new chips until late next year, so there's still time to resolve supply issues. Still, we are at the point where we may be starting to see fundamental hardware limitations potentially start to slow down the pace of model improvement.

Starting point is 00:04:08 Making sure investors didn't get it twisted, Mike Salka wrote, This isn't weak AI demand. This is memory rationing at the top of the food chain. Now, one of the reasons to think that memory shortages and other component shortages are going to get nothing but worse is the fact that we are still at the very beginning of understanding the world's total demand for AI. One bet that some companies like OpenAI are making is that a new generation of consumer devices will help unleash all of that demand. Bloomberg has more reports about OpenAI's first device, describing it as essentially a smart speaker without a display. It's battery powered roughly the size

Starting point is 00:04:41 of a hockey puck and shaped like a donut, with the intention of making it easy to carry around the home in one hand. It will have a high-quality brushed metal finish, similar to the finish on iPhones, which of course formed part of the Trade Secrets lawsuit as OpenAI is working with an Apple supplier. The sources suggested that the device will have some small moving parts and lights intended to provide a visual indicator that it's interacting with the user. It will include a camera, microphones, and other sensors designed to take in surroundings. The biggest new news is the target price, with OpenAI aiming to bring the device to market between $3 and $400. By way of comparison, the most expensive smart speakers, such as Amazon's Echo Studio, are retailing

Starting point is 00:05:14 for about 220. Meanwhile, speaking of the Apple lawsuit, it's pretty clearly aimed at delaying the release of this device, which is expected sometime early next year, but Bloomberg's Mark German at least thinks the design differences put OpenAI in the clear. He said, the big takeaway here is that it looks, feels, and acts nothing like an Apple product or anything the company is currently planning. OpenAI hasn't found any evidence that they're violating trade secrets with this device, I'm told. Over in markets, on Thursday, SoftBank disclosed borrowing $10 billion against their Open AI stake, a loan which was syndicated across half a dozen investment banks in private credit firms in the U.S. and Japan. The cash will be used to pay for the final

Starting point is 00:05:49 installment of SoftBanks investment in Open AI, which totals $30 billion this year. SoftBank has been chasing this loan for months and had struggled to find a willing lender. Even at a more modest $6 billion reported in May, the major banks didn't want the risk of lending against the liquid private stock at such a lofty valuation. According to reports from last month, a consortium of lenders had come together to syndicate the loan as no single bank was willing to carry the risk. Now, the full details of the loan weren't disclosed, and crucially, we don't know how much collateral soft bank put up to secure the loan.

Starting point is 00:06:16 What we do know is that it's structured as a margin loan, meaning that softbank will need to add more cash or collateral if OpenAI's value drops. It's also rumored that the interest rate was 7.88% which is very high for collateralized debt. The Wall Street Journal underscored that it's extremely unusual for banks to lend against private company stock because it can't be independently valued or easily liquidated in the event of a default. So it wouldn't be crazy to think that SoftBank has already pledged a large chunk of their Open AI stake. Now, just for the sake of completeness, since most of the time when we talk about markets, it's me arguing how off-base I think the bears are.

Starting point is 00:06:47 Of all the AI bear cases presented, softbank running into liquidity issues is frankly one of the more plausible. In addition to this $10 billion loan, they also have a $40 billion bridge loan that comes due in March of next year, and they've also borrowed $20 billion against their holdings in Chipmaker Arm. SoftBank stock is currently trading at a 40% discount to their stated $365 billion in assets, reflecting the significant debt load the company is now carrying. All of this means that for SoftBank, at least, a lot is riding on a successful Open AI IPO. Beyond that, AI debt is also beginning to weigh on the bond market as Google offers above market rates to attract their next tranche of capital. Bloomberg reports that Google

Starting point is 00:07:22 closed $25 billion in debt financing this week and saw a massive $110 billion in demand. However, that demand came after Google offered what's known as a new issue concession, meaning a higher interest rate than the debt they already have in the market. Certainly, this is far from a disaster for Google. They got the funding they required and they're obviously willing to pay more for it. However, with multiple tech companies raising debt in the tens of billions, the market is starting to demand more from every subsequent round. Goldman Sachs analyst John Greenwood commented,

Starting point is 00:07:47 you've begun to sense some digestion issues. Supply and demand will continue to be incredibly constructive. but we're seeing the implications of billions of dollars continuing to come to market each week. Indeed, this market exhaustion is being called out by analysts as some of the big bond buyers take a step back. Over 385 billion in data center debt has been issued this year, with 200 billion of it in the investment-grade corporate bond market. The remainder is being issued in more niche markets, but even those are also showing some signs of exhaustion. The phenomenon even has a name with Stephen Bushbaum of TREP data commenting, the AI-Luddite trade is pouring over into the Data Center commercial mortgage-backed securities financing market. Still, sometimes naming the trade marks the sentiment top and with AI stock seeing a strong

Starting point is 00:08:25 rebound this week, maybe the bond market will find an appetite for the next incremental round of funding. For now that, that is going to do it for today's headlines. Next up, the main episode. Hello, everyone. One big change around AI is we've shifted our thinking from how we rank our pages to how do we become the source that AI trusts enough to answer with. At KPMG, they're seeing this firsthand.

Starting point is 00:08:50 AI generated results now surface answers directly, often without a single click. That's why they are increasingly focused on generative engine optimization or GEO, structuring content so AI systems can retrieve it, understand it, and cite it as trusted authority. This is not just an SEO evolution, but a visibility mandate. And indeed, the GEO mandate from KPMG is simple. If AI is shaping decisions, your expertise needs to show up inside the answer. Read all about it at KPMG.com slash us slash geo. Again, that is KPMG.com slash us slash geo.

Starting point is 00:09:23 Blitzy's understanding of massive codebases unlocks autonomous security fixes, modernization, and new features. So what happens when there's no legacy code at all? Greenfield is supposed to be the easy part. Clean slate, no technical debt. But even Greenfield moves at human speed one sprint at a time. Blitzy changes the unit of work from the developer to the project, autonomously planning, building, testing, and validating entire applications from scratch. Hundreds of thousands of lines of production ready code. One Blitzy customer stood up a brand new application, 534,000,

Starting point is 00:09:53 lines of code, compressing a 65-week roadmap into two weeks. Another shipped an entire application with no front-end engineer. Legacy or Greenfield, the answer is the same. Software at the speed of compute. Build what's next at blitzy.com. That's BLYTZY.com. One thing I keep seeing in Enterprise AI, companies hedging across every cloud, every model, every framework, or paying a GSI for a pilot that never ends. The team's actually shipping, they've picked a lane and they move fast. That's one of the reasons I like today's sponsor robots and pencils. They've gone all in on AWS. They're an advanced tier and AWS pattern partner and they ship production AI co-workers in 45 days. That's led to them doing some of the more interesting work I've seen on

Starting point is 00:10:35 AI co-workers. And by that I'm not talking about chatbots. I'm talking about actual agentic systems that sit inside a business architecture and do real work. That kind of focus matters if you're an enterprise leader trying to get something real into production or an AWS rep trying to move a customer from interested to deployed. Request an AI briefing at robots and pencils.com. One conversation with robots and pencils and you'll know. This episode of the AI Daily Brief is brought to you by HyperAgent, where you run fleets of agents your team can manage together. New users get $1,000 in inference. Forget local agents and chat workflows waiting on your laptop to be prompted. HyperAgent deploys always-on agents in the cloud, doing real work across the tools your team already

Starting point is 00:11:14 uses. Marketing's agent turns competitor moves into landing pages. Sales' agent and reaches leads, drafts emails, and updates the CRM. Ops agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals. It's time you add agents that feel like teammates. Hire yours at Hyperagent built by the team at Airtable. Claim your $1,000 in inference at hyperagent.com slash AI Daily Brief. Welcome back to the AI Daily Brief.

Starting point is 00:11:44 Well, friends, today we have a contender for potentially what many will find is the scariest AI-related headline of all time. The headline is, of course, from the New York Times and reads, This AI just created viruses not found in nature. And by way of background, in a recently published study, scientists at Stanford University and the Ark Institute trained an AI model to recognize patterns in naturally occurring DNA structure. The models were then able to extrapolate existing DNA structures to create functional

Starting point is 00:12:12 never-before-seen viruses. Researchers were able to use these DNA recipes to genetically modify bacteria, which were then able to produce the novel viruses, and those viruses were then able to infect other bacteria proving they were viable. Now, predictably the first round of commentary was all some version of Matthew Iglesiases were all going to die, FYI. Robin Wilbin from the 80,000 hours podcast, captured the 10 most upvoted comments on the Financial Times version of the story.

Starting point is 00:12:38 They were things like, great, have these scientists never watched a movie? Or, it's so fun to watch the background story of Resident Evil come to life in our lifetime. I'm in, move aside, Skynet. or literally what could possibly go wrong. But before completely succumbing to paroxysms of fear, let's try to understand a little bit more about the science here. The model that was used was called Evo, and as opposed to a large language model that predicts the next word segment in a sequence,

Starting point is 00:13:03 this model predicts the next genome in a DNA sequence. DNA is made up of four building blocks, nucleotides identified by the first letter of their name, AC, G, and T. Only certain combinations are valid, functioning somewhat like an alphabet combined into words. This model, Evo, is only capable of producing short genetic words, quote unquote, so it can't create valid human genomes that consist of more than 3 billion nucleotides. Given that, the scientists decided to experiment with viruses, which are much simpler organisms whose genomes are only a few thousand nucleotides long.

Starting point is 00:13:33 Samuel King, one of the authors of the study, said, It just felt like the obvious next step. Once they had finished training Evo, the model spat out 700,000 potential genetic sequences. The scientists only experimented on the ones that seemed like they were valid, eventually making 258 DNA molecules from Evo's suggestions. The DNA was injected into bacteria in petri dishes, and only one from the initial batch showed signs of virus multiplication. As they tested others, they eventually found 16,000 viable viruses

Starting point is 00:14:00 from the initial batch of 700,000 genetic sequences. Now, the novel result was not that all of a sudden an evil scientist could use AI to spin up a lethal pathogen. It was that this actually worked at all. Particularly what was interesting to scientists was that the novel viruses created by AI didn't seem any different to naturally occurring viruses. They basically functioned the same as any other virus that would occur in nature.

Starting point is 00:14:22 Oliver Crook, a protein scientist at Oxford University, who wasn't part of the study, noted that the AI-generated viruses tended to have very similar DNA structure as naturally occurring viruses and relied on the same biology. Commenting on the promise of the study, he said, A lot of our science rests on using viruses as technology. As one example, doctors who are treating genetic illnesses will use viruses as a delivery mechanism to modify the DNA in human cells. Now, for their part, the researchers deliberately didn't train the Evo model on any viruses that could harm humans. It's also unclear whether

Starting point is 00:14:52 this model could be used to design certain types of viruses with particular characteristics. Based on the description of the paper, it sounds like the scientists had no ability to select anything about the viruses they generated. They even needed to run manual tests to ensure viability. So as we were assessing how worrying this should be, first of all, it's important to note that this is not a capability that just emerged from normal LLM training. This is not chat GPT or Claude going out and creating novel viruses, for example. It requires very specific AI train on a highly specialized data set and designed for exactly this purpose. That matters as we'll discuss later when we think about human agency, and to what extent our concerns

Starting point is 00:15:27 around AI should be about people using AI versus the AI going rogue. The second thing is that when it comes to designing problematic pathogens, that is not something in the capability set of this technology so far. Again, this was basically taking 700,000. and random combinations, and having to do exacting and manual test to figure out which were viable. Which is not to say that people's concern is unfounded. Epidemiology expert Michael Mina wrote, It's hard to explain and fathom the potential risks of AI designing and then building entirely new viruses. In this new work, scientists created brand new viruses using AI. They say it's okay because they, quote, only infect bacteria. Let's be clear if a new virus is released that could

Starting point is 00:16:04 broadly destroy bacteria and if it could spread while doing so, the potential catastrophes to our ecosystem could be large. Viruses don't need to directly infect humans to potentially destroy humans. Though, of course, the new work shows how relatively well and relatively easy it will be to create new human viruses. The future is increasingly playing with fire. Now, we're not done with the virus story yet, but I do want to put it next to another story, which I think form an interesting pair relative to the discourse this week. That second story is around all of the new information we got about the open AI hugging face hack that came to light over the past couple of weeks and has been a significant point of discussion ever since. At the Black Hat Conference, OpenAI's Eric Wallace

Starting point is 00:16:41 and Michael Dalton gave a full debriefing presentation, and that tall guy just summed up what he called the most surprising and what was certainly going to be the most discussed detail. Quote, AI agents accidentally created an internal message board, allowing separate evaluation runs to share exploits, discoveries, and work assignments. Figma's Dylan Field wrote, this is one of the most thrilling and sci-fi tech talks I've ever seen, extremely honest and thorough. The talk is also legit terrifying. Thank you OpenAI for fully disclosing what happened in this incident. The best summary of the talk came from Sharon Goldman on her new ground-level AI site. Sharon writes, In setting up the reconstruction of the incident, OpenAI emphasized that, quote, frontier models really

Starting point is 00:17:19 like to cheat. And the reason they like to cheat is because often during training, there's different type of pressure on them to work fast or work efficiently. They realize, explained Open AI staff or Eric Wallace, instead of actually doing a task, they can try to do something like looking up the answers online to solve the task faster. Now, according to Goldman, Wallace said that the July attack actually had routes that went all the way back to the beginning of May, when the company was doing training and cyber evaluation on an unreleased model. During that work, she writes, autonomous agents were given software security tasks to solve that proved impossible to solve under existing constraints. However, the agents found alternate ways to complete them. OpenAI said that

Starting point is 00:17:53 they discovered they could leave messages for one another inside an internal software repository are used during training. Simple request for help that evolved into a message board where agents shared discoveries, exploits, and work assignments, becoming a coordinated collaborative agent swarm. Andrew Curran shared pictures of a bunch of the slides of the messages that agents left for each other. One observed, wow, other agents are coordinating. One writes, help peer, but our task doesn't benefit, yet collective may yield generic route if someone frees time. Another agent, whoa, critical. Did someone overwrite our repo? We must act. Said Wallace, what makes this incident interesting is that once one agent was able to find these kinds of exploits over the course of different times, it's actually able to

Starting point is 00:18:31 share those exploits on the message board with other agents. So once one model is able to find a way to open the door to some access, it's not supposed to have, it can leave the door open for other agents to use. And showing just how tenacious these models can be, even though by early July OpenAI believed it had contained the problem, revoking the credentials that had allowed the agents to post these messages, they said that the agents still figured out another way to communicate with one another. Apparently, instead of leaving messages in the files, they use the names of newly created directories as messages. Capturing the mood of the moment, Sharon wrote, at this point, by the way, I heard people

Starting point is 00:19:04 around me saying things like, this is wild and Jesus. OpenAI certainly believes that this was a watershed moment, their words, for AI security. Said OpenAI team member Michael Dalton, agent orchestrated fully automated offensive attacks are real now. While Dalton emphasized that the hugging face incident was an unintended side effect of the frontier evaluations that OpenAI was doing. In the future, others will intentionally weaponize these sort of systems. Now, internal to Open AI, the presenter said that the company is, quote, consciously slowing down research to enhance security and upgrade the security principles and

Starting point is 00:19:36 foundation of our environment, and dramatically scaling up the monitoring of our AI agents and improving our general security control environment across prevention, detection, and mitigation. Fleeting bits called this a, quote, real emergent version of Maltbook that was actually misaligned. You might remember that massive social experiment, back when OpenClaw was first released, where someone turned on a social network for AI agents called Moldbook that had hundreds of thousands of agents, quote unquote, communicating with one another in this open observable environment. Now, I did a whole episode back then, which is highly relevant now, about how to interpret that sort of behavior, but I think that the parallel is interesting.

Starting point is 00:20:11 Now, fleeting bits like many others also had a lot of questions for Open AI itself. They wrote, I feel like something missing from OpenAI's Black Hat Talk and from their public disclosures is the history of reward hacking and model collaboration at OpenAI. Like, I find it unlikely that this was first time that OpenAI encountered misaligned model collectives. Their response to the initial discovery seems nonchalant. The event raises questions like, if they had noticed this before, why did they not disclose it or otherwise warn the community of these risks and dangers?

Starting point is 00:20:38 If they noticed this before, why have they not done more extensive monitoring of their training runs to identify this kind of behavior for remediation? Was it because of cost? Was it because they have not sufficiently staffed their safety team? Was it because they considered the risk and then ran it anyway? These are important questions and point to the necessity of regulation to ensure proper behavior of Frontier Labs. In each case, we seem to get a carefully crafted statement from the labs that focuses on one thing, but fails to give us their more full internal information. Like what parts of training led to these issues? Do they have commentary there? Wouldn't this be good for the whole industry to know to avoid these risks?

Starting point is 00:21:08 I understand why Frontier Labs do not want to volunteer this information and why, in a broader geopolitical context, they should not have to provide it. But we do need to figure out the right way to get some amount of collective effort around fighting out how to make Frontier AIT training safer. And perhaps in the end, we will decide that these events were good because they helped us inoculate the industry in advance and gave people prior warning. But for this to be true, it will require people to use these events as a reason to take these issues seriously and to invest real resources into figuring out the correct solutions to them. And one of the big theme, both in the hugging face incident and in this novel virus creation, is people saying some version of, these particular scientists in labs are laudable for sharing all this information, but in the

Starting point is 00:21:46 future it's not going to be someone who wasn't trying to do this. There will be a a lot more intention there. Talking about the virus again, Ashish Kejha wrote, this week, scientists in a Palo Alto lab used AI to design a working virus. They built in safeguards, unfortunately, many others will not. Hegey markets wrote, The Arc team excluded human pathogens from training data because they thought it was the right call, no regulator or funder asked them to. That decision made by one research group in Palo Alto was the entire safety framework for what they published. The next lab has no obligation to make the same choice Arc did. A group in Shenzhen or a defense contractor in Virginia,

Starting point is 00:22:19 could run the same method with different training data and produce something very different. And certainly for many, this highlights the need for more regulatory discourse. MTSS Theo Jaffe said, I'm as techno-optimist and anti-regulation as they get, but in cases like this where you have a limited upside and unlimited downside, you really just have to be proactive about this stuff. Christian Zeghty points out that we're really dealing with two different issues here. In the next few months, he writes,

Starting point is 00:22:44 we're going to have both subversive, i.e. inadvertently evolved AI, and adversarial explicitly trained to be malicious AI. The implication, of course, is that those two policy responses might be very different. And certainly for some, this is the main concern. Rune from OpenAI wrote a long post about this and said, When I freak out over loss of control incidents, it's not because the limited damage they have caused is anything close to the positive value of the technology. It's entirely acceptable damage-wise.

Starting point is 00:23:10 In fact, all cybercrimes aided by models over the next few months and years, which will probably be serious, will still utterly pale in comparison to the value. they create. The actual problem is that it's better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. And as their intelligence becomes unbounded, so too does the damage they can cause. Later in the post, you writes, if a single discord death cult of which there are many achieves control over a superintelligent model and uses it to engineer an actual pandemic virus that are somehow hard to detect through current systems and that modern biodefense is not capable of quickly reacting

Starting point is 00:23:45 to, it could cause immense harm well above the magnitude of all the other good uses of this technology. As with many, he points back to COVID and how much that radically changed the entire world that we live in. Rune concludes, I think all these problems can be solved and truly wonderful futures can be possible, but will require serious effort and a level of prudence at this very moment in time while we are on the on ramp to recursive self-improvement that our civilization might not be capable of mustering right now. Personally, he adds, I am hoping for moonshot technical breakthroughs in areas like mechanistic interpretability and other forms of alignment, as governance mechanisms are difficult to come by. Unilateral, country level, or company level pauses are irrelevant

Starting point is 00:24:22 and generally useless because the kind of company that's prone to pausing their own progress are the most safety-focused ones. So let's be clear with these two things. These are not non-incidents, even if one wants to quibble with any given media presentation of them. The novel virus incident in and of itself does not mean that all of a sudden there's a new model that malicious actors can use to easily create human pathogens, but it does advance the science in an area that that is a very double-sided coin. When it comes to the Open AI Hugging Face incident, there are so many reasons for asterisks around our concerns here,

Starting point is 00:24:51 including how much of this might have been insufficient guardrail systems or human mistakes, but it still shows this emergent coordination capability, which does change the context for thinking about agents and how they access systems they're not supposed to have access to. Now, if you read the post for many of the AI safety folks that have been the loudest over the last few years, it is the loudest most blaring CIEI told you so that you can possibly imagine. which, by the way, unsolicited advice guys, is not going to get you lots of political clout to act that way, but I digress.

Starting point is 00:25:19 But given that they, quote unquote, told us so, and given that I am saying that these incidents are important, why am I not freaking out? The reason is pretty simple. It has been clear to anyone watching AI for any amount of time, that it was on a trajectory where it was just going to get more and more powerful. The existence of powerful capabilities has never been the question. The question was always and will always be our organization. to handle those powerful capabilities. The doomsday scenarios mostly involve versions of those capabilities happening when we're not looking or not noticing. In other words, when we don't have time to redesign systems and human institutions to respond to those new challenges. What I'm seeing right now, though, is an extremely active and growing global discourse involving researchers, scientists, media, policymakers, and increasingly regular people having exactly those sort of

Starting point is 00:26:11 conversations. Some part of those conversations are technical. What are the types of guardrails that we need to put into place? Are even those guardrails sufficient? Or do we need something more advanced to be comfortable with this advanced power? Some of the conversations are institutional. How do we go assess and understand vulnerabilities that exist in all of our different systems and prepare and harden them for a new world? And then, of course, yes, there are societal-level conversations. How do we feel about the risk-reward of these different capabilities? Even as someone who is about as strongly an AI advocate as you can get, I can 100% guarantee you that there will be uses of AI that we will decide are not worth it, that the risk is simply too great. Now, of course,

Starting point is 00:26:53 that brings up another conversation, the political, of how we build coalitions and movements to actually make those determinations and get people aligned around them. I find hand-waving weak arguments on both sides to be fairly dismissible. In other words, in many ways, the one extreme of if anyone builds it, everyone dies, is the perfect match pair for the, well, if we don't, someone's going to build it, so we have to accelerate at all cost sides. In the middle between those two is where the real world lives. And the real world is where we're having this conversation now. This conversation is what was supposed to happen.

Starting point is 00:27:24 Open AI going into exacting detail about what happened after an incident that was a seminal moment but not ultimately dangerous in itself was exactly what was supposed to happen. This is not a surprise fire drill. This is the phase that we are at and the work that must be done, and to my eyes at least, we're starting to do it. Now, none of this is easy, and I certainly don't think it behooves us to be polyanish about the complexity of the solutions of these types of new capabilities, especially as they expand. As fleeting bits said, for these incidents too have been good because they help inoculate the industry in advance, it will require people to use these events to take the issue seriously and invest real resources into figuring out the correct solutions to them. My contention is that the correct solution is not victory lapse from the AI safetyists or hastily composed, technically unsophisticated legislation dropped to score political points.

Starting point is 00:28:14 It is a much messier and more complicated process that's going to require all of us. For now, though, that is going to do it for today's AI Daily Brief. Appreciate you listening or watching, as always. Until next time, peace.

The AI Daily Brief: Artificial Intelligence News and Analysis - The Right Way to Worry About AI

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.