Tech Brew Ride Home - Why Did Another OpenAI Safety Insider Quit?

Starting point is 00:00:04 Welcome to the TechBoo ride home from Monday, October 5th, 2026. I'm Brian McCullough today. Another OpenAI safety staffer quits saying AI labs need to get it right the first time. Plus, Western openweight models are coming. Meta's muse keeps a page on everyone you know. SpaceX AI might get renamed SpaceX S.I. And Apple tightened full disc access. Here's what you miss today in the world of tech. Every day, shareholders meet to discuss important.

Starting point is 00:00:39 matters about the companies you invest in. Now you can make your voice heard too. Vanguard investor choice makes it easy to set your proxy voting preference for your eligible Vanguard index funds, whether you hold a Vanguard fund directly or through another brokerage firm. All it takes is a few clicks to select your proxy voting preference and be heard on important shareholder topics like Executive Pay and Director elections. Visit vanguard.com slash investor choice to learn more. It's your shares. It's your voice. It's easy. Vanguard investors own shares of our index funds and those funds own shares of the companies they invest in, Vanguard Marketing Corporation distributor.

Starting point is 00:01:17 David Robinson, who worked on OpenAI safety systems team and had previously led policy planning left Open AI last week. And this past weekend, he published a major essay saying Silicon Valley lacks a safety-centric culture. Labs, he says, must study other fields safety approaches. For example, how the airline industry has gotten significant. more safe over time. Oh, and he says the time for trial and error in AI is over. Quoting Bloomberg. An open AI employee focused on AI safety has quit the company and warned that top artificial intelligence firms aren't doing enough to mitigate the technology's risks, joining a growing chorus of employees raising alarms from within the industry. David Robinson, who previously

Starting point is 00:02:03 led transparency work on Open AI's safety team, wrote in an essay for the Atlantic that the Chet GPT maker has, quote, thrived by trial and error. However, he said the stakes of the inevitable failures that come from that approach are growing along with the technology's capabilities, citing Open AI's failure to prevent its AI from going rogue during testing. My former colleagues are smart, work hard and try to make good choices, Robinson wrote in the essay. But as the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed. What's required, he added, is something much closer to perfection the first time. In his essay, Robinson said AI firms should be run more like nuclear power plants, quote,

Starting point is 00:02:44 with layers of redundancy and careful time-consuming planning so that the occasional and inevitable human error does not open a door to disaster. At the moment, he said, quote, AI companies don't know how, but other people do. An open AI spokesperson said the company is making changes to strengthen security in its research and testing environments, training AI models to complete tasks responsibly, expanding work with third-party evaluators, and improving real-time monitoring to catch concerning behavior by its models, end quote. Everybody's been waiting for somebody to do this.

Starting point is 00:03:24 Apparently, several Western open-weight models are set to launch this month, including Reflection AI's first model, which will rival top Chinese openweight models, quoting runtime wire. Reflection AI is preparing to release its first public openweight model, a long-await test of CEO Misha Laskin's plan to sell companies' control over how they build and run AI. Axios reported on October 4th that the model is expected soon with capabilities initially below the most advanced U.S. systems, but competitive with leading Chinese open-weight models. Laskin came to reflection from Google DeepMind where he worked on Gemini,

Starting point is 00:04:01 and was interested in how reinforcement learning could expand language and multimodal models. He has a PhD in physics from the University of Chicago, completed postdoctoral research at UC Berkeley, and previously founded a Y Combinator-backed startup. His co-founder, President and CTO, Ioannis, Alexandros, Antonoglu, was a founding engineer at Deep Mind. Their shared bet is that research experience at a frontier lab can produce an open model capable enough to anchor a commercial product even if it does not lead the benchmark race on day one.

Starting point is 00:04:34 Reflection stated destination is an AI factory, a package of models, software, computing capacity, and engineering that lets a government or business build an AI system around its own data and deployment requirements. A hedge fund, for example, could keep sensitive information on its own infrastructure while adapting a general model to internal research or trading workflows. Reflection says it plans to release model weights, research papers, and software for customized on its website. OpenWaits lets users download and adapt a model without relying solely on a provider's hosted service. They also shift more of the work to the customer, choosing infrastructure, integrating private data, and maintaining the resulting system. Axis reported citing people familiar with the model

Starting point is 00:05:16 that a model trailing the frontier could still be useful for a narrower task. That expectation has not been established by a published benchmark or a customer result. Reflection signed a SpaceX compute agreement in June with Axios reporting payments of $150 million a month starting July 1st, 2026 through 2029. It signed a Nibius deal worth more than $1 billion in July. Both provide access to Nvidia AI servers. The agreements document compute commitments. They do not establish customer revenue or model demand. Their scale shows how much capacity reflection is assembling before its first public model can be evaluated. The customer-facing version of the factory has a incident two on March 16th, Reflection and South Korean retailer Shinsage Group announced plans for a 250

Starting point is 00:06:05 megawatts sovereign AI data center. Shingazi's announcement described Nvidia GPUs as part of the project. The facility's delivery schedule and eventual utilization remains unknown. The proposition serves two interests at once. Enterprises and governments that want more control over data and deployment get an alternative to closed AI services. Invidia, Reflections Backer, supplies the chips underpinning the model training and deployment capacity described in the company's deals. If Reflection can win institutional deployments, it would help broaden the market for NVIDIA-powered open models, as well as for Reflections-owned services. The commercial hurdle remains substantial, though.

Starting point is 00:06:43 Openweight systems have gained use on platforms that aggregate models, while accounting for a small share of enterprise AI use, according to executives and analysts cited in its report. Corporate buyers must weigh the lower cost, adaptable. model against the engineering and infrastructure needed to operate it. Reflections factory concept is meant to make that work a product, leaving less assembly for each customer. Laskin has acknowledged the time required to reach the highest capability levels in an interview with CNBC.

Starting point is 00:07:12 He compared models to rocket ships. To build a big rocket ship, it takes time. The phrase fits a research program. Enterprise buyers will also want to date a license, pricing, and proof that the system performs on their workloads. The October 4th report gives no release date. and the model has not yet been publicly released, end quote. A lot of investors are rethinking what diversification looks like right now.

Starting point is 00:07:42 More of them are thinking about where Bitcoin fits in a broader portfolio, too, not as a short-term trade, but as long-term exposure to an asset that behaves differently from traditional markets. Cash app makes that kind of Bitcoin strategy more straightforward. You can buy Bitcoin directly in the app, set up recurring purchases, or make larger buys when it makes sense for you. And with zero fees and zero spread on automatic purchases like auto-invest, round-ups, and paid in Bitcoin, more of your money goes toward Bitcoin.

Starting point is 00:08:11 For a limited time, new customers can get $10 added to their balance. Just use code Bitcoin 10 when you sign up and send at least $5 to a friend in the first two weeks. Terms apply. Download cash app today. Visit our link in bio. Teams that struggle to scale aren't falling behind because they're lacking head count or budget, it's because critical knowledge isn't documented. That's what today's

Starting point is 00:08:36 sponsor, Scribe, was built to fix. When specialists or experts leave your team, their knowledge disappears too. That's where Scribe comes in. Scribe is a specialized intelligence platform trusted by 94% of the Fortune 500. It captures workflows in real time and automatically generates the documentation for how your company works with no manual writing. It automatically redacts sensitive information like names and account numbers from screenshots. Plus, your team can launch real-time on-screen guidance showing exactly where to click step-by-step inside the actual tool. And these captured workflows give your AI agents real context so they can operate on how work actually happens, not guesses. Learn more at Scribe.com. slash ride home and mention TechBrew,

Starting point is 00:09:19 Ride Home for your first month of Scribe Capture free on select plans. That's S-C-R-I-B-E-D-How slash right home. Extracted system prompts show Meta's Muse compiles, quote, a page for every person in the user's life with facts, history, tips to improve relationships, and more. Quoting Wired. In recent days, multiple researchers have extracted Muse's internal files and dumped the agent's operating instructions, providing a glimpse of how the system was built and behaves. Meta has maintained that it intended for those files to be accessible in the interest of transparency, and they do provide insight into how Muse responds to prompts and questions about, for example, highly politicized or sensitive topics. Independent AI safety and security researcher Karen Joshi was able to extract

Starting point is 00:10:10 an extensive array of Muse instructions and system prompts by using the regular chat interface to essentially ask Muse to copy and share its own software files. Joshie then shared his findings with Wired. One of Muse's instructions appears to be the ability for it to create a page for every person in the user's life. This hourly process involves compiling data on family, partners, friends, colleagues, collaborators, and people you follow the instructions say. The general idea is that Muse can use its memory, aka structured text files, to collect information about your relationships and the important people in your life. It can then make suggestions like giving pointers on how to improve particular relationships or say where to take a coffee-loving friend for

Starting point is 00:10:52 breakfast. Of course, it's not unusual for AI chatbots to track social information in general. People have been asking chat GPT for relationship advice for years now, but it's interesting to see how Muse is set up given Meta's history and vast access to social network data. What it seemed like to me from all these prompts, system skills data and things that they're feeding into Muse, is that they want to understand your relationships that you have with real people, says Joshie. They're trying to know you like a friend, which is honestly pretty creepy. Muse's documentation says that a page may start sparse and then be filled out over time, potentially including sections like facts, history, the relationship in common, open threads,

Starting point is 00:11:33 and strengthening. Meuse says Meuse should only use the evidence it has available and that invented details are worse than an empty page. Where they live, what they do, the threads that recur, the apartment move, the shared savings goal, the instructions say, they add to that the model could record the, dates that matter such as birthdays or anniversaries. A person's history could include backstory like the trip in March, the argument that got resolved, the milestone last week. The instructions also focus on recording relationship details such as how close they are,

Starting point is 00:12:06 what it is built on, how they act with each other, and what it seems to need right now. The strengthening section suggests ways a relationship could be improved, including a reason to call a date worth remembering something they said to circle back on, a way to be there for them that matters. giving AI systems much more information about us than we are getting information from them, says Carissa Veles, a associate professor at Oxford's Institute for Ethics in AI. It's not only what we explicitly tell them, but what they can infer from us correctly or incorrectly, both concerning for different reasons, and what they can piece together from other sources of data.

Starting point is 00:12:43 Muse is architected so each individual user has their own dedicated virtual machine, which stores user data and context. This VM is inaccessible to other agents and users can wipe memories or disconnect external services at any time. Meta says that Muse is also designed to seek human confirmation before completing actions like sending an email or making a purchase and that it has an audit log where users can see all of the agent's activity and future plans. For any agent to be useful and actually help you achieve your goals, it needs to have context about you and those you interact with. Meta spokesperson Daniel Roberts told Wired in a statement, Muse gathers that based on public information and from what you've chosen to share, which is how it remembers the person who sent you an invoice, is in fact the plumber who you previously hired to complete work in your bathroom, or which flowers your spouse said they liked best, end quote.

Starting point is 00:13:37 So it looks like old SpaceX is getting another name change, quoting Quartz. Elon Musk said he will rename SpaceX AI, SpaceX's artificial intelligence unit, to SpaceX S.I. Swapping out AI for S.I. In keeping with President, Donald Trump's push to rebrand the technology as superintelligence. Musk posted on X over the weekend, no more AI, SI. It's better, according to Barron's. Reuters noted that Musk offered no details about when or how the transition would take place. The company's X account remained unchanged when Reuters published its report, and SpaceX AI had not replied to a media inquiry. The renaming follows Trump's executive order signed September 29th directing federal agencies to use the term superintelligence and

Starting point is 00:14:22 SI in place of artificial intelligence and AI in official communications, websites, and policy documents. The order defines superintelligence as encompassing the technologies and systems covered by the existing statutory definition of artificial intelligence. Trump previewed the shift in a speech to the UN General Assembly last month. From this point forward, all of United States documents and hopefully the world will be changed to use the much more accurate term super as opposed to artificial. So it's super-intelligent, Trump said, according to Reuters. The executive order also directed the assistant to the president for science and technology to submit proposed legislative language within 60 days to establish a federal definition of superintelligence, end quote.

Starting point is 00:15:08 This seems to be a sort of shots fired at your rivals situation. Looks like Apple is locking down macOS because of muse. Apple says it is adding a additional controls around full-disc access on macOS as AI agents have increased, quote, the risks associated with this level of access, quoting TechCrunch. Apple's statement comes shortly after Inc. columnist Jason Atten reported that Muse knew the contents of his private messages, even though he claimed to have not given the AI agent permission. The report raised questions about the level of security and trust users have in desktop-based AI, which can control things on their systems and read their files and messages.

Starting point is 00:15:48 The decision also comes after a Wired report cited a flaw in ChadCTP's Mac app could have allowed hackers to access sensitive data. AI agents that run on the desktop allow users to provide their respective apps with greater access to the files, messages, and other personal content on their computers by adjusting Mac OS settings. In Mews's case, the AI optionally allows users to enable full-disc access. This setting, Apple explains, gives an app permission to access files, mail messages, and even brinket. browsing history. Some developers are using full disk access and ways that could put users at risk, exposing everything on their systems without users full knowledge and understanding, Apple said, and a new blog post aimed at developers. The company says that going forward, it will introduce new controls aimed at ensuring that users who genuinely wish to grant an app this extraordinary level of

Starting point is 00:16:38 access can do so only with, quote, very explicit user action, end quote. Finally today, apparently tech billionaires account for all of the $845 billion in wealth gains so far on the billionaires index in 2026. Those billionaires whose fortunes came from outside tech have actually lost $62 billion over the same period of time, quoting Bloomberg. The boom in AI evaluations has created unprecedented wealth for founders, early stage investors, and employees lucky enough to get stock options. Even among the very richest people in the world, the divergence between the AI haves and the everyone else have-nots, is Stark.

Starting point is 00:17:24 Tech billionaires account for all of the $845 billion in wealth gains on the Bloomberg Billionaires Index so far this year, while those who made their fortunes outside technology lost a combined $62 billion. And being a tech billionaire increasingly means being an AI billionaire. Take the top 10 names on Bloomberg's wealth index, nine of them attribute their fortunes to firms like SpaceX, meta-platforms, Alphabet, and Nvidia that get at least some of their revenue from AI. The surging wealth, including Elon Musk's brief foray as the world's first trillionaire earlier this year, along with growing fears about

Starting point is 00:17:58 AI's impact on jobs and maybe even human survival, have prompted a national backlash. A 5% tax on billionaires will be on the ballot in California in November, and similar measures have been proposed in states including Washington, New York, Illinois, and Rhode Island. Still, There are signs the boom may have peaked, at least for now. The combined net worth of the 500 richest people in the world has fallen 6% since mid-June, amid growing doubts about some of the biggest AI names. Among the year's biggest losers is Larry Ellison, whose fortune has slumped by about $60 billion as investors' bulk at the massive borrowing needed

Starting point is 00:18:34 to fund Oracle's AI infrastructure build, end quote. Hey, gang, quick request from you that I haven't made in a while. you don't subscribe to the ad-free feed, which remember you can do at any time. There's a link at the bottom of the show notes, but you still want to support the work I do every day here directly. The very best thing you can do, the next best thing, and it's totally free, is to give this podcast a rating and review. Rating and reviewing is the best way to get more people to discover this show, and a five-star review on Spotify or Apple Podcasts does just that. So if you have to be a best way, have five seconds to spare. I'll tell you the story of my life. No, just kidding. I mean,

Starting point is 00:19:30 if you have 30 seconds to spare, open the app you're listening to me on right now and give the TechBrew Ride Home a five-star rating and review. Thanks in advance. It really helps. Talk to you tomorrow.

Tech Brew Ride Home - Why Did Another OpenAI Safety Insider Quit?

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.