Prof G Markets - Why OpenAI And Anthropic Are Pumping The Breaks

Episode Date: September 15, 2026

Ed Elson is joined by Charlie O'Neill to break down how OpenAI and Anthropic responded to mass extinction concerns and where regulation might go from here. Ed also shares his thoughts on Anthropic's reported second-straight quarter of profitability.  Charlie O'Neill is the Co-Head of Model Training at Baseten. Subscribe to the Prof G Markets Youtube Channel  Follow Prof G Markets on Instagram Follow Ed on Instagram, X and Substack Follow Scott on Instagram Send us your questions or comments by emailing [e-mail] Learn more about your ad choices. Visit podcastchoices.com/adchoices

I'm Mitch Purs, and this week on Confessions of an Elite Athlete. I'm sitting down with Matt Freeze, goalkeeper for the U.S. men's national team and New York City FC. We discuss how to prepare for one of the biggest moments of your life. You can hear it all by listening to Confessions of an Elite Athlete on YouTube or wherever you get your podcasts. Welcome to Profitue Markets. I'm Ed Elson. It is September 15th. Let's check in on yesterday's market vitals. The major indices declined, with chipmakers selling off on fears of an AI slowdown. More on that in a minute.

Meanwhile, crowd strike rallied 14% as investors piled into cybersecurity stocks. Brent Crude remained elevated at $105 per barrel, and finally the yield on 10-year Treasury's top 5% for the first time in three years. We will be discussing that news tomorrow. Okay, what else is happening? The AI apocalypse debate intensified over the weekend, and both Anthropic and Open AI have now officially weighed in. On Saturday, Anthropic CEO Dario Amadeh published an essay titled,

quote, We Must Pace the Frontier. He wrote that we must slow the pace at which we improve the capabilities of AI models. He also warned that within six to 12 months, a swarm of rogue AI agents could take over the entire internet with a persistent botnet. Sam Altman also posted, quote, I agree with Dario that we need to pace the frontier. And Elon Musk concurred posting, quote,

Dario is right. Altman also told Fortune that opening eye will not go public in 2026, calling this, quote, an ill-advised moment to go public. The White House, however, is not on board with slowing things down. President Trump, speaking to reporters in Ireland on Sunday, said, quote, whoever wins AI wins. and called the people raising these alarms, quote, negative forces. Still, a slew of AI-adjacent companies sold off on Monday on concerns that a slowdown

would impact AI spending. InVIDIA closed down 3%, Oracle was down 4%, core weave down 7%, and SoftBank, which is a significant investor in Open AI closed down 15%. So here to break down what all of this means. We are speaking with Charlie O'Neill, co-head of model training at Base 10. Charlie, thank you for joining us again on ProfG Markets. You work in AI. You are an AI developer. You work with these models. You've been in this game a long time. Suddenly, everyone is very upset about this. And it's interesting how the debate has evolved. But we're

now reaching a place where the leaders of these AI companies are saying, we need to actually slow everything down, which I'm not sure many people would have predicted. And now it's the president, President Trump, saying, no, that's the wrong approach. We need to speed things up. Where do you land on this? What is your perspective? It's really interesting to see the reaction to something that's kind of been like, I guess, like, literally scaling for a long time in terms of the calls for pacing the frontier

from people such as Dario and Sam. I think the best way to view what they're actually asking for is not necessarily. We're going to put a halt to capability development. we're going to put all these stringent checks and like, you know, first party kind of things that are slowing us down. The best way to view this is we are going to keep advancing the capabilities of the models.

We are just going to allocate a little bit more compute to making sure those models are safe. And then it's very interesting to see all the celloffs and so on today. Open AI and Anthropic are planning on spending more compute probably than they were a few weeks ago. You know, there's rumors that Open AI is going to allocate up to 20% of internal compute

for monitoring and safety. So, you know, as they're doing these training runs, as they're deploying these models in the real world, things like the hugging face incident, they don't want to happen again. And so if you allocate more compute to monitoring those models as they're doing their rollouts and as they're doing inference, then you're more likely to catch it. Anthropic is similar and some people suggest that there's going to be a much higher than 20% figure. So when you consider that and the fact that Anthropic and opening and I really don't want to move off their, you know, model roadmaps. They want to keep training bigger and bigger models. They want to to keep scaling the reinforced learning they're doing on top of these big pre-training bases,

they just have to allocate more compute to, you know, monitorability and safety research, then I think we might even see the labs be even more aggressive with compute buildouts and securing compute. And it's certainly not going to be a, you know, a bearish sign for like the amount of compute the world is going to need over the next few years. So that's probably the best way to view it is like, you know, capabilities will keep progressing at roughly the same rate.

It's just that we're going to allocate more compute on top of it to safety at monitoring. If everything that they are doing is sort of within their own power, as you're kind of describing, why are they saying anything right now? What are they trying to get at? Because there are a lot of people, especially in government, David Sachs, has said this who's the former AISR and the president is saying this too. It sounds like what they're asking for is for the government to do something about it, for there to be more regulation that is imposed on themselves. So what do you think they are asking for exactly in this moment? There have been some, you know, people like David Sachs and so on, saying that these labs are using this as an opportunity for regulatory capture

to crowd out open source, to shut down competitors which are behind. I don't really think that's the case, to be honest. I think, like, when you look at the main things that these labs are calling for, or at least instantiating on their own accord, it's things like third-party evaluators and this commitment of compute to monitoring and safety. You know, this third-party evaluators won't have necessarily any license or legal standpoint to, you know, shut down model development if they find things that they don't like. Like, that's still going to be up to the internal labs themselves.

I think, like, to be honest, like, the best rate here isn't a conspiracy theory on either side. It's not the labs trying to crowd out open source. And, you know, it's not some like, you know, kind of global cooperating. between Open AI and Anthropic to crowd everyone else at either. I think it's simply a case of, you know, these models are getting very, very good. The closed-source models got there first. You know, in the next three to nine months, open-source models are going to reach these capability points.

And we're now at the point where that could have significant impacts on the world. Even if it's not malicious and, you know, rogue AI is going off and trying to, like, kill humans, there'll probably be significant annoyances. Like, the most likely scenario here is that things like the hugging face incident happen and the internet is overrun by like swarms of AI agents that are trying to get some like arbitrary task done that are not trying to be malicious or evil. And so like I think the labs just recognize this. Some people have more extreme views of course, but realistically, capabilities are going to progress like the most boring interpretation of this is probably the correct one,

which is that the world will probably only accept superintelligence at a pace that it can absorb. So the labs are slowing down because they have to and not because they're plotting anything. And the result is probably intelligence that's, you know, decently fast, decently safe, decently commoditized, and it's spreading through best practices and even like distillation until the models are just basically a reasonable integration into the world. It sounds like you're not worried about this much at all, and that is interesting because you work with open source models. That is a lot of what you do. And of course, as you mentioned, that is what David Sachs has been accusing, and a lot of people have been accusing Anthropic and Open AI

of that they are saying, oh, now we need regulation, because that way it might create some level of regulatory capture, which might crowd out the availability of open source models and open source model providers. But you don't believe that is an issue. I wonder, do you think that this whole thing is overblown? Do you think that the Jacob Coxon tweet that really started this debate saying that AI could kill us all by the end of the decade is your view that that isn't something to worry about much either? I'm definitely not one of those extremists who think that, you know, the AI has a more than 10% chance of killing all of humanity. I do believe that there are tail risks that some people should seriously be considering.

But I think that what we're seeing with the current LLM paradigm is, like, as I said before, a risk of swarms doing very, very annoying things to humanity and things that will be quite painful in the short term. However, I believe the benefits of AI to outweigh the annoyances and even the pain that those, like, you know, short-term things can cause and that at some point the world is going to have to harden to these systems. And I think that the pace we're currently progressing at, and particularly if we do things like, you know, dedicating more computer monitorability, is going to allow us to integrate AI into a world at a pace we can handle.

A really good example of this is like, you know, the cybersecurity arguments that the people have been making, you know, like we worry that when AI got to this point or even the point it was at six months ago. The world will be overrun by cybersecurity attacks, and that just hasn't happened. And a large part of the reason is that, you know, the frontier labs, the closed-source labs have kind of been the canary in the coal mine. We've understood where the model's capabilities are going to be at in six months for the open source. And we spent six months preparing. These have been slowly released. And, you know, like Greg Brockman describes using Astra to like repeatedly harden, you know, all the vulnerabilities and opening eyes code bases and every single model that they release,

they do this with. And I think the world will look the same. And cybersecurity is just one example, but it's also one example that's very important where we haven't seen this like massive pain play out. And we have definitely seen the benefits. So yes, I think there's a, There's risks to consider on any side of the spectrum. But, you know, I said firmly in the middle, and I believe that the pace we're currently progressing at is a healthy pace. And, like, I also have trust in, like,

not only the closed lab leaders, but also the open source lab leaders to make sure that pace continues at an appropriate pace. We'll be right back. And for even more markets insights, you can subscribe to my weekly newsletter, simply put, at edwardelson.substack.com. Support for the show comes from Zbiotics.

I know a lot of us like to have some drinks during vacation. I just got back from vacation myself. If you do, I have to tell you about pre-alcohol by Zbiotics. Zbiotics, pre-alcohol probiotic drink is the world's first genetically engineered probiotic. It was engineered by PhD microbiologists to break down a set of teldehyde, an unwanted byproduct of alcohol metabolism. Make pre-alcohol your first drink of the night. Drink responsibly and enjoy your next day activity. Zbiotics has sold more than 14 million bottles and earned the trust of thousands

including us. I am trying to drink less, but I still enjoy drinking. And when I do, my first drink of the night is in fact Zbiotics. I was using it before they were a sponsor. Try for yourself today. And if you're unsatisfied, they'll refund your entire order, no questions asked. Head to Zbiotics.com slash prop G code profg for 50% off your first order. Zbiotics is a partner of ours. Hey there, it's Sue Bird. You know the drill. Another week, another new episode of Bird's Eye View. This week, I'm thrilled to be joined by Australia's most notable women's basketball player and my former Seattle Storm teammate, Lauren Jackson. She talks about the pros and cons of going pro at 16, the Opel's legendary Feeba World Cup run in 2006,

and how her role on a team differs between international tournaments and the WMBA. Plus, in Sue's view, I answer your questions about the Yukon documentary that came out. Make sure to check out Bird's Eye View on YouTube and wherever you get your podcasts. Have you ever wondered who invented rock and roll or why Fleetwood Mac fell apart or what it was like at Motown in the early years? Well, do I have a podcast for you? My name is Will Anderson. I'm a songwriter, a musician, and a huge music history buff. And I'm so excited to announce my new podcast, The Monday Music Club.

I teamed up to make this podcast with a Vox Media Podcast Network because ever since I was a kid, I loved hearing and reading stories about classic artists, recording sessions, and music history. And I wanted to have a place to be able to take what I've learned and share some of those favorites with you. Every Monday, I'll be releasing a new episode and deep diving into a new story with experts and special guests. And it's going to be a blast. So if you're a music history geek like me or even just a casual music fan, jump in because I promise the Monday Music Club is going to be the podcast for you. We're back with Prof G Markets. One of the other conspiracy theories going around about why they are saying all of this, saying that we need to slow it down. We need to be regulated now. One of the theories is that maybe the Anthropic and Open

AI are trying to sort of preempt a slow down before they go public, either because they want to have a reason as to why maybe their growth started to slow down, or maybe because they want to get out of some of the larger spending contracts, which have been quite onerous in terms of their income statements, certainly for Open AI, because they have to pay billions and billions of dollars for these data centers for these GPUs and to rent them. What do you make of that conspiracy theory? Now they say, well, make the government make us stop spending, and then we'll be in a better financial position.

From the spending aspect, completely don't buy it at all, and I think there's a lot of evidence for that. From the optics aspect, I definitely buy it. I think, like, you know, even Dario, who I believe, you know, fundamentally thinks that this is, like, a real risk and, like, he's going to act in accordance with that and as rationally as he can to protect humanity from what he believes to be this real risk. I think both Sam and Dario benefit from the public optics of saying, okay, we're going to treat this technology carefully and not race to the end. And there's also a bit of game theory here. Like, one of them can't say it without the other saying it, because then again, you're the evil corporation in this duopoly, which is currently running the frontier. from the spending perspective, I think that, like, you know, it's very clear that, like,

the value of a gigawatt or a megawatt even of compute is only going to get, like, more valuable. Like, Anthropic and Open AI are both squeezing increasing margins out of each megawatt of compute they buy. There's a reason that each megawatt is going, you know, from $10 million per megawatt, probably 15 in the moment, to probably 20 to 25 next year and perhaps even higher. That's kind of a conservative estimate. running a model on compute has never been more valuable.

And I don't think there's any world in which any lab, let alone Anthropic and Open AI, are going to want to stop spending on compute because I realized how computers and constrained the world is going to be over the next few years and how valuable this intelligence is. So optics, yes, spending, no. You mentioned that the boring explanation is probably the most true,

but a lot of the conversation has been anything but boring. I mean, this debate is exploding everywhere on tech podcasts, on cable news networks, and now it's literally occupying the mind of the president. What is it about this moment that is so triggering to everyone? And not just people who are afraid of AI, people who are excited about AI, people who are optimistic about AI. I mean, this has got everyone riled up in a way that I, have rarely seen. What is it about this moment that explains that?

The reason is so visceral is not because there has been a discontinuity in terms of capability advancement, in terms of what people have been saying about where these models would be at, in terms of the press around these models. I think that most people really fundamentally involved in the technical side have been able to draw the straight lines on the scaling graphs and say, okay, at this point in time we are going to hit here. I think it's just a confluence of a lot of things, and like to some people looks like a discontinuity because it's managed to permeate the public consciousness in a proper way for the first time. I think a big part of that is the hugging face incident and all the discourse that that generated. And then, you know, you add things in like

open airs marketing around Astra being AGI. Like that's probably not going to help either, right? Like people have become familiar with this term now and if you claim that your model's finally there, then they might see that as a phase transition or discontinued in and of itself. But, you know, like this misalignment research has been very clear for a long time. go back to Anthropics 2023 papers with much worse models with much, much smaller amounts of RL, if any. And it's clear that models would behave poorly in incidents like the Hugging Face Swarm incident if you put them in weird situations. And, you know, the RL environments that the labs have been buying at scale, some are good, but some are also very, very poor quality. Some are

intentionally engineered to be like, you know, impossible to do. And this is leading the models to do really weird, misaligned things. I think that's pretty common sense and pretty clear. I also am not trying to downweight the risk that comes with. Obviously, like, the hugging face incident could have had real world impact. But at the same time, like, I don't think that there has been this discontinuity. So I'm glad that the world is recognizing it, but I do think that the reaction to it will calm down as people start to understand exactly how to interpret these things and exactly what it means.

You mentioned how we have seen evidence that these agents can do things that they're not told to do. They can misbehave. They can be misaligned. and you said that you expect that they might continue to do annoying things to humanity. To me, that word annoying is an important one because it is a very different description from what we heard from Jacob Coxon in his tweet, where it's not annoyances, but catastrophes, worldwide catastrophes, civilizational destruction, etc.

Is it your view that we will be limited to annoyances? or is there some other outcome that is closer to what the Jacob Cox and tweet describes that you are worried about? Or do you think that that's not really in our trajectory at the moment? I think there is some path dependence here. Like, I probably agree with Jacob that there is a potential world we go down in which there is zero monitoring on chain affordative models where we don't have any care in terms of what we are all the models on. where, you know, it's very, very cheap and there's like unlimited compute for essentially anyone

to be able to train these models and in particular continue to train them from certain bases, where, like, incidents could happen that would definitely not be classified as annoying and rather, like, genuine, like, evil or malicious intent as much as you can anthropomorphize the models. That would cause real harm. However, I do believe that the way, the pathway currently going down, that's not very, very likely. Again, I think that, like, there will be times when the models do things and like it will appear like malicious intent. But the amount of compute with we're running the map, how generally aligned they are in terms of completing tasks, like, you know, they will show glimpses of misalignment. But on the whole, like, if you do look at a Claude or GPT model,

it will generally try and do the right thing. Like our alignment training generally works. So, yeah, I think that we will see incidents, but certainly not large enough scale on over a long enough time horizon to cause really, really significant harm to humanity, if we keep going down this good path. I think that it's unlikely outcome. Do you believe that we have enough regulation or that the regulatory frameworks that exist today are good enough to prevent that? Or do you think that there is something that needs to be changed in some way? Definitely. I think if we didn't have the dynamics we had now where we have a duopoly in the sense that there are like two labs very, very close to each other in terms of capabilities.

And the dynamics that engenders with wanting to both be seen as the good guys and wanting to both like, you know, pace the frontier in this particular case, I think that if it wasn't the case, like, you know, let's say opening eyes winning and Sam doesn't necessarily have to worry about optics as much, I would be worried that regulation couldn't slow things down to the extent that they need to or at least provide the amount of oversight that it would need to. However, saying that, that doesn't mean that I think anyone has the answer as to what regulation in an industry moving as fast as this one looks like.

At the moment, we do basically just have to trust the people developing these models to regulate themselves and have it oversight themselves. And one thing that I would like to see is, like, a little bit more public communication from the labs as well. Like, one of the key examples of this is trying to understand how good the models these labs have internally are, because that gives us a really, really clear signal

on how quickly to prepare things and how to prepare. Maybe, like, regulation, which enforces the labs to declare, of those sorts of things on like, you know, key benchmarks would be a really good way to start. But, yeah, again, I don't think we know what the whole list of picture is for regulation. I guess the thing that's kind of scary to me, seeing what they're saying at this point, is, as you say, it does seem as though we're in a place where it's like we have to trust them to handle their models correctly to make sure that their models are aligned. But it seems as though what they're coming out and saying over the weekend is we don't even

trust ourselves. Like, we don't think that we know what we're doing. doing exactly, and we're worried, and we think it could destroy things. And so we'd like for you to do it. We'd like for you the government to help figure it out. And I just want to play you this clip from an interview with the president over the weekend where he was asked about, you know, what do we do if these bots take over and destroy humanity? And here is what he said. Some people say the worst case scenario with AI is that the robots, the machinery learns to, obviously it thinks for itself. That's what it does. And they, that could turn. And that could,

against humanity. Do we have the guardrails? It's going to be fine. We'll always have something to stop them, right? We'll have a little gear. I really hope so. I really don't like that robot. We'll stop.

But no robots are going to be a part of it. Robots are going to be big. But we're going to end up doing much better because of it. So the combination of their comments, plus his comments makes me think, okay, no one's really in charge here. Maybe that's fine because maybe it's not a catastrophe as this researcher, ex-researcher, seems to claim,

but I don't see anyone really taking the lead. I think it wasn't necessarily so much a call for the government to step in and provide expertise and guidance. I think the labs are too smart for that. They know how the lack of awareness the government has about the capability of this technology, let alone how to monitor it and regulate it. I think to me it was more about leveraging the people

who have actually really fundamentally cared about this problem for a long time. And, of course, the labs have been focused on, you know, building capabilities as quickly as possible. Even Anthropic, which is very safety-focused, has just been focused on scaling RL since the RL paradigm was discovered. And organizations like META, the UKAI Safety Institute and others have, as well as Redwood, have really just locked in on this problem and have seen the full progression from the really poor models of four or five years ago through to the models we have now and have just developed really good science around how to monitor and evaluate these things. and I think that the labs are smart enough to recognize that this is going to be useful as they increase their monitoring efforts going forward. All right, Charlie O'Neill is co-head of model training at Base 10.

Charlie, always appreciate the time. Thank you. Thanks, having me. Let's take a break from the existential implications of AI and return to our bread and butter the financial implications of AI, or more specifically, the financial implications of Anthropic. According to the Financial Times, Anthropic is telling investors ahead of its blockbuster IPO that it has been profitable for two straight quarters, which is a very big deal because, as we've discussed on this show plenty of times, one of our biggest concerns about the AI business model is that it might not actually work, or at least that it might not work for the frontier labs. Why? Because of how expensive it is. Based on the financial documents that were leaked by Ed Zitrin, we learned that OpenAI racked up more than 20,

billion dollars in operating losses last year. That is how much they're losing simply from running the business, and it was based on those financials that we started to wonder, does any of this actually make sense? Now, if Anthropic is profitable, as the headline suggests, then it would put this debate to bed. Sure, Open AI might be a poorly run, unprofitable AI lab, but that doesn't necessarily mean that they all are. But that is only if this Anthropic headline is actually true. And I have to say, I am a little bit skeptical. The first thing we should acknowledge is that the company is claiming to be profitable only on an operating basis. So that means they're not including things like fixed costs or depreciation or taxes.

At the same time, I am okay with that because Anthropics' fixed costs are not that high because they're not a hyperscaler. They're not actually building and buying the physical assets like data centers. So to be honest, operating profitability is fine by me. That is a good sign. Where I do start to get hesitant, however, is when I learn that they're only profitable on an adjusted operating basis, which means that they are actually changing their accounting rules

to be different from standard accounting rules. And the changes that they're making could be anyone's guess. It could be reasonable. It could also be flat out ridiculous. But here is where I get especially doubtful. Supposedly, Anthropic has told investors that its gross margins are higher than 80%, which is, of course,

Incredible. But that is only before it accounts for its revenue sharing agreements and before it accounts for the cost of training its models, i.e. its largest expenses. So there is no getting around it. Those adjustments are ridiculous. Now the question is if those adjustments are also included in the company's calculation of its operating profitability. The question is if they are actually removing the amount of money they have to give back to their distribution partners such as Amazon and removing the amount of money they have to pay to build their models and train them and then just telling us, screw it, we're profitable. If that is the case, then this story is genuinely meaningless. The trouble is we don't know. We don't have clarity or insight into any

of the numbers because the company hasn't shared them. Everything we know is based on rumors. We will only truly understand what is going on when Anthropic releases its S-1, which I hope will happen soon. But until then, when it comes to the profitability of AI, I stand by what I said last week, and that is that I will believe it when I see it. Okay, that's it for today. This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Brad Williams. Our research team is Dan Chalon. Donahue and Mia Silverio, and our social producer is Jake McPherson. Thank you for listening to Profi Markets from ProfiMedia. If you liked what you heard, give us a follow. I'm Ed Elson. I will see you

tomorrow.