The AI Daily Brief: Artificial Intelligence News and Analysis - Is OpenAI Secretly Training GPT-5?

Episode Date: September 4, 2023

Inflection's Mustafa Suleyman indicates that he is skeptical of claims that OpenAI isn't training GPT-5, and points out that even if they're not, others are busy training more advanced models. ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, we're looking at recent comments from an AI industry leader about whether or not OpenAI is secretly training GPT5. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.com to Breakdown Network for more information about our newsletter, our Discord, and our YouTube. Welcome back to the AI breakdown. Well, it is a holiday here in America. It's Labor Day for those of you who are not in the U.S. And so at first I was thinking maybe I'll finally take a day off, but there has been this point of speculation, of rumor, of innuendo that has been bubbling for a while but got a big update just before the weekend. And I think that given the significance to the industry of this particular question, it was worth doing a little bit of an exploration of and a long holiday weekend seems like the perfect time to do it. The question I'm referring to, of course, is whether OpenAI is training GPT5.

And before we get into the new context for that question, let's go back and give a little bit of context for it. Now, almost as soon as GPT4 was launched, rumors came out that OpenAI was already developing GPT5. Sam Altman, the CEO of OpenAI, went to pains to say that that just wasn't true. The Verge writes about Altman's appearance at an event in MIT at April, where he was queried about the open letter that asked for a six-month pause for all AI development of systems that were more powerful. than GPT4. From the verge, Altman said the letter was, quote, missing some technical nuance about where we need the pause and noted that an earlier version claimed that OpenAI is currently training GPT5. Quote, we are not and won't for some time, so in that sense it was sort of silly. Okay, so that was April. The next month in May, Sam Altman appeared before the Senate for the first hearing on AI

post-chat GPT, and once again reiterated that OpenAI was not training GPT5 at that time. Now, at that point we started to get a sense that this might be not just for technological reasons, but also for regulatory reasons. One of the ideas that seemed to be coming to the four in Altman's testimony was the idea that for models that were more advanced than GPT4, there might need to be some sort of licensing regime or approval process. And so given that that was sort of where the winds were blowing, perhaps OpenAI was thinking that it didn't really make sense to walk down the GPT5 path without that process becoming a little clearer. In June, Altman once again reiterated that OpenAI was still not training GPT-5.

At a conference hosted by the Economic Times of India, he said, we have a lot of work to do before we start that model. We're working on the new ideas that we think we need for it, but we are certainly not close to it to start. Now, since then, we haven't heard anything directly from Altman, but of course no update doesn't necessarily mean that they have now switched and started working on it. It could just mean that we haven't had an update

or that Altman hasn't wanted to repeat himself yet again. What did happen in the meantime is that ChatGPT released its code interpreter feature, which they've subsequently renamed because the name never really made sense. But basically, Code Interpreter was one of two plugins that OpenAI was working on themselves, the other being an internet browser. And in many ways, both of these new features. One, the ability to browse the web. And two, the ability to write code, which in many ways allowed GPT to expand the capacity

of things that it could be good at by giving it the ability to write scripts to help it with certain problems. Many thought that by and large, when you combine these two things or even independently, they add up to a model that is significantly more advanced than GPT4. Swicks from Latenspace podcast very prominently called code interpreter GPT 4.5, and others have had a similar analysis. What's more, people speculated that the reason that OpenAI wouldn't just call this GPT 4.5 was that regulatory concern. Another update that happened a little bit after Code Interpreter was released was the release of ChatGPT Enterprise. In addition to having Code Interpreter integrated, one of the things that ChatGPT Enterprise offered,

was, quote, unlimited higher speed GPT4 access. Some, such as Jan Peleg here, took that as an indication that they no longer needed to constrain their compute and that the most logical reason that they would no longer need to restrain their compute was if they had already finished training GPT5. So this is where the state of the conversation

was heading into the end of last week. But then we got interesting comments from Mustafa Saliman. Now, Salimam is, of course, the previous founder of Google DeepMind, and now the CEO of Inflection AI, but he's also promoting his forthcoming book about AI, and so has been on something of a media tour. In a conversation with Robin Wiblin on the 80,000 hours podcast,

this is what he had to say about GPT5. When asked if he believed that they were secretly training GPT5, he said, I don't know, you have to ask them. I like them very much, and I have huge respect for them, so I don't want to say anything bad if that's what they've said. But also, I think Sam Altman recently said they're not training GPT5. Come on, I don't know. I think it's better that we're all just straight about it.

That's why we disclose the total amount of compute that we've got. And obviously you can work out from that, roughly speaking, what order of magnitude of flops we're using. It's much better that we're just transparent about it. We're training models that are bigger than GPT4, right? We have 6,000 H-100s in operation today training models. By December, we will have 22,000 H-100s fully operational. And every month between now and then we're adding 1,000 to 2,000 H-100s. So people can work out with that enables us to train by spring by summer of next year,

and we'll continue training larger models. And I think that's the right way to go about it. just be super open and transparent. I think Google DeepMind should do the same thing. They should declare how many flops Gemini is trained on. So a lot that is packed in there, right? One, he basically intimated that it feels very unlikely to him that OpenAI wouldn't be training GPT5.

Two, he's basically saying that even if they're not, everyone else who has the compute is trying to get out ahead of where the models are now. In other words, to the extent that OpenAI has voluntarily paused, their competitors certainly are not. The AI safety memes account also put, hold some other related quotes from the conversation. One, they wrote, we're going to be training models that are 1,000x larger in the next three years. Even inflection will be 100x larger

than current frontier models in the next 18 months. Now, the response on Twitter to people posting about this was basically incredulous. I did a poll asking if people thought that OpenAI was training GPT5. And while far from scientific, 65.5% said absolutely, 26.7% said most likely, and only 7.8% said not yet. And reinforcing the other part of Mustafa's Sullyman's statement, basically the competitors were training GPT4 plus size models, even if OpenAI wasn't. On August 25th, Jason at AGI Kuala wrote,

overheard at a meta-gen-AI social. We have the compute to train Lama 3 and 4. The plan is for Lama 3 to be as good as GPT 4. Response, wow, if Lama 3 is as good as GPT 4, will you guys still open source it? Yeah, we will. Sorry, alignment people. Altman Sam tweets,

Meta wants to open source a GPT-5 level model and seems dead set on open sourcing right up until AGI. I want to be clear about what this means. One, there is no kill switch. If something goes wrong, an agent gets out of control or a bad actor weaponizes it, there's no easy way to turn it off. Two, safety research becomes meaningless. All the work people have done into making AI systems honest, aligned, ethical, etc., becomes mostly moot. The population of AI's out in the world will evolve towards whichever systems produce the most economic output, irrespective of what values or motives they have.

Now, I'll save the open source debate for another conversation. I think the point here is the sort of affirmation, at least anecdotally, and we should have the caveat that this is very anecdotal, of another competitor that is racing to get ahead of GPT4 level capacities, and is at least from that anecdotal evidence, still planning to release it publicly as open source. Now, just to put a fine point on how this competition could lead companies to act differently than they might otherwise. Another sub-theme from last week was a little bit of a back-and-forth between a blog that claimed that Google's Gemini would smash GPT4 and Sam Altman, who intimated that that blog was just doing Google's PR work for them. Summing it up, Insider writes, AI bros are at war over declarations

that Google's upcoming Gemini AI model smashes OpenAI's GPT4. Insider writes, it's never fun for a CEO to hear their product might get thrashed by a competitor. That might explain OpenAI boss Sam Altman's defensive response to a post published over the weekend titled, Google Gemini eats the world. Gemini smashes GPD4 by 5X, the GPU pours. Altman's tweet said, incredible Google got the semi-analysis guys to publish their internal marketing and recruiting chart, lull. Insider continues, much of the analysis, which the author said was based on data crunch from a Google supplier, boils down to Google having access to infinitely more top-flight chips and its model out doing GPT4 on a performance measure

relating to computer calculations known as flops. Now, as many people pointed out on Twitter, it is up for debate around how much just having more chips means better models. X-user Technuium said, I hope someone to Thrones open AI soon, but to say Gemini smash's GPT4 by 5X makes it sound like it's 5x better than GPT4. It's not its 5X compute, not necessarily correlated to quality. A hacker news user wrote, computational power alone is not the only resource.

It is also the training process itself and obviously the data and its quality. I will be convinced only after Google demonstrates that Gemini is better than GPT4 in some or all tasks. So where does this all net out? A few things. First of all, it's very clear that whatever Open AI is doing, their competitors are not standing still. Meta appears to be proceeding with Lama 3 and 4. Google is certainly banking on Gemini to put them back into this race in a big way. And I think that that word race is really relevant here.

Remember, one of the reasons that Jeffrey Hinton left Google was that he was concerned that the economic incentives of competition around these foundation models were already starting. to make companies act differently and with less responsibility than they might otherwise have. To the extent that Open AI isn't training GPT-5, they might find themselves in the very unenviable position of being behind where they once were ahead. Will that make them think differently about safety and alignment issues? It's hard to say. But the pressure is certainly going to be there. Now, the flip side, of course, is that as interest in AI has felt on a lower ebb over the course of the summer, one of the things that people anticipate bringing lots of people back is expanded capabilities of future models.

Avi Schiffman posted a chart of what people use chat GPT for. 29% programming questions, 23% education questions, 21% content questions, 13% sales and marketing questions, and asks, I'm curious how a multimodal GPT5 will change this. As with anything in this space, there is a lot to be excited about as well as trepidacious about. But for now, that's the story from here. Thanks for listening or watching as always. And until next time, peace.

Thank you.