The AI Daily Brief: Artificial Intelligence News and Analysis - Apple Spending Millions Each Day Training AI

Episode Date: September 7, 2023

According to new reporting from The Information, Apple is spending millions each day training AI. That said, its still not clear what they're actually planning to do in the space. Before that on the Brief: OpenAI announced its first developer event for November but says not to expect GPT 4.5 or GPT-5; Gavin Newsom signs AI executive order in CA; Google requires political ads to disclose use of generative AI, and more. Today's Sponsor: Supermanage - AI for 1-on-1's - https://supermanage.ai/breakdown ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, we're looking at reports that Apple is now spending millions of dollars a day training their internal AI models. Before that on the brief, OpenAI excites the developer community with the promise of their first developer event in November. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.netnet for more information about our Discord, our YouTube, and our newsletter. Hello, friends, one quick note before we dive into the brief. Today is my birthday. As a birthday present, I have one request. If you are finding yourself getting value from the AI breakdown, whether it's professional

or just personal, the biggest thing that you can do to help me out is actually leave a rating or a review. When it comes to recommending new podcast to people, Spotify and Apple, and all the other players, really, look at ratings and reviews more than just about anything else. I really appreciate all the people who have taken the time to do that already. And if you haven't yet, today would be an amazing day. I appreciate each and every one of you for listening, so let's now get to the brief. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes.

Today, we kick off with a small announcement that has people very excited. Yesterday, OpenAI CEO Sam Altman tweets, on November 6 will have some great stuff to show developers. No GPT5 or 4.5 or anything like that, calm down, but I still think people will be very happy. So what's going on is this is what OpenAI is calling its first developer conference. They write, the one-day event will bring hundreds of developers from around the world together with the team in OpenAI to preview new tools and exchange ideas. In-person attendees will also be able to join breakout sessions led by members of OpenAI's technical staff. The quote that they ran with from Sam Altman is similarly vague, saying, we're looking forward to showing our latest work to enable developers to build new things. Now, of course, this has got people speculating like crazy. Vion Williams says, okay, got it, agent systems based on GPT4. Max Kennan says some wild guessing. cheaper API, some kind of agent, or some kind of code model. Robert Scoble says, multimodal coming? November is going to be fun here on X. And Nate Chan points out, OpenAI has the chance to create a WWDC-type level of excitement event if they do this every year

and with as much care as Apple. Constantineus, however, points out that that's kind of far away, and that in some ways a two-month lull might represent an opportunity for someone else. He tweets, no new versions of GPT until at least November 6, so everyone can calm down, or is this an opportunity for others in the space to get ahead. Now, I think the big question from where I'm sitting is how much whatever gets announced there is really just for developers, things like new tooling, versus how much is something that those of us who are normies in the AI space will actually derive immediate value from. Of course, updates for developers eventually mean value accrues to the normies as they get to use

new features that are built on top of those new tools for developers. But in the meantime, everyone's going to spend the next two months just guessing. When it comes to the theory around multimodality, what's going for that idea is that open AI I had previously talked about bringing a multimodal version of chat GPT this year and had intimated that the only reason that they hadn't was an AI chip shortage. That would certainly fit with the pattern of doing things with GPT4 that souped up so much from where it started that it almost feels like a different model. One other interesting little stat before we move on, Logan, who does developer relations at OpenAI, tweeted. It's also crazy to think that more than two million developers

are using our APIs to build AI experiences. Just an interesting little nugget in the scale of who is building right now. Now, that open AI event will be in San Francisco, and staying in the realm of California, that state's governor Gavin Newsom has signed an executive order designed to start preparing California for the AI revolution. The governor's office writes, California is the global hub for generative artificial intelligence. We are the natural leader in this emerging field of technology, tools that could very well change the world. To capture its benefits for the good of society, but also to protect against its potential harms, Gavin Newsom issued an executive order today, laying out how California's measured approach will focus on shaping the future of ethical, transparent,

and trustworthy AI while remaining the world's AI leader. So what is actually in this executive order? There are a number of different provisions. One is a risk analysis report. This will quote direct state agencies and departments to perform a joint risk analysis of potential threats to, and vulnerabilities of California's critical energy infrastructure by the use of Gen. A.I. A second area is a procurement blueprint. Basically, this is all about developing a process by which generative AI can be used government offices and agencies. Beneficial uses of Gen. A.I. Report. In addition to just looking at those critical risks, Newsom is also directing state agencies to develop a report, quote, examining the most significant and beneficial uses of Gen A.I. in the state. Deployment and analysis

framework? Basically, this is a mechanism by which the state can start conducting pilots, as well as creating sandbox environments to test new projects. The executive order also establishes a formal partnership with the University of California, Berkeley, and Stanford to consider and evaluate the impacts of Gen A.I. on California. And one that's really interesting, is the category they call state employee training. The governor's office writes, To support California's state government workforce and prepare for the next generation of skills needed to thrive in the Gen A.I. agencies will provide trainings for state government workers to use state-approved Gen AI to achieve equitable outcomes and will establish criteria to evaluate

the impact of Gen AI to the state government workforce. Now, one of the things that I think is interesting about this executive order is the extent to which it is focused on preparing the government itself, the actual offices, departments, agencies that compare, comprise California's government, figuring out how to actually use and deploy generative AI themselves. I think there's something very positive about that, in that rather than just viewing this technology in the abstract, they're actually trying to put it into practice. It seems likely to me that actually getting one's hands dirty with these tools is going to end up leading to better policies around them. What's more, California has now created something of a template that other state governments

could use should they want to themselves engage in a similar set of experiments and research. Now, moving on to our next topic, but staying within the theme of governments and AI, Google has announced an updated policy around AI-generated election ads. Bloomberg reports that starting from November, Google will mandate that any election-related ads that feature AI-generated content will be required to have what they call a prominent label and disclosure identifying them as such. Bloomberg writes, advertisers must include prominent language like, this audio was computer-generated, or, this image does not depict real events. Notably, the policy does not apply to minor fixes, such as image resizing or brightening, but in many ways this is one of those just super, super obvious things. It is really hard in general to create mechanisms by which AI generated content can be identified, but platforms who are gatekeepers of content that is distributed on their platforms,

especially paid advertising content, can require this sort of disclosure. And so I think that while Google might be the first to make this policy, relative to the big advertising platforms, it seems likely that others will follow suit. if only because it's likely that they will be forced to at some point in the future if they don't do it voluntarily. Last up today, Time Magazine is doing a big cover story around artificial intelligence. One of the covers is an excerpt from Walter Isaacson's new biography of Elon Musk. That's all about how his opinions on artificial intelligence have evolved and why it started to dominate a lot of his thinking behind the scenes. And the other part of the cover story is a first-time list of the 100 most influential people in artificial intelligence.

There are a lot of familiar faces there, Sam Altman from OpenAI, Eliezer Yudkowski, Jan Lecun from Meta, and many, many more. I think that tomorrow I might do a deeper dive into who is on this list, but for now, an interesting reminder that for whatever summer lull we might have had in terms of the mainstream exposure of artificial intelligence, it seems likely that that lull won't last for a particularly long time. Anyways, friends, that is going to do it for today's AI breakdown brief. I'll be back soon with the main AI breakdown. Before we get into the main AI breakdown, I want to tell you about today's sponsor, Supermanage. If you work in a professional setting, you probably have some version of a one-on-one meeting,

either with the people that work for you or the people that you work with. Unfortunately, all too often, those one-on-one meetings become glorified catch-up calls. Don't you wish you could jump right to the stuff that really matters? That's where Supermanage comes in. Supermanage AI magically distills your team's public Slack channels into a real-time brief on any employee, any time. Catch up on contributions, work in progress, challenges they're facing, sentiment, everything you need to show up ready for a truly meaningful conversation. And it's completely free. Visit supermanage.a.i forward slash breakdown today to start making the most of your

one-on-ones. And thanks again to Supermanage for sponsoring the AI breakdown. Welcome back to the AI breakdown. Today we are talking about one of the biggest questions in the business of artificial intelligence, which is if, when, and how Apple might make their entrance into the race. We've just gotten a new report from the information who have become by far the most significant source of, at least Silicon Valley-based AI scoops of any news publication out there. The highlight of the report is that Apple is apparently spending millions of dollars each day to train a new AI model. But what are they actually going to use it for? Who's the team that's working on it, let's look and see what the reporting has dragged up. So a couple pieces of

information in here. First, the information reports that John Gianandria, who is Apple's head of AI, first authorized the formation of the team to develop conversational AI, i.e. LLMs, around four years ago. The team is called foundational models and is led by Roaming Pang, who is an ex-Gougler who worked with Gianandria when he oversaw Google's AI research arm. The information reports, quote, the team remains small, numbering around 16 people, but the budget for training Apple's most advanced models has grown to millions of dollars per day. Continuing, they write, the foundational models team at Apple plays a similar role to that of AI teams at companies such as Google and Facebook, where researchers produce the AI models and other groups then implement those models into the

company's products. Now, in addition, the information writes, quote, there appear to be at least two other relatively new teams at Apple that develop language or image models. A recent Apple AI research paper and employee profiles on LinkedIn, point to the existence of a visual intelligence team working on software that generates images, videos, or 3D scenes. This is probably not that surprising, given how much emphasis Apple has on augmented and virtual reality, given that their biggest release of this year is the Apple Vision Pro. Now, another team the information writes is working on long-term research involving multimodal AI. But what is Apple actually thinking about where this technology might come to bear in their products? One use case discussed in the piece is an LLM that could

interact with customers who use Apple care, another which is one of the most anticipated AI features from Apple, would be a fairly complete overhaul and upgrade of Siri. The piece writes, The Siri team plans to incorporate language models to let users of the voice assistant automate complex tasks in ways they currently cannot, such as creating and sending a GIF with a simple command. The information says that this effort hadn't been previously reported. In terms of how advanced the technology is, the information writes, people on the Apple team believe its most advanced language model Ajax GPD has capabilities exceeding those of OpenAI's GPT 3.5. Another person with direct knowledge of the model says that Ajax GPT has been trained on more than 200 billion parameters.

One issue, however, is how this would integrate with an on-premise implementation that would be more privacy preserving. As the piece writes, questions linger over how Apple can incorporate LLMs into its products. The company's leaders prefer running software on devices, which improves privacy and performance as opposed to on cloud servers. But, as they point out, an LLM with more than 200 billion parameters couldn't reasonably fit on an iPhone. Still, a lot of the reporting in this piece points to G and Andrea as just super skeptical of the usefulness of chatbots powered by LLMs. Quote, while he has repeatedly expressed skepticism to colleagues about the potential usefulness of chatbots powered by language models, a person familiar with the matter said over

the past year, he has come around to acknowledging the technology's ability to accomplish tasks after seeing a number of internal demonstrations. I don't know, man, it's a little hard from the outside to get a grasp on how much, to me, this muddies the question a little bit of the extent to which Apple not moving more aggressively into the AI space has to do with them having a clear picture of how they want to integrate AI into their existing products versus just kind of missing the boat on the consumer potential of LLMs. The piece also points out how much bleed there is in terms of talent between these big tech companies. They write, after he arrived at Apple in 2018, Gene Andrea helped recruit key engineers and researchers from Google. He also

favored using more of Google's cloud servers, including servers imbued with Google developed AI chips known as tensor processing units to train the machine learning models Apple uses to improve features in Siri and other products. The piece also points out that of the 18 people who have contributed to Axel Learn, which is Apple's internal software to train Ajax GPT, a dozen of them joined Apple within the last two years, and seven of them had previously worked at Google or meta. Now, obviously, Apple is a really cash flush company. So even the fact that they're spending, quote, millions of dollars a day training AI doesn't necessarily mean that some new product launch is imminent. Still, whether it's LLMs or something else, it feels very likely

to me that market pressure is going to get bigger and bigger on Apple to at least articulate what its vision for the AI space is, even if it's something very different than its competitors at Google and Meta. Speaking of Meta, the information also shared a piece a couple days ago called Inside Meta's AI Drama. The information once again writes, many of the scientists and engineers who worked on Lama have quit, embittered by a previously unreported internal battle over computing resources, with another meta-research team working on a rival model that the company ultimately abandoned. Specifically, the information writes, more than half of the 14 authors of the original Lama research paper published in February have since left the company, said Joelle Pinot, the head

of Meta's artificial intelligence research lab, quote, retention and attraction of good talent is probably where I spend most of my time. Of the research scientists and engineers who have left, a number went and founded a startup called Mistral AI that was notable for having raised a $113 million seed round, and others have gone on to join companies like Apple. It sounds like from the story that there were multiple different divisions within the fair or fundamental AI research team based in different locations that were working on different foundation models. In May 2022, one fair team that was based mostly in the U.S. publicly released something called OPT 175B. A few months after that, they started working on an even larger model. At the same time, a different fair team based in Paris had begun working

on a separate LLM that would eventually be dubbed Lama. The model was smaller than OPT, and as the information writes, the team believed a smaller model would be more efficient at inference, the process of generating responses to questions. And when push came to shove, the big tension was around computing resources. Quote, rivalry over access to computing power inflamed tensions between the teams. The Lama team in particular felt overlooked. They received less computing power than the North American-based OPT team, said people with direct knowledge of the situation. Now, apparently, and this makes sense, questions started to grow around why they had two teams working on similar projects that both required what were ultimately pretty finite resources. By February of this year,

leaders at the company had decided to start bringing together members of the competing LLM teams to focus on a single model which would become LMA 2. At that point, the OPT model was abandoned, and it sounds like part of the reason for that was that the team had just churned through personnel. Apparently around half of the 19 authors listed in a May 2022 paper about OPT have subsequently left meta. Now, you might think that given how significant and influential Lama has been this year in shaping how generative AI is developing, that some of these tensions might start to go away. However, according to the information, quote, despite the success, tensions are still shimmering among researchers as meta's attitude to AI research is evolving. Fair has traditionally had a bottom-up

culture led by researchers, with a mission centered on advancing breakthroughs in AI. But as Zuckerberg has become more intent on incorporating AI into meta's apps, Faire's focus has narrowed. It is canceled research that doesn't have a product slant, such as work on protein folding. Anyway, ultimately, just a great example of how much is going to be. on behind the scenes at these big tech companies as they jockey to figure out where they stand and what they can do in the race to shape the artificial intelligence field. A last example of this that we'll look at today comes from Amazon. Yet another piece from the information published on August 30th was called how AWS stumbled in AI giving Microsoft an opening. The piece begins. Long before chat GPT arrived on the

scene last year, Amazon Web Services was developing artificial intelligence software akin to the technology that powers the hit chatbot from OpenAI. AWS had hoped to unveil the software then known inside the company as Bedrock at its annual customer conference late last November, but had to postpone it due to technical snags. That ended up being a fortunate decision. A couple days into the conference, OpenAI released ChatGBTBT immediately wowing the tech world. AWS leaders soon realized Bedrock wasn't on the same level as OpenAI's AI software, but AWS had to do something its leaders felt. AWS was the number one provider of cloud services, while OpenAI had formed a tight relationship with AWS's biggest nemesis, Microsoft. In the following weeks, AWS chucked out its old

plan and attach the bedrock name to a new service that allows developers to connect cloud applications with a variety of LLMs. The change of plans the information writes, details of which haven't been previously reported, illustrates how ChatGPT's sudden surge in popularity caught Amazon off guard. Now heading into the fall, a lot of the focus is turning to Google and the extreme expectations that they're pushing around their Gemini model, which, based on the amount of compute alone, is being positioned as one of the first serious competitors for OpenAI's GPT4. This battle is quite obviously going to shape all of the tools that you and I use and dictate a lot about how the industry evolves, so it is one that we will keep our eyes on closely. Will Apple actually jump more fully into

this race? Only time will tell, but it certainly sounds like they're starting to spend the money that they might need to if they're going to get in the game. That is going to do it for today's AI breakdown. I appreciate you listening or watching as always. If you are getting value from this content, I would love it if you would go leave a rating or a review. It makes a big difference and I appreciate each and every one. Until next time, peace.