The AI Daily Brief: Artificial Intelligence News and Analysis - 2X Bigger than GPT-4!? Amazon Is Training "Olympus"

Episode Date: November 9, 2023

Sources are suggesting that Amazon is training a 2 trillion parameter model called Olympus. NLW looks at what it means for Google Gemini, OpenAI GPT-5 and more. Today's Sponsors: Listen to the chart-topping podcast 'web3 with a16z crypto' wherever you get your podcasts or here: https://link.chtbl.com/xz5kFVEK?sid=AIBreakdown  Interested in the opportunity mentioned in today's show? [e-mail] ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, Amazon is apparently training a new model called Olympus that has twice the parameters of GPT4. Before that on the brief, the Screen Actors Guildstrike is finally over after coming to agreement on the most thorny issue, which was, of course, AI. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.network for more information about our Discord, our YouTube channel, and our newsletter. Welcome back to the AI breakdown brief. All the AI headline news you need in around five minutes. Late last night, SAG AFRA announced that they were finally, after 118 days, the longest strike in Hollywood history, coming to a deal with the studios to end the strike. Now, there were a bunch of more Hollywood things about this. This strike was never just about artificial intelligence. In their announcement, the Screen Actors Guild said that they were able to secure a contract valued at over a billion dollars, that they were able to.

able to negotiate, quote, above pattern compensation increases. But the most interesting one for our purposes at least was the announcement that they had secured, quote, unprecedented provisions for consent and compensation that will protect members from the threat of AI. Now, as Wired points out, while the strike was initially about a lot of different things, quote, as this year's negotiations between SAG and AMPTP dragged on, generative AI became the major sticking point. Back in July, studios claimed they offered a, quote, groundbreaking AI proposal that protects actors digital likenesses. SAG countered that the proposal stipulated background performers could be scanned, paid for the day, and then turned into digital characters that studios could use for the rest of

eternity. AMPTP disputed this. Continuing, the issue was volleyed back and forth until last weekend when SAG reviewed the studio's last best and final offer and rejected it, claiming, there are several essential items on which we still do not have an agreement, including AI. A follow-up story in the Hollywood reporter revealed that the AMPTP proposal sought to allow studios to pay for AI scans of what are known as Schedule F performers, and following the actor's death, allow studios to use the scans without the consent of the estate or SAG. Schedule F performers include anyone who makes more than the minimum rate for TV series regulars or feature films. The Guild wanted compensation for reuse of the scans along with consent. So basically what you're talking about here is that reportedly

Hollywood wanted to be able to scan actors, working actors, just once, and use those scans for perpetuity. Meanwhile, SAG was arguing for one compensation when those scans were used. and two, for the actors who scans were being used to have to consent. Now, as a total outsider, it's hard not to be on the side of the people who want to be able to say when their likeness gets to be used or not and want to be paid for that, but as of yet, we haven't gotten the specific final terms of the deal. As I've always said, what makes this story notable is not just its Hollywood context, but about the way that I think it presages many, many of these sort of organized labor battles that are going to be fought around artificial intelligence. My expectation,

my base case, if you will, is that organized labor is going to get significantly more powerful and be seen as significantly more important because specifically of the rise of AI. Now, moving on to a very different topic, one of the big themes in AI policy this year has, of course, been the United States desire to cut off China's access to advanced AI chips. The first set of export restrictions went into place at the end of last year, but there were still some pretty meaningful loopholes. For example, if a Chinese company had subsidiaries in other countries, those subsidiaries weren't necessarily subject to the same restrictions that the China-based corporate leader was. Now, just a couple weeks ago, we got updated rules from the Commerce Department that represent

an attempt to close some of those loopholes. And we've also heard that the U.S. has also behind the scenes been pressuring companies like Nvidia and AMD to stop or slow down shipments of chips to certain areas of the world, particularly the Middle East, from which they think they might get sent on to those Chinese companies. Yesterday we discussed about how Baidu has been shifting its purchases away from Nvidia and two local suppliers. But now the Financial Times is reporting that Nvidia is developing another power-moderated chip that would come in under U.S. mandated power restrictions.

In fact, Nvidia has actually developed three new chips focused on China. The H20, the L20, and the L2. Information about the chips was shared in a document to prospective customers. Road chip consultancy semi-analysis. Invidia is perfectly straddling the line on peak performance and performance density with these new chips to get them through the new U.S. regulations. Now, this is the second time Nvidia has had to update or reconfigure chips for the Chinese market. And the question, of course, will be whether the U.S. actually allows these sales to go through.

Moving over to an interesting topic, which we actually haven't had much of a chance to talk about here on the AI breakdown, but which I expect will come up more and more. And that is the overlap of AI and investing. In an interview with Reuters, the CEO of Norway's sovereign wealth fund said that they're using AI to help manage their investments. Now, the reason that this is notable is that Norway's sovereign wealth fund manages $1.4 trillion. It invests the state's revenues from oil and gas production and is the largest sovereign wealth fund in the world. It holds stakes in more than 9,200 companies globally and owns 1.5% of all listed stocks. Said CEO, Nikolai Tangan, we are using AI now and how we deploy the capital. We use it to

reduce the trading we do because we are a near index fund and sometimes we have to make adjustments to the portfolio and these types of models can help us trade less. Tangan went on to say that he had set the fund an internal target to boost productivity by 10% over the next 12 months using AI, and it even had a conversation about how to do that with Sam Altman, who is, of course, the CEO of OpenAI. Tangan said that when he asked Altman if he thought that they could increase their productivity by 10%, Altman said, I think 20%, you can do. I have to assume that many other investors, including sovereign wealth funds and other private market actors, are going to be watching this experiment closely. Over in the AI safety world, scale AI has launched a new AI safety lab called the Safety

Evaluations and Analysis Lab or SEAL. The goal of the unit is to build a suite of products for scale AI's existing customers that can be used to help evaluate the possible risks of AI deployments, as well as implement red team methods to help find faults and biases in software before it's launched. The new SEAL lab is being read by a former Google researcher who worked on Google Bard. Now, it seems like part of the motivation isn't just an inborn desire to see responsible AI development and deployment, although I'm sure that's a big part of it. It also seems to be targeted based on their announcement blog post at helping companies get out ahead of what are expected to be more strict compliance needs in a world of Biden executive orders on

AI and AI safety summits in the UK. In the blog post announcing seal, they wrote, every enterprise and government deploying an LLM application is facing the need to adopt and comply with the forthcoming standards and regulations that will be put in place. Now, on that front scale was one of the companies that voluntarily joined the White House Responsible Development commitments back in September, said CEO Alexander Wang, to advance our knowledge and understanding of LLMs, and therefore advance their capabilities, we must understand their weaknesses as much as their strengths. It is imperative for the tech industry to take proactive measures like these, given the rapid

pace of technological development. Over in the world of AI products, a ton of buzz today is about Humane's AI pin. Now, we've seen little premieres of this device. There was a widely shared TED talk from earlier in the year, as well as an appearance at a fashion show recently. But the official launch event for the PIN is coming later today. And in advance, it's basically seeming like this thing is a phone without the screen. The device is reported to cost $699 and come with a $24 a month subscription. We won't get too deep into that now because I anticipate that it will be a big part of the show tomorrow, but something to keep an eye on for sure. Now, the biggest news today is undoubtedly reports that Amazon is worth. working on a model they're calling Olympus that reportedly has twice the parameters of GPT4, but if you want to hear about that, you will have to come back for the main episode. That's coming up next. And now a word from today's sponsor. Are you interested in how two top-of-mind trends AI and crypto can work together?

If so, I have the perfect podcast recommendation for you. Web3 with A16Crypto, the chart-topping show brought to you by venture firm Andresen Horowitz. Web 3 with A16Z Crypto is your definitive resource for the future of the internet. Whether you're already building in these spaces or simply curious about what's next. If you need a place to start, they recently released an excellent episode with Stanford Cryptography Professor Dan Bonay and former Google Xer Aliya in conversation with host Sonal Choxi about the intersection of AI and crypto. From fighting deep fakes and proving humanity to large language models like Chatchip-T,

they cover it all. I highly recommend checking it out, especially if you'd like to learn more about how AI and crypto will impact our everyday lives. Beyond crypto and AI, this show is for creators seeking more ways to truly own their work, for business leaders trying to prepare for the future today, and for innovators exploring trending tech topics. So go ahead, listen to Web3 with A16Z Crypto, wherever you get your podcasts. Welcome back to the AI breakdown. Today we are talking about the latest news and reports out of Amazon, which is, as I said in the intro, that they are working on a model code-named Olympus that is rumored to be about twice as big in terms of parameter size as GPT-4.

So today we're going to talk about the news, as well as the larger context of competition between the big players when it comes to Foundation and Frontier models. So Reuter sourcing has two people familiar with the matter, saying that Amazon's new codenamed Olympus model has two trillion parameters. Again, that's about twice what OpenAI's GPT4 model is rumored to have. According to Reuter sources, the team inside Amazon is headed by Rohit Prasad, who's the former head of Alexa, and who is reporting directly to Amazon CEO Andy Jassy. Prasad's title is head scientist of artificial general intelligence and has been able to pull in people

from the Alexa AI and Amazon Science team to work in a unified effort. Now, when it comes to the business motivation of this, despite the fact that Amazon already has bedrock, which is their platform through which they help AWS customers take advantage of or customize existing models, including things like Anthropic, as well as open source models, they believe that having their own unique model within the larger AWS ecosystem could be a desirable offering for their existing customers. Now, to get a little bit of a sense about the larger Amazon context, let's go to a piece from the information from August called how AWS stumbled in AI giving Microsoft an opening. What the story reports is that even before ChatGPT had been announced,

AWS had been developing a similar type of LLM. They had wanted to debut the that tool, which they were referring to inside the company as Bedrock at the time, at their annual customer conference in November, however at the last moment they had to postpone it due to, as the information puts it, technical snags. Now, in many ways that ended up being fortuitous, because even as that event was happening, OpenAI released ChatGBT, which, as we know, went extremely viral and became the fastest growing software at the time in history. It didn't take Amazon long to recognize that the Bedrock model that they had been working on wasn't even close to the model powering open AIs chat GPT, but at the same time, they didn't want to do nothing.

In the scramble that ensued, Amazon threw away their old plan of leading with a model-first strategy and instead came up with the approach to build a service which would allow developers to connect with various different large language models, taking an approach where effectively they were arguing from a business standpoint there wasn't going to be a winner take-all situation, and that especially enterprise customers were going to prioritize having choice and customization options to find the right models for their particular businesses. Subsequent to that, they attached that bedrock name that had been floating around for their model to that new service and started promoting it. Now, Amazon did include some of its own models in that bedrock environment, which they referred to as

Titan, but the review of those models has been extremely mixed, say the least. Now, perhaps because of the reputation of Titan, or just the general sense of how far ahead other companies like OpenAI are, I would say that there's a fair bit of skepticism around the potential for a launch of Olympus. The information summed it up in a section of an article called The Mountain that Olympus must climb. They write, making Olympus successful could be a challenge for Amazon, at least based on the reception accorded Titan. The launch of Titan was delayed last year because it didn't even hold a tiny candle to OpenAI's chat GPT, and since making one of the Titan models generally available to AWS customers, the feedback about it has only been so-so. They also pointed to a piece from

from Hugging Face technical lead Phil Schmidt, who, as they put it, after side-by-side tests involving Titan and other models concluded that Titan is significantly worse than available open source models as well as software from OpenAI. Now, in terms of when we might get more information about Olympus, it seems like one potential is that it could come at ReInvent, which is Amazon's annual company conference, which is happening in Las Vegas at the end of the month. As the information points out, if they don't debut it there, it might make customers in the press think it's even further behind OpenAI. Now, at the same time, Amazon is one of the very few companies who has the resources to actually potentially compete on size for these models, and that's certainly where a lot

of the attention has been focused if you look at the chatter on Twitter. John Paris writes size matters when it comes to training AI, and points to a piece from the bite, Amazon reportedly training AI with twice as many parameters as GPT4. The AI wars have become a parameter measuring contest. However, as meta-chief AI scientist Jan Lacoon pointed out, back in September, quote, a model with more parameters is not necessarily better. It's generally more expensive to run and requires more RAM than a single GPU card can have. Jan also points out that when it comes to GPT4, it's rumored to be a mixture of experts, i.e. a neural net consisting of multiple specialized modules, only one of which is run on any particular prompt. So the effective number of parameters used at any one time is smaller than the

total number. Now, another line of conversation on X is around the fact that this is an inevitable move. That's the way that Boris here puts it. He says, I think it's inevitable for them to start competing and they also need a state-of-the-art model to power Alexa. Additionally, having custom GBT's for every book, author, and topic in the bookstore sounds like a no-brainer. Now, a third thing that people are talking about or asking about is what this suggests of its investment in Anthropic. Vizal Sharif asks, is that just a hedge? And I think that there are a couple things going on here. I don't think that Amazon is totally full of it when they've been talking about the idea that there are going to be multiple winners in this market and that a service like Bedrock, which gives enterprise customer's choice, is likely going to be a

valuable part of the equation. Indeed, even despite their huge investment in OpenAI, we've sort of seen Microsoft shift to a similar strategy, for example, in the form of their partnership with Databricks, which gives companies access to more models than just OpenAI models. Now, the other thing about their Anthropic investment is that they're not the only big tech player in that company. Just a few weeks after they announced their partnership, it also came out that Google would be investing a couple billion dollars in Anthropic as well. That obviously suggests a much more blended affiliation and a much less clear business relationship than, for example, OpenAI has with Microsoft. However, for many, the biggest conversation that this brings up is actually a question,

summed up best by Playground AI founder Sohail, who says, where is Google? Now, you might remember that back in September, the information in other sources suggested that Google was nearing the release of their much-anticipated Gemini model. The news back then was that they had given a small group of companies access to an early version of Gemini, and that in general, when it came to Google, giving outside developers access meant that Google was getting close to incorporating it into its other services. Now, Gemini matters for two different reasons. For one, many think that it might be the first model to actually exceed the capacities of GPT4,

which has been the benchmark for the industry ever since it came out. I've talked before in previous episodes about what a significant moment I think that will be for the industry, and really is signaling that this first phase of generative AI post- ChatGPT coming out last November, has come to a close and something new has begun. However, it's not just significant for the industry in general as a waymarker of how it's evolving. It's also really significant for Google. I think broadly speaking, there has been a lot of surprise at how far behind Google has appeared ever since ChatGPT came onto the scene. Frankly, the longer the Gemini goes unreleased, the higher the stakes are for that company. Nathan Landz writes,

What happened to Google? They seem to have completely dropped the ball. There was a lot of hype about Gemini than nothing. Meanwhile, OpenAI hit it out of the park and XAI proved they have a decent product and are iterating incredibly fast. Invidias Dr. Jim Fan writes, expectation for Google Gemini is now ridiculously high. Gemini has to check off at least one of the following, 120% IQ of textual GPT4 or 100% of GPT4 but at half the cost or 2x speed of turbo, or 100% of visual GPT4 or natively support long videos and ship the API in Q1 of 2020. Now, notably, even that 2024 number is a delay. Kevin Zhu pointed to a recent interview with Google DeepMind CEO Demas Sassabas

and writes, Google Gemini's delay to 2024 reiterated. Reorg of Google Brain and DeepMind is taking much longer to work through than the market expected. Then perhaps somewhat snarkily, but also reflecting a lot of people's perspectives, he writes, having the team's CEO go on regulatory capture missions rather than shipping product doesn't help. The Macrosift.eft.eith writes, Google Gemini still in training and fine-tuning, suggesting 2024 launch. Said Demis Hesaba's CEO of Google DeepMind in an interview with CNBC, we'll see about a 2024 launch.

Now, he points to one prediction market that has 49% betting that Gemini comes out this quarter, with another 40% betting that it comes out in Q1. Metaculous markets have the betting at February 3rd, 2024, with a lot of concentration across the Q1 dates. And even with these delays, there are also, as Dr. Jim pointed out, pretty high expectations of how this model will actually perform. Metaculous has another poll, what MMLU benchmark score will Google DeepMind's Gemini model have on release? By way of explanation, the background info writes, MMLU, massive multitask language understanding, is a test to measure a text

model's general knowledge and intelligence. The test covers 57 subjects, which range in difficulty from elementary level to advanced professional level, and it tests both informational knowledge and problem-solving ability. Now, Metaculus expects the benchmark score of 90.13, which would put Gemini ahead of GPT4. GPT4, for example, comes in an 86.5. Now, Brian Romley also talked about where Google's Gemini might try to position itself. He writes, With the release of GROC and GROC in GPT4 Turbo,

Google's Gemini will try to place itself in the middle. The focus will be to solve hallucinations by using the search graph that has high weight to the mixture of experts output. Now, what Brian is getting at is that in addition to the known competitors, like Google and like meta, Amazon and their forthcoming Olympus also have to deal with Elon's XAI and their new GROC model, along with OpenAI and whatever comes out as GPD 4.5 or GPD5. One of the critiques that I saw of OpenAI's dev day was around their prioritization,

with some like AI entrepreneur Bindu Ready, suggesting that they really should have focused on these more advanced models. Bindu wrote, rumors are they are training Gobi, a multimodal model, GPT5, that will be way superior to GPT4. It's not clear when Gobi will be available, but I suspect it won't be until. next year. For what it's worth, Metaculous is betting that OpenAI will announce GPT5 in about 11 months on October 5th, 2024. What's clear is that the speed of innovation is increasing. This has been particularly on display recently with the launch of GROC. Suhail again writes, it's interesting that it only

takes four months now to train an LLM to GPT3.5 Lama 2 level from scratch. Prior to January this year, nobody had practically replicated GPT3 still. Doesn't seem like the lead of GPT4 will last too much longer. And so, as the information put it, Olympus has a heck of a mountain to climb. But at the end of the day, it's not particularly surprising that they're going to take this on. And so really the only question is, when are they going to officially announce it? Come join us on the Breakers Discord if you want to discuss that or anything else. But for now, we'll wrap there. I appreciate you listening or watching as always. Until next time, peace.