The AI Daily Brief: Artificial Intelligence News and Analysis - The State of AI In 13 Stats

Episode Date: April 17, 2024

Today's episode explores the state of artificial intelligence through Stanford University's AI Index, which distills key insights into 13 charts. The report highlights the growth of open-source AI models, the performance gap between open and closed-source models, and the industry's dominance in AI development. It also discusses the soaring costs of training high-performance AI models and the significant U.S. lead in AI development over other countries. This summary provides a snapshot of current trends and developments in the AI landscape, reflecting both the technological advancements and the economic dynamics shaping the future of AI. ** CHECK OUT THE JUST-LAUNCHED SUPERINTELLIGENT PLATFORM - 300+ AI video tutorials https://besuper.ai/ Consensus 2024 is happening May 29-31 in Austin, Texas. This year marks the tenth annual Consensus, making it the largest and longest-running event dedicated to all sides of crypto, blockchain and Web3. Use code AIBREAKDOWN to get 15% off your pass at https://go.coindesk.com/43SWugo  ** ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, the state of AI in 13 charts. Before that on the brief, Mistral is raising at a $5 billion valuation just months after raising at a $2 billion valuation. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown. Network for more information about our Discord, our newsletter, and our YouTube channel. Welcome back to the AI breakdown brief, all the AI headline news you need in around five minutes. Today we kick off with a fundraising story and exclusive from the information. France's Mistral, which is, of course, an open source darling, and one of the hottest LLM builders, is apparently in talks to raise fresh capital at a $5 billion valuation,

just a few months after closing money at a $2 billion valuation. Mistral closed on $415 million at that previous valuation back in December. So why might there be appetite for this next fundraise at a higher valuation? I think there are a couple pieces of this. First of all, in the subsequent months, Mistrel inked, big deal with Microsoft. That included releasing their first non-open source model through Microsoft's Azure, which could easily have suggested to investors that the company had a pathway to real revenue. More relevant, though, in my estimation, there are only a tiny handful of companies

that have any credibility as a plausible leader in the frontier and foundation model space. You've got OpenAI, Anthropic, Google, meta, grok, and mistral. Maybe you could nudge it open to data bricks and cohere, but the point is that this is a very, very, very, very, small group. What's more, it's a group that has gotten smaller. Inflection, which raised more than $1 billion last year to get the compute to actually compete in this area, still ultimately decided that it was too big a mountain to climb and ended up mostly going to Microsoft with that recent announcement. I believe that investors have a broad sense that the value for winning even a piece of this market is so tremendously huge that the valuation for credible competitors

in the space, that very small handful of credible competitors, is just not going to be the barrier. That doesn't mean that every investor is willing to sign up at any valuation. I know several investors, for example, who have blocked at the valuation of other companies on that list. But I think by and large, what this reflects is the sense of just how significant the pie is for winners in the LLM space. And despite being only a year old, Mistral is a serious contender. Now, in addition to that unconfirmed funding news, we also got more model updates from Mistral, who uploaded their 8X-22b model, as well as a brand-new instruct 8X-22b with function calling on hugging face. The model is fluent in English, French, Italian, German, and Spanish.

It's a 141 billion parameter model. It has a 64,000 token context window, and it's released under the same Apache 2.0 license that they've released previous models on, which includes, of course, an okay for commercial use. Mistral also dropped a chart showing the relationship between performance and cost, with three of their models, Mistral 7B, Mixtral 8x7B, and Mixstrel 8x22B, all being at the very best end of the performance and cost ratio. Bindu Ready of Abacus writes, apparently the new Mistral model beats Claude Sonna and it is a tad bit worse than GPT4. In a couple of months, the open source community will fine-tune it to beat GPT4. This is a fully open-wates model with an Apache 2 license.

I can't believe how quickly the OSS community has caught up. And if you're wondering why Mistral would be pushing hard right now, one of the reasons might be that, whether it's this week or very soon, meta's first versions of Lama 3 are soon to arrive. Mr.'s big competition in the open-source LLM space when it comes to developer mindshare is, of course, the Meta Lama models. So to me, overall, not that surprising to see a big fundraise, although going from a $2 billion to a $5 billion valuation over the course of four months by any previous industry standards would be slightly insane.

Next up, another big theme right now is the move to try to get models to work on local devices and local hardware. This is, of course, Apple's great pursuit as it tries to bring AI models to the iPhone and its other suite of devices, but it's happening everywhere. You've heard lots and lots of companies talking about the AIPC era, for example. AMD has this week unveiled a new set of chips that are specifically designed for these AI PCs and will be included in devices from HP and Lenovo. In making these chips for the AIPC market, AMD joins both Intel and InVideo who are also looking at the space. Another random little area in which AI is finding its way into hardware, Logitech is now shipping a mouse

that has a dedicated AI button. The Verge writes, tomorrow's AI PCs may not only have a copilot key on their keyboards, Logitech is introducing its own way to summon chat chip ET2. It's called the Logi AI Prompt Builder, and it'll use a dedicated button on your mouse or keyboard. What's more, it appears that this prompt builder doesn't just pull up a chatbot when you press the button,

but also offers an interface to actually improve prompts. So are AI buttons the next big thing in devices? I suppose we will have to wait and see. Finally, today, as every company tries, to figure out how to participate in the AI revolution without supporting or encouraging misinformation, companies are trying to figure out what their solution to letting people know that something was generated with AI. Basically, every big social network is thinking about how to do this on their platforms, and Snap is reportedly planning to add a watermark to AI images that are created with

Snap tools. Interestingly, this will not be an invisible watermark that can be picked up by a specific reader, as we've seen the approach being from some others, but instead will be a visible mark, a translucent version of the Snap logo with a white outline. The company says that removing the watermark will be a violation of their terms of service. The question, of course, will be whether that means they've just incentivized people to go to Mid Journey or Dolly or somewhere else to create an image that doesn't have a watermark. Right now, all these companies are still taking baby steps in this space because there's no regulatory pressure so far to do anything more. How much longer that stays the case remains to be seen. For now, though, that is going to do it for today's AI breakdown brief.

Next up, the main AI breakdown. Hello, friends. Quick note before we get back to the show, I'm so excited to share that Super Intelligent is now live. Super Intelligent is a platform for fast, fun, and super practical, useful AI learning. We have something like 300 video tutorials adding 30 to 50 each week, covering every topic in AI you can imagine from LLMs to image generators to case studies, use cases, basically everything that tells you how to use AI and what to use it on. In short, fast four to seven minute tutorial videos, which are paired with step-by-step instructions that help you actually use these tools as well. It's $20 a month for unlimited access, and I would love to see you

there. Check it out at B-super.a.I. That's B-super.aI. Attention, AI breakdown listeners. Consensus 2024 marks the 10th gathering for all things crypto, blockchain, and Web 3. However, importantly, this year's agenda will also dive deep into AI-driven transformation. And the speaker lineup includes the leading minds and innovators at the forefront of this digital renaissance. Don't miss the Consensus AI summit to cut through the hype to find where true transformation and opportunity lie. Listeners to this show can get 15% off registration with the code AI breakdown. Visit Consensus 24.com to learn more. Some of the folks will be at Consensus this year include Guillaume Verdun, aka Beth Jzos, founder and CEO of XTropic,

as well as spiritual leader of the Accelerationist Movement, Neil Stephenson, co-founder of LAMNAILEEN-IKE, the CEO of Brave Software. Again, go to Consensus24.coindex.com to learn more and get 15% off registration with the code AI breakdown. Welcome back to the AI breakdown. As you probably have garnered at this point, I am a sucker for a big think piece, especially if it has some numbers or data or anything that we can comment and react to. So today for our episode, we are looking at a recent piece by the Stanford University Human

Center at Artificial Intelligence Center called the AI Index, the state of AI in 13 charts. This is a sum up of the Institute's larger AI index, which is a 300 plus page report, and I think does a good job of giving the high levels. So what we're going to do is we're going to look at the charts, discuss Stanford's conclusions, and then I'll share if there's anything that I think differently around or just see it in any sort of different way. Their first note, they call a move towards open source. In 2023, they counted 98 open foundation models, 23 limited foundation models, and 28 with no access. They noted that of the 149 foundation models released in 2023, which was itself double the number released in 2022,

65.7 were open source that compared to only 44.4% in 2022 and 33.3% in 2021. If you've listened to this show at all, I mean, heck, if you listen to the mistral section in the brief today, this will probably not surprise you. There has been a huge push towards open source. LLMs, with companies like meta initially leading the charge, and in the process, surprising Google and OpenAI, I will say, and then the banner being picked up by others like Mistral as well. However, Stanford's second chart notes that Open comes at a cost of performance. They write, closed source models still outperform their open source counterparts. On 10 selected benchmarks, closed models achieved a median performance advantage of 24.2%,

with differences ranging from as little as 4% on mathematical tasks to as much as 317.7% on agentic tasks. This is a chart that I really don't like. It's not that I disagree that there is still a difference between closed and open models, but I think that this is an area where medians and averages really fall apart. Basically, who cares about not state-of-the-art models? Certainly no one that I know that's actually working with them unless they're making a choice for specific cost reasons. So a better comparison would be, what's the difference between the most performant open-source models and the most performant closed models? In other words, what is the gap between open source and GPT4 class performance?

I think the overall result would still show that gap, but it might be less dramatic than it seems here. Next, Stanford notes that, quote, industry dominates AI, especially in building and releasing foundation models. They point out that since 2019, Google has led in releasing foundation models with a total of 40, followed by OpenAI with 20, and academia far behind, with UC Berkeley releasing three and Stanford releasing two. Once again, no big surprise here, given how much money is at stake with these models.

Putting a fine point on that, they note that industry released 108 models last year as compared to academia's 28, industry academia collaborations 9, and government's 4. Next, Stanford notes that the prices for training models has gone up significantly. They write, one of the reasons academia and government have been edged out of the AI race, the exponential increase in cost of training these models. They note that Google's Gemini Ultra cost $191 million worth of compute to train, up from OpenAIs GPD4, which costs an estimated 7.5%. million. In comparison, they point to the original transformer model from 2017, which cost around $900 to train. The next one might be a little surprising given how much we talk about geopolitical competition around AI, but here's how Stanford sums it up. What AI race? They write, at least in terms of notable machine learning models, the United States vastly outpaced

other countries in 2023, developing a total of 61 models that compared to China's 15, France's 8, Germany's 5, and Canada's 4. I do think that these numbers are telling and important. However, I think that what's more of interest to people who are looking at the race dimension of this is less the number of models and more, once again, the state of the art and the performance of those models. In other words, to people who care about this race, it wouldn't matter if the U.S. released 100 times more models than China if China released the best models. Next up, they talk about how much more performant these models have gotten. They call the chart move over human. They note that when it comes to image classification, basic level reading

comprehension, English language understanding, visual reasoning, and multitask language understanding, AI has in general exceeded human performance at this point, and competition-level mathematics is getting very close. Of course, this brings up the question of what superintelligence in AGI actually mean, which is, of course, a debate that you see constantly on AI Twitter. Here's another one that might surprise you, given how often we talk about the big money going into AI on this show. Overall, Stanford notes that total AI private investment has actually gone down between 2021 and 2022. and again between 2022 and 2023. Specifically, 2021 saw $132.36 billion invested in AI down to $95.99 billion in 2023. However, generative AI has seen a massive surge, going from $4.17 billion in 2021 to $2.85 billion in

2022 to a 10x increase to $25.23 billion in 2023. So clearly the emphasis in what in AI is being invested in has made a big shift, commensurate with the change and the focus on chat GPT, mid-journey, and the like. Once again, the United States is seeing the biggest portion of that investment. In 2023, it saw 67.22% of investment. Next, almost 60 percentage points behind was China, which saw 7.76% of investment, the UK, which saw 3.78%, and then a set of companies including Germany, Sweden, France, Canada, Israel, South Korea, and India, which all saw between 1 and 2% of total private investment. Here's one that's particularly interesting to us as we are building super intelligent, a platform for practical learning around AI. Based on a McKinsey and Company survey, Stanford reported on how businesses

are using AI currently. 26% are using it for contact center automation, 23% are using it for personalization, 22% for customer acquisition, 22% for AI-based enhancement of products, and 19% for creation of new AI-based products. Surveys also saw an increase in organizations that said they were AI in 2023, reaching 55%, which was up from 50% in 2022. What about concerns around AI impacting jobs? Overall, Stanford writes that globally most people expect AI to change their jobs, with more than a third expecting AI to replace them. Gen Z and millennials are anticipating more substantial effects, with 66% of Gen Z compared

to 46% of boomers, believing AI will significantly affect their current jobs. Individuals that have higher incomes, more education, and decision-making rules also think that AI will have a bigger impact on their employment. The numbers they quote, 57% believe that AI will change how they do their current job in the next five years, and 36% see AI replacing their current job in the next five years. Stanford also looks at different countries' attitudes towards AI, and specifically where people were worried about it. At the top end of the concern is Australia with 69% of people saying that AI makes them nervous, while at the bottom end of the spectrum is Japan, who only had 23% worried. Lastly, they looked at the change in regulation. In the

United States, the number of AI-related regulations jumped from around 15 to 25 between 2022 and 2023, although of course none of those are any sort of comprehensive regulation, which remains the big question going forward. Overall, I think a lot of this probably reflects what you might assume if you're paying a close attention to this space, but it's still really interesting to see this data captured in this sort of way. So big thanks to Stanford University's Institute for Human-centered Artificial Intelligence for publishing this report, and for now, that's going to do it for the AI breakdown. time, peace.