The AI Daily Brief: Artificial Intelligence News and Analysis - Goldman Sachs Is Wrong About AI (Why AI Isn't A Bubble)

Episode Date: July 10, 2024

Recently, Goldman Sachs made waves in the AI space with their report "Gen AI: too much spend, too little benefit?" In this extra long episode, NLW explains what he sees as problems with the arguments of the skeptics featured in the piece, as well as why the report actually isn't nearly as negative as people are presenting it. Read the report: https://www.goldmansachs.com/intelligence/pages/gen-ai-too-much-spend-too-little-benefit.html Concerned about being spied on? Tired of censored responses? AI Daily Brief listeners receive a 20% discount on Venice Pro. Visit ⁠https://venice.ai/nlw⁠ and enter the discount code NLWDAILYBRIEF. Learn how to use AI with the world's biggest library of fun and useful tutorials: https://besuper.ai/ Use code 'podcast' for 50% off your first month. The AI Daily Brief helps you understand the most important news and discussions in AI. Subscribe to the podcast version of The AI Daily Brief wherever you listen: https://pod.link/1680633614 Subscribe to the newsletter: https://aidailybrief.beehiiv.com/ Join our Discord: https://bit.ly/aibreakdown

Today we were discussing why the Goldman Sachs report is wrong, why all the analysis around it calling BS on AI is also wrong, and why ultimately AI is not in a bubble. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes. Hello, friends, quick note before we dive into today's episode, this Goldman Sachs report has been absolutely dominating the conversation in the AI space. And frankly, it's been driving me nuts. And so today, instead of our normal episode, I am going to go very deep into, as I said, why it doesn't say exactly what people say it says and why AI is not in a bubble. Because of that, there will be no headlines in main episode division. This is just one big long op-ed. I hope you enjoy it. Welcome back to the AI Daily Brief. If you've been in and around AI conversational circles for the last week or so, whether it's on X or somewhere else, you've probably heard people squawking about this new Goldman's.

Sacks report. A representative example comes here from Ed Zittron who writes newsletter. Goldman Sacks is called BS on generative AI, and I believe that it's time that everybody follows suit. Generative AI is unreliable, unsustainable, requires an entire rebuild of America's power grid, and is most decidedly not the future. Now, it's worth noting that Ed's business model is bullshit and outrage, but he's far from the only one saying things like this. Roger McNamee also invokes the Sequoia report we talked about yesterday, saying, when firms like Sequoia and Goldman Sachs tell me there is a bubble I pay attention. When their argument is based narrowly on discretionary CAPX alarm bells go off.

Why? Because there are lots of other reasons to think generative AI is going to disappoint. On the flip side, there are people like Roheed here who says, I know it's become fashionable to go Gen AI as a bubble and where's the revenue with Goldman Sachs and Sequoia and more. But like, GBT4 came out barely a year ago and we've barely unwrapped the new data centers, so can y'all just give it a minute? His point isn't wrong, but I think there is a lot more here to get into. And so, of course, that is what we are going to do today. We are going to talk about why AI is not a bubble,

why these reports are not even saying what people think they are. If you haven't listened to or watched yesterday's episode, go check it out. In some ways, it's a part one of this, focusing on the Sequoia blog post from last week called AI $600 billion question. Now, at the center of this conversation is the Goldman Sachs Global Macro Research Report, issue 129 from June 25th, titled,

Gen AI, Too Much Spend, Too Little Benefit, question mark. It starts off, tech giants and beyond are set to spend over $1 trillion on AI CapEx in coming years, with so far little to show for it. Will this large spend ever pay off? They also lead with a couple of big pull quotes. Professor Darren Osamoglu is quoted as saying, given the focus and architecture of generative AI technology today, truly transformative changes won't happen quickly and few, if any, will likely occur within the next 10 years. Jim Covello, the head of global equity research at Goldman Sachs, writes, AI technology is exceptionally expensive, and to justify those costs,

the technology must be able to solve complex problems, which it isn't designed to do. And indeed, this is largely the tone of the executive summary. The promise of generative AI, they write, to transform companies, industries, and societies continues to be touted. Leading tech giants, other companies, and utilities spend an estimated $1 trillion on CAPEX in coming years, including significant investments in data centers, chips, other AI infrastructure, and the power grid, but this spending has little to show for it so far beyond reports of efficiency gains among developers. And even the stock of the company reaping the most benefits to date, Nvidia has sharply corrected. As evidence, they point to

Darren Asimoglu, once again a professor at MIT, who they summarize his point to saying, he estimates that only a quarter of AI-exposed tasks will be cost-effective to automate within the next 10 years, implying that AI will impact less than 5% of all tasks. He also questions whether AI adoption will create new tasks and products, saying these impacts are, quote, not a law of nature. So he forecasts AI will increase U.S. productivity by only 0.5%

and GDP growth by only 0.9% cumulatively over the next decade. GS head of global equity research Jim Covello goes a step further, arguing that to earn an adequate return on the $1 trillion estimated cost of developing and running AI technology, it must be able to solve complex problems, which he says it isn't built to do. He points out that truly life-changing inventions like the Internet enable low-cost solutions to disrupt high-cost solutions even in its infancy, unlike costly AI tech today. And he's skeptical that AI's cost will ever decline enough to make automating a large share of tasks affordable given the high starting point, as well as the complexity of building critical inputs like GPU chips, which may prevent competition. He's also doubtful that AI will boost the valuation of companies that

use the tech as any efficiency gains would likely be competed away, and the path to actually boosting revenues is unclear in his view. And he questions whether models trained on historical data will ever be able to replicate humans' most valuable capabilities. Now, if you were just looking at the reporting and the discourse around this report, that is all you will have heard. Ignore the fact that there are a number of other experts quoted in this who come to a very different conclusion. But before we get into them, let's actually try to take on some of these counter arguments. For this, I used Gamma, one of these useless generative AI tools, to take notes that I had put into Notion and create this presentation slash website in literally five seconds.

Let's talk about point one, which is the cost ROI question. As I mentioned yesterday in our discussion of the Sequoia blog post, I believe that the popular discourse around this question is actually conflating two very separate discussions. One is the return on investment for the companies that are building out massive AI infrastructure, the Microsoft's Metas and Googles of the world, and secondarily, and frankly separately, ROI questions for the businesses actually using AI. They are related in the sense that the infrastructure hyperscalers pass on cost to the businesses, but they are not entirely related, and not nearly so much as this discourse would have us believe.

So let's break the question apart. As I said yesterday, for the companies that are building out this infrastructure, it is, I believe, reasonable to ask the ROI question. These companies are pumping billions and ultimately trillions into AI infrastructure, and it is reasonable for investors to ask what the actual revenue upside of all of that investment will be. The conclusions that one might come to might make you believe that they are overvalued in the public markets, which could lead to all sorts of different investment decisions, like,

for example, going short on those companies. There are, however, a few caveats that make this a more complicated question than a simple one-to-one analysis of how much revenue they've made versus how much money they've put in. The first is the timescale calculus. This is on the one hand obvious, but I guess not obvious enough, given how little other people are discussing it. But when Wall Street is thinking about ROI, they are thinking in very short terms. And that's fine. That is their job. Yes, they are trying to integrate and price in the future value of a company today, but they're still thinking about the value of the company today. Meanwhile, the executives that are making these decisions to

invest incredible amounts of money in AI infrastructure are thinking about decades down the line. Mark Zuckerberg of Meta has always had this tension with Wall Street, empowered as he is to not have to care as much about Wall Street as other people in his position tend to have to. But now all of the companies, the metas, the Googles, the Microsofts of the world, are acting like Zuckerbergs. They are making bets on the long term and are willing to spend significantly in the short term to get there. So one issue we have here is a simple different. between the time horizons of the different people having this conversation. A second caveat to the ROI question is the one that's the least knowable so we won't spend as much time on, but it's the

AGI potential. The bet that these companies are making is not just on the productivity gains that you get deploying GPT40 across your organization at scale. They are betting on what you can do with GPT6, GPT7, and frankly even more what you can do on the other side of AGI. Of course, it is unknowable, how much new economic opportunity, how much change or disruption, those shifts will implicate. But I believe that the belief among many of the people making these investments right now is that the opportunity there is so enormous that it makes sense to spend even in a way that looks like overspending in the short term to get there. A third caveat for this question of ROI is the capacity of these companies for loss.

Let's not forget that if you look historically, bubbles tend to involve leverage. It's not just healthy companies with healthy balance sheets making bets on the future that leads to bubbles. It's unhealthy companies or other financial actors taking on more debt that they can afford, making bets that turn out to be wrong. The companies that are investing billions of dollars into AI infrastructure are extraordinarily financially healthy. In other words, they can absorb big losses. Of course, we can debate how big of losses they can bear. But the point is that these companies are not in danger of some big levered unwind like we've seen in bubbles in the past. And as a final aside, can you imagine if these companies chose to sit this one out?

The closest we've had to this, of course, is Apple, who didn't spend 2023 really talking about their AI strategy, and it was causing huge chaos in their board. It was having serious impact on their stock price. And so I think that the counterfactual, from a strategic perspective for these companies, is also worth thinking about. So if that is the AI infrastructure buildout ROI part of the conversation, and perhaps why it is less clear cut than these folks would make it seem, let's talk about the other side of the equation, the enterprise ROI side of the equation. It is absolutely true that at the current prices of the APIs of these models, this is a comparatively expensive new technology. That said, there's lots of evidence that, one, this is changing, and two, that businesses are figuring this out. What do I mean by businesses figuring it out?

A lot of the discourse in the enterprise right now is moving away from how do we always access the most state-of-the-art models into how do we match the models that we have available, including ones that are comparatively lower cost because they're not state-of-the-art, with the problems that we're using AI to solve. The information explored this in a recent newsletter under the header, how much do businesses need GPT-5? Stephanie Palazzo-O writes, it seems like every week we hear rumors about what GPT-5,

the next flagship model from OpenAI, could look like, but at a conference last week, a contrarian take popped up in many conversations. Silicon Valley is too focused on what capabilities the next generation of models could bring, without thinking about what we can do with today's models in some engineering elbow grease. A keynote from this week's AI Engineer World's Fair is a great example. In it, web developer Simon Willison, points out that even chat GPT has plenty of usability problems that could be fixed without needing GPT5. Then there was another piece in the information.

Businesses want slower AI models and that might hurt NVIDIA. This piece argues similarly, businesses have become much more focused on the cost effectiveness and returns of AI than on finding the fastest most advanced models. The example they give is batch processing. Basically, there are lots of business tasks that don't require immediate inference, which is costly. They don't mind waiting hours, days or even weeks for responses. And according to, once again, Stephanie Palazolo, quote, founders of cloud and inference providers have told me that there's growing pressure

from business customers for this kind of flexibility, otherwise known as batch processing. There's also evidence that we're going to see significant price competition among the model providers. An extreme example of this is happening in China. The economist last month wrote, price war breaks out among China's AI model builders. The emergence of hundreds of lookalike companies seemingly overnight is pushing down retail prices of everything from electric vehicles to bike sharing and bubble tea. The latest products to enter the ruinous fray are AI chatbots. The article points out

that as the entirety of the market gets up to GPT 3.5 and GPT4 level, LLMs are increasingly commoditized driving prices down. And it's not just China price wars where this is happening. Claude 3.5 Sonnet has undeniably overtaken GPT40 as the go-to choice for people in the know when it comes to LLM performance. For Anthropic, it replaces Claude 3 opus, and in addition to being better at basically everything than Claude 3 opus, it is available at one-fifth the cost, because it is a smaller, more performant model. On June 20th, right after Cloud 3.5 was released, Dan Shipper from Every, who had just released an AI product called Spiral built on Anthropics Tools, wrote the really fun thing about building Spiral is the product

just got way better and cheaper to run, and we did nothing. Anthropic just dropped a new model that's smarter and lower cost. All we have to do is change one line of code to take advantage. Keep this in mind as we come back to the arguments about how AI is unlikely to get cheaper. Today's episode is brought to you by Super Intelligent, the platform for fun, fast AI learning. Super has a ton of new things going on. We recently announced our partnership with Spotify, through which users of that app can now access Super Intelligent content directly from their mobile apps. We've also just launched the AI learning feed. In addition to seeing the tutorials that we're dropping, there are polls, news items with related lessons, and a chance for people

to show off the projects and use cases that are making AI come alive for them. We've also just kicked off the Super Summer Challenge, where each week we'll share a new challenge that you can use to discover new AI tools and use cases. Go to be super.aI and use code superfund for 50% off your first two months. That's Bsuper.a.i. Today's episode is brought to you by Venice. Venice is a private, uncensored generative AI app. It accesses open source models to enable text, image, and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concerns about it being monitored, sold, or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within your own browser,

never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off limits, no ideas are taboo. With Venice, you, you're you're in control of the AI, as you should be. Pro subscriptions are available for $49 a year or $8 per month. Try it for free without an account at venice.AI. There is lastly the sheer business reality that goes back to the point that we were making before.

The fact that the companies building out the AI infrastructure are in such a financially healthy position, that frankly enterprises who aren't getting enough value out of AI applications are going to be able, most likely, to push interim losses on to the AI infrastructure companies themselves. and those model builders will likely be able to bear it in this early phase. So summing up this section, question one, the big question of cost and ROI, there is clearly more nuance to the question of whether the AI model builders are really experiencing a massive ROI gap than is presented by these recent arguments. And I think it is even less clear that there's

any real big issue with enterprise ROI at this time. But let's bridge from there into the second pillar of these critiques, which is about utility and usefulness. Is it fair to say that enterprises are still figuring out what AI activities are going to be highest ROI? Absolutely. For a better and more nuanced take on many of these questions, I would suggest you read Benedict Devin's recently published piece, The AI Summer. His argument, which isn't even really fully an argument as much as a thesis, which he suggests is worth exploring, is that the feeling of magic when using chat GPT was so immense that it distracted us from the fact that there is always a process by which technology has to figure out its way into existing workflows that is a messy, time-consuming, and unclear process.

As someone who runs a company who spends every day talking to businesses of all types about how to use AI, I can say for sure that we are deep in an exploration phase and an experimental phase of figuring out where the most useful AI activities are going to take place. But the idea, which is really what the critique summarizations present, that this means that somehow AI is not useful, is just patently absurd. And if you really push on it, the argument for almost all of the productivity skeptics is some version of a belief that AI is not going to be good at fully automating complex tasks. In other words, there is an assumption that to actually realize productivity gains entire categories of jobs need to be automated away.

Now, we could have a whole discussion around technology and its relationship to productivity, so I'm not even discounting entirely the idea that for full productivity potential, there would need to be big categories of tasks and jobs automated away. However, I think there are a bunch of counterarguments that these productivity skeptics are not really engaging with. The first is that we are undeniably already seeing areas in which there is massive cost savings. AI, for example, is driving down costs in areas like marketing content production, advertising content production, in a dramatic way right now. This is not some future case. This is happening every single day.

It is beginning entire new categories of agencies designed entirely around operating within this new paradigm. This is just one example, but cost savings are not some future thing. They are here now and happening in big, important categories of significant spend. I think there is also an underappreciation of individual productivity enhancement. The way that AI is being adopted right now in the enterprise is in a massively horizontal way. It is individuals who are not waiting for permission to use AI, who are figuring out where it can save them time or improve what they do, and who are, in fact, not telling their bosses about it for fear of being told they're not allowed to use.

it. We've discussed this as the AI smuggling phenomenon in the past. Microsoft and LinkedIn found that 75% of knowledge workers are using AI and 78% of those are not telling anyone about it. I think these arguments are not appreciating how much value those people are finding right now. Third, I think that these skeptics don't have enough appreciation for the value of automating the most draining tasks in an individual's workday. One of the things that is very obviously happening when you dig into this is that people's first instinct is to try AI at doing the thing that they know best. If they are a writer, they ask AI to write for them, and often they will find themselves disappointed, because they could do it better. It's very understandable that this is where people would go,

but in fact, when you actually push on it, it turns out that a lot of the value of AI at this stage is not, in fact, experts using it to be better experts in their expert thing. It is about using AI to simplify, speed up, and improve the parts of anyone's job that they have to do that are unrelated fundamentally from the thing that they're an expert in, but are still part of their daily and weekly paradigm. Part of the reason that surveys suggest that AI is improving people's appreciation of work when they use it is exactly this, that they are spending less time on doing the things that aren't at the core of what they're great at. Fourth, and of course this is the least quantifiable in some ways, but I think that the skeptics have a massive underappreciation of the value of better work. Again,

when we talk to enterprises and individuals who are using AI at work, big categories of use are about things that are highly unquantifiable. Brainstorming, coming up, with new titles, coming up with ideas for content, coming up with new approaches to solving problems. These things aren't strictly speaking about time or cost savings in a way that would factor into an economic analysis. But they are improving people's work. And if you want evidence of this, we just posted a poll yesterday on Superintelligent that asked, what's your primary use case for AI tools? Now, keep in mind that this is a highly enterprisey, work-oriented audience. These are people that are paying for a subscription of a new product that's been out for

less than three months because they are so focused on learning and using AI. Guess what they're using AI for? When asked what their primary use case for AI tools is, I bet many of you would think it was writing. Actually, that's only 7%. Perhaps image generation and other content creation, once again, only 7%. Data analysis, once again, 7%. Workflow automation is a little bit more, 14%. And research is getting up there at 21%, but by far by a factor of more than two, the number one primary use case for our users. Again, an extremely highly enfranchised representation of AI users is brainstorming. 43% in this poll say that brainstorming is their primary use case for AI tools. It is not surprising to me that brainstorming is not showing up in productivity analysis by

economic analysts, but that doesn't mean that it's not real and that doesn't mean it's not valuable. We could also have an entire conversation about coding in AI. One of the new themes that's coming up from these skeptics is an attempt to argue that the benefits of AI-assisted coding are overstated and that people are perhaps walking back what they thought the real value was. This, I believe, is just patently untrue and represents the worst kind of argument cherry-picking. To the extent that you can say anything about AI and coding is that the adoption so far is much lower than you would think. However, for those who have used and who are integrating AI-assisted coding and co-pilots, they are absolutely 100% never going back.

At the risk of trying to make a general point with a specific example, our engineering team has recently found that when someone that they're hiring for a role hasn't developed the muscle to figure out where AI coding assistants could help them, that is actually a knock on our interest in hiring them. Not only are coding assistants not cheating, they are such productivity enhancers that we want people to know how to use them before they get to us. A third question that runs throughout both this Goldman Sachs report and the broader discussions around it has to do with the power grid. This is sort of being lumped in like it has a deterministic impact on things, even though it is a related but ultimately separate question. I think right now we are experiencing

a moment in time where because of climate change, there is a tendency on the part of some to see anything as using power as bad. I have lived inside these arguments for a decade with Bitcoin mining, but now they are back in an even bigger form when it comes to AI energy usage. This is Holy War territory, and again, we're pretty deep into a long episode already, but I think perhaps a more useful question is to ask whether AI creates economic incentive for rejuvenating the American power grid. The answer to me is obviously yes. And I think folks who think like me tend to believe that in fact AI could be one of the most powerful motivations for improving our system of energy usage that we've ever seen. Still, I think for the purposes of this episode,

what's worth keeping in mind is that ultimately this still is a question that is separate from utility. One could, in other words, believe that AI is incredibly useful and still have concerns around its energy usage. The fourth important discussion here is the press coverage of these pieces versus the pieces themselves. To read the press and see the discourse around both the Sequoia piece and the Goldman Sachs piece, you'd think that they were frothing full-throated arguments that AI is useless and hypey. It is, however, worth noting who has presented it this way. It is the media who are seeking a different narrative because they can only write about how brilliant AI is for so long. It is investors who perhaps have a financial incentive because they're trying to short the

market, and it's representatives of this new political class for whom AI is the latest boogeyman. In point of fact, the actual pieces themselves are much more diverse and nuanced than those analyses would have you think. For example, that's Sequoia piece that I focused on yesterday has a pretty important line. A huge amount of economic value is going to be created by AI. Company builders focused on delivering value to end users will be rewarded handsomely. We are living through what has the potential to be a generation-defining technology wave. And even in this Goldman piece, the reality is that they've balanced two critical voices with two much less critical voices. I haven't discussed the less critical side because it's not where the conversation has been,

but they have Goldman analyst Joseph Briggs, who writes, for example, we have long argued that generative AI could lead to significant economic upside, primarily owing to its ability to automate a large share of work tasks, with our baseline estimates implying as much as 15% cumulative of gross upside to U.S. labor productivity and GDP growth. Briggs goes deep on providing counterpoints to one of the big critic in this piece's arguments. They also interviewed two other senior equity research analyst at Goldman, Cash Rangan and Eric Sheridan, and when Cash was asked if he is as enthusiastic today as he was a year ago, he said, I am just as enthusiastic about generative AI's long-term potential as I was a year ago and perhaps even more. The pace of technological change over the past 12 months has been

mind-blowing, with hardly a week going by without reports of a newer and better AI model. Indeed, almost the entirety of the critique holding aside the environmental conversation comes from two people. The first is Darren Asimoglu, the Institute professor at MIT. And he is, indeed, skeptical of much of the hype around how fast AI comes to fruition. However, even within this, there is much more nuance than the headlines would suggest. I pointed out that the pull quote before was almost Paul Krugmanworthy, when he said back around the turn of the century that the internet was likely to only have as much impact as a fax machine. The pull quote from Asamoglu is,

given the focus and architecture of generative AI technology today, truly transformative changes won't happen quickly and few if any will likely occur in the next 10 years. But there is the pull quote, and then there is the full quote. The full quote is this. The forecast differences seem to revolve more around the timing of AI's economic impacts than the

ultimate promise of the technology. Generative AI has the potential to fundamentally change the process of scientific discovery, research and development, innovation, new product and material testing, etc., as well as create new products and platforms. But given the focus and architecture of generative AI technology today, these truly transformative changes won't happen quickly and few, if any, will likely occur within the next 10 years. Over this horizon, AI technology will instead primarily

increase the efficiency of existing production processes by automating certain tasks or by making workers who perform these tasks more productive. So estimating the gains in productivity and growth from AI technology on a shorter horizon depends wholly on the number of production processes that the technology will impact and the degree to which this technology increases productivity or reduces cost over this time frame. In other words, what Asimoglu is actually saying is that these big world-shaping benefits that people talk about are farther out in his estimation than we believe. And secondarily, he happens to believe that fewer tasks will be effectively automated than we think, but the criteria of how valuable AI is over the next decade is this question. And so if he is wrong,

in other words, that more production processes than he thinks will be impacted by AI, and or those production processes will have a deeper impact from AI. that calculus could change. The truest skeptic here is absolutely Jim Covello, the head of global equity research at Goldman Sachs. But if we were being honest about framing this then, we wouldn't say Goldman Sachs says AI is BS. We would say Jim Covello from Goldman Sachs says AI is BS.

What's clear when you read his interview is that Covello really just doesn't believe the use cases are valuable. For example, when asked, are you concerned about the cost of AI technology or are you also skeptical about its ultimate transformative potential, Covello says, I'm skeptical of both. Many people seem to believe that AI will be the most important technology

invention of their lifetime, but I don't agree given the extent to which the internet, cell phones, and laptops have fundamentally transformed our daily lives. Currently, AI has shown the most promise in making existing processes like coding more efficient, although estimates of even these efficiency improvements have declined, and the cost of utilizing the technology to solve tasks is much higher than existing methods. From there, from that baseline belief that it's not as useful as people think, Covello kind of backs into cost questions and cost justification issues as well. I will say that Covello has some very different beliefs than me and many people that I follow. One, for example, is that he doesn't believe that technology typically starts expensive and

becomes cheaper. He calls that idea revisionist history. He says even beyond that misconception, the tech world is too complacent in its assumption that AI costs will decline substantially over time. I would hear humbly only go back to that Dan Shipper post and the fact that literally within the last two months, the cost of using a state-of-the-art AI model has gone down by 80%. The argument then that the costs aren't going to come down just seems patently absurd to me. He also argues, broadly speaking, that AI is replacing low-cost labor with high-cost technology. First of all, this fails to address the clear counter examples that exist, like the marketing and advertising content example that we talked about earlier.

And second, it also rests on an assumption that, again, the cost of this technology isn't going to come down. So far, the evidence is just simply not in his side when it comes to that argument. I don't mean to rag too much on this one person's opinion, but since this is a question one person's opinion has been elevated to the status of defining an entire new trend, I think it's worth digging into. Frankly, his argument reads like someone who personally is annoyed by AI hype, and so has decided to become the loud voice on the opposite side. It ultimately suffers, I believe, from one of the most critical and obvious fallacies of AI analysis, which is to look at it

exclusively in its determination of what it can do today. For example, he says, people generally substantially overestimate what the technology is capable of today. In our experience, even basic summarization tasks often yield illegible and nonsensical results. This is not a matter of just some tweaks being required here and there. Despite its expensive price tag, the technology is nowhere near where it needs to be in order to be useful for even such basic tasks. The incredible level of hubris to make this argument broadly and blithely across all the domains of tasks is, I think, just absolutely absurd. I will now come back to my AI-created presentation, which took me seconds to produce to get to my final point, which is that to the extent that you are looking

for a true disagreement, an interesting and worthy discussion and discourse to be had, hold aside all of the fallacious arguments that we've discussed and perhaps poked holes in, there is one fundamental disagreement that comes up, and that is around scaling laws. Asimoglu, for example, was asked by the Goldman Sachs writer who was putting together this report, while AI technology cannot perform many complex tasks today, let alone in a cost-effective. manner, the historical record suggests that as technologies evolve, they both improve and become less costly. Won't AI technology follow a similar pattern? Asamoglu says, absolutely, but I am less convinced that throwing more data and GPU capacity at AI models will achieve these improvements

more quickly. Many people in the industry seem to believe in some sort of scaling law, i.e. that doubling the amount of data and compute capacity will double the capability of AI models. But I would challenge this view in several ways. What does it mean to double AI's capabilities? For open-ended tasks like customer service or understanding and summarizing text, no clear metric exists to demonstrate that the output is twice as good. Similarly, what does a doubling of data really mean and what can it achieve? Including twice as much data from Reddit into the next version of GPT may improve its ability to predict the next word when engaging in an informal conversation, but it won't necessarily improve a customer service representative's ability to help a customer

troubleshoot problems with their video service. This is a fine and reasonable point. However, it is not the only argument. For example, in a recent interview, AI pioneer Jeffrey Hinton, said that OpenAI co-founder Ilya Sutskever was basically always on the train that if you add more data to these systems, they will simply perform better, and Hinton himself had to admit that Ilya was mostly right. More recently, we have people like Microsoft CTO Kevin Scott, who have made a similar point. Here he is in a recent interview. We're not at diminishing marginal returns on scale up. And like I try to, I try to, you know, help people understand, like, you know, there is an exponential here. And like, the unfortunate thing is

you only get to sample it every couple of years because it just takes a while to build supercomputers and then train models on top of them. And so the next sample is coming. And like, you know, I can't tell you when and I can't predict exactly how good it's going to be, but it will almost certainly be better at, like, the things that are riddle right now where you're like, oh my God, like this is a little too expensive or it's a little too expensive. or it's a little too fragile for me to use. Like, all of that gets better.

Like, it'll get cheaper and, like, you know, things will become less fragile. And then, like, more complicated things will become possible. Like, that is the story of each generation in these models as we've scaled up. Professor Ethan Malik retweeted that and said, When I spoke to Kevin, he said something similar. I totally get some people thinking this is hype, but he has actually seen the next models, and it will be very obvious very soon if he is bluffing

and would reflect badly on him. I view this as a useful signal, but you decide. And ultimately, that's why I said this is of all of this conversation, the one piece that is really interesting to me and worthy of discussion. How far scaling laws hold will have a dramatic impact on ultimately how useful this approach to LLMs are and whether we need to build entirely different approaches to get to AGI. But obviously, this is a hell of a lot more nuanced than where we started, which is Goldman Sachs has called BS on generative AI.

So hopefully through all of this, you have a more nuanced perspective on this question. It is of course a vainglorious hope for our public discourse to be more nuanced than intelligent, but I have to at least try. That is going to do it for today's very long AI Daily Brief. Appreciate you listening as always. And until next time, peace.