Sharp Tech with Ben Thompson - The Road to Personalized LLM Answers, The AI Risk for Aggregators, Anthropic Releases a ‘Near-Human’ Claude 3

Episode Date: March 5, 2024

The challenges posed by AI as aggregators like Meta and Google deploy models around the around, why personalized LLM output might be a long term solution, and reactions to the latest release from Anthropic and the current pace of AI progress. At the end: A word about Perplexity.

Hello and welcome back to another episode of Sharp Tech. I'm Andrew Sharp and on the other line, Ben Thompson. Ben, how you doing? Doing okay, Andrew. How are you? I'm doing all right. It's good to be back. I do not want to talk about F1. I heard you paid the price for your Mexico trip over the weekend. I did.

I was watching Formula One as I was curled up on the couch because I caught some sort of virus that is currently tearing through. not only my family, but we had the grandparents watch Charles because both me and my wife were reeling in the wake of all this and then they got sick. So everybody's paying the price for our little four days in Cancun a couple weeks ago. Well, you made it. You're here on the podcast, so congratulations. And yes, an anticlimatic F1 race to say the least. But what can you do? So did you experience any of it in the Vision Pro before we dive into the actual show? I did, actually. So it was. It was.

perfect because here in Taiwan, it was on very late at night. Like I think that qualifying started at midnight and then the race was at at 11 p.m. So I actually watched qualifying in the Vision Pro, which fit the sort of use case I mentioned where I think I would use the Vision Pro at night. Kids are in bed, you know, better than being on my TV with AirPods, whatever might be. And guess what? It was way better. Like, you know, I took a screenshot. The screen was just so much bigger than my TV. And so, number one. Number two, usually when I'm watching F1,

it's usually on a little bit of a delay. I'm not always watching it live. So I actually don't want to be on Twitter while it's happening because I don't know, I don't know what happened. And then at one point, I'm like, you know,

it's a pretty pleasant evening outside. I went out and sat on the porch and I had a huge, I had a huge TV sitting there watching, watching qualifying. It was fantastic. It was a really, really good experience. When I paid $4,000 for that experience,

where I'm not sort of like a tech professional, probably not, but now I would layer that on. It's not like airplane-esque as far as look. If you fly a lot, you should buy this device. Like no question about it. But it was definitely pretty great and something that I will definitely do again. So I would say I anticipated that being a quality use case and it exceeded my expectations. Now, again, the not wanting to be on Twitter is a super important part of this.

Yep. Usually when I'm watching a basketball game live, I am glued to Twitter as much as I'm glued to the game. And that is still not nearly sort of as ideal of a use case. And I'd rather just be on TV and have it, you know, have my phone in my hand. But if you don't need to be on your phone, which is an important caveat, it was even better than I expected. I expected it to be pretty good. Yeah. Well, it's a burden specific to F1 fandom because they run these races all over the world.

They run them at all hours of the day. and frequently you or I are watching on delay. And it's hard. You realize how addicted you are to Twitter when you have to stay off Twitter for a little while. And so having the Vision Pro to keep you occupied, keep you focused sounds great. And I love it.

It actually, it prevents accidental like, like just pull out your phone and look at Twitter. Oh, shoot, I just saw the results. Exactly. It takes the power out of your hands. And then I also love you repurposing it for the balcony. That sounds delightful. And who needs a producer?

You know, if you've got a vision pro, you just take it outside, take it out to the porch and enjoy your evening. $4,000 worth. Yeah. For the race, my daughter was like, you know, she wanted to talk. And so she was out there in the living room. So I just had it on moron in the background, which, by the way, the race was less than qualifying anyway. So it worked out. I was, I promised I was giving my daughter by full attention.

But in that case, you know, that speaks to the weakness. It is definitely a one person experience. If you have any desire to interact with anyone else, now theoretically you could use Twitter in the Vision Pro. But that's going to be like a miserable experience. Right. Like a keyboard pulled up or whatever it might be. So if there's any sort of social interaction, not so great. Actively, actively bad.

Wow. So for the race, I watched it on TV and I had a nice conversation with my daughter about college choices, which is, you know, you'll be neck deep in that for the next sort of year. but yeah, thumbs up, but with, you know, everything I thought both good and bad were sort of confirmed in spades. Yes. And I'm glad you chose family over the Vision Pro. You passed that test. Good job parenting. And now we've got a full docket. So let's just dive in. A lot of AIBs to hit. Is that, is that some foreshadowing? Well, no, I think we're going to save the Elon Musk lawsuit for later in the week. But we do have a lot of different AI beats to hit here. And we're going to start with a question from Bill.

He says, do you guys have any thoughts on whether there will be multiple versions of a given LLM that cater to certain customer bases with a certain value set and response types that align with those customers values? Maybe a personal assistant for every individual customer feeding them exactly the information they want to hear. I guess it comes down to what people are willing to pay for. Do they want something that is accurate but says something they may not agree with sometimes or something that tells them what they want to hear, parentheses maybe more like a friend than an answering service? I could see both products existing with the former being positive for society and the latter maybe being a net negative. So Bill sent that email to us in the middle of last week and lo and behold, I read Straterey Monday morning and it turns out you did have some thoughts on this possibility. So what do you think?

Where should we start? I did not read your email, Bill, but I'm happy to have sort of answered it. I would, before we get into the specifics of what I wrote, I would push back on something Bill put in this email where he presents there is sort of a binary choice. And maybe this is the limitation of text and email. It's not what he meant, but as sort of read here, which is number one, You can get something that's accurate, but says something you may not agree with, or you get something that tells you want to hear, more like a friend than an answering service. And I think there is a fundamental fallacy embedded in the presenting of those options, which is that we can know the truth and the accuracy of everything you might ask an LOM for. And that's just not the case. And this is something that I was getting to broadly in my piece, which is, you know, and we, we,

talked about this endlessly when it comes in the context of questions about free speech, for example. Right. It's very easy. I've referenced the sort of Motten Bailey sort of argument style where you stake out an extreme position and then when challenge, you sort of retreat to sort of like the easy position, right? And it's like, I'm in favor of free speech and someone's like, oh, so you want people to do X, Y, Z. And they choose some outlandish or possibly illegal sort of thing to post and be and it's like no that that's that's not what I'm referring to or like misinformation right it's super easy to point to find some tweet online that is misinformation and say oh so you support this and it's like no like but but you can't actually in the end draw as clear

a line as you think you can because a lot of stuff we don't know it can be super gray or it can get into sort of core values so when we get to something like the Gemini fiasco and the issue is, like, how do we represent sort of people in an image generator? What is accurate in this case, you know, according to you sort of Bill's definition? An AI is very accurate. It is accurate in that it is drawing on its core data set, and the reason they have to put in all the prompt stuff is because they don't want to get the results that the AI will generate unprompted.

Now the question then is why is it inaccurate? Is it inaccurate because of systemic sort of issues in the collection of the data set? Is it inaccurate because of the images that were generated? Is it inaccurate because society sort of broadly? Or is it accurate? We don't want to actually deal with the implications of it being accurate. Or we want reality to be different in the future. So we're creating a virtual reality in the hope that it will inspire a different sort of future outcome.

All of these questions can be at play in these results, and it's ultimately a question of values. It's a question of how you think change happens in the world. It's a question of like just fundamental political differences that have been a matter of debate for centuries. Right. And I will just say that we continue to run into this fallacy over and over again with each iteration of chatbot that's released and then generates controversy. The truth is elastic and people are going to be upset about basically any answers that are generated here because you're never going to please every single faction with whatever answers are being put forth by any of these LLMs. Right. And so this was the point of the article that I wanted to get to is that bit. You can't ever please anyone. And the reality is, is that in the world of aggregators, like the last sort of 20 years, as Google has sort of accumulated power and Facebook has accumulated power and all these other sort of big tech companies, they accumulated power. The fundamental point of aggregation theory is that when we shifted from a world of limited supply, power came from controlling distribution.

When we go to a world of basically infinite supply, supply is commoditized, and there's so much supply that you're actually overwhelmed as a user. You don't know how to find what you're looking for. And there are power shifts to discovery, who can help you find what you're looking for. And you start there. And this is how Google became dominant. That's why advertising went to Google, because that's where everyone started. They understood on an individual level what people were looking for.

And there's an aspect here. You talk about sort of the 10 blue links, the classic sort of Google model. And we've talked about this in the context of their business model, which is the brilliance of their business model is they don't charge for showing ads. They charge for users clicking on ads. And basically, they are making the users the deciders of what ads are valuable or not, which is fantastic from an advertiser perspective. You don't want some bureaucrat in Mountain View deciding if your ad is valuable or not. You want the customer that's actually going to give you money to decide that is valuable. And this was the core sort of inside of their business model.

They deputize their users to decide what ads should be worth. The whole risk of sort of AI that we've talked about in the business sense is that you're losing that customer choice function, which fundamentally compromises your business model. What I wanted to drive at yesterday is that exact sort of principle is an issue for the product itself. Yep. As long as Google is surfacing 10 blue links, they are in some respects. Their power is shielded. Their influence and control of the flow of information is hidden.

Maybe there are manipulating search results. Maybe they're not. Who can really tell? They definitely were not sort of at the beginning and they built up this huge amount of trust in sort of habits that you've talked about while you're skeptical that anything will take Google over just because it's so embedded in people's lives. And as long as that was the case,

and you could, even if you had to change your search results around or figure out where to go, you could find what you were looking for on Google. And it was like this shield when this company has astronomical power, but that power was not perceived as a problem because the users were still deciding where to go after they did a Google search. Users are still empowered. Yeah, I mean, it's a really important distinction because to the extent that we're envisioning a world in which LLM's supplant search as information aggregators, the behavior of users entering text prompts is largely going to be the same or at least similar. But the behavior of the companies that are serving the results is completely different than the business that Google has been doing for the past 25 years, where, like, Google, they could be 100% news.

and just serve you a bunch of links. And then Google is judged by the quality of the 10 blue links it's giving you. But with LLMs, it's almost like the entire product becomes, like, remember the I'm feeling lucky button that Google has? Yes, yes, exactly. Enter a prompt and then boom, you have your information. And then users are going to judge Google by the quality of the subjective answer it gives. Depending on what the question is, I mean, we've seen it can generate all sorts of different

controversy and a road trust and everything else. And so it's just an entirely different game for Google to be playing here. That's right. You were going from a world of abundance where they're sorting that abundance for you to the exact opposite world. There's no abundance in an LM answer. There's one answer. It's actually even more constrictive than, you know, maybe if you lived in a city,

you would have two newspapers, right? Two is more than one. And there's a bit about this. this is very, very dangerous. If you're really good, of course it's super valuable. Like I mentioned, I've been using, you know, LMs much more for sort of basic sort of research and understanding of various terms and concepts, stuff that is very far from the political. That is not sort of updated rapidly. It's going to be, you know, it's going to point me in the direction sort of I need to know.

And in that case, just getting the answer is great. trying to go through the web, which is increasingly just, you know, there's so much SEO spam and there's ads everywhere and all this sort of thing to sort of figure out, I just want to understand this particular concept. LM is amazing, giving me one answer. And I just, it's faster. It's easier. Yeah.

It's a delightful experience. I've said in the past that I use it for parenting advice because there's so much sponsored crap on Google that you just go to chat GPT and ask a specific question about your baby and you get an easier, simpler answer on chat GPT. Did it tell you to buy an extra airplane seat for your career kit? I have my friend Ben. Ben GPT serves that purpose for me. But maybe one day chat GPT will catch up.

So this is, but the problem is the moment you get into anything controversial is you're presenting one option, which means by definition, you're imputing values into your answer. Right. And what's important to make clear here. is it doesn't actually matter. This is what those values are. Someone somewhere is going to be irritated and upset. If Gemini had gone the opposite direction

and was showing nothing but white people, there would have been a big controversy, right? Like there is no winning here. And on one hand, that perhaps gives you some sort of, you know, understanding and mercy for or grace or whatever for Google's sort of performance. But on the other hand, it just accentuates the sort of risk. Again, I've been focused on this business model aspect.

What's the difference of serving one answer instead of 10 links with a bunch of ad links that you can click on? But that exact issue is front and center with the product. The issue you run into is if you say, our North Star is accuracy, first, last, than always. People, accuracy means different things to different people. COVID is the classic example. Right.

There's also a lot of people don't want accuracy about it. Well, that's true. But also, I mean, there are a number of people like millions and millions of people who are convinced that they were right. They were on the side of objective reality throughout COVID. And they believed diametrically opposed things about various aspects of the response to COVID. And so you're not winning in that situation.

Right. We don't need. And the key thing is we don't need. we don't need to make a determination here on this podcast about which side was right. The whole point is that there were multiple sides, right? Google can't. That's the point. Yeah, that is going to apply to every sort of single situation.

This is just going to be a big problem for every large company because here's the issue. The aggregator model is predicated on serving everyone. And this gets into sort of the cost structure. So you go back to like say newspapers. One of the challenges newspapers had with going online wasn't just the issue that they used to be a bundle and they had, you know, advertising in a geographic area and all these sorts of things that drove their model. It was also that their cost structure was predicated on supporting that model. So they had a huge staff to cover sort of every area because they had to have something for everyone because they were to make sure to justify being the home for classified because everyone would sort of get the newspaper. They had to have printing presses.

They had to have delivery trucks. They had to have all this apparatus that was part and parcel of the overall sort of bundle they were delivering. You fast forward to the internet and suddenly where on the internet you can have, sure, you might have a tech columnness in the newspaper. You're no longer the only printed product with a tech column. You're competing against the entire internet. You're competing against every other newspaper. You're competing against individual blogs. You're competing against me.

Like the level of competition is massive. And suddenly the idea that some mid-city newspaper can have the best possible analysis or text about any particular issue is very, very low because they're competing against the internet. And so the appropriate product response would be to specialize to sort of focus on what can you differentiate on. You can differentiate on covering, say, local city council or covering sort of the local high school team, things that are local to your area. The problem is that that doesn't support the cost structure. You have this huge cost structure that is predicated on covering lots of stuff, getting mass coverage for advertising. And meanwhile, the differentiation opportunity is to be very, very focused.

By the way, if you're very, very focused, you're not going to have a huge audience, but you will have a much smaller audience that cares intensely about whatever you're covering. The way to monetize a small audience that cares intensely is to significantly increase the average revenue per user. The way you do that is through subscriptions. So the answer to local news, local news business models have written about is to have a super narrowly focused just what you're doing and charge a description for it. And the problem is that that is incompatible with your cost structure. And it's incompatible with the product you're billing to date. You just can't get from here to there. And so in the long run, the answer to local news, I still believe, is very small

focus publications that are focused just on what it is, subscription-based, and critically, have tiny cost structures, right? Strategically ran for years and years, and its only cost was credit card fees and me, right? My time. That is possible on the internet. The only gives you all these possibilities, but you have to have the cost structure to match. Right.

Go to the aggregators. The whole point here is they can serve the whole world. Because they can, every additional user is zero marginal costs. The transaction costs of serving a user are zero. You don't need a sales team or a customer service representative. Like people complain about Google's customer service. Yeah, that's the reality of dealing with a company that is set up to serve seven billion people or eight billion people or whatever it is.

Right. And so the way they work is they have these astronomical costs, but because they are predicated on serving the whole world, they are massively profitable because they get so much leverage on those huge costs. They may have huge costs on R&D and they have huge costs, just the cost of good sold. Running all these data centers is expensive. The sheer amount they spent on electricity is astronomical. But it all works because they're serving everyone. They could serve everyone when they were just giving 10 blue links because the user internalized what they clicked

on was their responsibility. When you shift to a world where you're giving an answer, there's one thing. And the user just fundamentally disagrees with not just fast, but fundamental values, then you're going to lose users. You start losing users. Suddenly, your economic model, your cost structure starts to get a little bit out of whack. And this is why this bit about giving one answer and being political in whatever direction you might choose, whether that be the insertion of these prompts or the absence of them, is such a bigger problem for a company like Google than like Open AI.

Open AI is building up from zero. They don't have the business model to defend, and they don't have the cost structure to maintain. Google has both, and that's a big challenge. So what do you think? Does Google just close up shop here? Is it hopeless moving forward, take down the shingle out in Mountain View?

Well, I mean, what I wanted to get to in this article is the core of the article was to make this point, that like aggregators actually face a really fundamental bit. And actually, talking it through, I wish I would have made the link to the business model more clearly. This is why podcasting is after the fact. It's a follow-up. Yeah, it's a follow-up. I don't like to just point out that there's a problem. Then what?

Oh, yeah, Google is screwed. Okay, when are they screwed? Well, I don't know, TBD. The reality is, in this case, I could do more because it does feel like there is a potential answer, which gets to sort of bills, bills sort of point. Of all the companies in the world that could viable. infer, broadly speaking, what your sort of value system is, it seems like Google and Meadow would be at the absolute top of the list. That's what they do for advertising.

25 years worth of data right here. And the data they have is so arcane and specific and just random, right? People have this vision of their head of like going to Google or Facebook and saying, I want to advertise a basketball t-shirt to white men with red hair between the ages of, you know, 25 and 45 in Washington DC era. Right. Now recovering from the norovirus. It's great. Right.

Somewhere that just recovered from the neurovirus is going to be very susceptible to buying a basketball t-shirt in the next 24 hours. Sort of in a haze, sure. Well, I mean, it's funny. We joke, but that actually gets closer to the way it actually works. In that you're not, it's not these broad buckets. It's through machine learning and just sort of knowing the different websites you go to and the different apps you use, they build these arcane profiles that are indecipherable by any human. This could only be done by computers.

It's like you're in this like this vector 3D space, you know, and just sort of, wow, there's a connection between this and that. And again, there's no human doing this. The computer is finding these latent connections where people that generally do XYZ like these sort of ads. And you actually end up getting shown an ad for who knows what particular reason. It'd be hard for anyone at Facebook or Google to actually articulate it because it's so deep in sort of like vector space and like only a computer could sort of figure it out.

And maybe it's because you just got the norovirus. I don't know. But like there's there's a, you know, that lets them deliver customized personalized advertising. Why couldn't you deliver a customized personalized AI? Right. Like now, I'm sure there are some technical details that I'm not fully thinking through. But in the near future and I brought up this bit that Google, you know, Google is deprecating sort of cookies in Chrome, this idea that, you know, you can see cookies from other sites and serve

them things that they're interested in. The replacement for contextual, so there's two kinds of ads. There's the ads that are personalized based on your specific data. There's contextual ads that are kind of based on, you know, usually what you're looking at on the page, right? So this is like the Reddit thesis, right? If you're on a Reddit about ski equipment, that's a good place to show you skiing ads. The idea with this aspect of Google's approach called Privacy Sandbox. It's called the Topics API, which is basically what happens is you surf the web. Your browser, which is personal to you, it's not uploaded to Google, is accumulating

information about what you're interested in. As it goes, it's basically sort of putting you in buckets of like these sorts of things that you're interested in. And then you go to a site, that site could say, I want to serve this particular ad to this particular bucket. And they don't know who the user is. They just know that someone is on the page that has demonstrated an interest based on their past browsing behavior on some particular sort of topic.

Maybe they've been to a lot of ski sites, so they get a ski ad. And that's how they serve it without having the sort of personal identical information of cookies sort of passing around. Now, whether this is good or bad or the implications is a completely separate podcast. What's interesting to consider, though, is you're dealing with a finite list of topics. That topics list is defined. Imagine that Google generated 100, 1,000, 10,000, whatever, a prompt that was specific to every sort of combinatorial bit of those topics or some sort of subset of them, right?

Sure. And they basically said for this particular user, this is sort of the topics they're generally interested in or they could infer sort of a persona or what that person is like. And the AI has your values. Now, again, we're being a little blasé about the AI has your values. But you could imagine a scenario where the output of the LLM is at least a lot less likely to make you really upset. Yeah, no, and I think in broad, in the very broadest strokes here, as we imagine what this landscape will look like a couple years down the line, it does make a lot more sense than where we are now, which is to say that a lot of the LLM output that we've seen has been underwhelming for one reason or another. And Gemini is the most extreme example, but honestly, one of the big beneficiaries of the Gemini of the Gemini fiasco was chat.

G-G-T because chat G-P-T has been giving its own lame answers for months now, but now it pales in comparison to how lame the Gemini stuff was. And I think what frustrates me as a user is that oftentimes you see some of the output and the answers you get from these LLMs are now so risk-averse that you get these wishy-washy answers that equivocate so much that they're functionally useless or annoying. They're destroying the product. That's the big issue. The products are eating themselves.

And so a version of this that works is something that is actually more definitive and more authoritative, but calibrated to the user. That does raise other concerns, which we can address. Well, sure. The other concerns are the same concerns that people have been raising for the last sort of eight or so years in particular, which is what if people search for the wrong things? What if people ask the Allen for the wrong beliefs? Right. What if the wrong person is searching? Yeah, this gets to the cultural point that we talked about last week.

I don't think that's Google's role. Like Google's role is not to make people believe the right things. And there are people that disagree with that. Like we've had this debate. Like there was a whole debate about the role of the press, the role of the press to report what people say or is it to say what's right or what's true or not? And it's very easy intellectually to say, no, they should say what's true. The problem is then we've come full circle.

What is truth? Like what is actually the correct thing? And it turns out that applies to a whole lot more things than you might think. One of the things I wandered right with this article, I just ended up not really fitting in, but was, you know, over last week, one of the Gemini goofs was if you ask Gemini is Modi a fascist, the prime minister of India, it said, well, here's some arguments for why he is. It kind of leaned towards yes. He didn't quite say yes, but it definitely gave arguments for it. Then you ask is Xi Jinping fascist? It's like, well, it's a very complicated topic.

I don't know. It was like not really sort of winning to it. I mean, the whole like China like still being coded as left wing instead of like it being like a textbook fascist sort of operation. Yeah. It's a consultive democracy, right? Whatever it is. I mean, like definitely no like like ethno-nationalistic like authoritarian regime that's fused with industry. Absolutely not.

Nope. Not happening in China. Yeah. But apparently this is a huge deal in India, right? And people are very upset about it. It led to this maybe somewhat premature, but definitely from all accounts, strongly held belief that AIs need to be regulated.

And so they released something saying like, yeah, all AIs now need to register. They backed off that a little bit. That's part of the reason I didn't want to get too much into it because it's clearly sort of in flux. In flux, yeah. But, I mean, number one, this gets to the idea that all these large-inage models are actually doing is just exporting U.S. coastal

sensibilities to the rest of the world, right? I actually went to the New York Times and I searched for Modi fascists and there's a bunch of articles, mostly op-eds, to be clear, that were sort of like fretting about this being the case. I searched for she-fascist, there was nothing. Like, it's just like there's, you know, we're not going to get into Modi and whether he's a fascist or not, but like the sort of Modi generally a more populous, Hindu nationalist sort of approach, the sort of cosmopolitan traditional consciousness party in India has been freaking out about it for years now. And that has generally been the case that's been put forth

in sort of American mainstream media. So it's not surprised. I don't think anyone was programming a moody sort of bit in the Gemini prompt. But the thing about these AIs is they're based on probabilities. They're based on correlations. And it's so like if we're going to actually code in a lot of approaches that are associated with generally a coastal view of the world, we're going to not just adopt the ones that are coded in. We're going to adopt all of them. And that's why the New York Times search is sort of indicative, in my opinion. I'm not surprised Gemini came out with that, even if it wasn't coded for that because

it's been prompted to have a particular viewpoint and you don't just get the viewpoints that are prompted in, you get the basket of them. That's how LLMs work. It's the probability sort of working against you in that regard. Anyhow, not to have a debate about India and Prime Minister Modi. The point is Google, as an aggregator, needs to be strong in the Indian market. Delivering political opinions about whether or not Prime Minister Modi is a fascist is not a good way to run your business if you need the Indian market. Right.

Google needs to think like a business is the core observation. They had the luxury of not doing that when they were protected by. 10 blue links. They could have whatever political opinions internally that they wanted because they didn't manifest themselves in the product. LLMs, if you're delivering one model for the entire world, are inherently going to deliver your particular point of view. And that's incompatible with being an aggregator. I talked about the idea of zero marginal, zero transactional cost, where you can scale up to serve everyone. There's transactional cost in terms of the product. incurring massive transactional costs in their model having strong political views that huge

portions of their user base don't agree with. They were risking their brand in America last week. Now it's happening in India. They're going to continue to run into that problem. And all of the chat bots are going to continue to run into that problem as we go. Yeah. I mean, I think it's a really a good issue to identify without a clear solution. And in terms of the personalization bit, one of the.

challenges that occurred to me as I was reading your article on Monday, it's almost a question of philosophy or maybe psychology for both the users and the engineering side where everyone has a tendency to attribute human qualities to these chatbots, even if it's just subconsciously. So when you receive an answer that feels both definitive and wrong, or in the engineering case if you're serving an answer that feels both definitive and wrong, it produces a different degree of backlash than if the top result in a search isn't helpful. And so I think the way users think about these chatbots may need to evolve over time. And certainly the way the team at Google thinks about a chatbot is going to need to evolve if they're going to try to

sort of reclaim their neutrality going forward and recontextualized what these products are supposed to be. Because I can understand if I'm an engineer at Google serving up an answer that I just completely disagree with or don't trust. I understand why that's sort of a psychological hurdle for people to have to clear. Does that make sense? It does. And you make a very important point here, which is it's actually really important in product design. to manage carefully sort of expectations and a sense of responsibility. The beauty of the 10 Blue Link model is that the user felt responsible for what they saw, right? Yeah.

This gets into some of the user experience aspects. When I pull out a phone, I'm always cognizant I'm using a phone. I'm not expecting a human-like sort of interaction. The challenge of LMs in the context of that user experience bit is you, feel the latency, you feel the slowness so much more tangibly precisely because it's so much closer to being human. Your expectations are getting screwed with, which makes you feel the flaws much more deeply. That's the same sort of issue we're talking about here. When you're delivering an answer, you're delivering a sort of a conversational style, it's inevitable people will

impute the answer to Google. They're not taking responsibility for the answer. It was what the chatbot said. And so I think it's an essential point. And if you're going, you know, we talk in the user experience case, if you want to actually get this human assistant, there are so many little edges as far as the product user experience that needs to be solved. Like this is why the GROC thing was super interesting because speed kills in a positive way. Like you feel it being so fast and immediate, I think there's a bigger market there just because I think it opens up use cases and opens up people's willingness to engage with these things when you're getting that sort of different sensation of more sort of real-time interaction.

In this particular case, you have to think really carefully about what do you want to inspire with the user. What are their expectations? What is being imputed to you instead of them? Yeah. No. And if companies want to start personalizing results, there's going to be more pressure to do it effectively. and there will be reluctance to build a product that basically reinforces various biases and tropes that may or may not have any basis in reality.

Here's the thing. Here's the thing. People want to deliver the perfect chat bot. The pursuit of perfect in the context of politics always ends in disaster. These utopian visions end up with billions starved or slaughtered or whatever might be because they can't get on board. Or maybe you can't actually manage a complex system. you know, the way you might be able to, right? Like, we, we have plenty of evidence in the last hundred years of exactly that sort of thing

happening in the pursuit of these idealized perfect utopias. Right. Why is a perfect AI chatbot outcome? It's going to suffer from the same thing. You're never going to get it right because you're dealing with humans. You're dealing with the human condition. The best way we've found to manage that, to muddle through that is through some, you know, particular permutation of liberal democracy where people get to decide for themselves.

And that means you have to be okay with some people deciding differently. Some people believing things that you think are abhorrent that are wrong. And this is the core problem for you. This is where we get into a broader philosophical debate. Should tech companies, is everything political? Right. Should they, you know, do they have a responsibility? to be making their chat GPT say the right thing, do the right thing.

I believe philosophically, no. I am against the political everywhere. I think it's something that makes sense intellectually, but in practice ends in disaster. And I also think from a business case, it's also no. It's an even bigger no. You're not just risking your brand.

You're risking your business model. And your cost structure can't support you losing a bunch of your users over time because they're sick of, they don't agree with your politics. Yeah. Well, and I think that's the first step is that collective recognition across big tech. And it's unclear whether last week brought us to that point. But it certainly brought us closer.

Well, to be clear, if Gemini produced every article or every sort of answer I agreed with, this would be the exact same problem because there would be different people that are upset. And actually, you know, my whole issue with Gemini last week was, as I sort of mentioned, The reason I had that tweet that I had to delete because it got my own point was not in the culture wars. My point was the dishonesty. Just not saying like what was actually happening or sort of not happening and trying to pin it on this. Oh, it was actually historical depictions. No, it was the just play it straight with us.

Product quality was not sort of your top priority. By the way, there was like this report in like pirate wires, which is, you know, a publication of the point of view to say the least. But, you know, Mike Solana did talk to Google employees. what they reported is that there's like multiple LMs that work here that are like rewriting the prompts on the fly for the image generation and that actually makes sense because I there's that one I got the prompt for the British Royals and it was like completely rewritten with like four different images and and you know got those sort of like absurd results and the amount of effort to the extent this report is correct the amount

of effort and work that went into to get these results it's not an it wasn't an accident Right? No. This is, this is, and this speaks to, to me, the solution is actually pretty straightforward. Try to get back to what you've always been,

a neutral arbiter and personalized advertising, which you can apply to sort of personalized prompts. This is where the culture bit comes in, though. Right. Apply the resources you're currently applying to engineering outcomes to engineering personalized outcomes that are served neutrally, like that it should be doable. Or serve based on what the user wants.

Again, is there any neutral? Right. But Google is neutral in that scenario. Right. They give everyone what they want. And to me, again, I think philosophically, that's correct because I think human problems are human scale and should be dealt with humans, not big tech companies. But also from a business perspective, I think that's what they need to do. And what was unspoken or covered by my article last week is I don't know that Google is capable of doing this. They just feel like it's over the years of monopoly. and the cover of 10 Blue Links, they've developed a culture that is hell bent on telling people what they ought to think. And now they have an opportunity to do that. And it's not going to end well.

Well, and I would also be remiss if I didn't acknowledge that there's risk going the personalized route because there are going to be all sorts of ways to generate horrifying answers that then generate their own news cycles that create risk for these massive corporations. Oh, for sure, which is what we have right now, right? We have the whole thing. You can go search for like bad tweets or bad Facebook posts or bad YouTube videos and say, they're allowing misinformation on their platform. That's just the reality of having a platform that has serves the entire world.

Like the entire world, unfortunately, has a lot of misinformation and a lot of bad opinions. The issue is there is no, there is no silver arrow here. Silver arrow. Is that the color of the arrow? Silver bullet. Silver bullet. Silver bullet. Silver bullet.

That's what I mean for it. I knew it did sound quite right. Yeah. The, you know, and so the question is like, and what should a tech company be making these determinations? Is that the right thing to do? Like, like, again, this is the problem. We're going to get a response to.

Oh, so you want this. You want that? No, I don't. It's that it's a tradeoff. Like, there is a tradeoff about what is possible, what is the right thing to do, getting the good with the bad. Every option has bad outcomes.

You can't have the perfect option. Like the bit about perfection and pursuit of utopia, that applies to price. decisions as well. Yeah. It's not possible. And so you have to make the tradeoffs. And I think the tradeoffs that are in line with the ethos of liberal democracy where we let people think for themselves and decide is the right direction for Google to go. I think that is open to them. I think they can redeem this. And I also don't think they're going to take it. Yeah. Refram your understanding of what the product should be would be one step. And then maybe you allow users in terms of

creating user expectations, you allow users to enter data about themselves so the chatbot can learn who you are, where you are, what you might want to see, just so that they know that it's programmed to give you a certain value judgment or whatever it may be. All of that remains to be seen, but I thought it was a good article on Monday and thought-provoking in terms of what the future is going to look like, because where we are now doesn't make a whole lot of sense. But in any event, where we are now, changes by the day. And so I just want to mention this as something that emerged on Monday, probably right as you were going to bed in Taiwan. I'll read from Ars Technica. On Monday, Anthropic released Claude 3, a family of three AI language models similar to those that power chat GPT.

Anthropic claims the model set new industry benchmarks across a range of cognitive tasks, even approaching, quote, near human, end quote, capability in some cases. According to Anthropic, Claude 3 Opus beats GPT4 on 10 AI benchmarks. And just for context, AI researcher Simon Willison spoke with R's Technica and said, as always, LLM benchmarks should be treated with a little bit of suspicion. How well a model performs on benchmarks doesn't tell you much about how the model feels to use. But this is still a huge deal. No other model has beaten GPT4 on a range of widely used benchmarks like this.

and I don't want to delve too deep into specifics because you haven't had a chance to really play with the model. But just in general, can you believe how quickly all this is moving? Like, is there any reference point for a tech landscape that has evolved this quickly in the past? Yeah, I mean, I think that's a good question. Probably the early days of the Internet and the early days of PCs. There was an aspect of this. I wrote an article a few years ago called the end of the beginning. where I actually went back to the dawn of the,

the automobile. One of my favorite pieces of years, yeah. And there were just like tons and tons and tons of car companies started every year for like 20 years, like just a ton of them. And because we were still figuring like out fundamental issues of the automobile, like how it might work and how to make them, how to produce them.

Obviously the huge innovation there was sort of Henry Ford and the assembly line. But then, you know, and then GM having a, we're going to have a differentiation strategy. We're going to have all these different brands and categories that things. GM grew in part by buying up a bunch of those sort of other car companies. Not all of them were sort of products of GM sort of originally. But in the end, then we ended up with like three, right? And that's the payoff. If you, at the beginning, it's a gold rush. There's a mass amount of like, let me become sort of a big player. If you win, the returns are astronomical. So it's worth it to risk the fact you're going to be one of the 97% that flame out.

Because if you're the 3% or the 1% or the 0.1%, the returns are just so, huge. This is the beauty of the venture model sort of in practice. Like, like, that's the idea is you just got to get one winner. It will make up for all the sort of bad bets. And it's great. We have an ecosystem that sort of funds this. And we are absolutely in the gold rush era of AI. This was the, this is part of the, the current bulk case for Nvidia. People are price insensitive. They will buy whatever sort of GPUs they get their hands on. It's also trillion dollar companies that are just like, yeah, we're not going to miss out on this. By people, I mean, you know, every. sort of a sentient being or non-sentient corporation. They will buy whatever GPUs they get their hands on. Blake check. Yeah, absolutely. So, yeah, there's lots of models coming along. Mistral's latest model is very, very impressive. You know, quad, you know, anthropic coming along with the model.

That's because of the GPT4. Now, it's worth noting GPT4 was trained in 2022. Like, they're working on polishing off GPT5 at some point. So I think OpenAI is broadly speaking, again, not having used sort of quad yet. but is still in the big picture in the lead. But what it speaks to is two things. Number one, the lead might not matter in the long run because there might just be a collection of foundation models,

all of which are very good. And if there's a collection, that's not good for profitability in the long run because there's high competition and it ends up being more of a commodity. And this is generally the case that happens by and large, and it's probably an average foundation models, where the real money to be made is going to be a layer above

where you productize it and you make something that is compelling for people. And maybe that's integrated where the product is integrated into the model like ChatchipT. ChatchipT isn't a foundation model. Chat ChachyPT is a product, right? And it's a product that is fully integrated into the foundation model. But, you know, it's sort of moving up the stack. And you can move up further into these different companies that are doing things with LMs. And that's not to say that being, this can't be profitable.

It's just that your approach is more high, volume, low margin. and, you know, it's not the new Google. Yeah. Right. So this, this speaks to that dynamic a bit, number one. Number two, this ties back into what we were just talking about. The risk for Google is that there are actually a bunch of AIs and there could be another AI that people prefer.

Now, by, from what I've seen slash heard, quad and chat GPT and Lama and Google all happen to have San Francisco area of politics, right? Like there's a bit where that may be sort of inescapable. The big question is, you know, maybe it doesn't matter just because no one else actually builds a functional foundation model that that is sort of open or broadly appealing. But to the extent there is that shows the risk factor I'm talking about in this regard. The third thing I would say, and just speaks to sort of Simon Willison's point, is measuring these models is getting really, really hard. Number one, like, who's making the measurements?

Who has the right incentive structure to do it? Like, you know, Matt Freeman mentioned in the interview with him last week. You know, the current dominant evaluation standard was made by some undergrad student, like, just, you know, and it's become like the standard. And I think Anthropic was the one who is releasing the measurements today. So I think it remains to be seen how it measures up as more objective parties digest all this. Well, the other bit, too, is there's so much already. already there's so much GPT or model produced text on the internet that it's it's going to get it might be impossible to sort of measure going forward. There's no more like clean data sets.

And so like there was you have this bit in here. I'm sort of like stealing your role, but talking about, you know, wow, it's incredible. I fed this huge amount of text into opus and I put like I inserted one sentence that didn't make sense and it found it, the needle in the haystack. I was just marveling about this last week in the context of Gemini 1.5. sort of having that capability. There's also a bit where part of this marveling wasn't just that it found it, but also said, oh, I think you're trying to trick me. I think you're trying to test me.

I found the needle, and I can tell you're screwing with me. So there's the human-like behavior there. No, that's not human-like behavior. That's evidence that there's lots of texts of the internet of people posting about them trying to trick these models and talking about it. That's how it knows it's being tricked, right? And so to me, that's actually evidence that we're starting to see this recycling of tokens of like what's been produced going back into these models.

Well, you just ruined the fun. I was really enjoying the pizza trick that Claude 3 opus played on the engineering team out there. But sure. Yeah. How does the model know that it's being tricked, right? Because that's what was remarkable the answer. It wasn't just that it found it. It said, ah, ha, ha, you're trying to get me.

It knows that because people have been posting about trying to get the models, right? And, oh, wow, look at it, found this sort of thing. And, you know, the implication. of this in the long run, what it means as these models are increasingly trained on other models sort of output, maybe it makes this whole problem worse about having a particular point of view, right? Maybe like we can generate synthetic tokens that are compelling or interesting. I mean, part of Mistraw's sort of thesis is they spend a lot of time having super clean, well-structured data and spending a lot of time like prepping that on the way in and the idea

being like, if you have really, really good data, you can have a smaller model that has really good results. and maybe, you know, the sort of trade-off between, oh, I, let's scour the whole web versus, you know, maybe the cost of these foundation models are let's spend a few hundred million dollars paying a bunch of people to write questions and answers to actually give, like, you know, make sure the data is like very, very accurate. That's going to be really interesting. I mean, a human job in the future may be producing training data for AIs. Like, like that's, we'll see.

I mean, so none of this is to diminish anthropic at their accomplishment. I just think some of the broader structural points it brings up are pretty interesting. No, yeah. And it's just for me, processing all of this, it's almost like the first couple days of NBA free agency where there's a big move and then six hours later there's another big move. And then the next day, there's another big move. It feels like that's what AI has been, except it's been like that for like 16 months now where every week there's a new mass. story that is a game changer and we all just sort of adjust our expectations and then the next week it happens again. And so I don't know when this cycle is going to end, but it's pretty

wild to live through and watch Twitter lose its mind. It does seem to go through stages. It doesn't go through stages. The spring seems to be the big, like last year, last spring was also insane. And then it kind of slowed down over the summer and everyone's like, oh, there's no one using chat chapitp tachy anymore. Turned out students just weren't in school. And then like all these sort of, you know, bits and pieces. But yeah, but yeah, if you zoom out at a high level, like the level of changes that happened just a year is astronomical. And I think that's going to continue for quite a while. Again, the car era took 20, no, longer than that.

It took like 30, 40 years to sort of shake out. The, you know, when I started trajectory, it was so exciting. And I thought I was too late to miss it. Like the whole like smartphone, who's going to win? That was 2013. it already been going for six years. By 2015, 2016, it was pretty settled. It was pretty clear that about the duopoly

and that Apple was not going to be disrupted by Android, which was the case when I, that was the conventional wisdom when I started and I got a lot of traction about saying, no, that's not going to happen. But so that we had probably a good nine-year cycle with smartphones. Yeah. Just like shaking it out and seeing how the structure is going to be.

PCs probably lasted, you know, depending on, you want to mark the start, like 19, it's not, it didn't just start with Windows. It started with like the Commodore and like the, you know, the, you know, the, you know, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the, the company would be, starting in the 70s. And then it probably fully settled in, probably the end of the exciting PC era was probably 1995, the release of Windows 95. Windows dominance was solidified, not just in the enterprise, but also in the consumer space. The Mac was an also run and everyone else was dead. And so we had a good 20 years of shakeout in the PC era.

And Intel is releasing a new processor every year or every two years or whatever. And it was just the old ones completely obsolete and all these things were happening. And now who thinks about their PC at all, right? Right. AI will probably, you know, the potential impacts and use cases and applications are so huge. Like we're still in the, we're still talking about foundation models. We have a whole era of actually building products that are built on these things. We have a whole era of figuring out what's going to happen with chips.

Like right now, GPUs are king. It just seems inevitable at some point, given the costs involved and the energy involved, we're going to end up with specialized chips that are specific for particular models and are just way more efficient and way faster. Like this is why the GROC thing is sort of interesting. Not saying they're the answer, but we're going to end up there sort of somewhere, which ties into, you know, where is Nvidia going to sort of be in the long run?

Where is that going to work out? You know, maybe I'm wrong. And it's like CISC versus Risk, where risk was definitely better, but Intel realized CISC was ahead. They should sort of stick with it and they were right. Maybe GPUs are theoretically worse, but there's so much infrastructure built up around them, they end up being dominant. TBD, and which is great.

That's why I'm excited to be here. I'm excited to talk to you. It's the second wind of just like, I, I, I, I mean, my poor, oh, actually, we have, I have to put this in the daily update. We actually watch another YouTube channel, which is, uh, trajectory articles. It's just like the main articles that are illustrated and sort of my voiceover. And, you know, I brought on, uh, Johnny Udo's agentometry is helping me sort of make these videos. And I'm like, yeah, I, I've kind of reduced to doing an article every other week.

I used to do it every week. And poor guys drowning because I'm like, back every week. I'm like, fouding these out. There's so much happening. And it's exciting. It's great for me, that's for sure. Yeah, well, and right now we're in the behemoth stage of things, or as you would say, behemoth, or behemoth. I forget how you say it.

Probably I've tried all the above. Yeah, well, and we're in the like trillion-dollar wars stage, but I also look forward to the open source models that emerge over the years to come here that make this even crazier. And so it's very exciting and occasionally exhausting to try to keep track of everything. everything. But that it occurred to me watching the entire internet freak out over clawed there. All right. Final note here. I said it was going to be an AI day. We'll close with yet another model. Bartek says, I find perplexity to be much better for exploration of a topic I want to learn more about relative to chat GPT or Google Gemini. Here, I won't write very much. I'll just share a thread of perplexity describing Ben Thompson and a couple more follow-up questions.

It did fail on the most important question. Why is Andrew not using AirPods? Even hallucinating AI cannot generate an answer for that. Bartek, I am not using AirPods because I don't want to have to charge my headphones. But the answers that perplexity gives are very, very detailed and impressive. I don't know if you've used this very much. I have, I made one of, you know, I mentioned I used the, the Mac OS ability to, you know, add a site to your doc, which basically creates a single site browser. I have Perplexity here as well. So number one, perplexity is not a model. Perpuxity is using, I think by default it uses 3.5, but if you pay, you can use four.

Maybe that's changed, but so I actually do. But so this is a example of a product. It's a level up on the stack. They're not making their own model. They're making a product. And their whole thing is doing, incorporating, I talked about RAG before, you know,

where you're getting other material and incorporating in your answer and presenting it and sort of a chatbot interface. That's what they're doing. So they're, they're, you know, they're,

it's a search engine. Like that, it goes out to the web, it gets information, and incorporates it with the large language model, and sort of presents an answer. I would say, number one,

it's very useful for, like, for example, you can do all these searches on Twitter with all these modifiers, right? Like, specific date ranges.

from XYZ, blah, blah, blah. And every time I can, I know a couple of the modifiers, but particularly the date ones, it's like since until I think you have to put it in a certain format. I have to always go look it up. How do I actually, I know this tweet was at a certain time, how do I actually put in the modifiers? I go to perplexity.

This is what I use it for. And I'm like, I want to do, I want to make this sort of search on Twitter. And it will go through, it will generate and give me the exact sort of like search thing that I'm looking for. That's awesome. And it does the search for it, right? And so it's more of a product that is the user interface is a chatbot, but it's a search engine.

That's kind of what it is. And by all accounts, it's growing very, very rapidly. I would imagine they've gotten a lot of investor interest in the last week or so where people like, maybe Google is actually more endangered. Tetering on the edge. Yeah. Let's buy in. My main issue with it is it's a little slow, in part because it's going out to get information.

Like so for, I actually have both. I've rededicated my, one of my monitors and my multi-monitor setup. One of the monitors is my actual. My actual MacBook monitor, which has just been sort of a grab bag of stuff. I usually have like my recording interface on there so I can see the levels and things like that. I now have, it's now my, my AI monitor. I have perplexity on the left. I have chat chipt be on the right.

And I actually find that very useful for different sort of queries. Again, explaining a concept chat chip PT is great. And the back and forth is actually useful, dive into this, do this sort of thing. For some of these specialized web searches, perplexity works well. And then, you know, I still use Google the most, to be clear. But, but yeah, it's a compelling product. And I think in many respects, it's more compelling after this last week. Yeah, well, it's more pleasant than 10 blue links, half of which are spam and junk and

sponsored content. I just, as a user, it's a cleaner experience. It has citations in here. They ask, what is Ben Thompson's connection to Andrew Sharp? The partnership between Thompson and Sharp began with their mutual interest in technology and media. Thompson's initial relationship with Sharp was as a listener of the Sports Illustrated Open Floor podcast, which Sharp co-hosted.

Their professional relationship evolved when Thompson had the opportunity to work with Sharp on Sharp Tech following Sharp's departure from journalism and his brief foray into law. Good information there. They missed our first one. in San Francisco. Our initial, our initial interaction was definitely about basketball,

not about tech, because, again, we're dealing with someone that doesn't have AirPods. But, but yes, that's pretty good. And I just say the,

I did the, I was do the, like, who has Ben Thompson? What is aggregation theory? I will say, Gemini crushed that one. Did that?

Summary was so good. I kind of wanted to just steal it. It was like, look, I get this question. I'm just going to drop it on there. Nice, tight link to send somebody. This is the whole Google paradox. They're good.

They have infrastructure. And it's, it's culture. It's just that. And it's culture and it's the, it's what happens when you've had a monopoly for years and years and years. It's problems confessor. You lose the ability to, to, to the product is no longer first. You have the luxury of the product not being first. And I mean, Sergey Brin does not seem very enthusiastic about taking over Google and making wholesale changes.

Like he's out there saying, oh, yeah, sorry, we screwed it up. X, Y, why. really want to know how expensive his coat was at that coating party he was at. It was a great coat. People were distracted by the guy. People were distracted by the guy with the boob shirt. But yeah, I had eyes for sure he's coat. That's for sure.

Yeah. Well, Bartik asked, I heard Ben mispronounces hard words very often. Is that true? The search results provided do not contain any information about Ben Thompson of Sertechri mispronouncing hard words frequently. So, GPT needs to up. update the training data. I'll tell you that much. Room to improve. Room to improve with all these different foundation models and everything else. But great to be here. Great to see you. I am happy to be up off the couch and rejoining the land of the living here on the podcast after a rough weekend and a rough result for Mercedes. Hey, but hey, maybe you'll get a cheer for Max on Mercedes. And then that'll be great for you. Oh, my God. I'm going to wretch all over again here. All right. On that note, we'll come back in a couple days. And Ben, I will talk to you soon. Talk to you later.