Big Technology Podcast - Amazon's Ad Business Soars + AI APIs — With Benedict Evans

Episode Date: March 8, 2023

Benedict Evans is a star tech analyst who’s spent years at Andreessen Horowitz and is now independent. Evans joins Big Technology Podcast to highlight some big, surprising new shifts in the tech industry, which he covers in a new presentation called the New Gatekeepers. In this episode, we discuss how media and retail are blending, with a focus on Amazon's ad business. Stay tuned for the second half, where Evans goes into depth about the potential for AI APIs, especially ChatGPT's. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [e-mail] You can find Evans' presentation here.

LinkedIn Presents. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Benedict Evans is our guest today. He's a Star Tech analyst who spent years at Andresen Horowitz and other firms and is now independent. Evans joins us today to discuss the state of big tech, breaking down his big new presentation, the new gatekeepers, which highlights some surprising new shifts underway in the tech industry. Did you know, for instance, that Amazon's ad business is now bigger than all of newspaper advertising combined, and that it's also bigger than Amazon Prime?

Beneath all of this is a trend that Evans points out, which is that the line between media and retail is kind of blurring. What's a media site and what's a retail site? At Amazon, and some of its peers. That's a bit unclear. That's the kind of stuff we cover in today's episode, along with a discussion of where the chatbot business is heading, looking into some of the APIs. My conversation with Benedict Evans right after this. Hi, Benedict. Welcome to the show. Thanks for having me. Great to have you back. Really had a fun discussion last time you were on. Back then, Apple was at $3 trillion. It feels like a completely different world today. And I guess that's just how the pace of tech moves.

There's just as many people online as they were back then, in fact, slightly more. But there's an old line about the stock market and the economy being rather like a dog going for a walk with its owner. And the owner walks in roughly the same direction and the dog kind of zigs back and force. But they both end up in roughly the same place. And I think that's what happened in the last couple of years. Like the dog went off chasing a rabbit and then just kind of slunk back with its tail between its legs. So you think now we've caught the dog and the human in this metaphor are side by side. Yeah, don't torture the metaphor. Yeah, no. We're in a heel state. Look, there was,

there's a chart early in my presentation of the market cap of Exxon and Zoom. And there was a moment when Zoom had a larger market cap than Exxon. And that was not sensible. There was a moment when, you know, the weirdest Silicon Valley metaphor I heard was people got ahead of their skiskees. people got you know we had a sort of certain loss of discipline a certain over enthusiasm but e-commerce penetration is still 20 plus percent five billion people out of six billion adults have got a smartphone today and so none of those things are winding back none of the actual underlying conversion to technology and you know destabilization recreation of markets is going away. We've just got a little bit of a shift of gear.

So, wait, do you think this entire big tech drawback, the narrative was overblown? Oh, look, you know, it's a long time since I was an equity analyst, and I think about, you know, valuations for as little as possible. Even beyond the stock, though, just like, it's a narrative right now that big tech is in retreat. I don't really know. I mean, I don't know what the narrative is really, I mean, I think there's an interest rate thing, there's a macro thing, there is, you know, there was a moment in 2020, 2021 when people sort of thought, wait, maybe we've just kind of jumped in time forward five years. And it turns out, maybe we jump forward six months. We're not quite back on the trend line, but we're not very far off the trend line that we're on

before. But it's important. This isn't like what happened in 2000 when, you know, the growth that people are expecting just disappeared. We are now roughly where we would have been if the pandemic can't hand in hand. And the valuations are kind of roughly back where they were before the pandemic. So like go figure. Right. So it just feels like there was this moment of irrational exuberance and now things are picking up a pace. They are. Yes. I mean, you know, guess what? Zoom is not a global platform that reinvents communication, but business travel is down probably 20% and we'll stay there. Do you think that some of the minimized valuations on these big tech companies will change their strategy at all? Like, does Apple at $3 trillion act substantially different from

Apple at, let's say, $2.25 trillion? Yeah, I think Apple is probably the least affected by all of this. I mean, obviously, there's a certain Apple. Apple, in the sense, is more exposed to things happening entirely outside tech. Like, you know, as we go into an actual recession, people's willingness to buy $1,500 phone slows down a bit. So they've kind of got kind of a different set of things to think about. For meta or for alphabet, e-commerce growth is slowing down a bit, advertising is slowing down a bit, advertising from outside tech is probably slowing down a bit, and so that has consequences for them. And obviously, Amazon more than doubled their fulfillment square footage from 2019 to the end of last year, and that was probably too much.

And so they're slowing down a bit. But it's not like, everybody who was buying online has stopped buying online. I think the interesting, kind of, there's a sort of a very Silicon Valley dynamic here around things like, I mean, this is obviously the one of the ways that people looked at the chaos at Twitter is why on earth did they have 15,000 people working there, or whatever it was, 10,000 or 11,000 FTEs and then like another 5,000 contractors. And the number they're at now, which is, I think, last I heard, it's like 2000 or something. That's probably not the right number, but the right number is maybe three or four. It's not 15. And you certainly would hear a lot from alphabet and from meta that they had hired an

awful lot of people. They also couldn't really fire anybody. So there were quite a lot of people kind of hang around not doing very much. It's a bit like the old joke, you know, how many people work at this company, about half. And that was starting to feel, to feel a little bit true at alphabet. bet. So, yeah, there's going to be a bit of a, you know, the kind of brutal phrase or anodyne phrase correction. But, you know, the underlying dynamics of the growth that we were having in 2019 hasn't gone any way. So you've mentioned it a couple of times that you have this new presentation now. And I think we've kind of gotten off to the races here to start off. But maybe we'd take a step back and talk a little bit about this presentation. And you make these big,

big presentations once a year that sort of wrap up your thoughts on where things are going in the tech industry, where they're coming from. This year is all about the new gatekeepers and talking a little bit about how, and let me know if I get this wrong or not, but how companies like Amazon are surpassing traditional retail and internet advertising is surpassing traditional advertising. You know, and as I read, first of all, there's so much good data in there that we're going to cover over the course of this conversation. But as I read, I also looked at it and said, wait a second. And so internet advertising, displacing newspaper and traditional advertising, okay, I feel like that's been something that's been going on a long time.

And e-commerce displacing retail, or brick and mortar retail also feels like something that we've seen for a long time. So is your thesis here that we're hitting an inflection point? Well, so there's a hundred odd slides in the presentation, each of them with sort of one point or one chart. And so there's not one thesis. Some of this is just like this is an important thing that you should know is happening. I think maybe two or three things,

sort of threads that run through some of the material is it's almost that I think it's useful to kind of step back and say, well, what are these definitions and why are we drawing those kinds of distinctions? So, for example, you know, when we get into a car and drive to go to a Walmart, we don't say I'm going to do some car commerce now. And, you know, when you drive to your home in the suburbs, we don't talk about my car home instead of my walking home. Those sort of distinctions, that's just one of very many different kinds of distinction.

And I think it's kind of, there's a chart in the presentation where I say, well, this is what American businesses spend on online advertising. This is what retailers spend on rent. And rent and advertising have now sort of become, two sides of the same question in the way that they weren't in the past. You know, you can say, should we open stores there or advertise there, which is not a question you could have asked before the internet. But then take that a bit further.

Well, there's advertising and then there's shipping and there's returns. And there's your payment charges, your credit card interchange fees. There's your trade dollars, your marketing expenses. There's all sorts of money that gets spent between the product leaving the factory door and the customer unwrapping. it at home. And those all sort of used to be different industries, different parts of the York chart, different budgets, and now they kind of become one question. How is it that we should be reaching our customer? And in that context, you know, a lot of most of the Amazon charts that I have

have, have Walmart on them as well. Because, you know, we do kind of, I'm just about old enough to remember when people thought that Walmart was at the end of Western civilization. And it's, you know, raised legitimate questions about what Moremark did to parts of America, but now we understand it's just a big retailer, and Amazon is kind of just a big retailer. I think the sort of the pair to that thesis, I think, would also be how many of these questions

are not tech questions anymore. I mean, I've said a number of times that I don't think the questions that matter for Netflix are tech questions, they're TV questions. The same thing for Sheen, which we might talk about, which is a fast fashion retailer that's now bigger than Zara or H&M. What are the questions that matter to them?

Are they taking questions or power questions? I mean, they sell on smartphones. But it seems to me that one would sort of think of e-commerce or the internet or streaming video, whatever it is, as a new channel or a new route to market. But who is it that uses that new route to market? Well, it's people in that market or it's people who might be new entrants to that market, but it's that market. So all the questions for Walmart or grocery questions and all the questions for

Sheen are apparel questions, but Walmart didn't get created by car people. It's not a car company. It didn't get created by people from Detroit. And I think, so I think that sort of shift of, you know, the technology has kind of changed how all of this works, whether that's apparel or retail or TV or advertising. But then they remain apparel questions or TV questions or retailing questions. And let's unpack some of this. And for me, there were some of these parts that really struck a nerve looking through some of the slides you put together, especially about how some of the stuff has merged, how maybe retail and advertising aren't exactly that distinct. And to just kick off with it, I mean, Amazon, I knew Amazon was an ad

powerhouse, but it just struck me reading through your slides about how big Amazon's advertising business is. I mean, first of all, you have a slide in there that says, Amazon's ad business is bigger than Prime, and Amazon incredibly sells more advertising than newspapers. So can you unpack this a little bit and talk a little bit about how Amazon's become so important in the ad space? So, yeah, so a couple of years ago, so it's always amazing how people who have strong opinions about a company, never seem to go and read the accounts. And so they were kind of loudly so. We wish this company would tell us what this number is. And you think, yeah, that has been in the back of the accounts for the last 10 years.

And so Amazon gives a breakdown of their business. In the P&L, they have AWS, and they split out AWS separately in their retail. But in the back of the accounts, they have these other things like, you know, what's the revenue for the physical stores? And they had this category called other. And the other number started getting really big. And the footnote, note to that said it was predominantly advertising. And then at the end of, I think, 2020, it got big enough that they had to break it out. They had to break out the advertising part. And it turned out it was all advertising. And last year, it was, I think, it was $38 billion. And I'm always kind of a fan of making charts that tell you what that means. And what does $38 billion mean?

Is that a lot of money? How big is advertising? If you don't follow this stuff, like it sounds like a lot of money to me, what does that mean? Which is why I kind of turned around and said, well, it's roughly the same size as prime. It is bigger than the global news paper industry, which as everyone does know, it's kind of collapsed over the last 20 years, which is maybe another conversation. It also makes them like the fifth or six biggest media owner on Earth, although obviously alphabet and meta are by far the largest. And so that was kind of one way of looking at this. I think it's a second way of looking at this, which really picks up to what I was saying a moment to go about challenging definitions and asking what these definitions really mean.

is that you can look at amateur. So you can start at this by saying, well, this is a kind of a broader category called merchant media or retail media. In which, you know, if you own a store, you can't really put advertising. I mean, you can maybe put a little bit of advertising in the store, but you're not a media owner.

But if you've got a website that hundreds of millions of people are looking at, that's media. And the square footage, so to speak, of that website, is inventory that you could put ads on, at least in principle. And combine that with the whole movement, of privacy in the last couple of years and the move away from cookies means that suddenly the fact that you've got this inventory

and you have some idea who these people are and you have some consent to that. And even if you don't, you know what they're searching for in an anonymous way means that all sorts of people have suddenly realized that they have had inventory that they didn't realize without inventory and that it's suddenly relatively much more appealing to advertisers

than it might have been five years ago because they've got this first party targeting data. and Amazon pioneered this, but Walmart will probably do $2.5 to $3 billion in 2022. Uber did a $500 million, hit a $500 million run rate at the end of last year. Target is doing this. Everyone's doing this because, and one of the quotes in my presentation was a quote from the CEO of Walmart basically saying, oh my God, look at these margins. Because if you're a retailer and you've got like two or three percent margins, well,

and then you suddenly add like 5 percent to your top line in. ad revenue at a 50% margin, then that's a meaningful change to your net income, which of course is what's happened to Amazon. So that's kind of a second piece, which is like any retailer was suddenly realized that they could try and do this. I think the third thing that's interesting, though, again, on definitions is, is this advertising or is this marketing? In fact, you could ask is Amazon, is Google search ad actually advertising or is that marketing? You're paying to be next to the till. You know, if you give a supermarket money to be next to the checkout, that's not advertising, that's marketing.

What is, is that, is that what, but that sounds like that, what is it that Amazon's doing? Is it advertising or marketing? You could also call this price discrimination, which is to say if you are a brand and you are paying Amazon a retail margin in order to sell your product, if Amazon comes, when you then buy search ads as well in Amazon, you're just giving Amazon a bigger cut. So what Amazon is really doing is saying, well, people who have a higher profitability, will pay us more because they'll still have the ROI to support it. And so you could say this is advertising, you could say it's marketing, you could say it's

price discrimination, you could say it's something else. Prime is mainly a marketing cost. That's how Amazon think about it, even though it's not in the marketing line. So all of those sorts of definitions break apart when it's all kind of happening on the same website and doing the same thing. Whereas in the physical world, like advertising of advertising and marketing is marketing because they're, like, happening in different places in different ways. Here, they're all kind of this merge into one.

That's interesting. But the question is, of course, like, you're not a media site when you're in Amazon. You are an e-commerce site. And the other side of this is that you can ruin your site by trying to become this media site and end up harming the usability, which, I mean, one look through Amazon tells you that this site is just much worse than it used to be. What's your perspective on that? There's a dilemma here.

I mean, Amazon has pick a number. It's hard to get a hard number, but they've got, you know, certainly hundreds of millions of scoos. At a certain point, those numbers sort of become meaningless because you've got 200 people selling the same thing, and you've got scoos that don't exist until you order them, and so it's not like saying how many scoos does the department store have, and what does Amazon have in comparison. But what you have here in essence is an attempt to capture everything that could possibly exist and could possibly be sold. at least certainly everything that isn't kind of grocery and doesn't have to be refrigerated which is again as a sort of separate conversation but in principle anything other than like

frozen food and vegetable fruit and vegetables could be sold on amazon as an interchangeable widget as an interchangeable school and of course what they've done to unlock this is create marketplace and so now something like 60% of what gets sold to where amazon isn't sold by Amazon. It's sold by third parties using Amazon as a channel. It has occurred to me that if you were kind of an innovative and thoughtful regulator and you wanted to take on Amazon, what you do is you say they have to offer wholesale access to the website and the logistics platform to anybody who wants it. And guess what they do? That's called Marketplace. That's what that is. It's as though any telco, it's like an MV&O. And so, but what that means is that they can scale

indefinitely because if they want to do such and such a product, they would have to hire people to go and source that product, but now they don't have to. It's like they basically created a free market in selling products on Amazon. And so if you want, how can I put this another way? Amazon has got lots of teams selling, who work for Amazon, who sell stuff on Amazon. So like make up in Germany of the team for the sake of argument. But now they don't need for something to be sold on Amazon, the people selling it, the team selling it doesn't need to work for Amazon. Amazon doesn't even need to know they exist.

They can just create their own marketplace account. So it's the original sort of strategic idea of saying, Jeff Bezell is saying everything has to be an API. Well, if everything internal is an API, then why not just open that up to other people outside and they can sell too? And so this lets Amazon sort of scale indefinitely to selling everything on Earth. Now, the challenges, and you see this in content, you know, you see this in newsletters, you see this in podcasts,

is if you create a system that makes it really easy for absolutely anybody to create, then everybody, that's a problem as well. You have infinite content. Right. And so that's a problem for music. It's a problem for movies. It's a problem for books. It's now effectively a problem for anything, for any product.

And I have a slide in the presentation, which kind of makes this point, that we now have effectively have infinite product and infinite media. And it used to be that there was a filter in how much could be stopped by Walmart or Macy's or whatever it is. and what they would choose to stock. So that was one filter. And then there was a filter on who could, you know, what brand could support a nationwide TV buy,

what brand could support putting an ad on the back page of Vogue. And so there was a media filter, you know, what, you know, how many things could Vogue or GQ or a car magazine actually write about anyway? So you had a filter in the kind of the discovery or suggesting channel and a filter in the kind of the logistics or the purchasing channel. And those filters are both gone. And there's infinite media.

There's infinite content. There's infinite product. And so whether you're a brand or a consumer, you've got this kind of like, suddenly you've kind of got the fire hose full in the face, which is what you're describing when you go to Amazon, but I suppose what I'm getting at is like that's a sort of inherent problem in not having a fixed amount of square footage per store and in deciding that you're going to allow anybody to sell anything. Exactly.

And there's another problem in being a marketplace, right? I mean, Amazon became Amazon because it had this first party marketplace. But now it's effectively, as you're saying, become a third party marketplace. in its retail operation and an ad house. And you look at the challengers. You have challenges like Shopify. And another number that struck me in your presentation is that Shopify has 45% as big as the Amazon marketplace. So if you can't differentiate yourself by a first-party marketplace and you play only in the third-party marketplace game, of course, there's fulfillment.

Do you then open yourself up to challenges from a company like Shopify, which, by the way, in your presentation seems a lot strong. than everything that I've heard about the company over the past couple years. Yeah, so, I mean, Shopify is, there's several things that are interesting about Shopify. One of them is a bit like Sheen, a bit like this new thing, Temu, a little bit like WhatsApp sort of 10 years ago or Skype before that. It's this thing that sort of became huge before anybody had quite noticed it. And those sort of, those things always sort of intrigued me. It's like, what the hell is this thing that seems to be in the top 10 of every app? on earth, but I've never heard of.

That's always kind of an interesting thing to look at. Shopify is a platform that lets anybody run a sort of first-class startup quality e-commerce experience. A whole bunch of their business is startups and small businesses and individuals, but another huge portion of it is big companies. So, you know, Facebook uses it, Snap uses it. Heinz uses it. And so it's a way of having, like, top-tier good quality e-commerce experience.

It's a way of unbundling Amazon. But of course, then you still have to tell people about it. So instead of putting all your money into rent, you put all your money into advertising. Then you have sort of the challenge that Shopify has is, are we just kind of an enterprise SaaS company charging a percentage or are we a network? At the moment, they're not a marketplace and they're not really a network. There's shop pay, which is a little bit of a network for the merchant in that they can say, will offer you shop pay and therefore you'll have higher conversion.

But it's not really a network effect from the consumer side. The consumer doesn't know that a website is on Shopify until they've already got to the point that they're making the purchase. And so what Shopify, I think, want to do is somehow work out ways of creating more network effects so that there are more reasons for merchants and or consumers to be on Shopify without upsetting the merchants by supplanting their brand, which of course is the trade-offs that you take if you go onto Amazon, then you're dependent on Amazon's as a brand. Now, if you look at the share price, was that a $100 billion company?

What was the peak valuation? It was $50 billion? Was that really a $50 million company? Well, if they had all that network and stuff done, then maybe without it, I don't know. Again, I'm not an equity analyst anymore. Or occasionally I see the CEO complaining about short sellers on Twitter, which I never think is particularly sensible. No.

And so you can argue about the valuation all day that to me, frankly, doesn't interest me so much as the sort of strategic shift of brands going direct. And it's actually something we should talk about, going back briefly to Marketplace, is this guy, I can't remember the guy's name, a Marketplace Pulse, which is always interesting. He reckons that anything between a third and a half of marketplace vendors, a Chinese manufacturer is going direct. And this is, you know, this was this thing in Atlantic about all these made-up brands on Amazon, that this is also a story about Shin, which is a part of what you've got here are Chinese

manufacturers that used to sell to Western American developed world retailers and brands, now trying to make their own brands and sell direct through to the consumer and because they don't need now to have their own physical retail presence or even put stuff into physical retail. They're a party physical retailers. They can sell direct through Amazon Marketplace. And so Anchor is, I think, probably a $2 billion business last year. And that's a brand that's been created almost entirely on Amazon Marketplace.

So you've got this sort of, you know, it's turtles all the way down. You've got unbundling and bundling going on in every direction. So, you know, you've unbundled from Best Buy. you've unb-into Amazon, you unbundled from Amazon into Shopify. Amazon bundles in your ad takes your ad spend and bundles it back into from TV or podcast or whatever and bundles it back into Amazon. It's unbundling and unbundling in every possible direction. And kind of part of the point is like there was this old relatively straightforward way

in which all of this work that was relatively sort of simple and well understood for like 50 years. And now all the cards have kind of been thrown up in the air and no one knows what the hell is going on or where they land. Benedict Evans is here. He is a preeminent tech analyst. You can find his work at bend dash evans.com. There's a great newsletter there that's subscribed to by more than 175,000 people. You could also find the presentation there. We've been talking about the new gatekeepers focusing on retail and e-commerce in the first half,

second half. We'll talk about advertising and generative AI. Back after this. Hey, everyone. Let me tell you about the Hustle Daily Show, a podcast filled with business, tech news, and original stories to keep you in the loop. on what's trending. More than 2 million professionals read The Hustle's daily email for its irreverent and informative takes on business and tech news. Now, they have a daily podcast called

The Hustle Daily Show, where their team of writers break down the biggest business headlines in 15 minutes or less and explain why you should care about them. So, search for the Hustle Daily show and your favorite podcast app, like the one you're using right now. And we're back with Bendict Evans here on Big Technology Podcasts. We're talking about sort of the resurgence of big tech that may not actually be a resurgence, actually just a continuation along the line of growth that we've seen from them. It's just kind of been skewed by the pandemic. Let's talk about Facebook and Google. We did Amazon in the first half.

Let's talk about Facebook and Google. You mentioned in your presentation that these companies. companies are used to 20% a year growth and they are not going to see that anymore or may not be seeing that anymore in terms of revenue. Do they look different when they don't have that type of growth and why is their growth tailing off? Well, there's many different ways to answer this question. I mean, one of them, I think, is there's a sort of a short-term macro pickup and I don't think one would just sort of presume that, you know, good times are over. I indefinitely, I mean, I don't think I would necessarily

presumed that like the long-term growth is just stopped and that's it now and these companies are just going to be flat. It's more like we've had a kind of a big macro shift in the last six months, 12 months. I think, you know, a second answer might be that when the stock is going up and you are in a war for talent, then you hire an awful lot of people. And, you know, these companies have hired huge numbers of people relative to where they were in 2010. you know, doubled or tripled in size. And not all of those people were necessarily found things to do or found productive things to do. And some of those people will now shake out and go and work for,

there's a sort of, you know, pan-glossy in view in the valley. This is all fantastic that all these people are now available to startups. You know, if you're working, you know, five hours a day at Google on half a million dollars, and now you're getting a job for a startup, we've got to work 14 hours a day on 100 grand plus stock options. That may not necessarily feel like a fantastic change. but like go figure that's like that's the way of the world um however um there's kind of another way of coming at this which is and it sort of speaks to what we were talking about

before the break which is you know what is google's tam is google's tam and it's also this this sort of point you know google and facebook have got whatever it is half to two thirds of digital advertising and digital advertising is now probably three quarters of global advertising um is that the right that but that's not a good way of understanding what's going on on here. What's actually happened is that the value of print advertising basically evaporated as a consequence of the internet. And five to ten years later, Google and then Meadow invented these completely new advertising models that biomlogs got completely different kinds of advertisers and made a lot of money from them. And the ad budgets that are going into Alphabet or

Amazon ads or Meta today, are those coming from TV ad budgets? Well, yes. Yes, but they're also coming from rent budgets and shipping budgets and pricing and margin and everything else. Again, you know, you kind of ask this question, should we spend our money on opening stores or advertising? Should we, you know, if we want to be, if we want people to buy our product, should we cut our prices so that we rank higher for cheap product or should we spend the money on an Amazon ad? What's the best way of appearing on the first page on Amazon? Having free shipping, having a low price or buying an Amazon ad. And the answer is, well, maybe it's all three, or it depends. But they're all on budget now.

And so to look at, and if you look at, you know, you imagine that the marketer at that company saying, you know, do I put the money into free shipping, free returns, Amazon ads, or low prices? And then you look at the Amazon ad revenue and you say, oh, they've taken all this money from the rest of the ad industry. Well, that's not what you happen at all. Like, no, that's not what's going on. interesting what's happening is that the ad budget is no longer a separate thing and obviously I'm kind of exaggerating for effect here but you know what's happening is that the ad budget

the rent by basically the entire margin of proctor and gamble everything below the gross margin line all blurs into one question which is how is it that we spend this money do we spend in advertising or on retail margins do we build a DTC business like what do we do so let me ask you this you know speaking of distinction disappearing and things that swallow everything else and you knew this question was coming, I have to ask it, what do you think the new chatbot wave is going to do to this stuff? Is there going to be a ripple here that it's going to make? Is it going to subsume things?

Or do you think it will sort of just exist on the side as a sideshow and everything continues to pace as normal? So several answers to this. I think the first of them is everyone thinking about this thinks that they are confused and do not quite have a good coherent understanding of what this is, what it means and what it turns into. And that's right down from the kind of the technical level of exactly what, what is it that LOMs are doing, why are they working, what are the limitations of this,

what are the things that don't work and how feasible it is it to fix them. So that's sort of basic like machine learning science questions. But there's also questions like, is a chat bot the right way to turn this into a product? Is that the right thing to do with it? is this is is general search is is this some is generalized search a good use case for this is this something that takes over generalized search or are all the vertical applications that are already kind of stringing up like mushrooms actually the place where most of this becomes most useful I mean I think it's kind of interesting to go back and think about all of the wave of voice assistance five and six

years ago, which I think we would all kind of now feel was, there was a moment when everyone was running around going, oh my God, this is a new platform. And I think we all kind of understood now, like, no, it can do about, they're useful for about five things. And some of that is because before LLMs, you actually had to write the answers one by one. So machine learning let you ask the question, and it could take the audio, transcribe the audio into text, work out the structure of the questions, so work out what you're asking. But it could only ask 50, answer 15 questions. anything else you'd ask, you just couldn't answer, even if it understood the question. And LLMs mean that, like, at least in principle, you can ask chat GPT anything.

Like, you don't have to write the answers one by one by hand. But it still doesn't follow that voice is the right UI or that chat is the right UI. There is a sort of a broad generalized question of like, what does it mean that these things, when these things are quote, unquote, wrong? What does it mean when they, quote, unquote, make things up? and I'm very doubtful that makes things up is actually the right way of describing what's going on when they say something that you don't think is true.

I think there's a really kind of interesting intellectual puzzle of yes, but what does that mean? There's a second question which is like, why the hell is this Bing-Thing gone crazy and how somebody very senior said to me like the Microsoft described this as Microsoft's galactic scale fuck up. Like, what is that?

is distinct from the kind of deeper problems in chat GPT. There's like a third question, which is, is how would you turn this into a search engine? And what would that mean? What would that do for market share? How would that change how the ad model works? What does that mean for copyright? If you are, I mean, I'm sure you're familiar with the idea of newspapers that if I send you a link to a newspaper story, then Google or Facebook are somehow supposed to pay the newspaper, and I think we would both agree that this was just completely preposterous. But if I ask Bing or Google, you know, hey, what's going on in Ukraine this week? And it just reads me out three paragraphs that it's scraped out of the BBC in the New York Times

and doesn't link to them or give them any traffic. Well, that's a whole other conversation. At that point, the newspapers are entirely right. Yeah, that's fine. And say, you can't do that. Can I swear on this podcast? You can bleep it out. The new, yeah, the newspapers, at that point, the newspapers, at that point, the newspapers,

they're entirely right to say, wait, you can't do that. So that's a whole question. And then there's a sort of a more fundamental point, which is like, is doing a chat UI the right way of doing this and is using this for general search? Does that even make any sense? Is that the right model? And so, you know, as I sort of said, the beginning of answering this, like, there's an awful lot of questions.

I don't think we quite know what all the questions are. I certainly don't think we know what the answers are. I mean, there's a whole other strand, which is like, what is the CAPX required to actually pipe all of Google through this stuff, which is, you know, again, I don't actually think anybody knows the answer, but the answer is probably quite a lot, although that will change radically over time. So there's an awful lot of questions. We don't even know what all the questions are yet, and we're kind of trying to digest this and kind of work out, like, what's the right layer of abstraction to understand it. The reason why we're asking these questions is that there is

some sort of they're there, right? That these things are pretty incredible. And without overhyping them, they're unbelievable to chat with. I spent a couple hours chatting with the Bing chat and I've spent more hours chatting with chat GPT. And it's, it is pretty unbelievable. So is it too early to say whether or not this threatened Google. Obviously, there's a this narrative that Bing is the new. Well, so any platform shift, yeah, and so any platform shift is a sort of, any sort of moment of discontinuity is a way of shifting consumer habits and patterns of behavior. It's a moment at which people can drop the ball. That's one answer. The second is, like, does this, is this doing something that's fundamentally challenging for them to do?

And, you know, the kind of classic example, of course, here is always the shift to the iPhone. And a kind of analogy I came up with randomly the other day was the first electronic calculator, which is powered by an Intel chip, the desktop calculator. It costs like $1,000 in today's money, and it's a calculator. And it looks just like the calculators that exist already, but the competition are electromechanical. So they're basically like horrendously complicated typewriters with like 3,000 moving parts inside them. And you take the lid off, and it looks like a kind of David Cronenberg nightmare. there. And they're doing the same thing, but they're completely different products. One of them has got, like, thousands of mechanical moving parts, and the other of them has one chip. And

when you take the lid off, you're like, oh, that's why the people who made the electromechanical calculators were completely screwed, because even though it looks like the same thing, it's not. And that's sort of what happened when Blackery looked at the iPhone and when Nokia and Microsoft look, they're like, this is a completely different thing. And we have no idea how to do this. For Microsoft, the problem was, okay, how does Microsoft beat the iPhone? Well, first of all, they'd be not be Microsoft because everyone's scared of them. Secondly, they have to make an open source operating system that they give away for free that has no commonality with Windows APIs.

Imagine saying that to Bill Gates and Steve Walmart. They'd look at you like you'd grown a third arm. And like they couldn't do it. There's no way they would have agreed to do that at that point in time. And so now we have this question, is there some reason that Google can't do this? Are they the electrical, are they the mechanical calculator company? Of course not. You know, they've got more, they've built more LLMs inside the company than Open AI and Microsoft.

Is there some like business culture disruption reason why they won't do it? Because like it challenges all the things that they believe. Maybe the question is, is the reason that they've so far not done. done it because they're right. You know, this is what you see, like Jan Lecun saying, which is looking at this thing and saying it doesn't work. I mean, I mean, I was going to say, I can monologue about all this all day, but like, I think it's really interesting to compare this with HoloLens because the HoloLens,

do you remember that? Anyone else remember HoloLens? Remember when this was the future of Microsoft and this was going to let them become dominant in mobile devices and this was a new smartphone? Of course. And, of course, he gave them, it's very cool. And it gives great demo. But it's got like a three-inch field of view, and it doesn't work in daylight.

And there was no path for those to get fixed. It also wasn't their technology anyway, which is another commonality with Bing GPD or chat, GPT. And here we are now, whatever it is five years later, the whole thing's kind of been quietly forgotten. But it was great PR at the time. And you kind of look at this and think, is the reason that Google and MetaSever, and so on haven't launched it because they don't get it or because they can't do it or they're dumb or their politics and their business model won't let them, which was part of the Microsoft problem. Or is it that they're looking at this and going, yeah, this doesn't work?

And one of the interesting things that you bring up is, you know, it may not be the core brand, right? Like, it's actually, I guess Bing needed to roll this out as the core brand because it was a demo or whatever it may be. But that was risky because you then attach it to the Bing brand. and they have rolled it back in ways that are surprising. But then you think about how do you make this available to others as an API? And maybe they take the risk. So what do you think about that? I'm kind of curious.

I imagine there's going to be some sort of war to provide infrastructure where other companies or other entities can build their own bots on top of this tech. Is that where you see it going? So I think there's an interesting kind of engineering slash product puzzle in that. in I'm thinking about it I'm thinking it's kind of like it's sort of at a right angle to the problem

put it like that which is I don't think anybody doubts that this stuff is going to be enormously world-changingly useful for certain specific vertical applications so I was looking at something this afternoon which was basically for developers in big companies

that have got mountains of code and say what is this module doing or you know summarize this podcast or do this or do that or you know the GitHub co-pilot is kind of an example here and I think the interesting common thread in those vertical applications is you don't run the risk of it going off the rails and accusing you of like trying to murder it and where it gets 5% of stuff wrong and quote unquote makes things up you can tell because you can see it doing it because you know how it works And you know enough about the domain that you can see the mistakes and fix them.

You know, if you get it to write code for you and you can write code as well, you can look and go out of that bit's wrong. Where the 10% or 5% error rate is dangerous is if you let people ask for medical advice and you're not a doctor and you don't know the bit that's wrong. And that's where you get into difficulty using it for generalized searches, almost by definition, people are using it for stuff they don't know. All of which is a way of saying, like, yeah, make the APIs and let people build dedicated vertical applications. where the people using it, understand what it is, understand how it works. You've built all the tooling around it to handle what it should be doing. It can't ask the stuff it shouldn't be doing. You don't have to worry about people trying to get it to give you instructions for suicide

because it's a coding assistant. You don't have that open-ended problem. And you're putting it in the hands of people who are able to assess the results and evaluate it. So when you make it vertical, like a huge chunk of the problems go away, or at least it's solvable. The challenge is if you try and use it for general purpose search, that's not really a brand problem. That's like, is that, at that point now you've got people saying,

do I have appendicitis and are not able actually to assess the reliability of the answer? Or people are saying, hey, what's the best way to try a noose if I want to kill myself and your filter doesn't catch that? Which is obviously a problem that Google had 20 years ago. Right. And it's the general search thing, trying to use it for general search is maybe where all the problems come from or makes all of those problems much more difficult to solve, which is kind of a paradox because you go back to voice assistants, the reason the problem with voice assistants is if you ask it for cricket scores and nobody at Amazon has written a cricket score module, then it's got nothing for you. and you have to write all of those things one at a time, which just became impossible and unscalable.

Whereas with an LLM, with chat GPT, because you've trained it on quote, unquote, the whole internet. In principle, it can't answer anything, except that maybe it can't, so maybe you need to build a special filter for cricket scores

and a special filter for football scores and a special filter for, and guess what, you're back at your rules-based system is trying to build 150 things. So there's almost like a paradox. And that's what makes it so magical. also is that its ability to go anywhere.

It is, except that then what does that mean? The magic and the disaster is two and one. And it may be that, you know, you give it 10x more data and 10x more compute, and that I have no idea what the actual error rate, but it's an abstract concept anyway, but you know, you get the error rate down by a couple of orders of magnitude and it doesn't matter. That may be one answer. The other answer might be actually, no.

you end up having to build 150 hand-tooled answers, in which case, what's the fucking point? Right. And you should kind of go back to building individual vertical things. I don't know. Let's run with this idea that it may be end up going, it may end up being infrastructure for individual things. Is there a company you think is better position there?

I mean, this is clearly, if that's the case, it will clearly be a battle between Azure and AWS, assuming Amazon releases something like this and Google Cloud. So do you see this, how do you see this battle shaping up and do you think it's a meaningful business opportunity to provide the back end of maybe not like Bing Chat, but travel agent chat, for instance, or code chat that works in specialized circumstances? So I think there's a kind of a useful lens of looking at this stuff is the kind of evolution of the first wave of machine learning, maybe it was the second or third wave of machine learning, but like the wave of machine learning that happened from a following on ImageNet in 2000. And so I was working at Andreessen Horowitz at the time from 2014. And you kind of had these waves of startups. So first wave of startup is I'm academic.

I've turned my resume into a PowerPoint. Can I have $10 million, please? I'm going to go and register a domain name and then Google will buy me. And at A16 Z, we're like, yeah, that doesn't really work for us as a venture investment, but it worked very well for them. And Google, we're not bought all those people. Like second wave is we're going to be an image recognition platform. And anyone who needs image recognition will come to us and use it.

And for most of those kinds of things, that ends up as part of AWS or Azure or GCP. Because that's like a basic primitive. It's like saying we're going to be a database provider. No, that's, you know, the right layer of abstraction for that is a large cloud provider. It's not like, there's not like a standalone market for just that one thing, unless it's something super specialized like Stripe. The next wave of abstraction is kind of comes into what, two kinds. The first, and I picked two A16 Z companies just because the ones I was familiar with.

One of them is we had a company that does legal discovery software. So you see someone, they send you a truck full of paper. What do you do with it? Okay, now they've got translation built in. Now they can do sentiment analysis and say, find me weird emails. They can do clustering. Some of that is using AWS or Azure or GCP. Some of it is theirs, but they're not worried about competition from Amazon.

They're building legal discovery software, and that's what they do. They understand how to build the software and they go to market. And so in that case, they're not even really an AI company. They're a legal software company. An AI just happens to be one of the things that they use to build one of the capabilities that they have. And so an awful lot of AI has sort of ended up like that. It kind of got absorbed into companies who were making some broader thing for an industry.

The other way that AI got kind of deployed is another A16C company is a company that does natural language processing on text going in and out of sales force to work out which sales pipelines are going wrong. And again, that's, there's a lot more AI in that, or the AI is much more kind of primal than what they're doing, because you just couldn't do that before. But that's not what the enterprise salespeople say. The enterprise sales people say, you know, we do sales process optimization and we tell you which sales pipelines are going wrong. And we, yeah, we use AI to do that, but like they're not an AI company. And so that's a sort of, and again, are they using GCP or Azure to do that? I don't know. I don't care. Is it running or AWS? Obviously, yes. Well, that

becomes a whole other conversation about, you know, should you run your own data center or not. But I am, you know, DCP is not going to go and build that product or unlikely to build that product. I don't know. Having said that, now said that, probably they are already building it. But you know what I mean? Like, what's the right layer of abstraction here? And so I think that's in principle, that's how this will evolve. That kind of the broad, generalized low level primitives that should be part of a big cloud platform will be part of a big cloud platform. But the more specific you get, the more industry specific you get, the more there's an abstraction you have on top of something like, you know,

please describe this for me. The more that has to get wrapped into someone who understands the right way to sell asset management software to large movie studios and understands what product you need to do in order to create that or whatever the actual problem you're solving for an actual specific industry is. And so you've got those kind of layer, that kind of pyramid of different layers of extraction. And I think that's how that, in principle, that I would, that's how I would expect this to evolve.

But I don't know what those categories will be. So let me, let me ask you what you think then the business opportunity is. I mean, do you think this is substantial? Like, is every company going to want to have their own chatbot? Oh, I mean, I sometimes say that the tech industry suffers from Tourette's only instead of people shouting swear words at random intervals and people, you know, shout out, voice, machine learning. NFTs, Metaverse, and obviously now it's going to be chat dpti. And yeah, you know, I was on, I've been on two calls today with people who are organizing events in industries that are well outside the tech industry, and they're all talking about chat TPT, and, you know, thinking about what you do with this stuff.

I mean, one of the slides in my presentation, I took a product shot from me Sheehan, and then I took the Sheehan description and typed in a stable diffusion and got three more images. put them up as a two-by-two image on the slide. That's pretty cool. You can kind of tell, but you can't really tell. And that's just like literally, that's just one-shot me, you know, not optimized or special. It's a general purpose model. I haven't played with it at all. I haven't done any prompt engineering at all.

You know, all I did was type in the Sheehan description and I typed in, I think, small red print fabric. Like, that's it. And so that, you know, fairly obviously tells you there's going to kind of a broad class of stuff where this will happen very, very quickly. But, you know, go back to thinking about machine learning, you know, when you dial a call center for your bank now, the bank will be doing five different kinds of machine learning analyzing that. So hypothetically, there's something that's trying to work out

if your calling pattern is weird and you might be a fraudster. There will also be stuff that's trying to work out if their call center agent is being rude to people or hanging up on people to get their numbers. up and they're you know that's being done for them by and that's not being done by google you know that's been absorbed into the industry that provides software for call centers and i think the sort of the same thing here and that's not a i you know and you you know that you don't you don't think of yourself as no i'm going to use the ai now i mean there's an old joke i mean it kind of goes

back to what when we're talking about you know is um is chat gpt intelligent um the old joke in AI research is that AI is anything that doesn't work yet, because as soon as it works, people say, oh, that's not AI. That's just a database. Right, but that's, that's all become the backend stuff. And I think that isn't the change here that you're starting to interact with these things? Or is that too simplistic? Well, I suppose, again, again, the question is that that's the way that we happen to have instantiated that as a product. Right. But I think there's a, you know, chat GPT is a bundle. You know, it is a bundle of, natural language input, natural language output, a chat thread-based UI, a couple of other things

as well. But I wouldn't presume that that's the only way you do this. I mean, you know, kind of trivial example, if you take a screenshot of something on your iPhone now in Safari, and then you tap a week later, you go, I wonder what that was. You tap on the URL in the URL bar, the picture of the URL. That's now tapable. The phone number is now tapable. Take a photograph of a bit of a poster. You can tap the phone number on the poster and your iPhone will call it. Now, there isn't a text recognition app on your phone that does that.

That's just what the Photos app does. Right. And so I think a lot of this will just sort of get subsumed into like, well, that's just how computers work. Interesting. Yeah. I wonder what will happen. Can we, can we end just by, I'd like to read one of your tweets and then we can wrap up.

I'm kind of curious if you could sort of explain what you were saying there. It says, you said, if Google is the new Microsoft and Microsoft is the new IBM, then Bing is dot, dot, dot. That kind of caught my attention. But what's going on in that one? Well, there's a very, very obvious answer to that, isn't it? It's so thinking about Watson. So, look, I mean, I think the observation that Google has sort of become the new Microsoft, I think is fairly obvious. It's become this very big, big, sprawling, rather slow-moving tech company that provides, you know,

the sort of endless, sprawling thousands of different projects, all of them sort of vaguely trying to touch consumer-e-enterprising stuff. Now, obviously, Google sucks an enterprise and Microsoft very good at it, that great relationship it. analogy breaks down a bit there. But yeah, you know, no question now, Google is the new Microsoft, you know, the operating systems in everywhere and all the different stuff that they do. Meanwhile, IBM, you know, what is it that IBM became? Well, they still sell mainframes. IBM's install base of mainframes is in fact still growing in terms of installed

compute base. You know, they sold record IBM, a mainframe compute capacity in 2020. In the hill holders, through the company, they'd never sold as much. Now, of course, this is partly Moore's Law. It's partly migrating VMs onto running on mainframes as all sorts of, you know, it's not the same mainframes that you're running in 1960. But what they turned into is a business that A serviced and sold basically the legacy product, and the legacy customers for the legacy product still use it.

I mean, you know, your bank still runs on mainframes. You know, the airlines still run on mainframes. You know, all the old mainframe stuff that IBM sold in 1960 is mostly still out. out there mostly still running, well, a lot of it's still running on mainframes. So that legacy business is still there. And then they built a huge pro-serve business on top of this. And they built this marketing machine with a pro-serve business attached called Watson and sprinkled some AI pixie dust on top of it.

But it was basically a bunch of like Accenture style outsource consultancy. They'd go and build your SQL database and say, hey, our AI successfully solved this problem for you. I know it's a fucking database. And that's all Watson was. And the same thing for Microsoft. You know, Microsoft has got this huge legacy business of Windows and Office, which is, I haven't looked at the numbers in a while, but, you know, they successfully migrated great chunks of that over to Azure.

They've successfully built a big kind of cloud business. And, you know, one of the slides in my presentation is there's a Goldman Sachs do a CIO survey every six months asking like, bunch of big company CIOs, how much of your workflows are in the cloud. And the number's been sort of stuck at 20 to 25% for like five years. And you see the same in numbers from Gartner and IDC. So, like, cloud has still got a very long way to go. And, you know, that's sort of what Microsoft does. That's what the act, that's what Microsoft actually is. They provide, they provide windows in office as accounting tools and business management tools to big companies. And then

there's a sizzily bit, which is the Watson bit. And so there was HoloLens. And there was HoloLens and then it's now it now it's being GPT and is this really the future of search? Well, I'm being unfair because this is not nearly as much kind of hand-wavy as what's-and-wots. But, you know, this is not, you know, it's very unclear that this is like a fundamental transformation in their business. Right. I mean, of all the reasons we've talked about, like, is this really, okay,

wind back a second, unquestionably LLMs are a transformative technology in the way that, you know, Watson was, there was no transformative, there was no technology at all in Watson. But does that mean that Microsoft takes over the internet and displaces Google and web search gets completely replaced by this? Like, yeah, that's a lot more difficult to say at this stage. Oh my God, could you just imagine this version of Bing GPT playing Jeopard? and the type of stuff that it would spew out.

I would pay pay-per-view money to watch that in action. Yeah. I mean, I saw a couple of examples. My comedy example, which is a slide in the presentation, is I asked Chat, GPT, write me a country and Western song about how I made lots of money in social media. And now I think it's destroying democracy, but I'm not giving the money back.

Right. I absolutely did not have a large private equity investor in mind when I asked that question. Ah, okay. Yeah, it's fun. But that comes back to the error thing. And, you know, the error thing is something that you could spend hours talking about. You know, if you are, I asked it to make a picture of Sundar Pichai and Satya Nadella as boxers.

In fact, I asked a picture for Satya Nadella as a boxer. And I put this in my newsletcher. I'll probably publish it in the next day or two. It's a really good picture of Satya. It's not a very good likeness. It's a really good, it's a convincing picture of somebody who sort of looks a bit like Satya Nadella. as a boxer two problems

number one the rings in the boxing the ropes in the boxing ring aren't quite right do I care no that's not the point it's a photo illustration problem two

he appears to have an extra arm growing out of his kidneys and also a hand growing out of his belly bun they're very well rendered they look like arms they've got boxing gloves

they look like boxing gloves except that people don't have four arms and is that wrong well what did I ask for it's sort of and I can see that it's wrong on the other hand

the rings around the boxing the ropes on the boxing ring are wrong too is that wrong no that doesn't matter what did I ask it to make well did I I didn't ask it for Satya and Adela but only two arms I mean kind of the example I keep kind of circling around

is I asked it to write a bio of me which opening I now doesn't let you do they've got all sorts of kind of filters on it. Right. And so it says Benedict Evans is a world-renowned, sort and hugely influential thought leader, which is obviously entirely correct. Then it says work for Andre's and Horrors. It's, well, yes, true. Then it says work for Bain. No, not true. And founded a startup. I actually never heard of that startup. And now I've never founded anything. And went to Oxford. No, I went to Cambridge. And has written books. No, I haven't. And so again, you can kind of look at this and say this

is wrong. You can also look at this and say this is an extremely accurate rendition of what biographies of people like Benedict tend to look like. It's a very good reproduction of the pattern. It's 100% accurate reproduction of the pattern. They always say what university, and there's always a prestigious university. They always say things like that. As a biography of me specifically, it's wrong, but is that what it's trying to do? No, what it's trying to do is say what would a biography of someone like Benedict tend to look like? I mean, this is the same thing with the footnotes thing. Have you seen this?

So if you ask it to write like a medical paper describing the symptoms of appendicitis, and it gives footnotes, and it gives you footnotes, but it doesn't know what footnotes are. And so it gives footnotes that reference universities and doctors, except that doctor didn't work at that university, and that other university actually doesn't exist. But it looks like a university name, and they look like doctor's names. It looks like a footnote. it's a very accurate rendition of what footnotes look like and that's what it's trying to do

it's not trying to make footnotes it's trying to make things that look like footnotes so I had to pick a couple of favorite episodes from the big technology podcast and this was Bing and Bing got some of them right but made some of them completely up and but did they sound like episodes oh my God I was like did we record that episode like did we have those guests on and we hadn't

and I was like oh maybe this should be an episode that we end up doing. Yeah, well, give it a year, and it can generate the episodes as well. Right, and then I'm really out of the job. That's what I was seeing with Sheehan. And so this is, it's like, this comes back to my kind of domain point. Like, if you were to ask, if I ask it, I talk about this, with someone this morning,

like, if I ask it to make a picture of Joe Biden wearing Roman armor, if you're the editor of the economist and you want that for your cover, you'll look at the picture and go, that's great. if you work at the British Museum and you're a specialist in ancient armour you'll look at that and know that's completely wrong so what is it that you're trying to do here what is it that you want it

why did you want it like the buckles are in the wrong place and that's made of steel and they didn't have steel then they had bronze but like does it matter well it depends exactly and that's the problem with using it for general search because if you've got it in one vertical then you can kind of contain and understand the question

and you know whether it matters or not But if you're just going to throw it at every Google search, you've got no idea whether it matters or not, or whether it's right or wrong, or what that would mean. It's just not ready. Yeah. So I think this is the puzzle, is what's the accuracy? What does that mean?

How would you change that? Can you change that at like a generalized level? And, you know, I'll listen to this. We won't, of course, we won't listen to this in a year. And I'm sure like two thirds of what I've said will be wrong. But that's kind of the point. Like, we don't really know what the questions are yet. It's so happening so quickly.

Hey, look, it's much better to start with the questions. And I think we will re-listen to it because I just went back to our last podcast and I listened to that all the way through. And at least I can say for myself that, I mean, I would love to have you back. And when you're back, I'll definitely be on this one. And then we can start going back and seeing how our predictions stacked up or how even the questions that we were starting to ask, how those stacked up with reality. So Bennett, there's been great chatting with you. Thanks so much for coming on. Sure.

Great to chat. Thanks for having me. Always great to chat. Do you want to let people know where they can find your presentation and sign up for your newsletter? Yeah, well, if you Google me, my parents had good SEO. So Googling Benedict Evans will produce the right results. Amazing.

Okay, great. Benedict, thanks so much. Hope to do it again soon. Thanks. And that'll do it for us here on Big Technology Podcast. Thank you so much, Benedict Evans, for joining. Always great to speak with you. And we can do this for next year's presentation as well.

Thanks to all of you, the listeners. Great having you come back week after week. Thank you very much. Thank you, Nick Gawotney, for handling the audio. I'm looking forward to seeing you in Austin. I think what, next week coming up for South by Southwest, an annual reunion that's well in store. So I'm pumped for that.

Thank you, LinkedIn, for having me as part of your podcast network. And once again, thanks to all of you for listening. Always great to have you here. We'll be back on Friday for a new conversation with Ron John Roy, who's also going down to Austin. That should be fun. And stay tuned for that. Okay, thanks again.

And we will see you next time on Big Technology Podcast. Thank you.