No Priors: Artificial Intelligence | Technology | Startups - The Copilot for Ecommerce with Shopify VP of Core Product Glen Coates

Episode Date: January 31, 2024

Building an ecommerce business is hard – it requires merchants to have a wealth of skills: technical, logistics, marketing, pricing, vendor management, finance and analytics. That’s why Shopify is releasing new AI features that help merchants tackle things like product descriptions, marketing suggestions and search. Today on No Priors, Glen Coates, the VP of core product at Shopify (and former founder of b2b wholesale platform Handshake), joins Sarah and Elad. They talk about the releases from Shopify Editions, why they are deploying “copilot” rather than “autopilot,” AI innovation-at-scale, how to change the basement of a house while people are living in it, and building a leadership team of entrepreneurs. Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @glencoates Shopify Editions | AI Section of Shopify Editions Show Notes:  (0:00) Background (2:22) Calling a “Code Red” at Shopify (4:04) Integrating acquisitions, entrepreneurial leaders (12:15) AI adoption (15:51) Deciding when to ship AI products, evaluations (17:33) Shopify’s risk orientation (18:50) Changing the core Shopify data model, enabling AI features (26:05) What’s missing from LLMs for merchants (28:47) Most interesting AI developments in the industry (33:22) What users want from LLMs and search (38:20) No Priors social

Hi, listeners, and welcome to another episode of No Pryors. Today, we're joined by Glenn Coates, the VP of Product at Shopify, where he leads the core developer platform, including all of their AI products that we'll get into today. Before he was a leader at Shopify, Glenn was the founder of Handshake, a B2B e-commerce that was acquired by Shopify in 2019. We're excited to talk about how AI is changing e-commerce and entrepreneurship, as well as innovation at scale and leading at Shopify. Welcome, Glenn. Thanks for having me. So lots of fun stuff today. Let's definitely cover your personal story quickly, since you're also a former founder,

one of several at Shopify now. Can you give some of your background? I was a comp side grad. I spent the early part of my career in video games, and then I had a weird left step in my career where I moved to San Diego in 2008 to run the U.S. operations of an eco-friendly shopping bag company. You know, like you take your own bags to the store, that whole thing. Anyway, this company did a whole bunch of wholesale. They sold a lot of bags through stores, and through that, I ended up getting to the idea for Handshake. And Handshake was a initially a very, like, sales rep focused B2B e-commerce product, but then became both for sales reps and for customers to buy online. So Handshake eventually became basically a B2B only version of Shopify, like a wholesale only version of Shopify.

So it's not hard to join the dots from there to after, you know, building Handshake for a better part of nine years here in New York, the opportunity came up to join forces with Shopify. And then I've been at Shopify for almost five, it'll be five years in May. I started out very much focused on on B2B and wholesale, which was obviously the point of the acquisition. I spent about a year doing that. I then moved on to focus. I ran a code red on checkout in 2020, which was the first year of the pandemic.

And then since the end of that, I've been leading the kind of core product group at Shopify, which is you can think of it as all of the built-ins of Shopify, the online store, the checkout, the back office, the developer platform, the app store. And yeah, that's what I do at Shopify. Lots of good stuff. What's a code red look like? A code red is when Toby sends an email to the entire company saying this thing is the number one priority, and he means it. From an actual operational perspective, what that really means is if the team that's working on this code red asks you to help, please drop what you're doing and help.

This is the number one priority. So that's how they work, but usually a code red is a symptom of some other, much more systemic problem that's gotten you to that point. at least in my case, checkout code red in 2020 was, hey, the checkout is failing in, I don't know, three, four, five different ways. And the checkout is a pretty important part of Shopify, obviously. So let's go fix that. So we, me and, you know, somewhere between, I think at its peak, it was probably two, three hundred people working on various parts of the problem. So we all like scrambled for a year and like did what it took to fix those issues. But then at the end of that year, we took a step back and said, okay, well, why did that happen? Like, how did we get to the point

where those problems even happen in the first place? And then that led to some of the reorganization of the company around less, like there used to be 12 to 15 of these kind of fairly small fractured business units. And now there is actually only like three or four, which is actually truer to what the product is. But of course, each of those units is bigger than the ones that were before. So it actually requires you to lead in a slightly different way once you start grouping that many, much of the product and the people together. I guess like two related questions from your background, I guess number one is since you were acquired into Shopify, one of the things I've been very impressed with by the

company is a degree to which it's been able to retain and grow the rules of founders that it's acquired. Because as far as I understand it, the majority of the management team at this point was acquired in at some point, or at least a very large fraction. And then secondly, as you buy a lot the things, you end up with a lot of disparate products, disparate platforms, disparate integrations, and all sorts of things can go right or wrong based on that. And so I was curious a little bit about the broader concept of integration, number one, from a team and individual perspective as a founder coming in, and why is it such a good home for founders? And the number two, like, once you acquire the tech and the product, how do you integrate that as well? Yeah, it's really

notable. So, like, when I look at the, so the, the sort of exact team of Shopify contains a number of founders, some of whom were acquired in and some of whom just were founders. And when I look at my management team, the people who report to me, who each of these sort of nine or ten people run R&D orgs that are like a couple hundred people each, and almost like a very high percentage of those people are also founders, right? And so you ask, okay, well, why are all these people ending up essentially running the main parts of the product of Shopify and Shopify's a product first company. So that's sort of the same thing as saying running Shopify. I think a lot of this comes from Toby himself, who has extremely strong opinions and has, I think, learned for the better

to no longer be shy about, like, saying, I want to aim in this direction, right? I think there was like a big culture of bottoms up decision making, power to the edges, like, whatever you want to call that school of management philosophy, which is like, you know, delegate and empower. I think a lot of former founders are actually coming back to the place where they say, hey, what am I uniquely good at? I'm uniquely good at, like, having a point of view, being willing to smash my head against a wall and like do whatever it takes to manifest that point of view in the world through building things, building teams, you know, saying the hard thing at the right moment. And I think Toby has kind of re-embraced that, right, in the past few years

of saying, like, hey, if I'm good at one thing, it's being extremely opinionated. And it seems that in this domain, at least for Toby, in this domain of like the internet and commerce, I'm right more often than I'm not. Like, I have a higher batting average than usual. Therefore, if I can help accelerate this company through like being decisive, that's going to be good. And so he sort of sets the example. And a lot of us, you know, see that. And we're like, hey, we remember being founders. We remember what it's like to have all bucks stop with you. And if there's a way that we can apply that level of decisiveness here and accelerate teams through like the otherwise like, you know, designed by committee bullshit that kills a lot of

companies, then like that's a net positive. Right. The other piece of buying And companies is sort of integrating different systems and platforms. And often you see people do one of two things. They either let things run independently. That was what's out for a long time at Meta, for example. Or you integrate in all the infrastructure. And usually that makes things more performant. You can have features that cross over, et cetera.

But the flip side of it is sometimes you see these giant projects that just stop a company from functioning. Like at Google, there was a famous sort of identity layer that was built. And for a year and a half, people just didn't ship things, right, in other areas. And so how do you all think about integrating and acquired companies and acquired technologies? But more generally, as you build and build and build a build, there's a big munch of stuff. How do you sort of consolidate that down in a performant way that's actually speedy to execute? And then I guess later on, maybe we could talk about whether AI has played a role in that or not. Yeah, it's a really good question.

The answer is it depends, like most problems. Usually when you encounter these problems, you sort of have to look at the stack. you usually don't want duplicates at the same layer of the stack, right? And the funny thing about this is sometimes you actually don't notice the two things are in fact at the same layer of the stack until much later on, right? So as an example, inside of Shopify there is a, there's the checkout, which takes, you know, someone's shopping cart and says, okay, let's take the business rules of the store and convert that to an agreement for a sale with taxes and duties and shipping

and all that stuff. So that's one engine. And then at different points in Shopify's history, other things that have been built that are sort of like adjacent to that, but not quite the same thing, right? So like a thing for creating like an invoice or a thing for like even editing an order after it's been placed. And a lot of these things kind of get built because you start with a small problem. Example. I just need to be able to add a discount to an order after it's already been placed because, you know, someone calls up and says, hey, I forgot to put the discount code on. Can you please enter it, right? And so someone starts a project over here, which is, oh, we're just going to support editing discounts onto orders. And you don't realize that what

you're actually doing is creating a second version of the negotiation engine, right? It's just that you started with a very, very small part of it. And then that thing kind of grows for a few years and you're like, oh, it's not just editing discounts in. It's also editing the shipping costs. It's also editing another product onto the order that they forgot to add. And before you realize it, you're like, oh, my God, we're reinventing the checkout over here, right? And by the way, in the meantime, all the teams that work at Shopify are finding themselves having to build to support both of these things, right? So someone builds a new feature and they're like, I'm going to add, I don't know, international pricing. And then that team is

like, oh shit, I have to build it into the checkout, but I also have to build it into that order editing thing. Oh, my God. And I also have to build it into that invoicing thing. And now it's like, to your point, it's like, okay, now the thing that should have taken a month to build is now taking six months to build because the layer of the stack that's underneath me has five things where there should just be one, right? And so sometimes it's hard to know what to do, right? Because sometimes the right answer is let's collapse those five things in the stack into a single thing, which is it sounds like what Google tried to do. But then you're like, oh, my God, it's like trying to take the foundations of a house

and like rearrange them while the house is still on top. And like that's, oh, God, this is going to be really hard because I can't just make everyone move out of the house, right? The real answer to how do you solve this problem is one is developing extremely good intuitions for when you are accidentally creating duplicates at the same layer of the stack. so you just never get into the problem in the first place. And then the second part is being willing to actually notice when you have created the duplication and eat the vegetables of saying like,

oh, shit, whoops, we did it. Okay, now it's time to eat the vegetables and collapse these things together because if we don't, every team that exists above us in the stack is going to pay this duplication cost, like in perpetuity. And the problem is it's not even just your RR. R&D teams that pay the cost, the saddest thing is that usually it's a combination of your R&D teams and your customers paying the cost because the reality is that not everyone will do the thing of building the five versions. They'll probably build like the top two or three most

important ones. And then all the customers who depend on numbers four and five will go to use the thing and they'll be like, oh, this is super weird. This thing doesn't work. Why is this thing so different to the other thing? I thought they were the same thing. And then they look under the covers and they're like, oh, I know why it's different because actually this part of the stack is all fucked up. Have you approached that from an AI perspective? So I think like Shopify was a very early, very early adopter of AI. And in general, whenever there's like a really sharp technical founder driving a company, they've been an earlier adopter, right? And Toby obviously fits that mold. But it's an area where I think you kind of have to play around with it,

experiment, et cetera, to really understand the capability set. But at the same time, to your point, you want kind of centralized infrastructure approaches. So have you thought about how to do that adoption as well as how have you thought about that in the context of launches like sidekick and Shopify magic and things like that? I mean, yes, this is obviously the space where everything's moving the fastest, right? Like, the world of invoicing is not moving at the pace of the world of AI, right? Depends on what you're invoicing, but yeah. Yeah, it's been an extremely interesting last 12 months of like looking at where can we

applied different models to parts of Shopify, I think the high level idea is, you know, Shopify's real focus and real mission is enabling entrepreneurs, right? And the meta strategy of everything we do is how do you simplify the process of starting and running a business so that more people who would otherwise not get over the hump, get over the hump and have an opportunity to try building whatever that business is. And like the historical strategy for that shop was build really easy to use software. Use software that is super simple when you start and it kind of reveals complexity as you need it. Like as you business grows and you have harder problems like the tools sort of like appear at the right time. I mean the world is littered

with the corpses of, you know, the companies that tried to serve SMB and enterprise at the same time. And this is kind of the hard problem of Shopify, how do you be simple but also powerful and scale up? AI is this really interesting moment where it's like, okay, well, Shopify built its position in the market largely by being the easiest to use thing as you build a business and scale, but in imperative mode, right? It's the, I'm going to give you all these switches, and we're going to give you the best switches. And we're going to show you the switches at the right moment in time, but this is the best switches, right? And AI brings us into the world of like, okay, well, what if we can give you the driver in the car who knows how all these switches work and can help you along without having to learn all the switches? It's probably one of the most powerful opportunities for more people who would otherwise be daunted by having to learn the switches.

Even though we try to make the switches as good as possible, there's all these people who can't quite figure out the switches. but they could talk to the, they could talk to the co-pilot and they could explain what switches they want hit, but they just can't do it themselves. And so I think we have an amazing opportunity to increase, like, the amount of entrepreneurs in the world who actually get a shot, right? And so that's the kind of headline. And then from there you go into, okay, well, which models do we apply to which problems? You know, you look at individual problems like, okay, how many people get stumped just writing the descriptions of their products? Because they're like, they're not good copywriters. And they're like, oh, this is embarrassing. I can't write a good

description for my candle. I'm not going to launch my store, you know. And there's so many instances like this where it's like, okay, well, can you give someone that extra bit of skill that gets them over the hump where they're really to hit the big green launch button, you know? Yeah, that's really cool. Glenn, it's really exciting how many obvious places there are to apply that to make it easier for the merchant or the would-be merchant. But I think one of the things that has struck me that you and Toby and the team has done is actually shipped a lot of that quickly when a lot of very large organizations are like, oh, that makes sense. The outputs are non-deterministic by nature. We have a process for measuring and evaluating quality of our products. Generally, that process does not apply. How did you

get to this is good enough and we can ship it? Well, one of the principles that we currently hold, This might change in the future if confidence intervals go high enough, but one of the principles that we have for all of the Shopify magic features today is that they're allowed to propose changes but not commit changes, right? So they can generate text, but they're not going to save it without you actually reading it and saving it. We might suggest a reply for the user that's, you know, someone writes in like, hey, what's your shipping policy? We can like suggest the reply based on us knowing what's in the store and like running that through an LLM.

But you have to hit enter, right? And so one of the things that, like, human is in the loop essentially right now is one of the ways that we're kind of mitigating risk here. And obviously, human in the loop is great because it gives you the feedback cycle. You actually get a three-part signal. You get which suggestions are accepted clean, which suggestions were accepted, but then with minor edits, and then which suggestions were outright rejected. And that's an amazing loop to be able to improve things through.

So we're always getting better, but that sort of, human in the loop, human must-click save thing is a big part of the strategy. Do you think that Shabahai has a different, like, risk orientation than other companies of its size? You still need guardrails, and even if there's suggestions, like, it's not, it's not perfect, right? Correct. Yeah. I think Toby and Kaz myself, we're all, like, fairly risk hungry people. I mean, I think that's just a little bit of founder culture, is like, you want to take risks like you don't really want to be safe you get bored you get annoyed when things get too safe

but there's always this balance of like we like taking risks that are risks to us we don't like taking risks that could like break someone's business and so if there's a way to do a thing that's like very risky for us but but is the risk is mitigated for the actual merchant like we're very for that and i think that's where that kind of human in the loop human must click save thing is is part of that, right? And who knows? Like maybe a year from now, maybe two years from now, like we get to like, you know, hey, you're, you seem to be accepting our suggestions without edits 99% of the time. Do you want to just go into like full auto mode? Like maybe we get there eventually. We're not there yet, but like you can sort of see that the hill climbing will eventually

take us somewhere close to that, right? Can you just describe for listeners some of the stuff that you think is coolest to add additions, if that's, you know, ImageGen or semantic search or even any of the foundations work. The foundations work is the thing that I'm most excited about, but I'm a commerce nerd. So, like, this is going to seem like very boring to the audience, but, like, just humor me for a sec. It's a pretty nerdy audience. So it depends. Yeah. The data model that is the most, most, most central to Shopify is the products data model, right? Like, how do you represent the products that you're selling? And Shopify has for, you know, basically like 10, 15 years had the product's data model hasn't changed that much.

It's like fairly simplistic. It's actually an amazing testament to how big of a company and how big of a, you know, the fact that you can get to like 10% of all US e-commerce on like a fairly simplistic data model is actually kind of amazing. But, you know, like Shopify has been a little bit on the weak side for dealing with like very large and complex products. So products that have a lot of options, a lot of colors, a lot of sizes. When it gets complex, we don't do as well.

And so we're updating our data model with support for just a much larger number of variants. And again, this is the nerdy part. This is one of these crazy things about tech. The reason this is so hard is because it forces you to go from unpaginated to paginated APIs when you start making the variant counts very large. And it's just a breaking change for like the entire app ecosystem. And it's just breaking changes are hard. Right. So I'm really excited about just the fact that we're unlocking that data model.

We've actually also embedded in not just like, hey, you can have more variance and more options, but there's also now a standard product taxonomy that comes with standard categories and standard attributes so that like when a merchant creates like a t-shirt, we're going to auto recognize, okay, this is in these standard category T-shirts, and that category comes with all of these standard attributes. And we're going to use AI to one, detect what category it should be, and two, detect what the values should be. So we're going to try and guess, okay, seems like the colors here are green and yellow based on the images you uploaded.

Seems like it's made out of cotton based on the description you had. And the great thing about that is when that merchant takes those products out to their own storefront, all of that data is going to make the search experience better. But more importantly, when they take those products out to like Google and Facebook and Amazon and all those places, having. that structured metadata around their products makes them way, way, way more discoverable, which is going to lead to them ranking higher in search results and basically making more money. So this is one of those weird things where like getting the data model right and having

the data quality be very high actually causes these very like important effects up funnel for the app for the business. So that's really cool. We are also releasing the first version of our image editing with AI thing. So you can take product images, you can replace backgrounds. Basically, you can take product imagery and get to a very, very, very professional standard on it without having to go through

very expensive photo shoots. Basically, a lot of what we do is like, how do you help someone who's like actually just, you know, my mom at home trying to start a business? How do you help that person look? like as professional as like the world's largest companies and then um yeah what else am i really excited about in the addition i think you just mentioned something sarah now i'm having a mental blank uh i mean i just have like i'm a search nerd and so i think like really yeah so here's a

crazy thing so i just learned what lbd means like a little black dress exactly i didn't know what lbd meant until recently um but uh very soon shop-fi stores will actually understand the meanings of things as you. So today's Shopify search is very, very literal keyword searchy. There's a video of me going around internally being like infuriated when I go to, there's a store that I actually buy things from and I went on there and searched for, um, LBD. Not LBD. I search for a sweater. And because that store, uh, only lists their thing as sweatshirt, I got like zero search results. I was like, I can't, this cannot still be true in 2023. Like this is insane that. this is happening. So that's a really easy example. But like, you know, if you go to a store

now and they haven't put little black dress in any of their keywords, if you search for LBD, it'll actually do the right thing. You can even do things like, I saw a crazy example from the team yesterday. They were like, Christmas themed shoes. And it would actually correctly go and find all the red and green shoes in the store. So that's kind of an edge case example, but it's like, you know, searching for things like something to wear to a wedding, something to where to the beach will actually do the right thing, whereas before would only would never do the right thing unless those keyboards happen to be in the product descriptions, which will never happen. So that's, I think again, that's one of those things where it's like, how can this

search experience on someone's tiny little storefront actually be as smart as like what you would get on Google? That's really cool. And are you doing like rag or embeddings or specific technical approaches like to that? Yeah. Yeah. So a lot of what we have to do is figure out what the right embeddings are to use, how to fine-tune the models. Like, obviously, there's categories in Shopify that are more, like, apparel, fashion, gift, homewares. So spending the time to assemble the datasets and do the fine-tunings and pick the right embedding models that are best for this category.

But it's also been really interesting with, like, especially some of the multimodal models that have emerged in the last, I don't know, two or three months. experimenting with how much weight to give like the text descriptions versus the images versus the taxonomy attributes and figuring out what mix of those things generates the best results is I mean we're still working on it honestly but it's a pretty cool and exciting space that's really cool yeah because I guess like a lot of people increasingly I feel are adopting gbtv or the vision model which allows you to upload images or understand or interrogate them And so it seems like there's a lot of information just resident in like visual imagery

that people aren't really making use of that now you can translate into a textual understanding and therefore tie that into everything else. It's associated with the merchants. That sounds very exciting. And it's an amazing thing for the buyer experience. I mean, like to your point, like the buyer experience is really cool when you search for things to wear to a wedding and it does the right thing. But actually think about it from a merchant experience point of view, if you're like,

hey, I've been running a store for five years and I've got like 10,000, products in there, and now you want me to go back and backfill like 10,000 products worth of five attributes for a product, like Jesus Christ, being able to actually apply the models to do something that turns out to be like a 90% correct guess is an amazing way to help bootstrap people into like the current moment, you know? What do you think is still missing from a capability's perspective to substantiate your perfect AI world for Shopify? Well, look, I mean, I think the exciting and frustrating thing about LLMs right now is that you can get an LLM, like, agent thing, like a sidekick-esque thing.

You can get it to like 75% in like 10 minutes. And then it's like this brutal hill climb to get it to like 95% like over time, right? It's just like catnip for like hackers because they're like, oh my God, I just 10 minutes. I got to a thing that's like pretty good. And then you're like, yeah, yeah, wait, buddy. Wait, wait for the next bit. The next bit's real interesting, you know. Yeah, the less 5% I think is the next five years of tech or something.

Right, right, right, right. And the irony is, like, you know, people talk about, they're like, oh, yeah, this LLM's hallucinating right now. And you're like, yeah, yeah, you know that's its job, right? You know that every time it emits a token, it's hallucinating, right? It's just that you like some of the hallucinations more than the others, right? And so I think that's been the challenge. for us. Like, it's even in the name, right? Like, Sidekick is, it is supposed to feel like

your companion on the journey, like your coach, your assistant, you're the person who's maybe seen, seen this movie before a little bit. Getting it from 75% to 95% is the process we're working through. And it's, and it's so exciting because, you know, we're doing stuff on our side where we improve the training sets. We, you know, we're building more and more conversations with feedback from users that help us dial it in more. But then every five seconds you look around and like there's a new model that you can take all your training set and all your evals and then it might be another step function. So it's kind of innovation really rapidly happening both inside and outside our R&D team. And you really know, you never know what

each week is going to bring really, you know. And you can be pretty optimistic in that like especially if you look at any one of these problems, I think there's a lot of focus on the core model capability and there should be. But like you're like, oh, well, embeddings models are getting better, right? And people are working on like synthetic data generation tools and working on tooling for the entire rag pipeline. And so I think it's, yes, like you hit the pain of reality of like, oh, no, I have to go beyond the demo. But a lot of other people are working on some enabling stuff. So I still expected a lot of the experiences to get better fast. counting on it from Shopify. Yes. Yeah, yeah. Well, I mean, we're at the cutting edge, you know. Yeah. Is there anything external to Shopify that you think is especially interesting right now in the AI world, be it startups or things that people are working on or projects or...

This is literally every single person has probably said this, but I think the, like, the rabbit thing at CES was pretty interesting. I mean, I'm literally wearing the shirt right now. Like, I'm a teenage engineering nerd. so like I was, I was hyped on the hardware, but I think one of the interesting problems with, like, LLM-based applications and agents in particular is, like, what's the actual interface they're interacting with, right? Like, in the case of Sidekick, right, there's a couple different places. Like, what is Sidekick using, right? Is Sidekick actually using the admin API under

the hood? Or is Sidekick actually reaching into the pixels of the web app? And, clicking around in the web app and doing stuff, right? Like, that's a pretty important question, like, what is the interface that the agent is actually learning and interacting with? And I thought the really interesting part of the Rabbit presentation was that they decided to treat the actual GUI as the interface and to try and, like, have a model that became very good at interacting with essentially web apps. If it actually works, the strategic brilliance of that is.

is they instantly have the world's largest app store because the world's largest app store is just the web, right? Yeah. Now, who knows if it'll actually work, but it's an incredibly interesting strategy. I was talking to Jesse from Rabbit about this. And I was like, it's a very,

it's a controversial, interesting strategic decision to say, like, I'm going to interact with the applications themselves versus use APIs. And his last company tried to unify actions on APIs. and his view was very much, like, well, there are implications on the, like, as you said, reach and ability to use different types of data and, like, whether or not you own your destiny if you are relying on those interfaces. And so, as you said, if it works, that's super interesting.

Yeah, and it's, even philosophically and from first principles, it sort of rings true to me, right? Because, you know, a lot of these experiences are, especially with, like, when I actually think about the sidekick UI, the sidekick actually runs sidecar in the admin next to you, right? Even the name sidekick is like, hey, I'm here with you. We're both here together and we're both using this Shopify thing to try and build your business, right? And so the idea that sidekick would literally be interacting with the pixels in the same way that you are, it's sort of almost like smells right in a weird way. And I mean, obviously, the nice thing about it is as long as the model can actually learn the UI, then every time you ship

updates to the UI, if the model's smart enough, it's just like, oh, yeah, my smart buddy keeps, it also can use this new button that just appeared today, right? So yeah, I think it's an interesting approach. We'll wait and see what actually happens. I also know people in the industry who are like, yeah, yeah, it's a cool idea. It's going to be insanely hard to pull off. But who knows, again, this thing changes every week. The other thing I think is kind of interesting, which is actually like Shopify relevant is, I guess I think, I'm curious what you guys think about this, but like the sort of the triangle of like perplexity, Google chat GPT of like, it's basically like LLM native search and like chat GPT really good at the LLM interface thing, not amazing at

search yet, Google amazing at search, not really figured out the LLM thing yet, perplexity is kind of squarely in the middle and saying like we think LLM native search is a really interesting like hill in the middle of this thing. I'm sort of watching this from the sidelines and kind of interested to see where it plays out. I think embedded in that strategic tussle is the question of what do people actually want from search? Do they want hard facts? Do they want opinions, how much lossiness are they willing to tolerate in order to get the compression of the LLM took 10 search results and summarized them for me. But maybe they hallucinated the summarization, right?

It's a really interesting place. Like, what do people actually want, you know? Yeah, I think it's a really interesting open question. And related to that, too, I've started to see the degradation of some performance of some other really early players in the market where there's a lot of qualifications or safety or you start to see the LLM go out to try and gather information. And it just really in some cases either slows down or makes the information worse because to some extent the reason you're interrogating an LLM is you want to get to an answer.

You don't want a web search or you don't want that, you know, poor synthesis of bad data from the web. You want the good synthesis of bad data from the web. And so yeah, it's been fascinating to watch and to your point, I think a lot of people have start to adopt perplexity for that specific use case because it has that middle ground of some form of IR plus LOM in a traditional sense. So yeah, I agree with you. It's going to be fascinating to watch how all the directions this goes. And I think the reality is people almost forgot that people from search, they actually want an answer. They don't want to do a search

process. They want to get to a result in many cases or a list of results. It's almost like people forgot the user need in some sense. Yeah. And it's funny when like, I mean, some searches are so specific, but some are almost like shared knowledge of humanity. So it's like one thing I found myself doing a lot like a year ago when I was looking at, okay, how can we improve search at Shopify, both on the storefronts but also on the shop app and just kind of playing with this idea. One of the tests that I found myself running over and over again. So at the time, I actually had this happen. I had a friend's daughter was her birthday. And I was like, Like, she was, I think she was six at the time.

And I was like, I want to buy a great classic book for a six-year-old girl. And as an experiment, I would go to Google and I would, like, say, classic children's book for six-year-old girl. And, like, the search results you would get back on Google shopping would just be, like, terrible, like absolute disaster, right? And then you would go to Amazon type, same thing, and you would get, like, nonsense back, right? You went to chat GPT and you said, give me 10, great. children's books for a six-year-old girl, what would come back would actually be amazing.

Like, it would be, like, really, like, the best 10 recommendations, right? And so I was like, huh, there's something here, right? Now, like, chat GPT, terrible, if you ask it for, hey, I want a pair of Nike AF1s in size 10.5 in a store in New York City, but really, really, really good at the general knowledge question of, like, great children's books, you know? Yeah, it's funny because I worked at Google years ago. as a Twitter or one of the teams that worked for me initially was search. And, you know, you tend to segment these things as you undoubtedly do a Shopify and a types of search, right?

Is it navigational? Is it certain types of information that you're looking for? Is it something else? And so, you know, you could imagine that in the LLM world, you effectively want to do almost like one boxes like you have at Google where you trigger off a certain types of keywords or phrases and therefore you end up with a different result. And you should be able to, especially have integrations with search engines to be able to serve the, I want these specific Nikes and where should I get them, right? Because that's almost like a form of navigational query in some sense relative to a commerce objective versus just give me some knowledge.

So it feels to me like a lot of stuff will come as long as people don't actually think that LLM has to do everything unless it can also function as a router or something. So it's really fascinating to think about. Yeah. And then that's when you get into the world of like things that are like matters of fact and things that are matters of opinion, right? Like is this shoe a Nike is not a matter of opinion, right? Whereas, is this book great for a six-year-old girl, that's a matter of opinion?

And how do you construct search systems that can actually sit correctly at the midpoint of the world of facts and the world of opinions, right? I think also going back to Alad's point on use cases, like one version of this, like from a scenarios of the future perspective, is it fragments, right? Sometimes I want facts and sometimes I want opinions, especially opinions like my own already. I mean, like a date on how people consume media definitely looks like that, right? And so I think it's very easy. I mean, I was an investor in a search company called NIVA. I think very highly of the perplexity team. I think it's very easy to look at the chokehold on distribution that Google has through its partners and say, like, that's very hard.

But we're at this very, I think, special moment where the technology and perhaps the user behavior, the expectation if we take out like slices of it where the expectation is very different and get people to think about pieces of search. Because it's not all clear to me. It should have been one market or it's permanently one market, right? You guys probably think about owning commerce search. Right, but even within commerce, there's so many different, like the way people search for clothes and the way they search for industrial parts are like not the same thing at all, right? So it's, even within that world, there's so much nuance of the way that people express. what they're looking for.

Glenn, awesome. Thank you for being here. That was great. Thanks, guys. Appreciate it. Good to see it. Thanks so much. Find us on Twitter

at No Pryors Pod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-dashpriars.com.

Thank you.