No Priors: Artificial Intelligence | Technology | Startups - How do we go from search engines to answer engines? With Perplexity AI’s Aravind Srinivas and Denis Yarats

Episode Date: March 23, 2023

With advances in machine learning, the way we search for information online will never be the same. This week on the No Priors podcast, we dive into a startup that aims to be the most trustworthy place to search for information online. Perplexity.ai is a search engine that provides answers to questions in a conversational way and hints at what the future of search might look like. Aravind Srinivas is a Co-founder and CEO of Perplexity. He is a former research scientist at Open AI and completed his PhD in computer science at University of California Berkeley. Denis Yarats is a Co-Founder and Perplexity’s CTO. He has a background in machine learning, having worked as a Research Scientist at Facebook AI Research and a machine learning engineer at Quora. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Aravind Srinivas on Google Scholar Denis Yarats on Google Scholar Perplexity AI Perplexity AI Discord AI Chatbots Are Coming to Search Engines. Can You Trust Them? - Scientific American Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @AravSrinivas | @denisyarats Show Notes:  [1:46] - How Perplexity AI iterates quickly and how the company has changed over time [5:46] - Approach to hiring and building a fast-paced team [10:43] - Why you don’t need AI pedigree to transition to work or research AI [14:01] - Challenges when transitioning from AI research to running a company as CEO & CTO [16:50] - Why Perplexity only shows answers it can cite [19:33] - How Perplexity approaches reinforcement learning [20:49] - Trustworthiness and if an answer engine needs a personality [23:05] - Why answer engines will become their own market segment [26:38] - Implications of “the era of fewer clicks” on publishers and advertisers [30:20] - Monetization strategy [33:20] - Advice for those deciding between academia or startups

So for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite. It's not just that we want to retrofit citations into a chatbot. That's not what perplexity is. In fact, it's more like a citation first service. This is the No Prior's podcast. I'm Alad Gail. We invest in, advise, and help start technology companies. In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

With advances in machine learning, the way we search for information online will never be the same. We're back again to talk about the future of search. This week, on the No Prior's podcast, I'm excited to introduce our next guest from Proplexity, AI, Proplexity, it's a search engine that provides answers to questions in a conversational way and hints at what the future of search might look like. Arvin Shrinivas is co-founder and CEO of Perplexity. He's a former research scientist at Open AI and completed his PhD in computer science at UC Berkeley. Dennis Jarets is a co-founder and Perplexity CTO.

He has a background in machine learning, having worked as a research scientist at Facebook AI, and also as a machine learning engineer at ORA. Arvin and Dennis, welcome to the podcast. Thank you for having us here. Thanks. Oh, yeah, thanks so much for joining. So the two of you alongside Andy Kowinski created perplexity around August or so of 2022. Arvin, do you want to give us a little bit of a sense of why you started this company

and what the core thesis of complexity is? Yeah, sure. Actually, Elad, you're our first ever investor who offered to invest in us. So those were the founding days. In fact, I remember the first ever idea we talked about in Noi Valley where we're sitting in the open space opposite Martha and I was telling you it would be cool to have a visual search engine

the only way to disrupt Google was to not do text-based search but to actually do it from camera pixels and you were like this is not going to work you need to think about distribution and search was always the core motivation for me and

Dennis and many others of the company we were just bouncing around ideas and then I was still open air around the time and then we left and Dennis also was still at Meta, then he also left, and Andy came in to help us sort of incorporate and get the company rolling. So that's sort of how the company started. The space of LLLN's was exciting, generative models were really exciting, and in general

we were motivated about search, whether it be a general search or vertical search, and we were bouncing around several different ideas. One of the ideas that you gave us was working on text to SQL. And we were pretty excited about that and started prototyping ideas around that. And I think Dennis also was hacking with us on building like a Jupiter notebook extension with like co-pilot for every cell. And then we were trying it with SQL around databases. But it's all like a bunch of nonlinear pathways to eventually get to where we are right now. Yeah, absolutely.

And hopefully I caveated whatever feedback I gave with. I'm probably wrong. but since I think I'm off and wrong on directions. No, I think whatever you said still applies. Search is tremendously a distribution game as much as a technology game. Yeah. I think one of the really impressive things about perplexity is the rate of iteration. And to your point, you've gone through things like text to SQL, copilot for the next-gen data stack.

And I've always been impressed by how rapidly you've just been able to point in a direction, iterate really fast, prototype something, see if it's working and then move on to the next thing. And to your point, you always had search in the back of your mind. I remember even as you're prototyping these things, you were talking about indexing aspects of Twitter or other sort of data feeds and then providing search on top of them, how did you end up building a team that can iterate that rapidly as well as a culture of fast iteration,

like other specific things that you all do as a team to help reinforce that? Yeah, I'll take the first part of this question, and then also that Dennis answered this because he's a big part of why this is happening. We both are basically from an academic background, So in general, the culture and academia is to, you have hundreds of ideas and you just need to try them out pretty quickly, run a lot of experiments really quickly and get the results and iterate. So we come from that background, both of us. And so that's not really new to us. It's just that when it comes to trying out in products, it's not just a result you get from running an experiment.

You actually have to go to users and make them use it and talk to companies or customers or potential people in a company who will be using. your product and get feedback. So there's that aspect of operational work that needs to be done to get results for experiments, and there's this aspect of quickly doing engineering to get it to a state where you can show it to people. So both of these things had to come together, and that's why the company exists. And Dennis is incredibly good at engineering and recruiting. And so we found our other co-founder, Johnny Ho, who's like a big reason my perplexity

operates this fast. and full credit to Dennis for helping recruit him. He was the world number one at competitive programming. It's like being Magnus Carlson of programming. He got pretty excited about all the stuff that Dennis was showing him. So it's sort of like few people help you accelerate a lot, and Johnny is like one big reason for that. And Dennis sort of continued doing that by getting more and more people in the full.

Are there specific things, Dennis, that you look for when you're hiring or when you're screening for great technical talent? because, I mean, the way one person put it to me is that you have a team that can build almost anything really fast. And so I'm curious, like, are there heuristics, or there are places that you look for people? Like, how do you think about hiring? Yeah, that's a good question.

I'm kind of like looking for the people who maybe have like this general interest in like applying this emerging technology like LLM, right? So like a lot of people hear about it. Maybe they don't have a specific background, but they're like very curious to learn about it and they want to put like a lot of energy into it. So I feel like that's my number one. indicator. So, like, personally, I would rather, like, get somebody who has this burning

desire to work on these things rather than somebody who already has a lot of experience and not going to put a lot of, or, like, as much effort as other people. I think another concept that we still using to this day, even though it takes a lot of time, it's we have kind of like this trial periods, I would say. So we basically get to know a person a little bit better. And once we see there's like good chance we might hire this person. We then ask a person to sort of like work with us for like some time. We kind of like intend to keep our number of employees very small. I feel like it's much faster to operate. And because of that, each hiring decision is very important, you know, and we want to like optimize for the best people. And that's

why we kind of like run this trial process where, you know, we basically making very sure that we want to work with this person. So I think that's been helpful. Is there anywhere you've been surprised as you've made hires where you feel like, you know, the signal was wrong or the trial surprised you? Yeah, there has been a few exceptions. Obviously, I think that's the part of it, but I feel like it's definitely, at least like compared to my prior experience, it's definitely smaller chance where you're going to get surprised. Normal interviews that, you know, big companies run, you know, you have like four or five meetings. like for 45 minutes each, and then you basically make decision after that.

Sometimes it works, sometimes it doesn't. But I feel like the way we do things, it gives us much more confidence that we're going to make right decision. And I think it's obviously, it's very useful for us to get the signal. But I also feel like for candidates, it also makes sense to, it's useful to make better decision as well, right? So they can understand, do they want to work on things like we do? do they want to work with the pace that we do? I think, like, one important thing for us,

and like many candidates sort of like don't want to maybe do this is the energy we're putting into perplexity. It's kind of like work-life balance, maybe not the ideal, but that's the only way to sort of like beat competition and sort of like iterate very fast and do great things. So that's why we kind of need to have this alignment at the beginning. Yeah, if you don't have, have a clear idea of, like, you know, which product you're going to build or which market

you're going after, and you still want to be giving yourself a shot at success. Eteration speed is the only thing that you can hope for. And so we decided, okay, like, until we actually figure out our business and product, we'll make sure that this is non-negotiable. Like, people should iterate really quickly with us. Yeah, I feel like the only real advantage that a startup has relative to incumbent is speed, right? because they come in has more people, they have more money, they have more distribution, they're more product.

And so really the only thing you have is speed. And the early days of mobile, you know, I was working at Google and I really helped get that team up and running. And then I started a company that Twitter bought in a Twitter, there was this weird meme internally that the only people they should hire for mobile were people with prior mobile experience, which I thought was a really dumb idea, right? It was more, you just find great engineers and they'll figure out how to write for iOS and stuff like that.

How do you think about that in the ML world? Because there is this sort of dogma right now, I feel, that people believe that only somebody who's trained in LLM before can work on an LLM center company or things like that. So I'm just sort of curious, you know, how you think about that. Yeah, actually, my thinking about this was already pretty clear because of having seen how Open AI went about this. If you look at the people who did GPD3, eventually they went on to start Anthropic, all of them are basically from physics background. The CEO, Dario is a physics PhD, and Jared Kaplan is a physics professor. He wrote the scaling law paper. So they basically, open AI, really succeeded in bringing these extremely talented physics people who wanted into machine learning.

And they came into the sort of language models and scaling. And that paid off well for them. And similarly, their whole engineering team, if you look at an engineering team, is just Dropbox and Stripe people. All like software engineers, solid people who can build infrastructure front and. and now they're doing AI work. So it's always been clear to me that in order to work in AI, both research and product, you do not need to already have been in AI.

And that's being shown clearly with Johnny Ho, our co-founder. He was not in AI. He was a competitive programmer, a trader. He'd worked at Kora for a year. But he's as good as anybody can get in picking up new things. So the other thing also is that LLMs are sort of in this weird territory where the people who use the LLMs for building stuff, understand it better than the people who actually did gradient descent and trained these models.

Like, you could find a PhD student at Stanford or Berkeley who would know a lot about how to train the model, but they might not be the best person to build a product of that. Yeah, I want to quickly add to this point. So I was at early days at Facebook Eye Research in Melo Park, the office, right? And at that time, and that was honestly one of my reason to do PhD as there was like kind of this very exclusive culture. So if you don't have a PhD, you're not going to be research scientist. And I didn't like that too much. So that's why I decided to do PhDs later on. But it turns out, through my experience, the best research scientists also very good engineers, like very good engineers. And

we've noticed like through deep mind and open AI is just like the companies that made the most progress over like last six years or five years were companies where they're like extremely good engineers and it's kind of like I didn't like from the beginning this I guess view from like academics that's just like if you engineer you're probably not you know either like smart enough or you're not going to like great things but turns out it's actually the other way around so that was also think like motivation and I feel like you don't need to be you know this like very impressive like academic with PhD to do great things yeah this also goes back to the thing Dennis said earlier, that you want to find the people who really want to get into AI rather

than who are already in AI. And every company that's gotten big has done this in their early days, including Google. Like, they got a lot of systems people. Jeff Dean was a compilers and systems person. And then they got, and Urs Roslow was like a systems professor. They got all these amazing people and they told them, hey, you know, guys, like we're having the most interesting computer science problems to solve here. and we can scale it up, so come work on search.

So it's not like you have to go, there are few people they are hired from information retrieval or search background, but most of the celebrity researchers that they have right now are just people who wanted to get into that space for the first time. It's funny because now the desired pedigory

isn't the PhD from whatever academic lab because the industry labs filled with physicists and engineers and people not necessarily from the domain made the most progress. And so if you want somebody who's worked on, I hear the argument, you don't actually need it. Like you want people who are really smart and motivated.

But a lot of companies are looking for somebody who has, you know, experience with X billion parameter training runs. And those people come from now, open AI and deep mind and such. So it's quite funny very quickly how the pedigree has changed. What's been the area of steepest learning curve with both of you coming from these like research engineering backgrounds? For me, it's like how to run company that's mostly what I'm doing here. I'm not doing much with core engineering.

Dennis does that. So that was not easy at all, but I've had the opportunity to learn from many good people, including Elad here. So if at all there is an easy way to do it, it's like getting advice, rapid advice from people. The other thing is also like when you're making a mistake, like being super brutally honest with yourself and listening to feedback. and quickly course-correcting it. That was also very new to me. Yeah, I guess for me, probably it was building a team, kind of like organizing everything in terms of like who does what,

how to prioritize things, what to work on. I think like being a small team, it's even more important to identify this one thing that you want to work on and like put all your focus on it. Early on, we were like had this sometimes made this mistakes that we'd shine to go after like several things. that's why even though we're like six months or like seven months old company we actually built so many things we had like you know twitter we had like sales force integration we have like hubspot integration

like many other things that we never like released one of the interesting thing is just like you have to have very precise focus and just like go after it yeah also as leader if you want to lead the company and if you cannot do everything the people will lose faith in you, right? They think you don't really have any clarity and that's why you're making them do one new thing every week. So you have more responsibility to sort of get it right and think more clearly. And even if you're wrong, like, do one thing wrong at a time and course correct rather than doing five things at once and seeing like which one wins. It's difficult for us coming from the academic background for this particular thing because in academia you're basically taught to hedge. You have like one first author project and like three or four co-author project. and one of them might become a big hit and might change your career. Whereas that's not how you should do startups. You really have to focus and iterate multiple times

on one thing or few things. So that took us a while to quickly learn. Yeah, makes a lot of sense. I think everybody goes through very similar paths as they start a company. One thing that I think is really interesting right now is we're at this point where consumers are really starting to wake up to how

machine learning and AI is changing search. And to your point, you all were thinking about this actually before chat GPT, and before the Sydney News and before Bing integrated, all these things. So you were quite early to realizing that

this is going to be a really core piece of search. Perplexity's mission, I believe, is to create the world's most trusted information service. How do you think about important product points

around factual accuracy, bias in presenting aggregate information for users and things like that? Yeah, so I guess you're also from a PhD background, right? So when you write your first paper, the thing your advisors teach you is you only have to write things

that you can actually cite. Anything else that you write in the paper is your opinion, not a scientific fact. And so that sort of stuck with us pretty closely. And that's sort of why we did the first version, where it's citation-powered search. So for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite.

This is a pretty subtle point here. It's not just that we want to retrofit citations into a chatbot. that's not what perplexity is. In fact, it's more like a citation-first service that it'll never say anything that it cannot cite. So if people have tried to play with it, as if it's like Chad GPT, where like, tell me who are you or like things like that.

And even for those questions, it would still go to a search engine, pull up stuff and come back with an answer. It's not going to say, I'm perplexity, I'm like a bot design, how are you doing or something like that? This is because of our obsession about factual accuracy. Like, even if it doesn't have a personality or character in it,

we don't care. we only care about the other thing, which is obsession about truth and accuracy. Yet, the second point you mentioned about aggregation of things, when you mash up multiple sources together, you might end up hallucinating. Like, for example, if there's multiple people who are the name Elad Gill, and like one of them is a venture capitalist and some others like a doctor, or let's say even if it goes to your own LinkedIn and things like, oh, the Elad Gil who did a PhD in biology is a different a little bit from the venture capitalist because someone might think that, right?

It's pretty unusual background. So then it might end up coming up with some entertaining or funny summaries that collate different sources together. We still haven't thought of a proper fix to this, but one thing is obviously as language models keep getting better, they're going to understand these things, these subtleties even better. And we're already seeing that. And the second thing is we are giving users the power to remove sources that they think

are irrelevant, just like how you can curate sources in Wikipedia. So we're sort of working towards this accuracy and the bias issues step-by-step at a time, but I feel like it'll take more iterations to give this truly correct. And I also don't think one LLM will just magically solve this problem. You need to build an end-to-end platform where users can correct the mistakes of an LLM. And that also means you need to design the platform where the incentive is right for the user, because this could also be used in the other way where users can use it hide information. So we haven't really thought through all these issues.

thoroughly, but we are committed to sort of figuring these things out over time. That makes sense. I guess in addition to that, or maybe related, you've done an impressive amount of research and reinforcement learning. What's unique about the way the perplexity uses reinforcement learning, and how does it tie into these plans? We like RLHF, like reinforcement learning from human feedback, where we use the contractors to, we collect feedback from the users on whether they like the summaries, the completions or not, and like, we use contractors to do the some ratings themselves. And these days, even LLMs can be prompted to do

the work the contractors do. Anthropics written a paper on that. So all these things are getting really very efficient to do. So that's sort of how we have been thinking about reinforcement learning right now. But we haven't gone beyond that to think of like agents and browsers and things like that. We'll probably focus more on the first part for the next at least six months to a year. One out in this aspect, full-blown like RELHF is definitely something we can look into that. But there is like several many steps that you can have in between.

that significantly can increase your quality. So, for example, I mean, even using something like a rejection sampling, you know, like discriminator on top, it's kind of, you want to like shift, you know, maybe you have several samples from your LLM and then you can rank some of them using different model peak only dose. It is in a sense kind of like one step of reinforcement learning, but it helps a lot. It's very effective. You guys have talked about how trustworthiness is more important to you than like. I don't know, personality, the ability to play with a bot.

Do you believe in chat as an interface? Like, where's the line between chat and search, given perplexity, does support, for example, like follow-ups and things that are more conversational? We think chat UI is the future. People are using it pretty heavily. At the same time, if you can try to get the answer right in the first attempt, you should, right? Like, you have a responsibility to save people's time.

It's not like Google doesn't do some kind of chat implicitly. Like, if you go to Google, you always get related questions, follow-ups, people also ask for and things like that. It's just sort of implicitly making you click on it and, you know, like you get a follow-up question that you sort of get an experience without the chat UI. So I feel like it's more like whether it makes sense for the particular experience you want to provide for your end user. And in our case, it does. Often you do not get the answer you want in the first attempt and you shouldn't feel the burden to get it also. Like sometimes you could ask a question. question by just asking for the keyword, and then you might realize, like, okay, this is actually

what I wanted to ask for. I've seen this in live and people use a perplexity, but they just couldn't know how to phrase this question the first time. So they use multiple attempts to get to the right question. So there are things like that, whereas the questions get more complex, the chat UI makes a lot more sense than Google's UI. But for like obvious questions, like, if you just wanted to know whether it's going to rain in Bay Area the next one week, why do you need to keep asking more, right?

Like, you just get the answer to the first attempt. And we want to support both these experiences and perplexity. More broadly, I'll ask a dangerous question. But what do you think the future of search looks like, right? Five years plus that. Do we still have monolithic, horizontal providers if the players change? Do we get more embedded apps? Like, contextual search as a feature in different places.

Are there agents that do things for you? Like, what do you think it looks like? I think there's this phrase that's becoming popular. I think Carpathie was the first one who tweeted it. and Satya Nadella is also using it. It's called an answer engine instead of a search engine that directly tries to answer your question instead of providing you a bunch of links

or just snippets from the first link. So we believe in that. Perplexity is the first conversational answer engine. Truly the first, I think nobody built it before us. I believe answer engines will become its own segment, market segment. Just like you have a default search engine, there will be a default answer engine over time,

if these things really work. and the burden of getting ranking and search right will reduce in the sense answer engines can do more heavy lifting than search engines. As these things get really good and way fewer hallucinations, and even if they do hallucinate, people can still go and click on these links, they will eventually prefer this experience over the regular 10 links or 20 links UI that Google has. So that's something I'm pretty confident about. And I think the sort of asking follow-up questions will become more of the norm. The number of queries in perplexity that go to at least one follow-up has been increasing ever since we released the chat UI.

So that will keep going up. People will get used to the sort of experience where they're encouraged to ask one more question and they're okay with not getting the answer right away. So that will happen. The third thing is like actions. People will be more deliberate in what they search for and try to execute an action on top of the search results they consumed. So that's definitely

likely to happen. It's already happening. If you go to Google and book a flight, you just fly from SF to Seattle, you just directly click on the book button. So that's going to happen more frequently in the chat UI too. And this will become an assistant more than just a search or answer engine. And I also think the fourth thing is

there will be some sort of like much fewer traffic to the actual content site. Like very few links need to be consumed. In perplexity, in fact, we don't even cite more than five links. It's a deliberate decision. A lot of people ask this, well, can you add 10 links or 20 links?

Can you just show all the links together? You put the summary at the top, but you also put all the 10 links, the usual, I want both. It's a decision just we just made that. No, no, no, that's not the right experience for you. You actually need to feel the difference here. So we only made only three to four links. In fact, the first version shipped it to like three to four links, I think, and like 50 word summaries.

And we expanded the summary more. So I think we just have fewer tabs open. We only open the tabs that we really need so that all this sort of behavioral change in the consumer is likely to happen. Whether it be through us or Google itself doing these changes, it's unclear, but this is where it looks like it's trending towards. When do you think we're going to have that transition from almost like a pull versus push model, right? Because I think right now you go and you ask for certain information and you may ask for it repetitively. I've been checking the weather every day versus a world where you have agents

that are effectively understanding your intent ongoing and providing you with an information and it's sort of a push-based way. How far away of a world is that? Is that a year away, five years away, 10 years away? I feel like we can do it now. In fact, like Google now was an attempt to do that.

If you keep checking for scores of a favorite football club, it'll automatically give you a push notification for their next latest match that you might have missed. They tried some of these things already. So I feel like we can do these things even better now with language models. So, yeah, I think it'll happen in a year or more than five years. And do you think it's going to be fragmented in terms of each? And I know this is all uncertain, right?

It's kind of predicting the future is never correct. But I'm just curious how you think about, is there going to be like an agent for your Google drive or an agent for your GitHub or an agent for your email or is it just sort of consolidated eventually into one central service? I mean, it's getting pretty clear, you know, with Google, the first few results, like at least like my pattern of using Google, I see like first two, three results I basically skip most of the time because it's here or like some ads. And that's like not ideal experience obviously for the user. And it's also kind of like why it might be very hard for Google to

fully launch this system like Answer Engine because it just like breaks miniatization strategy for them. As Arabian mentioned, you know, there's going to be like fewer clicks. And Google wants you to click on those things, right? So that's how they make money. Basically, I think there has to be this like new paradigm where you kind of like, you get your answer quicker, but then maybe you can monetize it better on it. Maybe now like you can help user to make your decision faster and then you can show like much more targeted and accurate ad. So then you know, like if user clicks it, it's going to be maybe they're going to buy something or whatever. whatever. So there is going to be few clicks, but each click is going to be more expensive.

And I feel like overall user experience has to be better because you just, you just get what you want. You don't have to do any extra effort. In this era of fewer clicks, right, if the summaries in chat that's just giving you the answer, how does that impact the relationship of search or answer engines with publishers, right? Does that remove the incentive to publish information on the internet? Does it become more adversarial? How do you guys think about that? Probably not. I feel like, so it's sort of like whether you cite a paper or not sort of thing, right? Like you would cite a paper if the paper was really good. It's actually going to bring back the whole concept of page rank even more. But the concept of page rank was inspired by

academic citations from what I read. Like a very important paper tends to get cited. So when you're in the sort of citation-based search interface now, the better content of a publisher has, the more likely it will get cited by an LLM, unless humans figure out answer engine optimization or LLM optimization, which I hopefully they don't invest effort into. But in general, it's unlikely that it's as easy as SEO with just keywords because LLMs are going to be much smarter in understanding relevance to a query. So I think it's just going to incentivize people to publish higher quality content

in order to get cited by an LLM-powered answer. like substack or things like that try to do like you want to own your content and you publish it and make sure it's high quality and you have your own like set of subscribers so people put a lot of effort into that more than writing tweets so something like that is likely to happen with this interface too but it's unclear exactly how to make all this monetized at scale like you know the click-based ad engine that google has and i think it's super interesting problem and it's amazing that many companies are trying this at the same time so even if one company figures us out, like, others can also, like, benefit from it.

If you look at a lot of the biggest consumer services, they took a while to figure out monetization. So, for example, Google's first attempt was literally to sell search results. So they got paid per thousand search queries that they'd syndicate to others. And then they built out an enterprise appliance, a literal piece of hardware that you'd install at enterprises to do search inside the enterprise. And then eventually they came up with ads and really realized that that was a future path for monetization.

And then similarly, you look at YouTube and people said that, would never make money, and Facebook would never make money, and all these things would never make money, and then, of course, they monetize eventually. How are you thinking about monetization, or is it simply too early, and it's more about just getting market share, and then, you know, at that point, you can iterate on the right model. Yeah, we have multiple thoughts about this. Obviously, just like Google was paid to sell the search results for 1,000 queries.

Like, I mean, Bing has APIs like that, too, and people ask, at least more than thousands of people have emailed us or messaged us in various different ways to ask for a perplexity as an API. And we haven't done that yet, but that's an obvious monetization strategy. And the other thing is obviously prosumerization of this, where like we're already sort of beginning to see our Chrome extension pick up rapidly, heading towards 100,000 users. And extensions like Grammarly have the sort of free version and the prosumer version, which has more features in them.

So they're based to do that through the browser extension as a productivity assistant. sort of thing. We already see some kind of search pilot. Every time you're on a site, you can ask you to do things for you. And then there's the whole as we keep getting more and more traffic onto our site, like say hundreds of millions of people come to

our site at one point eventually, that becomes a right ground for serving ads. But we need to not make the mistake that Google did of combining ads into the core search product itself and figure out an alternative pathway like Facebook did. And that

might work out better for us. Subscription-based search has been tried by other companies, and that's something that Chad GPT is also trying. So we don't know yet if it's high margin enough. And so if a bigger behemoth like Google or Bing

just put out the same or like even 80% as good as you for free, then you're never going to make it as a subscription product. So we are likely to stay away from that pathway, but we don't know yet. And the final piece is like if perplexity becomes like something that a lot of people want to use

for their own internal data, their links or their bookmarks or their company. And if we can make it easy for them to build that and become more like a platform which everybody can use, then that's likely to lead to monetization too. So there are like so many different rollouts possible here that we don't know yet which one we'll actually go for. But in the short term, we are more focused on growing the product, the users, the traffic. In fact, improving the whole experience. Like this,

I feel like Google and Microsoft will pretty much do the same thing we have right now as well as us. And as we discussed in the first part of the podcast, we need to operate with more velocity, ship more things, and stay ahead of them in terms of the core value

of the product itself. Yeah, that makes a ton of sense. And I think, you know, to the point before, there's probably lots of paths to monetize once you have a lot of usage. And so it's more just, you know, figuring out what's native and natural to the product. You mentioned earlier that one of

the things you learn from academia is fast iteration. And I feel like most academics I've worked with are almost the opposite. You know, like I actually feel like there's a lot of pre-planning and a lot of discussion and there's, there's less of a bias to action. And so I've been very impressed by the bias to action you all have. What advice do you have for researchers who are now deciding between an academic path or joining a company research role or starting a company? What advice would you give them, given that you've gone through a recent transition that's similar? Yeah. Firstly, we both share, Peter Abiel is also Dennis' thesis advisor and my advisor, and he's a pretty different academic from the others. He pushes people to get results pretty fast. So that's

a reason why we both are like that. And Dennis also worked in industry where you can't operate that slow. So for the advice to academic researchers, I feel like it's super hard to be an academic researcher right now, especially a PhD student is one of the worst jobs you can go for in a time when AI is so hot and so highly paid and you can do a startup or join a startup, you have to sort of give up all that and the buzz every single day you see on Twitter or other news journals and still focus on trying to build a future. It's more of a mental thing, I would say, more than like picking the right problem, like even having the composure and the maturity to sort of stay poor and like work on hard problems. It's super hard right now. Very few people in the

work and have the ability to do that. To be very honest, I only came to the United States for doing a PhD because there was no other way for me to come to the United States. Like, I couldn't take a loan for a master's or something like that. So PhD is fully funded and like sponsored. So that was the biggest reason I actually wanted to come here, more than doing a PhD. If I had the ability to get a job in industry, I've gone for that. So there's other reasons you might want to do a PhD, but okay, if all these things are sort of not a problem for you and you still want to do something, I think it's best to look for alternatives to the transformer, alternatives to like language models, that sort of radical directions than trying to improve

them because there's so much incentive for the existing companies to do that. A lot of people think open air has no incentive to improve GPT, the core architecture or like the model or something that's far from true. Like even if they have a lot of money, they would want to make it more efficient and train even bigger models that make better use of compute. So I feel like it's best for people to sort of look at places that are kind of controversial or radical. And that would mean even questioning Transformer itself, which is actually one of the best

research problems to work on. That's what I would work on if I would do a PhD. I would work on trying to write the next Transformer paper. I fully agree with this. I think I would probably want to also at least like from my experience. I think it's best to not go to PhD right after undergrad and kind of like spent at least a couple of years in industry. I think that goes to my point to me,

like the best researchers are those who are also very good engineers. Like you can get this like valuable experience of being, you know, a good engineer and then it's basically going to like propel your PhD. You can do things faster. And especially, you know, like now AI becomes a little bit more like engineering than it used to be, right? So it's just like, this skills are essential. You won't be able, especially if you want to work on like LLMs or like large scale stuff. You have to be very good engineer. But yeah, if you still want to stay on like very academic

side where you just like think like very deep ideas, then I agree with Arabian. You probably want to completely ignore LLMs and just do something very radical. There are some other ways to do stuff that's still useful and pretty different. I think Stanford has some students doing this like state-based models and flash attention. There's some really good papers like that coming out. And a flash attention is already being used to improve the efficiency of LMs. So you can sort of do such things or you can go work on video or video generation, stuff like that that's just still out there. That makes that sense.

Well, Dennis Narvin, this was a great conversation. and it's really exciting to see all the progress you're making on an important area. So thank you so much for joining us today. Thank you for having us here. Thank you for having us. Thanks. Thank you for listening to this week's episode of No Priors. Follow No Priors for a new guest each week and let us know online what you think and who an AI you want to hear from.

You can keep in touch with me and conviction by following at Serenormis. You can follow me on Twitter at Alad Gill. Thanks for listening. No Pryors is produced in partnership. with Pod People. Special thanks to our team, Cynthia Galdea and Pronav Reddy, and the production team at Pod People, Alex Vigmanis, Matt Saab, Amy Machado, Ashton, Ashton Carter, Danielle Roth, Carter Wogan, and Billy Libby. Also, our parents, our children, the Academy, and Open Google Soft AI, the future employer of all of mankind.