This Week in Startups - Inside Harvey AI’s $8 billion AI lawyer app, PLUS How OpenRouter unites the LLMs | E2207

Starting point is 00:00:00 Hey, everybody. Welcome to This Week in Startups. This is Alex. And today on the show, we have a couple of of Amazing Twist 500 interviews for you. And then at the end, Jason is going to jump on and do a couple of founder questions. So first up, Gabe from Harvey AI, you have heard of this company. It's worth a lot of money. It's growing very quickly. I wanted to learn more about why the law is such a ripe place to apply AI today. Then we were talking to Alex from OpenRouter and amazing startup. We have used their data on the show a number of times. I was really curious, though, what's the business behind the hype? We learned that and more. And then, as I said, at the end, Jason's going to pop on and answer a couple of questions.

Starting point is 00:00:35 All right, let's go. This week in startups is brought to you by Lemon.io. Hire pre-vetted remote developers and get 15% off your first four weeks of developer time at lemon. com. Uber. Bad data equals bad AI. Your AI is only as good as the data at least. learns from. Uber, that's right. Uber AI Solutions now works with enterprises around the world

Starting point is 00:01:04 to source, label, evaluate, and scale real world high quality data for every industry, everywhere, so that you can focus on building the next big thing. High quality data equals smarter and faster AI. Uber.com slash twist. And interpret. Interpret turns feedback noise into customer intelligence so your team knows exactly what to fix and build next. Head to interprets.com slash twist to book a demo and see it in action. Hey, everybody. Welcome back to this week in startups. This is Alex, and we are joined today by an entrepreneur and a founder that you have

Starting point is 00:01:39 heard of. The legal profession has become a hotspot for AI innovation. Lots of startups are raising money and growing very quickly. But at the absolute top of the heap, there's a firm called Harvey. You've heard of them. I've heard of them. I've long wanted to learn more. So please welcome to the show.

Starting point is 00:01:54 It's Gabe Pereira, the co-founder and president of the company. Gabe, how you doing? Good. How are you? I am actually quite good. I'm a little bit shocked, though, that you're not suited up. I thought anyone who interacted with the legal field had to pretend like they just stepped off an episode of suits. This is what Winston and I thought when we started the company and really early on, we had a customer event and we bought suits and showed up. And they were like, can you guys please just wear hoodies?

Starting point is 00:02:19 And so that's why I'm in a hoodie. you. Before we jumped on, we were joking. We were going through the fundraising history of the company, and you guys most recently raised $150 million from Andresen, a billion dollar valuation. And I joked, you know, did you buy a boat? I didn't think you actually had. But you let me know that you have a unique sleeping arrangement right now. Yeah, my mattress is still on the floor. So we got to, we got to fix that at some point. You really don't actually. I mean, you seem fine. Company's doing great. So I don't see any issues there. For folks out there who don't know what Harvey does, I think we should start with a little bit of kind of just brass tax.

Starting point is 00:02:56 So the way that I understand the company today is that it has two major components. One is a thing called vault that allows law firms to bring their information, their data, case history, just all of their stuff to the platform. And then you have a tool called the assistant that uses that information and does a lot of work for people doing drafts, that sort of thing. Is that a fair summation of Harvey today? Yeah, I would think of where we're at today is I think kind of how cursor built this ID or didn't build the ID but kind of added all this AI. I think the big challenge for legal was there didn't exist this kind of single workspace where you could work on these in legal. They're called client matters. And so a lot of what we've been building in the past three years

Starting point is 00:03:37 is how do we build a single workspace where either a lawyer or an agent could perform a client matter, whether it's litigation, an M&A end to end. And I think the challenge with that is Lawyers use kind of a wide variety of tools. You need case law. You need data rooms. You need e-discovery platforms. And so we're starting to partner with a bunch of these folks. And then we've built some of this functionality.

Starting point is 00:04:02 But I think the more or equally important piece is it's not just building the tools for individual lawyers. One of the massive challenge that these large law firms have is how do you manage all of these different client projects? And so if you're one of these large law firms, you're doing tens of thousands of client projects. matters, you have millions of historical client matters and you need to make sure all that data is separate. And so there's a massive part of the product that I think doesn't get seen as much is all of the governance and admin controls you need. And then another thing we're working a lot on is a lot of this work is not just how do law firm lawyers collaborate with each other, but how do they collaborate with their clients? And so now a large number of our customers

Starting point is 00:04:47 are clients of law firms and we're building a lot of functionality for these two parties. to work together on client matters. So if I'm a law firm and my client is also a Harvey customer, I presume there's kind of a bridge between the two that lets Harvey kind of talk across the sides. But let's say that I'm, I don't know, a private equity firm and my law firm doesn't use Harvey and I do. Does that create added friction or does that still help the overall legal process?

Starting point is 00:05:11 Yeah. I mean, I think what we're seeing, so when we sell to enterprise, I would think of there's two large things that our enterprise customers are using Harvey for. there is a bunch of internal legal work that these enterprises do that they typically don't collaborate with outside counsel. And so things like contracting and other internal legal processes. And so we're building products there. And then what we're seeing increasingly is when these large private equity firms or enterprises by Harvey, they're starting to ask their law firms to also

Starting point is 00:05:44 buy it so they can work together. And so I think we're starting to see a bit of this network effect from that. Really push your effective customer acquisition costs down, or is this more like an edge benefit that hasn't changed the economics of sales? Yeah. Oh, I think we've seen this make a huge difference. Like there are deals we get. And we see this actually in both directions where we see law firms going to their clients and saying, hey, we can work better with you if you buy Harvey. And then we have inbound from enterprises saying, hey, we want to buy this so we can work with our law firm. And then we see it in the other direction where we've seen some large enterprises buy this. and then reach out to their entire panel of law firms and say, hey, can all of you purchase Harvey

Starting point is 00:06:22 so we can collaborate? And so I think this is actually quite a big effect. That's fantastic. I mean, everyone wants distribution. And in the world of selling to large companies, it's very hard to get in the door, let alone to get something closed. But I presume that if your law firm's client is saying, please use this, that probably reduces the time to sale by a factor of 10, if not more, just for you guys. Yeah, exactly. All right. Now, I want to talk about the use of AI inside the legal profession because we all know that it works well in coding because there's a huge corpus of online information. There's tons of code you can pull in and learn from. My thought was that the world of legal stuff is a good fit for AI because it has its own

Starting point is 00:07:01 kind of grammar to it. It's structured in a particular way. There's a huge kind of corpus of case law and legal writings out there to pull from. Is that correct? And is there something that I'm missing that makes AI a good fit for AI. Yeah. Yeah, no, I think that's exactly right. Like, I think there's a ton of analogies between code and legal, and these feel like the two domains where you've really seen these types of companies be able to be very successful.

Starting point is 00:07:26 And I think it's for the exact reason you said. And then I think the one thing I would add is a lot of the job of these associates is just synthesizing massive amounts of information. And so a lot of times when you get staffed on a litigation or a merger, if you're a junior associate. It's not just, can you understand all the legal context? It is, oh, I'm representing this pharmaceutical company, and I actually just don't know how pharmaceutical supply chain works, and I need to like understand that and then understand how that connects to historical litigations. And so there's just this massive, like, learning problem.

Starting point is 00:08:02 And I think what we see now with things like Harvey is these associates can use web search to go understand the industry and then they can use Lexus to do this research and then they can connect it to here's a discovery corpus with all the emails and then use all of this to start drafting like a motion or something like that and so it's these really complex workflows across all these different data sources that you need to synthesize and you guys recently announced some sort of deal with Lexus right yeah so we have a partnership with them for for case law and so we're both we're working with them to kind of provide this in our platform and um kind of kind of of expand this as well.

Starting point is 00:08:41 The road, though, to applying AI to the legal profession started off pretty hot. There's a funny anecdote. I think it was somewhere on OpenAI's own blog, how you guys used GPT3 early on to do landlord-tenant questions from Reddit's legal advice subreddit and found that quoting, for 86 out of 100 questions, attorneys you talked to said they would just send it on. But then later on, OpenAI also said that you guys also tried things like fine-tuning foundation models and using RAG to try to bring more information in there. to AI models but found that that was kind of lacking. So it seems like there was like

Starting point is 00:09:13 initial fit between existing AI technologies and legal questions and then you ran into some complications along the way. So can you just kind of like chart for me how that progression is gone? If you're not listening to your customers and iterating based on their feedback, your startup will fail. So if you're drowning in feedback from support, reviews, calls, it's time for interpret. Interpret is the customer intelligence platform. that helps you turn random comments and feedback into actionable insights that help grow your company. Their AI system is already trained on your business and your product. It then reads all your support tickets.

Starting point is 00:09:51 It reads all your reviews, call transcripts, and surveys. Then their agents actually update your team in Jira, Linear, Zendesk, and Slack. So this feedback doesn't just sit in some email or messaging thread, but it actually helps keep your team moving the ball forward. Find out why Hanva, Notion, and Perplexity are already using. interpret, stay on top of what they need to fix or build next. So if you're ready to turn feedback chaos into customer intelligence, head to interpret.com slash twist to book a demo and see it in action. That's E-N-T-R-P-R-E-T dot com slash twist. Yeah, so I would think of the big difference

Starting point is 00:10:28 there is consumer versus corporate law. And so when we started Harvey, Winston and I were really interested in kind of access to justice and could we use this technology to provide legal services kind of more affordably but for consumers and the models at the time this was before GPD4 were already quite good kind of like you said of if you took a lot of these hey I got evicted here's my lease what should I do like so much of this is online that the models were quite good and and I think the challenge is there's a bunch of regulatory issues around unauthorized practice of law things like that that I think it's something now we're starting to work with, for example, the Supreme Court in Singapore, and so

Starting point is 00:11:13 governments and court systems to see if we can apply this technology, but I think the core business now is more corporate law. And that's where the challenges are much more difficult because most of that data is not public, right? If you think of what these very large law firms are doing for corporations, it's I have a massive litigation. I want to do a very large acquisition. most of the data you're working there is very sensitive client data that actually never goes into the public models. And so a lot of that is missing. And then the thing that you mentioned we did with OpenAI that I think is still a really big challenge is we are trying to make these models better at legal research. And so case law is semi-public.

Starting point is 00:11:59 And so the models have a lot of access to it. And I think what you see is their legal reasoning on things like Supreme Court cases is actually quite good. And my intuition is the reason for that is it's not actually because the Supreme Court cases are public. It's because there's so much public analysis of all of this data online, of all of these legal scholars talking about, here's the implication of this case, here's how it relates to this case. And so a lot of what you see the models doing is being able to interpolate between that. and then it looks like this very powerful legal reasoning. And then the issue we run into is when you try to extrapolate that to,

Starting point is 00:12:39 here's the more nuanced cases that are not Supreme Court cases, but a law firm would be researching when they're doing kind of a specialized matter for a client. There isn't that writing and that work around it publicly. And so the reasoning model, the reasoning of kind of the general purpose models falls off. And so I think we have this big intuition where a lot of the legal expertise is locked up in these law firms. And what we need to do is help them train these systems with their expertise. So each law firm can kind of put that into their own system. So you guys have a history with Open AI.

Starting point is 00:13:11 They were an early investor in the company. And then I think it was in May of this year, you guys added models from Anthropic and Alphabet to the product, kind of giving yourself a multimodal approach to the market. I'm trying to understand exactly what clients need to do. Do they end up training their own version of a general purpose model with their own data. Is this entirely rag? How does the architecture work to allow for multiple models, given the technical questions you just discussed? Yeah. So I would think it gets complicated because it depends on if it's a law firm or a law firm's client. And so I would think of in the

Starting point is 00:13:49 past three years, not just for us, but I think for most of these domains, a lot of the challenge is still not the post-training and personalization of the models. It's really connecting all of the context. And so part of why we've been doing all these partnerships with I manage net docs, kind of the document management systems of the law firms is it's still very hard to get all of the context that a lawyer needs into the model before you even start thinking of how do you customize this. And so the experience you want if you're a lawyer at a large law firm is you're always working on a specific client matter. And so you want to be able to query that client matter. And so if it's a litigation you want, here's all the emails, here is the e-discovery corpus, the case law,

Starting point is 00:14:34 our private documents and all the work products we're working on. And I want my model, my agent, to be able to interact with all of that. And so a lot of what we've been doing is how do you pull all that together? I thought model context protocol or NCP from Anthropics solved that for everybody, and it was a done deal, Gabe, and we were all done. I think it's a good abstraction. And then the challenge, I think, with all this enterprise stuff is it's almost never just a technical problem. It is, how do you connect all these systems? But it's moving there. So I think with law firms we're actually getting there,

Starting point is 00:15:05 where we are starting to be able to provide this experience where an attorney has all of that context. Once you have that, the next challenge is we can't train on any of that data. And so we're fully eyes off. We never use any of our client data. But even more so, the law firm can't do that. Because if you think of the data that the law firm has, it's actually client data from a bunch of different clients. And so the infrastructure we need to build for law firms is how do they partition all of that data?

Starting point is 00:15:38 And then for a specific client, can we build the infrastructure where if you have a large private equity firm and you do all their fund formation work, like then as a pair, you train the system on that if the client and the law firm are both okay with that. And so we're starting to have some of those conversations. And then if you think of it from the client side, what the clients eventually are going to want is I have one legal model and it won't probably be a single LLM. There will be some complicated LLM system that has access to all of my company's legal data. And it connects to all of my different law firms. And it can learn from the work that all of my law firms and eventually professional service providers are doing for me. Because you can probably learn from how people use the product and service, what they're looking for, when they look for it, what type of documents, you know, constitute a successful search, even if you can't look at the actual information in those documents because of privacy, you know, client litigator, confidentiality and so forth.

Starting point is 00:16:35 That's a big problem, game, because I think most of the people can can more easily kind of just hoover up the information and use it. You guys really have to be as delicate as the healthcare industry, it feels like. Yeah. No, and I think this is something from when we started the company, like we brought on the CISO of Goldman, and we brought on a bunch of these security advisors because I think the really big challenge we're going to solve is for banks, for government. How do you build these systems in a way where you can provide that Gen A.I experience, but it meets all of these kind of regulatory issues. And I think even before you get into the Gen AI, there's already these massive technical security legal challenges. And then when you think of the balance of like we want to use Gen AI, we want to get some of the

Starting point is 00:17:16 training benefits, but you also have all of those issues. I think that is going to be a really hard technical product, business change management problem. So not to be a brat, I know there's tons more work to do in the legal space, but it feels like you're building a bit of machinery that could also be applied to other industries where data is quite regulated, but they still want to use AI. Is there down the road a copy and paste into a different industry possible, or is this going to be so law-specific that it actually won't translate, as I'm kind of imagining, to something else? Yeah, we're actually already doing this in some adjacent industries, and so we do this for

Starting point is 00:17:52 tax, for example. And so I would think of the problem we're solving and the model where this applies is what are all the industries where you have advisors to companies? And the big technical problem that you need to solve is, as a company, I need to share my sensitive data with this advisor so they can operate on that data. And this is investment banking, consulting, tax, audit, HR, compliance, maybe insurance a bit. And so I think this expands to kind of a bunch of these different industries. It sounds very complicated and tricky. But the good news is that growth has been really, really impressive at the company. You guys scaled from, I think it was 50 million ARR, kind of at the end of last year, up to 100 million error this August. And I know you guys

Starting point is 00:18:38 have, I think, around 500 law firms that are clients. So talk to me about who's buying today. Are they domestic international? Is this just big law? Because I've had friends that have worked a big law. I'm curious what the customer mix looks like today. Yeah. So it definitely started. Our focus was the largest law firms. And we actually started outside the U.S. And so we worked with a lot of the large European law firms. Now we're in over 50 of the Amlaw 100. So kind of the top 100 firms and kind of a combination of, you know, a lot of the really large U.S. ones and then a lot of the really large international ones, midmarket is growing pretty quickly. And so we're starting to sell to smaller law firms. And then Enterprise, kind of the clients of these law firms, private equity and Fortune 500 is also growing pretty quickly. It feels like the addition of clients of law firms greatly expands your TAM because each law firm has, actually I don't know the answer to this, dozens, hundreds of individual clients. tens of thousands.

Starting point is 00:19:37 Yeah, see, I was way off. But that's a lot more potential contracted value for the company. Do you think that's going to become the majority down the road? I think for us, the interesting question is figuring out how to balance these parties. And so one of the big problems we want to solve is there is inherent tension of selling to both sides of kind of this marketplace. And what we're starting to figure out with law firms and their clients is the ideal win here is we can find a model. where for law firms, we can make them more profitable on this platform. And for their clients, they can also benefit.

Starting point is 00:20:12 And I think sometimes when people look at this, they say, okay, one of these parties needs to lose. But I think when you actually get into the complications of how these relationships work, we're starting to figure out ways that you can actually make this a win-win for both parties. And then I think to your point, it expands the TAM because once we go into Enterprise, there's stuff we are able to do for enterprises that is, not work they use outside counsel for. And so things like contracting, tax HR compliance.

Starting point is 00:20:41 So yeah. With AI, like everything else, the old saying is so true. Garbage in, garbage out. Without the right training data, you're not going to get great results. But my pals, that Uber, they're now working with enterprises all over the world to source, label, evaluate, and scale high quality real data for your startup. Uber is one of my favorite companies in the planning because they demonstrated their ability to scale and to build great products.

Starting point is 00:21:06 First in the ride chair market, then they figured out Uber Eats, and then they moved on to autonomous driving. And all of this was only possible because of a deep understanding of how to collect, label, understand, and analyze data. Now with Uber AI solutions, you can put the team and tools that turned Uber into one of the world's best companies to work for you, partnering with top talent and experts from Uber's Global Network. What better partner to help you scale your business

Starting point is 00:21:33 than the company that organizes over 36 million rides every single day. Book a demo right now by going to uber.com slash twist. That's ubeer.com slash twist. What is the tension, though? I'm curious about this because it feels to me that if a law firm uses Harvey and is more efficient and just better at their jobs because of it, their clients would be happy, not annoyed that they had this technology. And the other way around, if you could work more easily with your clients,

Starting point is 00:21:58 why is that a bad thing? So what's the tension you're describing? Yeah. I mean, I think the simplified version is kind of the billable hour where when you're charging for work on an hourly basis, there is some tension of the efficiency. But I think when you actually look at how these law firms operate, so like one good example I use is fund formation. And these aren't purely billed on billable hour. Like a lot of this work has caps and there is certain of the work that you have to do that gets written off. And so there's all these extra degrees of freedom.

Starting point is 00:22:32 But I think there is just a challenge, right? It's like if you think of that scale, if you are a very large company and you're spending $2 billion on legal, like these CEOs are giving mandates to their general counsels or their legal divisions and saying, hey, I need you to use Gen AI to reduce legal spend by 10%. Right. And so that obviously is going to put some pressure on the firms. But I think the part that's exciting is we've talked to law firms where they say, okay, maybe we're going to make a bit less on a client.

Starting point is 00:23:02 matter, but there's firms we're talking to where they say we can actually serve this diligence on a fixed fee or this litigation on a fixed fee, or we can, you know, take a bit of a cut here, but we can do that set scale with better profit margins. And so I actually, my gut of what's going to happen is you will see law firms that are 10 times larger than the current law firms, and they're able to operate at a much larger scale because I think Gen AI will let them decouple revenue from headcount. And so you'll be able to get closer to these software margins and kind of like scale beyond your headcount. So I think there is a win, but it's obviously a very challenging thing to figure out at this stage. I want to squeeze in a couple of quick ones about

Starting point is 00:23:45 the market itself. You guys, I think, told Fortune that your largest competitor is by far indirectly open AI and that as general model companies get better and better, you guys have to stay, you know, reasonably far ahead because, you know, you're doing something specific. How much competition do you think actually comes from the major companies versus the startups out there? Because, you know, Spellbook and Nexel and Lucille, there's a lot of companies that are coming after parts of the AI legal world. So who's really bigger in your kind of day today? Is it open AI in company or is it more of the startups out there that are extra hungry? Yeah. I think this is the challenge with building one of these companies is you have to operate on

Starting point is 00:24:25 all these different timeframes. So for example, on the day to day now, when we go into deals and things like that, especially with law firms, Open AI doesn't come up very much, right? Copilot doesn't. But some of the law firms use this, but it's really like we need a legal technology vendor because this is such a vertical industry. And then where you see some of it is when you sell to these very large corporates. And they're not using Open AI for their legal, but they are starting to think about how do we, you know, re-platform our company or connect all of our divisions using this. And I think that's going to happen, right? Like, you will, as a large company, you will have something like a co-pilot, a Claude, or a chatch EBT available to a large amount of your workforce, right?

Starting point is 00:25:11 And you will connect all of your data systems. And so we don't see the kind of the value we provide as a company as long-term, the individual assistant. But this creates friction as you want to, like, sell into these organizations. But eventually what we need to provide and is our value prop that I think will be very different than what the foundation models provide is like what we really need to sell to law firms is here is how you create a more profitable law firm. And it's more than just here is a product for an individual lawyer. It is here's all the infrastructure to transform your law firm into an AI first law firm.

Starting point is 00:25:47 And so it is the governance and the change management, the training, ways to deliver these services in new ways. And so that part feels like, you know, that's very differentiated. The horizontal players won't build that. And then connected with the enterprises, I think there is short term, and we had this historically with the law firms and it's gone away, the self-build, right? Where some in-house departments will say, oh, maybe I can just build this contracting solution myself. And my sense is what's going to happen is similar to what happened with Salesforce, where a lot of folks were like, oh, it's just the database table, I can put all my client data there. I'll just build that myself. And at scale, you really need something like Salesforce because of all the things you need to build on top of it.

Starting point is 00:26:32 And I imagine that's going to be similar of what happens with these in-house legal departments. Most of the, how do you optimize your legal spend is not going to be you deploy 10,000 super smart legal agents. It's going to be all the process management and the measurement connecting with law firms and things like that. And so I think that's the part. where we don't see too many competitors working on that, but I think eventually getting to that, there's friction kind of from all sides. And I think everyone faces that.

Starting point is 00:27:04 When it comes to bringing Harvey into clients, though, you guys are hiring forward-deployed engineers and investing a lot in customer success. I was just curious, as a final question here for other founders out there, how much work do you think AI-focused startups should be doing to help handle their clients, customers, and get them ready to go?

Starting point is 00:27:22 And is that amount of work going down over time as AI gets more advanced or is it still a pretty big cost for you guys? Yeah. We actually historically haven't had a large forward deployed motion. And so pretty much 100% of our platform is things that we have built that we are selling to every customer. I think like the way that we are thinking about forward deployed is as we go into new verticals, especially for enterprises, how do we figure out what are the right building blocks to put in the platform. So a good example is we are starting to build for private equity outside of legal. And a lot of these private equity firms want us to build essentially a fund operating system, right? Like a platform where they can manage all of their funds,

Starting point is 00:28:08 they can find all the documents. If they want to create a new fund, they can connect with their law firm. If they want to make an investment out of a fund, they can connect with the firm that does that. but then they can also do the investment diligence and things like that. And I think what we're finding is a lot of these tools that we built rhyme. And so a lot of the diligence stuff that we need to build for this is actually things that already exist in Vault. And so really figuring out what those abstractions are. I do think as we scale, especially for these enterprise customers, there is some of this you will need to do because historically, or the problem we hear from all these clients is just we don't know where our data is

Starting point is 00:28:52 both on law firms and enterprise. We have all these custom-built systems. We need to connect them. And there is some of this where you can't just go in and say, hey, just put MCP on everything and we'll just absorb it. Like, you really do need to sit down with these clients and figure it out. But it feels, it feels very scalable. Like, this isn't something that I think has been a big issue. And like, we've sold the same platform to all our corporates and all our big law firms. Awesome. All right. Well, Gabe, an absolute treat.

Starting point is 00:29:17 Thank you so much for coming on. I presume you guys will be a 200 million ARR by January, February, at the latest. So we'll have you back on when it's time. Harvey A.I. What's the website? And just for fun, is there a job you guys are looking to hire for you on a shout out to the world? All roles.

Starting point is 00:29:33 So, yeah, please apply. All right, Gabe. Talk to you soon. I appreciate it. Founders, let's be real. Finding the right developers is really hard when you're trying to run and scale your startup. But lemon.com. I.O. can save you time, money, and a ton of headaches. They've pre-vetted high-quality developers, and they've ensured that their experience result-oriented

Starting point is 00:29:54 and that they're seeking competitive rates. These are handpicked devs with a minimum of three years of on-the-job experience. And only 1% of applications to Lemon.io are accepted into their program. So forget the long, drawn-out hiring process. Lemmon's going to match you with a skilled developer in just a few days. And they're not just recruiting talent either. They're helping you integrate them into your team. And if anything goes wrong, Lemon will find you a replacement developer.

Starting point is 00:30:22 ASAP. We've had a ton of launch founders, work with Lemon.io. So go to lemon.com slash twist and find your perfect developer or a tech team in 48 hours or less. Twist listeners get 15% off their first four weeks. That's lemon.com.i. slash twist. L-E-M-O-O-S-T-E-M-O. and dot i.o slash twist.

Starting point is 00:30:41 AI is an enormous deal. Everyone wants to use it, but there is one startup out there that provides a unified API endpoint or hook, if you will, that lets companies pick and choose their models at will or helps them select. The company's called OpenRouter.

Starting point is 00:30:54 It's fantastic. I live and die by its data. So please join me and welcoming Alex Atala to the show, co-founder and CEO at OpenRrouter. Alex, how you doing? Doing great. Thanks for the intro, Alex.

Starting point is 00:31:06 Yeah, my absolute pleasure, man. All right, so open router, I think of it as a unified API that provides access to tons of different LLMs and inference providers. Is that a fair summation, Alex? Yeah, there's a new model every two, three days at this point. And most people are only familiar with one. So what we try to do is introduce the world to the wide variety of options out there. It's kind of learning that there are like multiple people in the world that you can work with as opposed to just one. I'm blown away.

Starting point is 00:31:37 And there are true nuances and advantages to exploring the diversity of language models and intelligence. So when we think about the models in question here, we're not talking about just, you know, Western closed source models, but also a deluge, if you will, of really high-quality open-source models from China. There's Mistrall over in France. Where else are we seeing new AI models pop up from? Yeah, there's a couple in China that are doing pretty well.

Starting point is 00:32:06 There's Kimi, K2 Thinking, which launched last week, Mini Max, M2, both really strong reasoning models. In France, Mistral, is really like major contender for European language models. We have in, you know, obviously in the U.S., like the heavy hitters, OpenAI, Gemini, Anthropic, and XAI. And we're starting to see a longer tail of mid-sized language models that specialize on specific types of tasks. Models are being used more frequently in long-running jobs. This could be like building something in the background. This could be using a bunch of resources. and these long-running jobs, they don't require deep knowledge about the world.

Starting point is 00:33:02 They require resourcefulness. And that's kind of a whole new skill and a whole new direction for, like, the AI economy to move in. Now, I thought it might be a good place to start by explaining how open router is different than Amazon Bedrock, Bedrock or Azure AI Foundry for folks who might be familiar with one particular cloud or inference provider. So for people out there listening, how do you guys compare to those and why are you better? Yeah, so the primary reason is that there's more options on OpenRouter, and you're able to observe everything going on in one place, regardless of where your credits are coming from. You can bring credits to OpenRouter, all of your existing deals, including things that you do on Bedrock. You can connect into OpenRouter and have it all show up there. So Open Router gives you a single pane of glass for discovering new models, discovering existing ones that you have access to. doing inference the exact same way through all of them. We normalize and standardize and fix errors with all requests. And we upgrade the quality of inference pretty significantly.

Starting point is 00:34:09 There are all kinds of paper cuts and ways to shoot yourself in the foot when you switch between providers or try to access a new model. And something we see very often is some company is struggling really hard with all models today on some problem. And a new model pops up and an engineer will be like, hey, I want to try this out and see if it fixes our issue. But there's a new provider. We have to like set up a new vendor. We have to do a new billing relationship. I know we have to do a new enterprise relationship. Does it have a data policy that like matches with all of our internal policies?

Starting point is 00:34:47 And a whole bunch of brain damage happens just to see if the problem is solved. So we delete all of that. and just make it so you can instantly test out new models and then use them very efficiently in production too. So I think a lot about APIs abstracting away complexity, much like how Twilio famously is the ER example, abstracted away telecom complexity. It just gave you a single API call.

Starting point is 00:35:13 It really feels like you guys are taking a wide number of options, providers, and so forth in the AI world today and really just making them incredibly simple and easier to consume. And my presumption there, Alex, is that if it is, easier to consume, easier to try more things, easier to experiment, people will wind up using net more total LLM inference or just more tokens, perhaps is the better way to say that. That is definitely a part of the thesis. Additionally, when you have more options, you kind of improve your leverage over the whole ecosystem. So, you know, if AI had zero competition,

Starting point is 00:35:53 There was only one company that mattered, or only one company shipping models. You know, there isn't a point to open router. But the more competition there is, the more like diversification of features that emerges, the more ridiculous it is for a developer to try to take advantage of all those features. And that's where we come into play. Like you get them all out of the box without you having to add complexity. to the work you're doing. Also, by making it easier to sample, to switch and so forth,

Starting point is 00:36:30 you also effectively increase the quantum of competition inside of the AI market. Because if I had to go to Open AI only and select them and live and die with them, I'm not really in a competitive marketplace. I have vendor lock, but you guys making it easier to swap between them. It makes your Googles and your Anthropics and your mistrales and your, you know, Kimmy's just more in competition with one another. regardless of where they're based and so forth. So it kind of turns vendors into just competitive blocks.

Starting point is 00:37:00 Like we also help bring up distribution for all of our vendors and all of our providers, though. So like a lot of the people who try out new models when they launched would probably never have done it had it not been really easy to just switch and explore it. So from the provider's point of view, you can think of us kind of like a distributor. where we're just helping you discover new users. And once a user gets really, really into a provider, sometimes we'll go to a direct relationship

Starting point is 00:37:31 and then connect everything in one place back on OpenRouter afterwards. But that way we kind of help people grow up and then scale up. Do AI model companies ever get a little bit annoyed that you're in a way disintermediating them from their customers? I mean, I know everyone wants inference, and so they're probably happy to serve the demand, but I would presume that Anthropic would rather have a direct relationship with Alex Incorporated versus me using OpenRouter's API to access Anthropic technology.

Starting point is 00:38:05 So in the early days, it was definitely a little tense because there was, it was, we were just kind of a strange new company. It wasn't a, it wasn't a category. and that was definitely, I think, a big question. And initially, our relations were on the 10th side. And over time, it's kind of gone completely 180, where now the model providers and the model labs, like, see the value in routers.

Starting point is 00:38:41 And it's now this kind of new AI product category. like routers are around to stay. We have like competitors too. And the product category like helps them get distribution, like I mentioned. It also helps them learn about their models. We do these stealth models where model labs will watch a model ahead of time and learn from the like our community and our power users and like the initial benchmarks that are runs on the model. And then we did this with GPT5, with GPD4.1, with GROC4.1, with GROC 4.

Starting point is 00:39:26 And currently a model called Polaris Alpha, which Alex, before the show declined to break confidentiality in his NDAs and tell me what it is. But one thing I did notice, just looking into its stats is that it's already seen around, I don't know, 30, 35 billion tokens process per day. I presume that companies come to OpenRouter with a model of, for some discrete testing because you guys have just a volume of users that will immediately tap into it and put it to work. That's a lot of tokens, Alex. Yeah, we have a good, I think, there's a couple communities that sort of compose on top of each other for OpenRadder.

Starting point is 00:40:01 There's this like LLM power user community that we really identify ourselves with. That's kind of how the company got started. And these are, you know, they test out all the models. They're building, they're often building things as side projects. And sometimes they're, they're like running internal e-vals at large companies. And they build a really good intuition for like when a new model should be used. We also have this, a wide variety of bring your own model apps. these apps, some of them you can see on our rankings page as well,

Starting point is 00:40:44 are apps where they want users to just be able to choose any model they want and have it work out of the box without the app having to change any code at all. And users love this because I think this is something the world doesn't quite get yet. Models are like content. Every time there's a new model, it's like a holiday. You know, there's a whole bunch of problems that might get solved. It's a new marketing moment for all the apps that suddenly can support the new model. It's like a new person being born.

Starting point is 00:41:20 And when that happens, people develop these like personal relationships with the model. And then they, they like understand the other things. And then they want to go to their apps and they want to bring that with them. And so the apps that support OpenRouter like that, can take advantage of all of those things for free. So that those also, that also helps the stealth models really understand the wide variety of use cases.

Starting point is 00:41:45 Because people have so many different things they want to use it with. Bring it to you guys, get a diverse use case set. It's not like 80 of your friends using it all for the same thing. No, that makes sense. On the venture capital side of things, you guys announced about $40 million worth of capital

Starting point is 00:41:58 raised across a seed in a series A earlier this year. I think Andreessen led the seed. Minload led the A. So talk to me about the business model. As far as I can tell, you guys charge a 5.5% platform fee for non-free access. Is that the main way that OpenRodder makes money? So it's, there's three ways. There's this credit top-up fee, which is, it's basically for off-the-shelf inference.

Starting point is 00:42:28 If you want to do volume with us, you can contact us through our enterprise page. get that fee down quite a bit. Okay. The other way we do it is by doing tons of volume on providers and, and just, like, working with providers to, like, do volume discounts. And then we often give those to users, too. Just to interrogate that. If you're offering, let's say, model X from model company Y at market rates, but you guys

Starting point is 00:43:04 negotiate a 15% discount, that provides really comfortable margin to open router itself. Is that what you're saying? I won't go into what the margin is, but that's like part of how we have a margin. That's right. And then the last component is that there's a whole bunch of inference adjacent software that people really want, things like observability, web search, like improves team management, the ability to like limit certain parts of your team programmatically, easy, like multimodal support, PDF parsing and PDF management, file context management,

Starting point is 00:43:47 all this stuff that like, it kind of like, it kind of becomes really powerful if you have it managed very closely with your inference. Those things, you know, we have like enterprise features that we charge for. And that's, I think, where like, the margin will shift over time. Okay, that helps a lot. Now, going back to AI models and such, you guys have hundreds of models on service. You have dozens of inference providers. And that has yielded a growth curve in terms of total tokens process that looks like this. If you're on the audio version, I'm sharing the OpenRouter rankings page. And as we can see here, Alex, pretty quick and steady growth across inference volumes. We're on track.

Starting point is 00:44:30 to set a new record this week. I think it's about $5.8 trillion tokens, 5.7, sorry. How quickly does this chart keep going up over the next 12 months in your view? Well, I think it's absolutely insane how underutilized LLM still are. There are, I mean, on top of just running into a lot of repetitive tasks that even engineers

Starting point is 00:44:57 continue to do today, I think the most exciting sort of glimpse into how many more tasks can be automated is the new types of training data that the model labs want. And they're really interested in figuring out how to be a product manager. What kinds of things do you need to know? How do you take a bunch of different contexts and distill it into action items? or like better ways of running agent workloads, understanding like somebody's calendar very deeply. And these tasks, like after a little bit of time,

Starting point is 00:45:41 they start to show up in the models. And what our job is, what we want to do is make that work visible to everyone. You know, if a model does a really good job managing your calendar, that should be clear in the ranking. using empirical database on how people are using it and then and also other signals that we kind of understand over time. So we're like, there's not that many categories that like model labs are known to optimize for today and it's just going to blow up. So I think the, I mean, I think it's pretty amazing how much more work.

Starting point is 00:46:27 there continues to be. Just doing that down there for folks who maybe are a little bit less head in this world as we are, you're saying that LLMs are going to get better at domain-specific tasks by better training by their makers. And so essentially, we're going to see LLMs perform better at different categories of knowledge work, product management, and so forth, and not just drafting legal documents and writing computer code. Yeah.

Starting point is 00:46:51 I mean, I'm not training models myself. and so I'm not the best person to ask about the exact ways they're going to improve. This is kind of my speculation and hope. But I think that's like a big signal that there's room to grow, the fact that nobody is tracking improvement on those things yet. You know, a key kind of muscle to build when you're entering the AI space is benchmarking. Everything is about themselves and benchmarks. And if you make claims without having an e-vail or a benchmark,

Starting point is 00:47:29 what are you really saying? And there's a lot of like, you know, there's just a lot of value to unlock by creating a benchmark. And it also guides the models in that direction when you do. So evals and benchmarks are very useful. There's other vectors you might use to choose which model, which provider, at which time. You guys built something called auto-router

Starting point is 00:47:52 that essentially does some of that work for your customers. I presume today it's an unbiased setup in which you guys are only selecting for the best option for your customer. But is there a way in the future in which people could sponsor that to get more people to try out their models for certain use cases? It's a good question. So far, we've stayed away from letting any providers sponsor more traffic. our goal is to be like a strong, neutral Switzerland-style gateway that gives everybody a fair shot based on real-world metrics and real-world benchmarks. But I do think that there will be interesting ways to shift people to new models in the future. We, for example, like, we do a little bit of model recommendation today, but very little.

Starting point is 00:48:54 You'll start to see that ramp up in 2016. You know, if you're 2026. 2026. I know we're getting old, but like, we have to at least get the tech hang right. You're good. Keep going. If you, yeah, yeah. Like, there are all kinds of workloads that are just like letting models that are way

Starting point is 00:49:17 too powerful and don't, you know, or you could save a ton of money. Yeah, you don't need to use GPT5 high if you can use, you know, 4.1 or whatever. Yeah, yeah, yeah, yeah. Exactly. And having the having like a really wide range of models and a lot of data about how they get used, I think helps there. But, but there's also, I think, good ways of, of like making very interesting model routers that we'll experiment with. It's kind of a good positioning for it.

Starting point is 00:49:51 In your series A announcement, Matt Murphy, one of your board members, said that he, quote, believes open routers value will grow not only because there will be more inference spending, but also because the startup will collect data on how the models are behaving. It feels like that's aligned with what you're saying here that you guys will be able to better recommend stuff for users and learn from them and kind of create a virtuous cycle in which people will always have the right model at the right place at the right time. Right. And today, like, our focus, so far, our focus has mostly been about giving you high-quality

Starting point is 00:50:25 inference for the models you want to try and to use in production and to help you discover new models based on your preferences. And, like, model discoverability, I think was, like, the key product strategy for us, where like, or goal, I should say, where we really wanted people to discover new models themselves and feel like they understood how to get there and then bring those choices really easily into their products. Do the models eventually end up so close in terms of performance that they become fungible or commodified? I was just looking at the artificial analysis, kind of like leading thinking index. And it's amazing how many models are within one or two points

Starting point is 00:51:08 of each other. There are vectors where there's, you know, big differentiation. Grot code fast one, ever since that came out on open riders taking a huge market share because it's cheap and fast. But if I kind of just think a year or two ahead, everything's going to be, you know, 10, 15, 20 percent better, whatever. And it all might taste like chicken. Is that where we're going? Or am I being a little bit too blasé about individual labs ability to create something that's distinct from competition? Well, you know, similar to like, you know, what I said earlier about making model shine based on these really like sometimes idiosyncratic skills that they spike on we're we're doing the same thing for providers so a provider that you know is extremely

Starting point is 00:51:52 at you know has extreme puts a ton of efforts into improving their tool calling ability should be shown accordingly and that's what we do with our exacto endpoints for example which uh focuses on tool use at the inference provider layer, right? It focuses, it's based on a lot of different inputs. We look at the models accuracy when calling tools with the correct schema. We look at the proclivity to call tools. We look at our own users preferences too. We have sort of like, you know, DoorDash style preferences that come in when users are using our API. So, You know, we, you know, those are kind of like votes. And at scale, they become a high signal.

Starting point is 00:52:45 And then we, we, we, all that data, we kind of turn into a big dynamic benchmark. Most benchmarks are static. And static benchmarks become worthless after a certain short amount of time. So we believe really heavily in dynamic benchmarks that are constantly updating and, and that like become products as a result. We productize them. All right. One last question before I let you go. I think co-founder of Chris Clark, he's your C-O-O.

Starting point is 00:53:12 He said that he believes that inference cost will eclipse salaries as the dominant operating expense for most knowledge-based companies in the next five or 10 years. I just want to understand this. Is he saying that as AI models writ large become more and more intelligent, we're going to spend more money having them do work, i. inference spend, then we're spending on human salaries because on one hand, that's very exciting.

Starting point is 00:53:34 On the other hand, I worry about everyone who's 20. Well, I don't know if he met operating spend inclusive. So I think there's definitely going to be some, this is kind of an ongoing debate in the community, how much human salaries are going to compare to inference spend. But I think it's kind of like more of a question of like what in the economy, like how much value in the economy, like how much value in the economy. should be done by AI models if they're improving like this over time, a bit faster, versus humans. And like, it's kind of tough to make any of these statements in the, like, current day.

Starting point is 00:54:26 Because you have to kind of like look at the curve and then just look at how the statement changes year to year. and what kinds of, what sort of domains don't seem to follow the pattern. So, like, it's, it's super early now, but my assumption is that there are going to be some domains where AI models are just kind of not scaling super quickly. And so we'll have, like, much more, you know, much more human salaries directed at those jobs. And those could be things like actually orchestrating models as an engineer, new jobs that didn't exist prior to AI, just focusing on coding, like deep infrastructure work, which the models are probably weakest at today and might stay up a little bit less quickly. And then there will be other jobs that humans mostly do today.

Starting point is 00:55:28 and probably going to be mostly replaced by AI and where that statement will be correct. Well, again, a super stoked, super worried, but I'm old enough that I'm going to dodge the damage here. I think about the future, but at the same time, people want to use AI because it's great.

Starting point is 00:55:44 So who's going to stop them? Well, Alex, it's been an absolute treat. When you hit your first 10 trillion token week, I'm going to be screaming about it from the top of my lungs here. Thank you for having the data so public so we can all use it. And it's openrider.

Starting point is 00:55:56 And before you go, Is there a job that you're looking for the right candidate for that you want to shout out to the audience so that way they can come find you? There are a bunch. Go to openrouter.a.ai slash careers. And feel free to message me on Twitter, too, if you have a, if you want, if you don't fall into one of those roles, but you think you should. All right. We appreciate it, man. We'll talk to you again in six to nine months.

Starting point is 00:56:20 In the meantime, Alex, thanks for coming on. Thanks, Alex. Thanks for having me. Jason, I found a friend. a really interesting question over on the startups subreddit. There's a founder that argued that it's hard to build a startup today because everyone is building things so quickly that no matter what you put into the market, people will essentially copy and paste using vibe coding tools and essentially collapse your pricing ability.

Starting point is 00:56:44 People have always done copycats, Ramp famously copied Brex to some degree down the line. So to me, this just sounds a little bit like cope from our friend. I'm curious what you think about quick vibe coded competitors and the potential for it being very hard to get traction when everyone can just roll out the Xerox machine. You know, as this person saying, it's now almost impossible to build the moat. Every product market fit turns into a red ocean in the blink of an eye. Before you even manage to build an MVP or get early traction, 10 other ventures are already doing the same thing, burning money and making any roadmap to profitability useless.

Starting point is 00:57:21 So that is peak bubble behavior, right? Like when we get to a bubble, there's a lot of money in the system. a lot of entrepreneurs, a lot of people chasing the same thing. So you'll see a bunch of knockoff companies. That's pretty normal. What happens with those companies is, though, they tend to give up. They tend to not be able to keep up in the race. So a lot of people start the race, right?

Starting point is 00:57:42 You might get 10 people who are doing ride chairing, 20 people doing delivery services, and then eventually DoorDash and Uber Reeds run away with deliveries for food. And people forget about postmates or, you know, Sprig and, just, you know, a long tail of delivery services, local, and all different modalities of it. So then it becomes a marathon. So who wins in a marathon? It's the person who has the most discipline, the most trained, well-rested, you know, best diet. And in those cases, it would come down to who has the best team, who is constantly shipping features. And then there's network liquidity. So not everything is about building the product. The liquidity of the network is also important.

Starting point is 00:58:32 So if you open up, you know, a ride-sharing app and it takes 10 minutes to get a ride or you open up Airbnb and there's, you know, a hundred choices in a city, you open up one fine stay and they haven't gotten to Lake Tahoe yet, okay, you're going to use Airbnb. So the liquidity of the marketplace becomes defensible. A sales team becomes defensive. If you look at Salesforce, they have many competitors, or Oracle, many competitors, many options, but they had incredible sales teams. So there are many other pieces to the puzzle that can become defensible. In the early stages of the startup, it's really going to be about using capital as a weapon,

Starting point is 00:59:13 being able to raise more money than the other competitors, and then do marketing and have more features or more distribution. So congratulations, yes. Everybody has the idea. Some people can build it, and then a smaller number of people can scale it, and then an even smaller number of people can maintain it, you know, and sustain it. So each of those levels, you lose competitors. Everybody has the idea.

Starting point is 00:59:41 Very few people can finish the product. Of the people who finish the product, very few people can keep the cadence up with distribution sales, right? And then on top of that, very few people can raise money at scale. So all of a sudden, you get down to a two. two or three horse race in every one of these businesses. So moats then are almost like concentric circles of not only product, but also internal capability, team capacity, and market liquidity.

Starting point is 01:00:08 All of these factors become important later on. So distribution is an important one. If you look at a product or service like perplexity, remember they were doing all these weird deals in the beginning? Like if you buy this, you get a free perplexity. It was like five, you get an American Express card. you get a perplexity account. You get Uber, you get a perplexity account.

Starting point is 01:00:28 They really were just, you know, throwing out. Yes. And then affiliate programs, like just annoying people with these things. Somebody on that team at perplexity was very good at doing annoying partnerships to just get the first crank started, right? So now the next person who built something that was 20% better than perplexies, sitting there and like, I didn't hire some affiliate marketing partnership program person. I didn't hire a social media person.

Starting point is 01:00:55 And this is where startups get really hard in competitive spaces. You have to have, you know, distribution and go to market strategies that beat the other people. It's not just enough to have Betamax versus VHS. It's an all reference to VHS videotapes. We used to watch movies with them. The better technology doesn't always win. Sometimes it's the one that's better marketed and better sold. And marketing and sales, all that is another art in business.

Starting point is 01:01:21 You have to learn after product market fit. or you know, you can, that's, the good news is you can buy that one. You can find people who've done it before. So you got to just get help. But it is worse now. So you got to deal with it. It is worse in a hot market. In a quiet market, it's hard to raise money.

Starting point is 01:01:41 And it's easier to get attention for your product. In a hot market, it's easier to raise money, but there's so many competitors. It's hard to get attention for your product. So in both cases, it's still hard to get attention for product. In one, it's because you have to pay attention to you. competitors. It's a competitive market. And in the beginning, it's because nobody cares about your product because it's a down market, right? Like, people are just not evaluating new products or technologies because their businesses are making cuts. So entrepreneurship is always hard, is what I'm hearing.

Starting point is 01:02:10 Yeah, none of it's easy. I mean, if it was easy, everybody would be doing it. And it's extremely hard. And it is a death march in some ways, you know, where people, there's just like Stephen King movie about some like everybody marches and it's like some sort of weird. The long walk. Okay, there you go. So I was fascinated by the premise and disturbed. Got to take this long walk. And yeah, the last person standing gets the prize.

Starting point is 01:02:41 Everybody else dies. Spoiler alert. It's in the trailer. I wonder if that was inspired by the Chinese Communist Party's Long March. It's an interesting historical question. It's a, this was a Stephen King book. Right. I'm wondering if he was inspired.

Starting point is 01:02:54 inspired by that debit of CCP mythos. Wow. Stephen King wrote it when he was but 17 years old. All right, everybody. That's another great question.

This Week in Startups - Inside Harvey AI’s $8 billion AI lawyer app, PLUS How OpenRouter unites the LLMs | E2207

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.