Latent Space: The AI Engineer Podcast - [State of AI Startups] Memory/Learning, RL Envs & DBT-Fivetran — Sarah Catanzaro, Amplify

Starting point is 00:00:00 Light and Space Night Toronto Fire Break up Lighten Space We're here with Sir Kerenzaro from Amplify. Welcome. Thank you. First time on the pod.

Starting point is 00:00:15 I know. I know. We've known each other for so long. Yeah, never made an appearance. It also made the transition from data to AI, I guess. I don't know if I did. I don't know if you were always like as deep on AI. But obviously, there's a lot of,

Starting point is 00:00:31 sympathical. Yeah, I've always actually kind of oscillated between data and AI. Sure. Like, arguably, I started my career in quote-unquote AI. It was just more like symbolic systems back then. But as you said, I think like they're, they're so symbiotic. Like, it's almost hard to divorce them. That's actually what brought me into data. I was like, I want to better understand what happens when I write a SQL query. Yeah. This briefly touch on data because I think, obviously, that's a lot of where you and I first met. DBT5Tran. That was so cool. I mean, or how do you think about the end of the modern data stack? Okay. So like a lot of people look at the like DBT5Tran merger and like talk about the end of the modern data stack.

Starting point is 00:01:18 And I think that is like a fundamentally wrong take. Both of these companies were growing, you know, very healthily. Both of these companies. Do you fund a DBT? We funded DBT. So like both of the companies were. actually, like, beating their revenue targets. I think what you're more seeing is a, you know, IPO environment wherein companies are expected to have far more than, you know, like 100 million revenue. And so... What would you say the bar is now? 300? No. Like, above 600. 600. Yeah. Yeah. And the combined company is 400? I believe that they'll actually be close to 600. I don't have the exact number. But they're clearly just getting ready for IPO. So, so, you know, basically, like, the merger was a way to accelerate that path to liquidity.

Starting point is 00:02:07 As you might remember. And there were the presumptive winners in their categories anyway. Exactly. Exactly. You know, I think one of the things that has actually pleasantly surprised me, and this speaks to, again, the symbiotic relationship between, you know, data and AI. Many of the Big Frontier Labs are actually using both DBT and 5Tran. I recall talking to folks at thinking machines, like within weeks of the company's formation, and DBT was already an important part of their stack.

Starting point is 00:02:39 Certainly, like, training datasets need to be managed. We need insight into what users are doing on these platforms. And in fact, like, the way in which you would analyze interactions with an agent or analyze interactions with an LLM is even more complicated. And so, while I think perhaps, like, the demand for analytics engineers, the demand for data scientists didn't explode in the way that some people thought. Like, analytics engineers are not one-third of personnel. But that doesn't actually mean that the demand for the tools is not still, like, very prevalent. But you go away you want it. You wanted to

Starting point is 00:03:15 democratize things. You got it. Yeah. Yeah. I mean, I guess we democratized things by perhaps reducing the need for the people. I don't know whether or not that is a good thing. But honestly, I do think that the fact that it is easier than ever from a tooling standpoint for people to make data-driven decisions is probably a step in the right direction. And I've become actually convinced that, like, well, every company does need analytics engineers and does need data scientists, they probably don't need armies of them. And probably having like a moderately sized data and analytics team is a good thing. Yeah. So you touched on an interesting thing wasn't plenty to ask, but this is interesting.

Starting point is 00:03:59 So I come from the data field, data was synonymous of analytics. Yeah. But you're now saying the DPT-Fir-T trend are being used for training data. Is there any notable differences in the workloads or the requirements? Undoubtedly, there will be. I mean, I think one of the things that we saw with analytics

Starting point is 00:04:19 that it was surprising to some of the people in the data infrastructure space was that, like, the workloads were actually quite predictable. They were quite predictable because, like, many of them were actually not being generated by humans, but rather by deterministic systems. So, like, a lot of it was, you know, like, BI dashboards that are, you know, Tableau that is actually hitting your database or maybe not Teplo, but like, Looker or, you know, hacks or something like that. I think with, like, analyzing, curating, preparing datasets, it's a bit more ad hoc. and so undoubtedly it will be less predictable. I don't know if that really changes the way that we approach developing data infrastructure.

Starting point is 00:05:06 I talked like some people are quite interested still in things like learned indexes, learned optimizers, and it's a bit easier to build a learned optimizer if you have more predictable workloads. And so it could change the way that we approach things like that. Yeah. Data catalogs, did it become more important? Are they transferred? Oh, man, like straight to the gut.

Starting point is 00:05:27 So that was something I got wrong. I'm sorry, I don't know the background. What did you? I just, I really believed that data catalogs were going to become an important part of, you know, the modern data stack. And the players are Atlin. She's Singapore. Yeah, yeah. There was.

Starting point is 00:05:50 Data World. Data World metaphor within our portfolio. They've all struggled as a category. They all have struggled a bit as a category. Many of them have been, you know, acquired subsequently, which suggests that, like, this was not, you know, perhaps a standalone category. As a data scientist, like, I spent so much time working on data catalogs. And so, you know, I kind of felt like this was, like, this was the thing I wanted.

Starting point is 00:06:19 Like, I didn't want to have to, like, build the, yeah. More to the point also, like, pre-training data, you have a lot more heterogeneous data all over the place. Yeah. And you need to keep on top of it. And you need to make it discoverable accessible and all that. So why didn't it work? So I think there were a couple of things.

Starting point is 00:06:35 I think we have seen some consolidation in the modern data stack, particularly around some of the key components, whether it was 5Tran or DBT or, you know, Hex or Snowflake. Many of these products offered kind of like data cataloging, capabilities as a feature. And I think for humans, that was good enough. Like, the data catalog that you had available in Snowflake was good enough. The data cataloging capabilities available in DBT. Like, those were good enough. They did... DBT, like, obviously, as they didn't build the cloud, they were going to build it. Yeah. Yeah.

Starting point is 00:07:14 Like, what else do you do? I mean, it's actually funny. In fact, my colleague Barr at Amplify was the, like, products lead on these kind of like metadata services. I think it's still not obvious to me, but I think one opportunity that might have existed and or could have been realized was the opportunity to build data catalogs not for humans, but for machines. This would look a little bit more like metadata services. I don't just mean for agents, although I think that opportunity is arising more. but even like microservices and things like that. Okay. Yeah.

Starting point is 00:07:59 So I do wonder at times, like, if we built data catalogs for the wrong people, and potentially even, you know, for the wrong use cases, like, I think a lot of data cataloging companies ended up focusing on like discoverability when perhaps like the real market opportunity was in governance. Governance very important. Any other comments just about what you know so far about the data stacks of, the large labs. You know, I guess obviously a lot of data people who might be listening would want to sell into them. Yeah. I mean, a couple of observations. One is that, you know,

Starting point is 00:08:33 they are actually paying careful attention to their data stacks. I think they're thinking about, you know, problems ranging from, you know, data discoverability to data preparation, to even things like the efficiency of data loading. Like if you're unable to load data to a GPU efficiently, then the GPU is going to sit idle and that's going to be a kind of like cost. Yeah, yeah, exactly. What solution handles that? I don't actually...

Starting point is 00:09:03 I mean, I get to talk about, yes, exactly. Plug my portfolio company is, we have a portfolio company called Spiral that has developed a file format called Vortex. And they make data loading super efficient. Specifically to GPUs? Specifically to GPUs.

Starting point is 00:09:21 Yeah, yeah. Good to know. One of the things that has surprised me, though, is actually that, like, so much data infrastructure has actually scaled quite elegantly to meet the AI use case. You would hope. You would, but, like, the scale of these AI companies, it's incredible. It's not as big as ads. Maybe, maybe, yeah. I think that could change, you know, like, as agents actually become kind of, like, more prevalent and are in. interfacing with each other and therefore, like, perhaps like the number of transactions explodes.

Starting point is 00:09:55 I have a friend who works on transactional databases at OpenAI, and I was like, so you must be, like, building databases. Like, this is like a paradigm shift in terms of like the scale that, like, databases are like going to need to handle. And he's like, no, we use Roxette. Like, it's fine. It's the one that acquire, right? Yes, exactly. Yeah, yeah. Very cool. Okay, let's just talk about funding around it because obviously that's like a big theme this year. What is comes to mind in terms Looking back at 2025, what stands out? It was crazy. Yeah.

Starting point is 00:10:28 You can give anonymized examples of like what does crazy look like? Yeah, I mean, I think crazy looks like raising upwards of $100 million. Seed. Like upwards of $100 million in a seed round where you have a long-term vision but not a near-term roadmap. This is something that I'm seeing. happening not just occasionally, but quite frequently. Yes. And it definitely makes me anxious because, firstly, like, when founders are asking me, you know, how much should I raise?

Starting point is 00:11:06 I'm typically saying, like, three, like, five. Well, like, what do you need to do? Like, what are your milestones for the next two? Let's call it, like, 12 to 24 months. What resources do you need in terms of, you know, headcount? compute equipment to unlock those milestones and then like maybe add like a 20% buffer or something like that. But doing that analysis requires you to like understand what you're going to build in the next zero to let's call it like 24 months. I've talked to some companies and they're like,

Starting point is 00:11:39 we're building a frontier lab for X. And I'm like, okay, cool. Like I get the long term vision. There is an opportunity to, you know, make AI more secure, make AI more humane, make AI more data efficient, whatever it might be. So like, I'm bought into the long-term vision. And that, for me as an investor, is super important. So let's talk about like what your team's going to work on in the next six months. They're like, uh, maybe we might build a consumer app. Like, you know, we're...

Starting point is 00:12:08 I feel like I know exactly the company you're talking about. But like I wish I was talking about like one specific company. I'm actually talking about like several companies. And look, like, I'd be a hypocrite to say that, like, I've never done investments like that. But I've done investments like that when, like, I really know the people. And I'm like, they're going to figure it out. What is frightening about this funding environment is that you meet a founder. They're like, I'm raising, you know, $100 million.

Starting point is 00:12:37 I'm raising like a billion dollars maybe at times. And you need to make a decision in seven days. And I can't tell you what I'm going to do for the next six months. And so, like, you have no way of even gaining conviction that they're going to figure it out because you only have, like, seven days to get to know them. I think what some of the founders are missing is, like, you only have seven days to get to know me. If you haven't figured it out, like, you probably want a partner who's going to be working closely with you to help you figure it out. I mean, they're absolutely viewing it as transactional, right?

Starting point is 00:13:08 Yeah. They don't care. No, they care about, you know, the most money at the highest valuation. I mean, the crazy thing is that they don't even seem to care about dilution. It's just like the most money at the highest valuation. Yeah, but, you know, it does send a signal that helps. So, I mean, yes, I think it does right now send a signal. Okay, I'll tell you how it affects me, and I hate it, I hate it, right?

Starting point is 00:13:32 Antithesis came out of stealth this week, right? And it's like the only thing I know about them is they do something, something in AI testing, and Jane Street led a seed round of $100 million. We invest in it too. I can tell you what they do, but they do the permanistic simulation testing. The thing that is the lead is the money. Yeah. And then like, okay, well, who else uses it other than Jane Street? Like, what do you do as innovative? Palantir. Okay. Okay. Work. Warpstream. Yeah. So, yeah. Anyway. So maybe, this is a bad example because they're actually legit. But like, you know, there's a lot of similar examples where they just lead with the money and like there's no no much substantiation behind it. Maybe it's just, bad storytelling and that's why I as a podcaster get to talk to the, I just talk to

Starting point is 00:14:16 general intuition and like once you spend some time with them, then you're like, oh, okay, this is why they raise $100 million. But like, without that context, it's like really hard to understand anything. Well, and like, I think there are some companies that are raising, you know, a hundred million dollars or more because they need it.

Starting point is 00:14:32 Like, a good example might be like periodic. In addition to, you know, yeah, they need to build out a wet lab and like designing a wet lab that can support high throughput biology, which is absolutely critical to their goals, that's costly. So, like, I understand why they need that, that funding. But again, there are others where, like, they don't have these near-term milestones. I think the thing that is a little bit, you know, perturbing to me, many of them are doing it because it makes it easier for them to hire.

Starting point is 00:15:04 because there are all of these candidates who like want to be, want to work at a company that is like a unicorn or a near a unicorn. They're pitching. Because the alternative is work at a big lab where, you know, it's, yeah, the prestige and the money is there. Yeah. Well, or the alternative is like work at like an early stage startup. But, but, but, but, but like there's something about like the big valuation that becomes enticing.

Starting point is 00:15:27 Yeah. They're also kind of pitching candidates. They have a compelling equity pitch where they're like, okay, maybe you're getting, you know, less than a zero point, like, uh, 1% of the company, but like given the valuation, uh, the value of, uh, your equity is already, you know, like $10 million or something like that. And, and they also, uh, guaranteed a dollar value. Yeah. You, you mean that like they'll offer them a loan to, to pay? Uh, a buyback. Uh, if, if, if it goes, um, if you want to sell it. Yeah, but, but, but, but, because they have so much cash.

Starting point is 00:16:04 But the thing, though, is that, like, the valuation is a made-up number. Like, valuation, until a company exits, it is an entirely made-up number. So, like, I could just be like, you know what, the latent space pod, that is worth $5 billion. And we could agree. Like, I as an investor could say, like, that is the price. And now, now the company is worth $5 billion. Like, do you think that, like, if you were to... Yeah, it's not real.

Starting point is 00:16:28 It's not actual... It was acted in any volume. And given the funding amounts that they're raising, too, Like, if they spend that and they, you know, get acquired for less than that amount, then, like, their teams are getting nothing. I wish people were kind of, like, more sensitive to this dynamic and thinking more about, like, what is the upside associated with the company? And, you know, more fundamentally, like, do I deeply believe in this vision? Because I think, like, joining companies because, like, they have a billion dollar valuation. It's just, it's not the right way to choose a job.

Starting point is 00:17:02 I hear you. Okay, so obviously we can go about that forever. Oh, yeah. And there's a lot of, there's also some stuff with like cyclical funding and all that stuff. But I do want to be more relevant to engineers and researchers. Yeah. What are the themes that are really strong, right? So one thing I'll point out is world models.

Starting point is 00:17:23 Oh, yeah. Just in general, are a really strong bet. I would say, so I have a, every near-Rubs, I go to this group of researchers and we take a vote on the top themes of the year. everyone's extremely skeptical about world models. I think it's a trailing indicator because L-L-L-LFs have been so enormously successful. You're like, I don't need anything else. I don't know if you're a take-on world models or any other top theme of the year. My take-on world models is that we have not yet defined what a world model is.

Starting point is 00:17:51 Oh, yeah, there's like three definitions right now. Yeah, I think there's a lot of confusion about like what a world model is and therefore, you know, what it should be used for. we're already seeing plenty of like market potential for video models, including for things as like perhaps like banal as like video editing. I think, you know, we're already seeing some applications of world models to things like autonomous driving and potentially even coding. But again, it really hinges upon like how are you defining world models? And I think one challenge that people have seen is that like world models perhaps designed for one specific use.

Starting point is 00:18:28 case might not generalize to others. So as an example of this, like, world models for, like, video game generation might not, like, generalize to, like, factory settings or robotics. I use the word might, like, strategically because I think, like, it is potentially a research problem that might be figured out. Yeah. So, yeah. That's part of the Genuantition podcast that we did. Yeah. Yeah. I think, like, it is possible. It's just we're not there yet today. Yeah. A theme that I've been spending a lot. of time thinking about is memory management and continual learning. I work with a lot of... Save startup. I don't see it. Okay. I think I know what startup you're thinking about as well.

Starting point is 00:19:11 But I actually, like, I see, like, a lot of market potential for memory management and continual learning. My interest in this is actually more driven by conversations with practitioners. Personalization is so important. important right now. I think what we're seeing is that a lot of AI application companies, they're growing really quickly, but they suffer from, you know, relatively low retention, relatively high churn. So, you know, if you're developing an app like cursor, how do you ensure that your users don't, you know, switch over to, you know, windsurf, yes, or, you know, cloud code or cognition or, you know, whatever else when they release new features. Yeah, cursor rules isn't enough, right? Like, it's like the shittiest form of memory. Yeah, yeah. And it's great.

Starting point is 00:20:04 But yeah, I agree with that, but also it's like, as a, I've publicly mused about this before where like memory is very poorly implemented today in a lot of surfaces. Like even chat chit, I wouldn't say like people are particularly excited about it. Okay. Yeah, yeah. You feel stronger about it than I do. Yeah, yeah. I mean, I wish chat ch ATPT had, you know, much better. Yeah, this is supposed to be the leading one.

Starting point is 00:20:29 I don't know. So, and then I think, like, just in general, it makes product management harder because what is the product? It's a combination of U-plus memory. And, like, when you have a bug, is it the memory or is it something core? And that's, as a user,

Starting point is 00:20:48 especially if it's consumer, there's going to be zero patience for any of this. I agree. But that said, like, consumers seem to be, like, tolerating products with like no implementation of memory today. So I think like better is still probably better than like what exists. Better is better than nothing, I guess. Would you agree with the statements that basically, let's say a key theme of 26 is this

Starting point is 00:21:12 personalization. I would call it kind of like the consumerization of AI in the same way that consumerization of enterprise was a trend like 10 years ago. Yeah. I mean, I think that is a good way to putting it too. Like I don't for what it's worth think like this is just. just a consumer or prosumer phenomena. If you are in enterprise that is adopting, again,

Starting point is 00:21:32 like a Devon or Augment or something like that, you probably also want your models to kind of like learn the like, I'm not limited to. Yeah. Like you start to like K factor, I had to explain what that is to so many founders. And, you know, like this, like if you're in normal SaaS, this is what you're obsessed over.

Starting point is 00:21:50 And to AI founders, they're like, what do you mean growth just doesn't just show up? Like, like, yeah. I mean, it has, though. But I think like it has because for a while, you know, AI has just felt magical. But like now we're getting more accustomed to the magic and it's no longer enough. And I think, you know, we need to revert to some of the like old tips and tricks for retaining people and, you know, bringing them in. Personalization is one of them. I always kind of intermingle like memory and. continual learning because I think like one interesting element of personalization is not just learning no facts about your or your preferences, but like actually learning new skills from interactions with you and learning as the world changes. Like there are new versions of languages and frameworks

Starting point is 00:22:44 and, you know, other repos that are coming out all the time. The world is changing all the time. Human intelligence is incredibly dynamic and yet like artificial intelligence is just so static today. Yeah. But like... So it must update weights. Yeah. For you.

Starting point is 00:23:00 But that also means that like it's an interesting kind of like systems problem because like if you must update weights, then like, you know, weights become stateful. And today like inference is not stateful. So, so, you know, I think there's going to be like a lot of kind of fun, narly problems to figure out as we figure out things like personalization and continual learning. That's also a fascinating infrastructure problem because you have to load and unload and, you know, cash and all the other good stuff. Yeah.

Starting point is 00:23:25 Yeah, exactly. One more thing, I think we have time for one more take on our all environments. Huge topic. Is it just a Docker container with some custom software loaded and logging stuff out? What are the good ones like and what are the average ones like? So I know I'm going on record on this and like I'm actually okay to be wrong, but I think RL environments is just a fad. Oh, God. Oh, no. They're all fake? I mean, like, people are like, okay, the thing that I, makes me take it seriously. The labs, I know, are paying seven, eight figures for our own environments, for other, like, and they could build it in-house they're not. And I don't understand why. I mean, they were paying seven to eight figures for, like, piss poor data annotation, too.

Starting point is 00:24:13 Yeah. And then data labeling before, like, the labs have a lot of money. I think perhaps like oral environments could create some value in the short term. But I think to the point about like what makes a good oral environment, what makes a bad oral environment, I think the best oral environment is, is, you know, the real world. Why would I, you know, want to buy a DoorDash clone when like I can just. use logs and traces from, you know, DoorDash itself. It doesn't mean that we don't need to blend. You can roll out in parallel.

Starting point is 00:24:51 Yeah. I mean, I think like using the real world, using real apps as like our RL environment is in fact the best thing. And this is what cursor does. Like they actually do use, you know, real user activity on their platform to, you know, significantly like improve both their coding agents as well as tab. And I think that's one of the approaches that has like made the platform so compelling. It doesn't, like, you still need to figure out, like, the right rubrics.

Starting point is 00:25:17 You still need to figure out, like, the right set of tasks. So there are some aspects of oral environment design, you know, at least as we're talking about it today, that I think are going to remain incredibly relevant. But, like, just building a clone of an app, I think is not that useful. Yeah. Yeah. Okay. That is hot. We have maybe three minutes for any other stuff that you think about just the state of startups in general, a state of funding. Yeah. So maybe I can talk about like just the archetype startup that is like most exciting to me. Yes. I press for startups. Yeah. Yeah. I love investing in, you know, infra tools, platforms, et cetera. And as we talked about with continual learning, I think like there will be opportunities for like new tools, platforms and in the future. I've spent a lot of time.

Starting point is 00:26:08 I'm thinking about like applications today. And specifically like the relationship between research and applications. An example of this is like, I think there were a lot of advances in RAG. And the biggest beneficiaries of these advances were the application companies for whom, you know, retrieval was a critical unlock. So as an example of this, you know, like Harvey, Habia. I knew you were going to say Harvey. Yeah. I mean, they, they have like really interesting.

Starting point is 00:26:38 rag implementations. They have hired researchers, like really good researchers to kind of advance the state of the art. And that enables them to build a better product. I feel this way very much about rule following and customer support. Rule following is like a hard research problem. But if you solve rule following, then you unlock, you know, better customer support. And I think a lot of Sierra's success can be attributed to like their focus on this. So I've been thinking about like, even for something like continual learning or memory, what is like the killer use case

Starting point is 00:27:12 where you can either offer a dramatically better experience by having a good memory implementation or you can do something that was just not possible today. I think you can also think about this in the inverse. And often the best company is emerge in this way. They're like,

Starting point is 00:27:30 I'm trying to do this thing. But in order to actually do it, I need to solve this hard technical problem. that that's kind of like the story of runway. I don't think they would have built models if they didn't have to. But I love that that combination of like, we're delivering something that is like better for consumers, better for prosumers, better for users,

Starting point is 00:27:51 but we're doing so by solving these like really nerly research and engineering problems. Yeah. I don't want to, yeah, go ahead. There's so much that I want to sort of dig into there, but we're a shorter time. But just thank you in general. I don't know if you have a general call to startups for like a page somewhere that you want to point people to. Twitter. Fax, whatever it's called.

Starting point is 00:28:16 Yeah, you can find me. You can find me there. We're in South Park. With the one I dog. I'm easy to spot. Oh, okay. Well, thank you so much for your time. I know you got to go.

Starting point is 00:28:25 But I appreciate it. Of course. It was great seeing you. Thanks for having me. Yeah. Thanks.

Latent Space: The AI Engineer Podcast - [State of AI Startups] Memory/Learning, RL Envs & DBT-Fivetran — Sarah Catanzaro, Amplify

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.