No Priors: Artificial Intelligence | Technology | Startups - Eradicating Machine Learning Pain Points with Weights & Biases CEO Lukas Biewald

Episode Date: August 3, 2023

How are ML developer tools helping to advance our capabilities? Lukas Biewald, CEO of Weights & Biases, joins Sarah Guo and Elad Gil this week on No Priors. Lukas explores the impact of ML in various industries like gaming, AgTech, and fintech through his insightful perspective. He discusses the impact of LLMs, puts them in context of the evolution of ML engineering over the past decade and a half, and tells the backstory of Weights & Biases' success. He gives advice for aspiring AI company founders, placing emphasis on customer feedback and using insecurity as a vehicle for better customer discovery. Prior to founding Weights & Biases, Lukas attacked the problem of data collection for model training as the Founder of Figure Eight, which he sold in 2019. He holds an MS in Computer Science and a BS in Mathematics from Stanford University. Show Links:  Lukas Biewald - CEO & Co-founder - Weights & Biases | LinkedIn   Weights & Biases Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @l2k Show Notes:  [0:00:00] - Lucas Wald's Journey in AI [0:08:16] - Startup Evolution and Machine Learning [0:18:54] - Open Source Models Implications and Adoption [0:29:54] - ML Impact in Various Industries [0:40:27] - Advice for AI Company Founders

We've talked to many practitioners who are pushing the state of the art. This week on the podcast, we're exploring the dominant ML developer tool, weights and biases. Elad and I are sitting down with CEO and co-founder Lucas Bewald. He has a knack for creating companies that support pain points in ML development. His first company, Figure 8, addressed the problem of data collection for model training. And his second company, Waits and Bias, has created an experimentation platform that supports AI practitioners at companies including Nvidia, OpenAI, Microsoft, and many more.

Lucas, thanks for doing this. Welcome to No Pryors. Thank you. Great to be here. Lucas, you studied at Stanford where I assume you discovered your interest in machine learning and under one of our previous No Pryor's guests, Daphne Kohler.

Can you talk about when you started working in AI and learning from Daphne? Yeah, totally. As a kid, I was obsessed with playing games and I got really into Go and I was super into the idea of, or thinking about how would computers win at these games? And so I actually sent Daphne an email, maybe as a freshman, being like, hey, can I,

can I work with you? Like, I'm really interested in games. I want to learn how to, like, beat go. And Daphne wrote me actually a pretty polite email being like, that's not what I do. Go away. A few years later, I took her course, and I was actually, I studied math at Stanford. And I have to say, Daphne cared about a thousand times more about teaching than even the best professor in the math department.

And so it was really just eye-opening. I just loved how much she actually cared about teaching, and it got me really excited about the AI that was working there. And I went on to be a research assistant for her. And the funny thing at that time was, like, nothing really worked. Like, it was just before kind of, you know, Google was thought to be really, like, page rank at the time was the thing that was making them work.

I think later, you know, it became clear that machine learning was a big part of that. But really, when I was doing ML, it was, like, searching for applications that were working. And Daphne was actually really obsessed at the time with a thing called Baysnets, which you don't hear about too much anymore, because I don't think they ever really, you know, worked for many applications. I hope I'm not offending anyone, but that's my understanding. I actually think, you know, the thing that I really took away from Daphne that really

lasted with me was, I mean, she's just one of the smartest people I've ever encountered, and she had this incredible clarity of thought and an intolerance for sloppy thinking that that's just like really served me well. And I think that's sort of separate from machine learning. You'd see like other professors would come and give like guest talks. And, you know, they would say something's kind of lazy. And like we'd all just be sitting there just like waiting for deaf to like eviscerate them. And I think her personality has mellowed a little bit over time.

But I kind of miss, I just miss that sort of like aggressive, clear thinking. And I really admire it. I don't think we got a taste of that. But we did talk about whether or not probably. holistic graphs are coming back a little bit. How did you, how'd you go from, you know, Stanford to founding figure eight? Yeah, you know, it's funny. I actually really struggled doing research with Daphne.

Basically, the things that I tried just barely, barely worked. Like, you know, I published a couple papers that I feel kind of ashamed of where it was sort of like go from like 68% accuracy to 70% accuracy in a task. Nobody cares about by throwing like a thousand X to compute. And by the way, like, kind of guessing the most likely answer is probably like 64% accuracy. So, you know, it just, it felt honestly kind of pointless and sad. Like, I love the idea of like computers learning to do things. But it's hard to sort of sustain the enthusiasm for that when everything you try just completely, you know, doesn't work.

And even the things that do work, you kind of wonder if you're like P value hacking. Like, okay, I tried a thousand things, you know. So I guess something's going to be like a little bit more accurate than a baseline. What tasks were you working on? Did you end up working on Go or games or anything? No, Daphne is not interested in games, let me tell you. And it's actually another, I kind of admire that perspective, too, as much as I love games. I'm a Go nerd, so I'm curious.

Oh, you are? Oh, me too. Yeah, I love Go. Yeah, Daphne was very not interested. She really was practical. And so I worked on a task that you really don't do now called word sense disambiguation, where you're trying to find out like, okay, I have a, the word plant, actually, if you look in most corpuses, because their government generated often

at the time, plant typically will mean like the power plant sense of plant, or cabinet often means the sort of president's cabinet sense of cabinet. And so you're kind of trying to figure out, like, what is the meaning here of these words, and then applied it to translation. It's a cool task. I mean, and actually, it turns out I think that these, again, nobody kill me. But my general sense is that these sort of like linguistic oriented strategies really don't work that. while it's kind of like by feeding more data in and sort of like working on outcomes, you can figure these things out much better. So a little bit of a dead end.

And actually, you know, I was so frustrated by that that I just really wanted to work on something that people cared about. I actually turned down an offer from Google because they didn't tell me what I would be working on to go to Yahoo because they were like, okay, you can work on, you know, search ranking in different languages. But that actually turned out to be incredibly fun, right? because it was super applied. It's actually a test that works really well.

And Yahoo is kind of in the infancy of switching from hand-tuned weights to machine-learned weights. And they really had no one, not many people actually, like, working on deploying this stuff. So I was, like, writing code to translate machine learning algorithms into C-code and then check it. Like, we would check it into our little code base and run this kind of like semi-hand-generated C-code in production. So that was super fun. But, you know, the thing I learned there, actually, which I think I'm not the only one that learned this, but I just felt it. I would go from country to country trying to switch from hand-tune weights to an ML model. And, like, I was sort of the messenger here.

So, like, sometimes it would work and sometimes it wouldn't. And so, like, people would either really happy with me when it did work, or they'd be really pissed at me when it didn't work. But I kind of realized, actually, the model that I'm building is, like, the same for each country. It's the training data, though, is different. So some countries would take the training data collection process really seriously, and they'd get a great model. And some would just, like, really half-asset or, like, you know,

have these crazy, like, issues in the data collection, and then the model wouldn't work. And so I just really kind of viscerally felt how much the training data process mattered. And I kind of felt like, you know, why don't they let me get involved in the training data process? Like, that would be a better use of my time than building these models. And so I wanted to make a company where the people doing the ML could actually have control over the training data collection process and really get visibility into it

because at the time I think the thinking was like, oh, this is sort of like a manual task that's like more of like an operations team should deal with this. And they would like, they would do this thing where you'd like make this giant requirements document it was so like waterfall. Like it would be like...

Yeah, it wasn't iterative. Oh, it wasn't iterative of all. And it'd be like you'd make like a 50 page document and like you know that the people doing the labeling are not like reading that document. But you kind of need that to cover your ass if they labeled something, you know, not the way you want. And it would have been so much better to be like, look, we're trying to make search

results, like, put yourself in the mindset of like someone, you know, who's like looking at this, like, is it good or bad versus trying to lay out in like excruciating detail what makes something relevant or not relevant. I think also at this time, like when you first started, I think originally was called Dolores Labs and then crowdflower and then eventually figure eight. Like, I think I met you in your Dolores Labs days or something. I know. I remember, yeah.

Yeah, yeah. And at the time, there weren't really solutions for data labeling externally, right? Some people are using mechanical Turk from Amazon to sort of run jobs and untrained workers. There wasn't like scale. There was, you know, there was none of these services. Yeah. And so you got really early to this idea of starting like a data labeling company and that that was actually very useful for machine learning. And so it'd be great to hear like, you know, what were the early days of that like and what was the industry like?

And how did you get all that running? Yeah, I mean, it was funny, right? Because back then I was coached actually quite a lot by Travis Calais. who's famous now for doing Uber and other things. But he was like, don't tell anyone that it's like AI, like VCs like don't want to hear AI, which actually good advice at the time. And it was good advice in the early days of the company. So I don't interrupt.

I think one interesting side note on that, just from a Silicon Valley history perspective, is Travis used to have these effectively like hackathons or meetups at his house called the hackpad. And, you know, I think you used to go to those, you know, a bunch of friends of mine used do. And so a lot of startups actually had some impact or influence from Travis in those days, like, due to his fact of, like, you know, being another founder in the scene and kind of getting everybody together. And so it's kind of an interesting moment in time or in history. And to your point back then, like, AI wasn't really as popular as it became later. So it's

kind of an interesting, like, side note. Well, I mean, not only was AI not popular, but like startups weren't popular, right? Like, my family didn't, you know, understand about startups. And I had graduated Stanford. You'd think I'd have all these great connections, but it didn't feel like that. Like, I had no one who knew how to, like, raise money from VCs. I didn't know any, you know, VCs or I didn't really know any, like, entrepreneurs, honestly. And we had this website for Dolores Labs in the early days just trying to get customers. And it put my, my personal phone number. Actually, remember, I was, like, the first user of Twilio because I needed to make a phone tree. And so I used Twilio software. And then, like, all three of the founders came to my house to, like, help me, like,

make that phone tree like work better, which is kind of amazing. It was like, you know, like, you know, one of those like, you know, 20-something, like, you know, grungy apartments in the mission. And then, and then Travis called in. But, you know, it's funny because the phone tree, we were just trying to pretend like we were a big company. And Travis called in because of the phone numbers on the website, not because he wanted to buy anything, but he just, like, thought it was, like, awesome. And so I'm just like, you know, I pick up my phone.

And then there's just like this guy in there and then just be like, oh, man, like, this is so cool. You know, I'm like, okay, like, who are you? You know, it's like, if you want to, like, get coffee. And that actually turned out to be incredibly, like, helpful. But then I think, like, the thing that was so different back then is that the people doing ML, there just weren't that many. Like, there were people, like, heavily investing in ML, but there, but it wasn't that many.

And so what happened was, you know, we got, like, eBay as a customer, which has really mattered at the time. And we got, like, you know, Google as a customer and Bloomberg. And then there just, like, wasn't. anywhere else to go. My board was always recommending like read crossing the chasm and we tried like a million different ways

to like, you know, grow the company and you know, I don't know, I hope this doesn't sound defensive. I mean, maybe I was just a bad CEO, but we had like years of like struggle because there was no chasm to cross, right? There was like nowhere else to go. So we tried all these different things

to like, you know, build more complete solutions for our customers and it just didn't work. And then kind of all of a sudden, you know, autonomous vehicles got popular, and that really actually suddenly caused our revenue to, you know, start to grow really fast again, but it was like an eight-year lull of, like, you know, really no growth, right? So it's hard because we started off fast, got everyone really excited, you know, kind of got like wamped for just like years and years and years. Actually, we had all these competitors. They all went away.

So at some point we had like no competitors left, right? Because like everyone had gone out of business. And then it was a funny experience because like scale came along and totally ate our lunch in the self-driving market, which is a market like I knew and loved. And so, you know, I was so excited to sell the company after, you know, so many years of struggle. You know, but then like right after that, we see like scale just like skyrocketing and revenue. It's like, oh, man, like I wish we had just like, you know, maybe held on a little bit longer. But then, you know, it gave me the space to start weights and biases. So, you know, who knows. I want to be like Daphne Culler and evaluate my decisions, like accurately and critically. But it also does seem like, you know,

I've had some good luck along the way. Yeah, now the market shifted so dramatically, and I think, to your point, self-driving was a first time that you suddenly had a bunch of systems at scale that people need a data labeling for. And then, of course, now we have this LLM wave, but it's all very, very recent. And I think a lot of people basically view ML as this sort of continuity and everything has always been kind of rising in a sort of almost linear way. And in reality, it's this very bumpy set of discontinuities in terms of the set of technologies and markets that people are adopting it in. And so it's not continuous. It's a discontinuous thing. And nobody thinks about it that

way. When you started weights and biases, you said something along the lines of, you can't paint well with a crappy paintbrush. You can't write code well in a crappy IDE, and you can't build and deploy great learning models with the tools we have now. I can't think of any more important, more important goal than changing that. And that's, I think, like, when you announced that you were starting waste and biases. And so I was just curious, like, what lapses and capability really got you going on 1B? And can you also just, you know, many of our listeners know what it does? But for those who don't, could you explain what the product does and how it works? Sure.

Yeah, so it's kind of constantly evolving, right? Because we're saying it's like a set of tools for people doing machine learning. We're best known for our first thing that does experiment tracking, which keeps track of how your models, like, perform over time as they learn and train. We also have a lot of stuff around like kind of data versioning, data lineage, you know, production monitoring, model registry, kind of the sort of end-to-end stuff that you need to do machine learning reliably. And I think the thing that happened to me was I had been running crowdflower for years,

and I always loved machine learning, but I was like really starting to get out of date. Like deep learning came along. And at first I was kind of skeptical of it because people are always saying, oh, I have a better model that's like magically better. And they're like wrong, wrong, wrong, wrong, wrong. It's just like really like data. And then but then they were right. So there actually was a sort of a better modeling approach that worked.

And I kind of realized, you know, when I was in my early 20s, I was really judgmental. of the people in their late 30s that hadn't adapted to machine learning at the time. Because like rule-based systems were kind of all the rage when a different generation was growing up. And I was like, wow, you know, I am actually getting out of date myself. Like I'm saying these kind of wrong things that were true 10 years ago and are not true now. And I honestly felt like really bad about myself. And so I did a couple projects to try to get up to speed. I started teaching free machine learning classes and deep learning classes to kind of force myself to learn the material.

And I actually interned briefly at Open AI, where I was just like, look, I will just do whatever work you want. Just I want to be like, I know that I need like an accountability partner essentially to force me to learn stuff, even though I love to learn stuff. It's like my favorite thing, but I always need accountability partners for anything I do. So I sort of used the students as an accountability partner and Open AI. And then what was happening was I was showing my old co-founder, Chris, like all the cool stuff. And he's like a really good engineer. And I'm like actually really like bad engineer. like, I'm, like, really lazy and, like, try to write the, like, you know, I'm just, like, people, my co-finders make fun of me all the time for, like, you don't really know how Git works. And I just, openly, I have no idea how Git works. I just sort of mash the Git keyboard until, like, you know, getting a bad state. And then I, like, call Kristen Begg him to, like, the CEO rebased. Yeah, I just, I don't know. I don't understand it. And it's like, my co-fathers just finders just find it, like, baffling that I wouldn't understand it. But I think it's like, um, for that,

them, you know, it's like, they're like, wow, this guy needs some basic tools, you know, like, because, you know, they're like, okay, like, reproducibility. Like, why don't you just use Docker? I think that's sort of the ops mindset. But I'm like, man, I don't understand Docker. I feel like I installed on my, like, laptop. And then it's always, like, taking up memory and stuff. I like, I don't really know what it's doing. And I'm, like, kind of scared of it. And, like, I don't know. So it's like, I just feel like it's adding weird complexity I understand. And so I think the tools kind of exist in a way, but they just weren't made in a way that, like, ML people could really use them because if you're like me,

you kind of come from a mathy background or a research background, you kind of didn't really learn to do like industrial style coding. And so, you know, I think companies have this idea that like the researchers are just going to like throw the thing over the fence and then it's going to be in production. But it doesn't really work, actually. Like I think that's a bad pattern that people sort of like imagine they're going to do and they don't ever really do that.

You end up like with researchers always the research code bleeds into production. in every company. So I think a better way is to give, you know, researchers and, like, ML people tools to just make their stuff more reliable. And it has to be simpler, maybe, or it's just a slightly different audience. Like, you can't just give someone, like, Docker. You can't just, like, you can't, I mean, a lot of people are like, hey, why don't you use, like, the Git large file system stuff to version your data?

And, like, there actually are some reasons. Like, it doesn't work well with, like, object stores. So there's some, like, ergonomics reasons. But it's also just like, man, Git is like complicated. I'm like willing to use it for code. But if you start making me like version of my data with Git, like I just want to like cry, you know what I mean? So like give me something like simple, you know what I mean?

Where I don't have to like think about it. Or I'm just going to start like renaming my data sets like latest, latest dash really, latest that's really for sure June 27. So I just need my stuff to be simple. That's kind of the mindset, you know, behind the company is like let's like make these like kind of simple, clear things that actually helped people. We were talking about how much you wanted to, like you were thinking through how much

LMs were going to change like experimentation and ML tooling when we last saw each other in person, not at the zoo, but before that. And you guys launched this prompt suite in April. Like, can you talk us through the sort of, you know, thought process of, hey, like, you know, I really admire this as a leader and as a technical person. You're like trying to stay really plastic about what is actually changing in machine learning. How do you think through this change? Well, it's really hard, right?

I mean, so what happened was we have a great business that, you know, makes like an ML, a set of ML tools for training models. And we actually helped most of the LLMs out there were built using weights and biases. And then we started to see, like, wait a second, some of these ML tasks, you could just ask the LLM, right? So instead of doing like a sentiment analysis model, you could just be like, hey, like, is this document positive or negative sentiment? Like, for structuring documents, you can just be like, hey, find all the names, like, in this document. And it actually works super well. And a little piece of me is a little bit sad about that because we have this, like, great, simple, relaxing business that grows revenue every month that I always dreamed of, right?

So, you know, part of me is like, shit, this is actually our kind of first real existential threat, I think. You know, and, you know, I went to my, like, leadership team and I went to my board and I was like, I think there's, like, a real existential threat here. And I think they were like, hey, you know, we don't like see it in the data. Like, are you sure? Like, maybe you're being paranoid. And I guess I do feel sure. And I don't want to say I'm like the only one or like paying myself as the hero. Like, you know, my co-founder is also seeing this and people talking about it.

But it's sort of like, you know, this threat is like now, right? And we have to actually like get the whole company to do this thing because it doesn't show up in any of our like metrics yet. But I just really believe that, you know, our customers are rational. and they're going to do a thing that makes sense for them. And so I see a lot of my colleagues being like, oh, there's going to be lots of different models. It's nice if it were true, but what I see everyone doing right now

on July 27th is using GPT. I see like 95% of the people out there using GPD for these ML tasks. And so it's like, look, we got to support that. And so we've really rallied the whole company behind it, and we pushed out prompts. We'd also, this is really my, my co-founders, my co-founder, Sean had really put a lot of effort into making our stuff really

flexible because he's like, you know, Lucas, like, there's going to be like changes, you know, coming. We don't know exactly what they are, but like, you know, kind of from the beginning, we really tried to build very flexible infrastructure. So this was kind of a moment where we could really sort of like flex that and get out a, you know, a product for monitoring the stuff. And, you know, now it's like, you know, kind of, it's our number one priority is getting out more tools for this new, this new workflow. Out of curiosity, because, you know, there's a lot of debate right now in terms of proprietary models versus open source models. And I think there's a really great quote. I think it's from Harrison from Lang Chain, which is, you know, no GPU until product

market fit, right? You should first, like, figure out if the thing works at all or if there's a customer need, and that means using GPT. And then once you prove it out, you know, you may use GPT4 or something for very advanced use cases. And then you kind of fall back to 3.5, or you start training your own model for things where you just want cheap, sort of high-throughput things happening. And it increasingly feels to me like people, the most sophisticated people who are at the farthest sort of cutting edge on this stuff are kind of doing both, right? They use GPT to prototype. And then in some cases, they're training their own instance of Lama 2 or whatever they're

using. Do you think that's where the world is heading or do you really think things kind of collapse onto some of these proprietary models, like over time? Like it's six months from now, it's a year from now, it's two years from now. I'm just sort of curious about how you think about adoption of open source. You know, it's funny. I feel like lately what I've been telling people is like, I'm just trying to see the world clearly as it is today. I can't predict the future and I can barely keep track of, you know, what people are doing today when I consider it like my full-time job. So I'm like scared to prognosticate like what, you know, might be coming. But I think you're right that that's what's happening now. I think like there are like a bunch of things that could change, right? Like I think like, you know, GPT is way far out ahead and it's hard to fine tune. not even possible with GPT4. And I think that that is like a little,

that's not like a technical limitation. I guess sort of like a business model in a limitation. So that might change. I think that there's a lot of hidden costs to running your own model. I think people are really enamored with the idea of running their own model. And I've kind of seen this before where I think at the end people do rational things, but it kind of takes them a while. So I'd rather sort of support what looks like the rational,

workflow. I mean, I think the insane thing must be crazier to be an investor in this world is like very, very few people have LLMs in production. Like there's probably more companies that have raised money as like LLM tools than companies that have LMs in production, which is like insane. It's just like an insanely saturated tools market with very few people getting things out. But it's because When you, Lucas, when you say, when you say LMs in production, you mean my own that I have fine-tuned that I serve myself. No, sorry, I mean like GPT, like using GPT in production. Oh, really? Okay.

Look, I mean, you may be like closer to this to me, but I- It's a small handful, yeah. I'm like desperately trying to find them because like these are our customers. Like we, you know, our stuff is just like, our ethos is like we want to help people do things in production. So it's like, if you're not in production, we're not relevant to you. So I like, I mean, back in January, February this year, we were looking for design partners that had stuff in production. And boy, was it hard to find, right? Like, you know, now there are more, but even when you, you know, you find people that are sort of like claiming to have these things in production, it's sort of like, well, it's like, you know, it's coming.

Like, you know, we have like all these like sort of like prototypes, you know, running. And so I think it'll change. I think it's changing quickly. But I think it's a funny moment where, I mean, I think if you actually looked at the tamp today of like tooling. for like LLM's like, I don't know. I bet you it's small. And I think also I think VCs maybe sometimes have this funny window where you see like all the companies that are using LLMs, but the enterprise adoption has been slower.

I mean, despite the fact they talk about it like constantly, like constantly, like everyone's talking about it. But in enterprises like, boy, I don't know if I've like used a product of like any enterprise that actually like was backed by an LM. And there's a bunch of things that make it. it hard. It's like, you know, it's kind of unfair because this stuff has only been out for like six months or so, but it is like, I think the adoption maybe take a little longer the short term that people think. I think that's a really key point because ultimately, you know, chat GPT

came out eight months ago and that was kind of the starting gun for all the stuff, in my opinion, and then GPT4 came out in March or something, right, which is three, four months ago. And if you look at enterprise planning cycles for large enterprises, it takes some six months to plan something, right? And so people often ping me and ask about adoption of these sorts of things. And it's like, well, Notion is seeing, you know, has adopted it in interesting ways already. Zapier has adopted in interesting ways. But it's basically these technical founder-led companies that jumped on it really early relative to everybody else. And the big enterprises are going to take another year or two because they're just in their planning cycle still around

this stuff. They just started really thinking about it and how to incorporate it and what to use it for. And then they're going to have to prototype and experiment for a while. And then they'll push it into production. And so that's why it's kind of asking a little bit about the future. I just feel like it's so early. Yeah. And we all talk about it, again, as if it's this continuous industry cycle, but it's really not. It's a disruptive new technology. And so, you know, I think a lot of it's still to come in really interesting ways. Oh, totally.

And there's tons of product issues, too, right? Like, you know, like Notion and Zapier both have these really compelling demos, and they're both products that I use. But then I actually don't use the LLM, like, piece of them myself. And I wonder, I have no insider knowledge of the level of adoption. But I think they're, I think they haven't gotten it, like, perfectly right yet, despite like a lot of thinking and really smart people working on it. Sure. For the Core 1B product, you know, you folks are being used for a wide variety of areas around autonomous vehicles, financial services, scientific research, media and entertainment. Is there any industry in particular that you think you're either surprised by adoption of the product or you're really excited to see sort of how people are using it?

Yeah, I mean, the one that stands out for me because this is the one that's really different than, you know, my figure eight days is pharma. So I actually think this is kind of flying under the radar a little bit, but every pharma company is making major investments in ML and not just on this sort of like, I mean, they do have these operations to sort of like sell more, you know, drugs to doctors that uses sort of like light ML. But I think the thing that's really exciting is like the actual testing of drugs, you know, before they have to test them in the physical world. And that's like obviously working, you know, super well. And I think I see this before, too, with like autonomous vehicles and stuff. It's like there's a big lag there right before you get something through like all the clinical

trials. So no drug developed by ML has gone through clinical trials. But if you look at the behavior of all of the big pharma companies, I can tell that it's working because they're hiring hundreds of people, right? Like, you know, like companies will hire like a few people for like an experiment, but they're all gearing up to like operationalize this stuff. And that's just gets me really excited. I mean, they could all be wrong, I suppose. And I don't really have any insider knowledge except for the seats that get bought on, you know, with advises. But when I see that, I get pumped. Because I just like, you know, the drugs that they're working on, you know, the diseases that they're curing, it's like the ones that like, you know, like our relatives

have, right? Like, you know, Alzheimer's and Parkinson's and these are kind of horrible things. And I think there's just a huge promise in being able to do physics, like inside a computer versus in the world. Yeah, I think there's a, I think that this is a really important point to It's actually commonly said, like, no machine learning developed drug has actually come to market today, but it's a backwards-looking metric in a very slow industry, right? Like, the clinical trial cycle is very long. And so I'm actually, like, quite optimistic on this. Yeah, and I think that stands out in pharma because it's very under discussed, but there's certain venture funds that have done incredibly well financially in pharma where there's one in particular I can think of that never shipped a drug until the COVID era and they were in business for 20 years. Wow.

And they made all this money and they funded all these companies and none of their biotechs ever launched anything in the market. Wow. So I think that's a broader sort of issue with pharma and we can talk about that, I think, some other time. But it's kind of interesting how little biotech has actually delivered. And there's been amazing deliveries, right, in terms of different drugs and things. But it's actually more common than just the ML side, I think. Yeah. Lucas, you, okay, so pharma is something you're excited about and you think has promise and growth in at least seats of 1B.

Figure 8, like you talked about, you know, Yahoo, eBay, like it's a very small set of people. Who else do you see in the weights and biases, like, customer base now? Like, how has that changed since it's actually incredible to me that you've been, you know, working on this? from the entrepreneurial side since 2007 because it's like, you know, pre-even deep learning revolution, right? And so I imagine, you know, you've got a much broader user set now.

Oh, yeah, it's so cool. I mean, the coolest thing about running weights and biases is the customer set is everyone. I really think every Fortune 500 company is doing something with ML that they, like, actually really care about. And it's always surprising, right? Like, we work with, you know,

most of the big game companies. Like, I'm not a big gamer. So, like, I'm vaguely aware of, like, Riot games and, like, Unity and stuff. But, you know, but they do all this cool stuff with ML to, like, you know, make the games more fun, to make, like, you know, models in the games. And this is, like, big investments they really, really care about because, you know, again, like, we're sort of the last step in your journey is to want good tooling for your ML team. You kind of need something to work, so you hire an ML team. You get into production. Then you, like, run to problems.

Then you come to waste of biases. So, like, we see stuff, you know, after it works. And, like, you know, like ag tech, like, you know, big aggregates. agricultural companies. I'd like never heard of some of them when they showed up. And then they're like these huge, you know, businesses that are actually using ML to find ways to do like cleaner farming. Like a lot of the reasons, you know, you spray a whole field with, with pesticides just because it's like so expensive to do something smarter. And so, you know, I think that like the crop yields and the, you know, the cleanness of the farming practice is about to like dramatically

improve. Like, you know, we worked with John Deere for years back from a figure eight days to, you know, weights and biases. And they're, they've deployed sprayers. that only target the weeds in fields. It's deployed. I remember for years seeing pictures on the wall and then showing me prototypes. And then one day they're like, yeah, you can buy this.

And it's cool because this intelligent stuff, it's like software, right? So it's not like a machine, you just like press copy and then you have more of it. And so yeah, I mean, we see that. We see like a lot of, you know, I mean, FinTech probably obvious to you guys, but they're kind of, I think, always out

in the forefront of the stuff for lots. I mean, like there's like, like consumer-oriented stuff that you'd recognize, like, you know, making chatbots not annoying, right? And then there's like, you know, kind of more, you know, financial forecasting and things like that. But yeah, I mean, it's funny. We don't do any vertical-based marketing

because there's not one vertical that's, like, dominant enough to warrant it. And our customers bounce around between verticals so much that I think the common thread here is people doing like ML and data science versus any particular application, which I just do is super cool. That means it's sort of like table stakes, you know, for everyone. You, you know, made jokes, I think jokes about, like, not being a terribly good engineer.

And now the weights and biases messaging is very much about developer first, right? Can you talk a little bit about how you think about, like, you know, and it actually, it is like, as far as I understand, it's like one of the most broadly adopted tools by developers work in ML. How do you think about, like, developer adoption versus, like, researcher adoption? and what did you do that, worked? Yeah, I mean, it's like developers and researchers that kind of blend together. But I think that what happened in the sort of MLOP space is that you got a lot of, well, the early companies had to sell to executives, which I totally understand.

Like, that's what Crowfler had to do. And the problem there is you kind of get stuck in these like multi-million dollar deals. And like, you just can't get out of that. Like, you can't switch to like a PLG motion. And so the early companies, I think, are kind of stuck, right? with these products that, like, CIOs love and the, you know, engineers hate. And that's just like, I just didn't want to do that with weights and biases, no matter how big the market is or how, like, juicy that is.

And the good news is it's, like, not a good market, like a developer-oriented sales better. When you look at, like, developers versus ML researchers, that line has really blurred in the time that we've been doing it. And I think that, like, there's sort of, like, subtle differences. But, you know, when Nvidia came along and these chips worked for deep learning, it just broke the entire stack. It was like a first time in my career where I'm running into like linker areas.

What the fuck is a linker? Like I vaguely like remember this from, you know, like a CS class I took, you know, like. And so it's like I think that ML researchers really had to kind of become software developers. And at the same time, you know, the AI class is the most popular class. So like all these software developers are smart ones

kind of become ML researchers. So I think that line has, weirdly blurred. But then I think there's a funny thing that also has been happening where like every DevOps person on the planet rebranded themselves as like an MLOps person all of a sudden. And so you get all these companies that come out of like every MLOP team then realizes they could raise like a shitload of funding. You know, and so like you got like every major company, their MLOps team like went off and like raised money to like make a new product in the market, which I think from an investor that's logical, right? It's probably they have a good

thing. But they're just like not good at connecting with actual developers, right? Because actually, like, DevOps is like a little bit of a different discipline where you're sort of obsessed with reliability. Kubernetes seems like simple to you. And that's just not like the experience of like an ordinary, you know, developer. Like, you know, like my co-founders are me. And so I think the joy of weights and biases is we're kind of making software for like ourselves. And I think it turned out that maybe in the median of my three co-founders was actually the target audience for us here. I think I skew more towards an ML researcher barely, but if I had to pick one end of that spectrum, and my co-founder, Chris, probably skews more to its software developer, and Sean's probably

somewhere in between. One of the things that's common to people or to developers is that they love to write their own tools, and they tend to really enjoy using open source over close-source solutions. how did you think about the open versus close source approach? And how did you think about making something that's valuable enough and good enough to overcome that natural inclination to just do it yourself? Well, it's funny. Like, I think the tools thing, I've always felt like,

I've always felt like kind of proud of making tools for developers. Like, that's always felt like really good because I think developers sort of know what quality is. Like, I mean, it's like, I kind of like making a tool for someone that could make the tool themselves because it kind of raises the bar. And this definitely, my grandfather was like a pattern. maker, which is like a sort of, you know, like the person makes a pattern for other machinists. And he had the same attitude of like, look, I'm making this stuff for like other engineers.

And like there's like an honor in that. So I definitely feel that pressure and love it. The open source for closed source thing was really just like we didn't know how to make an open source business. So, so like we kind of started off close to us because we just, we actually wanted to have like a working business. And it's had like a pro, there's been a major pro, which is that all our competitors are open source. And what that means is that they don't get to see how users actually use their software. And so I think our software is a lot more ergonomic because we have metrics on what people actually click on. If people aren't click on a button, we remove it.

If people like, you know, pick an option all the time, then we know to make that the standard option. As we've grown and you kind of can't just like rely on anecdotal user feedback. back that I think has made our product a lot better. People find it nicer to use. At the same time, I understand why people want to go to open source stuff, but honestly, I feel like it's a little bit of a DevOps mindset also.

I mean, DevOps people, they're obsessed with, like, open source. And usually like the ML ops, we talk to in companies really want like an open source piece, which is why our client is open source. I think that actually runs in your servers is open source. But, like, I don't know, like ML researchers aren't so precious.

in my experience, generally, they kind of want to get a job done. And I think they're kind of happy to like that we have like a stable like business that generates money in like a normal way and isn't going anywhere. Or at least that's what I tell myself. I think this is like the part about like the need for like ongoing telemetry and application feedback. Like there are a, you know, zero to marginal number of open source applications that have actually succeeded.

I think part of it is like the sort of, you know, hierarchy of honor of, like, the deeper in the stack you go, like, did people really want to work on, like, web UI in the open source or just, like, random business logic on a relational database? Like, it's not as sexy and exciting to, like, go put your, like, GitHub badge on. But I think the piece that you describe is actually really important, where, you know, you work on complex workflows. And if it's something that, like, somebody can just run in infrastructure and, like, you know, you, you, you, you get data back on like config files or yamil or whatever like that might that might work in terms of like one person's architectural point of view or some framework but i really don't think it works at the application layer for for these two reasons right like one total lack of feedback and to sort of the lack of interest in the i don't know technical brownie points you get for it yeah do you still pay attention i'm sure you do actually to like annotation like what do you what do you think happens to the data annotation space and like you know the land of lms and our L.H. Chess and such.

You know, I'll be like honest, actually. I'll just be like totally honest. I find it like incredibly stressful because I still feel bad that we lost the scale. Like it's still like, it's just like lingered with me. And I admire scale. Actually, I know hard that that businesses. So I have just like deep admiration for their like execution. But as a competitive guy, I kind of can't get over it.

So I've like always inundated with questions from VCs. Like whenever any annotation company's raising, I know about it because everyone like calls me. but I honestly try I know I should be closer to it, but I try to stay away from it just because it causes me so much anxiety to look at what's going on that I just can't deal with it.

What were some of the things that you did differently with the second company? I feel like, you know, I've started two companies and with the second one, there's all sorts of lessons I applied immediately. Were there two or three key takeaways that when you started away some biases

made the second time around easier, was it harder? How did you think about, you know, key learnings or how to apply new things? Yeah, I mean, I think like one thing was like extreme clarity about who we were serving. So I'm surprised I don't hear this more because like the ways that biases started with a with a customer profile. And I think it's actually a nice way to start a company because, you know, especially as like a founder, you have to spend so much time with your customers. You have to seek them out.

Like picking a customer that you love, I think is a really good thing for your like mental health, you know. And so that was like a big thing. And then I think like I think I've just been a more confident person in myself. Like anytime I start thinking like, okay, like long term or short term, it's just like you always want to think long term. Like everybody wants you to think short term. Like everyone's going to push you to think short term. They wouldn't say it like that. But it's like, you know, it's like people can see like A or R growth.

They can see like user growth. It's harder to see like product quality. Right. And so I think like I think I'm a competitive. guy who likes metrics and likes accountability. But I actually think that can get counterproductive for me where you start sacrificing short-term things to grow these external-facing metrics. And I just really try to fight that myself. I think everybody like chases, every entrepreneur chases like short-term like AR numbers like in quarter. But then it like hurts your growth rate the next quarter. It's like,

it would actually be better always to like push out deals. But like nobody thinks like that, right? You can't think like that. But it's, I don't think it's totally rational. Is there any advice that you'd give to founders who are running their first AI company or just getting up and running? Yeah, you know, the advice I always give is like, it's like the generic advice that everyone says, it's like even truer than you think. It's even truer than like I know, even though I like deeply believe it. So it's like caring about like if you're making something people want. Like everybody knows it, but like no one cares about enough, right? Like people just, they get distracted. They do other weird stuff. Even I do it. I understand. But like, you should.

should care more than you think, no matter how much you think. I've never met anyone that cared too much about that. And then spending time with customers, it's like, it's so critical. Everyone says they do do it. But I don't really believe it. Like, I feel like I'm obsessed with this. I mean, like getting like, when you're an early company, getting like three customer calls in a week, that's like tough, man. I mean, you've got to like scrape and claw and like beg to get those meetings. And you know, like, two of them are going to like cancel. So I don't know, People tell me, oh, I met with like 30 customers this week or something. It's like, really, did you?

Like, I don't know. I tried it's really hard to get customers attention. So I don't know. I'm just feeling that nobody does enough of that, but I don't really know. I think people are all lying to each other, but how much, like, actual kind of customer meetings they're doing. And then it's like, you know, when you get to a customer, it's so precious. It's just like, man, like, show up prepared and like ask the tough questions. Like, I think, like, I feel like one thing about me is, like, I always, like, default to, like,

wanting people to like me, and it's a terrible trait in a CEO. You know, it's like, I feel like I've all these, like, coping mechanisms for myself to, like, not just, like, kind of flip into that mode. But I think it's good for customer discovery because I'm always, like, so afraid that they secretly, like, hate my product, you know, that I get, like, really insecure. I'm just like, okay, like, tell me, like, more, you know, like, are you sure? This is really, like, working for you. Actually, it does actually help in that one important, like, entrepreneurial process.

To lean into your insecurities with your, with your early customers. Lucas, this has been great. Is there anything you wanted to talk about that we didn't cover? No, this has been fun. I mean, I just, I think the message that I'm trying to tell the world is that we're really trying to make tools for this new LLM workflow that people are calling LLM ops. And so my, my advertisement for weights and vices is like, hey, if you knew us and liked us for our MLOP stuff, try our LLMOP stuff called prompts. I think it's, I think it's not amazing yet, but I think it's kind of ahead of the market. And it's about to get a lot better because we are like investing every. every resource that we have into making as good as possible. And we're really listening to feedback and iterating. So if people want to, you know, email me directly and tell me some issue they had with prompts, I really want to hear it. Is it, is it Lucas at 1B.com?

Yeah, Lucas or the K, yeah, at 1B.com. Okay. You're going to get a flood. Well, I'm awesomeistic. You're such a pioneer here. Thanks so much for doing this, Lucas. That's great. Thanks for us.

Yeah, thanks for joining. Thank you.