This Week in Startups - How open-source & distributed models can win AI with MosaicML’s Naveen Rao | E1754

Episode Date: June 1, 2023

This Week in Startups is presented by: Vanta. Compliance and security shouldn't be a deal-breaker for startups to win new business. Vanta makes it easy for companies to get a SOC 2 report fast. TWiST listeners can get $1,000 off for a limited time at vanta.com/twist. Trovata. Starting up is hard. Trovata makes managing cash easy. Start automating your cash management at Trovata.io/TWIST. Use Code TWIST for 30% off one full year of premium features like AI forecasting. The Microsoft for Startups Founders Hub helps all founders build a better startup, at a lower cost, from day one. Startups get up to $150K in Azure credits, access to free OpenAI credits, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more. There is no funding requirement, and it only takes minutes to join. Sign up today at aka.ms/thisweekinstartups * Todays show: MosaicML Co-Founder and CEO Naveen Rao joins Jason to discuss the open-source vs closed AI debate, the profound impact of AI on society (41:06) AI’s rapid pace of change, and its implications for the future of employment and education (40:42). They wrap the show by breaking down the potential problems with centralized regulation (54:37). Follow Naveen: https://twitter.com/NaveenGRao Check Out MosaicML: https://mosaicml.com * Time stamps: (00:00) Naveen Rao joins Jason (2:54) MosaicML and its purpose (5:10) Obtaining datasets and incentivizing creators (8:30) Vanta - Get $1000 off your SOC 2 at https://vanta.com/twist (9:37) The process of using your data with MosaicML (11:55) Defining tokens and prompts (16:53) Fine-tuning the AI model and reinforcement learning (19:27) The competition with open-source models (24:26) The cost of running AI models (26:08) Trovata - Use code TWIST at https://trovata.io/twist for 30% off one year of premium features, like AI forecasting (27:35) How the GPU crunch has affected cloud models (32:13) Why demand will not cease (34:21) Specialized models vs. general models (39:12) Microsoft for Startups Founders Hub - Apply in 5 minutes for six figures in discounts at http://aka.ms/thisweekinstartups (40:42) The impact AI will have on employment (48:49) The impact AI will have on education (54:37) Thoughts on OpenAI becoming ClosedAI * Read LAUNCH Fund 4 Deal Memo & Apply for Funding Buy ANGEL Great recent interviews: Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland, PrayingForExits, Jenny Lefcourt Check out Jason’s suite of newsletters: https://substack.com/@calacanis * Follow Jason: Twitter: https://twitter.com/jason Instagram: https://www.instagram.com/jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis * Follow TWiST: Substack: https://twistartups.substack.com Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin * Subscribe to the Founder University Podcast: https://www.founder.university/podcast

AI, in my view, is the next evolution of what humans can do. You know, language was a big technology that humans used to pass knowledge. That exploded what humans could do and the influence we could have on the world. AI is going to be that next inflection point. But how are we going to make sure that everyone has a place in that world? How are we going to make sure that the demand that's created by the increase in efficiency is commensurate? Or does it collapse? This Week in Startups is brought to you by Vanta.

Compliance and security shouldn't be a deal breaker for startups to win new business. Vanta makes it easy for companies to get a SOC2 report fast. Twist listeners can get $1,000 off for a limited time at vanta.com slash twist. Trovada. Starting up is hard. Trovada makes managing cash easy. Start automating your cash management at Trovada.io slash twist. Use code twist for 30% off one full year.

year of premium features like AI forecasting. And the Microsoft for Startups Founders Hub helps all founders build a better startup at a lower cost from day one. Startups get up to $150,000 in Azure credits, access to free open AI credits, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more. There is no funding requirement and it only takes minutes to join. Sign up today at AKA.m.m.S. slash this week in startups. All right, everybody, we are really focused on AI and this crazy revolution that started really with GPT3 and 4

having a moment for open AI and people starting to realize, hmm, this stuff is going to impact everything. Since that time, Dolly and stable diffusion and showed what's possible. And now people are incorporating generative AI, the ability to generate some type of content intelligently from a prompt, into every single product,

whether it's Notion or Microsoft Office or Gmail. We are going to have AI, companions, co-pilots, and every piece of software. But you're left with a big question. As an entrepreneur or an enterprise, do you do this on your own? And do you own the IP? And do you control?

your destiny, or do you partner with existing platforms that are out there with me today is Naveen Rau of Mosaic ML. He's the co-founder and CEO, correct? Naveen. That's correct. Got it. So you heard me sort of teeing this up. You've been in machine learning for quite a time. You had a company before your current company that you sold to Intel, I believe. And so that was Nirvana. maybe you could explain what you're doing today with Mosaic and why. Yeah, so what we're doing today is really bringing these capabilities of large-scale machine learning,

which is generative AI in my mind, to many organizations. I think one of the things we've done, even with my previous company, was trying to really bring these capabilities to more people to create the world we want. I see success as people that disagree with me being able to build models equally as good as me, right? I think that's how we're going to make this world work. And it's become front and center now with debates around regulation of AI and, you know, putting some sort of, you know, government licenses and this and that. I think really this is solved more in a market as a market solution where many people can build this stuff.

Many people can imbue these models with the biases that they see fit. and we'll let the market decide where things should be, not some sort of centralized regulatory agency. So your company allows my organization to take our data, put it into a language model. This is called Mosaic ML. This is the software that will let me train my own model and then host it easily on AWS or whatever cloud I choose, I suppose.

That's correct, yeah. In fact, we even collapsed the experience across different clouds. I mean, you can start training on 512 GPUs and AWS and then move it to 1,000 GPUs in Azure. We actually make it very easy to move things around and be modular. And really, that enables people to use their resources more effectively, but also not have a lock-in from the provider and really just kind of own their IP. I mean, I think it all stems from the fact that respecting data privacy, I think, is important. Data is, you know, arguably an expression of your company, of your IP, and

building solutions that respect those balance and enables people to build on top of that data and own that thing that is built, i.e., the model, is very important. And I think that's what we aim to do at Mosaic. So this harkens back to the, I don't know, the thorn and my paw that I have been screaming about since the beginning of this, which is, hey, what did you train these things on? And are those people being compensated? We now have a handful of lawsuits and letters that have been either filed or sent Twitter to Microsoft about the training use of their data, Reddit coming out and saying,

hey, this is our data. If you want to use it, we're going to need a fee. And of course, people trained on it without those two sources permission. And then, of course, you have Getty images, uh, versus stable diffusion. You have open source, uh, the community, uh,

or open source contributors versus co-pilot my GitHub's, uh, composer essentially or, um, you know, co-pilot for developers. So if I was Disney and I own Marvel, every comic book ever written by Marvel,

every film, every piece of dialogue written, every treatment ever written, things that made it onto, uh, the screen, things that didn't, make it onto the screen. Things that were spec scripts that were written that were never done.

You can be sure they've got tense scripts for every one they actually produced. Disney could take that Marvel or Star Wars corpus, put it into Mosaic, compose a library, never have to worry about having put that training data into a public entity that would then go use it for a future or maybe even claim some IP ownership of it. and then they could let their writers on the Marvel series ask questions, hey, tell me about this character, Dazzler, was she ever part of the X-Men? What is she known for? What does her dialogue sound like?

Can I get some backstory or whatever? Something the writers are fighting against, but putting that issue aside, this is a pretty compelling case for a company like Disney to start this work now and to keep OpenAI and Google's hands off this data, correct? That's right. I mean, I think that's one sort of really flagrant example, like kind of incentivizing content creators to keep creating content. There's even more, I'll call it mundane things where, hey, I'm a company that has, you know, huge data sets on the behavior of my customers. And I want to build something that gives me a competitive advantage in MySpace, right?

And I don't want to share that with my competitors. I want to build a model and express that competitive advantage directly. but I don't know how to build models, right? I'm not an expert at doing that. I can't hire a team to do it. They can use our tools to go and do that and leverage their data for a competitive advantage. But yeah, I think it all comes down to this

sort of similar way of thinking is that we have data that gives us a kind of an economic incentive to keep gathering data, to create new content, and then we have some way of expressing an advantage in the market. And you need to build to create your model.

that could be proprietary data about consumer behavior, or it could be, you know, I pick the most iconic IP of all time, Star Wars and Marvel, or at least in recent history that have, you know, generated massive, massive profits for those companies. If you're a SaaS or services company

that stores customer data in the cloud, then you need to be, uh, SOC2 compliant, you knew that, from a third party, and you need that third party to close big deals. And if you want to get compliant, easier and faster you need to use. V-A-N-T-A. V-A-V-A-V-A-V-A-V-A-V-A-V-A-V-A-V-A-V-A-V-A-V-T-A-V-T-C-A-V-T-C-T-C-T-Compin in just two to four weeks. Prepare that to three to five months without Vanta.

And Vanta can save you hundreds of hours of manual work and up to 85% of compliance costs. This is a total no-brainer, and Vanta does more than just SOC2 compliance. They also automate up to 90% compliance for GDPR, HIPAA, and more. You can't afford to lose out our major customers. We all know that. Listen, it's a hard year. Last year was hard. You can't lose those major customers because you don't have your compliance dialed in. Just work with Vanta. Get your compliance automated and tight and tight is right. Lock down those big deals. Here's the best part. Vanta's going to give you $1,000 off. That's 10 hundies. Get $1,000 off at vanta.com slash twist. That's Vanta.com slash twist for $1,000 off. Your sock, too.

walk us through for non-technical people who are listening, you know, maybe founders of companies or capital allocators, exactly how I would start this process of taking my data. And I'm going to use myself as an example. It'll be easy for people who listen to this show to understand. We have thousands of meetings and notes from thousands of meetings. We have thousands of applications of startups asking us for funding. We now have those in Notion.

we have Zoom's calls with transcripts and summaries that we've done with AI. And there are external data sources like Crunchbase or LinkedIn that have signals that a startup has done well, i.e. downstream funding. It's not perfect data sets, but there's pitchbook, there's Crunchbase. There's how many employees they have listed on LinkedIn, not perfect, but directionally correct. So I want to see if I can moneyball startup investing. I don't know if it's necessary for us, since we have incubators and accelerators that let us place a lot of bets. But I do want to do this at some point just for the giggles and to see what comes out of it. How would I start that process?

I have a non-technical team, let's say, of investors and technical in that we can talk about technology, but not developers. How do I start this process of taking a corpus of data? Let's call it a thousand meeting notes or five thousand meeting notes and applications of startups. How would I, what would I do? Walk me through it. Yeah. So, I, guys, first preface this with, our tools are meant for technical audiences like ML engineers, data engineers. But there are sort of conceptually three major ways to modify the behavior of a language model, right?

So we'll start with language because we're talking about text. So when you're in a smaller data regime, when I say smaller, I mean that which was fits in a book, like less than 100,000 words, say, then we can use things like prompt injection to actually change the behavior. So actually the model we released less than a month ago called MPT7B has essentially an infinite prompt window. So we tuned it to have 64K tokens. The token is about three quarters of a word of a prompt. Explain what tokens are for folks and prompts just so we really can explain what's happening here with machine learning. in this process.

Yeah, a token is really the important part of a word. So if I said evergreen, we think of that as one word, but it's ever and green are tokens. The vast majority of words, the token and word are the same thing. So, you know, that, boy, girl, those are all single token words. So that's why we kind of give this ratio of about 0.75 of, you know, tokens to words. Yeah, words to tokens rather, sorry. So when we, the way these models work is that we, tokenize the language into these blocks that are meaningful, a word or some piece of the word.

And then that becomes the input to the model, which we call a prompt. That prompt sets what we call a context. It's like, I'm telling you, hey, Jason, we're going to talk about evergreens. So now if I say something like naked seed that makes sense to you, right? So because I set the context. Really, this is what a prompt does, it sets a context for a model. And then it can sort of recall knowledge that has been trained. upon from that context. This is why prompt length is actually quite important. So what we enabled was a very long context window that you could actually feed it a whole book. We in fact fed it the entire Great Gatsby and we asked it to write the epilogue, made up epilogue.

And it did so. And it can do that across the entire context of the book. It's imagine, imagine like reading the whole book, keeping it all in your mind and then writing it out. That's what the model is doing. So I could take the, I could take the, I could take a successful company like Amazon or Netflix, have some research on that company,

like a research report that was written, plug that in and say, of these thousand meetings I've done, do you see any companies that would correlate with this company in some way? And I could use the prompt of a 50,000 word Gartner report or Goldman Sachs report on Amazon or something from 1999 or 2000. I could take Bill Gurley's reporting on, you know, Amazon from the 90s or 2000s Merrimacres and start using that as prompt engineering for looking for patterns and startups, huh?

Something like that? Correct. That's right. Prompting and context windows are, I'll call them the weakest form of learning in a sense, where you can take information, put it in the prompt, and have the model, you know, do some analysis on that. The problem is sometimes there are weird conditions where, let's say, there's conflicting evidence from where the model was trying. trained versus what was inserted in the prompt, you might get, you know, kind of undesirable behavior in those cases. So that backs us up to one more version of how we modify the outputs is what we call fine tuning. Fine tuning allows us to kind of condition the model to act in certain

ways. Like if I ask a model, you know, racist questions, maybe we want to say, hey, I don't, I don't, I don't want to talk about that subject. So we can condition it to do that. It's very similar to a human. Like if I put a human in a call center, I know they're talking about customers. I'm like, hey, don't talk about our competitors' customers, right? Or don't talk about our competitors. Just talk about our products. Yes. Don't talk about politics, right?

Don't use swear language. These are things that I might tell a human, right? And so we can actually use fine tuning to condition the model to give us outputs that are like that. You can even imbue new knowledge through fine tuning as well, but I would argue that doesn't work quite as well. The real way I think to describe or modify the behavior of a model in a very profound way is using pre-training and data mix. So pre-training is where we take a model that doesn't know anything and we train it on a bunch of data. And the way we do this is actually an optimal mix of maybe some domain-specific data along with some general data. It's actually kind of similar to education, right?

I have a child. I'm going to put them in school. I'm going to teach them about history and politics and math and science. And at some point, later in life, you start to specialize, right? It's actually a similar process with LLMs. And so in this example where I'm dumping in my startup data, what would be then the next steps for me to get value from it? What would I do once I've got the model set up?

Yeah. So I think the first thing is to analyze how much data you're throwing at it. So if you're under that 100,000 word limit, then you're probably in the regime of tokens. sorry, prompts into tokens. If you're in the call it 100 million so range, we can start talking about

fine-tuning. If we're now in the billion range, we can talk about pre-training and layering in this data. So that's really the analysis that we kind of walk our customers through typically. It's like which method you want to use depends on how much data you have to throw at it, typically.

So in your example, you said 5,000 transcript, something like that. That's probably in the prompt regime. We're probably not doing anything beyond that. Maybe able to use some light fine tuning to actually condition the model to act in certain ways. Like, hey, I want this kind of information pulled out. Like, I want to know something about the quality of the founders.

I want to, I want you to focus on that as an output, right? Got it. I can condition the model to that with fine tuning. Got it. So I could say, hey, what's the problem? You know, because typically if you backed out of a deck, the deck structure was architected to convince investors. Investors were optimizing for big problems, solving big problems with high margin businesses

with people who had great backgrounds who could execute. So you could actually like the deck having a competitive landscape or the total addressable market or the problem and the solution, those are the things that investors would go to first. We know this because when you send a doc you send or some of these tracking software is a little bit creepy, but it will tell you how long people spent on each page, which pages they zipped right over, like, advisors, who cares? You know, uh, you know, right. There's a lot of stuff is, uh, you know, thrown into decks just for performative reasons, but the problem and solution,

and the background of the founders are paramount. The number of customers and the pricing, paramount, the business model. So you fine-tuning would be essentially that process of trying to tell the model, this is important. That's not as. important. That's right. And you can even link it to outputs, right? We call this process reinforcement learning with human feedback, RLHF. And actually what you do then is you say, well, the inputs are all this decked material, say.

And then these companies did really well and those companies didn't. Right? You could actually start to link it to an output and you can start saying, hey, show me companies that you think are going to do well. Right? And it can actually kind of pick up some some patterns. And it would be different. For a C-stage investor, it would be they got through a billion dollar valuation. For a late-stage investor, it might be this company went public or got bought for over a billion dollars.

And so you could actually have two different outcomes could be defined as success. Totally. You know, for Y Combinator or our accelerator at launch, like, you know, success might be the company gets past $200 million because we're investing in low single-digit valuations, what companies are just starting out and they're just ideas. So you have a totally different approach. there. So you're in competition with some of these open source projects. Is your solution open source? And maybe you could speak to who's going to win ultimately having the great language

models. Is it going to be the person with the greatest data, the person with the largest open source community, fine-tuning the open source projects to analyze that data? Who, in your estimation, is going to win the day? or will it be parity? Where, you know, having a Web, a CDN, a content delivery network,

sure, there's on the margins, some that are faster than others, and you can probably have debates with a SIS admin all day long, but the fact is in 2023, you throw up any of the top 10 CDNs, your site's going to work really well. There's parity.

Right, right. Yeah, yeah. I mean, so our models are open source. We open sourced our 7B model, a little over a month ago, or a little under a month ago. And we are going to continue to do that. The reason being that we want to give people great starting points to get going. I think for us, what we're learning is our customers are on a journey here. This is all very new, right? It's new for every company right now. They all want to do it. It just comes, people come at it from different

starting points. Some people are like, okay, I'm 100% in. I'm going to budget $10 million. I'm going to go do this. Okay, great. We can help you with that pre-training side of things. Some are like, well, we're dabbling. We want to understand how we can add value to our customers. Can we start with a smaller bite? So we want to meet them where they are. And open source models are a great way to do that. They can start with the open source model. They can fine tune it. It's relatively cheap. And then eventually start integrating to the application and then customizing even further. So we want to, we have the whole breadth here. And open source is a very big part of that. I think who's going to win out of this is the one who can serve their customers the best.

I don't think those principles are going out the window. Everything you've talked about over many years, those things are still real, right? I mean, at the end of the day, you've got to give your customer something they want. If that means taking an open source model and fine tuning it and that's good enough, great. If it means that you need to pre-train something, that's fine too. If you're chat GPT and OpenAI, like, yeah, they got to go build their own thing because that's their competitive advantage. I don't think that's true everywhere. But what we are seeing now is that the game is ratcheting up pretty fast.

Like if I have something that I put in front of customers that interacts with specific kinds of data, getting really good at interacting with that data probably means you need to own how that model works. If you don't, your competitors can buy that thing too. If I'm integrating an OpenAI API, maybe that's a great way to get started. But I don't have much of a competitive advantage because my competitors can go do exactly the same thing, right? You basically have decided to be on par with everybody. and everybody will get to the same place, whereas if you have your own proprietary model

and you're tweaking it and tweaking it, everything you do past that open source moment where you use the open source software, you own and those are accrete to your product or solution, not to. And this is distinctly different than just hosting, picking where to host your server. When you pick to host on Amazon or Google

or Rackspace, Azure, whatever, the act of hosting on Azure doesn't make Azure or Google Cloud or AWS better. But the act of hosting on ChatGPT or Barr does make those models better, correct? And that is a subtle point where it can, I guess, yeah. Yeah, yeah. Yeah, so the physical infrastructure, it's interesting. I mean, Nvidia clearly had a huge bump recently. They are the backbone of all of this, both training and inference.

Right now, I mean, we're actually. actually encourage, we encourage many different types of hardware vendors to come to us and we want to run on their stuff. Nvidia is great. They build really good products and we are running on top of them. Then the clouds are sort of the channel through which you get GPUs, right? They also have some types of differentiation. I mean, network interconnect and, you know, reliability, failover, all this kind of stuff. You know, we find that it comes largely down to availability and price is the biggest differentiator along with some of these other more minor things like network capabilities. And really, customers want choice. Right now, they want their cloud is a relationship that they do a lot of things on

because they run their business on it. And they want some choice here. It's like, hey, you know what? I don't want to be like in a vice with one vendor because of this relationship. I want to have some choice. And so that's where the multi-cloud thing actually became a pretty good value prop

from their perspective. Not every customer, but some of them. In that case, I'm building AWS is giving me a great price but Azure just dropped the prices massively they're trying to win our business I've got to keep running this model growing it. It's not cheap

to run these models. Can you give us an idea of what my the job I gave you of my 5,000 meetings? What do you think this all costs to run these models at scale to add a thousand you know a thousand

new startups a month to it and really keep growing in? What is this going to cost. Well, I think there's some misnomer's out there that some people believe it's like, you need to be at $30 billion to build a model that even matters. That's not true. But I think like in the level you're talking about, let's start with your, you know, a few thousand documents. When you're talking about hundreds of thousands or even maybe tens of millions of words, it's really pretty cheap. This is on the order of $100.00 bucks, we could get, we could do a lot in terms of fine-tuning. A thousand bucks, you can do really a lot.

So it's really not that hard. But when we start talking about pre-training, building models from scratch, I'll give you the numbers. Our 7 billion parameter model was trained on one trillion tokens, one trillion with a team. So that's approximately 750 billion words. Crazy. A very long book has 100,000 words. So you can kind of do the math. It's a lot of content.

That model took nine and a half days on 440 Nvidia, A100 GPUs. and it costs about $200,000 to build from scratch. Just to build that one model one time and you run it again. You have to run it again. And you don't own all those. You rent those. You time share them on other platforms and these platforms in the cloud. Trovata is a cash management platform that helps you keep tabs on your runway,

which is super important when you have to answer investor questions. and this is just going to gain control over all your financial data for you. Trovada scales from seed rounds all the way up to your IPO, and it makes it easier than ever to manage multi-bank liquidity with a single source of truth. You know, everybody now is putting their accounts and your money. You're splitting it across multiple banks, so you're protected with that FDIC insurance. Startups shouldn't be managing their lifeblood, aka your cash position in a spreadsheet. No, don't opt for bulky solutions.

that take months to implement when the banks now have super-fast API connectivity, with investors like Wells Fargo and JPMorgan, Travada, has pioneered the largest library of corporate bank APIs, tons of unicorns like Carta and Fanatics Trust Travada, to gain visibility into their multi-bank data. It's the Cash Command Center that helps you analyze report and forecast cash like a pro. Recently, they also launched Travada AI. This is the first generative AI for FinTech that uses GPT to automate cash

boarding and business intelligence while keeping your data private and secure, of course. So here's your call to action. Go to Truvada.io slash twist to get started for free. Use the code Twist for 30% off premium features for one year like AI forecasting and reconciliation. Educate the audience on the utilization rates of these in the cloud right now because we're hearing, hey, A100s, H100s, whatever. There's a line around the corner. We saw Nvidia have this huge spike, a couple billion dollars and, you know, unexpected orders came in. So they're doing fantastic. But if you want to run one of these models for your company, are you waiting in line to get access to them?

Do you have to reserve them? Is there like a line out the door to just use them? Or can you just use them anytime you need to? Well, yeah, we are in a GPU crunch, no doubt about it. And that's not going to alleviate for a while. I'm happy to talk about why that is as well. Yeah, please. a lot of time in the semiconductor industry.

But right now, if you're willing to sign longer-term contracts, you can generally get them. So we, we as a company, actually have blocks of GPUs that we buy, and we can bring to customers. We call that a 1P deployment, a first-party deployment, where we basically create a tendency for our customer with GPUs that we already have contracted. So that actually works great. They basically pay us for a block of time, and we can run those things and get them access and do it very efficiently and effectively. The other way we deploy things is within the tendency of a customer.

So a customer has a relationship with AWS. They believe they can get AWS to give them GPUs. We can run our software stack inside of their tendency without ever seeing their data. That's something that people like because of the security and privacy. But as he said, the shortage of GPUs starts dictating how people go here. And so we actually do have a large number of GPUs. I don't necessarily want to comment on how many, but in the several thousands range that we can bring to bear.

Now, the reason this is, this is an issue is that we're seeing scale, scaling up these neural networks matters, right? For a 7 billion parameter model, I needed on the order of 4 to 500 GPUs. That wasn't true two years ago. People weren't doing this. And all of a sudden, it's like what you would do on 4 or 8 or 16 GPUs, now you're thinking I need 400. And so the demand just went through the roof.

The new H100, that's the latest GPU from Nvidia, is going to help a bit in the sense that each one is faster than the previous generation, so you don't need as many. But I think what will happen is it's sort of like goldfish, you know, you grow to the pond you have. Like, as the capabilities of the hardware gets better, people just want to use more of it. And I anticipate us using routinely 1,000 GPUs for customer workload. So that crunch is going to continue. Now, why do we have a crunch?

I mean, can't we just crank out more silicon, right? What I think the world doesn't realize is that there's really three places in the world that can build state-of-the-art silicon. TSM, Taiwan Semiconductor. That's the biggest one, and that's where Nvidia has a deep relationship. Samsung is another one that Nvidia also fans on, and then Intel as a fab. And Intel primarily focuses on CPUs for their fabrication capabilities. And then beyond that, there's something called high bandwidth memory, HBM memory. It's packaged within the same physical package as the GPU.

The process of packaging and getting memory together and making it all yield is actually the biggest bottle deck. There are two places in the world that make HBM memory, Samsung and S.K. Hynix. So this is your supply chain for these things. and there just ain't a whole lot more left of it. And to build out capacity means you got to build a whole new building, you know? And that's part of what the Chips Act was trying to do here in the United States is to create some redundancy, have some of these on the North American continent, and maybe have less dependency on regions that could be impacted by geopolitical events. Taiwan, fill in the blinds. Yeah.

And so ramping those fabs up is underway, but this is a non-dominous task. It is a significant task to put one of these. To stand one of these up, this is a couple-year process. Yeah, I mean, two to three-year process and, you know, in the order of $10 billion of investment to build a save-of-the-art fab, if not more, these things. So it's not small. And it takes time. I think that's the other part is that, like, if you want to react, to a change in demand, which there has been a big spike in demand, the reaction time is a minimum

to two and a half years just to build the capacity. Then you got to deliver that capacity. So it's another year beyond that. It's like a three year minimum kind of thing. But the software and the models are getting so good. Hugging Face has like a leaderboard of the models. You're in the top 50 models. And you have all these different players trying to make language models, open source them and make them better. So is it not trying to make true that these base models are going to be built and a lot of the demand to use them is not going to require, they're going to get so good that maybe you're just not going to require to do as many new models or is it just induction where people are like, well, I can make a new model,

I should run a new model for my vertical, etc. Yeah, I think what's happening now is where one can't get the resources, they're just going to take the other approach of, I'm going to use an existing model. Great. It's a practical approach. They are giving up performance knowingly. If they had the capability to build their model, they can and will. So I think the demand is not going to tap out because of this. Like if I want to build a better model to be competitive in my space, I will. And if I can get the resources, I'm going to go do it. I might be strapped by resources, not be able to get them. We are, we've taken a fundamentally different approach to a lot of companies in that we focused on efficiency of compute from the get-go. So meaning that can I do more with less? Can I build a big model and make it cheap? So the reason that our model is state of the art and only $200,000 is that we put a lot of engineering time and research time and making it very efficient.

We use that GPU completely. You know, it's like when I kill that animal, I'm going to eat everything, you know, kind of a thing. And we're going to continue to do that. So we're getting more and more efficient with it. But honestly, the demand is going so fast that even with our efficiencies, which bring nearly an order of magnitude of efficiency compared to what it was a couple years ago,

there's still not enough GPU compute. And I think we are an absolute requirement to make this happen, still not enough. People are going to be seeing that they can build a better model and get an advantage, and there's going to be an economic incentive to do it. It's just they can't buy the GPU. Talk to me about the difference between specialized models

and the general models and how this is going to play out, because, you know, Reddit, Bloomberg, Twitter, Quora, these are very unique datasets. Not only are the unique

datasets, they've already have built into them some amount of categorization, i.e. a subreddit, i.e. a topic on Quora that this is a legal topic versus a health topic. I know it's the model

can figure that out itself, but the fact is these are very structured sets of data that have been built for decades that are really unique in the intent in building them. Stack Overflow would probably fall into this. So how does this, I guess, balkanize or manifest itself in the next two, three, four years? Is Reddit just going to have a Reddit GPT and Cora already has their own GPT, basically, an interface, maybe Twitter has their own, Bloomberg created their own, a small investment I have, a small company I have skipped, SKIFT.com, created their own based on their reports of travel.

companies. They're kind of like a verticalized B2B travel publisher, plus the transcripts of all their interviews, plus all the research and the companies that they cover, and they've made their own narrow language model. How does this all pan out in the coming years? So I'll give an intuition first

before I go into an answer here. I think the way to look at it is, you know, if I want to be a, if I want my kid to be a famous violinist, what do you do? You don't start them at 20 years old. you start them at four, right? Arguably, if they're going to be a virtuoso and violin, they're probably not going to be a finance virtuoso, because they're going to put a lot of time and effort into making their brain very specialized toward that task. Even with everything that biology has given us in our brains, we still need to specialize to be really good.

So right now, we're at the very beginnings of this. Yes, you can talk about, I can build a model that can do a lot of things. it's going to be a jack of all trades and actually not a master of anything specifically. And that's okay. There are tasks where I want something general, right? If I want to take over multiple tasks that maybe people do or find mundane, I maybe want something general for that. And those general models will work for that.

But when I really start getting into healthcare, being a co-pilot for a doctor or a nurse, or a co-pilot for an investor, then I'm not. I need some really kind of specific knowledge. And it's very difficult to make a general model do really well in specific tasks. That's one. The economics of building a general model that could potentially be very good at every task start getting kind of out of hand. I mean, just the training of itself gets very expensive. Then, because that model itself has to be so large, serving that model just has very unfavorable economics, especially when we're talking about compute being so scarce, right? Training and inference compute is basically the same kind of chip. So

now I've got to start thinking about, well, all right, if I want to actually serve this model to my, to my customers, I need to think about the economics of serving that model. So I think we're going to be in a world where there are going to be some large general models and they serve some set of use cases and the costs to serve them is justified. There's going to be a whole tale of multiple expert models that are much smaller, that have much more favorable economics, maybe you're very good at particular tasks and not and less good at other tasks. Like, if I'm building something that's going to do customer support. I really don't want it to philosophize by why Rome fell, right? It just doesn't need to do that. It needs to talk

about my products. It needs to get the user, you know, to fix their problem, ASAP. That's it. I don't want it to do anything general. So I think this is what we're going to see is this world where everything's kind of coexisting and solving different problems. We're already seeing that now. I mean, I talked to the founder of a company called Perplexity AI, which is doing like, kind of, you know, search and, you know, finding knowledge across different sources using LLMs. And they're doing a whole bunch of different things. They use every possible model they can to best serve the task that they have, right? So sometimes they use a general model to do filtering and they use specific models to condition the output the way they want them.

So I think we're going to see this world where everything kind of coexists, which is going to be a bigger market. But our bet is that people constantly building experts on their domains is going to be the bigger bet. And the other one will be a consumer thing. Maybe it'll be different. I don't know. But I think in this world that's coming, we're going to see just a proliferation of all of these capabilities out there.

And the markets are going to be enormous. So it's almost like not worth sweating the details right now. All right, everybody. Our friends from Microsoft are here. Tom Davis, a senior director at Microsoft for startups. and you're a former founder. So, Tom, tell me the Microsoft for Startups Founders Hub. What is it?

And what are you offering startups? Run us through the bullet pointed list of all these incredible benefits. There's lots of them. So we start with up to $150,000 worth of credits for Azure. That is not just traditional Azure, but also the Azure Open AI service, which is all the rage at the moment. You get benefits as well for productivity tools. So Microsoft 365 with Teams and Office.

in there. Developer tools, GitHub, Visual Studio, but also third-party benefits. So like LinkedIn services as well, you can get access to bubble. But as well, we have a special benefit with OpenAI, up to $2,500 with OpenAI. So you can leverage the latest and greatest models that are coming out from Open AI. And when you want to go into sort of production and reliability on services, you can shift across to the Azure OpenAI services that you get with $150,000 worth of credit. Amazing. Well done. And if anybody wants to sign up for that, do it now while you are in front of your computer, aka.m.s slash this week in startups, aka.m.S. slash this week and startups, well done, Microsoft and well done, Tom. It's part of our mission to democratize access to

innovation. So the more we can do for startups, wherever they are, whoever they are, the better it is for society in general. What do you think, as an insider, the impact is going to be on employment? So we'll go big picture now. We got into the details of these models. Congratulations on being one of the top 50. Yeah, it seems like you've got it dialed in. There's going to be tons of use for this. But what people are sweating is, hey,

and I got my own feelings on it, but I'm curious yours. Do you feel like even in your own, I think you have 60, 70 people in your startup? Do you feel like you need to hire as much? Or do you feel like as the CEO founder, co-founder here, your time is better spent, taking the 67 brilliant people you've already assembled and just trying to make them 30% or 20% more efficient

using AI tools, where do you spend your time? Hiring the next incremental person or making the existing team better at what they do? I still spend a lot of time hiring. Great people are still very hard to substitute. I mean, these models can do something that at a 20th percentile human, I need 99th percentile players.

Got it. Right. So I think there are things that 99th percentile players can that very few other humans can do. So I spend my time on that. So elite is still elite in your world view. The elite are not impacted by this trend. Well, okay, let's go down the, the, uh, let's go down the rabbit hole.

At least not yet. Okay. But I think, but I think to your point, right, even if I can make those elite players 20% more efficient, that would mean, I would imply I need 20% fewer of them, right? Right. I mean, making Steph Curry, that has it wrong. If you made Steph Curry 2% more efficient, it would just, destroy the league. Like this, he's already too efficient.

Yes. Right. So if you're thinking about all stars, can you imagine making LeBron James 20% more efficient? I mean, what happens to the league? It's insane. Yeah. No, it's insane. And I think throughout my career, and I was here before the dot-com bubble, and I've been a

tech maximalist. I've felt that tech made the world better. Efficiency made the world better. Sure. I've changed that a little bit because of this new world. And it's, and I'll tell you why, it actually has nothing to do with. the technology, but more about the pace of change. What worries me is if I make the 50th percentile player 30 percent more efficient across the board, I have, the change in demand won't be as fast as the

change in supply. And I think that's going to create this window of time for 30 or 40 years where we haven't figured it out as a society. And I don't know what the answer is. And that's the thing that worries me, to be honest. I do think the tech is going to happen. I think, It enables humans to do more and to, you know, strive to solve bigger problems. AI, in my view, is the next evolution of what humans can do. You know, language was a big technology that humans used to pass knowledge. That exploded what humans could do and the influence we could have on the world. AI is going to be that next inflection point.

But how are we going to make sure that everyone has a place in that world? How are we going to make sure that the demand that's created by the increase in inefficient is commensurate or does it collapse, right? So I don't know the answer, but that's kind of why I've taken a step back. You're worried at this velocity, if I can summarize it correctly here and reflect it back to you, which is an important thing to do in discussions. The speed at which the efficiency is going to impact, you know, the 50th percentile below could be so violent, so fast.

It could happen so quickly that those people, the demand for those people who don't make the jump could be so low that they can't catch up in time. And then they've got some number of years of their careers where they are sideline marginalized or otherwise not needed, which is scary. That's right. And it did happen quickly with things like the typing pool in, I don't know if you're all enough to remember, but law firms or, you know, many businesses. would have a photocopying room, a mail room, and a typing pool. And a filing room, right? And the filing room eventually gave way to Box or, you know, Google Drive or whatever. The typing pool, everybody just typed their own stuff, and mail became email and docusign.

And those rooms, the photocopy room as well, went away. They don't exist in a modern office, whereas those were half of a modern office's floor space previously. but that took how long 10 years maybe 15? Yeah, something like that. I'm talking about something that could change in three years, right? Wow. And I think the other part of it is that, you know, if you look at the mail room and the filing room, right, perhaps that increased efficiency sum total for the business 5%, let's say, and it took 15 years.

We're talking about increasing efficiency by 30% in three years. That shift is so fast that like, okay, so you. Yeah, or even a year, right? It could just be like, boom. You just can take on a new tool and all of a sudden it goes away. So what happens then, right? So we increase efficiency and delivery of goods and production of goods. But now there's fewer people that can pay for it. So what happens, right? As a society. And this is where I think some people's minds go to UBI and then other people's minds go to entrepreneurship. And it does really depend on, I think, your framing or worldview. If you're an entrepreneur, I think you're mind goes towards, well, start a business, or find more customers, lower the price for whatever service you're doing. You think about radiologists who, you know, look at, you know, x-rays or computer, you know, generated x-rays, et cetera. It's pretty obvious that AI will absolutely do a better job in, you know, for most of that job in the next year or two, if it's not already done. So then what happens to those folks?

Well, we could do more MRIs. We could lower the price of an MRI. We could let people take more MRIs or CTs or PETs, all these different tests. What if we lowered the cost of those tests so that when your doctor was making a decision, she wouldn't have to say, I don't know about that it's worth it. It's like, who cares if it's worth it? Yeah. It's not $900.

It's $100. So go do it. Yeah. We can do it 10 times as many. Yeah. And that's the world. I want, right? And that's where I would restore my faith in tech maximalism, right?

Where we can do that. And we actually just do better at everything. We did it already. There's an example. cheaper, faster. Remember food insecurity? Yeah. This concept of food insecurity.

And now what do we have? Obesity. In the 80s when we were growing up, I don't know how I'll do are, but, you know, but in the 80s, you know, we had live aid and we were trying to feed Africa that was like, oh my God, this was the cause of, you know, Africa has no food. And, you know, now if you don't have food in the modern era, it's because some dictator in all likelihood has blockaded food from reaching you.

And the biggest drug in the world right now is our Zampic and Mugovie, because we have an obesity problem in abundance. We could have an abundance problem. We could have an abundance problem in health care in our lifetime. Too many doctors, too many nurses, too many beds, too much available. You're going to be too healthy because we just figured everything out. Kind of like the abundance angle. Again, great. I want that to be the case. And I do go in my own mind to entrepreneurship. I just don't know if everyone's wired like that. Is this what worries me?

Every human has the motivation to become self-reliant, radically self-reliant, and hunt for their meals as opposed to punch a clock and get their meal ticket. It is a... That's right. That's right. What do you think is going to happen with education? This is the one I think is super fascinating. Because I'm learning so quickly right now. I agree. It's just using chat chTPT as my default browser.

When I open my browser, it's my default window now. And I'm retraining myself to use chat chpt4 as my first line. And man, I'll be on a podcast. And I, like, when I'm talking to you, I might, like when we were just talking, I said, what Y College jobs would be most impacted by AI? and I saw radiologists on the list and I was like, you know,

just for brainstorming, I was like, yeah, that's an obvious one. And that's how I did that throughout of a conversation was AI got to radiologist before I would. Pretty amazing.

Interesting. Yeah. No, I think for education, right, it's, it's going to be, I have,

I have kids. I have kids in high school. And, you know, there's a traditionalism in education that again, kind of goes on a very long time scale, right? People are like, oh, liberal education, you need to do this, you need to learn that. I take a different approach where it's like, all right, you know, chat GPT is here.

My kid told me he submitted a paper written by chat GPT, and I said, look, I don't want you to be dishonest. I'm okay with it, as long as your teacher's okay with it. So if you tell your teacher and she was okay with it, I'm perfectly fine with it because that's what your world is going to look like, right? And learning how to wield these tools and make them really effective is going to be how you differentiate yourself. So I think education should be more about exposure to these tools and in solving problems directly, as opposed to sort of memorization of knowledge, which was sort of human 1.0, right? We had writing and human 1.0 was like, okay, if I can memorize stuff, I know something. Others don't. That's gone, right? I have Google. I have chat GPT. I have all that stuff right now and I can, I have access to everything that every human, yeah, has ever, has ever written in a scientific paper.

Yeah. And you can get to it quickly. You know, the thing I find is interesting about kids. And I'm, I'm big on this Montessori and like base level learning and like this regio learning where you follow the kids instincts. And if they're really into something, obviously the aperture for learning goes way open. You know, if it's about something. Some, you know, orcas, my daughter was into, you know, killer whales for a little bit. And she, you could teach her anything with killer whales. She would do math, physics, as long as it was with a killer whale, you know, as the thing we're weighing or the thing and the force of the killer well. Like, she's going to be really into it. So that's awesome. But just personalized learning and unlocking student creativity as but two measures here. You could take any personal lesson plan.

And I could say, hey, take this lesson plan for history. And, you know, let's have an approach to it that includes superheroes. And it's like, what? How do we include superheroes? And it's like, oh, well, yeah, they did actually use Captain America to, you know, study like, you could make a Captain America going through different world wars and or Spider-Man doing it. It would actually make total sense, actually, to that person. And then they would be drawn into it. Spider-Man teaches you physics.

Great. What could be better? Right? Yeah. The personalized stuff to me is amazing for kids' brains. And that was what the Vulcans were doing. You remember in Star Trek when the Vulcans would go into those little pods? There was like one episode of the Star Trek series where like, I don't know if it was one of the reboots, but, you know, like they just put Spock in a pod and he's sitting there in a pod and the computer is just throwing information out.

I mean, he's learning. Like, I just see that as the future. The AI knows what you know and what you don't and is going to present the next lesson plan that you're most. open to and will be most accretive to your life. That's wild when you think about it. I don't know. I find myself optimistic. It's like we're hacking a human learning process, you know.

Are you optimistic right now watching this? Because the pace, you've been in this for a while, but the pace was very slow and then it's suddenly breakneck, which, you know, Elon and some other people did predict that this will be slow until it's cataclysmic. What do you think? pretty accurate, pretty accurate and why? Why is that so accurate if you think it is accurate? Well, I, okay, so cataclysmic, I think is the wrong word.

I think it is breakneck. I am very positive. There are things I still worry about, but I'm still positive. And I think we are, what's happening now that I think is a bit annoying is that cataclysmic kind of rhetoric is being used in self-serving ways, you know, in anti-competitive ways. example. I think we're still in this phase.

Open AI, closed AI. Well, I mean, I'm perfectly supportive of them being closed. They should be able to have their own competitive advantage if they want. Totally fine with that. What I don't like is talking about regulatory agencies issuing, you know, certificates of you may now go train a model. I mean, come on. Really?

We're so much at the beginning of this whole journey. We don't even know the value of a model. We don't even know how we think about the data. that went into the model. We don't even know the use cases for most of this stuff. Let's let the flowers bloom a little more. And then we'll start understanding the bounds of where the incentives break down when people don't own things and all that kind of stuff and get to there. I don't think where, you know, the end of the world is nigh. I really don't think we're that close to it. I think people are using that fear right now to. Why do it start going so fast? Are they using that

to do regulatory capture? Yeah. Exactly. Maybe pull up the ladder behind them. I do have to say, I do find it questionable that opening I became closed AI. The whole premise, and I've told Sam this, and I've said it publicly like, the whole premise was this is too dangerous for people not to see what's going on. And then they said, well, it's too dangerous for people to see what's going on. So how do you make that crazy shift? It's almost like this we know better than everybody else.

But if you go on Hugging Face and you look at the 50 open source models, they're doing it open source. So what's so unique about Open AI that they get to make this decision? And in fact, Google engineers, you must have seen this in leaked documents said, and I'll just quote, we've done a lot of looking over our shoulder at Open AI who will cross the next milestone, what will be the next one. But the uncomfortable truth is we are in position to win this arms race and neither is Open AI. While we've been squabbling a third faction is quietly eating our lunch. I'm talking, of course, about open source. plainly put, they are lapping us.

It's crazy. Yeah. Why is Google so scared of open source? I think it's hard for any single organization to compete with them. Well, because it's hard to compete with the whole community, right? There's this unleashed creativity from many, many people. You just can't, you can't compete with it. That's been traditional in software, right?

I mean, you've seen this Linux, all this, like, you can try to centralize it. And maybe that's the activation energy to get it over a hump, but to compete with it is very hard. And I think that's why everyone's scared of open source. But I think back to a philosophical point, I 100% agree with you. It's like, why do you have the mandate to dictate what this technology will look like? That's the part I have a problem with. And I think the way to solve that is actually distributed capabilities. Many people having these capabilities, right? It's like, yes, there are economics involved and it's expensive, but we can make those economics a little bit more favorable by time slicing. It actually looks very similar to semiconductors. We actually call ourselves the LLM foundry,

very similar to, yes, there's a large investment required to build a foundry, but once you do that, the incremental cost of making a chip actually isn't terrible. And by enabling many to build these chips, you build, you know, Apple builds their own chips, Qualcomm builds their own chips. You enable this whole ecosystem. And I think that's how we solve this is through almost a market solution, and not centralization, and it's like, you know, sort of paternalistic centralization. That's the issue I have.

And maybe that's just the, that's the entrepreneur in me. I hate when someone tells me that I'm allowed to do something or not, right? I mean, there's, I think it's great that we're having conversations about how fast this is moving and the impact it's going to have on society

because usually everybody's very late to that party. And so the fact that we're doing it in real time for the first time, it felt like they played catch up with, you know, social media. They play catch up with the regulatory framework for a crypto. But here we are. We're looking at AI, which will have certainly a bigger impact than crypto did, obviously.

And it will, yeah, I think it will probably have a bigger impact than social, even though social has impacted governments, media, people's help, their psyches. This should be a bigger thing. And it's actually good that we're having the conversations. if some people want to regulate it for nefarious reasons or to pull the ladder up, I think we can see that happening, but at least we're aware of it. And it's great that you're building something that democratizes it a bit and levels the playing field. So companies, individuals, nonprofits, whoever can start building their models in a more open source freeway and then portable, right?

I mean, the portability is also super important that no one person owns this hardware stack. And that's not going to happen, right? It's not like there's any hardware advantage that's going to accrue here. There's going to be many competitors to Invidia in the coming years. Do you think? Or you think they're going to run the table. Well, I think what Nvidia has done well is just they've executed really well against those competitors that have tried to come out. The would-be competitors.

And that's why they've continued to maintain advantage, which is, again, they did exactly what they should do. And I would argue it's because of Jensen's leadership being. you know, here, this is what we're doing, right? Very tops down, very strong-handed. But I think there are going to be competitors. And for the simple reason that we need more supply. Yeah. What worries me, however, is that the supply, even if there were 10 Nvidia's out there,

we may not have that much more supply simply because the bottlenecks back in the supply chain are further back. Yeah. The memory chips. You know, memory and packaging, right? Yeah. So, but yeah, I'm encouragement. I encourage anybody to build. competitor and make it. You know, what might be interesting about it is if the, if it turns out the hardware

stack throttles this a bit. And that could be a built in throttling. Then we don't need the government to get involved. It's like, hey, we're going to be able to make so much progress here, you know, without the hardware stack dramatically increasing. So, all right, listen, amazing job, continued success. You're hiring. You mentioned, All-Stars 99th percentile. How can people learn more or how can they see what jobs are open, who you're looking for, et cetera? Let's get you a couple of employees for coming on the show. Yeah, absolutely. Go to our website.

We have several listings on there for careers. And, you know, basically people who are builders, innovators, this is what we need. We're a small team and we rely on, you know, highly creative individuals who are amazing at what they do and you can implement them fast. And if you feel like you're one of those and you want to make a difference, come talk to us. MosaicML.com slash careers. All right. We'll see you all next time. on this week's service. Bye-bye.