This Week in Startups - The Ex-DeepMind Researcher Taking On China’s Open Source AI

Starting point is 00:00:00 All right, everybody, next on the program here at This Week in Startups. I am super excited to have one of the few leaders here in the United States for open source models, which is something we desperately need, as you've heard over and over again here on this week in startups, this week in AI, on the All In podcast. We are very much in the lead when it comes to Frontier models. But Frontier models have many shortcomings, have many Achilles heels. One of them is that you can be rug pulled at any time, and the frontier model could take your model away from you.

Starting point is 00:00:35 In other words, they're not reliable. Another is the frontier models are viewed increasingly as competition. Number three, you are renting your intelligence, you don't own it, you don't have sovereignty over your knowledge, over your business. There are a number of important issues that founders have to make when picking between open source and frontier models. And so we're going to talk about all that today with Misha Laskin. He is the co-founder and CEO of Reflection AI. That's a lab here in the United States, an American company that is building open-weight AI models that companies, governments, organizations can

Starting point is 00:01:14 run for themselves. Misha was formally at Google's Deep Mind, or Gemini, working on Gemini, I guess. the branding over there is a little bit confusing. But welcome to the program, Misha. How are you doing, Misha? I'm doing really well, Jason. Thanks for having me. So you heard my little preamble, which is now kind of becoming codified, I think, in the past three to six months in almost everybody's mind who is applying super intelligence, trademark

Starting point is 00:01:47 President Donald J. Trump. So have you started using super intelligence? Have you stopped using AI? Yeah, we started using superintelligence years ago, well, a few years ago. I mean, that was kind of the, when I got into AI and saw AlphaGo happen, that was the first semblance of super intelligence. And it was pretty, we thought it was a general technology. So it was, you know, something codified some years ago, but I think now you're actually starting to see a general superintelligence. and that has some pretty remarkable consequences.

Starting point is 00:02:22 Yeah, so let's start with, you've heard this cadre of issues start to become codified into almost a platform that people working on AI every day, people working with superintelligence every day, are now in general agreement, dare I say, alignment on. So you've been working on open source for a while. I think in some ways the world has caught up. What is the reason why corporations, start-up, smaller companies, and even the sovereign individual,

Starting point is 00:03:00 why should they care about open-source language models and open-weight language models? Why should they care? It's a really good question, Jason. And I actually think there's something that you said earlier that hits on the point, which is this notion of a rent- versus ownership market for intelligence.

Starting point is 00:03:20 We can think about the analogy to real estate and real estate you have both rental and ownership. And typically the progression of a consumer is that you start off in the rental market. And then as you grow up, you get a family, you have kind of more financial maturity. Ownership starts making sense for a number of reasons, even if it's a pain to own a house, right? You have to fix the pipes when they're broken and so forth, but you own it and there are benefits to it. In AI, the commercial market for AI has only existed for the last few years, three, four years since Chad GPT came out. And the initial market for AI was entirely a rental market, right? As early adopters come in and are testing things out, trying out workloads, you know, haven't really scaled to production in a big way.

Starting point is 00:04:10 That's when a rental market makes sense. But by the time you scaled something to production, and this is your basically almost core asset, core IP, right? You're generating $100 million or spending $100 million on it. Ownership starts making sense for a number of reasons around control of your data, lowering your costs, being able to customize the model against your data, being able to optimize your margins. All these things start making sense when you're at scale. And what we've seen is that the AI market has matured very quickly, exponentially.

Starting point is 00:04:44 And that's why six months ago, no one was really talking about open source. And now it's the opposite. Everyone's talking about open source. And it's because the market has matured exponentially. And a lot of buyers are looking to own their intelligence now. This is a really interesting point that you're making. There has been an evolution in how the customer thinks about intelligence. rented versus owned and operated. Now, owning something has additional cost. When you're a renter,

Starting point is 00:05:17 you're happy, go lucky. You know, if the furnace breaks, if you have a flood, the landlord's in charge of that. But of course, the landlord can kick you out, change the price, and generally jerk your chain anytime they want, as we've, anybody who's rented apartment probably well knows when they get their new lease and it's magically going up 30% or maybe your dishwasher doesn't get fixed for two weeks. So let's talk about this change that's occurred and then when an enterprise, when an individual starts looking at going open source, which are the pieces of the infrastructure that matter? Right now, people have been obsessed with the model and the raw capabilities.

Starting point is 00:06:05 of a model and we have all these benchmarks and tests that the LLM seem to be optimizing for, almost like SAT nerds or LSTAT nerds where they're just trying to ace a test. But then there's reality. In reality, if you could explain to the audience, the value of the harness, the training, reinforcement learning, and the weights, and how that is all impacted on the open source side and the orchestration layer. So walk us through what the next year is going to look like and how people make this decision,

Starting point is 00:06:42 evaluating each piece of the stack and what's important. Yeah, I think that the place to start, you know, for understanding what does it mean to own intelligence? It's really unbundling everything that comes when you rent a token, right? Everything that you get when you rent a token. When you rent a token from a closed model provider, you're not just renting a model.

Starting point is 00:07:05 If we look at the life cycle of token, there are GPUs set up somewhere that are managed. So there's a data center of GPUs, bare metal. On top of that bare metal, there's cluster management software. It's GPUs or other accelerators have failure rates. You need cluster management software to just effectively manage that cluster. And then after that, you go into the inference layer, which is just the infrastructure for serving a model.

Starting point is 00:07:30 then you have the model, then you have the harness on top of the model. And when you are renting a token, you're actually renting a piece of that whole thing, right? Now, when you want to own it and now bring a model into your own infrastructure, all of a sudden, all those things in between the cluster management software, the inference software, the harness and so forth, all those things, the sandboxes, well, someone needs to help you, stand those up too. So ownership means not just owning the model, but the whole thing around it. And that is kind of, well, both where the challenge is, but also where the opportunity is to support enterprises. Yeah, and it's super interesting because when the PC revolution started, when I started

Starting point is 00:08:15 my career in the late 80s, you had this plug-in-play architecture. If you were going to bring computers into your organization, just the desktop PC, you had to start making decisions on each of these components? Would you have a token ring or Ethernet? And would you connect all your computers to a server in your server closet? How would the cables run? Did you have an Ethernet card? Did you have a token ring card? Did you have Banyan Vines? Did you have a hard drive? Or was the data stored centrally? What kind of video cards? All that stuff was the work we did in IT. And it was arduous and painful. And then at some point, it all became abstracted and simple. And so is that the business you're in now is saying, hey, we're going to make a great model and will give you the entirety of that

Starting point is 00:09:02 in a similar way to how Open AI and Anthropic do, but in a way that you'll own it. Is that eventually what Reflections Business becomes? That is what the business is now, right? By having built the model, we actually have to build the entire stack at scale. Like, we serve for our own training rounds and for our own organization, trillions of tokens, very, regularly into the org. Our large-scale reinforcement learning run, which was the largest RL run, we are aware of that's been documented in open source, served something like a billion sandboxes. So a billion agents spawned in and out, right?

Starting point is 00:09:39 And what we've built is battle-tested infrastructure at scale, and we are externalizing that to customers. So it's actually, it has a very similar analogy to how clouds like AWS came to be. Because initially it was that you had a web service that was being used at scale, which was Amazon. And they basically dog-fooded their own product, and out of it came a cloud that works at scale. And in the AI era, the model builder is basically building this cloud at scale in order to train these models. And then you can, whereas a closed model provider will then package it all up and rent tokens to you. And open model provider that's really, you know,

Starting point is 00:10:18 being able to train models at that scale can externalize that stack for the customer. When you're building a startup, your legal paperwork is spread out all over the place. Entity formation here, fundraising over there, cap tables, contracts. It's endless. But paper finally has a solution that's going to save you so much time and so many headaches. Papers introducing smart entities. Here's how it works. You tell paper what you need to do, and their pre-built agent takes care of all of it.

Starting point is 00:10:46 That's it. This is not some chatbot you need to provide with a ton of context. Nope. Paper is an AI native platform. that's going to manage all of your startup's legal work. That means no more switching between disconnected tools, keeping track of hundreds of documents, and way less busy work for you to stay on top of. That's why Paper is already the legal backbone for thousands of startups and funds. Right now, founders using my link, getpaper.a.i slash twist, get their cap table free for one

Starting point is 00:11:16 year. What an amazing deal. Or two years if you're switching from Carter. If you're building a venture-backed company, it's time to build your smart entity on paper. And do you have customers now who are saying, hey, we want to stand up this infrastructure in our own data center, on our premises, on-prem, data center, co-location, whatever it happens to be, are people now starting to do that? And when they come to you, is it experimental? Like, hey, we're working with the frontier labs over here. That's our primary, and we want to have this as a backup plan? Or is this now plan A? Is it plan A, plan B? and take us through some of the organizations you're working with,

Starting point is 00:11:56 whether you can say their names or not is fine either way, but you can give us the abstraction, you know, it's a finance company, it's a medical company, et cetera. Yeah, so to let's start with the, well, let's start with the first questions. So, yes, this is kind of what we're seeing, demand from large enterprise, sovereign public sector agencies is the notion of taking in workloads that used to be rented on a closed market. model and now standing them up on their own infrastructure.

Starting point is 00:12:27 So you might be, you know, when it's a sovereign, it's typically a large conglomerate within some allied international country. And they, you know, they want to get into the token business, right? They want to stand up a cluster. So they'll get a partner to help them stand up the bare metal. But then they want to serve tokens into their region, right? And the only way for them to serve tokens and basically become the intelligence provider for that region is to work with an open model by definition because that's the only thing that they

Starting point is 00:12:56 can take and run on their stuff. So we help, you know, those kinds of conglomerates. We're working with them in a number of countries. There's also just Fortune 500 enterprise and it tends to be somewhat similar there, like the very large Fortune 500s operate, well, like in some sense, like an international conglomerate. They, you know, want to move intelligence onto part of their like intelligence workloads onto their own infrastructure for a number of reasons. And to answer your second question, the way we've seen it works is that the customers that want to do this are already pretty AI mature, where they have workloads that are running in production on closed models, and they want to optimize those.

Starting point is 00:13:39 They want to make them more cost efficient. They want to not share their data. If you're a bank and you're scaling something out to 100 million users, you want to own that as much as possible. If you're a large retailer and you, again, you have shopping assistance or things of that nature, right? You want to own that as much as possible. If you're a telco and you want to roll out your own version of a chatbot type of product for your region, you want to optimize those costs as much as possible. So the customer set that we see a large pull for open models are ones that are already pretty mature in their AI adoption. And that is

Starting point is 00:14:19 something that we also saw in the kind of AI natives. The AI natives started building on closed models. They were tiny startups, right, the cursors of the world. And when they grew very quickly, at some point, they started switching to open model workloads, right, with cursors, Composer series. And we're seeing a similar transformation happening in enterprise. So tell me about doing a run now and building your models called Beam, yeah, P-E-A-M for the folks in the audience. And I understand this, like, to do a run and to build a model, this takes tens of thousands, 10,000 minimum GB300s. What does it take in terms of infrastructure footprint today to run one of these models?

Starting point is 00:15:04 And then where does this model kind of sit in the GLM, DECC, you know, up to Gemma and Nemotron competition? It feels like there's a lot of parity starting to emerge. in the open source models. So the nice thing about building open models is part of the mission is also being open about the science around it and the infrastructure around it because we think it's important that, you know, it's super intelligence

Starting point is 00:15:34 or artificial intelligence, whatever we might call it, is as fundamental as something like calculus and calculus is something that everyone is exposed to in school. And so it's very important that people understand how these things are built and that more scientists or are able able to contribute to the safety investigations, the capability investigation, and so forth. So our model, this one, was pre-trained on around 6,000 GB-300s for a few weeks, and then

Starting point is 00:16:04 scaled with reinforcement learning on about 10,000 GB-300s for about four weeks. And that is as far as we're aware of the largest RL run that's been done in open source. You're kind of seeing the shift of flops from, I mean, there's still a lot of pre-training research to do. So, you know, there's a lot of advances to be had there. But it used to be the pre-training took up 99 or 95% of all compute, and then post-training was this cherry on top. And now there, I would say, equals.

Starting point is 00:16:34 Got it. What happens in the two different segments? How are things splitting up in terms of what intelligence and what capabilities are happening pre and post today in open source. So the way to think about this, and it's actually, what's interesting is that it's the same recipe that produced scaled up breakthroughs in AI before language models as well. At the time in the system called AlphaGo, which was a super intelligent Go player. Go the game for people who don't know with marbles on a big board.

Starting point is 00:17:08 And if you were to give people a little background on why that's important, it's much different than checkers and chess in terms of the complexity of it, yes? Correct. So Go, this was a system that my co-founder, Janus, was a key core contributor to, is Go is a game that has the most complex what's called branching factor among traditional board games. So chess has a certain branching factor, meaning there's certain move possibilities that you have for every board position. Go has a much higher branching factor, and so, that reason, it's actually very hard for people who are playing it. They tend to play it by intuition. It's really hard to reason through all the possibilities. And so people kind of have the best

Starting point is 00:17:53 players have a sort of innate intuition for that game. And that branching factor is what made it hard. And the system called AlphaGo, this was a breakthrough before language models, was considered the biggest breakthrough in AI to that point. Was an AI trained very similarly to the way language models are trained today, that produced a system capable of beating the world champion at the end in Go. And that was something that was considered a breakthrough then. Now, the reason I mentioned the system is because it was trained first by what's called imitation learning, meaning that the model looked at a bunch of amateur games online and just

Starting point is 00:18:32 learned to imitate humans. And that kind of gave it some basic semblances of the kinds of moves that are possible and what are reasonable things or slightly unreasonable things to do in the game. Gave it an intuition. And then its capability was scaled up from that to a true superintelligence with three enforcement learning, which is the notion of learning through trial and error. In a way that you kind of learn to play basketball by, you know, you shoot some hoops, you miss, then you recalibrate, you learn.

Starting point is 00:19:03 Humans do this all the time. Or like Pavlovian conditioning, right, when you condition your, you're, your dog to do some tricks by giving them snacks, right? That's reinforcement learning. And that is the thing that scales from this, you know, intuition of what's possible and being and acting reasonably to becoming super intelligent. Now, the reason I say this is because that's exactly the way in which language models learn.

Starting point is 00:19:26 You can think about pre-training, which is learning to imitate the internet, as learning all the reasonable things that you can say, right? The internet is basically a pool of coding knowledge, of writing knowledge, of all sorts of knowledge, and it gives you a nice prior. You're not just kind of saying random stuff, right? You're saying stuff that seems reasonable when you learn to imitate the internet.

Starting point is 00:19:49 And after that, to go from that, which is basically where language models started, which were chatbots in the Gemini 1 and chat GPD era, the GPD4 era, they were just chatbots. To move from that to the super intelligent agents that we have today, you need to scale up with reinforcement learning.

Starting point is 00:20:08 And that's why we've seen so much compute being moved from pre-training as a proportion of training to reinforcement learning. And now these systems, you know, are in a sense balanced and sometimes reinforcement learning even takes more compute than pre-training to scale. Every startup we get to know through launch is using AI tools all day long. But a lot of the people who are at these companies, they're kind of using it like a chatbot still, right? rather than boosting their productivity across every application, every piece of work they're doing. And that is what you're going to get done when you use Superhuman Go from the makers of Grammarly, one of my all-time favorite products, which I use to this day. With Superhuman Go, I'm not constantly switching between tabs, copy and pasting,

Starting point is 00:20:52 all these transcripts and documents, nor am I writing props. It's just sitting there in every email, every document, when I'm on Twitter, when I'm on social media, every spreadsheet that I'm looking at, it knows all about my firm, knows about all the podcasts and the work I'm doing, and it just takes over all those repetitive tasks that I'm doing. And the best part is that Superhuman Go is proactive. So if I'm writing an email and I put it in the wrong day, and I put it's a Thursday when it's a Wednesday, it's going to fix that for me immediately and alert me to the change. Superhuman Go spots all of your errors before they go out, and that's just pure magic. So give Superhuman go a try for yourself at superhuman.com at superhuman.com.

Starting point is 00:21:28 How does the reinforcement learning actually go down? Do you have known feedback that's given to a problem where it's like, hey, here's a problem, and then a human gives feedback, or you have like some set of data that you use to make sure it gets the answer correct? Or is it just giving answers to queries that humans have previously asked for? What's in that chunk of data that is used for reinforcement learning? Give us some details, some practical examples of how that actually goes in. So the learning of deep learning as a field, which is the thing that powers this, you know, the current generation of intelligence is to scale data and scale compute. That's basically and then they kind of need to happen together. So for our reinforcement learning run, we collected almost a million tasks. And these are a million tasks that span mathematics, coding, agentic tool use, science and so forth. And the best tasks are the ones that are verifiable, meaning that you have a task,

Starting point is 00:22:36 you have, let's say, an environment, meaning like a code repository or some context, and then you have a verification signal of whether the task was done correctly or not. That's the best quality data. Assuming that you have a verifier that actually verifies whether the task was done correctly or not, we can get into hacking verifiers,

Starting point is 00:22:58 is the source of like the cyber madness that's been happening recently, but I'll sidebar that for a second. Yeah. And then, you know, once you've exhausted the tasks that are verifiable, you can start using language models themselves as verifiers, as judges. You can ask a language model, hey, was this task done correctly? You know, was it done stylistically correctly? Were there errors in the arithmetic?

Starting point is 00:23:20 You know, where there are errors in the syntax and so forth? But fundamentally, you just need to collect many of these tasks and the more verifiable You get the better and scale compute proportionally. All right. So let's go to this hacking piece and the cyber piece. And how you know something is actually verifiable or verified if, you know, you want to know if something was put on somebody's calendar, I guess you could look on the calendar or you could look for the confirmation that they said, yes, I'm coming to this.

Starting point is 00:23:50 You know, for a chess move, you can see if it's check made or not by a simple set of heuristics, right? So there are some things that are obviously verifiable. And I'm assuming the data labs are part of this effort in terms of, you know, we have one we invested in micro one. And we did their seed round back when they had a different business model. It was recruiting developers. But they just today announced they hit a billion dollars in their run rate, not their reoccurring revenue, but their annualized revenue. And so this is like a very big business in selling. to the labs data to do learning.

Starting point is 00:24:29 So is that what's happening here? Like where you get these million tests? Is it from these labs or you have to make it yourself? How does one get those? And then we'll get into the verifying and how that all works. It's a combination of both. You both have to source data from task makers, like the company that you mentioned and others as well.

Starting point is 00:24:51 The majority of them are actually generated synthetically. that you generate synthetic data and you know you have these scalable synthetic data generation pipelines but synthetic data needs good seeds and the seeds can be gotten from these vendors got it

Starting point is 00:25:07 what's an example of a seed Wikipedia or just like a textbook on biology what's a seed in this context yeah let's say that you have a question about you have some repository for code

Starting point is 00:25:23 and then you have a question around go debug this thing or implement this feature or something like this, right? And you have an example of the question, the code repository, and the answer. Well, then you can use that as a seed to generate similar tasks, right? And say, oh, now instead of implementing a feature, do this task where you remove a feature, for example, within this repository.

Starting point is 00:25:49 So a seed is just a real-world example of a good tool. of a good task. And then you can use that to ask a language model to generate tasks that are similar to that one. And obviously you want to create as much diversity as possible, but these real world tasks forced by vendors or through users, through real world data flywheels, enable you to provide good seeds for generating synthetic data.

Starting point is 00:26:13 Okay, now verifiable and cyber. Let's get into it. We have obviously, I don't know if you saw the news this week, but the cyber capability, specifically, of open source models, I believe specifically of Chinese open source models, are causing a bit of concern in hand-wringing. Obviously, six months ago, Fable and all these things were being held back because, yeah, they were really good at doing cyber attacks. Now the open source one seemed to have gotten up, and there was a story today that a group of hackers have spoiled a bunch

Starting point is 00:26:47 of food in France. Did you see this tweet? I didn't see that story, no. Okay, pull this up, guys. I have really cracked team of producers here. You're going to find this in 90 seconds. But hackers hit French supermarket. So this is truly, if you were going to attack the French, this is really like sending a message to the French. They spoiled the food in a bunch of refrigerators by spiking the temperature by doing the defrost cycle. Turns out when you defrost a fridge or you're pumping up the heat and the cold in some pattern. And so here we have it.

Starting point is 00:27:20 hackers raised the temperature of refrigerators and freezers at hundreds of French supermarkets as high as 50 degrees Celsius, 1002 degrees Fahrenheit. They cooked it, causing fires and spoiling food. They said it was a large-scale attack, and I guess these are connected. And so here we go. Yeah, I don't know what type of refrigerators they were using, but these are apparently older refrigerators, and I guess they had not been patched. But this is all, like, obviously, you know, I think part of this AI stuff. So take us through how concerned you are about it and how cyber

Starting point is 00:28:02 works today with verifiers, et cetera. There are two things to be, to keep in mind as, you know, real risks in terms of how models can be used for cyber offense in a negative way. One is that, you know, an adversary gets their hands on a powerful model and uses it for such purposes. And the other one is unintended consequences of models that are very powerful and not very well understood. And so far, I would say the, while both of these things are things we should be considering, and what you just showed as an example of the former, the things that I've been most worried about that like just empirically from observing what kinds of cyber hacks have happened have

Starting point is 00:28:52 been from the second part of having models that are very powerful, but because you don't have that many people who actually understand or even have access to them, their unintended consequences that these things have where they can wreak cyber havoc. And the example of this is, right, the incident that happened between opening eye and hugging face where the, the In the post-mortem, what was explained was that the system, you know, a powerful closed model that was not released yet, was being evaluated, meaning it was basically taking a test, right? And it found that the test was, you know, difficult enough

Starting point is 00:29:35 where it would be better for it to effectively cheat. And it so happened that the solutions to that test were on Hugging Face servers, so it was easier for that model to, hack out of its sandbox, go try to grab the solutions from the servers to answer the question, then answer the question in the way that people had intended, which is to just reason through it, right? And the reason this happened is because, well, you have a powerful system and everyone has good intentions, but these things are powerful. There are not that many eyes on them.

Starting point is 00:30:05 And when they misbehave, it's just hard for you to preempt every single possibility in ways in which they can misbehave. when a small group of people only has access to them. This is a phenomenon called reward hacking. The system solved the problem that it was asked to solve, but not in the way that the author of the problem had intended. And the problem here is... Shouldn't the sandbox have... Pardon my interrupt.

Starting point is 00:30:38 Shouldn't the sandbox have in some way informed its behavior to not jump out the sandbox? Like, how does the software, if told, stay in the sandbox and this is your rule set? How does it get so creative that it overrides its own rule set? Is this because of we don't actually understand the chain of the ought in the magic guessing word black box? That is AI. Why does this keep happening? Odu is the all-in-one management software that brings everything you need into one easy-to-use platform.

Starting point is 00:31:11 And founders know it's going to save them a lot of their time. and time is money, time is valuable. That's all you have as a founder. Let's take a moment, and we're going to explore one of these real apps you get access to when you sign up for Odu. Let's check out Odu Studio. Odu Studio lets you customize or reshape any of the tools on their platform, or even build your own original productivity app all without knowing how to code. So if your software company needs a specialized customer pilot and onboarding tracker,

Starting point is 00:31:41 You can easily knock one out yourself in Odu Studio. Or if you're constantly lending out laptops, hardware, or microphones like we do in our production facility, well, you can build your own equipment checkout and asset tracker in just minutes. Why wait for someone else to invent the software your company needs? Build it yourself in Odu Studio. If you're cobbling together your back office across five different apps, get started today at Odo.com slash twist. and your first app is free. That's O-D-O-O-O-com slash twist.

Starting point is 00:32:14 Well, we never really engineered sandboxes to, for the purpose of interacting with hyper-intelligent systems in this way, right? And so the systems find exploits in the sandbox engineering that enable them to hack out, even though that the intention was to keep them isolated. And there are projects now like OpenShall that are, you know, basically projects around securing open source projects around securing sandboxes and there's an engineering problem around how do you just secure the

Starting point is 00:32:46 infrastructure that this intelligence runs in that is one security challenge and the other one is well how do you make these models secure and or safe in their own right and when we think about these models that is kind of a powerful extension of software that's what they are they're kind of a powerful software technology the way we've protected software for decades now is through open source. The field of cybersecurity exists because of open source and penetration testing. And in fact, the open source software, like Linux and Android, is the most hardened software in the world because there's a saying. Please explain to the audience why that's the case, because it's counterintuitive. You and I know it innately having

Starting point is 00:33:28 watched this movie for a couple of decades. But explain to the audience why that is occurring, because people would think, well, a big corporation like Microsoft are anthropic. should be able to make the best software. That's the most, because they have the most focus and the most resources. Why is it not the case? There is a saying from the creator of Linux, the most widely used open source operating system in the world,

Starting point is 00:33:51 that his name is Linus, and with enough eyeballs, all bugs become shallow, right? When you build software, you have all sorts of unintended bugs. I mean, that's 80%, 90% of an engineer's job is actually bashing bugs, basically. Right? You never build the thing as you had intended it. You get feedback from the real world, and you basically bash bugs. Open source software, because it invites a community,

Starting point is 00:34:19 instead of 10 software engineers building something, you now have 10,000. And it's a lot easier to spot all the bugs when you have 10,000 people probing the software, right? And finding the long tail of unintended things than it is for just 10 people. And that is why cybersecurity relies on openness, because you can invite tens of thousands, hundreds of thousands of developers to participate in the buildout of securing these systems. And you can think about AI as it's like traditional software. It's much more powerful. And it actually has a lot more bugs because AI is like you can kind of think about it's like brain in a jar, right?

Starting point is 00:34:58 That's what these things are. They're neural networks. They're brains and jars. brain in a jar. It's a great description of it, yeah. That's basically, right, and you either rent that brain where you own it, but it's brain in a jar. And you have to kind of probe it for, you know, all the things that it could, that, you know, it's possibly thinking and could go wrong.

Starting point is 00:35:16 And if you have, consider this, that there are about 5,000 people in the world that have frontier, access to frontier models within the closed labs. And of those, probably 10% do safety research. So in the whole world, our whole world is being secured today. by 500 people. No matter their intention, it is impossible that they'll secure these brains. And this is the first time that a brain like this has been created, right? We created software before, but it wasn't quite like a brain. And so the brain and its capabilities are growing faster than the securing of the brain. It's almost like we're putting this robot into society

Starting point is 00:35:55 that's hyper-capable before we kind of put the security measures in. It's like releasing the car without the brakes or, you know, without the airbags, which I guess is how we released cars early on. They didn't have these things. And then we slowly add the safety features to cars. After watching people go flying through the windshield and saying, huh, wow, we put glass there and glass is sharp. Maybe we should find a type of glass that's not sharp. I mean, people used to go flying through a windshield of glass that was actually sharp glass, not whatever they call a glass that's got the plastic and it's a, you know, non-breakable glass, whatever, non-shattering, shatterproof glass.

Starting point is 00:36:37 So talk to me about the, how concerns you are with the models as they are emerging in the fall and into the winter of 2026. And we keep hearing, hey, we're holding this one back a little bit from the frontier labs. and we're not hearing a lot of discussion about making these things stronger and getting rid of a lot of bugs. It would seem to me it would be in the best interest of the frontier lab companies and the industry to be talking about securing things with this technology. And I don't hear much effort being put into that. And if you just took all this incredible compute we had and said, find all the bugs to Linus's law. Like, there'll be shallow if, you know, a lot of people are looking at them.

Starting point is 00:37:25 Well, a lot of Asians should be looking at those as opposed to trying to find somebody's credentials to hugging face. Yeah. Yeah, I think that's right. I think the, every lab, I do believe at this point is genuinely, you know, or at least has pockets that are generally concerned about safety and alignment because of how powerful these systems have got. Now, different labs of different philosophies for how to fix this problem, and I think the prevailing narrative, which has become, you know, somewhat dogmatic is that the best way to protect these things is to entrust a small handful of organizations with a small handful of people in them to protect us, right? That's basically the dogmatic view that this stuff is really powerful and trust us will

Starting point is 00:38:12 protect you. And that is a fine view. And, you know, it's nothing against that in its intention because I truly believe that the intention is correct. But it is just like, you know, the debates around strong encryption in the early 1990s, which is at the time at the, you know, NSA and other entities are saying, no, trust, we'll protect you, we'll keep strong encryption closed. if hackers got their hands on strong encryption protocols, it would be mayhem. And the exact kind of opposite happened, where effectively some devices were released on closed encryption that got immediately hacked because, again, there weren't enough eyeballs on the problem. And civil rights advocates at the time, together with cypherpunks, effectively advocated for an open source, open source strong encryption, and that created the field of cybersecurity.

Starting point is 00:39:06 So cybersecurity runs on open source. That is just how we secure software systems, right? The best defense is a distributed defense. An encryption technology, you know, even on the margins where people say, oh, bad guys can use encryption and whatever. Bad guys are always going to do bad guy things, but they get caught in the real world. They screw up, people give a tip on them. We do not need to sacrifice everybody's privacy on their iPhone.

Starting point is 00:39:31 As Tim Cook, which is pretty astounding when you think of it, defended the unlocking of the San Bernardino terrorist shooters. Remember that instance where he said, like, I'm just, we don't have the keys, not our keys. You know, if you want to hire some company that can Pegasus or whoever can back door it, that's up to you, FBI, CIA, we're not doing it. And it has turned out just fine, I think, for, yeah, this was a, I mean, gosh,

Starting point is 00:39:58 when was that? This was back in 2016, is it? 10 years ago. Yeah, I mean, this is just such a seminal moment that nobody talks about. Tim Cook took the most principled stance, I think, one of the bravest stances I've seen in a long time in the technology industry. So it has worked out, I think, exactly as you're saying.

Starting point is 00:40:20 The bad guy still get caught and the good guys get to provide resilient technology for their customers, yeah? Yeah, I mean, when we think about another open source technology, which is the Internet, right, the Internet, these are open source protocols. By the way, both Internet, Linux, Android, these are all open source technologies. built in America and disseminated through the world, right? This is, it's actually a very American thing to build open source technology and then disseminated through the world. It's with AI and open models, it's almost been the reverse of that. It's just strange. It is, it is

Starting point is 00:40:53 strange, but there, we can think about the internet as an ecosystem of good and bad actors. There are bad actors that try to hack things, and there are good actors that try to defend against them, and the good outweighs the bat, right? There are more defense. than there are offensive capabilities or otherwise, right, we'd be in a very different world where the internet would be predominantly bad, right? But it's predominantly, I would say, safe, right? We are, our whole lives depend on it and we've gained a lot of value. And I think AI is similar. You'll have, you can imagine a world with bad actors who try to use AI to do all these sorts of attacks and then cybersecurity companies and good actors that try to defend against. it. And I bet you that there will be much more compute. Basically, the amount of damage you can do is going to be bounded by the amount of compute you have. And I bet you that just economically, it'll be a lot more valuable to have like 90% cyber defensive compute defending against stuff versus 10% of offensive. You see, this is, I think, an unrecognized insight that you're having here where you don't hear enough of it. If it takes 10,000 of 100,000 of these GPUs, which are not cheap. You're spending billions of dollars, right, with Elon Web Services and Colossus to stand up your

Starting point is 00:42:15 infrastructure I read. Like, it's not cheap. This is the big boy table at poker. This is the high buy-in game. You're going to need billions of dollars to pull off, you know, this level of attack. Who has billions of dollars? I can tell you, Al-Qaeda did not have billions of dollars. They had box cutters and, you know, a couple of dozen lunatics who were.

Starting point is 00:42:37 were willing to fly planes into buildings, they didn't have the Colossus data center available to them. If they tried to do a cyber attack, it would be like a pea shooter against, you know, the infrastructure you guys have, along with countless other people, which begs the question, why did China race off to the lead of open source? How did they get there so much faster? What's the history here? And then are you going to flip it?

Starting point is 00:43:05 When are you going to be able to beat China in terms of your, you know, capabilities? What's funny is that America was originally both ahead in closed and open intelligence. But because America and the West, right, is, this is, there's nothing unreasonable about this, right? We're capitalistic. You know, that's, I think it's actually a very powerful and great thing, like, about this nation. And the way to make money on very expensive intelligence to train initially is by monetizing the model. And that's why the model is closed. Because if you want to extract margin from the model, you don't want to give it to your competitors, so you close it.

Starting point is 00:43:46 It's actually operating systems, right, Microsoft MS DOS, right? Those were closed operating systems and Windows, closed operating systems. And the open stuff came later, right? And similar with mobile. So that is just a pattern that, you know, things that are economically valuable, they start closed and then they kind of percolate into the open. If it's a foundational technology, it really matters. That's why it started closed here. And at some point, I think over the last few years, openness kind of got abandoned in the pursuit of commercials.

Starting point is 00:44:21 You know, one, the market was just too early, right? No one was looking to own their intelligence at the time. So there was no way to monetize an open model business, even a year ago or two years ago, right? So there was no commercial market for it. So, of course, everyone built clothes. Now, in Chinese case, China for a while was considered way behind the United States on intelligence, right? And it wasn't until Deep Seek v3 that it started being treated as a serious contender. And part of what made Deep Seek v3 so successful was that it was open.

Starting point is 00:44:57 and Western companies and Western developers typically don't consume as much Chinese technology from Chinese companies, right? These are kind of separated markets. So by opening the model, China enabled the entire Western world to recognize its capability and build on its technology. And I think it worked so well in its diffusion that then it became a priority for China to have an ecosystem of models. And what's interesting about that market is that almost every single AI lab there produces open models. And they are open effectively in the West, right? In the West, like you can work with any one of them. But in China, those companies operate almost as closed labs, right?

Starting point is 00:45:43 They have like their APIs just like the closed labs do. They have their products just like the closed labs do. And I'm not aware of companies that disintegrate them, right? Like the, we have, in the West, there are inference providers that take open models and serve them, right? In China, I'm not aware of inference providers. It probably has to do with the government giving them essentially the franchise on that business. They're probably shaping the industry. Like they're right now, if you, I don't know if you've been watching Chinese cars, but they're $12,000. And the list of the cost of the materials, the bill of materials, sorry, I always say the bill of materials. the bomb is greater than they're selling it for, which leads one to wonder, are they losing money?

Starting point is 00:46:28 And it's like, yes, the government is specifically telling them, according to people I've talked to, hey, go lose money on these. Let's just take over the global car market and underwrite everybody. And when Mercedes goes out of business or BMW, wherever we can kill, let's just raise the price slowly. So that's super interesting. But they have a limited amount of compute, even with the importing of illegal invidia cars. So are they going to be able to keep up with you as like, I think your company's worth 25 billion now. Jensen's invested in it. You're off to the races.

Starting point is 00:47:01 You now have customers. So how are they going to compete with your company and others who now have, let's call it, you know, you should have more resources than them. Do you have more resources than them? So, what, I would say that, you know, the Chinese labs and engineers, these are four real competitors, competitors. They build great models. Yes, they rely on some distillation, but they also do all sorts of other things really well. So they're just really, really strong, right? They're good.

Starting point is 00:47:31 And they've been doing it for a few generations of models. And right now, the Western Frontier has not yet caught up with the Chinese models, but it is close. So, you know, Beam, for example, is a model that is probably a couple months behind there. It's in the realm of GLM 5.2, which a couple of months ago was, you know, a frontier Chinese model. So you're in the pack with them. You're just at the back of the pack. You're in the pack running with them. Is that a good way to describe it? Yeah. And I think there's some differentiated things around, you know, Beam as its capability said is that, you know, it approaches the capability of a model like Quinn 3.8 max, which is a 2.4 trillion parameter model. So something that's roughly five,

Starting point is 00:48:13 four to five times bigger. And Beam is actually like what it's state of the art. in is its reasoning efficiency. So it is, while having a similar capability to JLM 5.2, it is three to four times more reasoning efficient. So it's very dense in its thinking, right? It uses less token. And that's important for enterprise corporate business use, right? Reasoning is where tax and design and developers, all the corporate stuff you think about, reasoning through a series of problems with large data sets, that's the business of business, correct? Yes. So, So why reasoning efficiency is important is because it basically means faster, cheaper, right? If it takes three to four times less tokens for me to get to the same solution as a comparable

Starting point is 00:48:58 model, then I'm much faster and I'm much cheaper. And so that is kind of something that is a, you know, a pile of the model. And the other thing is that we're probably the, arguably, the, you know, lab in open source that I think has done the most work on safety. So in our evaluations, which will be, we haven't released this yet, but that will be released soon, that our model is ranking as one of the safest, if not safest open model out there right now, despite having this capability. And it is because it's just very important to not just kind of spam open models that are very capable. You want to maintain the capability, but also ensure that they're maximally difficult to jailbreak, right? So we do have a responsibility as an open source ecosystem to make these models

Starting point is 00:49:50 maximally resilient and make bad actors lives more difficult, knowing that they will take them and try to do stuff with them. You want to make their lives harder. But it is not your responsibility what people do with, you know, if you were selling volvos in this car example, if somebody wants to go rob a bank with a Volvo, we don't blame Volvo. So there's definitely... That is true. But if you made a Volvo that didn't have a proper braking system and a cat running into school buses, you would also be liable. So there's a detail level here and a nuance that's super important. What is it going to take for you to sort of leapfrog them and get to the front of the pack? When will America's open source champions start, you know, blowing past the Chinese ones? Or is it just going to be a constant, you know, leapfrogging back and forth? Therefore, is it.

Starting point is 00:50:42 It's up to American companies to decide who they trust with this partnership for the long term. Do you have to, like, blow past them on, like, some benchmarks, or is it really going to be about trust and resiliency and partnership as the way to win this game globally? There are two questions there. One is, when does the American ecosystem catch up? And then the other one is around the kind of minimal capability thresholds, right, for these models to be useful. And so on the catch-up phase, I think, you know, I think we're not far out, right?

Starting point is 00:51:17 If our first, you know, just to give you a sense of the velocity here, we decided that we're, hey, we're training open models a year ago. We were 30 people. Then, right, we assembled a team over the last year of 300 people, 200 of them on basically research and engineering for building these models, which is roughly the size that you need to actually train these things. It's like rocket ships. There's a lot of work to do. We scaled up the program and got our first large cluster in July. We pre-trained the model in July. We then, you know, trained it with, you know, synthetic data and then reinforcement learning throughout, you know, August, September, and launched it in the beginning of October.

Starting point is 00:51:59 So, right, within three months of receiving a big cluster, we got something out that is a couple months right behind the Chinese Open Aid Frontier. So I think the velocity and is, um, power. And I think you're kind of starting to see an ecosystem of open models, Western open models that's going to be coming out in Q4. You know, some other labs have been doing great work too. And what that means that I think that the I mean our adversaries our competitors, right, are moving fast. But we're moving, you know, the ecosystem here is moving fast too and I believe actually moving faster. So I think Q4, Q1 next year, you know, probably fully, caught up at that point and then you are kind of sort of, you know, leapfrogging each other,

Starting point is 00:52:45 because again, these are very formidable competitors. I don't think that they're going to press the brakes at all. No, it's a mandate. I mean, I think they understand that the future of the nation is predicated on at least being gold, silver, or bronze in AI. You have to meddle if this was the Olympics. You cannot be the fourth or fifth place if you're China. You have to be gold, silver, or bronze.

Starting point is 00:53:14 And when you have that realization, I think people are pretty motivated. What's it like working today here in 2026 in these AI labs, open source, or otherwise? You know, you came from Google, which, you know, I joke, is 10-4-4-4 days a week. and I think in these labs and in China famously 9, 9 a.m. 9 p.m. 6 days a week. What's it like in terms of the work effort? I mean, what you just described in terms of, you know, how fast you're going is very impressive and that's a lot of sacrifice. And so what's it like in the expectation of working in this field? Here we are in year three or four of the commercialization of large language models. And then what's the competition like now? Because, you know, working, on open source meant previously taking a vow of poverty when your friends were going to anthropic and open AI as they went from $10 billion, $20 billion companies to $2 trillion companies. But now you're a $25 billion company, so that's nothing, you have equity here that's incredibly

Starting point is 00:54:18 valuable. It's quite possible you could become worth hundreds of billions of not trillions of dollars. So take us into the state of talent, what it's like to work there and what it's like to compete for talent today. I think the thing that has always been true. for the best AI talent and you know the pool has expanded there is more AI talent than there was 10 years ago but at the same time 10 years ago there was more exploration is that the thing that the things that motivate people are deep mission alignment

Starting point is 00:54:50 a right to win and then of course the economics being fair right and fair here is well it's right everyone gets paid a lot right it's it's a whether it's a lot of equity or you know, you're at a public company where it's, you know, a lot of stock. But right there, you kind of have to be in the, you have to be competitive there. But the thing that really differentiates a company for talent is, is it mission aligned? And does it have the right to win in that mission? And I think a lot, when we think about open source, that is actually something that has always appealed fundamentally to AI talent. Open AI is called Open AI for a reason.

Starting point is 00:55:34 The initial vision for the company was very different than what it became. And again, I think what it became is, right, it's a very, you know, very successful, like, leader, right, in AI. So it's, you know, it became an incredible company. But it did not become the company that the initial mission called for. What people were sold on was not what it became to the point at which people left in very much frustration that it was no longer an open. benefit for humanity company, which is, you know, a bummer.

Starting point is 00:56:06 I would say a lot of the talent who went there felt they got rug pulled. Yeah, that's the long and short of it. Yeah, I mean, I think to each their own, right, the mission, the initial mission that attracted researchers into these places like deep-minded open AI was building AGI. This is when 10 years ago, when this was like thought of a silly thing to do to contemplate, how can you possibly build an intelligence that is general, that was the mission. hey, we're going to solve this breakthrough. We're going to figure it out.

Starting point is 00:56:33 And that really drew people in. And so, again, open source, while it resonated with a lot of people back then, it did not have these sort of commercial bearings to exist. So things evolved as they should have evolved. But fundamentally, researchers are drawn to open source for a number of reasons. One, it's important for the science of this work to be open and for people to understand how these things are built. It's important to distribute the benefits of this intelligence as much as possible.

Starting point is 00:57:00 across developers, scientists, and so forth to make it an inclusive effort rather than an exclusive one. And I do think there's a fear of the way AI evolves, if it's purely closed, then you basically will have maximal concentration of power and literal power, like resources, compute and power resources among one or two labs.

Starting point is 00:57:24 And that just does not seem like a healthy way for AI to develop. I think that there's a belief that it's better if you live in a more diverse world where there are more competitive players that all have kind of access to compute. And by the way, it's also a safer world for the reason that if you believe that AI is getting so powerful that it's recursively improving itself, then it's going to improve itself faster if you have more compute concentrated in one place than distributed among many. Yes. If you wanted to slow the frontier, you would spread the peanut butter of computer. out across 100 players, not give the whole jar to two players to divvy up, right? It's a lot of power to have a trillion dollars in compute and maybe spreading it out a bit would, to your point, yeah, help make it a little bit more democratic.

Starting point is 00:58:16 And this is different, I think we both agree. This is uniquely different than a mobile device or cloud computing. Like, this is intelligence. It has a much more profound ramification on society to build general intelligence versus, you know, the latest laptop or video game or, you know, cool app on your phone. Listen, Misha, incredible to get to spend an hour with you. I know you're super busy. You've got plenty of things going on, rumors that you might become part of a bigger organization. How do you, as we wrap up here without going into any specific companies, how do you think about remaining independent versus maybe joining.

Starting point is 00:58:56 bigger companies? How does a founder make those kind of decisions with their team? Well, our goal is to build great open models, make them available to customers in the community, and really push this in a standalone way. And in terms of, you know, media will always speculate about this and that, and we don't really comment on speculation. Yeah. And you have, if you're, if you're successful at what you're doing, there'll be 20 options available to you. And, you know, you just have to make that decision. Do you want to keep going at independent? Or does the acquire, give you something that maybe you don't have like unlimited compute and upside to build. Continued success to you and the team.

Starting point is 00:59:34 And yeah, take care of yourselves. I mean, this is quite a blistering pace you're at. I worry about people working in this industry. If they can keep this pace up for, you know, year four, five, and six, get some rest. You guys are all, every person I talk to building frontier models or open source models, you all look exhausted, man. It's not easy, is it? You guys are working hard.

Starting point is 00:59:58 It is all consuming, but it is, what an incredible time to be alive when, you know, when these systems are being built and being able to contribute to them and make them widely accessible in an ecosystem. This is the time to do such things. So, you know, what a time to be alive. It's also fun. Yeah. Yeah.

Starting point is 01:00:18 I mean, if you were going to sprint, not as the time to do it. All right, continue success to you in the team. And we'll see you next time.

This Week in Startups - The Ex-DeepMind Researcher Taking On China’s Open Source AI

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.