This Week in Startups - SO MANY THINGS need to go right just so you can watch a TikTok! | E2215

Episode Date: November 26, 2025

Register here to join Founder University Japan’s kickoff! 👉🏻 https://luma.com/cm0x90mkToday’s show: On TWiST, Alex is chatting with Neubird CEO Gou Rao about the intricate complexity of IT Ops, and the vast amount of layers and components that have to be delivered very quickly to your phone in order to stream a simple video clip.Neubird has developed Hawkeye, an AI “engineer” that uses “context engineering” to isolate and troubleshoot problems, while still keeping costs down. Loading up a video to show your friends may SOUND straightforward, but it’s actually, as Alex notes, a miracle anything on the internet works at all.PLUS we’ve got Subject AI CEO Michael Vilardo stopping by to tell us how the app uses gamification to keep kids learning and engaged. NOTE TO FOUNDERS: Mike got invited on the show because he sends email updates to his “dream investors,” including Jason. It’s a solid best practice!Finally, we’re flashing back to a remarkably prescient 2019 interview with then-Scale AI chief Alexandr Wang, who tells us why he thinks the AI Doomsday scenarios — AND the elaborate AGI promises — are both way overblown.Timestamps:(3:19) Neubird asks: What if we took AI tools and applied them to IT ops?(5:17) Why does IT ops seem to get so much more complicated over time?(6:14) Gou walks us through all the various components that need to be delivered when you post a simple TikTok video(8:41) Why agentic systems (like Hawkeye) are the ideal tool for monitoring metrics, logs, traces and alerts(10:05) CLA - Get started with CLA's CPAs, consultants, and wealth advisors now at https://claconnect.com/tech(11:17) How is IT Ops different from Site Reliability Engineering (SRE)?(12:36) Is Hawkeye an AI Agent or an “Engineer”? How we decide what to call our “co-pilot.”(16:07) How “context engineering” helps Hawkeye isolate problems, and keeps inference costs down(17:27) Is Hawkeye plug and play? Or does it need to be trained on a specific system?(18:59) How Neubird measures Hawkeye’s overall effectiveness(19:59) Perspective AI: Real insights, straight from your customers, and your first two months are on us. Just go to http://getperspective.ai/twist(21:47) Hawkeye can diagnose and make recommendations, or go in and fix the problem itself, if you like!(24:27) What Gou thinks about the pace of AI innovation: Has it slowed? Are we in a lull?(25:41) LLMs already have a lot of knowledge, but how do we enhance their EQ?(28:58) TIP FOR FOUNDERS: Send updates to your “dream” investors! That’s how Mike got on the show!(30:20) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at https://Uber.com/twist⁠(31:21) The importance of branding… Why Mike pivoted to Subject.ai(33:43) How Subject helps bring teachers and students together for more 1-on-1 interaction(35:48) Mike walks us through how Subject works, and how gamification keeps kids engaged(39:56) Is anything lost when an AI grades a student’s test, rather than a human?(42:09) Subject can serve as a “teacher of record” in classrooms(46:28) Why Mike and Subject AI are back in the office(47:40) TWIST FLASHBACK with Alexandr Wang of Scale AI from November 2019(53:10) Alexandr explains AI to Jason (back when it was still called “Machine Learning”)(1:01:08) How AI will help people focus on “higher-value work”(1:04:48) Why Alexandr thinks apocalyptic AI scenarios are unrealistic(1:13:28) Are the big promises around AGI overblown?*Thank you to our partners:(10:05) CLA - Get started with CLA's CPAs, consultants, and wealth advisors now at https://claconnect.com/tech(19:59) Perspective AI: Real insights, straight from your customers, and your first two months are on us. Just go to http://getperspective.ai/twist(30:20) Uber AI Solutions - Your trusted partner to get AI to work in the real world. Book a demo with them TODAY at https://Uber.com/twist

You know, what engineers do is troubleshoot, fix problems. They create problems, and then they fix the problems they created. There's a lot of interest now in how to use AI, especially Gen AI, and how that can ease the burden of IT operations. The moment you get on a phone and you're looking at a TikTok video, I mean the number of layers that are involved to deliver that, notwithstanding the physical infrastructure to deliver the audio and video content, but the infrastructure software and physical infrastructure that it takes to store the media,

It's a very complex environment and people want more of this faster. Just for fun, can you walk me through some of the software that goes into making a TikTok appear on my phone, for example? The videos have to be stored somewhere. So there's some quality control. So there are software associated with that. And then when the video gets edited and uploaded and stored, you want to make sure that there's no data loss, no integrity compromise on the video. And then the distribution aspect of it. And there's edge caching.

You don't want somebody looking at the video to hit the back. and server each time, and so there's what's called content distribution networks involved. This week in startups is brought to you by Uber. Bad data equals bad AI. Your AI is only as good as the data it learns from. Uber, that's right. Uber AI Solutions now works with enterprises around the world to source, label, evaluate, and scale real world high quality data for every industry everywhere so that you can focus on building the next big thing. High quality data equals smarter and faster AI. Uber.com slash twist.

Perspective AI. Surveys, they never capture what customers are really thinking. That's why we use prospective AI, real insights straight from your customers. And the first two months are on us. Just go to get perspective. AI slash twist. And CLA. Innovation takes balance.

Our CPAs, consultants, and wealth advisors can help you get from startup to where you want to end up. Get started now at CLA connect.com slash tech. Hey, everybody. Welcome back to Twist. And if you are in the United States, tomorrow is Thanksgiving. So I presume you are listening to us on the plane from the car. Wherever you are, we're glad you're spending the day before the great holiday with us. Now, on the show today, Newbird, a Twist 500 company that is building what it calls

and Nagintic A-I-S-R-E. What does that mean? Well, it means they're applying modern AI technology to helping companies ensure that their cloud set-up actually works. As the cloud becomes more complicated, there's more wires and tubes to connect, and this company wants to make sure

that you never get them crossed. Then, Jason talks to Subject AI. It is a really cool company in the ed tech space, like some others, it's helping students have more personalized educational tools and also helping teachers get rid of the busy work so that way they can spend more time one-on-one with the kids. Then to close off, we're doing a little bit of a flashback all the way

back to November of 2019 when Jason sat down with Alexander Wang, then the CEO of Scale AI, talking about the AI market and more. It is a fantastic time capsule and look into the prior era of AI. Let's have some fun. I am paying a lot of attention to applied AI. Sure, it's great to keep up with the newest models, but if you really want to understand the health of the AI market, I think you need to go and look where the rubber actually meets the road. So that's why we've talked to startups like Harvey, which is bringing AI into the legal realm. And today, we're talking to another startup working in applying AI, albeit in a very different realm. New Bird AI, that's an EU bird, ask the question, what if we took modern AI tools and applied them to IT ops?

And if you want to know why that matters, well, just think back to the recent AWS outage. Well, maybe these things are very important to the running of the economy. Anyways, let's learn some more. Please join me and welcoming to the show. It's Gu, Rao, the CEO and co-founder of Newbird. Hey, how you doing? Doing well, Alex. Thanks for having me.

Oh, my absolute pleasure. I have to say that when it comes to hands-on experience in IT ops, I have precisely in exactly zero. And I think a lot of folks out there, founders and all, probably are in the same boat. So maybe the best place to start, Goo, is just why do we need an AI, IT operations engineer? Allow me to give you just a little bit of background. I'm in engineer.

You know, what engineers do is troubleshoot, fix problems. They create problems, and then they fix the problems they created. And creating and fixing the problems, for some way, shape or form typically happens at two in the morning. And you don't want to wake up at two in the morning, waking up to an IT outage saying that the website's not working or the payment processing system is down and you can't take transaction. So this is IT computers or something that have moved us into the new frontier over the past year, or 20 years. And, it's going to be more and more essential and critical to all of our lives. And so getting ahead of this problem and having infrastructure in place that can keep the operations running smoothly is of the utmost important. So there's a lot of interest now in how to use AI, especially Gen AI, and how that can ease the burden of IT operations. Before we get into how you guys are doing that, though, is it correct that the state of IT operations is one of increasing ramping complexity?

because it feels like with the advent of like multi-cloud setups and so forth, there's just a lot more pipes to keep connected. And so my impression, as a layman in this context, is that this is probably a problem that's getting harder and harder as time goes along. That's exactly what's happening. As things mature and there's more demand for new and faster software, faster products, better features, complexity increases.

And people are moving at such a high velocity now engineers in creating product that there are many layers of technology, many components that interact with each other to deliver the features that you like. The moment you get on a phone and you're looking at a TikTok video, I mean the number of layers that are involved to deliver that, notwithstanding the physical infrastructure to deliver the audio and video content, but the infrastructure software and physical infrastructure that it takes to store the media, it's a very complex environment.

And people want more of this faster. Just for fun, can you walk me through some of the software that goes into making a TikTok appear on my phone, for example? So I think that's an interesting thought experiment to kind of talk through. First of all, there's the content creation and distribution aspect of things. The videos have to be stored somewhere. A lot of times this content is being curated by people that are brand, they have a tight control on their brand. They're not just putting any video up there.

So there's some quality control. So there's software associated with that. And then when the video gets edited and uploaded and stored, you want to make sure that there's no data loss, no integrity compromise on the video. And then the distribution aspect of it. This now goes to people are watching this on their Verizon or T-Mobile and audio and video format doesn't need to get delivered in. And there's edge caching. You don't want somebody looking at the video to hit the backend server each time. And so there's what's called content distribution networks involved.

You guys are all familiar with the Cloudflare outage that happened a couple of months ago. And so these things, there's many layers involved. A video blogger goes out there and says, I put this content out there, but my customers can't reach it. Is the problem with the cloud distribution network? Is the problem with you running out of space on your AWS account? Like, how do you even go troubleshooting these things? It's a complex problem. It takes many engineers to get involved and troubleshoot these things,

whether you're an engineer at TikTok or you're supporting the blogger's website. Is it more surprising that the internet, as we know, it works today? Or is it more surprising when it breaks? Because I feel like given what you just described, I'm kind of shocked that everything stays online, most of the time go. It takes a lot of hard work to keep things running. It's great that the infrastructure work. Look, I'm surprised that any part of infrastructure that works. Like when my train comes in on time, I'm really happy with it. Kudos to the people that are managing that. And so

it's taken many years for us to perfect how to get trains running on time. And if you were even look at the aircraft industry, right, or the air industry, that's a huge logistics issue. And so people have perfected that. And now we're getting into moving away from manually keeping things online to using AI to keep these kind of systems working. But the complexity that we described generates so much information logs, essentially, that I presume that AI is a good tool to be kind of a first set of eyes on all the information that's coming in because me, the individual human, even if I replicate myself and sit myself at 12 different desks, probably can't look at all the logs that are coming in from even a fraction

of TikTok's overall digital footprint to stick with that one example. 100%. That's where agentic systems come in. Let me define what an agentic system is. An agentic system is something that leverages gen AI to do what I think you called it applied AI, which is a really good term. So an agentic system takes generative AI and applies it to a very specific task domain, a problem domain. And in this case, we're talking about IT ops. And so to your point, in IT ops, what an engineer has to go through when they're debugging in issues, they're looking at a complex array of data sources. They're looking at logs. They're looking at metrics.

They're looking at traces. they're looking at alerts, and each one of these have their own data structure complexities, and there's a lot of this data. Data has always been the Achilles Heel for an enterprise. So, yes, while people can do this, it is tedious, takes time and effort for people to go through logs alerts traces. AI can do this very quickly. All we do here at Newbert AI is exactly that.

We leverage large language models, extract the reasoning capabilities, and interface them with the complex array of metrics, logs, alerts, traces so that they can solve an IT issue much faster than humans can. And this product is called Hawkeye. You guys call it an AI ITOps engineer. I think you've also described it as an SRE in some capacity. Can you just explain to me the difference between IT ops and site reliability engineers why you, I think in one release said one and one release the other.

This might be a small point, but I'm just intellectually curious. One of the themes we talk about over and over again on this week in startups is making sure you do your chores. I'm no expert on these things. I have some experience. Stephen Estes from CLA is an expert. Let's talk about being cash efficient. Tell us about efficiency and what you see in the top tier startups in your practice. We're seeing kind of an interesting trend out there where companies aren't needing to raise quite as much as they had in the past. You really have to be careful as a founder to only take on as much money as you really need. You've got to do the forecast and you've got to do the modeling and you've got to dialed in and get it.

it right. Otherwise, you're going to end up either not raising enough capital to get to where you're going and you're going to have to go get venture debt or go back, have an extended to the round, or you're going to give up too much of the company because you just didn't recognize how much money actually needed. Yeah, very important to get this stuff right, folks. And that's really a bummer when startups don't do things in a button up. I always have a great partner. A good partner to have on this adventure. While things change, my friend Stephen over at CLA. Visit CLA connect.com slash tech. Don't forget to mention that your boy, J-Calcension.

That's clyconnect.com slash tech. Start today. Site reliability engineering was a coin turned by Google for very large enterprises where there are specific engineers focused on maintaining site operations. Now, depending on the size of the company you talk to, there are dedicated SREs that are completely focused on making sure that the IT operations are running up smooth. These are guys that respond to things like Pager Duty Alerts,

like it goes down, they get an alert. they have to go in and resolve the issue. Now, as you get into more mid-market, there's a blend. A lot of times engineers that are working on product or in DevOps share the role of maintaining their site operations. So they're doing double duty, so to speak, these products, agentic systems address both use cases. So to talk about agentic systems in the context of site reliability, engineering appeals to very large enterprises, but at the same time, we want to make sure that the same technology is helping mid-market as well. So essentially, I should think about SREs as very, very specialized IT-OPS engineers, normally the largest enterprises. But if you're going to sell

to the mid-market and up, you're going to want to speak to both the IT-Ops and SRE community. 100%. A lot of times, you know, the engineering team that's writing code, they're on call duty. Like if the site goes down, this engineer, you have your project to do. You're working on your coding, but you're also on-call support if something happens. And so you want this product to be able to address the needs for both. You guys describe Hawkeye as an engineer versus an agent. Is an engineer in this context kind of like three agents in one trench coat? The terms and how these things resonate with people that's still evolving. Everybody has a different expectation on what the term is called, what it does. Is it a co-pilot? Is it an agent? Is it something autonomously? Am I interacting with

it? These are all things for people to figure out how they're going to live with agentic systems. We call it an engineer because Hawkeye initially when it's deployed, it is autonomously and asynchronously figuring out what's going wrong with your IT ops and then figuring out what the issues are and creating remediation solutions. If somebody wants to interact with it and say, look, there's nothing going wrong now, but what if I were to do this? Can you go figure this out? People can do that too? Do you call that an engineer? Do you call it an agent? Engineer is a specific term for a general category that I would say is distinct from agents.

I think co-pilots, if you go back to the early days of the post-chat GPT era, when Microsoft was talking about cop pilots, they're a little assistance. They help you. I think agents, to me, feel more like single-shot tools that go out and do a thing like, I'm going to send my agent to Amazon to buy myself candles. The reason why I kind of like that you guys are not calling this an agent is that it does feel to be like a discrete set of systems that I can work with, but also work for me, even when I'm not directly tasking them with a, please go do A on site B, right?

So it does feel different to me in a positive way. You hit the nail on the head, Alex. And look, this is a little fluid, but that's the feedback we're getting to. You're right. A co-pilot is like you have to distinctly carve out a body of work for it to go in and do, or you're asking it to cross-check the work that you've done, which is kind of not what's happening here. What we're really trying to do is streamline yours as an enterprise's IT operations

where they're getting far fewer alerts, or like 80% of the alerts are being handled by Hawkeye and the U.S. is done, it's automatically resolved. I'll give you a simplest use case if somebody's wondering, like a very large company here in the Bay Area that operates in the cloud. They run in AWS. Obviously, cost operations is a big issue for them. All they want Hawkeye to do is see if their today's spend is projected to be greater than 5% of yesterday's baseline. And if so, why? You mean greater than 5% differential? Differential from yesterday's baseline. That's it. simplest of use cases, you would think like, hmm, that sounds easy. But then you realize what

engineers have to do. They have to log in, look at every instance that's launched, which instance change size, or maybe it's not even instances. Maybe somebody's consuming more storage capacity, or maybe somebody's doing a lot of egress traffic and downloading a lot of five. Like, it is not that easy. And most importantly, it's kind of boring at the end of the day when you find the issue. So you just kind of let these agents do this on their own. seems to me expensive because when I think about using AI to do a lot of stuff, I just begin to think about inference costs. And you guys talk about having integrations with Elasticsearch, Prometheus, IBM, PagerDuty, MongoDB, Red, Snowflake, everybody, which means that if I'm going

to send out Hawkeye, and I'm a big cloud user who has a lot of different things to keep an eye on, it feels like that Hawkeye would need to ingest so much information and process that it would be relatively heavy or expensive to run, go? Am I over-indexing on gross margin fears here? Or is it actually as expensive as I think. You are dialing in on one of the core problems that we need to solve for. Anybody can go in and download a whole bunch of logs, paste it into chat GPT, and say what's wrong over here. What you've actually done is you've isolated the problem down to a certain area and you're asking for a solution. I'll answer your question. The term for that is called context engineering. Quite honestly, more context, while these large language models have very large context windows

in garbage out. To answer your question on cost, but doctor were to charge you by minute, okay, just assume, and you go in unprepared and you say, I don't know what my problem is, it could be in my head, it could be in my back, it could be in my legs, well, you're paying a lot of money. So what you will do then is before you go to the doctor, you're going to say, I'm going to ask myself, what is my biggest issue, isolate it, go to the doctor and say, I have this problem in my knee right here when I touch it like this. And you'll get a very good answer from the doctor. That's called context engineering. That's all we do at Newbert. We specialize in context engineering because of two reasons.

We don't want the customer to incur very large inference costs. More importantly, we don't want the LLMs to come up with garbage answers or what's called hallucinations. When you give a marketing document to a chat GPT and say, I did this campaign last week, next week I'm going to reinvent, create a new campaign, it'll do a great job. But you can't do that with IT systems because you want Hawkeye to give you exactly what's going wrong with your website and why it's failing. not a what if, not a possibility, but exact.

Does the system have to be trained on a specific company's setup, or is it kind of something that you can plug into any cloud or IT environment and it will automatically know how to look around and observe? The latter. It's plug and play. And the reason for this is we believe these foundational models have seen so many different IT scenarios that the specific ailment you're facing is not something that hasn't ever happened before. We're not solving for that minutia of possibilities. So then all we have to do is take all this

knowledge that exists in the LLMs and apply it to your context, your data. That's what we do here at Newbert. We are context engineers. We pick the right alerts, logs, metrics, traces so that the LLMs can diagnose the problem using reasoning. And we apply a different kind of reasoning methodologies. We have these models argue with each other, cross-check each other, consult knowledge basis to see if the reason that it's coming up with is in the realm of something that's happened before. It's a complex system to build agentic systems, but it all relies on good context engineering. I've always felt that once we could get AI models to kind of fight amongst themselves as discrete entities, we would have some really positive friction that would yield better end results.

And I think that's one reason why I'm so bullish on building up more compute capacity, because I think it would be great to be able to be wasteful with inference, because then I think we can do quite a lot. But that's down the road. You guys launched Hawkeye, general availability like a year ago? Yep, a year ago. Yeah. So we're sitting here 12 months later, Gu, how's it going? Pretty good.

Look, we intentionally launched this with limited availability so we could learn with key design partners. We picked enterprises of all shapes and sizes, some very large logistic companies, some very large financial institutions, pharmaceutical companies, manufacturing two all the way down to low end of the mid-market. Why? Because we want to see if the product resonates well with the whole spectrum. What type of problems do you have? At the very large enterprise, a lot of noisy alerts. At the very low end, they want even more surgical because they're more tied into DevOps. What part of my code change is actually causing this issue? And so we've learned a lot. We've had a lot of good success. What do I mean by that? We're achieving close to 90% plus reduction in the meantime to incident response and resolution from the RCA's that we're creating.

Meantime of incident to respond. So that means that if it used to take me 10 hours to get from problem to solution, it now takes me one. You can't make your product better without listening to your customers. But how are you supposed to actually figure out what your customers want? You can send out a survey, of course, but people just want to give the right answer in that case. They're not really giving you their honest opinion. Well, at this weekend startups, we actually found the perfect answer.

It's called Perspective AI. Their expertly trained AI conducts one-on-one interviews with your users based on your prompts and questions. Just tell their system whatever you want feedback on in simple language. And they take care of the rest. We've learned so much about our audience since teaming up with Perspective AI. Reginald, for example, is a founder who listens and wants more quick-hit, faster-paced content. And an anonymous guy from Canada gave us tons of helpful feedback about some audio-sinking issues we've been having and we fixed them. This is the kind of feedback that's invaluable when you're,

You're a founder who is product obsessed like you should be. Sign up today and get started in just a few minutes at getperspective. comaI slash twist. And you'll get your first two months free. That's getperspective. com slash twist. Exactly. And by the way, that's not uncommon to.

And it's not just time. It takes an army of people. Look, you can imagine that any one engineer is an expert in networking and Kubernetes and storage. It's not possible. No. expertise in different areas. And thankfully, these LLMs sort of have cross domain expertise. They have a lot of knowledge, but they need to be told what to do. And that's where the context engineering

comes from. In the hardest of problems, we've been told that Hawkeye is able to save people a lot of time and at minimum tell them all the things that it did look at, which are something that the on-call engineer shouldn't look at and dial into where the problem could be. Even if Hawkeye doesn't have access to all the information, sometimes it doesn't. I'll say I've taken care of all of this, this area, I don't have visibility, focus on this. Does Hawkeye recommend methods of resolving the issue or can't actually go in there and make the changes or updates, corrections that I need to resolve it autonomously? Both. So it depends on the customer's comfort level. Typically, this has been the case when people deploy Hawkeye, they're looking at it

from a consultation perspective. Tell me what the remediation is. Give me a root cause. They will create the remediation report, code change or corrective actions, but people may not give it right access to take the action and it can submit a PR and somebody would have to hit a proof. That's normally the case. In some areas where feature flags and things like that are involved, because that's low-hanging fruit, people will just, OK, go take the feature flag, enable or disable it. But it sounds like maybe a good way to track how smart our overall AI intelligence is becoming as, as more and more Newberg customers allow Hawkeye to make those changes, it'll imply that there's increasing base model intelligence and also increasingly sophisticated applied AI techniques that you guys decide to make this entire

thing feel more like magic and less like integrated systems. 100% Alex. And the other thing that I'll say here for your audience, and I'm not talking specifically about Newberg, I would imagine that this is in all facets because I think agentic systems are going to be a thing in the enterprise, not just in IT, but in all facets, marketing, sales and people should treat agentic systems as not, you know, off-the-shelf software where you have a known outcome. You should treat it like it has a persona because these things are built out of human knowledge. So what do I mean by this? You have a chance to interact with it. So it's not a binary answer. You're not evaluating an agentic system for saying it's good or bad. There's a range in

between. So a lot of times these agentic systems can come back with an answer. You have time to interact with it, give it feedback and say, go do this. I like what you did, but you felt short here, go make these changes. It is like another employee. You asked me how have things been going. I've had evaluations come in where an SRE manager came to me and said, your product is doing really good. Hawkeye came up with an answer, but my SRE disagreed with it. Okay. Turns out both were correct. Hawkeye found an issue, which was correct. The SRE found an issue. And the Sari had never thought of that. So it offers you different insights into your environment. It is bringing diversity. It's bringing a different chain of thought because it's been trained from so many different scenarios.

And, you know, the manager was pleasantly really happy with that. And ultimately, solving both issues, improve the site ops multiple fold. I think it's funny how we go from, you know, deterministic computing systems to more probabilistic. We do have to do more talking to it. Like, great, try again. I spend a lot of time with my child GPT instance being like, excellent. Now, please do it without being lazy this time.

Thank you. Very, very fundamentally. are we still seeing the pace of AI model improvement that you were hoping to see? Because it does feel a little bit like from where I said that we are seeing improvements in Chinese open source models. We're seeing some improvements in American and European closed source models. But it doesn't feel as fast good as it used to it. And I'm curious if I'm just wrong about that or if you guys are also kind of noticing a bit

of a no, no, at some point, everything starts to plateau. And so I think you're acknowledging the fact that the, rate of new features is starting to reach limits. And I think this is expected in any new system, you're evolving so fast and you're putting out features so quickly and you're training these models. And when you have billions of parameters that these models have, it takes time to train them. But then the number of parameters are limited. And at some point also, having more parameters may not even make sense. I mean, you may just start getting weird answers. So in that sense, the knowledge in the models is reaching at this point.

some sort of convergence or, and it'll keep increasing, but the rate of increase, I think it's expected to not be at that same angle of trajectory that it was. Well, that's too bad. I was really hoping it was going to keep going straight up for a while there, because I could use some more free intelligence to kind of layer on to myself, you know? I would love to be smarter. But see, the intelligence is going to come from external systems is what I'm saying. I think there's knowledge in these models to do a lot of good work. Now it's on external tools and external context. We have these things called MCP. Model context. protocol. It's an anthropic project, yeah. The core of the brain, if it's steady, then augmenting

its intelligence or sort of call it, if it has enough IQ, let's get it EQ, which is the external knowledge that it would need to do awesome tasks. And that's where I think the next two or three years in AI, at least as it comes to, as you put it, applied AI is going to be around building the context and the support system around this really great set of brains that we have. One last question, before I let you go. We talked about there being a problem, Hawkeye notices it, figures out what's going wrong, and maybe either suggests or execute some remediation. Are you guys going to eventually go in front of the issues and begin to do like proactive scanning and checking to see what might break in someone's IT infrastructure? Because I feel like that

would be a natural next step, but I may again not be not be correct. We actually do that. When somebody interfaces with Hawkeye, there's three phases to their journey. The first phase actually is retrospective, the opposite of what you said, hey, I want to try Hawkeye. Last week, I had an issue. I did what every person on the planet does. I rebooted my cluster because I don't know what the issue is. I want to get out of it.

I got out of the issue by rebooting. What was the actual issue? So retrospective analysis. And then Hawkeye will say, I'm glad you got out of the issue, but this is what you should have done. Number two, real-time analysis. How is my environment running? I have an alarm.

What's the issue? And that's the primary use case. Big use case for Hawkeye is what you said, which is trends. What could happen? If things are progressing the way it's progressing today, what will happen next week? A very good use case for that.

A customer will say, I'm planning to roll out new software. Here are my changes. Is my environment ready to absorb these changes? Hawkeye will tell you, hell no, it's not. You got a lot of work to do. Yeah, it could be that,

look, you have this much reserved memory capacity, but your pod spec is requesting more capacity than you have allocated. Your pods will fail to start. Sounds like a reasonable answer to me. Sounds like a very reasonable answer to me. It's newbird. Dot AI, if you want to go check it out,

N-EUBird.a.I. And do you promise to tell me when you're closing your next major round of capital before we left. So I'll leave that to you. We are very well funded. Fortunate to have great investors. We're funded by Mayfield, Microsoft.

We have enough money in the bank that we don't need to. But then, you know, we're always talking to investors. And there are strategic relationships that we're forging. Without disclosing anything, I think there will be some good news down the line soon. All right, well, we'll see you on January 15th, give or takes. That's what that sounds like to me. Go, an absolute treat.

We'll talk to you soon. Thank you so much, Alex. All right, next up on the show, we have Mike Vlardo from Subject AI. Jason, this is an AI education company, started off by making courses for online consumption, but has since pivoted into more tools for the classroom. We've talked to other companies in the space like Magic School, but this one is blowing up. So let's talk to Mike. Mike, welcome to the show.

Hey, thank you so much for having me. Longtime listener, first time caller. I'm excited to be here today. Great note for startup. founders, Michael, for some really intelligent reason, decided to put me on his updates to investors. This is a really great technique for founders. We train people to do this at the launch accelerator, which is you make two lists. Your investors, who you want to keep updated, so that's your investor update, and then your dream investors, these can be the same email or slightly different.

With your existing investors, you might say, Alex, hey, here's where we're going, and here's some challenges. This is the problem. Here's the legal issue we're dealing with. Okay, right, we're getting sued by a former employee. We got a patent dispute. Okay, great. That's for the internal, like, 20 investors, right? Then there's the prospective investors. Now, this is just where you shine. Right. Hey, this is the third quarter in a row that we've had 20% on average growth. We are now on pace to hit 5 million in AR, 10 million in AR by this year. And we have cash in the bank, and we've just reached infinite runway because we're profitable. And then somebody like me gets this email.

Now, I'm BCC on it. And I'm like, wait a second. I've invested in 600 companies. Did I invest in Michael's company? Oh, no. Then I do a search. And there's like an email from Michael pitching me that I never got back to from four years ago or a DM. But anyway, well done, Michael.

You've been doing this technique for some time or is this the first time you did it? With AI, like everything else, the old saying is so true. Garbage in, garbage out. Without the right training data, you're not going to get great results. But my pals, that Uber, they're now working with enterprise. all over the world to source, label, evaluate, and scale high quality real data for your startup. Uber is one of my favorite companies in the planet because they demonstrated their ability to scale and to build great products. First in the ride chair market, then they figured out Uber Eats,

and then they moved on to autonomous driving. And all of this was only possible because of a deep understanding of how to collect, label, understand, and analyze data. Now with Uber AI solutions, you can put the team and tools that turned Uber into one of the world's best companies to work for you, partnering with top talent and experts from Uber's global network. What better partner to help you scale your business than the company that organizes over 36 million rides every single day? Book a demo right now by going to uber.com slash twist. That's ub-e-r.com slash twist. I'm doing it a little bit more often, but I do have to give you a shout-out.

I believe we were mentioned on a previous twist episode when we rebranded from a meal learning to subject and got subject.com. And you mentioned it as startup of the week and seven figure domain. Now, we didn't pay that much, but I do believe on the market, it could be worth that. So I always appreciated the way you gave us the nod then. And that's when I actually reached out via Twitter or X and we're really excited at. Finally, we're here together. How much did you pay?

How much? You can't say not seven. Undisclosed amount, but, you know, low six figures. So we're really excited about that. Three or four hundred grand for that is a pretty good deal. I have to begin right now. Less than that, I can confirm. So less than that. And we also have subject.coma now as well. So we really take a lot of pride in our branding and name recognition. And, you know, we want to be synonymous with elite premium education for any age and any part of the world. Hey, so why don't you show us what you're building. You can share your screen if you like and just walk us through the website or the product.

I don't know if we gave you a heads up to do that, but it might be nice to just show the product a little bit and what's working. Because I know you started pre-AI and then you wound up. up here with building AI tools, right? So, yeah, tell us what you're working on. I could quickly voiceover and really give you up to speed. So we were really the Netflix of education subject. I started this during my MBA experience. It was 2019.

I was at UCLA. It brought me to Los Angeles. 2020, I got thrown into Zoom University, the worst experience of my life. 70 grand a year to sit on a Zoom lecture like this. Everyone, you know, have their screen blacked out, slacking each other, the answers. And, you know, in Los Angeles, it was pretty extreme lockdown. Couldn't even eat outside of the restaurant for some period.

So what did you do after six, seven hours of Zoom? while you watch Tiger King, Last Dance, all these Netflix shows, that was the saving grace and got us through that time period. So my co-founder and I really said, hey, how can we make education feel like Netflix? How can we help students be engaged? I'm a millennial. Imagine a Gen Z learner, or now Gen Alpha in the middle of North Dakota and Nebraska. They might not have nearly the sports that UCLA was providing me. And so that was our original passion, premium cinematic quality.

I'm in our office in Los Angeles. We have four production studios here, 10,000 square feet where we do all our filming. But then since the launch of AI, we've really graduated past that and really providing much more adept, personalized tools for students and teachers, save them thousands of hours a year and allow all of them to feel like they're learning from their favorite teacher every single day in the classroom. So in the same way, Waymo and Uber and Neuro are trying to make like the perfect driver. You're trying to make the perfect teacher with AI, yeah? 100%. I'm really here to amplify the heroes of society, which are teachers and coaches. You know, I had a lot of great teachers and coaches growing up. And our goal is to give teachers the time to spend with their students in person,

making magical moments happen. And we're seeing that firsthand now with Subject AI. They're able to service three times the amount of students on a daily basis by using our product. And that allows for them to have more one-on-one interactions like this, while Subject AI can be the core experience in the classroom. This used to be called adaptive learning in the industry. It was like really hard to code, et cetera. But it's gotten easier, yeah, to do adaptive learning?

Yeah, adaptive learning or commonly referred to also as, flip classroom where you walk into a classroom and every student is engaging with subject videography, personalized AI tooling. And then the teacher is doing small group or one-on-one instruction with students who are behind or way ahead, making it much more impactful. And actually on the Andreessen Horowitz podcast, Mark and Ben have a really great episode on education. And they talk about this is very commonly known in the industry. The highest impact on a student's journey is how much one-on-one tutoring time they get. The problem is most people can't afford one-on-one tutoring. But when you work within the public school system and you allow subject to be that,

that core methodology in the classroom, now every student has a chance of getting one-on-one time with every one of their teachers. So flip classroom is the technical term. You learn the repetitive stuff on your laptop. And then when you're stuck or you're excelling, then the teacher comes and says, oh, yeah, hey, you're way far ahead. Here's some additional coursework for you to go on. Or if you're behind, let's get, let's break you through whatever the blocker is. Yeah. Oh, 100%. I mean, think about it from a first principal standpoint. Why should a teacher be doing the same lecture three times a day year over year. And how much variance is there? You know, depending on your previous day, anything going on in your personal life, the teacher may be for worrying at their

best or worse on that given day. How can we give the teacher the time to focus on that one-on-one instruction where they love it? They get so much energy from talking to students in person and the mundane as well as the grading and feedback is all taken care of by AI. And there's no excuse in this world of AI now to not be able to provide these teachers much more supports so they can have a much better work experience as well. Can you just show us? Because I got, yeah, I want to see it. You can see it's very much like a Spotify, Netflix-esque vibe when you go through the LMS here. We really wanted to always feel like premium consumer technology or social media. Our goal is to bring the best trends from those industries in education.

And, you know, we did the shout out earlier about you being an Uber investor, me being an Uber ex-Uber founder. You know, that's what Travis and Emile, who are huge, huge inspiration for me. Amiel is actually one of our investors. They brought the best people to work on transportation across all industries. Our goal is to do that in education and we're super excited about that. You could also turn into light mode with a lot of the teachers like when they put on a projection screen. But we'll go into dark mode here. So we're super excited about the product right now.

So you could quickly see here how we're able to engage students like never before. This is all premium cinematography filmed in our LA studios. You know, most of the legacy players in our space are either no videography or leveraging 30 plus minute videos. Everything we do is short form. You know, especially GenZ Engine Alpha, they're looking for 30 seconds to a minute clip. All of our clips are around five minutes or less now. and you can see a quick 10 second clip here. Hello again.

I'm Miss Holly, and I'm back to discuss two important ideas to help us understand stories better. The central idea? Really engaging teachers in videography, but videos are only a small part of the course. As you can see on our left-hand panel, this is what a typical course will look like. We'll have one video and then a variety of quizzes,

Chuck for understanding in exams. That way you could be able to take the entire course asynchronous and use it in a live classroom setting to be able to get through the curriculum. One of the most exciting pieces, though, with AI now is now we have all of our courses have video games, which really engages the students. Right now, 96% of our students complete at least one course on subject. That's record high engagement in the industry. And so that means when a student starts at subject.com or subject

dot AI, 96% of the time they complete at least one course. And these new AI video games are a huge driver of this. All right. So you're doing a video game of some type to teach storytelling and fiction. Yes. Got it. Wow. And these are all made with AI, I take it, or you just hire everything with AI. I got it wrong. That's not a good look for me. And I'll quickly show you one last teacher tool so that way you can get a good snippet of that before we jump back in. You know, the educator portal and everything we do for educators is absolutely critical for our success. We live to serve teachers.

We live to serve teachers and students, but teachers are the core methodology we able to get through to these students. When you work with a district, it's very uncommon that a student would, you know, revolt to a school and say, hey, if we don't get subject, I'm dropping out. Typically not a lot of agency around the student voice. The key is to have teachers say, hey, my 20 students are crushing on subject. You need to get subject memberships for your students as well. And so we're really big on empowering teachers. But this is all done with AI. And this is why we're saving so much time for teachers today.

We do all the grading, lesson planning and scoring and feedback for them in our console here. And so quickly you could see which students aren't passing their course. Within seconds, you're able to see, oh, wow, I have five pages of students not passing. How can I do something in class today to help? So it's really got the whole classroom. information data across all of their project progress in the adaptive learning. And you're just using a chat interface for the teacher to solve these problems. A hundred percent. And this is what's really powerful as well because, you know,

we are in a highly regulated industry. Every state has its own nuance. You could be able to go into our product and get state specific suggestions, scoring and feedback, lesson planning and worksheets within seconds. And so we're really excited about the advent of AI there and how we've been able to really accelerate our growth. So started with the premium videos, made them more short form every single year. And now everything we're really doing from here on out has been AI native. We really made the transition in early 25. We launched subject AI in June of this year. Big inflection point for subject. That's really, really interesting. I'm kind of blown away how in depth it is. It feels like you've replaced the entire school framework with a series of videos,

quizzes and so forth that's highly automated. Does this mean that individual teachers, can therefore better serve more students at once? Because I feel like if they're just focusing on one-to-one, the actual kind of quote-quote class size becomes less meaningful. Well, so we're really excited about it. And we see this in the data. One, we want our students to spend more and more time on the product because that shows they love it.

And right now, our students are spending around three hours a day on the product. They love subject. But teachers, we want them to spend less time on the product. We want them to be with the students in person. We want them to be hanging out, making those magical in-person experience moments happen. And so while we've seen teacher usage stay under an hour over the last two years, we've seen triple the amount of students served. That's exactly what we're going for.

Higher leverage, more opportunity for teachers to have better access to student and person experience. And let us do all the mundane admin work. Let us do the grading. Let us do the scoring and feedback. Let you do what you love. That's actually the only thing you've said this far that I've kind of like freaked out about a little bit is the grading element of this because, well, actually, maybe a better

question to ask is what are the underlying models powering the teacher side? of subject AI. We've used a variety of models right now. We're leveraging Claude, which has been fantastic for us. But, you know, we've used chatGBT, Gemini. We're constantly looking to be able to see what's the best. And we have easily, you know, on our code base being the ability to swap in and swap out. I think the big thing for us, and, you know, especially when you look at legacy competitors in our space, you know, in coursewear and such, a lot of them have had the thing of being clicked for credit, you know, a lot of multiple choice, true, false answers. We never wanted that adage with subject. And so previously, we were having, you know, a teacher.

of having to grade so many of these open form essay responses. Yeah. Now with AI, this is speeding things up and allows for us to have a lot of authentic assignments while keeping up mass speed and delivery to these students to get instantaneous feedback. Is there anything lost though when when Claude grades my assignment versus the teacher who knows me and perhaps even the con. I can imagine a teacher's syllabus being partially verbal and not entirely

written down and therefore ingestible into the AI model itself. And so like therefore the grading system might not entirely match up with what the teacher once. It just seems like a place where personally I wouldn't put as much AI versus on the teaching side. Yeah. So when we do digital curriculum, everything we do is a recommended grade. And so the teacher is the final say no matter what. When we do teacher of record, we typically are the final grade, but there's still always opportunity for them to amend edit the grade if they have some feedback for us. And so we're always working with our district. At the end of the day, the customer is always right. And so whatever the district wants to be able to do for their students,

we're here to serve them. Can you actually explain teacher of record to folks? Because I don't think that's actually a term of art that most of our listeners know. Totally. And especially any industry jargon, acronyms, et cetera, it's always like, okay, let's explain it to the greater masses. You know, when I first started in this industry, I had no idea what these terms meant. So basically, we're accredited by two major governing bodies, Wask, Western Association of Schools and Colleges. This is very common, you know, Stanford, USC Cal, they're all Wask accredited. There's six regional bodies across the U.S. And they were also Cognia accredited, which is the international accreditation that took, you know, many years to get. And we're really proud of that. With those

accreditations, we have the ability to service as teacher record for our districts. And that means that without a credentialed teacher in the classroom, we could step in on the back end and provide a credentialed experience through subject.com or subject.com or subject.com. And so with AI, that's made it a lot more scalable and high leverage. And so when they do teacher record, it's typically not a credential teacher in the district. We're providing the teacher of record credentialed instructor on the back end. And this allows for AP classes and such that might have a low student count, but then they can be brought to schools that wouldn't otherwise be able to support them was my understanding. Yeah, I mean, even though we were the first cryptocurrency accredited class in the country,

that's exciting taught by Chad Copeland, Copeland, literacy is a big one as well. It's a high usage for us. Brandon Copeland teaches the class. He teaches that same class at UPenn. So you have an Ivy League instructor teaching the class for high school students across the US.

We're super excited. But then you also see, and this is one we're really passionate. And you know, I came from a small town, Kerry, Illinois. A lot of small town schools, it's hard to fill all, especially STEM courses, science and math. And so when you lose a science teacher in a rural community, it's very hard to backfill that. And so we can step in and help support that district by providing a four-credit course with our teacher of record AI. What's the impact on the student ability to learn and be college-ready? There's been a lot of reporting lately about people get into pick your brand name

school with the unable to like divide fractions and very basic things that I thought we'd handled. How much can you guys help get our graduating high school students up to snuff? I feel very confident our ability. I think that's the big transition we're going through with education technology in general is how can we be able to be more part of the solution and making sure all the rigor is behind it and not allowing students to easily cheat their way through platforms. And so that's why we take academic integrity to the most seriousness and make sure that students aren't just passing. We don't want to be a credit mill. And that's again, one of the negative terminologies in our industry. We want to make sure that students are actually learning and prepare for the next step.

There's over 40 million adults in the U.S. who have college debt, but no degree. We never want to be a of that opportunity. That is super aggravating to me. It motivates me every day to make sure we're helping students either go to top colleges or go direct into the workforce. And that's what we're doing now with the career technical education product that we're debuting in 2026. So, hey, you're hearing it here first. So you're going to help people be essentially STEM ready out the gate. Wow, that's fantastic. How do you make the decision to go sell into schools, which is a slow, arduous process versus just doing what master class did or just go direct, like just go direct to consumers. How do you make that decision? So in 2021, we were all direct to,

to consumer and we're only AP classes and we got a lot of great momentum raise a great round from Kleiner Perkins. Annie Case, our partner there love working with her. Then over the summer, we saw a huge dip in signups because naturally direct to consumer in K-12, a lot of folks aren't active in the summer. And so we're like, hey, do we want to always engage with this, especially you're part of K-12, B-2B is going to be the stickier business model, but also, which is really important from the mission standpoint, it's not just affecting affluent families, whereas B-2CAP was almost all affluent family. When you're selling into districts, you know, we work with some of the most poverty-stricken areas of the country, which means the world to me.

You know, like, we're in some incredible districts. How long do they take to, like, actually get on board? When we sell the product? Is it a year sales cycle, two years? Because there are, you know. In the beginning. And so we really struggle when we pivoted. And, you know, it was a couple tough years, but those years made us stronger. And now we're seeing sales cycles of under six months, which is really great. But in the beginning, learning to sell to schools, especially being outside the industry, there was so much to learn. And we readily acknowledge

that. And we definitely had to really learn the hard way. But now the last couple of years, we have an incredible acceleration momentum. It just took us all of really 22 and 23 to understand the sales cycle and understand who even the buyer was. What were the key stakeholders? What was ICP? And now we have great alignment on it and we're full scale mode. Awesome.

Listen, continued success. Thanks for coming on the program. Everybody go check out subject.com or subject. com or subject. Either one, any positions you're hiring for it? So really talented folks who are willing to be in office five days a week in either Los Angeles or Austin. We have two offices now and we're super excited about that.

And the biggest thing for us, I would say as top level engineers and data folks, we really want to continue to improve the efficacy of our data. And we believe that's the best for unlocking more sales and more product delivery. So data and engineers, please find our way. We have a lot of job openings ready to rock. We love to work with you. You're an in-person guy. All about in person.

Tell people why. We launched in 2020 is a pretty sensitive HR environment, say the least, with COVID going on and whatnot. And so we were very flexible for the first few years. And, you know, we struggled. We didn't have a lot of momentum. We weren't scaling quick. And so I really said, hey, why don't we try just running the company our way rather than trying to please anyone? And I come from a sports background.

And so, you know, hey, when you're playing high-level sports, everyone's training together in person. So we went full hardcore, five days a week in office and the number of skyrocket. And it's 100% not the right fit for everyone. I'm totally empathetic of that. But for a subject, if you want to work with us, we're really passionate about building in person as a team. And, you know, one of our core values is championship sports team mentality. And so we love being able to build together. Love it.

Awesome. continued success and we'll see you next time. One of my favorite parts of being here at Twist is that we get to look back at figures that are currently making news back when we had them in the studio before they were famous, before the tens of billions of dollars. One of those people is Alexander Wang. Now, he was in charge of Scale AI and then eventually sold about half that company to Mehta and became the new God of its superintelligence team.

But back in 2019 on episode 105, Jason had Alexander in the studio to talk to about all things AI. And keep in mind, this is several years before chat GPT came out. Please welcome back to the show. It's Lon Harris. Lon. We're going to do another flashback. And this interview is a straight banger. Yeah, this one was great. I think what's so interesting about this is 2020, it was like, I think spring 2020 was when all of these apps like chat GPT and all the like stable diffusion, like the first generation of image to text ones were blowing up. To me, that feels like when all of us on like the ground level became aware of the AI revolution is real and it's happening.

And these apps are like they exist now and you can use them. So it's fascinating to go back to a time before that and hear people talk about what we were all going to be talking about within a few years. But really like none of us understood at all. Like Alexander at some points in this is sort of explaining to Jason like, what is training? And how do you train a machine learning system? and like, how do they learn? And all of this information is now embedded in all of our minds because we've been thinking about it every day for years.

But back then, it was like very theoretical. You could see Jay's still like wrapping his mind around it in some ways. Oh, absolutely. I mean, asking the difference between AI and ML and where does the data come from and what do you need to do with the data? Can you share data between providers? Alex was like, absolutely not. But let's jump in, Lon, where do you want to start?

Right away, they talk about, you know, Alexander being only 22 years old, already raising $100 million. I like that Jason felt a little bit of kinship with him as a person who had also been a young buck founder who got asked a lot about his age and being so precocious. So let's take a look at that. I guess the thing that most people would think is remarkable,

is candidly that you've raised over $100 million in your last round of funding. That's a lot of money. Quite a bit. It's quite a bit of money from Founders Fund that I think you're 22 years old. 22. Yeah. 22. That's annoying to be young and successful because then every interview starts with your age.

A little bit. It's annoying. I had it happen to me. It was like 23-year-old publisher of CyberServe or 25-year-old publisher of Silicon Airporter. And I was like, why does my age matter? Now, I tell you, when you hit about 35, 40, they don't mention it anymore. Because they're like, well, you're 40.

You should be doing interesting things or be successful in the world. But you've been running this company since you were how old? 19. 19. How did you get into the game? I have a fun little history. I grew up in Los Alamos, New Mexico. So both of my parents are physicists, and they worked at the National Lab in Los Alamos. Tell people about that lab. It was the lab where the atomic bomb was originally built. So the Manhattan Project started in Los Alamos. It was very secretive at that time.

What do they call that lab? Los Alamos National Lab. Yes. It's pretty boring. It's pretty boring. Yeah, yeah. It's a government sponsored lab. Exactly. Yeah, totally government funded. And then growing up in high school, I did a bunch of program. I did all these coding competitions. I was getting recruiter in bounds in high school. So after high school, I actually came out here to work. I worked at this company, Quora, for a couple of years. Yeah, we know it.

They do a Q&A site. Yeah, Q&A site. How'd you get that job? You just applied, and they saw your code, and they were like, okay. They recruiter in bounds, because I was an anonymous person on these coding competitions. And then... So you could just go into a coding competition. Nobody knows your age.

Do you teach yourself how to code? I guess the internet. You just looked it up. You found courses online on YouTube or... It's hard to remember. I think I just Googled around. Anyway, I worked at Quora for a couple of years doing engineering, infrastructure, et cetera. No college.

No. Well, then I went to college after that. Ah. I went to MIT and then got basically bored after a year and started scale. So you left? I left, yeah. I remember Lon back when I was young enough to be considered a rising something or other, and now I'm just a middle, middle, middle, middle person.

But Alexander Reagan is still quite young, so his career trajectory kept going straight up. If I had given you $100 million when you were 22, how well would you have spent that money? not at all. I mean, this is always a fascinating thing for me looking into the tech business where everybody gets started young. And it's all these like prodigal talents who started coding when they were 13. Like, I had no idea who I wanted to be or what I wanted to do when I was Alexander Wang's age in this video. Probably still not even Alexander Wang's age today, but definitely not when he made this video. I was like, I just graduated college. I was floating around. Maybe I want to go to grad school. Maybe I want to be a journalist. Maybe I want to write screenplays. I don't know. I was working in

post-production in Hollywood doing subtitles for HBO DVDs when I was age age. So, no, I don't think I would have been in a good position to raise $100 million and decide what to do with it. One more thing I wanted to pull up there from this opening, though, I thought a really fun little tidbit was, Alexander comes by being a prodigly gifted STEM guy pretty naturally. Both of his parents are physicists who work together at the Los Alamos Laboratory that, of course, famous if you've seen the film Oppenheimer is where the Manhattan Project was

and they were doing the Trinity Testo. He grew up in that kind of like a hard science environment around people who were doing like fascinating high level research. Next up, Alexander explains how scale provides extremely accurate data to the autonomous vehicle companies.

This interview lawn really spent a lot of time talking about self-driving because in the pre-jad GPT era, we didn't think a lot about, you know, chatbots and that sort of I think AI agents weren't even a phrase. We were trying to get cars to stay on the road. Take a listen. The one that has really captivated the world's attention is autonomous vehicles, right?

It's a compelling example because, first, nobody likes driving, but also driving is very unsafe. There's a lot of risk in driving. Sure. And so the- A lot at stake. Yeah, exactly. And so the captivating sort of machine learning model is one that can take in all of the camera data and other sensor data from the vehicle, understand everything that's going on

around it, something that's very easy for URI, but currently, or at least before machine learning, was very difficult for machines, and then can determine the best path to take and figure out how to drive on its own, basically. Got it. So we see the lane markers, double yellow markers, double white markers on the highway. We know keep the car between those two lines as smoothly as possible. We see somebody deviate from their lane into ours. We know to slow down, give them some room, maybe they're drunk.

Machine doesn't know that inherently. We have to teach at that. Exactly. What does scale.com do that Tesla and Waymo don't already do? Because they're solving that problem. Do they use your software? And do they need to? The core problem, as you just laid out, is that machines don't know what to do unless they

have data that actually tells them what they're supposed to be doing, right? And so what that means is one of the huge bottlenecks for machine learning is data. It ends up being like data that tells these algorithms, tell these models what they're supposed to be doing. That's where scale comes in. What we are is sort of this data refinery, if you will. We accept a bunch of raw data from our customers. We go through and process it and we sort of, we tell the machine what it should be doing.

For example, given an image taken by a self-driving car, we would outline, these are where the people are. These are where the cars are. These are the lane markings, et cetera, so that over time these algorithms can learn those things. And I see you are highlighting cars. You're highlighting people. Exactly. And the machine is figuring out, okay, that's the approximate shape of, you know, a Dodge pickup truck. That's a Toyota Prius. And these look like the silhouettes of people.

But that's a human telling the machine that's what it is for now. Yeah, exactly. So the core way that our whole pipeline works is that a lot of work is done behind the scenes by machines and our own AI models originally. And then humans basically give input and correct mistakes to make sure that the end data is extremely accurate. Because that ultimately is what's important for the safety of these systems and for low bias, et cetera. All these things that are needed imperative for machine learning to be performing. So you would go to a customer. So they would give you videos of their cars driving and then you would annotate it for them and put that data into a database somehow? That's exactly right. So for example, if they gave us a video like this, you'll see

originally the first step was a human drawing a box. Yep. And then a machine learning model that's already pre-processed through all this data has determined the path of that vehicle over time. And then we confirm that all this is correct and then send that data over to the customer and they train machine learning malls on top of it. Got it. And this is how I guess one of the cars got fooled. Somebody drew on the ground like an arrow turning and a car followed the arrow, which a human would do too. But they basically drew a turning signal to see if like it would fool a self-driving car. And of course it did. I didn't see this news. I would believe that that is how that's what would happen basically. Yeah. And that's what would happen to a human by the way. So I thought that was the stupidest prank ever. They're like, look, we can. fool a machine that's driving cars to make a wrong turn. In a lot of ways, they will have,

they will have some of the same challenges that humans have been driving. I was wondering, when do you think we'll have a self-driving car in a major city like San Francisco driving, a major route? What year would this be possible? 2030 and over? Or under 2030? When do you think we'd first see this? This is the million dollar question, which is when are... It's actually a trillion dollar question. A trillion dollar question. Let's be real. Where we're at, like fundamentally, the technology is getting better and better every year. The algorithms that that perceive their environments are getting a lot better, like asymptotically better every year. And then the algorithms that figure out what the car is supposed to do, the planning algorithms, et cetera, are also getting

better. So it really is, it's sort of only a matter of time before we get to the point where these kinds of routes are possible and we'll live in a safer world. So that seems to me that you're thinking less than 10 years from now this will be happening with regulation and all that counted in. I don't think regulation is necessarily going to be that tricky. Why? There is precedent for, like if you think of when autopilots first came about, I think there's precedent for how to think about a lot of these things. You mean autopilot in the airplane sense or in the Elon sense? In the airplane sense. So the FAA and whatever regulatory bodies were like, okay, we get it. Autopilot works better than a human. Exactly. It's pretty obvious. Plenty of room up there to operate

when you're up at 30,000 feet. A lot less room to operate when you're going through the tendal and there's six people in the middle of the street, though. The challenges are a bit different, but I think there's precedent for how to think about these things. I think the technology, once it gets good enough, will be clearly extremely good. And so I don't think it'll be that big of a deal. Andrew, so you think 10 years may be less. I'm excited for the self-driving future. What's so interesting here. I mean, obviously, Jason having to be sort of told or instructed about, here's how we're training these AI models, you know, like the cars record the video and then they send it to us, and then we annotate it with all the, like, helpful things that the computers need to learn, and then

send it back. It's interesting just that Jason needs to be sort of like, this was probably the first time he encountered, you know, the differences between what Tesla's doing with their full self-driving and what Waymo is doing with their LIDAR. And like just thinking all this stuff out, again, it's become so second nature for us today. We've all been in Waymo's. We've all, you know, tried this sort of out, or at least everybody in the tech industry has sort of been thinking about it, whereas it was still so new. And they were on this, you know, horizon of this really exciting new time, which they then go to talk about, like Jason asks him what Alexander says is the trillion-dollar question, which is basically predict the future of self-driving cars.

In terms of the future is, many players, several years down the road, all doing incredibly well. Waymo expanded their geographic footprint recently. The Tesla Robotaxi project is doing well. Zooks is in Vegas. Everyone's working with everybody and it's going very quickly. So this AI thing, I think, worked out, Lon, all this hype from 2019. There is one other very prescient moment where they're talking about,

the next 10 years. Of course, this was already six years ago. How often are we going to be in self-driving cars in 10 years? Alexander didn't want to like set an exact. He didn't want to be nailed down. But one thing that he does say is that he predicts that regulation's not going to be that tricky, that like we have some precedent for the idea of self-driving vehicles. He mentions autopilot in planes, but you could also think of like trains or other things that are on tracks or monorails or whatever. At the time, that must have seemed like a very bold prediction. Like, you think cars are going to drive themselves around cities and cities aren't going to want to, like, obsessively monitor that for safety and they're not going to, like, regulate the

hell out of that. And I think he's been sort of proved right. We're doing these pilot programs. People sort of roughly believe in this technology and trust it. And they're able to sort of roll out these tests to more and more cities all the time and expand it. And there's some pushback. But I really don't feel like the regulation of self-driving cars has been super onerous. And it's stopping them, collecting the data that they need. To the contrary, regulations for self-driving cars in the U.S. are so lax that many Chinese self-driving companies are training here. So there you go.

That was a pretty remarkable thing to predict in 2019 when I think most of us would have expected cities to crack down very hard on this sort of thing. Yeah, it turns out that humans are terrible drivers. And if we're going to allow 16-year-olds to drive tanks down the highways at 100 miles an hour, maybe letting the smart computers do its better idea. Lon, the next thing I want to grab is speaking of pressures and kind of getting things right. The two of them discussed the importance of helping humans do higher value work. Now, when we talk today about AI agents, increasing model intelligence, we're always thinking

about what's this going to replace in the human labor pool. Is it blue-collar work, white-collar work? But often we're seeing AI today turn up in very high economic value areas, coding, legal work, that sort of thing. And so when Alexander Wang discusses where he sees the AI market going, it's really interesting. Take a listen. I think there's like helping people focus.

on higher and higher value work. I mean, that's sort of like the core of human progress in some sense. I strongly believe will be the actual story of AI and machine learning. And it'll have to happen more and more and more for us to be comfortable with it. A great example is like truck driving. So there's all these automated truck driving companies. Yeah, lots. We work with a lot of them and bark, Ike, et cetera.

The naive view is that, hey, they're just, they're going to automate truck drivers. And like, if you look at the map of the states, like truck driving is a top profession in a lot of states. And so it seems really bad. But actually, if you look at the system as a whole, there's a shortage. There's a national shortage of truck drivers. And the median age is like 50 or something crazy. There's this kind of like instability in the market because of all of this stuff. The automated truck driving systems, actually what they would do is automate the long haul

middles of these truck driving trips. The boring parts. Which are the boring parts. Arduous. That displace people from wherever their homes are, et cetera. And allow the current truck drivers to focus on these like higher value trips that are like warehouse to a meeting point or whatnot. Yeah, drayage to the factory or even the last mile. I mean, who knows? Like, maybe these trucks will change their form factor and be half the size, be automated.

And when the truck gets off the road, the same truck. Instead of using 18 wheelers, we might just use smaller mid-sized trucks that'll be electric and solar powered. So you have more of them. When they get off, they become the delivery truck. And they just automatically start delivering. It could be a much better model. The sort of like the introduction of machine learning to improve the efficiency of the economy, It'll be slow because of how in general free market economics work, it'll take effect in areas where there's an acute problem today. It'll happen in those places first. And it'll allow the current jobs that exist to become higher value more impactful.

Yeah, I feel like his truck driving example here is really good. It's a very smart illustration of the larger point that I think makes it easy to sort of visualize because what he's arguing is, you know, they're not going to fire all the truck drivers overnight. No. It's not like AI's here. AI could drive the trucks. We don't need you anymore. It's like there are different kinds of truck driving.

There's the kind of truck driving where you're in a straight line down the highway for like three days in a row. And then there's the more intricate. You're going through a neighborhood. You're going around some sharp turns. You're the last mile where the truck needs to go to actually make the delivery or what have you. And so he's saying, you know, like, well, AI would take part of that, the like long haul

boring part that's sort of uneventful. You don't really need a human. for it, and that's the part that makes a human drive three days away from their house and then be away from their family for the whole week to get that haul done. And so he's like, well, you could have the AI do that part and then local truck drivers take care of the more intricate parts where it helps to have a human and they get to stay in the same place. So it's like, AI makes everything better and takes care of the stuff that we don't want human staff

to do. And it's like, I don't know if that's actually how all of this stuff is going to play out. But that strikes me as like the best case scenario, like the use. utopian vision for how AI would integrate into our lives. I'm a little more pessimistic about human labor than that, but I'll take the bullish scenario. Why not? I want to grab one more thing from this from my end, Lawn, and then if you have anything else, we can go to that.

But they discussed the dangers of AI. And what I found very interesting in this was just how much unbothered Alexander is. I mean that really, a completely non-joking way, as a real positive. I mean, I do think we went through a period of AI excitement, a period of AI. concern. And now mostly we're talking about economic concerns. You know, is the data center build out a bubble? But we're not talking as much about like, is AI being used and existential risk to humanity? And so to hear Jason, this is back, you know, five, six years ago, taking the tack that he did, surprise me because this is not how he talks today. So take listen to this.

Well, we believe the true narrative will be extremely positive, actually, versus the current narrative, which is like AI and AI, et cetera, are going to take over the world. There is a possibility that AI could get out of control at a certain point with exponential computing, that's not far-fetched that it could do something crazy and stupid. You only think that's not far-fetched because you watched a lot of these sci-fi movies. If you were to train an AI to work on a drug to kill cancer and you didn't program it properly, it could create a drug that was too aggressive because you didn't tell it, well, in the process of killing cancer, please don't make the person blind or all these other things.

So you could just forget some edge case and some general AI might think, if you said to the general AI, you should work on things that make the human species better and goes, okay, yeah, let's kill cancer. And then it's like, oh, yeah. Let's cure this communicable disease. Great. The best way to cure communicable disease is to kill everybody who have the disease currency, so it can't be communicated.

This sounds far-fetched, but there will be instances where they will make the wrong decision, right? Or it will be just too slow of a ramp up for us not to catch it. The thought experience always go like, oh, you'll make an errand comment to an AI, and all of a sudden it'll take over the world and do something. that you really don't want it to do. I mean, in reality, like, there's a lot of oversight over these machine learning systems right now. There's, like, tens, hundreds of people who look at these models. They look at all the data that comes in and out. They, like, analyze everything. And

they try to figure out, okay, what is this model doing well? Was it doing poorly? And how do we adjust to that, et cetera? So I think that could happen in a world where it's like, we have low oversight of these systems. So oversight is always important in any new technology, right? It's like when we started having airplane autopilot, for example, it would be crazy to just say, okay, of airplane autopilot, just let it fly. Did we put any oversights over the Facebook and social media companies? They have, to be clear, they do. They didn't have oversight.

The FCC, like giving a fine in the review mirrors oversight. It's not oversight. They had no oversight. What regulation is there of AI right now? There's none. You're acting under zero regulatory environment right now. And China's got a negative regulatory environment. It's true that like-

So you should be regulated to your admission. No, no, no, no. That's not what I'm saying. Well, wait, wait. You just said that you should be regulated so that we don't have problems, so which isn't? I do think that there are a lot of important issues about how we deem what AI systems are appropriate, how we look at what they're supposed to be doing, et cetera. I do think governing bodies, the U.S. government in particular, for example, has to take a deep look, understand the technology, determine what is reasonable, what is not reasonable.

But even in their case, they're looking at the miles driven and the accidents. But they're not looking at the code that you guys are writing. They're not looking at anybody's code. They're not looking at the AI systems. They don't even have anybody on staff who could even write an algorithm. them, right? That's also changing, to be clear. Is it? In general, I think they're looking at any lines of code in any of these systems? I'm not sure about the answer to that, but I do think they look at a large amount of data.

So, for example, these... Okay, they do, yeah? Yeah, in Europe, for example, there are all these ADAS systems, right? So there are these driver assistance programs or driver assistance systems and a lot of, like, high-end vehicles that you buy today, right? Keep you in the lane. Yeah, exactly. They keep you in the lane. If you have, like, stopping on traffic, you don't need to do anything. Yeah, adaptive, cruise control, lane change warning. These systems exist. People buy these systems. People rely on these systems. And in the EU, for example, where a lot of these car makers are, where BMW, Audi, VW, etc. are. They have a responsibility to actually both have a large data set. They have collected themselves that is able to validate

that these systems are performing well, as well as pass a series of sort of trials and actual... Oh, really? Yeah, yeah. Different forms of data that governing bodies place in front of them. Well, that would be very interesting. Now to think about it, we do crash tests for cars. You're required to give three cars or something to the government for them to just destroy in their crash tests. But we don't require those cars to go into a lab, get taken over by the governing body, and force them to go into real world testing environments because there's some real world testing environment where you do self-driving up north here, I think. A lot of these companies buy cheap real estate. They outfit them into these mini-town so they can create these funny scenarios. Have you ever been to one of those?

I've never been, but I've definitely seen the video from there. The videos, yeah. Yeah, it's pretty cool. they have like children come darting out, like little cardboard cutouts of children to see if it hits it. This way they can do that in private. It's interesting that at some point the government's going to have to have people who are developers and coders actually getting into the data and understanding some portion of this. At the very least, they'll have to create like the driver's test, for example, the driver's license test for a self-driving car. I mean, that will exist.

And I think we've all just become a little bit less concerned about that because when's the last time you heard Jason talk about, I mean, Terminator? Or like they're going to use the AI to make like a biological weapon. That's going to blind you, yeah. We are definitely like less concerned about all that today. And I think, yeah, Alexander was, he was a bit ahead of the time on that. I enjoy the fact that he is a little bit skeptical of science fiction. I feel like so many of the tech visionaries of the moment love science fiction and obsess over it and try to make science fiction reality.

I mean, I think Bezos and Elon Musk are great examples of guys who were like, you know they grew up reading a lot of this sci-fi and a lot of their ideas about space and robots and brain implants are like i want to bring about star trek and make star trek's future happen and i feel like a lot of what alexander would say was more like aware of the sci-fi element of it but a little bit skeptical like he's a little bit skeptical about a g i and like we're we think we're going to get there faster than we do and and he sort of put forth a very fascinating very sci-fi feeling but he feels the most plausible path to AGI, which is you would create an artificial life form

by simulating evolution in a computer. We can't just design the final form. We would design the like the cells or the whatever was in the pond for of AI. And then the computer would like naturally select that up to be like a life form like a human that was sufficiently evolved. And like you could tell a guy who's thrown around ideas like that, read his sci-fi, but has it led it like, I'm going to cloud, like this is, I'm going to just try to make what Robert Heinlein was envisioning. Like, he's really thinking about it in a practical way and trying to integrate it into what we really know about science and technology.

I thought that was fascinating. If you want more about this, the life cycle of software objects, a short story by Ted Chiang, which was put together in his latest collection of short stories is really well worth your time. It's funny how much there is a shared language long, thanks to our science fiction heritage. And I think it's, it's fun for me because I've read all the authors that are kind of in question here. So when they make illusions and references, it's kind of like my, my personal hobby. So that's a real banger. For the sci-fi news, we should point out, Ted Liang also wrote the story that became that movie, Arrival. Yes, stories of our life.

But the thing that I want to just double click on before we play this general AI clip is that he talks about how just more compute is not going to get us there. And I think that that's become less of a commonly held view in the let's get the newest immediate chips and get 20,000 in one building and then push really hard. Meta in particular, XAI in particular, have been investing a lot in building up their compute power as a way to create more intelligent systems. And it's funny to hear him here say that that's not going to get us all the way to the mountain top.

What a helpful thing. That made me feel less crazy because I feel like sometimes there is that assumption that we don't need to like code things anymore. We just need to put enough chips in a room and then they, will figure it out for us. And to me, that's always, I'm not a programmer, folks. So, like, maybe that's how it'll work. But that always felt very counterintuitive to me.

Like, every other piece of software ever, we, a human had to, like, figure out how it was going to work. And now we're suddenly coming with this idea that, well, if you just put enough compute power together, it'll take care of all of that for us. And I've never really bought that. So it was nice to hear a technologist agree with me in that way. Like, oh, I'm not crazy. This is a big assumption for us to make.

It'd be fun to hear what he has to think about this now, but here's Alexander Way. You believe in general AI that will hit that at some point? AI that is just generally smart can do anything a human can? I believe in some sense. I believe in the sense that, like, for most technological things that humans can conceive of, that aren't physically impossible, if humans survive, they'll happen at some point. Like, I think humans are like infinitely creative, infinitely ingenious, et cetera. I think it's very overblown the timelines that people are talking about general AI happening.

There's a lot of things that are wrong about the common arguments. One of which is people say that if Moore's Law keeps going, then we'll have all this exponential compute. It's only a matter of time before we produce these general AI. Moro's Law is going to be dead. And then quantum computing is very far away despite recent press releases, et cetera. I think that leg of the argument is not actually that strong. And then I also think it's not even clear that if you have infinite compute, you'll be

able to produce general AI. I think that's very unclear. So infinite compute helps narrow AI because you're doing a number of scenarios and playing out every scenario in Go, the game, many more permutations than poker, many more permutations than chess, which is a finite data set. So yeah, more compute power on those things, certainly get you quicker ability, or even just throwing people into a random video game, like OpenAI is, sure. Definitely. But generally, I, taking somebody who've mastered chess and then saying Master Go and then Master Impressionist painting, it's different. It's very different.

One of the arguments goes that once you have enough compute, you can create. artificial life by basically simulating evolution. It's one of the more vogue arguments. You have so much compute power that you can say start with this tiny piece of bacteria, whatever, then grow it and grow and grow an entire evolutionary system to the point at which there is a human-like species and then grow that human-like species in whatever number of scenarios with a big brain into whatever comes after us. Even if you just grow the human-like species that's as intelligent us, then that's kind of good. That would be general AI as read as well, right? Because general AI I would normally, most people would define general AI as not even being smarter than us,

but being as smart. But somebody would have to code that and program that and build the systems to do that. It's not just going to magically happen, right? It's very unclear if that's even possible. That's an argument. I honestly think that's the most plausible argument. It's very much so science fiction in the sense that we're not close to being able to even validate the hypothesis.

So I don't believe in general AI anytime soon. Lon, I just love how many people we've talked to over the last decade, two decades here on the show. So our archives are deep. We shall mine them. But that's all for Twist today. you're the best we'll talk to everyone soon goodbye bye bye