The a16z Show - The Next Frontier of AI Video Is Control

Episode Date: September 17, 2026

a16z General Partner Jennifer Li sits down with fal co-founder Gorkem Yurtseven and Head of Engineering Batuhan Taskaya to discuss what changes when generative video becomes fast enough to run in real... time.They unpack the technical work behind H3 Max, fal’s post-trained version of MiniMax’s open-weight video model, and how combining model post-training with systems and hardware optimization significantly reduced generation time while maintaining quality. That speed has enabled experiments with continuous video, including streams that can remember previous scenes and respond to new directions while they’re running.They also discuss why the next challenge may be less about speed and more about control, from camera movement and lighting to characters, motion, and lip sync. And they explore what those capabilities could mean for professional creative workflows, where artists and studios need predictable tools rather than simply generating a video from a prompt.Resources:Follow Gorkem Yurtseven on X: https://x.com/gorkemFollow Batuhan Taskaya on X: https://x.com/isidenticalLearn more about fal: https://fal.aiFollow Jennifer Li on X: https://x.com/JenniferHli  Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Transcript
Discussion (0)
Starting point is 00:00:00 Generative media is along with the coding agent market. What we call is token market fit. Everyone's waiting for a large consumer moment in AI. I believe H3Mex makes it possible. Were you surprised by the speed up and the gain you could get from post-training this model? We have a version called H-Termax Turbo that's public that can generate like a 5-second video in like 1.5 seconds. From a cost standpoint, it's also like 2x less. People starting creating these beautiful scenes using an LLM model, GPT, Astra, in Blender,
Starting point is 00:00:36 and all of a sudden it unlocked the whole new workflow for Hollywood and professional people. We have been very, very focused towards speed, performance, quality, and now we have a really good base model. The next month or two is going to be fully focused on. What happens when AI video becomes fast enough to generate in real time? A16Z general partner Jennifer Lee sits down with Fowl co-founder Gorka Mirdsevin and head of engineering Bhatuan Tashkaya to discuss H3Max and the rapidly changing generative video stack.
Starting point is 00:01:07 They unpack how post-training and systems optimization made video generation significantly faster, opening up new experiences where video can run continuously, remember previous scenes, and respond to direction as it plays. But speed is only part of the story. They also discuss the push toward greater control over camera angles, lighting, characters, and motion. and why those tools could make generative video more useful for professional creative workflows. Welcome Gorkham-Botwan to our podcast again. We did the last one last year. This is long overdue, and we have such an exciting model to talk about, which is false H3 Max.
Starting point is 00:01:46 The day when I came out, I was calling it, it's really in the league of its own. Like, it's so funny to see the benchmarks where you have the dot of this model on the far left or far right, and then everything else is on the other half. And that graph is actually log scale. It's actually further, but we had to fit it in. We had to do Luxcale. That is hilarious. The time portion, the quality is not.
Starting point is 00:02:07 For sure. The internet noticed, for sure. There are so many viral tweets about it. Like people really played around with this model. Maybe just give us the backstory of what inspired you to post-trained this open-weight model from Minimax. And how did you get the quality and speed to where it is? First of all, the Minimax H-3 model is the first three. truly open source, very capable, like, latest generation video model out there.
Starting point is 00:02:36 So even though we work with some of the other model labs to run inference for them, we never had this capability, like had the right to add this capability on top of it. So Minimex came up with their very capable open source model that is truly last generation, can take references, like very familiar architecture to any other video model. we thought this is a great opportunity to go all in and see what we can do. And again, we did like many different things that we are going to talk about that combined gave the results that you show on the graphs. But the biggest reason why everything came together for this particular moment
Starting point is 00:03:15 was because H3 was the first truly next generation video model that's open source. What is the idea given like FAA has been known to be like a general media inference serving platform? Like what is the idea to get into post-training open weight model? Like you talk quite a bit about it in the blog of combining the system work with the model itself. Maybe talk more about the work behind that. Like generative media is, I would say, along with the coding agent market, what we call is token market fit. And the way we define it is as can a single person productively spend a lot of tokens?
Starting point is 00:03:52 And the amount is like 10K amount, something like that. So there is incredible amount of demand in the market to generate video, to generate many things at the same time. And a person who is doing this for their daily job, they spend in front of a computer and do this all day long, and they spend thousands of dollars, lots of tokens. And since around April, the whole industry and file itself, we've been compute constraint.
Starting point is 00:04:21 We are growing as much as we are adding compute. Like there are things we do here. and there, but the whole industry has been compute constraint. And we've always been looking for efficiencies where we can relieve that a little bit so people can use this more. So that has been the idea behind everything we've been doing since April. And this just came at the right time because this makes everything maybe an order of magnitude more efficient. So it gives more compute for other models or even more tokens can be generated. using H3MX.
Starting point is 00:04:56 I think Pottaan would agree on that. Yeah. Like just from a system-wide optimizations, which is what we have been doing for the past three or four years, you can maybe make the model 2x, 3x faster while producing the same quality,
Starting point is 00:05:09 right? Because it's at the end of today's same model, same architecture, you have the same constraints, you're just trying to optimize what you can get out of the chip itself. And there is a roof line there. And we have been approaching
Starting point is 00:05:21 that roof line more and more, especially like lately, because our entire team has been focusing on how do we get out more video pixels from a single chip as much as possible. And this new set of post-training related optimizations with system slash model co-design enables us to go beyond that roofline by an order of magnitude. And like we just felt the pressure. We have been working on it on top of open source image models before the video models.
Starting point is 00:05:49 We did one version with ideogram. We did one version with flux. So we have been like experimenting with how can we build? post-training infrastructure to take an existing model, build kernels and systems design around it to run it very, very fast for a specialized version that can beat anything else that we would get just by running the model itself. And, you know, combination of that plus just like getting a frontier video model on our hands and all this expertise, we were able to call by like an order of magnitude in terms of speed. Incredible. Let's dig into that. I mean, get some of the numbers wrong,
Starting point is 00:06:22 But there's like efficiency numbers, there's cost numbers, there's speed up numbers. Right. Not everything means efficiency, but it all adds up to be very efficient. Yeah, I guess what is stunning to me is there is like a magnitude lower cost and also much faster. I think it was like 35X speed up. Yeah, compared to the original minimax H3 endpoint. Well, at the ELO score, you didn't really sacrifice quality.
Starting point is 00:06:46 So yeah, just revealed a little more of the secret sauce behind of, is this more of the type of system work you have done, like, did you have to do like model architecture change? Is it a system work that really brought down the cost and latency? And how about like the next generation of chips like GB200 fits into the whole story? It's just like compounding effect of like multiple different optimization variables that we have been targeting. The first one is obviously, okay, you go from like the base model to a model that's like post-trained to be like more efficient.
Starting point is 00:07:17 For diffusion models, this is just essentially how do you go from running 50 steps to running something like 20 steps, right? Like, you're just trying to optimize that pipeline. But as soon as you go from 50 steps or 20 steps, you lose quality. So you need to target in the optimization scene. Okay, I want to improve the quality and then I want to apply the optimization. So we have like checkpoints of this that has significantly higher quality, but obviously slower.
Starting point is 00:07:38 So what we initially did was, okay, let's run our post-raining and our L pipelines so that we can improve the model's quality and then apply the optimization stack on top of it so that the end result gets you the same quality or even like higher quality than the original model. but at the same time you're like an order of magnitude faster. So most of the gains come from post-training this model to be like compatible that you can run this on like less amount of steps. But on top of that you add like all the kernels and systems engineering work that you do
Starting point is 00:08:08 that brings your like hardware utilization from 30, 40%, which is like standard in like many inference workloads to like 70, 80%. And 70% on theoretical MFU, which is like impossible to reach. So you're essentially at the roofline of what you can get out. And then these models are not just like a single, oh, you just give a prompt and you get a video back. They're actually pipelines underneath. You need to take a prompt.
Starting point is 00:08:29 You need to like run an LLM, like a very large LLM to go expand that prompt to a format that the model was initially trained at. Generate the video in the latency space and then decode those latency back into pixels. And then depending on the workload, there might be an upscaling component involved. So there's like multiple components. And every single component by default is unoptimized. There's still like lots to be gained there. And like we just looked at it.
Starting point is 00:08:52 from a perspective of we are going to get the maximum out of every single component. This made us go around LLMs at super high speeds, right? There is that component. But for a different workload, this is not like something like an agent decoding LLM workload where you have very high cash rates, where you have higher sessions. It's a single shot. Give a prompt. You get a prompt back and there's no caching.
Starting point is 00:09:10 You're operating at low batch sizes. So there's a completely different set of optimization on the prompt expansion side, completely different set of optimizations on the diffusion model, completely different set of optimizations on the VAE that you take from latence to pixels. and you just combined all of these to have an effect that compounds. From a hardware standpoint, going from something like hoppers to black valves, you see something like 2 to 3x improvement by itself. But from a cost standpoint, it's like pretty comparable
Starting point is 00:09:36 because the cost is also like in that league. So I would say it only reduces your like wall clock time, but not just the efficiency itself. But it obviously helps if you want to go significantly beyond real time. If you want to generate five seconds of video in less than three, two seconds, then you need like some of these like latest generation. hardware today to unlock that possibility. Maybe this is a detailful question.
Starting point is 00:09:57 Is the model being served on single GPU or like? It is like majority of the video models today run in a single node configuration, which is 8 GPUs because once you start scaling beyond 8 GPUs, the efficiency gets less and less because of the communication overhead. And existing both like the existing Minamax H3 endpoints as well as like other video models are probably getting served at like, you know, single node configuration. Same with this. It's like running on in parallel across 8GP.
Starting point is 00:10:22 use. And do you think there will be more efficiency gains in there that you can either, you know, optimize more of the steps in between by sacrificing maybe some of the, like, narrowed down the user experiences, let's say, like, the different type of inputs and outputs or like, as you're thinking of parallelism, like, is there more choose to squeeze? Maybe that's the question. We released a turbo version of H3 Max. So initial idea was calling this H3 Turbo.
Starting point is 00:10:48 And we were like, we don't want to call this turbo because the quality is like better than the original one, right? Like, this needs to signify how good at home achievement it is. So we released H3Max, but like a week later, we had like, you know, like our team was like, we can run this 2X faster at like 97th percentile of quality. Like we run evals.
Starting point is 00:11:05 They're like almost the same, right? Like there's still like there's like a noticeable, like there's a small noticeable loss in quality. But we have a version called HG Max Turbo. That's public that can generate like a five second video in like 1.5 seconds, which is like insane. And that's also like 2X, the 2x like from a cost standpoint is also like 2x less so they're like depends on like how okay you are
Starting point is 00:11:28 with like losing quality you know you can go down and like today these models are so cheap and so fast that I don't think people need any faster or like any cheaper like it's already like at a point where the from a cost standpoint compared to the frontier itself it's an order of magnitude cheaper compared from like a speed perspective it's more than an order of magnitude faster and And like, you just like enable all the experiences. I think we would need to see, like, I think we would need to see what else levers that people would need. But my bet today is we just need to improve quality more than like the speed at these speeds. Right. Like let's fix the speed and let's try to push for quality and controllability of these models,
Starting point is 00:12:09 which is like, you know, what we have been pushing in the past two or three weeks. I think controllability is key. Like when you first did it, we did text to video and then image to video and then reference. came later, which adds a ton of controllability, and it's basically the default mode, how people use these models these days references. And then we are now adding different Laura's fine tunes of the base model as well. We are working on like a lip syncing version. We are working on a different camera angle Laura, different style Laura.
Starting point is 00:12:43 So again, open source adds a whole ecosystem around the model and it really, really helps. Were you surprised by the speed up and the gain you could get from post-training this model? Like I saw it as a little bit of a surprise that like one day, I think it was a Saturday, you'll launch the model and the Sunday people put it on on Twitch and become a real-time model. Like that's the interesting part of like when you reviews. We did e-vals. Like we spent a ton of money doing e-vails on our own. I don't know, like tens of thousands of dollars even.
Starting point is 00:13:14 And like the results were unbelievable. And then like, the plan was to just release the model without doing external e-wels. And then, okay, we decided let's hold off. Let's not tell people that this is like so much faster and so much better before we have some external validation. So we waited like three, four days to all these other like e-wel platforms to actually run the evils. So we match the results that we have externally as well.
Starting point is 00:13:46 And that's how we launched it. Because as you said, results were a little too good to be true. Yeah. And it was. I guess, well, you're taken by surprise that the real-time use case that came out of it, or what are some examples that you think this model has, like, unlocked of the experiences, the prior models couldn't.
Starting point is 00:14:05 This happens at Fall once in every couple of months where, like, the whole company gets hold of something and the creativity just explodes and everyone is just working on a, new little app or a different optimization Laura, whatever it might. Like the whole company gathered around this model and like some front end engineers
Starting point is 00:14:31 started working on like interesting applications. We can talk about our like world model accelerator team which is brand new. They started working on the live experience. RTC. The WebRTC live
Starting point is 00:14:47 experience. So like they were five, six different parallel little projects within the company. And like, I think we broke a record on Slack that day. How many messages were sent in the company? Because like, and like we have a distributed team. We have people all around the world, like mostly in San Francisco. But it's like incredible when you see like the 24 hour development. Like when people like work 16, 17 hours and then someone else wakes up
Starting point is 00:15:19 and picks up that, like, and that went on for like three, four days. And that's when we released all these projects. Take me into that. It's so interesting because, like, you imagine, like, a model or product launch being, like, planned out, like having, again, like having all these, like, eval vendors being ready, lined up and, you know, ship something out. And then, like, you let the world or the external users take it and then experiment and, like, build experiences put online.
Starting point is 00:15:48 Yeah, it seems like. you know, people internally who are very creative just like took this, drop everything they were doing, like launched an experience that got really popular on Twitter. Do you want to tell us about that one? Yeah, of course. One of our engineers, Rehan, just by himself
Starting point is 00:16:03 completely, started streaming a live stream of continuous generations of H3MX from his laptop. He was... He's computer. His computer, exactly. He was doing some like prompt tricks, trying to keep
Starting point is 00:16:17 like a coherent story. and then he started live streaming that on Twitch. In parallel, Levozio, famous Twitter influencer at this point, had a similar idea, and he reached up to us that he has a website ready already. He wants to post the streaming himself
Starting point is 00:16:36 and have a website that does infinite streaming. Internally also, we had another team who was working on a continuous version of H-T-Mex. So H-T-Max is, like the ReHans version and Levels I-O version were independent clips. It's still very fast, but the clip starts, it ends, and then you take the last frame of the clip, try to put it in the next one, and try to create a continuous. And there's no memory.
Starting point is 00:17:07 Like the second clip doesn't really remember anything from the first clip other than the last frame. But internally, the ML team was working on a version where, The transition is more seamless. There is like two minutes of memory. So like you're in a scene and when you direct the model or someone else enters the room, it actually like everyone looks at that person entering and the scene is continuous. So internally we were working on that. And then another team was working on an experience we called Fall Live for the Continuous version.
Starting point is 00:17:42 So we had three parallel efforts going on that were all independently going. viral on Twitter. And these were all like spontaneous. Like you didn't plan for it at all. Yeah, exactly. And they just became products
Starting point is 00:17:55 and experiences in the next, in the following days. But tell you let's talk about how we made them all more continuous. That was very surprising to me because I've never seen that actually work on a video model before.
Starting point is 00:18:11 Yeah. So going back, we have been like very, very focused towards role models and essentially like action controlled or like, you know, action-driven, real-time continuous streams of video. And the problem till like, you know,
Starting point is 00:18:25 something like H-Tremax was, quality was not good enough at all. It was just like, you know, it degraded a lot. It didn't remember the past before. But we built the infrastructure. We built the infrastructure that we can not go stream video, had people control it in real time, being able to like multiplex it to multiple people,
Starting point is 00:18:42 very low latency. And at the same time, our ML team was essentially trying to take every single video model and try to apply this set of optimizations and tricks to, okay, how can we make this generate instead of a five-second video, 15-second video, 30-second video. But you were always like below the real-time factor where, you were always like, you never could generate like five seconds under five seconds. Once H-Tremax unlocked it, the ML team was like, this is insane, which are like separate teams
Starting point is 00:19:09 internally. We have a research team. We have an M-L team. They saw this and like, this is insane. We can apply all these like, six. set of learnings that we had in previous models where we attempt to do this, where instead of trying to generate a five-second chunk, let's try to generate, you know, like a 50, like 10-second video, and then the five seconds from previous one is still attended, we still remember it, and
Starting point is 00:19:31 like as the video goes up, we can like extend that memory up to two minutes. And you need to do extremely clever optimizations because attending to a two-minute video is just extremely, extremely, extremely compute-intensive. And just like, it goes up exponentially, from like a compute standpoint. So we did like lots of optimizations there, but that none of the state, we were able to, okay, we can remember back to two minutes, which is like generally good enough from a memory perspective. And then obviously with like prompt tracks, you can still like continuously remember more finer, like the grain details above the two minute mark. And you can essentially stream infinitely. We captured at an hour from that perspective.
Starting point is 00:20:08 And then that team just like released that model under H3 Max director, which is public for people to use. And I think it's the only model that can generate. like, you know, up to 60 minutes, continuous videos. That is action control. You can, like, you know, start with a prompt, say, like, there's like an office setting and someone is, like, you know, working. And then, like, 30 seconds later, it just imagines by itself. 30 seconds later, you can, like, say,
Starting point is 00:20:31 a woman walks through the door. Like, it can take the prompt and reflect it immediately, which is the most fun part. And the office is still the same office. Same office. The camera can pan back to the original person and the original person is still there in the same state. Yeah.
Starting point is 00:20:45 So, you know, we released that and it got like, we did this like fall live website to just like demonstrated because it's like, people need to see how cool this is, right? This is a new technology. I don't think people are like really aware. And it got also like very viral immediately because we also let people vote on what the next section is. It was like, you know, like a form of crowdsource. Like the chat was controlling whatever was happening, which is fun.
Starting point is 00:21:10 But obviously, you know, we limited on like the options and then they could pick, oh, like a banana enters the office instead of a movement and it's like more fun and like people start like having this and we start adding more channels and like every channel had a concept there's like a channel where it's like full chaos there's a channel where it's like cartoons from like 80s and like the model is like extremely capable and it just like remembers like so many different concepts and it has like a big big memory from like styles and like you know concept perspective so it just became like a very fun experience underneath again like there's so many really incredible experiences coming out of this. Like H3 Max director was just another huge surprise to me. It's like
Starting point is 00:21:49 I found it interesting in the in the gen media market that you, it's not like, you know, like language model, you have like this linear graph of like just continuously compounding on like, you know, intelligence capability and so on like feels like in the field you're operating in. It's always like a few months of like sort of quiet time, but like a lot of things are bubbling. But like in a very short period of time, like everything bursts, like all these things in combination come together of like the base model being good enough.
Starting point is 00:22:25 Like you can get the latency down to the point where you can like... References. Yeah. Yeah. Like get the real time experience, but also like apply controllability on top of that real time experience. Like this just opens so many, you know, opportunities of like live experiences
Starting point is 00:22:41 where like end user can control, happening on the screen, which is incredible. Like, we have imagined a lot of these experiences, but never been able to, like, really play around with it. Maybe just, like, tell us more about what you're seeing from the market of, like, how are people using, like, the director capability? Like, what are you seeing creators of creating that you haven't seen before? And what do you think that unlocks as far as, you know, what people can do with this medium?
Starting point is 00:23:10 Yeah, it's been, like, almost three weeks since we, released H3 Max and already it is the most popular video model on the platform on the file platform by like double almost like more than double in terms of like volume
Starting point is 00:23:27 so in a lot of other platforms it's also becoming the default model that people interact with because it's so fast so cheap it just makes sense if you if you come to a platform this is the experience that that you want to see.
Starting point is 00:23:44 So in terms of like popularity and volume, it's taking over at least from our vantage point. And for Max director, again, there has been, I don't know, tens of different versions of these live streams. Some of them are still going on and like becoming more and more popular. We are trying to work with some like AI IP holders,
Starting point is 00:24:09 people who have like AI shows on Instagram and TikTok and do train Alora on their style and do a live version of their show. So we have a couple lined up already. So that's going to be very exciting. And like the way
Starting point is 00:24:25 people like if you talk to a creative technologist prompting voice has already become something that like they use all the time is like using whisper flow or the chat GPT voice mode. Yep. And and now like you can
Starting point is 00:24:42 keep talking to the model and it's like almost as if it's a rear director in a real movie set directing like the camera directing people where to go you can do that and like our creative engineers started using these models like that so we'll see like a lot of interesting experiences are built as we speak very interesting as you like the video is playing you are you are like talking to the video and what's what's being displayed changes are That's incredible. And talking about, like, how the memory piece holds now, like, are, again, this may be a technical detail, like, the capability of remembering what happened in the last scene or in the last couple minutes of scene. Like, are you remembering that through, like, the frames, the images, or is it, like, through text?
Starting point is 00:25:33 It essentially, no, it's essentially, like, it remembers through all video, obviously very, very compressed because you can't attend it of all video, but it essentially, knows like most of the details happened in the past two minutes from its own generations. And above the two minute mark, it has like, think of it as like a evolving system prompt on top of the two minute mark from two to 60 minutes where it knows like the overall structure, overall detail. So it remembers like the last few scenes. If you think a scene is like 15, 30 seconds, then it's remembers like the last four to eight scenes.
Starting point is 00:26:04 And then on top of that, there's like a continuously evolving, gradually evolving system prompt that like keeps remembering the. like the overall coherence of the of the world. Everyone's waiting for a large consumer moment in AI. Now it's like good enough and cheap enough that like a truly novel social AI experience can be built on top of it. Maybe let's talk more about the economic side of this.
Starting point is 00:26:34 Like what is the I guess one just like talking about serving cost for like same minutes of video with its 3 Max. And how has it changed your thinking around like your footprint of like inventory of chips? Like how do you want to have like different steps of experiences serving to the end user? But I mentioned this a little bit. Like everyone talks about how complex the next generation LLMs are. But video models are actually very complex as well because the pipeline has different components and sometimes they require different hardware configuration for efficiency, things
Starting point is 00:27:16 like that. So if you were to do this even more efficient, let's call maybe even cheaper, we would probably run different parts of the pipeline in different types of hardware. Another interesting thing would be to run it on consumer hardware for people to run it in their own machines at home, like optimizations don't translate 100%, but translate somewhat, somewhat close to that. And then we can do extra work to translate more of it. So doing these optimizations in different types of hardware and combining the pipeline in a way that it's even more efficient, I think that's what we are going to do in the next coming weeks.
Starting point is 00:28:03 Amazing. So you will have people like Rohan that can stream a partial. of the experience from this computer, but also having like the director and the control plane we're living on the... Exactly, yeah. On the cloud. Makes sense.
Starting point is 00:28:19 So we talk about all the consumer experiences this model could unlock. And it seems like Bertuan is happy with all the efficiency, like squeeze out of the GPUs. Now we're talking more about how do we like improve quality and controllability of these models
Starting point is 00:28:34 so that like, you know, the high end of the market, the Hollywood creators, directors can take this to the next level. I saw some demos. Coincidentally, like, you know, this model came out of, came out the same week or week prior to Astra. People were combining the Blender experience with H3 Max from Foul, like talk about how it's going to impact the Hollywood world.
Starting point is 00:28:59 Using Blender with one of these AI models together is an extremely popular workflow for professional work. Basically, you render a low resolution of your scene what you want to do using Blender, previous non-AI technology. And then once you add that video as a reference to an AI model, you basically get close to 100% controllability. And this is an incredibly popular workflow
Starting point is 00:29:31 for BFX artists, people who are doing this professionally, because they want, exactly, they want to get exactly what they put in into the model. And as you mentioned, a week after we launched H3Max, people starting creating, generating these beautiful scenes using an LLM model, GPT, Astra, in Blender, and all of a sudden it unlocked a whole new pipeline using an LLM to create a blender scene and then passing that to the H3MX model or any video model.
Starting point is 00:30:06 But it works very well with H-T-M-X because it's extremely fast and you can like try many things all at once in parallel. And that unlocked the whole new workflow for Hollywood and professional people. And it gets you to like close to 100% controllability. As I said, we have been very, very focused towards speed, performance quality. And now we have a really good pace model. I think the next month or two is going to be fully focused on, okay, how much controllability we can add to these models. So that professionals at studios, professionals who want to actually produce, like, produce content that fits their use cases perfectly can leverage these models. The team has been working on an amazing, you know, like a lip synchronization model where, you know, you can just supply the audio, you can supply your, you can supply like a video or an image reference.
Starting point is 00:30:58 And then it can like synchronize the lips perfectly. Same with like motion controls. You can just take a motion of someone dancing and apply to like a video. your AI generated character and it fits perfectly. And this is like, you can like get these results with like basic prompting and you're going to get like 80%, 90% reliability. What we are targeting is like 99.9% reliability in the outputs so that you can actually trust the model did every single aspect of this generation perfectly.
Starting point is 00:31:26 And that's like what we've been pushing. One big launch that we had last week was the camera controls, which is essentially you can direct where the camera is going, within the video perfectly to the degree. And this is by like describing in the prompt or like generating the... You just essentially like underneath you give a JSON of like, I want camera at like zero zero at T zero. I want camera at like 90 degrees angle at T1.
Starting point is 00:31:53 Like you essentially supply a structured description of where your camera needs to be at any point in time. And then the model is like perfectly conditioned to regard it as like the only source of truth. doesn't like hallucinate, like, where the camera should go. And it's just like, you can essentially reconstruct 3D scenes from a single input, like because the model itself is a very good video model. But at the same time, you know, it's like perfectly adheres to the camera itself. And this is because the base model itself already has the understanding of the camera angle that you can. It doesn't respect it.
Starting point is 00:32:29 It just under, like, you need to tune the model. You need to tune the model to a signaling degree. And this is like what enables like at large scale, you know, post-training infrastructure. we now have the infrastructure to take H3Max at any capability to do it. Same applies for any new model, right? There's a new video model. We essentially spend most of the time building it
Starting point is 00:32:47 as an infrastructure than just like one of training runs so that we can build like services around this for not just like open source models but for like frontier close source models as well because we see in the market this is like the biggest gap is just how controllable these models are. First we start with text video where you put a prompt, you got a video back,
Starting point is 00:33:06 It was good, but like you never could describe the perfect character for you. And we had image to video where, you know, you used an image editing model and then generally like, you know, the first scene and then the model was like obviously much more fitting, but you still couldn't like say, oh, I want this new character appear at like second three. You need to put it to your first frame or like you can't like you can prompt it, but it's not never perfect. And then we had reference to the video where, you know, you can provide like an initial starting frame and you can also provide, I want these characters with these voices.
Starting point is 00:33:34 Like, you know, that's also like a big unlock where you can essentially say, this is the voice for this character. And now, like, you know, we are adding, oh, within this scene, I want camera to look at this degree at like T0, on camera to look at this degree at like T3. And then we are adding lighting controls where you essentially say where the light is coming from. These are all all compounding on top of each other. And like we just have the unified infrastructure to just apply this to any model at this point. That's incredible. Hollywood is our fastest growing segment.
Starting point is 00:34:00 And there's a lot of noise about how AI. I might disturb Hollywood, but Hollywood usage was nonexistent a year ago. And in the past year, it grew, and now it's the fastest growing segment. Like Amazon MGM Studios in their conference, they released their NARA tool. It's mostly backed by file infrastructure behind the scenes. And we are seeing incredible, incredible pool coming from Hollywood. And exactly what they need, these small points of, rather than generating everything from scratch,
Starting point is 00:34:37 they want to be able to extend the video a little bit. They want to be able to change the camera controls. They want to change the lighting. And someone has to build these solutions for them. What Hollywood needs and what the creators actually need and what the research labs are working on, there's a little bit of a disconnect there. And we believe we can come in and do these little post-training projects
Starting point is 00:35:01 to close that gap. because we work with all the Hollywood studios and we hear from them what they need and these are exactly the things they need, these small point solutions that actually make them more efficient, push out more video, and AI can actually close that gap very nicely.
Starting point is 00:35:22 Maybe say in a little bit different way, like we have been starting at this problem for the last three years as well. Like we see like companies trying to like build a, you know, movie director, like a video model by either pre-trained or post-trained on the video side.
Starting point is 00:35:40 But what I'm hearing is like different people expressing the way they want the output to come out very differently. Consumers talk about it and then like write the prompt and generate the results very differently from a Hollywood director which is obvious, right?
Starting point is 00:35:55 Like professionals want to talk about like you know these camera angles. They want to talk about the lighting. Like you sort of have built a library or like a collection of post-training, like I'd call it data and toolkits that can apply these, any model that you can like grab the weights on so that they are adapted to like a different audience
Starting point is 00:36:20 where they can express their creativity in a bit different fashion to control the model when it unlocks a lot of capability underneath. And half the problem was capabilities all these models. We are solving that. The other half of the problem was legal and data residency, things like that. We made a ton of progress there as well. We now have a system of people can apply with their own IP,
Starting point is 00:36:48 and we unlock their own IP in the models. We are going to grow that, and that's going to be a very powerful thing we do with Hollywood Studios. Also, we now have C-Dance, US hosted as well. We already had previously other Chinese models. Seedance was the missing part. Every Hollywood studio wanted us to have it U.S. hosted. Now that's available.
Starting point is 00:37:12 So there are no obstacles in front of these Hollywood studios now. Everything is ready. And we believe they are going to 10x, 100x their AI usage in the coming months. It's such an exciting world for movie lovers, consumers, people who consume a lot of video and creative content. And we have our conference, Generative Media Conference, next week. This is the second time we are doing it.
Starting point is 00:37:40 Last year, it was mostly consumer AI. There were, like, maybe a couple Hollywood executives here and there, just curious about it. And now it's dominated by studios. New AI studios, who are, like, offshoots of the bigger studios trying to do, like, only AI, AI shows, but also like the biggest of the Hollywood studios are also there
Starting point is 00:38:02 because now they have big plans integrating AI into their workflows into their existing systems. So you can see the change in the attendance of the conference as well. That's awesome. Well, for the audience, check out the content coming out of the Gem Media Conference. It's going to be very, very exciting. And thank you so much, Gorkham and Betwan coming onto our show. It's super exciting time for Gem Media.
Starting point is 00:38:27 Thank you. Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify, follow us on X at A16Z and subscribe to our Substack at A16Z.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only.
Starting point is 00:38:58 should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.