The a16z Show - World Models, Robotics, and the Future of 3D AI

Episode Date: September 13, 2026

World Labs co-founder Justin Johnson joins MTS hosts Theo Jaffee and Sofia Puccini to discuss Atlas, World Labs’ latest world model, and the broader case for AI systems that understand and interact with the physical world.Justin explains how Atlas approaches three core tasks: generating new worlds, reconstructing real environments from images, and simulating how objects or robots might behave within them. Underlying it is a bigger thesis: just as language models became general-purpose engines for working with text, world models could become a horizontal layer for visual and physical intelligence across industries from entertainment and gaming to construction and robotics.They also explore how world models could change video games and creative tools, why precise spatial control matters, and the potential for “real-to-sim-to-real” robotics, where a few photos of a physical environment could eventually be enough to build a simulation and adapt a robot to that specific space.Resources:Follow Justin Johnson on X: https://x.com/jcjohnssFollow Theo Jaffee on X: https://x.com/theojaffeeFollow Sofia on X: https://x.com/schisofreniaFollow MTS on X: https://x.com/mtslive  Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Language models are these general horizontal engines for processing streams of discrete tax, discrete tokens. Those have had tons of applications from everything that we know and love today. And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality, they should be applicable to tons of different industries, right? From entertainment to VR, to construction, to robotics.

We live our lives in this physical built space all around us, and we need models to help us with these things as well. Language models gave AI a way to work with words. What happens when models can understand and simulate the physical world? World Labs co-founder, Justin Johnson, joins Theo Jaffe and Sophia Puccini on MTS to discuss Atlas and the broader idea behind world models. They explore how models could generate and reconstruct environments,

what separates them from video generation, and applications across gaming, VFX, and robotics, including turning a few photos of a real space into a simulation for training a robot. We're live with Justin Johnson, who is a co-founder of World Labs, which is a startup that builds spatial intelligence products. And just today they announced Atlas,

the world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstruct them in 3D. Model the world, move the camera, and simulate space and time. So congrats on the launch. The launch video looks really cool.

Welcome to M.S. Thanks for having me. Absolutely. So tell us a little bit more about what this product is specifically what you can do with it, what people will use it for. Yeah, so this is not a product launch. This is a model announcement. So the products are coming later.

This is the base model. It's going to be used to power our future products from World Labs. But at its heart, it's a world model. It does basically three different kinds of things, generation, reconstruction, and simulation, right? Generation is you want to generate new worlds that don't exist, right? Like I can start from a text prompt, I can start from an image prompt, then I can direct the camera and sort of fly through that world and whatever, have the model generate a world that never existed before.

So that obviously has tons of applications across many things in creativity, right? From VFX to gaming to all the kinds of use cases that people are using there. The second major capability of Atlas is reconstruction, right? There, sometimes I don't want to generate a new world. Sometimes I have an existing space in the real world that I like to model and bring into virtual reality or the virtual world in some way. Right. Right.

So there what Atlas can do is sparse reconstruction. I can take just a few photos of a space, as few as one, up to 100 or more. And more photos you give it, the more you can reconstruct that existing real world space accurately. And the third thing we can do is simulation, right? And this is going after more robotics. Yeah. Well, both VFX and robotics. But there we want to be able to take an existing space, recreate that space in 3D,

then simulate how a robot might behave in that space. And we show some examples of all these different use cases on the blog. Yeah. Is the ultimate vision for World Lab something more like environment for teaching robotics how to work? Or is it more like a metaverse that people will like spend time? All the above, probably. Or is it's just kind of all the above. Yeah, I think it's all of the above.

The central thesis here is that there's a new, there's a different kind of horizontal model that we can build called a world model. Like language models are these general horizontal engines for processing streams of discrete tax discrete tokens. Those have had tons of applications from, you know, everything that we know and love today. And our thesis is that there exists another category of model called world model. that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds. And if we can build these with the right generality,

they should be applicable to tons of different industries, right? From entertainment to VR, to construction, to robotics, you know, we live our lives in this physical built space all around us, and we need models to help us with these things as well. Right, so the model is the input that would go to all of those industries, like the input intelligence. I'm curious, like, how this has evolved and why Atlas is a big jump here. So I would imagine like one of the difficulties with procedurally generating worlds is that like after a certain point, like, I feel like, you know, with video models like 10 seconds in, I feel like the output could start to get jumbled, especially if you have like a reference image.

And then it starts to take its own direction. It starts to like hallucinate in its own way. So how do you fix this thing of like at a certain horizon, things would start to get jumped or distorted? Yeah, that's a great question. So here we built Atlas specifically around spatial control and the notion of a spatial context. So when you give Atlas reference images, they aren't just images. They're actually grounded in 3D space. So you can stage those images in 3D and say, this reference image should be here.

This reference image should be here. This reference image should be here. And when you generate a video from Atlas, you're not just giving it a text prompt. You also go in and steer the camera and say, I want the camera to fly this way, this way, this way, with exact precise pixel control. So then we show some examples where even up to a minute of video, we give it more reference images and sort of leave breadgris. crumbs of those images along the trail in 3D, and then you get that direct tutorial control over the video that you're generating. So the idea here is that even as you go to really long generations, it's not a slot machine.

You're generating and directing and really controlling what the model outputs. Yeah. So, yeah, we were just talking right before we went on about our friend Tice from OpenAI, did this project that went kind of viral that was like GTA, but at San Francisco. So it, like, I think it was like a giant Gaussian splat of the whole whole. city or something and you can like walk around in it you can steal cars and drive around you can fly around it's not very high resolution definitely not as much as atlas you think ultimately this is something you'll be able to do with atlas yeah i think so and i think there's actually two

different futures here of like the tech path that gets us there um one is sort of the explicit 3d route that you mentioned right like the idea of generating a gaussian splat reconstruction then importing that explicit asset into a game engine or something else um this is something atlas can do right Like it can be used to generate, you know, 3D Gaussian splat scenes, and then you could import them and generate and build games around them. Another direction where this could go is, you know, you skip the engine entirely and go directly for frame generation, right? And I have the model directly outputting the frames without any explicit 3D in the mix anywhere.

And Atlas is built to do this as well. So it's one model that both handles, you know, 2D when you need it, 3D when you need it, and can handle both modalities in a native way. If you're somebody using this world model and you have all these inputs, Like, how would you steer Atlas to do what you intended? Like, if you're like a film producer or a video game developer, how would you work with it? Yeah, we have some internal tooling for this, but it's going to take a little bit more, a little bit more iteration on the product side to figure out the right way to expose that.

But the internal tooling is actually pretty awesome, right? You can basically take images, drag them into this interface, have this sort of spatial 3D canvas for all your input images. Then they end up with this like proxy geometry in 3D space, and you can steer your camera. kind of with an FPS-type control to like decide exactly where you want this thing to go. And that's a pretty complicated user experience or user interface. So I think we'll probably iterate on different ways to expose these capabilities in nice ways. How does it differ from Marble, which is World Labs' first product? Yeah, so it's a totally rebuilt stack of models, right?

I think the central thesis is the same. We want to build multimodal models that can input many different modalities, output many modalities. This is something that Marble did as well. But one way in which they differ is that Marble really put Gaussian splats really at the center of many things that it did. So Marble could output videos, marble could output 3D scenes, marble could output meshes, but everything was kind of bottlenecking through a 3D Gaussian spot representation. So whenever we

output videos or images from Marvel, you were always generating a Gaussian scene, then rendering those Gaussian back to images or videos. And this is really nice, but we wanted to break that dependency. So Atlas, you know, was built fully from the ground up to both rethink the architecture to make it more unified, more easy to scale, and to have it, have that sort of that branch between 2D and 3D happen earlier. So that when we're generating 2D outputs or video outputs from Atlas, you're not bottlenecking through a Gaussian splash scene anymore.

The model is directly giving you those beautiful 2D pixels. And then only you lift them up to 3D only for the applications that you actually know that you need 3D. So I guess at this stage, like are we outgrowing Gaussians as a technology? Or like what is their role within all of this? I think Gaussians are a tool just like anything else. And you've got to use the right tool for the job, right? Like, if you're working on something like VFX, then the ultimate output is a 2D image,

or a 2D image or 2D video, and maybe that's all you need. But for other applications, especially real-time applications on embedded devices, like I think Gauchans and other explicit 3D representations still have their place. Right. I think there might be a world in the future where we're all streaming pixels directly off these models in real-time from powerful server farms. But, you know, we're not quite there yet. And even if we are, it's going to be pretty expensive. So something like a Gaussian or a mesh is really useful for client-side

rendering, especially on mobile or VR devices, and also really useful for integrating with existing workflows and pipelines. I think we all believe in the future of technology and we're excited about building these solutions that give us whole new experiences. But the reality is that it takes a lot longer for these things to permeate than we expect. So people in gaming and VFX and design and architecture, they have their existing workflows that work on 3D representations. And instead of trying to switch them over all at once to fully AI-native thing,

if you can get with AI into their workflows and meet them where they are, then that's another great use of splats and other explicit 3D representations. Got it. Yeah. One of my favorite video games ever is Microsoft Flight Simulator. Yeah. Which is you get in a plane and you fly around the world and it uses like a lot of not quite real time, but it uses satellite data in order to like reconstruct a 3D model of the world, basically in reasonably high fidelity, not incredibly high, reasonably high. You think Alice will eventually be able to, like, what will it take to get Atlas from where it is now to able to dynamically generate content quickly enough to do what Flight Simulator does?

And like, is this even like the right tech tree for that? Imagine like Google Maps in this world. Like you could just traverse like any space basically. Is it kind of overkill though? I mean, I think it'll happen sooner than you expect. Right. And that'll happen in one of two ways. Like there's one sort of, and actually, you know, Atlas was built to, to, to, to, to,

handle both of those tech trees. One is the explicit 3D route, right? Like maybe we're generating a Gaussian scene or a 3D point cloud scene and then interpolating that on the client. That's something Atlas can do because of its native 3D outputs. The other version, like I said, is just directly streaming the frames, right? And that's something that I think we'll be able to do pretty soon. Okay. I'm also really curious about real to sim for robotics. So, you know, taking like a real video and then building a simulation. So how is this useful and how do you see this possibly being like helping evolve the field. Yeah, so I think there's, there's a couple different ways that this is useful.

One is for evaluation, right? Like, you definitely want to evaluate robotics in the final situation, in the final environment where they're actually going to end up. Right? If they're going to, if you're going to have a robot working in a particular factory environment with a particular type of machine in a particular environment, like you want to make sure the robot works exactly in that intended environment. So there's a real, so there's a really useful role in simulation at that like last mile

that last mile part of robotics training and evaluation, where we build a simulation that matches the final use case as accurately as possible. Then even if you've got your big pre-trained robotics foundation model that knows about a lot of robotics actions in general, then you could either fine-tune that thing directly in that final simulation that matches the end application or even do evaluations to make sure that it's going to work.

Yeah. Right? So that gives you a different avenue than you could just collect a ton of demonstrations of a robot either working in that space or a person teleoperating the robot in that space, space, but simulation gives you a different angle to simulate those environments. Yeah, interesting. So is there a world where like, you know, you have your like general pre-trained robot, but then you always need like a realtism of a specific environment?

And could that get us basically to like a working like general robot faster by scaling this? Maybe. I think there's a couple different tech trees that people are working on. One is this notion of fully in context learning, right? Like maybe I've got a robotics foundation model, then I can kind of demonstrate the robot once how a task should be performed. And that's enough for the robot to figure out, like, stick that demonstration in the context, and now the robot can figure out that task. Another version is more this, you know, real to, what we're calling real to sim to real. There, maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this,

like, particular environment, right? Like this studio, like this table, like moving this microphone around. Yes. Then there's a version of that where you could come in here and take just a couple of casual videos with your phone or a couple images with your phone. use Atlas to reconstruct the space and now stage all kinds of robotics interactions in that, in like this studio space in particular. And if you could lower the barrier to doing that, like literally you could do it in five minutes to like take five photos,

upload it to Atlas, have a general simulation, maybe describe a natural language, what you think, what kind of task you want the robot to do, have an agent go build, you know, build that simulation for you, fine tune your thing, like RL, fine tune your general purpose robotics foundation model. And now you've got a robot maybe in the span of a couple minutes even that could come in and is perfectly adapted to this space. Yeah, like a very fast onboarding. Exactly. Yeah, that'd be great.

I'm kind of curious about all the way back in 2023, 2024, when the original version of SORA came out, opening eyes paper on it was called like video generation models as world simulators, which seems like very similar to the vision of world labs. And then, you know, they did SORA as that. They did SORA 2 basically as like a TikTok clone that didn't seem that interested in the like fundamental research direction of video models as world simulators. It seemed like a consumer product instead. And then they dropped SORA and now they're not working on world models at all.

So why do you think they decided to drop it? Well, I don't want to comment on Open AI in particular. I don't really know much about their internal strategy or discussions. But our thesis is that world models are this general horizontal platform that can be used to go after a lot of of different applications and there's going to be a ton of value unlocked there. I think there is maybe a sense of a bit of innovator's dilemma, right? Like you're sitting on LLMs and you've got this as a really powerful thing and you're leading the pack there.

It doesn't necessarily make sense to try to invest in that next thing. But for us, we think that there is another category of models out here. There's a ton of opportunities out here and we want to be the ones to go and grab them. Yeah. Do you think that there's like, I'm curious about the mass like consumer applications of world models or the first ones? I feel like VR or AR feels natural, and then video games feels very natural. I feel like there could be, like, we could start shifting in a direction where everyone's, like,

vibe coding video games and world models could be a part of that. Whereas right now we're just like vibe coding websites. We're already kind of doing that, right? Yeah, I think we're moving towards that. Do you see, like, world models being, like, that rapidly deployed so that everyone can kind of play around with them? Yeah, definitely. And that's what we want to do. So we want to get this, that's kind of like, fits into our thesis about these models,

as horizontal, general things. So if we want to expose these models to a lot of people, let them be consumable by humans and agents alike, and let them interoperate with other tools. So if we can provide a model that has all these general capabilities in simulation, in reconstruction, in generation, in explicit 3D, in 2D, and then we can have a coding agent go and stitch all these capabilities together.

Then we can let people really quickly iterate on really cool experiences. So I'm really excited to see, once we get this into people's hands, what kinds of cool experiences people will build with it. Right. Yeah, I'm also curious, like, how you think they'll naturally work with LLMs. So are we all going to basically be, like, adding another subscription to our AI stack? And, like, do you think, like, agents are very easily able to, like, work with world models in theory? Yeah, definitely. I don't see why not, right? Like, if you want, once you offer something on an API, like, you know, you get certain things in, certain things out, like, you can, they interrupt it with other pieces of software. Like, I think, I think we've always wanted to build these things for both humans and agents to consume. Yeah, I do wonder if like GTA 6 is just the last great piece of like human written software before like all future video games are kind of generated on the fly with stuff like Alice. I don't know.

I'm I disagree with that. I think there will always be like human in the loop with because video games are an art. I'm not saying there won't be humans in the loop, but I'm saying like fully hand stitched to like video game. Yeah, like you know, you won't need to like, you know, physically place like millions of different. assets in a game engine or anything. Yeah, but this is just the evolution of technology in general, right? As we see like, you know, from the early, like some of the early Disney animation films where they go and they hand paint every cell, like that was already a new type of technology that

enabled a new way to make movies and creative experiences at that time, right? Or some of the early Disney films figured out how to sync audio and video for the first time, and that was a really transformative experience. But, you know, new technologies come out and creative people always find really interesting and innovative ways to enable new experiences. So moving from, you know, live, like live action silent films to like animated, animated early Disney films to, you know, the kind of computer generated films, like, you know, the Pixar era of CG, you know, into AI.

I view these as just new creative tools and there will always be people, like, creative people behind the wheel that will be interested in creating new experiences. Yeah, I can imagine like a Pixar but in 3D, like in like a 3D navigable space sort of thing emerging. I think this is going to be really good for creative specifically. Yeah, I think so. And what we've wanted to build on all these things is this aspect of creative control, right? We never wanted to be building these generative slot machines where you just like pull the thing

and hope you get out a good generation. Or you don't feel like that generation was yours or that you controlled it or that you knew what you were getting out. Like we want to build these tools that have really deep control, right? Like whether it's 3D control, whether it's text control, that you feel like you're a director in the thing guiding this model and deciding exactly what it outputs rather than just like the model spit something out and maybe I can stitch it together after the fact. Yeah, totally. How does Atlas perform with like difficult physics stuff? Like, for example, if I wanted to do a simulation

of a black hole or something, like, would it be able to say like, oh, this is like a scenario where like normal object collisions don't really apply, you'll need to do like spaghettification or whatever? Yeah, black hole's a good one. I haven't tried that. Maybe I'll try this afternoon and tweet it. But I should do that. You should. You should. I'd actually really curious to hear. Yeah. My guess is it probably won't do something super exciting for black holes. We'll see. We'll see. Yeah, is the value. Just like hair or something.

Hair is pretty good. Like, I think one of the most amazing applications we got of these out of this thing, that was actually a bit of a surprise to us, how well it worked, were these bullet time shots. So we can do this thing or you can take like a couple regular iPhones and stick them on tripods, just like three iPhones. Take synchronized capture of some event happening. And then you can use Atlas to do these bullet time reframing shots, like the like freeze frame fly around. And this work better than any of us we're expecting. It's completely insane. So you've

one of our headliner examples there is a strawberry dropping into a bowl of oat milk. And then we can like, as soon as the strawberry hits the milk, the milk splashes up, we freeze time and like go in there and fly all around it. Like it's just incredible this works from just three input views. Yeah. I'm also curious about that like added capability of being able to like adjust where the camera is. Does that help like what part does that help with? I can imagine like that helps if trying to make like a movie or something. But yeah, how is that a useful capability? Yeah, I think it's very useful as a, so as, so I think there's a couple notions,

places where 3D is useful. One, like we already talked about the output side. Like sometimes you want to have that explicit 3D output to plug into a workflow or a game engine or something like that. But I think there's an underappreciated value of 3D as on the input side as well, right? Like even even in a situation like a VFX shot or something like that or the ultimate output is 2D, it's useful for the model to offer these two, these 3D control surfaces to let the human director figure out like what kind of model what kind of

seen the model should output right so in this case you know we can do this you know there you can control like precisely where does the camera move because I don't just want to write in text like pan in pan out truck left yeah I want to be able to grab the camera and make it like zoom exactly as I like and there's really no way to describe that explicitly in text you really want to be able to just like grab the camera and make it do exactly what you wanted so Atlas has camera control as a native 3D as a native input type right it's multimodal and one of those modalities is cameras yeah so if you think it wouldn't do a black hole correctly now like what more would be required in order to get it to handle all kinds of edge cases in physics oh i mean so i think

there's a couple different regimes of physics so like anything that's like roughly newtonian like we that was in the pre-training data so anything that's like roughly newtonian that doesn't work at is probably just a quick sfti away um if you want something that's sort of non-neutonian right like black hole or like, you know, microscopic stuff, once you're down in the quantum mechanics realm, I think then you'd have to rethink a bit, the architecture and the data side. But the black hole example is really interesting. I'll have to give this a try. Yeah, yeah.

I would definitely be very curious to see. Or like at a nanoscale. Yeah, that's probably not going to work. Yeah. Not yet, not yet. But one of our dreams we joked when starting the company is you want to build these 3D world models and how cool would it be to use a 3D world model to design a chip that we then use to train the next generation world model.

I was literally just thinking that. Yeah, yeah, yeah. Yeah. Is we're talking about quantum stuff? You have your like semiconductor. Also, to me, what feels very natural is that like the natural modality for navigating these and even prompting the world models is just going to be VR, basically. Yeah.

Because you can like, I guess, do W-A-S-D and then like move your mouse around to traverse a world. But I think eventually you will have to basically like. Well, not just eventually. We've had a, we've had marble in VR for like almost a year at this point. That is a really transformative experience if you guys haven't tried it. I haven't tried it. So then you really like stick on the headset and like I'm in this complete AI generated world that I created. Like it's quite, it's quite amazing.

Yes. Yeah, yeah. Yeah, I can't wait to see all of the amazing stuff that people are going to do with Atlas once it's fully out into the world. Yeah. Yeah. Justin, thanks so much for coming on MTS. Yeah, thanks for having me. Absolutely.

This was awesome. This was really awesome. Yeah. Thanks for listening to this episode of the A60s podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X, A16Z, and subscribe to our substack at A16Z.com. Thanks again for listening, and I'll see you in the next episode.

This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z. Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.