No Priors: Artificial Intelligence | Technology | Startups - What does AI-powered content creation look like? with Runway ML’s Cristobal Valenzuela

Episode Date: February 9, 2023

For a long time, AI-generated images and video felt like a fun toy. Cool, but not something that would bring value to professional content creators. But now we are at the exciting moment where machine learning tools have the power to unlock more creative ideas. This week on the podcast, Sarah Guo and Elad Gil talk to Cristobal Valenzuela, a technologist, artist and software developer. He’s also the CEO and co-founder of Runway, a web-based tool that allows creatives to use machine learning to generate and edit video. You've probably already seen Runway's work in action on the Late Show with Stephen Colbert and in the feature film Everything Everywhere All at Once. Show Links: Watch Cris Valenzuela’s 2018 thesis presentation at New York University’s ITP program. Read how Runway is used on the Late Show and in Everything Everywhere All at Once on the Runway Blog. Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @c_valenzuelab Show Notes:  [1:50] - Cris’s background and how he doesn’t see barriers between art and machine learning [6:46] - How Runway works as a tool [8:36] - The origins and early iterations of Runway [12:22] - Product sequencing and roadmapping in a fast growing space [15:43] - Runway as an applied research company [19:10] - Common pitfalls for founders to avoid [22:35] - How Runway structures teams for effective collaboration [24:22] - Learnings from how Runway built Greenscreen product [28:01] - Building a long-term and sustainable business [32:34] - Finding Product Market Fit [36:34] - The influence of AI tools in art as an artistic movement

I was speaking with this director who was working on a film, who was using runway. And he came up with this idea of when he was chatting with his editor. He was like, we should just runway that. I'm not constrained by the time and the cost. I'm constrained by whatever idea I think works the best. And that's just phenomenal, right? And so our goal, and I think our goal still is,

to build this kind of like autonomous like systems that don't engage in any sort of like relationship with humans or with creatives. On the contrary, it's like, you have humans coming up with great ideas and they want to express those ideas. How do you build systems that will help them get there really quick? This is the No Prior's podcast.

I'm Sarah Gua. I'm Aladgal. We invest in, advise, and help start technology companies. In this podcast, we're talking with a leading founder and researchers in AI about the biggest questions. We're thrilled to have Krista Ball Valenzuela on today's episode of No Pryors. He's the CEO and co-founder of RunwayML. Runway's a web-based tool that allows creatives to use machine learning to generate and edit video.

You've probably seen Runway's work in action. Visual effects editors have used Runway to create visuals on The Late Show with Stephen Colbert and the movie Everything Everywhere at Once. Chris, welcome to the podcast. Thank you for having me here. I'm super excited to chat with you. So can we start all the way back? I think you are the only person I know with degrees in economics, business, design, and then also went to art school.

I had that happen. And then how did you stick an interest in ML in there that became very real at some point? Yeah, that's an interesting question. I've always been very curious about just things in general. And so we're trying to find ways of channeling that curiosity. And I'm originally from Chile, and I study in Chile, a combination of, like, business and econ, and then went into design. And it was a very particular design kind of, like, program. I spent a lot of time with physical computing, which is, like, working with hardware with, like, electronics, mostly applied to design and, like, art.

And while I was doing that, I was also consulting. So it's kind of, like, for a moment, I thought I had, like, two lives. I was like doing art on the one end with like Arduino and electronics and on the other side I was like consulting for these banks, which is very like different. But I love it. I think it's perspectives and worldviews that are very opposite at the same time you gain from being at both. Long story, I kind of like falling in love or was

experimenting with early computer vision models in 2016, 15 and then went into Rabbit Hall, apply and got a scholarship at NYU and then spend like two years in art school. ITP, that's the name of the program. It's a very unique program for me. It was very fundamental kind of like piece in my career of like understanding how to breach business, design, art, and technology in a cohesive way.

Amazing. And now you have one life that combines those. But in the art side, how should I picture Arduino electronic art? Media arts probably is the best way of describing it. I think for me, media arts is a way of like, expressing a worldview using technology, like any other form of art. Like you're just kind of like experimenting and reflecting and expressing a worldview using a piece of like a tool.

And in this case, like it happens to be that we like to express it via like computers and software and writing software is a form of art and making hardware. It's also a form of art. One thing I remember early on my career when I was doubling between like art and business, I met this very famous Chilean artist, he's a photographer, and he was just, like, mentoring me and, like, we were chatting. And he was speaking with him, and he was like, Chris, this is the same world. We all live in the same world, right? It's the same, right?

We just build, like, silos and, like, arbitrary definitions of what is what. I think he just said it. I really, like, stuck with that. And I think that's how I like to league of the world. Like, it's just the same world. You can apply different points of views and perspective on that. We build arbitrary, like, definitions of, like, this is art. and that's like design and that's econ and that's business.

But I think true creativity and curiosity comes from just like looking at it as a whole and taking things that weren't supposed to be part of one thing and then adapting them. And sometimes it's hard because you need to learn things that you've never done before and it's uncomfortable and it's perhaps you feel like an impostery. Like you haven't been doing this. I've learned not to care, to be honest. I just like, you just drive back curiosity. Like you'll figure something out and I really like that.

That's super cool. Yeah. It seems like a lot of the history of Silicon Valley actually ties in really closely with art and the art scene. So if you go back to like the Stewart Brand World of the 70s or some of the early things that were being done on the Mac, or you even look at some of the people and technology where the art side of them is understated. You know, like Paul Graham obviously wrote a whole book on this, hackers and painters and is a painter himself. But there's people like Sep Comvar who started a company at the Google Bond and has done a lot of crypto-related things. He was a co-founder of Sello. And he's exhibited digital art at the MoMA as well. And so it just kind of feels like it's almost under-descent. discussed now in terms of this overlap between technology art and the two scenes, except for, you know, occasionally when people go to Burning Man or something, they bring it up. Other than that, it seems like it's very under-focused on. Yeah, I agree. And for me, has always, to be honest, a bit new. Like, I've been in New York, like, six years and a runway now is going to turn four years.

And I was also new to, like, just the tech world and SF, like, I've never been to SF like three years ago, right? So I'm relatively new to the space. But I think what I just, how we approach it was with that same level of curiosity. of like, I'm going to figure out, I'm going to learn about it. And I think that there's two, like, sites of that. The one is that it takes time for you to adapt to that because it's just new. Like everything else, you just need to understand it.

You need to understand the patterns of that, that subject, that domain, that area, right? But at the same time, I'm looking at it with fresh eyes, with things that have perhaps the ecosystem itself has considered, like, norms. I don't consider them norms. I just, I want to try new things, right? And I think that opens the door, again, to do new things. and experiment with new things. And that has, I think, being a consistent, like, path in both my career, but also in Runway as a whole.

But we look at things, we try to look at things with, like, very fresh eyes and, like, pretty much with, like, a first principle's kind of like mentality to it. It's like, okay, where are we doing this? Like, but really why? And they go to the basic aspects of it and then innovate. I think that's a lot of innovation comes basically from that way looking at the world. runway is, I think, a very creative shape of product. It's not the kind of product you can come up with if you're just,

like casting around for a good idea. It obviously comes from creativity and discovery. And maybe what you could do for our listeners actually just, can you explain how runway works as a tool and what people do with it to set context? Yeah, totally. Also, happy to set a bit of context of the company itself. So I think that better helps contextualize the product itself. The best way of describing runway, I would say,

it's to think about it as an, I'd apply eye research company. We do core fundamental research on neural networks for both content creation and video automation and journey models. We then transferred those models into an infrastructure, a system to deploy those algorithms and systems in safe ways and in ways that will make us build products that are useful for people, right? And those products can take different shapes and forms.

We have around 35 different what we call AI power tools or magic tools. And those tools help serve a wide spectrum of creative tasks from traditional like editing, editing, videos or just audio or images has been a very expensive, time-consuming, and sophisticated process. And so we build systems that help you do that. So we have tools like Greenscreen, for example, which a lot of broadcasting companies and film studios and post-production companies use to reduce the time of photoscoping, which is,

if you ever speak with a filmmaker, that's the one thing no one's to do and just no one's to do, but you have to do it. And so we basically just help you reduce that time. And we also have tools that help you ideate and design and craft. And we have a set of like suites for generative image editing, for generic video editing. So it's a, the best way perhaps to think about it is it's a creative collection of tools and systems that just help you augment your creativity in any way you want. From an origins perspective, like you had this thesis project, which were all of these creative tools. And it was really, I remember like watching the presentation as around accessibility of, you know, the increasing number of algorithms.

could help people in this sort of creation and editing process for different modalities. And when we met in 2019, you framed it quite differently as this kind of desktop app store for ML models. Can you talk about the iterations from that collection of algorithms you were experimenting with to like the app store idea to where runway is today? Yeah, totally. A lot has happened, I would say, over the last decade or so. When I started building runway, it was perhaps like the Alexnet, ImageNet kind of like

moment. There was image classification was the kind of the thing. like the big thing and the breakthrough, and a lot of interesting applications were coming out of that time, but it's still very early. Like, TensorFlow was just perhaps a year old. Pythorch might not even have been released at the time. I think Pythorch was 2016.

Guns were just, like, very early, early, early, like inception time. But what I kept saying was, like, there's this neural, like, aesthetic, this neural, like, capabilities that are impacting, not just, like, the visual world or, like, the perhaps industries and markets, like, solve driving cars that are using, a lot of these technologies and hardware, but the outputs are very interesting from a visual perspective, right?

There seems to be a correlation and an approximation towards the visual domain. And so I sort of just experimenting with what what does it actually mean, right? What do you mean by how do you experiment with this sophisticated algorithms that were very early that had all this like obscure CUDA dependencies and C++ like libraries that were just very research-centric because they were basically research, right? like core research. But it was just fascinated by the outputs of the research elements.

And at the same time, I mean, everything I would say that we consider like a baseline today wasn't really like there yet at the time. Things have progressed radically. The space has been growing exponentially. But systems and like software and obstructions to tap into that potential wasn't really there. So our first intuition and our first kind of like product experimentation was let's build like thin layer, right? Basically, let's Let's take this research set of models and the amount of models that are coming up. It's just like so interesting. Let's add a thin layer of accessibility to those models, specifically target and aim at

creatives, right? And so if you're a designer, a filmmaker, an art director, a copywriter, you might want to tap into some of these things. So you want to experiment with them, but they're just very hard to get started with. So we built what at the time was a model directory, it's an app store of models, right? We had around at some point like 400 different models. It was one of the first, like, I would say, model apps. I think there are a few out there now that you can tap into and use them.

These are very, very early. And we built a whole system around it. We build an SDK. We build systems for, like, deploying those models into real-time applications. So we build a restful API systems where you can use a model, train a model, and then deploy that model. And so people were building web apps and, like, interactive GPT1. Someone was training a model and, like, fine-tuning a GPT2 modeled on a specific corpus of data and then creating an API to build a text generation app.

And we had all these, like, very interesting layers of applications that, to be honest, for us, was just a way of learning, learning a lot about the space and a lot about what was visible, what was possible, who was kind of like interesting in like building more of this. And from there on, we've kind of like continuously iterating. We've learned a lot from that model registry or like model hub. We still use a lot of those in our infrastructure kind of like parts.

on the app, but also we gather a lot of insights on how to build these kind of systems in scalable ways. How did your technology stack or the approaches that you took transition over time? Because I think when I look at the evolution of the area, to your point, you know, a lot of people are doing like CNN and RN-based things in GANS and all the sort of early things in neural networks. And then, you know, the analogy may be, I know a lot of people who started companies right before AWS launched. And their whole like infrastructure stack got stuck on the past set of approaches, and then later, a subset of them transitioned on the AWS and a subset just continued with our own private clouds.

And I'm just sort of curious how you thought about it as, you know, obviously diffusion models, I think were invented around 2015, Transformers 2017, but it took a couple years for all this stuff to catch on. And so when did you start transitioning architectures or have you or how have you thought about this sort of whole evolution of the field relative to the tools that you provide and reinventing them over time and everything else? No, that's a great question. Something actually would think a lot about when you think about

product sequencing and roadmap, which is just, I would say, one of the most important aspects of product building. It's like, how do you sequence everything you have to do? And specifically in infrastructure, like what makes the most sense and how you spend time every single day, like, means a lot in a startup. I think for us was a few realizations, to be honest. One is that the moment something gets released, like let's say transformers or a particular piece of technology that you think would be interesting, it could be worth experimenting with. I think it takes a collective set of months, like 12, 24 months sometimes to understand the implications of that, right? And we've seen this with like language models. GPT3 has been around for some

time, but they took like a collective 24 months of like just tinkering and experimenting to truly understand like, okay, where can you go and what can you build and what's possible. So I think that we embedded that and we always keep that in mind. The second thing I would say is things are changing really fast, right? And so if you're thinking about building a long-term business and a long-term product, which we are, you always kept the desire of like, okay, what are long-term birds versus short-term bets? And I think a lot of building and software, engineering, and developing products is just saying no to a lot of things.

There's customers might want to ask you for to build something and could sound good. It could be revenue and some growth, but it actually might move you away from like a more consistent long-term plan. I think for us was a decision of those kind of like things. And then the third one I would say is the third component of how we think about that stack is really understanding our users, right? Who are we building for? And so early on, it was more a technical product. So you had to know Kuda and Docker containers and managing your Docker and Vita GPU cards and, like, you know, all this like sophistication that I think it's in some part natural when you things are so early because it's the only way of making sense.

And also you have to build more things. But for us, we've always been thinking about artists and filmmakers. and creative, it's hard, and really those things don't really matter that much. What matters is, like, your idea and how you execute that idea. And so from the stack perspective, we've iterated a lot on the kind of like back-inside of things, but from a user perspective, we iterate even more on how to present those things and what obstructions and metaphors you need to build to really aim to solve the things that you want to solve.

But, yeah, it's a fast-growing space, so there are a lot of things that are changing. In an area where the research, like, nobody can keep up with the papers, right, the progress is mind-blowing and has been. You referred to a runway as, I think, an applied research lab. Is that the right term? Yeah. Like, where do you decide, given the progress in the community, like, when you need to do in-house research and push the state of the art versus exploit what's out there?

Yeah. I guess going back to that sort of learnings early on, I think one thing that we realize is models on their own are not products, right? A model is a research component. And taking a model and productionizing that model, it's, It's a different problem that actually building one single model, right, or one single task or problem or improving a metric in a specific kind of direction. There's a lot of nuances of how that model will get deployed, it will get built, how users would interact with it. The unit economics of running this kind of like systems as well is very important, right?

So they have all these complexities. And as we started like leveraging perhaps open source solutions at a time or trying to build our own, we quickly realize that having control is like key. Like, you need to be sure that you can understand your stack and you can understand and know how to fix your stack, right? Because if things are changing really fast and you think about going in one particular direction, but it then happens to be the case that there's a breakthrough somewhere else, you need to react really fast, right? And you need to be able to incorporate that. And if you're just relying on third parties or some other solutions, then it might be very hard, right? And so for us, it was a survival realization that if we really want to make and move the standard of creative tools in the ways and vision that we had, we had to own our stack. And so we started building this research team, right?

And this research team has very deep, like, understandings and knowledges and perspectives on how to build models. And we've done this when we've collaborated and contributed to, like, breakthrough moments in, like, the creative eye space. But most importantly, we have these researchers working really closely with creatives. Half of our team have arts backgrounds, right, which is very unique. And we put a lot of emphasis on finding those very unique, like they're very hard to find folks that can speak both worlds. Like I just went back to the world analogy. And so in one single table, you can have a PhD scientist that's been contributing to

like fundamental research on the space, working really closely with someone who's working on video for 20 years, right? Who's been editing and post-producing films or content, right? And the things they learn from each other is just so, it's so unique. It's so radically different. And it helps inform how we build products, right? And so we don't treat research as a standalone kind of like department that comes every six months with here's a paper and just like do something with it.

We see it as an applied thing. It's like it's at the core of who we are and like how we drive the product forward. And it helps just drive the product in a different way. I think that the only thing I've learned is that building that muscle takes time, right? It's not that it's something you can just like, I'm going to hire a bunch of creatives and a bunch of researchers and just put them in a room and like you figure something out. It's it's a lot of learning and a lot of processes and like frameworks of how you make decisions, how you understand what's really possible versus what's visible. And there's a lot of just nuances of how to do that. It seems like there's a lot of founders now who come from the research community in the AI and ML world.

And, you know, you've navigated that extremely well in terms of saying, okay, let's be very product-centric and yet still capture the best of what new technology has to offer or new research has to offer. What do you think are common pitfalls that research-centric founders should avoid or things that they should think about more as they sort of start their own companies? Yeah. I think it's just phenomenal to see, like, that progression of more researchers that being perhaps in academia for too long progressing or moving into just the operational

world, like building products, I think it's a great realization of you're working or something for six, eight months a year. But you see something else in the world of someone using something very similar to what you just built and impacting the world in very meaningful ways. I think that's great to see people transitioning more. I think we need more of that.

I still think that there's a lot to be learned around the difference between a model and a product. And again, there's a lot of back and forth of how you embed models into usable products. And so coming up with training a model or improving some sort of quality of benchmark in some particular way. Even you have a very cool demo, it's a long way to go to actually build a business

and a reliable system that will continuously iterate over that. And so I think having that more product perspective is always just good. And releasing and working with real people as fast as you can, I think that's just key. I think a lot of researchers just assume how people work and how great is work and just, oh, we'll just do that. But the release might be very different. And so having tools being used by people is, I think, the best way of learning how to develop products.

Yeah. Are there specific areas of research that you're especially excited about when it comes to video or images right now? Yeah, for sure. I think, I mean, everything we've seen on the explosion of diffusion has been just so exciting to see. I think I'm particularly excited about multimodalities and, like, combining different input or, like, outputs in ways that they are yet to be explored. I think we're moving away from, like, very silo domains, so, like, someone who could be an NLP researcher. and computer vision researcher, right? I think we're, like, starting to see them gradually converge and mix.

And so building a diverse team that can understand, like, those multi-domains is really interesting. And I'm excited to see how that's going to play out in video and in images. And I like to think also of how you translate, again, and go back to product. I'm a bit of product obsessed, but how you translate that into products that are useful, right? I think a common natural evolution of just the creative stack or the creative software solutions are they tend to be very specific to domains of content. So you have a tool that specialize on image editing.

And then you have a tool that specialize on vector graphics. And you have a tool that specialize on motion graphics, which is different from video editing, which is different from like compositing, and you have all this like very sophisticated software stacks. And I think the very interesting aspect of what it will like to see and what will probably see more with multimodal systems is that you're able to merge all of those.

And what I really find interesting about that is that's how we humans think, right? You don't go to a movie and watch the video first and then you stop and you hear the audio and you stop and you read the subtitles. It's a combination of all of those things, right? And our art director thinks in all of those things at the shared time as well, right?

So having systems that can translate ideas and tax descriptions into videos and then having a conversation with what's the input of those videos into like audio. And then I think that's the kind of like creativity and set of tools that I'm really excited to discover and build. And I guess how do you organize your product efforts?

Because I think to your point, you have a really unique approach in terms of, you know, effectively turning research into products or being product-centric in terms of what you're asking from the research organization. Is there a specific structure? You know, for example, at one of the companies I started color, we basically would embed somebody with a very deep bioinformatics background with the systems team so that they basically inform that team around the needs of what they had and then the rest of the team would build it.

And it sounds like in your case, you have people who kind of are in both worlds. Is there a specific structure where you're like, I always put. three, you know, full-stack engineers with a researcher, with a product person, or the researcher is the product person, or how do you kind of approach all that? Yeah, we're a small team. We'd be consistently historically a small team. And until like two weeks ago, we didn't have a product person. Product was led by a combination of research, design, and engineering.

And I think that drives a lot of fundamentals of truly understanding the things that need to be explored. We've iterated a lot on building squads or building, like, teams, or having more autonomy, I think it really depends. I think you tend to have a different company every like four or five, six months. If you've successfully built stuff, it's a continuous like process. And the thing that worked when we were like five people sitting at a table, it's not going to really work when you're like 20 and you have new technologies

and system things available. And so I don't think there's one answer in particular. I think pretty much with how we think about product, we like to iterate a lot. Right now we're working all over with squats. And so we've come to a kind of like a place in time where the organization can have a bit more of like domain expertise and like instead of having like very journalize engineers, we tend to like more specialized a little more. So you can still jump and be and collaborate,

but you tend to have a bit of a focus of area. And we're iterated with that and seeing how that works. Maybe we can talk about an example of like what that iteration looks like. So you mentioned rotoscoping and like a green screening as like a like one of the magic tools that runway creates when we were building that feature like what was hard? What were the iteration processes like? Yeah. I think that green screen is a great example of how to build and how to deploy useful AI products at scale. When we were building that model directory and were just like

early stages of understanding limitations and capacities and directions, we quickly realized a type of like user that was coming for segmentation models, right? And at the time, we didn't have a green screen tool. It was just like an image segmentation model. And those folks were coming from a specific domain, and they're actually applying a model that was image-based into a video task. And so they were, like, exporting themselves with FFMPEC, creating these sequences of images to then render them back in video.

And they were like, why are you doing that? Why are you? What's going on? And the thing is, like, image models don't really work really well with, like, video. And so we started interviewing them, and we got to a point, I was like, wait, it seems like this could be something we could, like,

improve, and we're bringing our research team, so we started, like, iterating more on that. But no one ever asked for one one-click solution for Grinscreen, right? If you ask people what they wanted, they wanted a better alternative that was faster to create masks from their current stack, right? And they're probably using something like rotorbrush too, right?

So what I would really like would be like a better brush to just brush over my frames, right? And I think customers and people are really good at telling you, like, what their problems are. They're really hard at verbalizing, like, solutions. And so you aggregate that amount of data, you see what falls off your research, you see, you chat more with people,

and you start prototyping a lot, And then we came to, like, the realization that we could build, and we have the expertise to build a system that will help you automate that, right? And most literature around video objects lamentation, which is that in filmmaking is basically known as rotoscoping or green screen, right, was around, like, fully automated systems, right? You fit in a video, and the video automatically, like, understands, like, subjects, and then rotoscopes or segments, right?

One specific central object or two, let's say. But just a few minutes of chatting with a professional film. maker, you'll probably discover that that's rarely the case because the shots, the scenes, and the compositions, and the camera angles really depend. If you have a shot of 10 people, you might want to rotoscope the one on the left. Depending on your idea, right? It's a creative tool, right? So it should be general.

And so what we did was we've, instead of relying on fully automatic systems, we embedded a human in the loop kind of like component in it, right? And we thought it would be great. If before you start doing that, you can, the model. Like, you can tell what kind of, like, selections or areas of the video, and you can swim in and define you want, right? And that also really help us train the model, because we train a model on, we build a probabilistic model of, like, human simulated, like, human clicks on a mask, and the model was trained on that knowledge, right, from the very bare bones.

And that helped the product itself, because people were using that model in that particular way. And that decision was, I would say, like a combination of different things. It was some research knowledge and understanding of what was feasible. Can you build that cementation model? What data sets? And what do you need to do it? Who would be using it for? How are we going to test if it works?

And the first version of green screen was working at like four frames per second, right? It was like incredibly slow. It was like not as good as the one we have now, which is incredible. But it didn't matter. It was significantly better than anything else that was at the time. Right? And people were, like, scrambling to use it just because it proved to be a percentage of amount better than anything out there, right? And people were hacking things and we're trying to, like, incorporate it.

It's like, great. That means that, like, that you hit something. And then we start iterating a lot. And so we keep iterating a lot on it. But the fundamental piece of how we build product is still pretty much similar to that. Very cool. Let's zoom out and talk about runway as a business. So, as you said, now you're very intent on building, like, a long-term door.

business, who uses and pays for runway today? We're devoted to like storytelling and like creative exploration and ideation. And that's a wide spectrum of people where you can consider work in the storytelling business, right? On the one end, you have professional, really professional people that have been doing this for years, right? Folks working in post-production agencies, BFX agencies, broadcasting companies that are creating video as their main business.

Like this is basically what you do, right? its entertainment is sometimes sports. You know, that's kind of counterintuitive because like one of the sort of beliefs of many people who look at the research, which is fast progressing, is like you can't get the quality level for like the sort of highest production value type assets with today's research. So it's really interesting that like you're talking about VFX studios and sort of that type of content. Yeah.

I think I think that realization for us is like what the goal is, right? If you're trying to automate the entire process of like the whole end to end to system of making a movie, yeah, like, we're not there, right? We're very far from that. There's a lot of things that have to be developed, that have to be the, like, research and kind of like understood and tested. But going back to the green screen, if you look into the processes and the nuances of how video is created and you look at the inefficiencies of how people are doing it right now,

and you offer these people like 100, even like 10% or 20% or like whatever percentage of like speed and cost reduction. It's just so radically better, right? And it's radically better for two reasons. Of course, it has helped reduce the cost of, like, you can do things faster, so it's just easier. At the same time, you can explore creatively more, right? And this happens a lot. I was speaking with this director who was working on a film and was using runway.

And he came up with his idea of, like, when he was chatting with his editor, he was like, we should just runway that, right? Just runaway the thing that you want to, like, do. And before runwaying something, they had to. marry themselves or like just lock one specific idea, right? Because if we try to do two other things, it's going to take us too much time and we just don't kind of afford that. Like every creative is always on a deadline.

It's very waterfall era, right? You must choose a direction and do the whole thing. Exactly. And now he was telling me, like, now I can do the three, right? I can just see the three and pick the one that I like the most, right? I'm not constrained by the time and the cost. I'm constrained by whatever idea I think works the best. And that's just phenomenal, right?

And so our goal, and I think our goal still is. goal still is. It's not to build this kind of like autonomous systems that don't engage in any sort of like relationship with humans or with creatives. On the contrary, it's like you have humans coming up with great ideas and they want to express those ideas. How do you build systems that will help them get there really quick, right? And sometimes what you need is to get 80% there, 90% there. And in research going from 80% to 100% is really hard. I think that you've seen that in like autonomous vehicles where like it's always like two years ahead and always 80% but like the 20, 10% is just really hard.

But it's really hard in that domain because if there's a 1% failure, someone might die, right? In creative domains, it's not the case. Like, even if you're 80% there, the 20%, like, sure, I mean, it could worry about it. I can improve it. I can, like, find ways of work with that. But you've made an incredible progress, right?

From that perspective. I think that's actually an interesting, like, filter for what domains are interesting for applied research today, like areas where there's built-in tolerance for, you know, lower levels of accuracy is one way to look at it. Yeah, and you always integrate with, there's ways of, like, combining existing tools, right? So for ROTUSCOM, for example, you can get 80% there. And then if you're a professional filmmaker working on Nuke or Flame, you can do the 20% in that stack, right? But you still save yourself, like, days of work, right?

So it's still better than anything you were using before, right? So it depends a lot. And I think over time, more models will get to, like, higher numbers and we'll have higher outputs. But there's a lot yet to be developed. And I think we're still scratching the surface of what's coming. What was the moment? You know, you mentioned there was an evolution, both in terms of the number of tools you provided as well as their relative quality in terms of, you know, 80% versus more or less and things like that.

Was there a specific moment where you really felt that you had product market fit or where you felt that, okay, this is something a lot of people want and they want to use? Is that, was it immediate? Was it after a specific tool came out? Like when was that moment for you? Yeah. I like to think of product market feed as a spectrum of like you have either really strong product market fit or weak product market fit. And as you build new products and your research, you're always seeking to be very on the strong side of things, of course.

I think for us, there are a few factors that we've kind of like realized that what we were building was beyond just like a niche. Because I think we started with a very niche, like, audience and everyone like dismissed a little bit of what we're doing like as toys. She's just sort of like art student building like some toys. And I think you shouldn't dismiss toys. Toys are like very interesting to learn a lot. And I've learned that over time. But it's by the time where you're building those, of course, it's just like you're focusing on the output and they're glitchy and they're like abstract

and it's just weird. I can make sense of it. And it's only 128 by 128 pixels. Exactly. I was actually, I remember like we had a version of a fairly early, early like gun system that did text to image translation. We actually still have the demo online. And the output was like this 120 pixels like exactly you were saying images that were

just blurry. It looked like abstract paintings, right? You just like, I mean, you type, I don't know, a blue ocean and we get like a blue form with something. So we'll see if you close your eyes, like 10 meters away, maybe. And you saw beauty or the future. I really like it. But at the same time, I remember like showing it to advertisers. And like I went to like this, like, say to a meeting at this top agency in New York. And it was like, here, guys, here's the thing you will be using to work, right? And they were like, Chris, this is a toy, like great. I mean, fascinating technology, whatever, but like, we have work to do. Come on, like, move on, right?

And I think the main mistake for me was, like, you're looking at the singular moment in time of that technology. You should really be looking at the rate of progress, right? That thing that I can type a word and stand an image wasn't feasible a year ago, just didn't excess, right? Now we have this. So just compound and imagine where we'll be in like four or five, six years, right? But the thing is that's really hard because you can't imagine it.

And I remember people at the time, like when I show some of those demos, specifically for genetic models, people were asking me like, Hey Chris, how are you collaging these images? You're taking existing images and you're pasting them together, right? And it's like, no, this is, you're generating them. This model has learned patterns around, for sure, a data set,

and you're then generating them on the fly, but these images don't really exist. Just don't exist. And so there's a lot of, I think, mental models that need to be adjusted to really understand it and we've been adjusting that those mental models. And from a product perspective and from a product perspective, product market feed perspective, I think there's the right moment for the market to use technology. And I think that moment has mature and we've seen it more as more people have been

exposed to the models and the potential of them. And for us, it's still like, there's a lot to build and to develop and to kind of like improve. But a few realizations were like when people were starting using runways a bird, right? You just runway that. Okay, that means something. Then you start seeing people just like creating tutorials and like speaking about the product online, right? with no, we don't, for a long time, never had a, like, a marketing team or a content strategy team. Like, everything was just basically people making things and then sharing them online. I think that really drives, I would say, really seems, okay, we're into something. Like, people are using this.

Every day, they're coming and they're sharing with their friends, right? And they're thinking about it every day. And they were like, I remember an artist and a person who early runway adopter, which just fall in a song we love with a product, he sent me like, he painted a picture, like, and he just sent me the picture to my home. And it's like, here's like, just once you have the first piece I ever made with AI. And that was like 2018. What was it?

A cat? No, it was an abstract painting, where it was like he generated something that was very abstract, right? And then he painted it on a canvas and then he used like mixed techniques to just like improve some sounds and like change some colors. It was very new and novel at the time. It was like, wow, that's just, I don't know, interesting and fascinating. One thing I'd love to get your perspective on simply because you have such a unique mix of background and skills and customers and everything else is, you know, there's this emerging debate in the art world about the role of AI and art. And I think if you go back through art history, there's always been ongoing questions and

contentious, not just around technology and art, but the role of an artist relative to the art they create. And I think like the sort of old school canonical example was Marcel Duchamp signing the urinal with Armut. And I think it was called the fountain or something, right? It was a piece that he submitted and it got refused and it created a bunch of sort of scandal at the time or, you know, Andy Warhol had like the factory and other people would assemble a lot of the art actually with sort of him overseeing it. And so it seems like there's been a long history of sort of different approaches to art that at the time seemed very controversial. And now you're just like, yeah, of course, that's how you do things or how things were done. What do you think

about the debates right now in terms of art and AI and, you know, what do you think of the important threads that people are talking about? And what do you think of the areas that in 10 or 20 years people look back and say, yes, it was just part of sort of this art history debate. But it, in hindsight, wasn't really that important. Yeah. I like to think a lot about what, I guess, previous moments in history and time, as you were referring before, that has thought of something about how to both understand art and look at the tools that we use for art.

For me, art is the way of looking at the world and expressing that view of the world in a particular way. And an artist's role, I think, should be to explore and experiment with different mediums that would allow you to express that in the best way you think possible, right? And so people explain it with different techniques and different systems and different, like, structures and pigments and, like, tools themselves, right? Even before, like, Duchamp and even before, like, Warhol, you had previous moments in times where technical revolutions enabled people to look at their world in very different ways and then express those views of the world in very different ways to whether it was feasible at the time or possible at the time. And an example I go back to often as this idea of, in the 1700s, like before even painting was a massive thing that you can do in any condition, situation, like, location, painting was the realm of these very sophisticated painters that were painting in studios, right? Painting was the realm of people who can afford and were able to understand and master the techniques of the masters, right?

And more importantly, from a tool's perspective, it was really hard to get pigments, right? It's a very practical thing, but like, you couldn't just, pigments didn't access, but you could just go to a store and get red, white, yellow, and like, I'll print something and I have a canvas. The way you mixed pigments was this very sophisticated thing, where you had to hire a master that knew, like, these obscure techniques, and you were, like, measuring them, and then you store them in, like, this sophisticated bladders, and you seal them, and it was an incredible, like,

complex and expensive process. And then someone was like, hey, we should just, like, build a tube and then, like, have this. and carry it around, right? And maybe it's easier. And it was, and it was a very radical innovation, very simple at the time, it's very simple for us now, but that would allow it was like for a whole new generation

of artists to look at art and be like, great, I want to take this painting, and there's a mountain that I really like there. You don't know what's it when the canvas. I'm gonna paint in plan air, which is that, which is a thing. You paint in plan air, right? You're painting in air, in the wild. you're able to look at the world and the sky and you're able to like quickly brush the light,

right? And just being outside of the studio was just not physical. It was just 10, like before that and then gave birth to impressionism, right? And impression was like a whole revolution. Like impression was not really well received because it's like, hey, this is, this is not art. These are just brushes of like things. Like they're not, I mean, no, right? And then impressionism really started to big up. People like started to really understand the medium. And then it evolved. it continues evolve and evolve, right? And you find similar moments in time where the paint of Betelphor becomes relevant

and, like, photographer for me was a very similar one, right? And then cinema for sure. And then the digital world, the transition to like film to find out to digital is another one. And every single step of the way you have artists experimenting with the technology and using them to put a perspective of the world.

I think right now where we're seeing right now with AI, I like to think of two AI, art waves. There was like the 2015 to 2022 where like the VQ gun and like the early gun experimenters and there was a lot of artists experimenting with it. And then now the diffusion kind of like and the transformers kind of like world has enabled a whole new wave of people to expand with it. But at both the first wave and now this particular way, I think we're in the paint tubes moment, right? Where people are taking it and are using it to express something, right? To think of the world and then type that on the world and generate something, right?

I think the artist still remains pretty much at the center because that's what really art is about. And these are just tools, right? It's hard to understand them first because you're just new, like every new piece of, like, yes. And I think, well, I guess to your point, like what are we going to be asking ourselves in like 10, 20 years, 30 years?

I think it's the realization that we'll look back and we'll look at this moment as in like, yeah, it meant it was a natural transition and we needed it. It allowed us to do so many things that we just couldn't thought of before. Great that we had it. And I think we're still early at realizing that. Yeah, it seems like an extremely exciting time from a arts perspective.

And I remember in 2018, the first sort of GAN-based artwork sold at auction. I don't know if it's Christie's or Sotheby's or something. And then it almost felt like everybody got really excited and then there was silence, you know, until this next wave of like diffusion-based models and, you know, everything else. Is there anything that you think is needed to encourage that art scene or do you think it's literally just time now because we have the tools and we have really interesting things happening? Or do you need to be able to print the artist a certain way? I'm just sort of curious, like, what are the obstacles for this becoming a bona fide sort of fine arts moment or movement?

I think it's convenience, and it needs to be, like, accessible and usable and understandable by people. I think in the analogy of the paint tubes, we're not yet at the stage where you can just buy a paint tube and use it. We're still, like, in the stage of where transitioning from, like, this, sophisticated pigments to, like, some sort of like a paint tube, right? But early, like, Gans and like Roby Barat, which I was thinking, I think it's the artist you mentioned that was behind all of the early works on that auction on 20, I think it was 2017-218. It was very hard to just get started with a model, right? It was a very sophisticated process. And now you can just do it from your phone, right? You're coming closer to the like that.

We're putting the cab on the painter, right? Almost there. I think it was just a rate of, like the rate of progress and the expectation of it will become easier and better. And I would say two things. These models and the systems need to become really expressible and controllable, which is somehow the way I like to think about alignment is like you have an intention and you want to express that intention in a very controllable way, right? These models are yet not controllable, right?

Not exactly as we would like them to be, right? And the reason why because of that is a pursuit very early. Like there's a lot that they had to be invented to control them and have them be very expressive and have them work in the way that you really want them to work. So the art movements that you mention, there are art movements, but they're also like cultural movements fundamentally, right? And like we talked about the tools because you're a tool maker and it's like, we got to have to paint too.

But if you take impressionism or futurism or something, like it's also like it had an aesthetic, it had tools, but it was also very Italian at a certain point in time. And it was about optimism about like urbanism and cars and everything. Are there like schools or philosophies or scenes that you think are like worth paying attention to right now. Yeah, I mean, I'm biased because it's been part of a particular scene in New York, the media art scene that I particularly think will heavily, has heavily influenced a lot, other of these learnings in the stage.

To your point, I think every art movement sits in a particular cultural context and historical context and futurism was like in particular moment in time about like technology and like also fascism was around and there's a lot of like things and you just look at the world in like this particular way and you're expressing this particular way and there's an aesthetic and a line and a system. that if you look back, it was like, oh, of course. And cinema was the same. Like, movies early on were a way of perceiving the world and expressing them

because it was in a very contextual, like, historical moment in time. I think for me, if you apply that same kind of like principle, now, I would tend to look a lot at the weirdos of tech, right? People who are at the fringes, people who have always considered like, oh, you're just towing around. This is like an experiment. Like, there's a lot of creative coding communities. and people from experimenting with COESA's art,

there's like a lot of conferences and these communities of people, baby castles in New York or WordHack or like AIO or you have all this, they're just like interesting, just art. It's just very, very, very highly creative and niche. I think those folks will define a lot of what we'll see it next in tech. Yeah, well, New York or otherwise, weirdos are a pretty good bet in general

for people who come from the technology world. Yeah. Chris, this has been amazing. That's all we have time for today. are looking forward towards unlocked creativity and the next paint tubes. Thank you so much for joining us on the podcast. Of course.

Thank you for having me here. It was great. Thank you for listening to this week's episode of No Priors. Follow No Priors for a new guest each week and let us know online what you think and who an AI you want to hear from. You can keep in touch with me and conviction

by following at Serenormis. You can follow me on Twitter at Alad Gill. Thanks for listening. No Priors is produced in partnership with Pod People. Special thanks to our team, Cynthia Galdaya and Pranav Reddy, and the production team at Pod People. Alex Vigmanis, Matt Saav, Amy Machado, Ashton, Ashton, Danielle Roth, Carter, and Billy Libby. Also, our parents, our children, the Academy, GovGBT, and our future AGI overlords.