This Week in Startups - Giving NPCs the power of AI with Convai's Purnendu Mukherjee | E1839

Episode Date: November 2, 2023

This Week in Startups is brought to you by… The Equinix Startup program offers provides a hybrid infrastructure solution for startups, including up to $100K in credits and personalized consultations and guidance from the Equinix team. Go to https://equinixstartups.com to apply today. Fitbod. Tired of doing the same workouts at the gym? Fitbod will build you personalized workouts that help you progress with every set. Get 25% off your subscription or try out the app for FREE when you sign up now at fitbod.me/TWIST. LinkedIn Marketing. To redeem a $100 LinkedIn ad credit and launch your first campaign, go to linkedin.com/thisweekinstartups Today’s show: Convai CEO Purnendu Mukherjee joins Jason to discuss creating AI-powered NPCs using Convai (6:05), the birth of generative AI (22:17), open-source vs. closed AI (36:01), and much more! * Time stamps: (0:00) Convai CEO Purnendu Mukherjee joins Jason (3:54) The motivation behind Convai (6:05) Purnendu breaks down creating an NPC character using Convai (10:16) Equinix - Join the Equinix Startup Program for up to $100K in credits and much more at https://equinixstartups.com (11:36) Demo of character creation using Convai's unique toolset (17:18) Integrating AI Characters into Game Worlds with Convai (20:50) Fitbod - Get 25% off at https://fitbod.me/twist (22:17) Convai’s business model and the issue game developers have with moving from flat rate to usage-based charges (25:00) Purnendu reflects on his time at Nvidia (29:57) The birth of the generative AI boom (34:30) LinkedIn Marketing - Get a $100 LinkedIn ad credit at https://linkedin.com/thisweekinstartups (36:01) Navigating the Future: AI Regulation & open-source vs. closed AI (40:11) The countdown to self-creating AI (43:13) Purnendu breaks down data collected for AI and the uses for an AI-enabled Amazon Mechanical Turk Check out Convai: https://www.convai.com Follow Purnendu: https://twitter.com/purn3ndu https://www.linkedin.com/in/purn3ndu * Read LAUNCH Fund 4 Deal Memo: https://www.launch.co/four Apply for Funding: https://www.launch.co/apply Buy ANGEL: https://www.angelthebook.com Great 2023 interviews: Steve Huffman, Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland Check out Jason’s suite of newsletters: https://substack.com/@calacanis * Follow Jason: Twitter: https://twitter.com/jason Instagram: https://www.instagram.com/jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis * Follow TWiST: Substack: https://twistartups.substack.com Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin * Subscribe to the Founder University Podcast: https://www.founder.university/podcast

Open source can win if there is an incentive system for people to contribute their niche data. Ah, interesting. So if it has a license to use Falcon or a license to use this open source model, and then if I put my data in, I own Getty archives. I say, I'll put my Getty in there, but I want, you know, 3% of the pool or whatever it happens to me. So that requires some centralization, which, you know, obviously. Oh, okay. Not necessarily. So, and this, this is, Don't say blockchain. Don't say blockchain. I was going to say blockchain. Oh, no.

Are you telling me that after all this crypto drama, it's coming back? A hundred percent. The blockchain is coming back. Oh, God, no. This week in Startups is brought to you by The Equinix Startup Program provides hybrid infrastructure solutions for startups, including up to $100,000 in credits and personalized consultations and guidance from the Equinix team. Go to Equinix Startups.com.

to apply today. FitBod. Tired of doing the same workouts at the gym? FitBod will build you personalized workouts that help you progress with every set. Get 25% off your subscription or try out the app for free when you sign up now at FitBod.me slash twist and LinkedIn marketing. To redeem a free $100 LinkedIn ad credit and launch your first campaign, go to LinkedIn.com slash this week in startups. All right, everybody, welcome back to this week in startups. Everybody knows, Jenner of AI is having a massive impact on the creative industries. Chat GPT, amazing to, you know, brainstorm or workshop, some dialogue, maybe work on your screenplay.

Mid Journey, Dolly, generating amazing images from text that you can then use for a logo for your company or in a magazine or a website. And of course, we're seeing all these AI video editing tools to make clips from your podcast or your web show, put captions on them, make the host of the show speak a different language. It is amazing how fast this has happened. And one area where AI is making a huge impact, we all know, is gaming. Making games is incredibly, incredibly expensive. And Convey is bringing multiple tools together to create a platform where game developers can create AI-powered NPCs. What's an NPC?

These are non-playing characters. It's used as kind of a joke in online culture. If you hear somebody say he's an NPC, she's an NPC, basically means you're a mid, which is another cultural reference

and a way to say somebody's like an average person or unimportant. So NPC, non-playing character. If you were playing Grand Theft Auto, that might be the shopkeeper, it might be the bartender, or if you're on TikTok,

it might be Pinky. Now, of course... Ice is so good. Ice cream's so good. Pinky, who slays is not an NPC. She's pretending to be an NPC because NPCs are boring and do repetitive things over and over again. So the fact that she can mock them and make hundreds of thousands of dollars a year doing TikTok Live and people giving her all kinds of virtual gifts shows you how popular the concept of an NPC is in popular culture. Well, my guest today is the co-founder and CEO of Convei. And it's

spelled C-O-N-V-A-I if you're looking for it. It's Convey, as in Convey something. The CEO of the company and co-founder is Pernendu Mukherji. Did I get it right? Mukherjee. That's right. Incredible. See, this is the thing.

Pronndu. The harder the name is, like India names, Sri Lankan names, Greek names. I get them. And then sometimes like weird English names, I get them wrong. So anyway, welcome to the program. You heard my little preamble there. How did you come up with the idea to make tools to make NPCs? Were you in the video game industry before?

How did you come up with this opportunity? Yeah, absolutely. I mean, the thing is, a multiple things came together, which made it so compelling that I was like, I have to do this or I'll be recruiting. And those multiple threads, if I were to like say the most important ones were I've been thinking about how the human mind works since I was a kid and was pretty fascinated of how our mind works and I've been spending a lot of time with that

and then I did all my thesis work on question answering systems, natural language processing and after my graduation I started at Nvidia as a deep learning engineer and did similar work there primarily focusing on generative AI, large language models, and that's where I face the problem myself. You know, where one thing is like you were working on text-based systems, right? But when you want to create a character that has a consistent backstory, that has some expertise,

you want to add voice, you know, like have voice-to-voice conversation, and then you want to connect it with an avatar or a non-playing character, let's say, become the mind of that character, right? That's a challenging task, especially you have to do that in low latency with different voices and different, you're basically trying to replicate a mind that can be almost human-like. So now the problem that I faced and my passion and interest and what I have been doing,

I was also a competitive gamer before, you know, as growing up. So like all of those things came together and I was like, yeah, my life has been gearing up towards this. I have to do it. So last year I started conveyor. This was before the chat GPT boom thing, but it started in April.

So interestingly, the NPCs are going to become more like main characters because of your work if you're successful. So in a way, you're the savior of NPCs. You could say that. Making them more human-like, more less like the NPC meme.

More human than human, perhaps. So show, yeah, show us how you make one and then, yeah, how they act in the game. Because the kind of joke of NPCs is they, usually they have like a script that, you know, as pinky, doll was showing us here. It's usually a script of like six or seven items.

And yeah, after you go to the six or seven items, it gets repetitive and funny. Absolutely, yeah. I could walk you through that. I actually have the whole process. Maybe I give a brief on that and then we create one. Yeah, sure. Okay, so I actually have a...

You're going to share your screen. And for people listening, we'll describe what's happening on the screen here because this is, again, a cool to make characters in a video game. And these NPCs are typically the non-playing characters are typically boring. So make them a little less boring is the goal, yeah?

Yes. So what I'll do here is I'll present this particular slide in all the steps that happens. So this is all the step all the way from creating the character primarily on top of an LLM to adding voice, to adding avatar, to putting into the game engines. So first thing, when you on top of the character, you give them a backstory, they will have personalities, they will have emotions, they can even have memories. you can put guard drills around them in terms of things that they can or cannot say. They can speak different languages. You can add vector databases

which augments the knowledge and make them expert on certain topic. You can follow a particular narrative. And once you've added all of these over text, which we will see right after, that you can add voices on top of them so they can listen to you, understand all different kind of language,

speak different languages, and speak them with emotions. As you see, my voice is having internet, and whatnot. Once you have this whole voice-to-voice character, you connect them with the avatars, right? Whatever kind of avatar, you can use them with metahumans, with various different kind of avatar systems that are there. We are avatar agnostic, but we support all kinds, and we also support all kinds of engines, right? So you can integrate them with Unreal Engine Unity. Anywhere you can create 3D words, you can embody these characters with the AI that you just created

with the character, right? So lip sync will work, facial expressions, gestures, work. And then finally, once these are like avaturized, you have provided the mind, it means that they can like perceive the environment around, you know, carry out different actions. You know, they will move around the world. You know, you can say, hey, go pick up my, pick up something and they will bring it to you. You know, like, so they can do all of these things, right? So we solve the whole pipeline end to end. Pause there on that last part before you go further. On that last part, you're going to have them go interact with the environment. Yeah. Which means you don't know.

exactly what their response is going to be because I as the player could tell them, hey, I want you to go find me all of the loot and bring it back and they go do that for you. Okay, fine. Yeah, yeah, yeah. But you could say, hey, I want you to go and insult everybody else playing the game and, you know, in the style of... You could do that too. You know,

Jeffrey Ross, the famous insult comedian. Yeah. And I want you to use a Trump voice or something and go insult everybody. So now you've essentially created chaos inside the game. So I guess, you're you're unleashing a little bit of craziness, no? That is true, but like, it's up to the game designers that, you know, will you allow them to have different voices? Usually they will probably stick to a singular voice so that it stays consistent with

the character, but maybe somebody might design something like that, you know, to try that. But it's going to hit some language model. And as we've seen, you can kind of convince language models to change their mind about things. So things are going to get a little weird. Yeah, things will. But I think game designers... In a good way, hopefully, yeah.

In a good way. But the thing is, this is a developer tool, and the game designers are going to spend their time trying to perfect the character so that it stays in character. Somebody may want something like that. But most cases, 99.9% of the cases, they would want a consistent character. But yes, they will have the dynamism where you can tell them,

hey, go tell this thing to this friend of mine on the other side of the map. And they will go running inside the world and tell them whatever message you. send them. Yeah. Awesome. Cloud computing has revolutionized startups over the past decade. We all know that. But reality is a fully cloud-based solution is not right for every startup. Sometimes you need a hybrid solution. Like, let's say you're working with sensitive data that you just can't put in some random cloud, right? Or maybe you need to connect multiple cloud providers at once. Or maybe you just

want a more cost-efficient solution, right? Because sometimes these cloud bills can get crazy. If that's the case, you need to check out Equinix. Equinix Medal will give you direct access to physical servers, but you still get the benefits of the cloud. So there's no need to rack your own servers. Equinix provides on-demand infrastructure in over 25 major cities, and here's the best part. They have an amazing startup program. The Equinic startup program offers personalized consultations, guidance from the Equinx team, and wait for it, $100,000 in credits. That's not a typo. So here's what I want you to do. Go to Equinix Startups.com to apply.

And when you apply, James from Equinix will reach out to you directly. That's how serious they take the startup community. That's EQU-I-N-I-X startups.com. EQU-I-N-I-X startups.com. Equinix startups.com. Get a call from James. Well, I mean, if you asked it, I want you to speak French and I want you to bark like an English bulldog, it could go find in the language model

those attributes and start talking French and barking like a bulldog. Yeah, you can, you can. So if you want to try that, we actually have support for multiple languages where if you speak French, you can speak French. So here we are, we're in the interface, and this looks like a piece of software,

you got a bunch of different characters. We pick a character. We can figure a character or we can create a new one. Okay, so create a new one. Let's say, maybe I can create you, Jason. Okay. So I put Jason, character's voice. I'll just choose a male voice for now.

But we can even add your voice later on. How do you want to say it? Can you use AI to make character voices? Like, could you say, give me a pirate and even if you didn't have pirate already in there, we'd go find pirate? Yeah, we can. We can add voices in this as well. But yeah, let's go with the. default one. Maybe I'll just say, Jason

L. O'Connis is the host of All In Podcast. And I'll see what GPD3 generates. Oh, there you go. Okay. Put it in there, hey, this is

All In Podcast where you've mastered the art of captivating your audience with your insightful and thoughtful, provoking discussions, your existence is anchored in the world of tech. You just pasted that in? No, I just generated that. Oh, you just generated it. Yeah, yeah.

So based on Jason Calacanus as a host of All In, it generated the paragraph. Fantastic. Yeah, yeah, yeah. So we'll create that. And once the character is created, it basically gives a placeholder avatar, but we can actually have your own metahuman character inside the game engine when we take it there that will look like you, feel like you, and all that. So we have like all of these options to craft it further.

So like you were saying, it currently speaks English. So let's just start with it. Hey, how are you? Hey there, I'm doing great. Thanks for asking. How about you? Almost sounds like you. Almost. If I had a very deep voice, it would sound a little like me. Yeah. Yeah. What do you do? I'm the host of the All In Podcasts, where I discuss tech and investment topics with industry experts.

I'm also a venture capitalist and entrepreneur, so I'm always looking for new opportunities to invest in. Pretty accurate, yeah. True. True. I think it's all true. You have this knowledge bank. Let's say you have your own set of files. Let's say all the all-in podcast stuff that's tagged by your name.

You can upload that and they'll know everything about... You can upload every episode or something. Yeah, sure. Exactly. And so this is the state of mind, like how you are feeling as I'm asking this. So let's say, and this will drive the facial expressions, the tone of the voice and all of that. So let's say I say, my dog passed away. which is not true, but I'm just testing it.

That's terrible. I feel so bad. So let's see if it changes. The emotion. I'm so sorry to hear that. Losing a pet can be really difficult. Is there anything I can do to help? So you see it kind of goes into the grief mode.

Yeah. So you're showing the emotion wheel. I forgot what they call that emotional wheel. Yeah. It's called Plotchicks wheel. Plot chick's wheel. What? Chicks wheel.

God. Neat. So it shows you the emotional. state. Yeah. Wow, that's interesting. You can also craft the, um, the, uh, you know, like how is your personality? Are you an agreeable person or? Yeah, you can, it's giving you, like, a little slider and you can say how openness, meticulous, extroversion, agreeableness, sensitivity, like kind of a Myers-Briggs. And you can just put me as like a hundred percent extrovert, whatever, medium, emotional, whatever.

And it will just kind of build my. psychographic there, my personality trait, yeah. Yes, yes, yes. So, exactly. This is the big five personality traits. We initially thought of doing Myers-Briggs, but it seemed like this would be more appropriate to craft this. Yeah, and then there is like you can have long-term memory settings that you can turn on.

Oh, interesting. You can have, you can have guardrail. So this is the part it protects from saying certain things. So let's say you can hold this. Yeah. So it says limit responses, stick to character description and knowledge bank only. and then all the other end is talk about other areas. So you can just go crazy or you can go somewhere in the middle.

So that's just how on script the person is going to be. So you're basically creating Westworld here. When Elon or somebody figures out robots, you can put this into them and see if they'll stay on script and always stay a samurai or always stay a cowboy. Interesting. Exactly. And then we have this narrative design aspect,

which is basically like, you know, you can create a narrative. one after the other, where this thing could be connected to the next thing, and you can craft a story, basically. Wow. Yeah, so maybe I can show you an existing one.

So let's say. So you're making, what do they call that, a story tree or something? Yeah, like a dialogue tree or story tree. However, the difference from the traditional ones would be that this is going to be open-ended. So it will still stay within the story. It will try to follow the story, right, or whatever your objective that you have provided it,

and yet be open-ended, right? So that's another feature, so that they can play the role that you ask them to, right? So once you have done all of these things, right, you take this character ID, which is the mind of the character that once you've crafted it, now you take it to the game engine, right? So maybe I'll show you a quick one.

So let's say, I'll go to this. So you've got a token, a unique token that represents that character, That'd be very hard to guess. You plug it into the Unreal engine. Exactly. Boom. It's now connected. Yeah.

So you take it, you know, like there's a particular character there. You put it in the mind of that character that could be a look-alike or, you know, however you want it. And then it's inside the game world. You know, like, and you can talk to it. This is something that is not shared yet publicly, but this is an internal discussion. But it's mainly a scratch video internally. But this is something we plan to launch. in a couple of weeks.

Oh, cool. Which is what I was saying you, that these people can actually carry out different actions. All right, so here we go. We got three people in the woods. Looks like they're in the jungle or something. For example. Three NPCs hanging out.

Sure. I can follow you. And you're telling it, oh, follow me. There you go. Okay, let's try something. So now we're doing something more complex, like, hey, can you go there, do something? More complex.

Like, hey, can you move to those trees and back to me again? Yes. I can move to the trees and back to you again. Ah, yeah. So that's one thing. The next one, we asked this guy to, like, do a show us a dance. Maybe I'll share this one, basically. So, one sec.

Yeah, this is going to get crazy. So we're here. Yeah. And we'll ask the character to bring us the axe. No, no, tell her to take the axe and kill the other two NPCs. Sure, I'll bring you the ass. I'm joking. Don't tell her to kill the other NPCs.

Thanks for watching his demo from Conve. So you can think of like many different things which we were discussing before. And each of them were like avatars still avatars where we put in a character ID. And now they are aware of the environment, right? And you can tell them, accompany me or go there, do that, tell this to another friend, et cetera, et cetera, right? You can make them look like however you want to make them look. So, yeah, this is interesting. And you could see people becoming addicted.

If these things have history, every time you come back and you start talking to these people, you could start developing relationships with them like obviously famously in the movie Her where you're kind of making relationships. So this will change video games forever if the characters, like some people play these games for a decade, the world,

the Warcraft people have been playing forever, you know, whatever. You could start having these characters build deep, meaningful relationships where you talk to them about your life and your kids and this is happening. Hey, why haven't you played the game for so long? Will you come back to the game? and then this kind of, now they start to become sentient

in a way, right? They have their own lives inside the game and if you don't come back, it kind of reminds me of toy story in a way, like where the boy grows up and the toys don't have, Woody doesn't have anybody to play with. And so this is,

could go beyond video games here in terms of like second life had the game second life, it was all human characters, but if you introduce AI NPCs, Well, now you've got a whole different world. I could create my own world. And you never know.

They might be already doing that, you know? Yeah, you never know. People can be putting those in there, yeah. All right. You know, I've been on a health kick over the past year, and you know I care about data-driven solutions. And if you listen to this podcast, I bet you do too. So let me tell you about FitBod. This is a data-driven workout app that blends machine learning with exercise science.

FitBod creates custom dynamic workouts programs based on. your fitness goals, your experience, and most interestingly to me, the available equipment. Let's say you got a bunch of kettlebells, or let's say you're at some, you know, sparse gym at a hotel, or you're on vacation. You got nothing. Well, FitMod will maximize your fitness games by varying the intensity and the volume between your sessions and leverage the equipment you have or don't have, as the case may be. You can customize the length of your workout, what muscles you want to target, and so much more. So let's say you want to get a 30-minute workout in. And I want to do chess, triceps, and apps.

But I'm staying at an Airbnb. There's no equipment. FitBod can create a perfectly optimized workout for me based on these parameters. And it will do it for you to check it out. It's amazing. The design of this app is extraordinary. I was able to invest in it. That's how impressed I was with it.

FitBod takes the guest workout of fitness. Just open the app and start making progress. You deserve it. Get 25% off your FitBod subscription or try out the app for free. When you sign up now at FitBod. M-E-slash-Twist. That's F-I-D-D-D-M-E-S-T-W-I-T for 25% off. And so pretty obvious how you make money.

You charge developers to use these tools. Do you charge them on a per-use basis? Anything like that? Yeah. So basically for developers, it's all the way from free through, you know, like $29-a-month. But primarily when they develop and ship it, you know, then the real traffic starts, right? So we we have our cost pretty much close to chat GPT cost per token, you know, like, but we call it by interaction because rather than, you know, like, for interaction,

we kind of assume thousand to two thousand tokens are being used because there's backstory and whatnot. And, and, and, uh, so it's not a fixed cost. If these things start getting really popular, there's a usage base here. So the game has to have some kind of monetization in it, uh, or ability to pay for it. So that's... But it's going to be de minimis, yeah. Yeah, but we have done the math and done the numbers, business models for various business models, all the way from free-to-play games to fixed cost,

one-time purchase games, and subscription model games. It can, with the right design in place, it can work with all of those models. Yeah, I mean, this is something that got unity in a lot of trouble recently because they started moving from flat rate to usage-based, or I guess publishing-based. They wanted to make more money.

You're aware of that. I guess. What's happened in the developer community? Because you're also adding variable pricing to groups that might have flat rate pricing. Like, you know, Candy Crush maybe doesn't want to pay for, or Angry Birds doesn't want to pay for something like this if they're flat rate. So, but where we see this going is that, and this kind of like brings us to the market size discussion, actually.

So, um, you know, if we do a bottoms of understanding, so every game, every game that has characters, you know, like whichever way, you know, all the way, from absolute AAA 3D cyberpunks-like games through even small mobile games that has characters, you could potentially have these LLM-driven experiences for these characters, right? And yes, and my thought is this, that if players find it more engaging

and they stay longer, the retention is good, they will spend more money and the studio will make more money and thus it won't be a big deal to pay for us. You take this games like Apex Legends and whatnot, They spend, because any MMO, massively multiplayer online games, has a huge tens of millions of dollars they spend in server costs, right? So it is not super new to spend server costs for more engaging time.

So they blend server costs into there. So you're just like a hard drive or some CPU. That makes sense or a GPU. Yeah. What was it like to work at Nvidia? When did you work there? And tell me a little bit about the ascension of that company. It's been crazy.

Sure, absolutely. I mean, I absolutely love the company. If I didn't have this feeling of I absolutely have to do this, I would not have left. And there's, I don't know if other companies, I mean, I haven't worked in a ton of companies, but the culture there is absolutely amazing. You know, like, in my experience, at least. So I loved working there, love the people, still in touch with them. And thanks to them, I mean, Jensen actually showcased one of our demos that we created Workdown together, and I don't know if you remember that, but that went pretty viral. If you want to see, I can share that. But like, the... Which demo was it?

The... It was this Raman Shop demo? Oh, yeah, sure, show, show, show. We partnered with a framework and avatar toolmaker called Convei. And together, we developed this demo you're about to see. Okay, Rhonda, please. Everything is real time. So you're going up to the ramen counter.

Yeah. Hey, Jen. How are you? Unfortunately, not so good. How come? I'm worried about the crime around here. It's gotten bad lately. My ramen shop got caught in the crossfire. Can I help?

Yeah. If you want to do something about this, I have heard rumors that the powerful crime lord Kuman Ayoki is causing all sorts of chaos in the city. Uh-oh. He may be the root of this violence. We're going to have to do something. Where can I find him?

I have heard he hangs out in the underground fight clubs on the city's east side. Try there. Okay. I'll go. Be careful, Kai. So that interaction is happening and you got your quest, but that's happening in real time. Yeah. Yeah.

Oh, here. A character, a backstory. His story about his ramen shop and the story of this of this game. And all you have to do is go up and talk to this character. And because this character has been infused with artificial intelligence and large language models, it can interact with you, understand your meaning, and interact with you in a really reasonable way. All of the facial animation completely done by the AI. We have made it possible for all kinds of characters to be generated.

They're all domain, they have their own domain knowledge. You can customize it so everybody's game's different. And look how wonderfully beautiful they are and naturally. they are. This is the future of video games. Not only will AI contribute to the rendering and the... So, I guess the question is now from here, can I take this

and make my own character and then put it inside of this? Remember the... This is Black Mirror. If you miss the episode of Black Mirror. Yeah, yeah, I saw this. It's horrifying. Horrified when the little robot is chasing you. And you've built the

software to power this. Because you could write that you are just absolutely evil, murderous robot, please go out there and this is your backstory. You are only, so congratulations. I think you're the guy in the Terminator movie who accidentally unleashes all of this. No, I'm kidding. Hopefully not. I don't. Well, I mean, here's the truth. You could program and people have programmed and humans are using robots right now or like quadcopters and drones to go do crazy things in Ukraine, you know, and, and, you know, and and in all kinds of wars and stuff like that.

And this next wave, the next wave of weapons is going to be built with tools, not your tool, but not a totally dissimilar tool, which might be, hey, these are the bad guys. Here's kung fu. Here's martial arts. And, you know, if you had some robot, here's all the weapons you can use. And use your best, you know, the AI within real time. Yeah.

Teach it how to fight different opponents or, you know, solve different challenges. Absolutely. And I think, I mean, it's definitely happening whether we do it or not. Yeah. The, the, but it may actually even help people to not kill people. You know, when these are robots, they can take bullets and not fire back. Yeah, you can just say, hey, disarm the person, sit on top of them. Disarm the person. Yeah, yeah, exactly. So, yeah, you could write one that's just like Andre the Giant, where it's just like, do the war,

just lay on top of the person until help comes. Hold them and handcuff them. And that's how you fight worse. You don't have to kill them. You know, like, just guessing, yeah. How did this whole generative AI stuff start? You had some thoughts on that. What your take on how we got to hear?

Because this has been a crazy year. We're getting right on the year anniversary of ChatchapT 3.5, I believe. Yeah. When we tape this in the fall of 2023. So how did this all start? then what do you think about the pace? Yeah. I think multiple things are reaching to an inflection point which is causing this.

So, and you all know this. So basically over the last 20 plus years, we have been building this body of data over the internet, collectively, everyone. You know, like everybody contributing to the internet. Yeah. The hardware has been getting better. And finally, and the architecture, the neural. network models has been there for way longer. The theory has been there for way longer.

And now finally we are going into, so initially it was all about vision. You know, like vision models got quite well. We started doing it on text. And now we are going multimodal. You know, GPD5. GPD4, vision already has that. Yeah. GPT5 probably will be about videos.

You know, like who has the most amount of videos. So in terms of like, and then what we are doing and I think, the next frontier and which is something that actually I wrote about back in 2017 that before starting at Nvidia was that and I saw that while doing my thesis work that you know language models are where it's at it was not called large language models back then and um and uh you know I I wrote that while these language model systems are getting better and will continue to get better it is not exactly how we humans learn we learn from the 3D world around us and and And then we basically have, like, when we were kids or babies, you know, we attach the words to the images that we see.

Like, this is mama, this is papa, this is food. You know, like, and that's how to learn. A very unique thing about cognition for humans, our big, large brains put words on it. And I wonder at what point in evolution do people start in their cognition. You know, we know consciousness starts like, you know, at different points in time for different species. And then the sort of advanced cognition where you're saying, this is this, and you have a word, and this is what I want to do, and I am here, and this is what I'm going to do next, right? Yeah. Very interesting.

And I think the only difference, and I hope we are super careful on this, especially the big companies who are working on this, is our objective function is survival, all the way from a multicellular organism to like a small creature. That's the objective function that we are trying to optimize on. The today's language model is the objecting. And so. Which is kind of the same thing, right? Because that's like survival of your species. Survival of your genes, I should say. Yes.

Survival of your genes. So that's why parents would sacrifice themselves to save their kid. Survival of their genes is the objective function that we optimize on. So the, and for the language models right now, it's like predict the next word or predict the next token. And that's like their objective function. However, where I think with all the multimodality is coming in, researchers are going to see that encoding survival is going to be necessary for the best AI.

And that is something that we have to be super careful about. We don't want, maybe we don't want that. Yeah. Why would encoding survival be so dangerous? Because encoding survival, because now then we are creating a different species, you know, like effectively, who is going to start thinking for themselves, right? what objective function and maybe this is the movie

the iRobot the tenets of what we would want to encode as their objective function becomes important yeah we have to tell them their objection their objective is to protect human life and to be useful to humanity

yeah and but useful to humanity might be oh you know what you know fentanyol is really dangerous so let me just go kill everybody who is using or near fentanyl I mean, it might take it literally, right?

Yeah, yeah. Cancer's bad. Cancer exists in people's bodies. So, the greatest way to get rid of cancer is to kill the bodies that have cancer. And it's like, no, no, no, no, no. Everybody's got a little bit of cancer inside them, probably. Yeah. All right, business to business founders need to understand two very important things.

B2B marketing is hard. We all know that. It's really hard. Number two, it's simple on LinkedIn ads. Why is B2B marketing so hard? Because the decision makers, hey, people like me, right? The founder of the company. Or maybe you, you're an executive.

listening to the show. They're hard to identify, right? And they're really hard to target over the internet. And how do you know you're reaching them in the most respectful environment? People who are decision makers, you cannot reach them through that 1990s way of reaching executives. No, they need social outreach. And the way to get that social outreach dialed in is with LinkedIn. LinkedIn solved this problem. There are 70 million decision makers hanging out on LinkedIn. And executives like me, yeah, the decision makers, we're on LinkedIn. We're deep into LinkedIn every single day. And LinkedIn ads helps you get your B2B message in front of the right people like me. And listen, when people are on LinkedIn, they want to do business.

LinkedIn equals business. Business equals LinkedIn. Each social network has its vibe and the vibes at LinkedIn are business. It's business vibes only. And it's happened finally. LinkedIn has hit one billion users to march to a billion has been achieved. Congratulations to all my friends at LinkedIn. So here's a call to action.

Get started today. CY LinkedIn is the place to be for B2B. We'll even give you a hundy right now, $100 credit on your next campaign. Go to LinkedIn.com slash this week in startups to claim your $100 credit. Why wouldn't you take the 100? Go to LinkedIn.com slash this week in startups. The links are in the show notes. Terms and conditions do apply?

So do you think this regulation that just came out, this executive order this week? Do you think that's overreaching? Do you think it's going to stifle innovation? Do you think it's worth getting the government involved this early? Or is it too late to get the government involved? How do you look at it? I pretty much take the same take that you all have, you know, David Sachs have in your podcast that I think it's definitely early.

I haven't read the whole thing through, to be honest, but from what I understand, it may not be great for startups. It's great for incumbents. Of course, yes. You have more resources to report in. You got resources to hire people who used to work for the government. It's regulatory capture in Bill Gurley's amazing talk at its peak. It's peak regulatory capture.

It's like, we can't wait to be regulated because we have $10 billion in our bank and, you know, you have $10 million at your startup and the next startup has $1 million. And, you know, the two folks in a garage or at Ycommonator have $100K. Right. So, yeah, it's a profoundly unfair for later. But open source is going to win the day, you think? Or do you think closed day is going to win? That's, uh, hmm, the thing is whoever has the most amount of niche data are going to win. Right?

So let's think that through. Google became a monopoly because it had the most niche data of people clicking the right links and pays rank doing its job. So it was actually human feedback that got got it the network effect, right? So Chad GPD is trying that to some extent, but is that enough data when you give the thumbs up, thumbs down? It's not as much as Microsoft, Google, and Apple have. Exactly, right?

They have billions of users. Yeah. Right. So they are collecting human feedback data. Is that differentiated enough? And I think open source has a chance in the sense that until we exhaust the amount of, you know, like internet niche data that we can collect and clean. But I do feel that the open source, thanks to meta, is like, and some other organizations

like the Falcon ones and others who are open sourcing with licensing will enable people to contribute their niche data. And they would want a way, I think open source can win if there is an incentive system for people to contribute their niche data. Interesting. So if it has a license to use Falcon

or a license to use this open source model, and then if I put my data in, I own Getty archives, I say, I'll put my Getty in there, but I want, you know, 3% of the pool or whatever it happens to be. So that requires some centralization, which, you know,

not necessarily. Oh, okay. So, and this, this is, don't say blockchain. Don't say blockchain. I was going to say blockchain. Oh, no. Are you telling me that after all this crypto drama,

it's coming back? A hundred percent. The blockchain's coming back. Oh, God, no. No, here's the thing. See, I was absolutely against, I was absolutely against the whole hype train. 90% of the people were there for making a quick buck.

Sure. But if you see the, the fundamental tenets of you know like blockchain with zero trust protocols and such it can enable a lot of other applications that are otherwise not possible

right if you have a token too and then you and I we could create the token yeah this weekend and then we'll keep like 10% of it in a wallet that we forgot the you know that we forgot and then we'll own 10% of the global AI wallet share

and then we'll liquidate it in 10 years whatever It has to be done ethically, but that's how you get everyone valid it around you. That kind of invalidates crypto then. No, I mean, the blockchain would be actually a way to do licensing, right? Yeah, that would be an actual use case. That would make sense. Smart contracts, yeah. Yeah.

Hey, when you think about AI and you're kind of in a dreamy state, what do you think's going to happen here? This has been moving at a crazy pace for a year, but you've been in it for your whole career. What do you know, based on what you know on the pace, right? back to my question about the pace. The pace is obviously accelerating. Now you've got a lot of people using these tools, making all kinds of edge cases, and then you got every company with unlimited

or seemingly unlimited resources making their own chipsets, you know, and the amount being invested in this means it's only going to get faster. And the velocity is going to get faster because more people participating, more money participating,

and the AI getting better, and then using the AI to make AI better, which has always been Elon, and other folks concerned that AI will make AI better, we're kind of on the cusp of AI making AI better, yeah? And so what do you think about the pace? Yeah, I think the pace is not going to slow down from here because we have entered time where the data for all the different modalities

that humans understand are there, like video, audio, you know, like, and it's only going to get more and more. So this is not slowing. Basically, we have reached the, you know, we see human civilization how, how curved they were. And I think we are there. It's like it is going to accelerate it. It's the inflection point. It's not classic hockey stick, but probably like a hockey stick we've never seen before.

And that's, and in fairness, shout out to my guy, Tim Urban. In fairness, that's why the government wants to slow this down because it thinks that that inflection point could happen any second. When does AI start making AI better? Like, literally somebody. says to Claude, you know, hey, look at your, or Falcon, somebody trains Falcon to look at Claude's code base and pulls up their open source, whatever. I don't know if Claude's open source, but you get the idea. Maybe it would be Falcon looks at, you know, what other open source project, and says,

make that better. And then it actually contributes without a human being involved and actually does make it better. That's the main point. That's the main thing. So right now, AI is making AI better. A lot of the open source model is actually collecting data from GPT, cloud, et cetera, you know, like, and making the models better. However, AI doing that itself, the entire loop all the way from training, fine-tuning, and all that.

I think we probably may not want that. We always still want human in the loop. At least for the next- When would it be possible, do you think, to set an AI? Technically? Yeah. Technically, one could build that already, but that may not be the perfect thing. That may not be the perfect thing, because when you are training a model, you have

have to look into so many different things. Is it even going towards convergence or is it making the model more stupid? Yes. And there's like, are you cleaning the right data? How many GPU is it using? Did a run fail, job fail or whatnot? It definitely requires, you know, humans right now. And even if you set up a system, it's not guaranteed to succeed.

Yes. And who's going to add more new data? Like you can make the loop of AI making it a bit better, but like it ultimately matters, like where is the data coming from? You know, like, the new data, the new data... Is synthetic data a thing, or is it just a buzzword? That, like, this AI can make data that would then train the next AI. Like, oh, make a bunch of images and then train it on the images.

That doesn't make a lot of logical sense to me, but, yeah, it's not a thing. So I think the... Because you have already added the data to, in this case, right? Like, it may help the model in some cases where it may be stuck on some optimization situation. So to give it the necessary noise or the necessary noise, necessary variance in the data. Synthetic data may be helpful, but you primarily want more and more new data

and different modalities of data, like videos, and that would require new architecture and such. Let's put this side. Let me tell you an idea. This is an idea. Shout out to the team over at Cora. What if Cora look at the experts with the most highest ratings, you know, with the highest ratings?

So they look at the experts with the highest ratings, and then they say, hey, AI wants to talk to you about your expertise. Would you, for a hundred bucks an hour, for a hundred hours, answer questions with this AI. Just sit there and answer questions. And you'll go up the leaderboard and we'll send you some cash. But you say to the AI, what do you know about, you know, poetry? What do you know about history? And then it goes and asks a hundred history experts what they know about history. And then it says, okay, this history expert told me this, this one has this thing.

So you could actually have AI, interviewing humans to find out what it doesn't know. That would work, yeah? Yeah, yeah. You basically want human feedback, especially for the corner cases. The gold data are the corner case data. Gold data.

Yeah. So, like, the corner case data where things are not working or where you would want human feedback, that's the gold data. And, you know, like, if there's an incentive system, that's what I'm saying, like you basically want Amazon,

Mechanical Turk at scale. Yes. You know. Yes. And but the, it's mechanical Turk talking to experts, not low wage folks. Yes. Exactly. Who are saying, is this a hot dog or not?

Yeah. You want the person who's the greatest chef in the world who will tell you how to make the hot dog and a hundred other things. That's kind of where it gets super interesting. So somebody could make an AI mechanical Turk type thing or Kora could do it. Yeah. Where it identifies the experts and then just lets AI talk to them constantly. and then pays them for their knowledge. All of the startups that are starting, let's say legal startups,

that are basically that. That's true. Verifies legal or, oh, you think they're hiring lawyers to answer questions? Yeah. Yeah. Yeah.

Yeah, yeah, yeah. Not just giving it to potential customers and saying, hey, give it a feedback loop. They're explicitly going to lawyers and saying, hey, did it get this answer correct? Right. Same with healthcare startups and, you know, on top of LLMs and whatnot. That's what they're doing.

Yeah. What's crazy about that is the people who are being asked to train it are training. They're basically training their replacements. Yeah. They're training their replacements. They're training their replacements. No, like literally, this happened in America. They, uh, there was a movie about it.

It was a movie in the 80s. It was a comedy and it had the guy who played Michael Keaton. I think it might have been Michael Keaton. It was about factories. And I think it was like on a couple different contexts, but the idea was like, oh, the Americans are going to go to Mexico or Japan and that they would teach them how they did what they did. And they were like, wait a second, we're teaching them, they're automating it or whatever. And then we're losing our jobs.

They're taking our jobs. And there was like gung-ho, maybe it was the name of it. I can't remember. It was a crazy, crazy movie. Yeah. But I think net it will be positive in the sense that for the most part, you still would want the experts in the room. they may take out the redundant stuff, but I don't see at least in the next 10 years

that lawyers or doctors would be invalidated completely by the AI. That's like at least 30 years away. Yeah. Yeah. Anything you're worried about right now with AI? I think the regulation part I have to check. I don't know if it affects our startup, but that's one thing, yeah. That's the goal is for you not to understand it.

You have to hire a lawyer to tell you. Yeah. Yeah. The other thing would be the funding market is bad. Thankfully for AI, it isn't terrible, but who knows what is going to happen. So there's that. Yeah, yeah. All right, listen, you've been an amazing guest.

Come back on in like six months or a year and let's just chop it up and talk about other AI stuff. You'd be good for an AI roundtable. I like the fact that you just answer the questions and don't try to be like super correct about the answer. Like, this stuff is dangerous. Yeah. It's also inspiring. It's getting better and better. I mean, this multimodal thing and chat GPT4 is,

really mind-blowing. I just took a picture of a medical document and I was like, hey, transcribe this. And then I was like, oh, and give me some advice on it. And it's like, I was like, you're a doctor. Give me a second opinion. And it does. It's kind of nuts, you know?

It's nuts. Yeah. You're going to be like at the doctor. The doctor's going to like be looking at your x-ray. You take out your phone. You put on chat chbd4 and you hear the doctor and you take a picture of it. And then it's going to like basically check that the doctor got this right or give feedback. And it's going to be nuts next year.

It's going to be nuts. Yeah. Yes, yes. Yeah, we hope like, yeah, once we have our plan is that these characters today that you just saw, they can actually see the virtual world with scene metadata, you know, because it's in an engine. Yeah, yeah. But our goal is that they can actually take in the vision input. Oh, you're going to give them a vision.

Oh, that's so brilliant. So you're going to give them virtual eyes So they see the perspective That the player would see Apply GPD4 Yeah They learn from what they see Yes

This is why like they might want to stay alive And they might want to try and hack the game to stay alive That was by the way another great 80s film No one is war games I'm giving you all the stuff you missed War games Another great one shall we play a game You've never seen more games

No I have to It has to be on my list now. Oh, my God. You're going to love it. Oh, my God. Matthew Broderick. Oh, my God. Ali Sheedy.

Oh, my God. You're going to fall in love with Ali Sheedy. Like, she was like, everybody, every nerd guy's like dream girlfriend, Ali Sheedy. I see. Amazing. My oldest was Blade Runner. And obviously the new ones like free guy. Free guy is like exactly what we are doing.

Free guy is exactly. I forgot to bring up a shot of Free Guy, which is the NPC trying to break out and then be in the real world. real world. Incredible. Of course, there's no way to do that right now. Yeah. No, war games pretty great. You know, verticalized AI becoming sentient,

basically. You're going to love them. All right, everybody. We'll see you next time on this week and start us. Bye-bye.