No Priors: Artificial Intelligence | Technology | Startups - How Replit’s AI Tools are Changing Software Development with Co-founder and CEO Amjad Masad

Episode Date: September 21, 2023

Replit’s develop-to-deploy platform and new AI tool, Ghostwriter, are breaking down the barriers to entry for beginner programmers. Replit’s CEO, co-founder, and head engineer Amjad Masad joins hosts Sarah Guo and Elad Gil to discuss how AI can change software engineering, the infrastructure we still need, open source foundation models, and what to expect from AI agents. Before co-founding Replit, Amjad Masad worked at Facebook as a software engineer, where he worked on infrastructure tooling. He was a founding engineer at CodeAcademy. Throughout his career, Masad has been an advocate for open-source software. Show Links:  Amjad Masad - CEO & Co-founder of Replit | LinkedIn   Replit Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @amasad Show Links:  Amjad Masad - CEO & Co-founder of Replit | LinkedIn   Replit Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @amasad Show Notes:  0:03:55 - Impact of AI on Code Generation  0:11:09 - Breaking Down Barriers to Entry in Development with Replit 0:14:35 - The Impact of Open Source Models, Meta/Llama 0:20:32 - Bounties, Agents who Make Money 0:24:26 - The Missing Data Spec-to-Code 0:32:29 - Building the Future of AI, Money as a Programmable Primitive

This week on No Pryors, we explore Replit and its new AI tool, Ghostwriter. Replit now has more than 22 million users and recently started a partnership with Google. With Replit, anyone from beginners to experts can quickly build and release fully functional live apps and seconds, now with the help of AI. I'm Jad Mossad, CEO, co-founder, and head engineer at Replit joins Alad of me this week on No Priors. Welcome. Thank you. Judd, just to start, you were working as an engineer at Facebook. What made you decide to start Replit? My journey with Replit actually starts way back in college. I was really interested

in program languages, and I found it really difficult to just keep setting up the development environment. I didn't have a laptop at the time. And so I was like, you know, why can't I open a browser window and start coding? Turns out nobody has done that. And I was kind of naive and thought I I could probably try to do it and got a prototype up pretty quickly, but my friends loved it in college. Everyone started using it. It was just like a simple webpage with a text box and a button that says run, which is still kind of in a lot of ways of our upload is. But then that started me in a multi-year journey to actually build the first kind of in-browser sandbox. And at the time, we had a bit of a breakthrough.

We were the first to compile Python and Ruby and other languages to JavaScript. I open source that project. That got me a job at Code Academy. I was a founding engineer. And after that, I went to Facebook, and I was one of the founding engineers and React Native. We, like, made React and JASTA BABEL. And a bunch of tools that JavaScript developers used today,

our goal was to make mobile development as fun and easy and fast as web development, which I think we succeeded at. You know, towards the end of my time at Facebook, I just wanted to see what to do next and kind of looked around and, you know, whatever happened to this online coding space. And there was a bunch of online editors, but none of them really felt like web native. Like you can just share a URL and someone can hop in and code with you. Kind of like how Figma or Google logs work. And we decided to start in 2016. You have made a huge bet on AI as a company.

Can you just talk about Ghost Rider and how it came about and the investment in this area? Yeah, throughout my career working on code that handles code, right? Whether it's at Code Academy for teaching programming, whether it's my own project, whether it's React and building the runtime around React Native, I always felt like our tools that were handling code, whether it's like, compiling it, parsing it, minifying it, all that stuff, we're very kind of rigid and very laborious. In a lot of ways, you're building sort of a classical intelligence system,

very algorithmic, and a lot of heuristics and all of that. And I always thought that you can probably apply machine learning to it, and I started reading around whether anyone had done it. There was this seminal paper in 2012 called On the Naturalness of Software, and basically a bunch of researchers try to apply NLP, to code. And what they found is that actually code can be modeled like any language. That's why they call it naturalness. Because, hey, code is kind of repetitive, like language. You can do things like Ingram, basic Ingram model can actually start to generate code that is compilable. That had a

huge impact on me. And every year or two, while starting Replit, I would go look at the state of the art on ML on code. And nothing ever really worked all that well up until GPT, And you can like take GPT2 and fine tune it and like try to make it right code and it was like kind of okay. But obviously GPT3 was like, okay, it's here. And that's when we started building what became Ghost Rider. Initially it was a bunch of tools kind of sprinkled all over the Replit ID, explain code, you know, generate code here. And then we wrote our kind of auto complete product, which is kind of like co-pilot as you're coding. It kind of gives you suggestions.

And then we added the chat product and that all became sort of ghostwriter, our AI suite. It's amazing that this idea of type ahead code autocomplete with local context has become as widely adopted as it has. Like your tool and co-pilot are sort of a handful of tools that people like very much accept have changed workflow. Maybe if we just project out a little bit, where do you think that the like next big leaps in productivity in software development come from? in terms of AI. What else is going to happen? It's an amazing tool, but I think it's still very primitive. And we haven't fully explored the full capabilities of even GPT, but also transform models applied to code in general. There's a lot to do. I think there's a lot to build on the sort of the layer just around the models. So they're basically how to give models

better context, how you give them tools, the ability for the model to actually be able to go out and read a file, install a package, evaluate code, the ability for these models to be more agentic and be able to kind of write entire features by themselves. And so that's all the work just around the model itself with the model in the center. Then there's a lot of work on the models themselves

that haven't been fully explored. The way we train models today is we just give it a large corpus of code, but you can imagine different ways of training models. And we've played around with different techniques at Replit. There are ways to give models more intuition about how code execution happens as supposed to just static code because you're training them on static code. They understand the structure and syntax of code, but they don't really understand the semantics of code.

So you can imagine a way to train models where you're not only feeding them code, but you you're also feeding them the results of the evaluation of that code. You can imagine ways of training models, not just to be good at code, but also be good at the editor, the debugger, everything that comes around code itself. And so I feel like we're in the very early beginnings of it. I think for professional programmers, perhaps it's like a marginal improvement. And like some people claim it's a huge margin based on our research of how people are using our products, we think it's 30 to 50% more productivity on coding itself.

Obviously, there's a lot of things other than coding, such as debugging, deployment, meetings, being able to translate specs into code and everything that we do just around the active coding itself. AI hasn't touched yet, and I think it will touch every part of the software development life cycle. I think actually the people that are getting the most amount of value today tend to be the beginners tend to be the people who don't have the skills typically to like start a startup. And we're seeing a lot of those on Replit just today.

We've profiled an entrepreneur that has a quarter million ARR startup in a few months. And they haven't, you know, earlier this year, they didn't know how to code. And I think this is where most of the impact is right now. But I think if you project forward, those people will also get a lot more tools and going to get more productive. are going to hit superhuman productivity. But the professional programmers, this is where actually you start to have that 2x, 5x, 10x, multiple in productivity.

What do you think is a timeline on that? Do you think that's a year away, three years away, five years away? I'm just sort of curious about when we both see really massive productivity gains and then also when we move towards this more agentic world that you mentioned where you have AI go out and do things on your behalf in terms of generating a set of features or grabbing up a package from somewhere or doing other things for you.

My feeling is this is within reach, even in the current capabilities. If you hold constant the current capabilities, this is within reach in the order of six to 18 months. We're in the demo era of GPT models. Right now, we're doing the hard work of building infrastructure on Replit. We are building this service that sits inside the container that embodies the AI in the development environment, such that it can do things like go to the internet, like install a package, like read a fire, write a file, start another project, get clone, do all of that stuff. That's a lot of work.

And that's not just one model, it's probably multiple models. And so within the current capabilities, there's a lot more to do on the infrastructure side. So I think we're within months of actually having really cool basic agentic experiences where you can give a high-level task to a model or to an AI. such as a build a login page, and for it to do a decent enough job to give you a starting point for that. Now, if you try to project like the actual advancement in AI itself, increased context size, increased performance, better prices, potentially like even net new advancement or new ways of training code models, it becomes really hard to predict. I think software could fundamentally

changed in the next three to five years where it looks totally different to someone sitting today and looking at it into the future. Maybe it looks something like programmers tend to be more like engineering managers. Even ICs, you're just like working with employees. Those employees are mainly AI and most of the time you're reviewing code and giving instructions and you're not actually like spending 50% of your time typing. It's hard to know. But I'm confident to say within the current capabilities, there's still a lot to do over the next six to 18 months. And then I guess one of the thing that you have is a real advantage for your product is just you have such user liquidity, right?

So you have, I think it's on the order of 22 million users or something in that range. And that obviously can create an enormous feedback loop for training your own models, for reinforcement learning, et cetera. There's a paper that recently came out from Google comparing a very specific type of reinforcement learning through human feedback versus AI feedback. and it roughly netted out, right? You could start to use AI for certain tasks to effectively train itself. Like, how do you think about that in the context of code and sort of where you're heading?

I think it's easy to overstate data modes. You need to be measured about that. I think the real advantage of Replit is actually just the platform, just the end-to-end platform. Because I think, you know, you can be, you can try to build these code models, but without having the platform to be able to apply it on, to be able to get feedback from, it gets very difficult. But also, even if you just have a code editor or editor extension, that doesn't give you all the feedback that you would want,

because you would want feedback from deployment. You would want feedback from execution of the code. And not just user feedback. I think user feedback, you're right for our Lachaf type things. I don't think that's the real advantage. The real advantage is the end-to-end journey from the, the first line of code to the deployment, to getting crashes in production, to making edits, to all of that feedback cycle, which typically in most cases is divided between GitHub and

VS code and AWS and all of that. And the real magic of Rapplet is putting all of that together. And this is, I think, where we're going to get the kind of richness of training data that allows you to train over a large action space. And that's really exciting. Can you talk about the decision to train models versus use existing APIs? Like, was it just a latency thing? How did that happen?

So when Microsoft did it, like they took GPT3, they distilled it down, and they hosted it on their own infrastructure. They did a bunch of caching, a lot of things like that. If you're someone who's just using the Open AI API, you couldn't do any of that. And it was very expensive. It was very slow. And even now, you know, there's no. completion models. Now, all the models are chat models, you know, they're not going to be

releasing any completion model going forward. And so if you wanted to build a completion-based product, it's actually fairly difficult to do it using commercial APIs. And we felt like we can we can make a model that's both cheap and fast and good enough for that auto-complete use case. And we found that the 3 billion parameter size is kind of the nice sweet spot where you get enough raw IQ from it because like $1 billion felt kind of too dumb in a lot of ways. And it was going to be cheap enough and fast enough to host and be able to do a fast inference from. And it was a time after the chinchilla paper first came out and Lama had been announced.

And the idea of just training them longer had just been in the air and we're like, okay, what if we apply all these open source tokens on a smaller model? And then we applied some more tokens from Replitt's data that gave us a 50% improvement over the model that we open source. So we open source the base model because we think that's sort of the ethical thing to do because we trained on open source data and we want to give it back to the world. And what state of the art model was better than most open source code models at the time, even things 20 exit size. But the need for it came from a product need. Like we needed to build that product. I think a lot of founders now see it as a point of pride or status to train a model.

And the moment we put out of the 3 billion parameter model, like three or four copycats came out after that. And I'm not a fan of what's happening in the open source world where everyone's competing over these silly benchmarks. I think it's important to start from a product and start from a user and customer need and then go from there. right now, if we could do what we can do with Lama, we'll just use Lama. If we have to train a model, we'll train a model. And that's how we always approach building this company is start from the customer and the use case and then solve any technical challenge that you get along the way. One of the interesting debates I've been hearing about Lama recently is how long will Facebook

continue to sponsor them as open source models? And so far it seems like Lama 2, for example, is the one that a lot of people are viewing is the most performant with Lama 3, perhaps being a big step forward in terms of catching up relative to GPT4 and sort of other close source models. What do you think happens if Facebook stops open sourcing? Do you think somebody else steps into the void? Do you think everybody's kind of on their own? I'm a little bit curious how you think about the open source world relative to models,

given how important Lama has become so rapidly. So when I came to the realization early 22 that we can't build the product, that we want to build using commercial APIs. I was kind of depressed for a moment. It was like, oh, we didn't have a lot of funding, whatever. It's like, okay, Microsoft's going to be the only one is going to be able to build these kind of products. And I got a meeting with Zuck, actually.

And I pitched Zuck on this idea. I remember around the time, he did the OpenCompute project. So what OpenC compute was, like Google and AWS are going to keep all their data center secrets because it's a company advantage. But Facebook did not sell compute, did not sell cloud. And there's a lot of other companies that did not sell cloud. Cloud was a complement for them, was not a competitive advantage. And so they pooled their resources together and they worked on this open compute project

to be able to optimize computers for data centers. And it was a huge success. And what I told Zuck at the time was like, hey, why didn't you do that for LLMs? Obviously, open source AI is going to be great for you because you can build. better product, it's a complement. And so it's sort of the traditional, like the classical like commoditizer complement a business strategy.

And at the time, he just kind of nodded his head. And I don't know if it was a novel idea, maybe he was already thinking about it. But that's basically what he's talking about. I think in one of one of his interviews, he talked about open compute as an analogy for what they're doing with LLMs. So I think it makes a lot of sense for them to do that.

What I'm actually surprised by is how little industry participation they've got. And I think the difference between LMs and OpenCompute is that the AI safety angle makes it a little toxic for a lot of companies to touch. Like they wouldn't want to be associated with something that spews something that is toxic or they don't want to touch. So that's another wrinkle that I didn't appreciate it. And it takes a certain amount of guts to release an open source language model, especially with political heat that Meta's getting from that. Now, what happens if they stop? Yeah, I think it would be bad.

I think it would be bad for the ecosystem. Will there be another player that will emerge? The problem, a lot is that, as you know, there isn't a lot of guts in the industry. That's our problem. It's like, and Zuck has balls, right? Yeah, no, I think it's been super impressive what meta has done with Lama, and I think it's really helped push forward the entire industry in an amazing way. And so really kudos to the company for doing it.

I think beyond guts, which I agree with, there's also the cash side of it. I actually think the money exists in the ecosystem to pay for open source and to help train models. I think the talent base is much harder. And the magic of what I think Facebook did with Lama is they had an amazing talent base and then the capital to train something and then the guts to release it. And if you look at every other open source software wave, almost every major one, had a big corporate sponsor, right? Linux was sponsored by IBM to the tune of like a billion dollars a year for some period of time. WebKit was obviously sponsored by Google and Apple. There's lots of the really big things that were important in terms of fundamental pieces

were sponsored by a company at one point or another. And I think in the absence of meta providing not only capital and guts but also talent, there may end up being a gap. But that's one of the reasons I was asking the question is they have that really amazing intersection of guts, money and people. Right. And it's unclear to me what the replacement there is. Yeah, it's also unclear, like you said, which other companies would be able to participate in that. And I'm sure they try to do outreach and they try to work with other companies. I think on a long enough time scale, obviously there's going to be open source models that are as good as commercial models. But the current landscape makes it hard to kind of imagine how it would play out.

And also there's this whole question of like, if the architecture gets more and more complex, it actually makes it harder and harder to open source, right? Like there are tools and there are services. Tools especially are easy to open source because it's a binary, you can run. And basically that's what we have today. You know, you can just run, it's like a binary blog, right? But if there's like a mixture of experts and there are components, stateful components for running this stuff, then it becomes exceedingly harder to create an open source alternative.

I'm curious if we can talk about bounties for a second. Sure. Because you're somebody who like clearly believes in agentic development. And then you stuck a marketplace in your product that allows developers to collect dollars for code that they've produced. Do you think we end up doing that with development agents? agents? Or how does that play out? So the idea behind Boundaries is that at Rathlet, we have all this talented people across the world that are learning how to code and that are actually like

becoming great programmers that kind of no one knows about. And I felt like it was almost our duty to, you know, make it known that there's so much talent around the world that you can to kind of leverage. And a lot of these people don't have any resumes, don't have a lot of skills to be able to go get jobs themselves. So part of it was this community building activity. Can we get all these people money? And like we have people in Africa, India, Thailand, Serbia, all sorts of countries now that are like making money on bounties. The other thing was my view at the time or my thesis that I wanted to test was it's going to be a lot easier. And it's going to be a lot cheaper to be able to get basic software tasks done

because AI, like I said, will have in the short term the most impact on beginners. So beginners are actually getting a two, three, 10x multiplication on their productivity. You learn a little bit of code, and now you can actually build a product that makes 200KRR. And so the idea was that, okay, how do we leverage this opportunity? How do we monetize it? And so, okay, human plus AI, a human who knows a little bit of code plus AI could actually be a really great developer.

And so, yeah, turns out it was right. You can go to Replit and get a startup prototype for a startup or an MVP for a startup for as little as like $50 or $100 in some cases. My hope as well for Replit in general is that there is some kind of currency or way to transact built into the. the platform, and that was upstream of the idea of Bounties, Bounties was kind of the first step into that direction. My view is that there's something broken about open source where the currency in open source is sort of stars, right, like how many stars you get. It's sort of a fake currency and creates all sorts of weird side effects. We talked about how open source tend to be the most impactful open source projects tend to be corporate sponsored, but that doesn't

really have to be the case. I think the software economy is missing some native currency for exchange of value. And open source is good and great. We want more of it, but also we wanted to be sustainable, and we wanted the people to be rewarded who are contributing to it. And then there are all sorts of things you can do with a platform that has concept of money and transactions built into that. So if your development environment and your deployment environment has a currency built into it, what kind of collaboration and ways of doing things doesn't unlock? One idea is that you sort of publish a module or publish a component or publish a service to the network and you kind of price things per function.

And I can sort of call these functions and be able to have a wallet in my application, be able to kind of spend money on the different functions and different databases, different services that I want to compose to create an application that someone can consume. And AI was an interesting sort of additional component to this because I think agents in the future will also be able to use money to do transactions. do things in the world, right? So right now, our AIs, again, we're using them as these co-pilots, right? They're basically something that's helping you do some kind of action or task.

They're predicting the next step. They're literally doing the, you know, auto-aggressive action. But what if we get to this place where we actually have these agents that can act semi-autonomously, what would you want them to do? Obviously, we want them to interact with a wider internet so that, They're coding, they're evaluating code, but you also would want them to be able to spend money and make money. In the future, I could see people creating bounty hunters that are fully automated. So can you create an AI that can go earn money for you while you sleep?

And I think that that would be a really cool, cool idea. And finally, I think on the data side of things, one thing that I thought was missing from the data landscape around code, is complete project spec or product spec to actual end product. And so we have this data now where you have a literal, like, complete end-to-end project spec and the actual code accompanied with it. Yeah, we host this college hacker group for my fund for this commit hackathon. And it is very cool seeing how sure some of the 19-year-olds are that AI is going to change development such that they don't really need to, like, learn to code in the same way before, right?

Like, and if you ask them, like, well, do you need to know architecture? And they're like, nah, like, you know, more prompt guidance. Like, do you, like, who's going to debug? And they're like, oh, another agent, right? And these are people who, you know, 20-year-old MIT kids that are, like, designing and building their own GPUs. They're very interested in computing. But the orientation toward, I would rather orchestrate some of Amjad's, like, bounty hunter-type developer bots is very strong. So I'm excited to see if that happens. I think there'll be some of your first users.

I think embedded in their answer is this attitude of like, if it becomes a problem, I will learn it. If the AI can't do it, I will learn it. And I think that's the right attitude for hackers is that you learn just in time to be able to do that task. Yeah, absolutely. I'm not going to like move myself up the MIT computer science curriculum hierarchy. I'm going to go figure out what.

I want to build and then kind of work backwards and look, I can jump 20 steps forward because of Ghost Rider now. It's a very cool setup. One of the things that you mentioned was, you know, different aspects of design, payment, different activities, etc. Do you think from a payment perspective, anything needs to change in terms of financial system? Is it crypto? Is it just centralized payments? And it's just the standard old rails? You know, I'm very pragmatic about like technology. If Stripe gets us there, Stripe gets us there. If it is Bitcoin, then it's Bitcoin. I think it's a combination of both. I don't view them as that separate. Actually, like, one interesting historical tidbit is that

Stripe, Square, and Bitcoin started the same year. And Satoshi was behind all three, it turns out. Right. Maybe. It's actually Jack. Yeah, it's Jack and Patrick working together. Right. But, you know, it was obviously in the air, this idea of like the internet is missing this core future. Money is a coordinating mechanism right. Like, there's so

many cool things you can do with money as a primitive. Not only for people to be able to kind of buy and transact and value, but you know, all the things around the people in the Ethereum community that sort of invented with staking and when I was

designing bounties, I really didn't want it to be the like rigid thing where people apply and you accept them, whatever, what if you could like stake your currency and say, I'm going to do this task. And if I don't do it, you can take $100 for me. I'm going to do it in five hours. And for every extra hour, you can slash my funds by X percentage or what have you. And so you have this market dynamic where people can kind of bet on how confident they are they're going to be able to do the task. And then the coordination it happens sort of semi-automatically because like the biggest bet, you know, takes that task and so on and so forth.

So money could be a programmable primitive. And I think that's true. And it's really hard to argue with. You know, I think there are a lot of drawbacks to the current financial system and all the problems that come with it. I still believe I think Bitcoin is going to be long term. The TCPI, IP of money, but there's going to be a lot of services that could also be Stripe and others and Square that it's built on that, like, on that sort of rails. Cool. I think we covered a ton of ground. Thanks so much for joining us today. It was a really great conversation. Yeah, likewise. Yeah. Thank you for having me. Find us on Twitter at No Pryor's Pod. Subscribe to our YouTube channel. If you want to see

our faces, follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for email. or find transcripts for every episode at no dash priors.com.