Invest Like the Best with Patrick O'Shaughnessy - Scott Wu - Building Cognition - [Invest Like the Best, EP.402]

Starting point is 00:00:02 Hello and welcome, everyone. I'm Patrick O'Shaughnessy, and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. Invest like the best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at join colossus.com. Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of positive sum. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of positive sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc. My guest today is Scott Wu. Scott is the co-founder and CEO of Cognition, which is an applied AI lab that has created the first AI software engineer,

Starting point is 00:01:04 which they called Devin. In just a year since founding cognition, Devon functions at the level of a junior software engineer, capable of handling complete engineering workflows from bug fixing to submitting poll requests. Scott is a former competitive programming champion, and he describes the engineering field as simply the art of telling a computer what you want it to do. Scott predicts AI will surpass the world's best competitive programmer within one to two years and sees this technology not as replacing programmers, but as democratizing software creation. We discussed the bottleneck in software development, the future of AI in various industries, and the challenges of leveling up Devon.

Starting point is 00:01:40 The implication of Scott's work and Devon itself is enormous. Please enjoy my great conversation with Scott Wood. Scott, since not everyone listening will know cognition intimately, can you just start by describing what the business is and does? It's quite young. It's made a lot of progress very, very quickly, which is going to be a topic of our conversation in general. But maybe just start by describing for those listening what cognition does. Yeah, absolutely. And thank you so much for having me. We got started actually just around one

Starting point is 00:02:10 year ago. And we are focused on building AI to automate software engineering. The product we're building is called Devon. And Devin is the first fully autonomous software engineer. And what that means is that Devin is basically able to do all of the parts of the software flow. And so you work with Devon the same way that you would work with the coworker. And Devon is in your Slack, Devon is in your GitHub, uses all of your tools and works in your local developer environment. And so you can just tag Devon. You can say, hey, at Devon, I've got this bug that's going on in the front end. Can you figure out what's going on?

Starting point is 00:02:43 Devon will do not just the parts of writing the code, but actually all of the parts around it as well, which includes getting the local environment running, reproducing the bug itself, taking a look at the logs, figuring out what's going on, testing the bug, adding in some unit tests, and then making a pull request for you as the engineer to go review. What's the right way to think about, if I think about Devin as an actual engineer, it's level of sophistication. Maybe you could put it in like salary terms or something or years of experience terms or some other thing. Like help manage my expectations for how much I could have this thing do without any human involved. It's very much a junior engineer today. That's how we think, but that's how we describe it.

Starting point is 00:03:24 I would say right around when we got started, high school CS student or something like that is probably fair. maybe six months ago, an intern, and I think today it's very much an entry-level junior engineer. There's going to be a lot of tasks. I mean, for example, if you wanted to say, like, all right, I need you to re-architect my entire code base, rewrite the whole thing, make it all 10x more efficient, and then I'll check in in like a week. That's probably not the right way to use Devon. Whereas actually a lot of it, it really does function, turning every engineer into an engineering manager, where you get to work with your own team of Devons and handoff tasks. And so a lot of these little things which you would want to delegate and spread off, you would say, hey, Devin,

Starting point is 00:04:02 take a look at this, take a first pass. Maybe you have to fix a few things or maybe Devin asks a few questions that you have to answer or things like that. But the whole idea is that you're working asynchronously with your team of Devin's while you're doing your own stuff or doing things separately. Should I think of Devin as a singular or a plural? The way that it's all set up is we bill by usage. So we have a plan. The plan is 500 bucks a month for engineering teams by default. and then obviously as you use more and then we build more. And the whole idea with it, obviously, is that you can have your own army of devons. And each devon is its own thing.

Starting point is 00:04:35 And let's say you have five different bugs or issues or something that you're trying to work on today. A lot of our users do this where you just spin up five different devons and each one is tackling a different one. And obviously, they ask you as they have questions or put up code for you to review or things like that. But now you're actually jumping between your different devons and tackling these tasks in parallel. It seems like GPTs are GPT's general purpose technologies. And one of the cool things about general purpose technologies is it's very hard to predict how the world will use them to build other things. Now that you've had Devon Live for a while in private and now generally available,

Starting point is 00:05:10 what has surprised you most about the ways in which people use it to do things? A lot of the immersion use cases are things that we wouldn't have predicted ourselves. One of the first things that companies that we work with have been using Devon4 actually is a lot of these migrations, repletforms, modernization and version upgrades, things like that. For example, we work with bangs, we work with healthcare companies. And many of these folks have code that is literally decades old. And they would love to be on the latest stack, on the latest technologies, you make things faster, make things more efficient, it would make things more secure. It would make things easier for their developers to work on as well. But it's just such a pain, as you can imagine, if your codebase is 30,000 files to actually go through and do this kind of modernization effort.

Starting point is 00:05:55 And so it's the kind of thing that's very tedious. And it requires a little bit of active thought and effort to get into. But it's obviously there's just so much volume of it. And that was one of the first things where we really, really saw folks using Devon. A lot of our customers did their own internal studies of the usage. And they were finding that every hour of an engineering's time using Devon was equivalent to eight to 12 hours actually without Devon. So what that looks like is you're just asking questions and reviewing the code rather than having to write the code yourself. And so some of these usage patterns where a lot of the kind of more routine tasks that needed to get done, Devin was just really able to do in one fell swoop.

Starting point is 00:06:29 Can you explain the key differences between those innovating in the IDE like a cursor or something that just make it much easier for a programmer to program versus something like Devin where it feels more holistic like I'm handing off a whole thing and coming back and getting a complete. piece of work. We've been doing this, call it, this agentic approach, like a full automation, ever since we got started. And the way that I described it, I think the biggest step function change is about asynchronous versus synchronous. So a lot of these code assistant tools to call that, which you mentioned a lot of great examples, really great tools, honestly. What happens, obviously, is you use language models to do it auto-complete or you have simple things where write certain files or functions for you or things like that. And if it writes the right things, You hit tab and confirm those or confirm the suggestion that gets given.

Starting point is 00:07:20 It saves you that time of typing. I think in general, what folks have seen with those kinds of technologies is they see something like a 10 to 20 percent improvement across the board. So all of the different software work that they're doing, it makes them a 10 to 20 percent faster engineer, basically. By the way, engineering is huge. That's not a small number at all. But with Devin, what we kind of see is there is a certain level of task, obviously, that Devin

Starting point is 00:07:43 is capable of taking on. You're not going to ask it to do the entire re-architecting or something like that. But for the tasks that Devin can do, it's more like a 10x rather than a 10% because it really is you just handing off the task to Devin. And you go ahead and do your own stuff, you work on other things. Maybe you're running other Devin's as well. And in the meantime, Devin is really just a co-worker that you've given the task and works on that and gives you a full review and a full piece of code. There's a lot of meaningful differences in the product experience that all stem from that. So Devin has to have the ability to test its own.

Starting point is 00:08:15 own code, to run things live, to go and look up documentation, to be able to pull up websites and stuff by itself. You have to be able to talk to it in your GitHub or your Slack or your linear or whatever technologies you're using as a team to communicate with each other. It's quite a different product experience. And honestly, we've seen from folks that it's complimentary. We use the assistant suite of tools as well. And then we also use Devon. A lot of folks use both. Because of where you sit and your own personal experience, I'd love to use the opportunity of chatting with you today to really talk about the past, present, and future of software engineering. I would love to start with the past. Maybe, you know, in my mind, it's this growing abstraction layer

Starting point is 00:08:52 where it gets simpler and simpler and you have to know less and less things down close to the bare metal itself of semiconductors to do what you want to do in a computer system. Maybe you can describe your perspective on the history so far to this point of computer programming and like the major milestones that we've reached before we talk about today and then the future. Well, I love the way that you actually just described it, which is, I think software engineering, this whole time programming or whatever we want to call this, has always been basically just the art of telling the computer what you want it to do. That has been the goal. It is really that simple. And it's kind of funny, obviously, because the form factor that has changed so much

Starting point is 00:09:26 over time. And obviously, like a long, long time ago, it was all about punch cards. But even before that was about these gas tubes or things like that. That was programming. And then there was assembly. And assembly, that was programming. There was basic and there was C and these other languages and frameworks and technologies. We had cloud, we had mobile, a lot of big broad shifts in the technology. I think the undercurrent that's going through this all is over the last 50 to 100 years, we have been able to automate so much of the processes that we need in our economy. And automating that is essentially, it boils down to being able to explain to the computer exactly what you want the computer to do for you. Really, really fascinating to study a lot of

Starting point is 00:10:08 the history of it because I think often it actually was the case that many of these technologies, at the time they came out, they were very interesting technology. And then there's a question of how do you make it useful? What is the use case? And then sometimes it was kind of surprising. I mean, probably the most fan's example that folks know about is the ENIAC back in the 40s, which was the first real automated computer. And the initial use case, obviously, that really got it there was missile trajectories, ballistics. And all the physics said you want to calculate with that, if you want to be very, very accurate and doing 10 digits of precision or something, you don't want to be doing that by hand. With punch cards, for example, the first real

Starting point is 00:10:41 big thing that really caused that to be valuable was the U.S. Census. And that was the thing that needed to be automated. The first Macintosh, for example, like a lot of the initial use case actually was for graphic designers, actually. And then number two was actually the spreadsheet was the killer app that made it useful. With each of these, it was this technology that came about, and then it took some time to really make it useful. We're seeing a very similar paradigm here where language models came out, and obviously they took the world by storm. ChatGBT, in November of 2022, almost exactly two years ago, it was the moment when everyone realized what was possible. And now we're seeing what are all the natural applications of that. And I think

Starting point is 00:11:17 one of those obviously is software itself. Accelerating software itself is just this recursive thing where you get such massive continued gains. You, I don't know what age you were. I think you were 17 or something. Won the competitive programming challenge. So literally you were the best programmer alive or at least in some major way. At that time, and maybe today, what distinguishing is, you hear about this idea of the 10x or 100x engineer. What are the qualities of those people that make them better than others? In those competitions, what is the competitive frontier? What are you trying to best your opponents at in that field? Well, thank you so much for the kind of words. There's these competitions, which are kind of more like algorithmic problems.

Starting point is 00:11:59 There's software. But in all these things, I honestly think the soul of programming the whole way through, even as we're saying, it's changed in form factors so much. I think the soul of it is really just being able to understand, deeply understand these abstractions and just work at this abstract level. I think that's true in software today, where the sole of software engineering is the part of taking a customer's problem or your own products problem or something like that and really just figuring out, okay, here is exactly the solution that I need to build. Here is how I'm going to represent it and here's how I'm going to architect it. Here's all the cases and the details that I need to be thinking about. Here's the flows maybe that I'm going to be setting up

Starting point is 00:12:34 in my website. Here's the database schema that's going to be. to make this all work and make this efficient or something like that. And the thing that's kind of funny about it, obviously, is that is the part that every engineer loves about software, you know, and it is also, I think, as you're saying, it's a thing that really distinguishes somebody from being a 1x versus a 10x versus 100x. At the same time, it's probably only about 10% of what the average engineer does. Because the other 90% of their time, obviously, is debugging, implementing, setting up

Starting point is 00:13:01 testing, dealing with their DevOps, figuring out what went down with their database, was that just crashed or all of these kind of various things, which look a lot more like implementation than this creativity. But I would say even despite that, the creativity and the problem solving is actually really what defines programming. In those competitions, how did you get better? Maybe just walking through the competitions themselves, how they were structured, what you did to try to win, how you won. I'm really interested in that chapter of your life. The competition was called the I-OI, the International Olympiad of Infomeratics. It was very much the Olympics of programming. So every country.

Starting point is 00:13:36 sends their top contestants. There's gold medals, silver medals, bronze medals. You qualify at your local level and then you make it to the team selection at your country and you qualify. And yeah, I mean, that was my entire life growing up actually. I'm from Baton Ridge, Louisiana. They're not that many people who are into competitive programming in Baton Ridge, Louisiana. But I just really, really loved it. My older brother actually did it first. And I got super into it through him. And it was the number one focus of my life for the first 17 years of my life, honestly. At the core of it, It's really just problems solving. And you have code as the medium by which you build the solutions and all the things

Starting point is 00:14:10 that you're designing around and so on. But the core of it is really whether you can figure out these algorithmic problems and understand the right approaches. I wonder if you could sketch for us what your best guess is as to what will happen in this world in the next five to 10 years. Computer programming writ large, obviously you're directly trying to affect that trajectory and what the answer is. But you also occupied this cool seat with maybe five or ten others.

Starting point is 00:14:34 at the very bleeding edge frontier. So not only are you pushing that frontier, but it probably gives you the chance to see a little bit further, as you look out, what do you see? What do you think, maybe you could draw confidence intervals or something for us,

Starting point is 00:14:46 best case, worst case, or something? But if you think forward five years, where might we get in terms of what's possible with engineering? I mean, for one, you're asking about competitive programming. The best programmer in history is actually a guy named Gannati Korokkavich,

Starting point is 00:14:59 he's got from Belarus. And I would imagine that we have AI that can beat Ganadi in the next one or two years. Beating the best programmer in the world at this programming competition, I think that is going to happen. And I think it's going to be like an AlfaGo-style moment where it's obviously like, holy shit, like this is truly there. The tech today, honestly already is folks have been putting out things.

Starting point is 00:15:19 Google put out something on the IMO, the Math Olympiad, and Open AI put something out on the I-O-I. But they're already at a level where they're competitive with some of the best competitors in the world. Getting to number one is honestly not too far from that. I think what happens to software engineering as a whole is I really come back to this point of what we're saying that all of programming, it really is just about telling your computer what to do. A lot of this work that's happening with AI and AI agents as truly just the next generation of human computer interface. And so what I mean by that obviously is if you think about a programming language, let's say Python, for example, perhaps the most popular language out there.

Starting point is 00:15:55 Python is great, it's beautiful and everything. But if you really just look at it, it is such a stark compromise that you have to be making for both. both sides. From the human side, obviously, you have to learn how to code. Ninety-nine percent of people in the world don't know how to code. And that takes years to learn all the little details, and here's how you do like a Lambda function, and here's how you do this little particular thing. And these are all the things you have to learn just to be able to talk to your computer. And from the computer side, Python itself is notoriously actually slow and inefficient and so on, even compared to other languages, but obviously compared to how fast you could truly

Starting point is 00:16:28 run things on bare metal assembly, if you knew this is exactly what I want to do, you can make all of the systems honestly 10 or 100x faster. And the way I think about it is Python is maybe the best language out there by a lot of people's definitions. And really it's this awkward compromise that we have to make because this is the only way that humans and computers are really communicating today. One of the great things I think that AI will really solve is essentially this translation problem where a human, any human, doesn't have to be somebody who is trained in code or something can describe precisely what it is that they want to build. And the computer will build the efficient and feature-complete version of that that actually

Starting point is 00:17:06 just works and does all these things. It's going to take some real time to get there, obviously. I think that's the future that we're headed towards. Meaning, just to use a simple example, I using natural language, want a very, very specific application to stick on my phone or something that maybe within five years will be at a point that me with no particular CS experience can get exactly what I want in a way that works with high fidelity without having to rely on human programmers? Do you think that's feasible? I think that's right. If I had to put down a number right now, it's obviously very hard to predict.

Starting point is 00:17:39 I think five to ten years is like the reasonable realm for that. One point I would make on that, even the way that we talk about applications, we kind of assume, obviously, that there's some application that has to generalize. One way to put it maybe is software engineering is so, expensive today. It is so, so expensive that the only software that you can actually build that's worth building is basically applications that reach millions of people. It's like, okay, it was worth it to code Facebook because of how many people use the exact same software every day, YouTube or something or DoorDash or whatever. And obviously, there are somewhat more niche things, but there's honestly lots and lots of things out there where it would be better

Starting point is 00:18:16 served with code. There's communities of, call it a couple thousand people. There's little groups or the little niches or things like that, there's personalized products even for one person. And so whatever your day-to-day chores are, they really are better served with code. That code can do all of the things that you might need to do and just make all the execution of that much cleaner. The reason we don't have that, obviously, is because there's not really anyone in the world for whom that makes sense to write custom software just for their own personal use case every time they want to do something new.

Starting point is 00:18:46 But I think we will get to that point where you are just able to execute these things. And maybe it's code on the back end, but from the perspective, of us, it's really, yeah, just explaining to the computer exactly what we want and seeing that happen. There's going to be a lot of middle steps, but yeah. I'm curious what you think about one layer down or up, depending on how you think about it, which is the same question for agents specifically and whether or not all of these systems are going to cause applications themselves to be less common. And agents, we interact with through voice or through text or through other means, to do more and more of the things that we currently use applications to do. Since you're building,

Starting point is 00:19:22 an agenetic system, I'd love you to just teach us about agents. Everyone's saying that now. You hear the word agent in every single call. I feel like that scene in Seinfeld talking about write-offs. You don't really know what that means, do you? Tell us what agents means to you and why it's such an important construct in this coming age of technology. Like we were saying, GPT came out two years ago. It's huge surprise for everyone. And I think for a while, all of the products that came out of that were basically what I would call these text completion products that work off of that. And by the way, it makes sense, obviously, because the native form of a language model is put in this input text, tell me what the output text is. But that's kind of where you saw every application at first.

Starting point is 00:20:01 GBT itself is obviously like that. You had a lot of Q&A products, marketing, copywriting, hey, I need you to write this paragraph for me that explains X, Y, and Z. Can you just put together a nicely written thing? Boom, here's the output. Customer's asking something. And then it's like, what would you say here? Here's all the information you need to know. Boom.

Starting point is 00:20:18 Code, that was the same thing, too, right? where the first big code products, I mean, GitHub Copilot is probably the biggest of these, is a text autocomplete where it's like, here's the file so far, predict me the next line, and if it is correct, I'll just hit tab, and it'll obviously save me the time to type the line. It doesn't save me the time of thinking of the line

Starting point is 00:20:34 because obviously it's still me that's deciding what's correct and what's not correct, but it does save the time of typing the line. And I think what we're now seeing and what everybody is talking about now with agents, although, of course, it's very early, is this new paradigm where models can actually act and make decisions in the real world.

Starting point is 00:20:50 There's a lot to it, and it's how humans do it too. It's not like a human just produces this perfect paragraph. Here's everything that I needed to give you, and then you're all done. A lot of it is going and interacting with the real world, looking up websites or something, trying out things, seeing what works or what doesn't work. In the case of code, obviously, running the code and looking at the output or looking at the logs and things like that.

Starting point is 00:21:12 And I think maybe the simplest way to put it is in all of these verticals, I think we're going to see this where basically, hey, having a legal Q&A bot is great, but the next step is to have a lawyer. Having something that can answer questions about math problems or something that you have is great. The next step is to have like a full teacher. And obviously, in software, it's the same thing where it's like having the text auto-complete for coding does make you faster yourself as an engineer.

Starting point is 00:21:36 But what if you had a whole engineer that you're working with? That's kind of the core of this adjective work. Maybe you can describe then how literally Devin is built. Behind this question, I'm also curious about taking. your experience as advice for other builders who are using core foundation models as a key part of the thing that they're building and building in such a way that is complementary to those underlying foundation models and won't just get subsumed by them. Everyone asks about this. I'm sure every VC asks every founder like, how well, oh one, not just do this. So tell us about how Devin is built

Starting point is 00:22:10 and how you think about building with and alongside these evolving foundation models. We are 100% laser focused on solving all the problems along the way of software engineering. And the truth is software engineering is just very messy. My point with that is just that there's this idea of like solving these math problems in a vacuum, for example. There's a lot of research that's going towards that. But there's also this entire thread between of how do you take that IQ and turn that into something that is actually meaningfully useful for software engineers in their day-to-day work? So there's a ton of stuff that needs to be done on adapting to the capabilities for a particular. particular tooling and particular stack. And then there's also a ton of stuff that has to be done

Starting point is 00:22:49 on the product side as well. I'll just give several examples of that. A simple one is, obviously, Devin has to be plugged into all your tools. Solving problems in a vacuum is great, but Devin has to work in your GitHub. It has to be working in your codebase. It has to use your confluence docs or whatever it is. That's where the information is. It has to read the bugs on Jira. And on top of that, obviously, Devin needs to learn with the code base over time. And so even a really, really, really sharp software engineer. They join your company on day one. They're probably not to be all that productive, at least compared to somebody who's been there for five years and knows the entire code base in and out. Devin needs to learn the same way of understanding, hey, here's what this file does,

Starting point is 00:23:27 here's what these functions are for. If I want to do the scam of migration here is step one, two, three to get that done and all of that. And then there's obviously all of the actual step-by-step decision-making that goes into it, which is somewhat different. We were talking about fixing a bug, I think a very natural flow with fixing a bug is, okay, I got a report for a bug. First thing I'm going to do is I'm going to try and reproduce the bug myself and see what's going on. I've got to go run all the could locally and try to make the bug happen. If the bug does happen, I'm going to go and take a look at the logs and see what the error was or what happened exactly what the variables were.

Starting point is 00:24:01 And then from there, I'm going to go read the relevant files. Maybe you can make some edits, test things out, and whatever. And then I'm going to go test it again and see if it works. And if that all works, then I'm going to add a unit test to make sure it doesn't fail again and whatever. And all this step-by-step decision-making, it's a very different problem, obviously, from text completing what would somebody really smart on the internet say. How do you use the tools available to you and make a sequence of decisions that maximizes your chance of success? I think the other question you asked was how, as a builder to think about this in relation to,

Starting point is 00:24:31 obviously, this steady stream of improving models. It's a really interesting one because a lot of other technologies that we've had have been a lot more step-function. mobile phones came out. And the first question was, what are you going to do when everyone has a smartphone in their pocket? In some ways, the core of Uber or Airbnb or DoorDash or all these things.

Starting point is 00:24:48 And you can kind of think about, okay, this is the technology shift. And let's think about what happens over the next coming years now that we have this shift. AI is a lot more of a very incremental thing. Here's GPT3 and 3.5 and 4. And each little technology gets a little bit better over time.

Starting point is 00:25:03 And part of what that implies, I think, is just really thinking about what are the models not going to do by themselves over time? And a lot of the stack and a lot of the things that we care about, obviously are around building capabilities and the product experience specifically for software. I have no doubt that the foundation models will continue to increase in IQ and increase in IQ and so on over time.

Starting point is 00:25:25 But a lot of the detail of all of the tools that you work with, how you make decisions with all of that, even how you communicate with the user. What is the user interface where you're talking with Devin and giving feedback and seeing what Devin's up to and checking in and making sure nothing's wrong and all these things. There's a lot, a lot of detail, obviously, in that flow that really brings out the inherent IQ, to put it that way.

Starting point is 00:25:46 We've always thought about it in that way of how do we make sure everything that we're doing is complementary with the model increases. And by the way, complementary with everything else. The hardware is getting faster all the time. The whole stack is getting more efficient everywhere. And we have our part of the stack and we have always thought about it as how do we do as well as possible in our part of the stack specifically for software. What improvements are you hoping for from the foundation model companies that would make Devin more effective than it is today in the future?

Starting point is 00:26:13 I should mention, by the way, the foundation model labs are all super, super excited about agents, and it makes sense. I think one thing, which is kind of interesting, just practically, is let's say you asked GPT a question, who is the fifth president or something. You ask one question, you get one answer, and that's a single model query. And obviously, there's a consumer pricing and everything, but if you're going by the API pricing that costs $1 million, of a dollar or something like that, something ridiculously low. Whereas if you're asking Devin, hey, I've got this new feature that I need you to build out. Can you build it and test it and make sure it looks good? Over the course of that, Devin's probably going to be making hundreds, if not thousands, actually, of decisions over the next half an hour or something while it's going and

Starting point is 00:26:51 building that out. And it also just practically means it's actually just a lot more inference on GPUs. And so for that reason, what we'll see over the next while, honestly, even the next one year, I think agents are going to become a lot more widespread. And what we'll see actually, is I think the dominant usage of these GPUs, of these language models and everything will be for agents, largely because every single Asian query is just so much more magnitude of underlying language model calls than a single Q&A query. And I think what happens next with that, which I think these Foundation Model Labs are all thinking about, is, yeah, how do we build our models to be as optimized for agents? And so I think a lot of these things of working with multi-turn traces, being able to handle long content,

Starting point is 00:27:34 context outputs, for example, like context is a big thing that people are spending a lot of time thinking about. And with an agent, again, it's like, if you have half an hour of decisions, obviously, it helps a lot to know what's been going on in the last half hour. So some of these particular technologies that will really help with that. How do you use Devin in your own building of Devin? We use a lot of Devin, actually, when we're building Devin. It really is this new workflow, a very asynchronous workflow. And so I'll just give an example. We do all of our communications in Slack. We have a few channels. We have a crash. channel where every production crash gets reported.

Starting point is 00:28:07 We have a front-end channel where we talk about any like front-end feature requests or maybe like, hey, let's switch the flow around this way or, hey, this design is a little bit off or things like that. Front-end things that come up. We have a bugs channel where users are filing bug reports of, hey, this thing didn't work or something or this thing broke or whatever. In all of these channels, one thing we do every time something gets reported, we tag Dev.

Starting point is 00:28:30 You can just go through the channel. Hey, this thing's broken. At Devon. Hey, somebody requests. to this at Devon. Hey, we just had a crash at Devon. Devon's not perfect. Devin's a junior engineer and often does need a lot of guidance. And I would say probably about 50% of the time, Devin's output on one of these is just like a pull request code that's straight out of the box ready to merge. And the other 50% time, obviously you can make the call of, okay, do I want to

Starting point is 00:28:52 talk with Devin and coach Devin through this? Do I want to go and make some changes myself and just touch up whatever needs to be fixed? Or do I want to just go start from scratch? If it's especially far off or whatever. But having this be part of our workflow, you had to report the bug anyway. And so you might as well to just tag on and at Devon at the end, and then you get a first pass for free.

Starting point is 00:29:11 And so a lot of workflows like this, a lot of it is just thinking about how to put it into the asynchronous workflows. A similar one, honestly, is if one of our engineers is building some big new project and they're going and implementing things or building the IDE, for example, this happens all the time as an engineer,

Starting point is 00:29:26 you're going to go through the code and you're going to see all these little things that are like, man, why did we write it this way? this is so messy or like this thing is technically correct, but it is not the most stable way to do this, or lots of little things. You get caught up in a lot of these little things as you're building your bigger project. It does distract and it takes away your focus. And what we do now, obviously, is you just kick off Devin from the IDE for each one of these. And it's like, hey, can you go refactor this one real quick? Hey, can you change this to be type safe? Hey, order this in a different way so that there's

Starting point is 00:29:53 not going to be a race condition or something. And you're doing all these little fixes and you're kicking them off async. And then you're doing your own project in the meantime. And both these cases, a lot of it is really just figuring out these asynchronous flows. It seems like part of the answer, too, in the near term, you said earlier that maybe 10% of the engineer's time is spent on what a friend of mine calls positive engineering, building things, the creative part. 90% might be on make sure things work, more almost like a mechanic than a creative, productive engineer. So does that mean that in the near term, what Devin and other systems like it are going to help us do is eat that 90% and keep the programmer focused on the 10%?

Starting point is 00:30:29 And then what happens beyond that? If we beat the world's best programmer in two years, why do we need programmers? Will there be programmers? What will they do? So I think it's exactly right, eating the 90% of all the miscellaneous implementation and debugging and iteration and whatever. I think what happens then is, honestly, you just get to do 10x more than 10%. And I think the cool thing about it, we should say it all right. Obviously, there's a lot of fear in AI about what's going to happen and what happens next.

Starting point is 00:30:55 But I think one of the really beautiful things about software is there really is 10 times, more software to go right if we just had 10 times of capacity. Over the last however many years, we've always been bottom-necked by supply. If you talk to any engineering team, I haven't met a single team who said, yeah, we built everything that we wanted to build. There's actually no software wants to do. Every team out there has 20 projects that they want to do. They have to pick three to take on this week because there's just so few software engineers and software just takes so long to build. I honestly think we're going to see a lot of this Jevon's paradox where we're just going to be building more and more and more software as it gets easier. It's what we've seen already,

Starting point is 00:31:29 maybe to a somewhat lesser scale. Building software now versus building it in the 80s, it probably is about 10x easier or so. And this shift will happen a lot quicker, obviously, but over that 40 years, we've gotten a lot more programmers and we've built a lot more software. And I think we're going to see a shift that looks like that as well. I think in the long term, I think the soul of software engineering, again, is each person getting to decide what are the problems that I want to solve and what is exactly the solution that I want to build. AI is not magic. Obviously, it's not going to go and solve all your problems for you, but it will give you the power to specify exactly what the solution is that you want to have

Starting point is 00:32:02 and just be able to have that happen and build that out. Do you think in general that all this means that business people need to start thinking harder about demand and distribution? If we've been supply constrained and that's about to go away, does that mean that the people that are going to win in the future are those with better demand and distribution? I think in software especially, we have always been kept by supply and not demand. Obviously, there are a lot of things that are more fixed in terms of demand. I think there's a lot of interesting questions of what happens to the economy with this

Starting point is 00:32:31 and a lot of interesting second order effects. The simplest way to put it, where does all the money in tech go? Where does the money in venture capital? What does it go into? And it is primarily software engineers. So when you're getting 10x more bang for the buck for every software engineer that you're bringing on, it does change the landscape a lot of how much people can build with products and how much people can do. And then I think down the line, you see a lot of pretty interesting effects with businesses. I see a few different categories of businesses, broadly. If we're being blunt, I think the toughest category to be in is business that's heavily reliant on switching costs and lock it.

Starting point is 00:33:04 A lot of businesses out there, software businesses, that the reason they make all the revenue that they make is because it is so painful to switch to the other products or the other technology or something like that. If Devin is just able to do all that implementation and migration and those costs go to zero, obviously it's going to be a lot more about having the better product. That's going to be a real shift. I think that companies are going to deal with. At the same time, it's not an nihilist view.

Starting point is 00:33:25 There are a lot of things that will be just a strong. if not stronger. Network effects, I think, will always be golden. I think data and personalization is going to be even more powerful. So a business that owns the infrastructure, for example, or a business that has all the personal relationships and the data and so on, being able to serve for each person, an individualized product that is truly optimized just for them, instead of this kunky thing that kind of works for everybody and touches all the millions of people, but is not the perfect product for anybody. I think a lot of those businesses will be able to do very well. What kinds of questions are becoming most common when you spend your time talking to the senior most people at the big established technology companies?

Starting point is 00:34:05 There's definitely a lot of folks asking, how do I think about my engineering headcount and how do I work with this? I think we're very much in the paradigm where people like to say this. AI won't replace you, but someone using AI better than you might. And so a lot of it is how do you get smart and how do you make sure you're on top of the latest technologies and you're ready for the shift, for sure, honestly, going to quicker than most people imagine. and getting your whole team set for that and getting things built for that. I think there's also a special case actually with these technology businesses themselves, folks who are building SaaS products or things like that, where we talk so much about agents.

Starting point is 00:34:37 I actually think one of the kind of interesting product changes is that a lot of these companies are actually going to be serving agents in addition to humans. A lot of great businesses out there that are basically meant for this collaboration and coordination, a lot of interfacing. We talked about companies like Slack or GitHub or Datadog or. or things like that where there's a lot of work in compiling that they do, and also a lot of it is about the interface with how humans interact. I think the way I think about it is right now,

Starting point is 00:35:04 those are really valuable tools for agents to be using as well. Sure, you can imagine this nihilist view where an agent is just building everything on its own from scratch, but the truth is there's a reason that we look at logs, the way that we look at logs. The reason that we set up our documents, the way that we set up our documents, and it does just make it easier. It makes the problem easier of getting the context that you need.

Starting point is 00:35:24 So what will happen actually is we're going to have a lot of agents who are using these products as well. So businesses that are in that space, a lot of folks are actually thinking about how do I make my product ready for my new wave of agent customers? And in addition to all the human customers. I was talking to Martin Casado about these climbing cost curves that technology creates. And the first one that's clearly happened already in AI is like the cost of marginal content creation has started its march towards zero in all different forms of media. It seems like with you and what you're building, there's that same curve story happening in the cost of software itself. How do you build a software business in that reality? If I fast forward 10 years or something sufficiently long, why should Devin cost anything?

Starting point is 00:36:08 Why won't there just be like an open source Devin that does everything for me? And what do I need to pay for software if we're entering an era of marginal cost of creating software being very close to zero? I mean, I think right now Devin is really the only one out there in the market that's doing that. But as you're saying, I'm sure there will be lots and lots of competitors pretty soon, even. And folks that want to do similar things, InC is great for the consumer. We were just talking about the 10x engineers and things like that. The truth is that every level of decision-making quality and every level of skill that you can have in your software engineer actually really does have this exponential reward curve. And it makes a big difference.

Starting point is 00:36:44 And one of the things that we're getting, obviously, is this ability to really, really personalize Devin to each of their customers. And so, like, Devin, at a lot of our customers today, when we work with Devin, for example, it's just like, Devin knows the entire codebase in and out. Devin wrote half the files in the codebase. And so there's all these details. And obviously, it's not just the lines of code. It's the decision making that went into it and thought process and so on that Devin has internalized and working with the entire engineering team.

Starting point is 00:37:08 And it really is that tradeoff where it is about here's an engineer who's maybe smart and sharp and stuff, but it's starting day one on your code base today versus here's an engineer who's been with your codebase for years and years and understands every file and has built all of these things and knows exactly what your engineers mean when they ask certain things and there's a real trade-off there. It comes down to these things where it's kind of like pure technology switching costs, where it's migrating all your files to some new thing or moving to some different platform. Yeah, I think that is going to get cheaper and cheaper over time. At the same time, obviously, if you have the level of personalization, if you have the network

Starting point is 00:37:43 effects or whatever, it's not about the cost to switch. It's really just about you're able to provide a better products. Maybe you say a bit about pricing. This is such an interesting question. It seems like someone at Open AI put their finger in the air and decided 20 bucks was a reasonable price to charge and a whole business ecosystem has spun up and that seems to be the default pricing. You made a very different pricing choice. It's 500 bucks a month. Open AI just released the $200 a month option. So we're marching upward. But talk me through how you thought about and think about pricing in this new world. Yeah, we're going to see more of that obviously. I mean, I think for us, it's funny. We had a lot of questions with our launch, for example, questions of

Starting point is 00:38:22 like how much do we anthropomorphize Devin? How do we explain the structure of it and so on? But I really do think it is a pretty different paradigm where with agents, it does come down to usage for better or force. There's reasons for both the inputs and the outputs for that. But the inputs, like we're saying, I mean, it is just more expensive to run that that is the case. You're doing more queries and more work. There's more GPU compute that goes into each job or reach task or something like that. But I think with the outputs as well, obviously, when you're starting to take on tasks that are real end-to-end tasks, it's just much cleaner and much clearer to measure the concrete impact that something like this has. I think the proxy that we essentially

Starting point is 00:39:02 want to have is value-based pricing, where it's basically we want to make a lot of the projects or the bugs or the features or whatever that you're building 10x cheaper to build. And so we want the cost of Devon to run on those to be 10x less. It's obviously hard to have a perfect proxy for that, but the proxy that we have essentially is usage. We have a unit called ACUs, agent compute units, and it's basically all the decisions that Devin makes, the box that Devin uses, the code that Devin runs or things like that. And it roughly corresponds to basically for every hour of Devin's work that you're doing.

Starting point is 00:39:31 Again, it depends on different machines, but you're paying something like $8 to $12 or something like that. Obviously, Devin can do a lot in an hour. The setup is such that we want it to be $10 cheaper than having to do those yourself. But I think that is going to be broadly the new paradigm when we start to have AIs that are doing tasks instead of answering questions. Any thoughts on this big fun almost has become like a cocktail party style question around whether or not we've achieved AGI or not?

Starting point is 00:39:56 And whether or not that's a question that you care about. I'm curious what you think about this one. I'm in camp number two on that. We already have AGI. Someone said this in October, in 2017, you ask if we have AGI, the answer is no. In 2024, you ask if we have AGI, the answer is, well, you got to define AGI. I think there is a somewhat recursive definition of AGI, obviously, that people like to say, oh, it's doing 80% of human labor.

Starting point is 00:40:20 It's like, well, if you could do that, then the humans would be doing the 20%, so it wouldn't be the 80% anymore. There's some recursiveness. But the way I would say it practically is I think there's going to be a tale of things where we will push the capabilities, and we'll push the capabilities. And for a while, there's going to be lots of little things that are like, well, actually humans are still better at this one thing. And then we'll solve that thing.

Starting point is 00:40:40 And then people will be like, oh, well, humans actually can say, do this one thing. And we obviously have a lot of pride as humans and our own species of wanting to find the things that make us unique and so on. But I think that like practically, what matters for the world, obviously, is, is the AI good enough and is it built into products and distributed well enough that it's actually affecting the world and affecting the economy and affecting people's day-to-date lives? In some sense, that question matters more than is it doing 95% or 96% or 97% of the humans' workflow. It's more a question of, are humans actually out there doing 25 times more because of the AI

Starting point is 00:41:18 technology? And I think that actually is much more of a practical question. And in some sense, it's like we're talking about the IQs and things like that. The math problems that these base AIs get solved already, honestly, shockingly good. And among humans would be in the upper echelons of humans, if not quite number one in the world. And I think the biggest distinction is actually not whether we are number one in the world or only top 0.1% human or something like that, the distinction is a bit more of, is that level of intelligence actually used by every human out there in the world and done in a way that's

Starting point is 00:41:50 actually giving them a lot of value? That's going to be the big shift that actually gets us to the massive GDP growth, massive efficiency gains, quality of life, whatever you want to call it. Investors mostly are obsessing over scaling loss because it just has such big implications for public equities, for companies building new products in the private markets. And so that's where they tend to focus. If you are training your attention on what's going on in the foundation model world, what do you spend your time wondering about, asking your friends about learning about, where's your eye trained in that world? Yeah, my hot take from this is I think scaling laws are a little bit of a myth. I think we have made continued progress on these models

Starting point is 00:42:28 and push the capabilities and things like that. And there's been a lot of new technologies, obviously, that we figured out. And there's way better post-training and there's some of this stuff, obviously, that's going on reinforcement learning and things like that. Sure, there is return to scale, but I think a lot of what we're seeing is basically it is also the introduction of new technologies that are making this more efficient or allowing it to grow with scale beyond that. And so people sometimes like to talk about scaling laws as if you just sit back and as long as you got the hardware for it, it's just going to get better and better and better. And I think really, obviously we have needed more and more hardware, but it's also been

Starting point is 00:43:00 innovations that have actually gotten us there. I think practically with foundation models, we always obviously want to have a pulse on what's going on and what things are going on next. We work very closely with all the foundation model labs. We do evaluation of early models with them. We do custom training of models with them, things like that. It's like we're saying it is an incremental thing. These things will get better. And a lot of the question as a decision maker is to basically just have a very clear sense on what are the capabilities that will get better and what are either the capabilities or the product or the human experience of it that will always need to complement these for you to

Starting point is 00:43:36 deliver something of value to your customers. I'm really curious to learn a bit about the formative experiences for you as a business person. Obviously, you had, we've talked a lot about formative experiences as a technologist building cognition and also I-O-I and these other competitions for the first part of your life. I'm sure everyone's seen the video of you when you were 12 or something, winning math competitions. Obviously, you've had some pretty cool formative experiences. What have those been on the business side? For example, I'm interested to hear about your experience with lunch club and what you were building, why you were building it, and what that experience or that chapter of your business career taught you?

Starting point is 00:44:12 Yeah, Lunch Club was my first company. I dropped out of school, actually, to start the company. It was a great experience, first of all. There's an AI for professional networking. We made millions of meetings happen, and it was super fun team and product to build. I would say one of the big things, honestly, that I learned is, it sounds simple, but it really does come down to the fundamentals.

Starting point is 00:44:30 I feel like the common pieces of startup wisdom that everybody talks about, In some sense, they're so obvious that it's not obvious. It's not even worth mentioning people feel that way where it's like go as fast as you can. Everybody says that. Everybody knows that. Never compromise on recruiting. Stay as close to your customers as possible.

Starting point is 00:44:47 Focus on building a product that people love and want. These are almost taught logically true. But the thing is, the reality is there's always another gear that you can kick into with your fundamentals. And that's how I've really thought about it now, having gone to see it. Simple examples, like everybody says you. can go fast. In our case, we incorporated the company in January. We were building the first versions of the products. We got it out to the first group of initial users in February. We did a

Starting point is 00:45:13 launch in March. We raised a big round in April. We did a big partnership with Microsoft in May. While we were doing that, we got on a lot of enterprise customers. We grew the team. No matter how fast you're going, and every startup in the Valley, obviously, thinks they're going quite fast, it's still worth it to really push for even faster. And no matter how close you are to your customers and how deeply understand their problems, you're probably not going to regret spending even more time with them and understanding that deeper. And no matter how high your bar is for hiring,

Starting point is 00:45:41 you're probably not going to regret trying to go for folks that are even higher still. And I think that is the biggest lesson I've learned with startups. The competitive landscape, does it feel like this big, exciting greenfield, or does it feel like battle lines are being drawn and insanely competitive for every available opening and, you know, sharp elbows. Characterize what the playing field feels like right now. You all work out of this amazing house that I've been to. It's a cool environment, which feels so startup-y. And yet the next day you're with Microsoft, the contrast here of real true startups in startup houses and raw cultures

Starting point is 00:46:20 that are emerging, they're interacting like you are with the biggest players in the world of technology inside of a few months. It's just like a bizarre set of circumstances. How would you describe the playing field to an outsider? I think it's still very green field. I mean, the technology itself is so early. We're like one or two years in, in some sense, to all of this stuff really being valuable and getting there. One thing that I think about a lot is software is just so big.

Starting point is 00:46:47 We obviously know that, but there's so many facets of software and so many details and so many different things that can be done and can be built. And so one way that I think about it is each of the use cases that we're talking about, there are enormous and really, really meaningful businesses that are built and just do that use case. A lot of folks are talking about AI for data observability, for example. Datadog is a great business. It's making billions of revenue.

Starting point is 00:47:09 It's something like a 50 or $60 billion market cap to do it very, very well and growing quickly. And as you can imagine, there's so much to do with AI for data observability in some form. Incident response is a huge thing that's only going to get bigger and bigger. And there's a lot of great businesses in the space like Patriot is a great business. But AI is going to make these things again, orders of magnitude bigger. It's true with testing.

Starting point is 00:47:29 It's true with modernization products. truth. Migration, a lot of these big guys like Microsoft, Amazon, Google, or anyone, the amount of effort they have to go through to migrate customers onto their platform, even today, is enormous. It's billions and billions of dollars every year. So all of these are just such enormous spaces, and there's still so many questions even about the product form factor. And as the technology gets better and better and better, the kinds of use cases that we'll be able to support as a space are only going to get bigger. What do you think the world is overestimating and underestimating right now about your whole world, about the automatability of software engineering?

Starting point is 00:48:06 The speed of it is probably a big one. One thing I always think about, if we're just looking all the way across AI, for example, I think agents in general are going to really work everywhere. Code is one of the first for a lot of reasons. But I would almost argue one of the first agents, if you want to call it, that actually was self-driving cars. It's an actual AI that's making decisions in the real world and taking in feedback and iterating with that.

Starting point is 00:48:27 People are familiar with the story of self-driving cars where I actually lived in Mountain View in 2014. I would see the Waymo's driving around all the time. I never got a driver's license. I had just gotten started working at the time. I'm not going to need one. They're basically there. I'm sure I'll be able to use it. We did finally get there.

Starting point is 00:48:44 I think there's still a lot of room to grow, obviously, but it did take a lot of time. I think the difference between self-driving and code and honestly, a lot of other applications I think we'll see over the next year is, I think with self-driving, you really, really need to have it be a 99.9.9. 9999% sufficiency. And there's not a world where it's like, or there is, but it's not as big of a difference where it's like human and machine combined

Starting point is 00:49:06 can do a lot more together. And I think that's people's typical example. It's like, okay, you know, there's going to be all the edge cases to figure out and all that, which is true, by the way. But the nice thing with software, for example, is there actually is just a really great way

Starting point is 00:49:18 to do human plus machine and do a lot more. That's the case with Devon. Devin is not the 99.999% solution. It looks a lot more like the 2014 Waymo or honestly, probably even a little earlier than that. But the thing is, having Devin take a first pass or having Devin send you the code and you take a look and give a review or whatever it is like that, where Devin is doing the 90% of you're doing one nine instead of six nines, it's still super, super useful. It actually does save you 90% at the time, whereas you're not going to get into a car

Starting point is 00:49:47 that is only 90% consistent. And so I think there's going to be a real form factor difference in how soon we're going to see it actually really transform practical software, not just because of the capabilities, but also just because the product space is better set up for that. What was the lesson that Lunch Club taught you about the world of entrepreneurship? Was it something around market? Was it something around product? If you think back on that experience, what's the big takeaway and how are you applying that learning to what you're doing at cognition? Probably the biggest one that stands out, honestly, is sometimes it's easier to solve a bigger problem than it is to solve smaller problem, which is quite. And, you know,

Starting point is 00:50:27 people folks talk about this. If you're going after something that's truly enormous and it's the thing and it's the most exciting thing, you'll be able to bring together a group of people who's all really excited to make it happen and is really passionate and dedicated to pushing the frontier in a way that actually sometimes make it easier. Our team, for example, our team is about 20 people right now. Of those 20, I think 14 of them actually have been founders themselves before. And it's an interesting one where we've all done companies and honestly, it's a very important. really, really talented team. A lot of these folks would get a blank check to raise money if that's what they wanted to do and they wanted to do their own thing. But a lot of it is just how do we

Starting point is 00:51:05 come together and build something that's really great. I think that's one of the big lessons for me. What has been the most difficult or stressful moments so far in the one year cognition journey? Oh, there's been a lot of it. Let's do a few then. I love these. One of my favorites, all of these launches, you know how it is with startups, obviously. Things are so last minute. So we had the GA launch this week, and then we obviously had our initial product launch back in March. And for both of those, the few days leading up to the launch

Starting point is 00:51:35 were some of the most stressful, but just the most memorable experiences. And our launch in March is huge. There's so much you have to do to get into it. And obviously, there's the product itself and testing with early users and making sure everything is truly working the way that it's supposed to, getting the info ready for it, setting everything up. And then there's obviously all of these other things with filming the VIII,

Starting point is 00:51:56 video, think about the blog post, think about the content, getting things out there, working with customers to get testimonials and things like that. And our March launch, actually, we were doing all of that in addition to, we're doing a fundraise right before the launch as well, actually. And so we were doing that. We had candidates who were in the pipeline, pretty meaningful candidates that we were working on and figuring out and everything during that as well. And so that weekend was a very fun time.

Starting point is 00:52:17 That launch was actually, there was an article that went out by Ashley Vance, Bloomberg article about our company. And that was the article, obviously, that marked the launch. It was supposed to go out at 6 a.m. Tuesday. We asked them to push just a little bit later. We got it to be 9 a.m. Tuesday instead of 6 a.m. We needed those extra few hours. We weren't done filming the video for the launch as of that night at 1 or 2 a.m.

Starting point is 00:52:41 We were still recording lines. And then there was all the blog posts and putting that all together. It was a crazy weekend. Yeah. They did not get very much sleep that night, basically. It was just a really fun moment to get together and really do everything together. It's one of the reasons actually, I think, that us all just sleep. living and working in a house together is great. It's because there's something about the shared experience,

Starting point is 00:52:59 yeah, sharing the late night energy and everything. I feel like it wouldn't be quite the same at an office. And yet this launch as well, you know, actually, we were all in Utah, which was kind of hilarious, actually. The last minute decision, it's like, okay, we're going to Utah. And we got on a plane that day. We found a cabin that would fit us all. Got into that and we were all just working on the launch. And it was the same story, always. Even having done this, we got no wiseries. And so we we were still filming at 2 a.m. the night before and putting things together for the launch. And getting all the content ready and getting the info ready. And obviously, in this case, it's a full GA thing too.

Starting point is 00:53:29 So we got to be ready to handle 10X load or even more than that. So all these kinds of things that we had to do to get ready. And those are some of the funnest moments, honestly. I'm just extremely interested in launching things. I think there's so much rolled up into it that it's like business encompassed or something. Can you tell me the best and worst part about each of those two launches in retrospect? Honestly, the experience with the team is the best part. just getting to build it together and do it together with the team. It really is the soul of

Starting point is 00:53:56 startups. Yeah. The worst, I mean, there's scares and stuff that happened all the time. The night before, it's just like, hey, we did our load testing. This shit didn't work at all. All the info went down. There's no way we're able to handle the load. We were expecting to handle this many concurrent Devon sessions per minute. This did not work at all. And the last minute things we have to do. The onboarding flow, for example, last minute changes that we had to make. My co-founder, Stephen, actually put out a post talking about how we shipped a whole thing where you could do a Slack connect with us and work directly with our team. That got started about 1 a.m. the night before the launch and shits the morning of the launch.

Starting point is 00:54:30 And Devin built a lot of that because obviously we needed the extra bandwidth from Devin. In the moment, every single thing that you're optimizing for is how do we make this not incredibly embarrassing for us all? And sometimes you just have to put it out there and do your best. How do you think about something like 01 Pro Mode and people using that to do coding? versus Devin. Yeah, I think this tools are great. I think a lot of companies are thinking about code and are focused on code. And like we're saying, code is such a big thing.

Starting point is 00:55:00 There are a lot of cases where, yeah, sure, it's like you want to build something, a single file of code out of the box and you want to think through particular problems or whatever and have that be a really great single file of code or something where, yeah, you can just go to O1Pro and consult it or an AI stack overflow, if you will, where it's like you can paste in an error and it'll give you its best guess from what it has. here's what I think the error is. With Devin, there's a ton of just these iterative flows that make a lot of natural sense. And so, for example, if it's not just writing a single file and that's all,

Starting point is 00:55:31 but it's like, hey, I need you to work in my codebase and build a new feature and plug it into all the existing stuff, but test it out to make sure that it works, then obviously all of those iterative steps, those are the things that Devin is doing for you. And similarly, if it was a debugging thing, like an AI Stack Overflow thing, for example, Actually, one of our first Devon moment, I want to say, is we started building this as more like research type technology and just trying to understand multi-turn agents, iterative decision-making and stuff. And eventually it made sense to build into the Devon products. But one of the first things that really convinced us is we were trying to set up our database for our own work. We were setting up MongoDB and anyone who's done it knows what this flow looks like, basically, where it's just give you a set of commands, you try it, and then it doesn't work.

Starting point is 00:56:13 And then you read the error and you copy, paste it, and you try to figure out you Google what's going on. You find something and you run that, and then you run into a different error. You do that 10 times until the thing works. Everybody's setup is always a little bit different, and there's always these little details that will get you. In this case, we'd spend a while trying to set a MongoDB, and it wasn't working. So we just gave it to Devon. We're like, hey, can you just please get this working locally?

Starting point is 00:56:34 And then Devin did it. And that was the thing to what we were saying earlier. It's not just a stack overflow where it just gives you the one answer of, oh, you ran to this error. Oh, maybe you should try this. Like Devin sees the error and says, okay, let me see what's going on. let's check what ports are open. Let me see what the schema of this database is.

Starting point is 00:56:50 Let me see if the socket connection is actually set out. It's going and running all those commands to go look at that. And then it uses that information to decide, okay, what's the next thing I'm going to try? And then it tries that. And if it run into a new error, then it's able to debug that. The integrated autonomous flow is, I think, the biggest difference. How hard is it to build that? I'm not technical enough to ask the best version of this question.

Starting point is 00:57:11 But if I think about part of the secret sauce being the iterative agentic nature of the whole thing, What are the hardest parts about building that capability? The obvious one is the model capability itself. The models are roughly like, here's what a smart person on the internet might say. But if you're now thinking, okay, I'm doing a step-by-step process to make decisions to maximize my chance, not now, but after 100 steps that I'll have solved the issue, then it's obviously a pretty different thing that you're solving for. And so a lot of it goes into the capabilities.

Starting point is 00:57:40 But I think beyond that, too, there's so much with the infra and with the whole experience that gets quite involved, as you can imagine. And so, for example, Devon has its own machine that it works in. And a lot of the features that you'd want to support, for example, like rolling back state of the machine or dealing with plot loading these quickly and being able to spin up a Devon instance with a new machine immediately, being able to go and navigate through all the previous decisions

Starting point is 00:58:03 and go back and forth between these different things or the ability to work with different Devons in parallel or things like that. Yeah, there's a lot of infrastructure problems that go into that as well. And I think it just comes down to this, where the software engineering is so messy. And so it really is just so different from doing problems in a vacuum and building all of these high-touch components

Starting point is 00:58:24 with the real world, where you're actually able to do these things in the real world, you're actually able to use all your integrations, you're actually able to maintain your own machine state or work within your Get checkout or all these things. A lot of the mess of that is the part that I think that really comes out from us just really focusing

Starting point is 00:58:41 only on software engineering. If you had to put your most sci-fi lenses on for the future of technology in the world, what are some things that you maybe talk about with your team just for fun that don't seem feasible yet, but might be feasible in the future that get you the most excited. My co-founder Walden has this great line, which I've always really loved, which is we've been playing in Minecraft's survival mode for so long, and now it's going to be time Minecraft creative mode. Describe what that means for those that don't have 10-year-old boys.

Starting point is 00:59:10 Yeah, yeah. And so basically, Minecraft is a sandbox open world game, and it's meant to be a simulation of life. You can go and grow food. You can go and forage. You can mine. You can do all these different things. You can build stuff.

Starting point is 00:59:22 It's kind of funny because there's two major modes. There's one, which is survival mode, which is you are actually just to deal with the enemies that come at to do. You have to go build your own shelter and make sure you have enough food to stay alive and all these things. And there's obviously creativity within that things that you can do, but you are also constrained by how hungry your own character is, or whether you're being attacked or something like that.

Starting point is 00:59:41 Whereas Minecraft Creative mode is basically a mode where you're able to put together whatever you want. It's not like I didn't find any iron or jadey, and so I'm not going to be able to build this thing. It's like, I want to build this thing, I want to try this new thing, I want to set up this really cool thing. There's folks who have set up entire computers and computer programs in Minecraft, which is insane. It's basically just focusing on the ideas and the imagination rather than the execution. It's not about, okay, now I'm going to have to go spend another two hours going and collecting gold ore because I want to build this thing. It's like, I know exactly what I want to build, and now I have the freedom to just make that come to life. That's honestly, I think the big dream of AI as a whole is to be able to free up humans.

Starting point is 01:00:22 The point of AI isn't to go and just solve all of our problems for us and leave us with nothing. I think it's to give us the power that or the kinds of problems that we want to solve or the things that we want to build to give us the ability to actually just turn that into reality. Yeah, it's so funny to imagine, I'll use my own experience. When I visited you, you asked, I think the demo is incredibly powerful. So you asked me to come up with an application idea. I think it took eight minutes to build and we watched it do it. Watching the iterative nature of it feels like a real aha moment because it really does look like it's in hyper speed doing something that a person would do as it builds something.

Starting point is 01:00:55 But maybe the more profound realization I had was it built the thing that I described in natural language. But it wasn't until I saw the thing, the little stupid application that I made up, that I realized why that idea was. was bad. When I said the idea, it sounded good. And when I saw the thing, I knew why it was bad. And so I would say the thing that excites me is not that I can make my ideas, but that I can get better ideas, which I think is pretty underrated part of this whole thing that just like a computer program is going to be faster now, anyone with ideas is going to be better and faster because they can see the thing and see why it stinks. Does that resonate? That's a great point. That's a great point. it's very much the idea of turning 10% of time that you get to spend on your ideas until 100% of time,

Starting point is 01:01:46 whereas before you've got to get a team together, they're going to go and build the whole thing, and you spend a couple days or something building this whole thing, and then you get to do your next step of idea iteration. And by the way, Devin's still, honestly, a lot of room to grow up and it's going to take a lot of progress, honestly, to get to this future. But to get to get to a point where it's basically just you get to do the ideation work, you get to iterate against that. You get to do the new idea, you get to do the new idea and so on. And instead of these 90% implementation with the little 10% chunks in between, you just really

Starting point is 01:02:11 just get to spend all your focus on the ideas. I'm super excited about that. It seems like one of the defining aspects of cognition is the degree of friendship and trust between the leaders is extremely unusual. Maybe you can describe the origin of that. But when you're moving fast, trust is really valuable. And I'm just curious to hear about that side of your building experience so far. Pretty much all of us on the team have known each other for years and years and years. Stephen, Walden, Russell. I've known him of these guys for a long time and the whole crew as well. I had always wondered about, okay, is it going to be awkward to work with your friends? There's lots of things that come up. And if anything, it's honestly the other way around where it's easier to have tough

Starting point is 01:02:51 conversations because you know, obviously, we were friends before all this. All of this that we're doing is coming from a place of we are just trying to take care of each other and make something great together. When you both know that's true and when that undertone is always there, it's a lot easier to just call out stuff like, hey, like, I think this is totally wrong or we need to change this thing right now or things like that. What in your experience talking to investors, present company included, what do people not ask you about the product, the company, or the vision that they should ask you about? Yeah, it's a good question. I think one thing that's definitely different talking about it versus feeling it is that folks sometimes talk about or ask about

Starting point is 01:03:29 really is just kind of the experience with the team. I think the day to day, it's getting the product out there and seeing what folks are saying about it and talking with our users and things like that, obviously. But ourselves, I think, just how we do it and how we build it as a team is truly special, I would say. I think we have some of the smartest and most ambitious and most capable people, and yet somehow they're all such low ego people as well. I think that is truly the, that's the soul of it for me is the 20 of us in a house together trying to do something meaningful. It's a big difference in talking about it versus really seeing it live. And so one of the things that we often do as we're really pushing with it is having folks at the house just come by

Starting point is 01:04:13 and see what it's like. It's going to be 1 a.m. or 2 a.m. and stuff and people are still building and there's a lot of discussions and stuff that's moving quickly and everything. And that's probably the thing about building the company that I'm most grateful. What uncertainty interest you the most. That's outside of your control. Interesting. Yeah. One of the uncertainties I actually think a lot about is broad adoption. It seems very clear to me now that the technology will get there or maybe already has

Starting point is 01:04:41 gotten there in a lot of cases to be extremely impactful. And I think the rate at which we see it happening in different industries is going to be pretty fascinating. We can both think of things like medicine or law or things like that where you can certainly imagine it will take time to do that. And then there's obviously just the whole world of distribution. I will call it a win where everyone in Baton Rouge, Louisiana, is actually having a significantly better life because of the work that AI is doing.

Starting point is 01:05:07 It's a different curve everywhere. Obviously, I think we're lucky in software that developers are just kind of inherently curious and always want to try to be. But I do think the rates of adoption, it's going to be a real question with this technology because I think the pace of progress is going to be such that technology in some place is going to be way ahead of reality. When we were first together, you did. this insane card trick.

Starting point is 01:05:28 Are you willing to do it for us? Yeah, let's do it. Tell us the origins of this trick. Yeah. So this is a card trick. So I wouldn't even call it a car trick. It's more of a card game. And so growing up, I always used to play this game called 24.

Starting point is 01:05:40 It's a little math day. And I think it actually helps a lot with just getting the fluency in math. I think it's one thing to learn, okay, here's addition, here's multiplication, and whatever, which is obviously super to build a lot. But also a lot of it is just this creativity and this ability to reason with numbers and experiment with numbers. And so I'll just show what the game looks like. And so the idea is you have four cards and they're four numbers.

Starting point is 01:06:03 By the way, Jack is 11, queen is 12, Kings 13. So it would be four, seven, eight, and 12 are the four numbers that we're dealing with here. And the idea with this is you can use the four numbers in any order, in any combination. You got to add, subtract, multiplied, divide any of these. And your goal is to put together an equation that uses all four of these and makes the number 24. And so here what I would do is 12 minus 8 is 4. 4 times 7 is 28 and 28 minus 4 is 24. And so you can see how it uses a lot of this creativity where it's like,

Starting point is 01:06:35 hmm, messing around this number for 24 is a multiple of 4. And so if I could just make a 6 from the other 3, then maybe I could do something. Turns out that you can't quite do that, but you can make a 7 and subtract another 4 out, things like that where you're playing with these ideas. And so at these math camps and programming camps, we used to play this game all time.

Starting point is 01:06:51 And eventually we got to the point where you'll learn all the combinations of 24. There's only about 1,300 total combinations that you have to know. Not all of them are possible, but for the ones that are possible, you know them. And so what we did next was we did a thing where you take six cards and you're making 163. And so I'll just steal another six cards. And so in this case, for example, what I would do is 11 times 2 is 22, 42 minus 3 is 19, 6 plus 10, 16, 16 times 9 is 144, and 144 plus 19 is 163. And so as you can imagine,

Starting point is 01:07:23 we would play this at the math camp during the team selection training for the U.S. team. Basically, in between contests, we would just be playing this game all the time and doing this. And eventually you get to the point where 163, you have a lot of the patterns down and you see things.

Starting point is 01:07:37 And so then the next thing that we did was we do eight cards and we make the current year. And so now it's 2024. And so let's just go ahead and do that. So 13 minus 12 is one. And this is actually relatively easy. I'm just going to take one times one times one

Starting point is 01:07:49 and we still got a one. And 13 plus 10 is 23. 23 times 11 is 253. 253 times 8 is 2024, which times 1 is still 2024. Incredible. And eventually you learn all the patterns with that. And so obviously the next fun thing that we have to do is we have to do a custom number. How many digits?

Starting point is 01:08:08 Four digits? Yeah, yeah. Let's do 7532. 7.32. Okay. So it's a little bigger. So we're going to do nine cards. Okay.

Starting point is 01:08:17 Okay. So 11 times 7 to 77. 77 minus 2 is 75. We got two tens here, which is a nice hundred. And so that makes it 7,500. And then from here, 9 times 4 is 36. 3 plus 1 is 4. 36 minus 4 is 32.

Starting point is 01:08:31 So 7500 plus 32, 7532. Pretty amazing. I don't know how you do it. What are you doing? What is the method behind calculation? Or is it multiple methods? A lot of it is an intuition, I guess, that you build every time. With 163, for example, a lot of these little things where I know,

Starting point is 01:08:49 okay, 11 times 15 is 165. So if I can make an 11 into 15 and then I can make a two left over, then I can do 165 minus two, and then I'm searching for that. And so based on the numbers, I see I'm solving for something like that. And it depends a lot on the patterns that you get and everything. It's actually, I think, a great game for kids. This is my favorite game as a three-year-old kid or whatever playing 24 with the four cards, because a lot of it really pushes the number sense. It's not just a question of can you add two numbers, but can you really reason around that And like, you know what that means and be creative with the numbers that you have, which I think is the core skill that you really want to learn, not just the mechanical part.

Starting point is 01:09:26 I think what you're building is so cool to me because effectively what you're doing downstream is unlocking the creativity of a lot more people than currently have the ability to build things through computer science and programming. And I just think that's an incredibly cool and amazing thing for the world. I hope you and others like you are successful in doing so. When I do these interviews, I ask everyone the same traditional closing question. What is the kindest thing that anyone's ever done for you? The number one thing that I come to throughout my life is honestly just all of the mentors that I've had and how much effort they truly put in.

Starting point is 01:10:01 Even as a kid, as you can imagine, like training for the U.S. team and for the international thing, it was funny because I never had a coach. I never had formal this is the person. But I did have a couple people and one in particular who just really invested in me and believed in me. And he was the one who really just pushed. I was a high school kid. I had a lot of other things that I wanted to do besides just doing math. couple of time. And he were always pushing me and be like, you got to practice, you got to do this thing. Every time it would get something wrong, you'd be like, yeah, you can't make that

Starting point is 01:10:26 mistake. Here's how we're going to do these things. And all the time that he put in is everything, honestly, comes from that. I feel like I'm really only the sum of all the mentorship that I've received. And I think the same is true in startups. I can think of a few people. And again, I can think of one in particular who's really just coached me through a lot of this and guided me through a lot of these things and been the one to give me the harsh criticism and the feedback and hit me with the things that matter and push me to do better and to do more. And people often think of kindness as sacrifice. And I do think actually that often a lot of the greatest kindness is about building more together and doing more together. Scott, thank you so much for your time.

Starting point is 01:11:03 Thank you so much, man. If you enjoy this episode, check out join colossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

Invest Like the Best with Patrick O'Shaughnessy - Scott Wu - Building Cognition - [Invest Like the Best, EP.402]

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.