No Priors: Artificial Intelligence | Technology | Startups - Cognition’s Scott Wu on how Devin, the AI software engineer, will work for you

Episode Date: May 2, 2024

Scott Wu loves code. He grew up competing in the International Olympiad in Informatics (IOI) and is a world class coder, and now he's building an AI agent designed to create more, not fewer, human engineers. This week on No Priors, Sarah and Elad talk to Scott, the co-founder and CEO of Cognition, an AI lab focusing on reasoning. Recently, the Cognition team released a demo of Devin, an AI software engineer that can increasingly handle entire tasks end to end. In this episode, they talk about why the team built Devin with a UI that mimics looking over another engineer’s shoulder as they work and how this transparency makes for a better result. Scott discusses why he thinks Devin will make it possible for there to be more human engineers in the world, and what will be important for software engineers to focus on as these roles evolve. They also get into how Scott thinks about building the Cognition team and that they’re just getting started.  Sign up for new podcasts every week. Email feedback to [e-mail] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @ScottWu46 Show Notes:  (0:00) Introduction (1:12) IOI training and community (6:39) Cognition’s founding team (8:20) Meet Devin (9:17) The discourse around Devin (12:14) Building Devin’s UI (14:28) Devin’s strengths and weakness  (18:44) The evolution of coding agents (22:43) Tips for human engineers (26:48) Hiring at Cognition

Hi, listeners, and welcome to No Pryors. Today, we are talking to a very good human software engineer and the co-founder and CEO of Cognition, Scott Wu. The team at Cognition calls themselves an AI lab focused on reasoning. And recently, they released a demo of Devon, an AI software engineer, that can increasingly handle entire tasks end-to-end, with leading results on SwayBench, a software engineering benchmark of real-world GitHub issues. The demo broke the internet, at least among tech Twitter. And so we're really excited to have Scott.

Welcome, Scott. Hey, great to be here. Thanks for having me. So you have been coding and then coding competitively since you were a kid. What first got you interested? I always really, really liked math growing up, actually. My older brother, Neil, was the first to get me into programming. So I think I learned how program when I was around nine years old.

I just fell in love with it. I think the ability to take a lot of it. ideas and then make them into reality was always really exciting for me. I did a lot of math and programming competitions, you know, throughout school. And by the time I finished high school, I was pretty set on going into tech. Can you explain for our listeners who may, some set of our listeners that are not familiar, like what I.O.I is. Yeah, sure. So I.O.I is the International Olympiad of Informatics. So it's the Olympiad of Code, basically. And it's, it works just, like the other Olympics. Every country sets in their own team of their best coders, and it's

a competitive programming competition. And, you know, there's gold medals, silver medals, bronze medals, and so on. And, you know, the whole idea is it's very algorithmic problem solving. And so you're given tasks to solve and, you know, you have to figure out the optimal algorithms for those and then also implement those into code. It's like an interesting topic for somebody who was world class at I.O.I. Like, what is practice like for getting better at competitive programming? Yeah, I used to, you know, programming and math competitions used to be my entire life when I was growing up.

I, you know, it's what you think about at the shower. It's what you're spending all your time on. It's, you know, how, how, every problem in life that I run into, I would frame as an algorithm's problem, basically. And a lot of it is, like a lot of other disciplines, it's just, you know, putting a lot of effort into it and being willing to think really. analytically and, you know, be very brutal about your own shortcomings and, you know, focus on the things that you're not doing well and just continue to push and improve it.

Is it like a known space of like domains and algorithms one can learn or is it really like learning to solve problems algorithmic really well? And it could be completely new problems or domains in every competition. Yeah, there's definitely, there definitely are standard algorithms that exist. to like, you know, shortest path or something or, you know, binary search trees or things like that. And so it helps a lot to learn the fundamentals. But the whole idea is that every problem in the contest is totally unique. You know, it's a new problem that's never appeared before. And, you know, the beauty of the contest itself is in the creative problem solving that

you're doing to figure out the right algorithm. Right. And so, you know, while the fundamentals are very helpful, a lot of it is figuring out how to use each of these pieces and, you know, reduce the problem to a shortest breath problem or how you, you know, modify, you know, certain algorithms to make them work for different use cases. Yeah, I feel like there's a whole generation of people now who are basically people who grew up with the internet being their hometown in some sense, you know, where a lot of their early community or people they interact with are sort of come from that world. And I feel like that's a lot of people in the AI world today. Was that true for the IOWI community as well? Was it largely like online interactions and trading tips with strangers

who became friends and all that kind of stuff. And then if so, how's that impacted your career or your life or working with others over time? Yeah, definitely. I mean, I grew up in Baton Ridge, Louisiana, and so there were not a lot of other people who had the same kind of interest in math and programming

that I did. And so, you know, a lot of these competitions were the first time that I got to meet others who had a lot of these same interests. And, you know, the competitions are once a year, and maybe there's training camps or things like that that are a couple times a year, but for the large majority of the year, we'd be talking online. And, you know, we had all of our own communities where we'd talk about, you know, competition

problems, but also kind of, yeah, yeah, we became very close friends to that, too. Yeah, it's an incredibly smart community of people, at least, you know, competing at the highest level. And I think you said before that CP has more in common with entrepreneurship than one might see from the outside initially because it looks so, like, structured as a competition, but, you know, one of the reasons we met was you knew very well somebody from high school that I backed previously. And I found out recently that you, I think you knew like Alex Wang in middle school, too. And I was like, oh, that's a small world. Yeah, yeah, no, it was a very, very tight-knit group. A lot of, honestly, a lot of that group,

many of them became founders. And a lot of them went into AI. And so, no, it's cool to see. there's, you know, I think there's overlaps for sure. I think they're the more subtle kind of overlaps, obviously, where it's, you know, you're not literally using shortest path or something to, you know, to build your company. But, you know, a lot of the same constructs and a lot of the same principles are there. And I think one of the really big things that I think about all the time is, you know, a lot of the really great math or programming problems are kind about, you know, what are all these assumptions that you take for granted and which one of those might actually not be true, you know? And a lot of these really tough problems and really

interesting problems are about finding this creative idea that's actually something that's, you know, you would not intuitively think of at all at first, right? And I think startups are much the same way, you know. I think a lot of building a great startup is about, you know, finding something that the world doesn't believe is true yet, but actually it's, you know, and kind of learning how to think independently, you know, how to come at problems from the very first principles mindset. And of course, you know, as with any discipline, how to just really push yourself to grow and continue to improve over time. I think there's a lot of commonalities. It seems like a lot of what you mentioned really impacted what you ended up doing a cognition

and working on Devon because when you launched it, a lot of people saw behavior that could exist today that they didn't think could exist. And a lot of people were waiting for the next word to step up and an LLM or foundation model to get there. Can you tell us a little bit more about cognition and what Devin is and how it works and how you think about it in the context of rethinking what's possible today? Yeah. So cognition got started around November. And the initial group, a lot of them actually were, you know, many of my friends from this math and programming contest community. And, you know, we had all done all of those. But since then, we've all actually had our own kind of journeys in AI. And so, you know, I started a machine learning company

called Lunch Club that I ran for about five years. You know, others on the team. My co-founder Walden, you know, built a lot of early cursor, for example. And my other co-founder, Stephen, was one of the first engineers at scale AI. And our whole founding team was kind of like that. You know, we had grown up with a love of math and programming and problem solving, but we'd also spent our last, you know, five or ten years at these different AI and AI infra companies. For one, I think code has a very special place in our hearts, but I think for two, I do think there's a lot that can be done with code.

You know, I would even say, like, most of the progress in the world in the last 20 or 30 years has come from software. And it's crazy when you think about how actually everyone, everywhere, needs more software engineering, not less, right? I mean, every company is hiring for more software engineers, you know, everyone has more ideas than time to build them. We really felt like there was a lot to do with accelerating, accelerating pace of code. For anyone who hasn't seen it, can you describe like what Devin does today? Sure, yeah. So Devin is an AI software engineer. What I mean by that is that Devin is fully able to make all of its own decisions in the same way that a human software

engineer would. And so, you know, obviously writing and edited code is a huge piece of that, but also So, you know, being able to work with the command line, being able to use the browser, being able to read documentation, being able to deploy or test or debug or all these pieces, David does autonomously. And so, you know, you can give Devin a simple prompt of what you'd like to build, and Devin will just go through and do that all end-to-end. And you can interact with Devin as well. You know, you can see what Devin's looking at or what Devin's working on and give feedback on them too. And so it's very much meant to mirror the experience of, you know, looking over the shoulder. of another engineer to be able to see what they're doing and guide them or get feedback. What has been the reaction to Devin and Cognition online in the popular discourse

and how do you think people are interpreting the product that you launched? Yeah, I mean, it's really fun to see all the reactions to Devin. We got a lot more views than we expected. It was honestly quite something to see some of my middle school math videos on Twitter all over again. But yeah, no, I mean, it's great to see the reaction. And as with anything, I mean, there's obviously the healthy mix of there are some who are skeptics. We're saying, oh, there's no way this could possibly work. There's somebody saying, wow, we're all going to lose all of our jobs. Why do you think, like, engineering doesn't disappear as a class of work in the future?

We're obviously very excited about Devin. But, you know, I think if anything, there's going to be more engineers, not less. And I'll kind of give two reasons why. I think the first is there's just so much demand for engineering out there. And so much demand, honestly, even in ways that we, we don't always think about, right? A lot of using Excel or a lot of, you know, working with various tools is, in some ways, is there because there's, because engineering is hard, right?

I think there are so many problems that could be solved with code. There's so much more that could be built with code that I think multiplying every single developer is going to give us more developers, not less. You know, the other thing I think is that Devin is very much not the type to decide what to do, You know, and I think there is always this core part of how you decide what exactly to build or what problems to solve or, you know, how particular things should work. Rather than engineering going away, I think engineer actually becomes a more pure, you know, abstraction of those things, right? I think the average software shares today might spend 20% of their time thinking through all these like really fundamental problem solving questions and 80% of the time writing that in code. And I think they'll be able to do 5x more.

And they'll be able to spend all their time doing this really creative problems on it. And that's what we're really excited about. As a really loose analogy, I think of like human calculators previously, right? Like you probably don't want people doing that work and it allows you to expand to much more capable human beings that would have done that work. And, you know, one of my core optimisms around AI is that it is actually very democratizing, right? Like we are tapping into latent demand for lots of different. activities where it was just unaffordable before where like the software doesn't match the workflows that people want or it's not at sufficient quality or it's not as sophisticated as

could be for an unlimited number of things so I'm very excited about that actually yeah definitely that's largely the case I think in knowledge work everywhere even right where um you know there's there's so much to do and there's so many problems out there to solve and I think um enabling every every human to do more is only going to help us all accelerate faster. I think this is one of the really interesting things about the product was, you know, I feel like there are more companies showing up now, kind of showing me Devon like UIs. And before a lot of people built agents and you'd start the agent and it would go off and then

30 minutes later would come back with something that you didn't like. And in the case of Devon, you know, because it feels like a lot of agents today are almost like really eager interns, right? there's an intent to build what you wanted to, but then sometimes it goes off the beaten path. And the ability to steer it back on to what you wanted to do matters a lot. And in the context of Devin, I think it's really interesting that you had those four tabs of, you know, planning and here's what we're going to do and sort of check that off. And then the shell and the code and then the browser and showing like, you know, where

where it's interacting with the world. And I think that's a really powerful paradigm. And so I'm a little bit curious about what was inspiration for that. Was it literally you thought of it as, hey, I'm peering over somebody's shoulder. or was it something else that really inspired that UI, which I think is starting to catch on in other areas now. Yeah. Yeah. I mean, I think one of the nice things about the space, obviously, is that we get to build for ourselves in a way.

And we're all obviously software engineers. We ourselves use Devon all the time at our own work, even when we're building Devon, working through each of the pieces, basically, and figuring out what it took from us to work well with Devon. As you kind of mentioned, if an agent is, you know, you give it the task, and then it goes off and does it, then you, you know, it basically only has one try to do it all and to do everything perfectly. Whereas if you have a human intern or a junior engineer or whatever it is, you know, if they're working on a project over the course of the day and, you know,

you have to check in four or five times and just give a quick, you know, 30 seconds of feedback on what's going on, you know, for one, it helps them learn a lot. And for two, you know, beans that they can still, you know, give you a lot of really valuable work over that time. And obviously, it's much, much faster to be doing that than, you know, to do all of those pieces yourself. I think we worked backwards from, you know, what made sense and how we would want to interact with an AI coding teammate and thought through technically how we would be able to get those pieces to work. What is Devon today better and worse at than human software engineers? How does that change how you guys use it internally? The encyclopedia knowledge is obviously really, really

useful. I think honestly with DevOps and dev setup, I think there's a lot that it's very good at. You know, I think DevOps is just kind of hard for humans. Actually, one of the first, the first really exciting moments with Devon was we were trying to set up, you know, get a database spinning, and get Kubernetes up and whatever else, you know, for our own purposes. And we were stuck after like an hour or something, you know, going down the rabbit hole of debugging errors and stack traces and whatever. And we just asked Devin, hey, can you set this up, please? And let us know how you did because we can't do this right now.

And Devin actually did. And that was one of the first, like, really exciting moments for us. I think the, you know, the step-by-step flow of, you know, editing and working with things, you know, running commands live, looking at the errors that come up, you know, all those pieces of, hey, like, do I have this port open or like maybe I need to install this package or whatever, you know, work very, very well in an agent workflow. So that's a big one.

I think data analysis is another big one, which we've seen a lot as a good use case. A lot, you had a few use cases in that bucket, actually, in particular. But that's another very end-to-end flip, right? often you're trying to find the right data sets from the internet or, you know, pull out a CSB file, then you're sanitizing the data, that you're running, you know, whatever your exact analysis is and that you're building a visualization of that analysis. And, you know, being able to just do that all from start to finish with an agent is another really good use case.

I mean, I think as far as things that it's worse at, you know, Devin is very much not the one who is going to be deciding what to do, if that makes sense. And so, you know, we very much think of Devin as you provide the precise formulation of what you want built or what you want done. And Devin is the one that is doing the thoughtful execution of that, right? And so, you know, you wouldn't be able to just go to Devin and say, hey, you know, built me a great business or something or built me, you know, something like that. Because, yeah, I mean, I think a lot of Devin's focus is understanding how to take precise ideas and really formulate them into code and do that. entire, you know, testing process and debugging process and installing packages and deploying and whatever else that are involved with that.

What can you tell us about how it works and, you know, what work happens in the foundation model layer and the system around it and, like, how you guys invest in improvement? Yeah. Yeah. I mean, unfortunately, I can't go too much into the details of how it works. I mean, I think it's, you know, it's obviously our, one of the, one of the areas that we spend the most effort on. But, you know, I think a lot of it comes down to understanding the interface of the problem and understanding how exactly to optimize the problem that we're solving for, right? And I think one way that I'd frame it, for example, is, you know, suppose you were given an issue, you know, you were given a GitHub issue and you needed to solve it. Maybe there was some bug that you needed to

fix, right? You know, there certainly is a view where you could take the entirety of the code base and you could figure out from the bug exactly what the diff was that needed to be done and write out that diff and clean text. And you could do that, right? And so, you know, some perfect intelligence, either human or AI, in theory, would be able to do that, right?

Because it would be able to figure out exactly what's going on and understand exactly what needs to be fixed. Right? But I think practically for humans today and, you know, we think also for AI's today, the cleaner path to do that involves, you know, running the code yourself, right? Reproducing the bug or, you know, adding debugging statements and print statements and

rerunning the code, or taking a look at the logs, or, you know, asking Stack Overflow or all of these things, right? And a lot of the problem that we solve is figuring out how to, yeah, how to get Devin to think in that way and to make decisions in that way. So a lot of this planning and evaluation is what we spend our time on. How do you think this all evolves over time? So, for example, say we extrapolate out one or two years and then maybe five years. Like what proportion of code do you think is written by agents like Devin or what do you think changes in terms of how software engineers work in both the shorter term as well as longer term?

Yeah, so I mean, I think in the longer term, like I think on the five to 10 year horizon, I mean, I think what we call software, I think changes a lot, you know? And I think we think of Devin or the steps that we're building with Devin. has kind of the next generation of human computer interfaces, right? Where, you know, I think 10 years from now, people will look back and think, wow, isn't it crazy that, you know, you had to learn all these esoteric languages and you had to, you know, work through all these stack traces or whatever just to be able to communicate with your computer, right? I mean, at the end of the day, software engineering today is a discipline about

being able to work with your computer and to have the computer do what you want to do. And I think that, you know, what happens with software engineering is, I think software engineers everywhere get to spend all their time on, you know, the really fun part of the problem, which is taking any problem that you're given and figuring out exactly what the form of the solution should be, right? And so all of this work of, you know, what are the cases and the edge cases and the details, what are all the flows, what are the architectures to build to solve each exact problem. and, you know, taking it from that to that code implementation of exactly that, you know, is something that the Devons of the world will solve, right? I think the really exciting thing is there's so much, so much demand for software, right? And so, you know, it's crazy to think about how, you know, there are 30 million software engineers in the world, but, you know, in acting that still only 0.4% of people, right?

Giving those people the ability to do much more and also enabling so many other people to be able to work with software. It's something that we're really excited about. In the next one to two years, I think, you know, I think obviously there's a lot of kind of broader changes that will happen. I do think, software engineers, I think, very quickly, you know, who work with AIs and are kind of very AI-Native on this trend are going to be able to multiply themselves and do more. It's a really interesting time to be an AI. You know, I think there's a lot of rise in tides and, you know, the hardware is getting better all the time, you know, the foundation models are getting better all the time. Obviously, all the agent work that we do is getting better and better all the time.

And so I think, yeah, I think things will move pretty quickly here. What do you think are the things that will improve agents the most going forward? Is it the underlying foundation models? Is it reasoning, forms of memory, self-play, something else? I'm just sort of curious at a high level, even putting aside what Devin is doing. But like, agents in general, like what is needed in the field to push these things forward? Yeah, I honestly, my honest answer is yes to all the above. You know, I think all of those things will be great. You know, I think, you know, even like infrared speedups are going to be very useful for the end experience.

I think, as you said, I mean, I think better reasoning, for example, on the base models is going to be really, really great. I think, you know, figuring out some of this planning and, you know, better tool use and so on. The way I'd kind of describe it is, you know, I think the agents today are a combination of all of these factors. And in some sense, I think there's a bit of a race between all of them where a perfect intelligence AI, in theory, wouldn't need anything else, right? You'd be able to just look at the code and tell you exactly what should change, right? At the same time, you know, some perfect tool user, for example, perhaps might also be, you know, a great solution. I think what we'll see is, you know, all of these are going to be a rising tide, and it'll be a matter of, you know, which of these spaces see the most progress as far as how much impact they have on agents. What do you think is going to be important from a human software engineer or just like human technology person five years from now?

I realize that's a really long time scale in AI, but it's certainly not like encyclopedic knowledge anymore, right? Yeah. Yeah. I mean, I think there's there's a meme that, you know, the hottest new program, language is English, right? And I mean, I think there's a lot of truth to that. But with that said, I think that, you know, the software engineering fundamentals are obviously still super, super valuable, right? People, you know, for example, like, I think, you know, the internet today is something that we all kind of are able to use and kind of take granted, but people who work with these networks, it's certainly very helpful for them to understand the details of TCP, right? And I think similarly, I think, you know, I think we'll be able to communicate our

ideas in English, you work with all these things, but, you know, understanding the internals of how computers work and understanding logic gates and, you know, a lot of these core pieces, like these core foundations, I think will still be very useful, right? And so, you know, whether that's, you know, algorithms or technologies or logical reasoning or things like that, like, I think the, you know, I think the role of a software engineer, five or ten years from now, it looks something like a mix between a technical architect and a product manager today, where a lot of what you do is, you know, you take problems that you're facing or that your business is facing or whatever, and you're really thinking about and breaking down what exactly the solution should be. How do you think about it on an even farther time frame? Because when I, if it was five years ago, I would have told either my kids or people who have kids, you know, you should study computer science and math. 20 years from now, I'm not as certain. So I'm sort of curious how you think about the future of this field if much or all the work, including a lot of the planning,

is actually done by machines at some point. Yeah. I mean, I love that. So I have to say it's a worthwhile experience, even if it doesn't end up being practically useful. But no, I mean, I think a lot of these fundamentals will stay useful for a long time. There's obviously a lot of questions that come up about, you know, super intelligence and singularity and all of this.

And, you know, it's very hard to predict. I think everyone in AI, it's, you know, we've all made our predictions and, you know, tried to make our guesses. But I think it's hard to be very high confidence. But with that said, I do think that, you know, we're going to see AI's concrete impacts on work and economy and people's lives, I think, a lot sooner than that. You know, I think the way that we think about the problem is that, You know, even with the tools that are available today and the technologies that exist

today, there's so much that's possible to really impact people's lives, right? And, you know, we're still very, very early in this whole AI revolution. I mean, you know, chat GPT was about a year and a half ago at this point. And there's a lot more to do and a lot more to build, you know, both on the research side and on the product side. That's the best guest I have. I had lunch, Scott, as you know, with one of your new hires today. and he's like, it just makes sense to me

that Scott Wu is going to be like the last human software engineer slash labeler, right? And I'm like, all right, like, without a better plan, I'm having my kids like be the best labler they can of that data. Yeah, yeah, yeah. No, I mean, I think, I would guess that, you know,

I think good reasoning and good fundamentals and, you know, good understanding of technology, you know, are going to be useful for quite some time to come. I mean, maybe eventually there will be a point where, you know, you just upload all this knowledge, into your brain or whatever, you know, you just become a perfect rational decision maker. And I'm sure that it'll be a very exciting future. But I think, you know, with what we have today, I imagine that, you know, a lot of these logical French conscious will stay relevant. Yeah.

Maybe on that note, I think one of the things that I really admire is just the like a broad strength of the founding team and the early team at Cognition. Can you just talk about a little bit about like your approach to hiring? Yeah, no, thank you so much. I mean, I think the, you know, I obviously think our team is really exceptional. I think we, yeah, I think a lot of the last, you know, five or six months has been really thinking about just the size of the problem that we're going after, you know, and software, software is massive. You know, it's a huge driver for change and it's also a very deep and interesting problem. And a lot of our team today, in fact, I think almost everyone on our team today

I think was either a founder of themselves before or were explicitly thinking of starting their own company, if not for cognition. And I do think that working with people who are very high ownership, you know, very interested to drive and, you know, much more focused on outcomes than they are on, you know, politics or anything like that is very important. On top of that, of course, I think, you know, I think creativity, you know, work ethic, you know, good communication. All of these are super useful. I think

we've, you know, thus far been mainly kind of go through a very close network of people that we've known well and worked well with and kind of grow in our team that way. Are there specific types of people they are looking to hire now? Yeah, I mean, we're always looking for really great engineers and researchers. You know, I think we're still very, very early, and it's a very big mission of what we want to do. And so I think, yeah, I mean, great, great technical thinkers who are excited to, you know, to work on multiple different parts of the business, I think, are what we're largely looking

for, you know, I think between customers and general strategy, research, engineering, product and all of this, you know, it's, we're kind of all doing everything, which is how it often is and an early startup. And so, yeah, I mean, I think we, yeah, we'd love to bring on more more people that think those archetypes. Awesome. Thanks for being here, Scott. Yeah, thanks for doing. Yeah, thanks for having me. Find us on Twitter at No Pryor's Pod. Subscribe to our YouTube channel if you want to see our faces, follow the show on Apple Podcasts, Spotify, or where you listen. That way you get a new episode every week. And sign up for emails or find

transcripts for every episode at no dash priors.com.