The Pragmatic Engineer - AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django)

Starting point is 00:00:00 Every programmer who works with these models the first time it spits out like 20 lines of actually good code that solves your problem, it does it faster than you would. There's that moment when you're like, hang on a second, what am I even for? And then I tried this new feature of chat GPT that they launched last year called code interpreter mode. And I asked a question and it flawlessly answered it by composing the right SQL query, running that using the Python SQL Lite library and spitting out the answer. What am I even for? Like I thought my life's purpose was to solve this problem. That was a little bit existential thread. It is scary when you think, okay, I earn a very good salary

Starting point is 00:00:34 because I have worked through the trivia of understanding Python and JavaScript and I'm better at that trivia than most other people and now you've got this machine that comes along and it's better at the trivia than I am. I feel like there's a pessimistic and an optimistic way. The optimistic version, I can use these tools better than anyone else for programming. I can take my existing programming knowledge

Starting point is 00:00:53 and when I combine it with these tools, I will run circles around somebody who's never been, written a code line of code in their life, I can just do this stuff better. Welcome to the Pragmatic Engineering Podcast. In this show, we cover software engineering at big tech and startups from the inside. You'll get deep dives with experience engineers and tech professionals who share their harder and lessons, interesting stories, and practical advice that they have on building software. After each episode, you'll walk away with pragmatic approaches you can use to build stuff,

Starting point is 00:01:22 whether you're a software engineer or a manager of engineers. In this first episode, we go into a really timely topic using Gen AI for coding. Now, there's no shortage of AI companies hyping up their capabilities, but we'll size up all of that. I turned to longtime software engineer Simon Willison, who is safe to refer to as an independent investigator of large language models, because he's been using them so much to improve his personal productivity for the last four years. With Simon, we have a refreshingly honest conversation on how these tools actually work for us developers, as of an hour. We talk about common LLM use cases like fine tuning and rag, Simon's day-to-day large language model stack, and misconceptions about large language models. This is the first episode of many such deep dives to come.

Starting point is 00:02:08 Subscribe to get notified of when new episodes are out. So Simon, welcome to the podcast. Hey, it's really great to be here. So it's great to have you here. You're an experienced software engineer, and you've definitely been around the blog. So some people will know you from your prolific open source content. contributions, co-creating the Django framework, which is a rapid web development tool written in Python. You're also the creator of a dataset, tool for exploring and publishing data.

Starting point is 00:02:36 And then you're also a startup founder, right? So I remember you were the founder of Lannard, a conference direction site, which was funded by Y Combinator, acquired by EventRite. And then you were there for six years as an engineer as a manager. So you've really done all of the things, open source, founder, working at a large company. Yeah, I got to do the startup to large company thing is particularly interesting, you know, like moving from moving at the speed of a startup to moving at the speed of a much larger company

Starting point is 00:03:06 where bugs matter and people lose money if your software breaks. When I started to notice you more is when around the time when chat GPT came out, and you were very hands-on in trying out what this works for your development workflow. You shared a lot of things on your blog. And really, this is what we're going to talk about today. Your firsthand learning is about how this AI enhanced development helps your specific workflow, where it doesn't help and what you've learned through this. How many years has it been two, three years of non-sub usage?

Starting point is 00:03:40 Well, so I was on GPT3 before ChatGPT came out. So I'm at about, I'm verging on three years of using this stuff frequently. It got exciting when ChatGPT came out. GPT3 was interesting, but ChatGPT, that's when the whole. world started paying attention to it. To kick off, I'm interested in how you got started with these large language model tools. What was the first time you came across them and you're like, all right, let me go as a go. So I've been paying attention to the field of machine learning as a sort of like side interest for five or six years. I did the fast AI course,

Starting point is 00:04:20 Jeremy Howard's course back in, I think, 2018. And then GPT2 came out in, was that 2019? Yeah, it's 2019 GPT2 was happening, which was the first of these models that you could see there was something interesting there, but it was not very good. You could give it text to sort of complete a sentence, and sometimes it would be useful. And I did an experiment back then where I tried to generate New York Times headlines for different decades by feeding in, say, all of the New York Times headlines in 1950s, then the 1960s, and then giving it stories to complete now.

Starting point is 00:04:57 And I poked around for a bunch. The results were not exactly super exciting. And I kind of lost interest at that point, to be honest. And then GPT3, which came out in 2020, but sort of began to be more available in 2021, that's when things started getting super interesting. because GPT was the first of these models that was large enough that it could actually do useful things. And one of the earliest code things I was using it for was, I think I was using it for JQ, the little JSON programming language, which I've always found really difficult.

Starting point is 00:05:32 It just doesn't quite fit in my head. And I was finding that GPT3, if I prompted it in the right way, and this was a model where you had to do the completion prompt. So you don't ask it a question and get an answer. You say, the JQ needed to turn this into this is, and then you stop and you run that in the model and it finishes the sentence, which I think is the reason most people weren't playing with it. That's a weird way of interacting with something. Like in many ways, the big innovation of chat GPT was they added a chat interface on top of this model.

Starting point is 00:06:02 And so now you didn't have to think in terms of completions, you could ask a question and get an answer back. But yeah, so it was very clear back then sort of, and that was running up for about 12 months before chat chapti came along, There was something really interesting about this model and what it could do. And that was also the point where it became clear that code was actually something was surprisingly good at. And this, I talked to somebody OpenAI, asked them, it's like, were you expecting it to be good at code?

Starting point is 00:06:29 And they said, you know, we thought maybe, but it wasn't one of our original goals. Like the original goals of these models were much more things like translation from one human language to another, which they do incredibly well. But when you think about it, the fact that they can write code well isn't that surprising because code is so much simpler than like English or Chinese or German. Yeah. Like grammar rules. A trivial.

Starting point is 00:06:51 What we know, I think it's pretty obvious. And I think, you know, we'll talk about implications. But let's just jump a little bit ahead. So I think, like, I personally had, wow, this is amazing moment with LLMs. And then I've also had a bit of a, like, scared moment of like, is this, could this actually replace part of what I do or not? And you had a really interesting story with that a proper like, this is scary moment. Can you talk about that?

Starting point is 00:07:20 I mean, I've definitely, I've had a few of those. I think every programmer who works with these models the first time it spits out like 20 lines of actually good code that solves your problem. It does it faster than you would. There's that moment when you're like, hang on a second. What am I even for? But I had a bigger version of that with, actually with my main open source project. So I build this tool called Dataset, which is a, it's a, it's a, you know, interface for querying databases and analyzing data, creating JSON APIs on top of data,

Starting point is 00:07:49 all of that kind of stuff. And the thing I've always been trying to solve with that is I feel like every human being should be able to ask questions of databases. It's absurd that everyone's got all of this data about them, but we don't give them tools that let them actually, you know, dig in and explore it and filter it and try and answer questions that way. And then I tried this new feature of chat GPT that they launched last year called Code Interpreter Mode. This is the thing. where chat GPT, you can ask you the question, it could write some Python code, and then it can execute that Python code for you and use the result to continue answering your question.

Starting point is 00:08:23 And Code Intercept Mode has a feature where you can upload files to it. So I uploaded a SQLite database file to it, like just the same database files that I use in my own software. And I asked the question, and it flawlessly answered it by composing the right SQL query, running that using the Python SQLite library and spitting out the answer. And I sat there looking at this thinking on the one hand, this is the most incredible example of like being able to ask questions of your data that I've ever seen. But on the other hand, what am I even for? Like I thought my life's purpose was to solve this problem.

Starting point is 00:08:55 And this thing, this new tool is solving my problem without even really thinking about it. Like they didn't mention, oh, it could do SQLite SQL queries as part of what it does. It's just like Python. And that was fun. And well, no, that was a little bit existential dread. And the way I've been coping with that is thinking, okay, well, my software needs to be better than chat GPT code interpreted this particular problem if I mix AI features into it. So I've started exploring what do plugins for my software look like that add large language model based, like, build, run a SQL query against this schema, all of that kind of stuff. But it's interesting.

Starting point is 00:09:30 Like it did very much change my mental model of the problem that I was trying to solve because it took such a big bite out of that problem. This episode is brought to by Kodium, the AI tool of choice or professional software developers. That is CODE IUM. Codium removes TDM from your development through a suite of state-of-the-art AI capabilities, available via extensions in all of your favorite IDs, such as VS code, JetBrains, Visual Studio, Eclipse, X-Code, Neovim, Jupiter Notebooks, and more. Uniquely, Codium is fully enterprise-ready. As proof, it had multiple regulated Fortune 500 companies, companies counted within its thousands of enterprise customers. Join the 700,000 developers using Kodium's individual free plan

Starting point is 00:10:14 and ask your companies to consider a free trial of the enterprise plan. To learn more about Kodium, visit codium.com slash pragmatic. That is C-O-D-E-I-U-M dot com slash pragmatic. And what I notice is you have been experimenting a lot with trying out how different L-L-L-Ms will work. You've been running models locally. we've been obviously trying a lot of, like, you know, there's the usual suspect tools, but even beyond that. Can you share a little bit on how your initial impressions were?

Starting point is 00:10:45 Because you were already on the early versions of the tool from Chad GPT to co-pilot to some other things. And how your stack has changed or refined to actually make you more productive, because it sounds like you are more productive now. Yes, very much so. I mean, so I've been calling myself an independent researcher when it comes to this kind of stuff. because I've got the time to, I can dig into these things. I write a lot. Like I've been blogging about this since when I first started investigating it. And yeah, I mean, like I said, GPT3, I was basically using it through their playground interface,

Starting point is 00:11:20 which still exists today. It's the API debugging tool for this stuff. And it was fine. And I was using it to solve. I experimented with having it like write documentation, but I've always felt a bit funny about publishing words that I didn't write because I do so much writing myself and little bits and pieces of code

Starting point is 00:11:39 but I didn't really get into the coding side until after ChatGPT came out and I did the advent of code that December the sort of month long programming challenge. So this was 2022 December right? Yes, November 30th is when ChatGPT came out and so I spent December

Starting point is 00:11:57 trying to learn Rust with its assistance which didn't, it was interesting I got a reasonably long way. Rust is actually, I still don't know Rust. Rust, the memory management in Rust is just difficult enough that language models still have trouble with it. Like, one of my tests of a new language model is, okay, can it explain the Rust borrowing

Starting point is 00:12:18 to me? And they're getting to a point where I'm almost understanding it, but it's an interesting sort of stress test for this. Whereas if you use these models for JavaScript and Python, they're phenomenally good. There's so much more training data about JavaScript and Python out there than there is for a language like Rust, that honestly, they just completely sing.

Starting point is 00:12:37 And that's great for me because the languages I use every day are Python and JavaScript and SQL. And those are the three languages that language models are best at. So I'm perfectly positioned to have these things be useful and helpful for me. And they've also got an inch...

Starting point is 00:12:54 I tend to pick, like I said, boring technology like Django, which the language models know already. You know, if you're sticking... If you stick with Django, they're going to be able to do pretty much anything that you ask of them. But yeah, so I tried learning Rust, and that was a really good exercise for just every day trying these things out and seeing what could happen.

Starting point is 00:13:13 One of the key things I've learned that I think people don't necessarily acknowledge, these things are really difficult to use. And there's a lot of, it's not just skill, there's a lot of intuition you have to build up in order to use them effectively. Like, if you just sit down and ask it a question like you'd ask on Stack Overflow, you'll probably not get a great response. And a lot of people do that, and then they write the whole thing off. They're like, okay, it didn't give me what I wanted.

Starting point is 00:13:39 This is all hype. There's no value here. The trick is, firstly, you have to learn how to prompt them. You have to, more importantly, you have to learn what kind of things they're good at and what kind of things they're bad at. Like, I know, because I've spent so much time with them, the Python JavaScript they're great at, Russ, they're not quite as good at yet. I know that you shouldn't ask them about current events

Starting point is 00:13:57 because they've got a training cut off in terms of what they're understand. I know that they're terrible at like mathematics, math and logic puzzles. Don't ask them to count anything, which is bizarre because computers are really good at maths and counting and looking things up. And language models, those are the three things they're not good at.

Starting point is 00:14:15 And they're our most, supposedly our most advanced computers. But so you have to build this quite intricate mental model of what these things can do and how to get them to do those things. And if you build that mental model, if you put the work in, you can scream with them. You can work

Starting point is 00:14:30 so quickly at solving specific problems. When you say, oh, this is the kind of thing that language model can do, and then you just outsource it to your, I call it my weird intern sometimes. Whereas other things, you're like, okay, well, it's not even worth trying that on the language model, because I know from past experience that it won't do a good job with it. So, like, as a software engineer, I mean, we do have a bit of an enduring mindset, but, you know, there's, when you see a new technology, I mean, you know, clearly this is, this is here. It's not going on a way. But there's two ways you can look at it. One is, I think you kind of explain. You start playing with it. You start stress testing it. You see where it works,

Starting point is 00:15:04 where it doesn't. And the other one is you start from a theory. You understand how it's built, how it works, what's behind the scenes. And then you start probing. And then you have it. You know, I think this is a little bit with the way computer science is taught. Like if you go to university, like when I went to computer science, we started with algebra and, and some like formal methods and and language and kind of coding was a little bit we got there by the end and they're like, well, yeah, I guess I now know what happens underneath the compiler, but obviously there's the other route as well. In your, like, you know, view, like, was there, sounds like you kind of like jump straight

Starting point is 00:15:43 into like, let me see how this actually works and let me not overthink the theory, which at the time it was a bit unclear. Right now, if you start with the theory, it will hold you back. Like this specific technology, it's weirdly, it's weirdly harmful to spend too much time trying to understand how they actually work before you start playing with them, which is very unintuitive. Like I have friends who say that if you're a machine learning researcher, if you've been training models and stuff for years, you're actually a disadvantage to start using these tools than if you come in completely fresh because they don't, they're very weird.

Starting point is 00:16:22 You know, they don't react like you expect, like other machine learning models. Machine learning people always jump straight to fine tuning. Fine tuning on these things is mostly a waste of time. Like, it takes people a long time to get to the point with like, you know what, there's no point to fine tuning at my own custom version of this. And then just next month, it'll not just to break it up for fine tuning because I think like we hear this word a lot, but by fine tuning, like, you mean that you take, you know, the model and then you add more training to.

Starting point is 00:16:52 You run wrong training cycles. And it's a very confusing term because, yeah, so the idea with fine-tuning is you take an existing model. It might be one of the openly licensed models or actually, like, I think Claude has this now, and opening have APIs where you can upload like a CSV file of a million examples and they spend a lot of money with them and they will give you a model trying tuned on that. And it sounds so tempting. Everyone's like, wow, I could have a model that's perfectly attuned to my specific needs.

Starting point is 00:17:20 It's really difficult to do. It's really expensive. And for most of the things that people want to do, it turns out it doesn't actually solve the problem. Lots of people think, I want the model to know about my documentation, my company's internal documentation. I want to take questions about that. Surely, I fine-tune a model to solve that.

Starting point is 00:17:39 That, it turns out, just plain doesn't work because the weight of all of the existing knowledge the model has completely overwhelms anything that you try and add into it with fine-tuning. The models, they hallucinate much. more on questions about things, if you've done that extra fine-tuning step to add knowledge, which is a surprising thing. Where fine-tuning does work is for sort of tasks. If you want a model that's just really good at SQL, you can give it 10,000 examples of here's a human question and the SQL schema and here's the SQL query. And that will make it, that will give you a model

Starting point is 00:18:11 that is stronger at that kind of activity. But for adding new facts into the model, it just doesn't work, which confuses people. And so then you have to look at the other techniques for solving that problem. There's a thing called RAG, which is a very fancy acronym for a very simple trick. It stands for retrieval augmented generation. All it means is the user asks a question, you search your documentation for things that might be relevant to that question. You copy and paste the whole lot into the model. And these models can take quite a lot of input now. And then you put the user's question at the end. That's it, right? Super, super simple. Don't get baffled by people. It's so simple. I actually wrote an article about it. And I had a, whatever the, the,

Starting point is 00:18:50 people who guest wrote it built an open source tool to, well, just a tool to do your own rack training and you could plug in chat, GPT. And, you know, I did it. I understand the code. And the code itself was very simple. And I was like, is this all there is to it? Like, you just break it up into, you know, chunks. You get some embedding so you can figure out where research will end you. And then you just add in that extra thing. And the only thing, obviously you can go down to the rabbit hole, but it's for simple rag is you decide on the context window size for the most part. And I was amazed at how well, as you said, like, it seemed so simple. So I looked at the code and I said, well, this, I mean, I'm not expecting much.

Starting point is 00:19:27 And when I tried it out, it worked really well. So one of those counter, it feels there are some counterintuitive things. Yeah. So rag, it's the hello world of building software on top of LLMs. Like you don't get it to pre-Hello world. You get it to answer questions about your documentation. I've implemented like 30 lines of Python. I've got one version that's like two dozen lines of bash, I think.

Starting point is 00:19:48 It's very easy to get the basic version working, but getting good rag working is incredibly difficult. Because the problem is that if you built the system and you know how it works, you're naturally going to ask questions of it in the right kind of format. The moment you expose it to real human beings, they will come up with an infinite quantity of weird ways that they might ask questions. And so the art of building good rag systems, the reason that it could take six months to actually get it production ready, is figuring out, okay, they were all of these different ways. that it can go wrong. And the key trick in RAG is always how do we fill that context? How do we pick the information that's most relevant to what the user is asking, which is really hard?

Starting point is 00:20:28 That's actually like it's an information retrieval problem. It's what search engineers have been trying to figure out for 30 years. And there's a lot of depth to that field. So RAG, just like everything else in language models, it's fractally interesting and complicated. It's simple at the top and then each little aspect of it gets more and more involved the further you look. One of my favorite difficult problems in this is what's called in the industry e-vals, right? Automated evaluations. Because when you're writing software, we write automated tests. We write unit tests, and they tell us of our software works, and that's great.

Starting point is 00:21:00 You can't do that with language models because they're non-deterministic. Like, they very rarely return exactly the same answer. So we don't even have unit testing, but with things like Rack, we need to have automated tests that can tell us, okay, we tweaked our algorithm for picking content. better? Like, does that do a better job of answering questions? It's really difficult. I'm still trying to figure out the right pattern of this myself. And I talk with someone who's working at an AI company. The weird thing that I would just, it just feels it breaks all that we know is they have this evile test suite, which runs against their model. Whenever they make a change, they run it.

Starting point is 00:21:38 And she told me like, okay, it's, it costs us $50 to run this every single time. Wow. And this is just something I don't think we've been used to. Like, you know, like I run my test, like as a software engineer, I'm a unit test integrated. I know how much time it costs me. But suddenly, obviously, they're using different APIs, whichever vendor. This is just, it feels like there's a bit of a, this clearly used to be the thing before my time, at least, like back when there were, you know, servers or mainframes or computing time was expensive. But suddenly, like, this is just yet another interesting variable. So. Yep. Yeah. So you don't want to run those on every commit to your repository, that will bankrupt you pretty quickly. It's also funny that, um,

Starting point is 00:22:18 With evils, one of the most common techniques is what's called LLM as a judge. So, you know, if you're trying to say, okay, I'm building a summariser. Here's an article. I want it summarized. Here's the summary. How could you write test against a summary to check that it's actually good? And what a lot of people do is they outsource that to another model. So they produce two summaries and then they say, hey, GPT4, which of these two summaries is best?

Starting point is 00:22:41 And I find that so uncomfortable. Like, this stuff is all so weird and difficult to evaluate already. and now we're throwing in another letter of weird language models to try and give us a score from our previous language models. But these are the kind of options that we're exploring at the moment. Yeah, it's interesting. Speaking about options, so you've experimented a lot with trying out different tools, including building your own and obviously copilot and other models. I saw you mentioned Claude, for example, when you're playing with. What is your current LLM stack and, like, day-to-day, how do you use?

Starting point is 00:23:17 it for actually coding on a dataset or on your projects. So my default stack right now is my default model is Claude 3.5 Sonnet, which is brand new. It came out maybe three weeks ago. I heard it's amazing for coding. It's amazing for everything. It is the first time somebody who is not Open AI has had the clearly best model. It's just better than Open AI's best available models at the moment. The team behind it, the company behind it, Anthropic are actually a splinter group from Open

Starting point is 00:23:47 They split a couple of years ago. And apparently it's because they tried to get Sam Altman fired, which you can't do. We saw this happen publicly six months ago. But they were like, they were early adopters two and a half years ago. They tried to get Sam Altman fired. It didn't work. They quit and spill up their own company. And they were some of the people who built GPT4.

Starting point is 00:24:07 So it's actually the sort of GPT4 original team. But anyway, Claude 3.5 Sonet is unbelievably good. it's my default for most of the work that I'm doing. I still use GPT40, which is OpenAI, probably their best available model for mainly because, mainly for two features, it's got code interpret mode, this thing where it can write Python code and then execute that Python code.

Starting point is 00:24:32 So sometimes I'll throw a fiddly problem at it, and I'll watch it try five or six times until it works, and I just sit there and watch it going through the motions. So I use that a lot, and then chat GPT, has the voice mode, which I use when I'm walking my dog, because you can stick in a pair of air pods, and you can go for an hour-long walk with the dog, and you can talk to this weird AI assistant and have it write you code because it can do code interpreter, and it can look things up

Starting point is 00:25:00 on the internet and such like. So you can have a very productive hour-long conversation while you're walking the dog on the beach. This, I was not expecting how the honest. That's that is the most dystopian sci-fi future thing as well. Like the voice mode. And this is the, this isn't the fancy new voice mode they demoed a few weeks ago. This is the one they found for like six months. It's so good. Like the intonation, the, the voices, it's like having a conversation with an intern who can go and look things up for you. And then so you mentioned the stack, but like if I imagine your data, you know, you've got your terminal or your coat. There's more to my stack. So it's, those are the ones I'm using in my browser and on my phone.

Starting point is 00:25:41 Got it. I use GitHub copilot. I use GitHub co-pilot. I've always got that turned on. I've been building this open source tool called LLM, which is a command line. Just a question. A copilot, what features do you use? Because it now has a competing feature. It does have a chat window if you want to use that. It has auto-complete.

Starting point is 00:25:57 Which ones you find most useful for your use cases? Mostly autocomplete, like old-school copilot. I've recently started using the thing where you can select some lines of code. There's a little sparkly icon. You can click that and then give it a prompt to run against those lines of code, and it'll do that. I don't use the chat window at all. I use Claude-free, I use Claude in the browser for what I would use that for. And it's great, you know, it's, it's, co-pilot's another interesting one where you hear from

Starting point is 00:26:24 people who like, I turned it on and it just gave me a bunch of junk and I turned it off again because it's clearly not useful. And again, copilot, you have to learn how to use it. Like, there's no manual for any of this stuff, but especially not for co-pilot. And that you have to learn things like if you type out the start of a function name and give it carefully named parameters with their types or type annotations, it will complete the function for you. And if you add a comment, it will, like,

Starting point is 00:26:49 you can, you can, you learn, you prompt it through the comments that you write, essentially. Yeah, I've actually started to use that. It's actually, again, no one tells you that. But once you figure it out, it's, it can be useful because that's how you can generate, like, either a small part, for me, just a small part or a function. Right. It just gets it. And again, like, as, well, I mean, it's not surprising, but the more context you give in

Starting point is 00:27:09 the comment, the more it'll, it'll, kind of do what you want if you're lucky. I think the other thing to know about co-pilot is that it's actually running rag. It's got an incredibly sophisticated like retrieval, a rag mechanism where every time it does a completion for you, copilot, it tries to include context from nearby in your file, but it also looks for other files in your project that have similar keywords in them. So that's why sometimes you'll be in your test. That's really interesting that you say that because we're going to get to the misconceptions,

Starting point is 00:27:39 But we've been running an AI survey. And one of the things that people really complain about saying is I use copilot, because it's the one that's easy to turn on in your ID. And people said that it only uses my files and I wish it would look at the project or understand the whole project. But it's interesting to say that because I think a lot of people don't realize that it is trying to do it or in smart ways. Most people, or not most, but a lot of people assume that it just only looks at whatever

Starting point is 00:28:05 you're seeing on the screen. No, it is looking at bits of other files, but it's, undocumented and it's weird and it's trying to do semantic similarities and all of that sort of stuff. What I do a lot of is sometimes I'll just copy and paste a chunk of one file into a comment in another so that it's definitely visible to copilot. That's great for things like writing tests.

Starting point is 00:28:23 You can literally copy in the code that you're testing into your tests. Dot pie and then start writing the tests and it'll fill in the detail. I'm now starting to understand. You know, when you said you need to learn how to use it. It sounds like you kind of are coming from the other way instead of like trying out and saying yea or nay. And, you know, like, because I guess you're working for yourself.

Starting point is 00:28:39 So it kind of makes sense that you want to make yourself productive. You figure it out how these things can actually make you more productive. Right. Absolutely. And like it's so much work. Like that's the, I think that the biggest sort of misconception about all of this is that you'll get this tool and it'll make you productive on day one. And it absolutely won't.

Starting point is 00:28:57 You have to put in so much effort to learn, to explore it and experiment and learn how to use it. And there's no guidance. Like I said, copilot doesn't have a manual, which is crazy. Claude, to its credit, Claude is. the only one of these things that actually has documentation that's really good. Like, if you want to learn how to prompt LLMs, the clawed anthropic prompting guide is actually the best thing I've seen anywhere. Open AI have almost nothing. There are so many hypesters and blogs and tweets and LinkedIn posts full of like junk advice, you know, all of the things like always tell it that

Starting point is 00:29:30 you are the world's greatest expert in X before you ask you all of that kind of thing. It's mostly rubbish, right? But there's so much superstition. Because this stuff isn't documented and even the people who created the models don't fully understand how they do what they do. It's very easy to form superstitions. You know, you try the, you're the world's greatest expert in Python thing and you get good answer. So you're like, okay, I'll do that from now on. It's kind of like if your dog finds a hamburger in a bush, every time you walk past that bush for the next two years, they will check for a hamburger, right? Because dogs are very superstitious. And it's that, but for software engineering. And then going back to

Starting point is 00:30:09 stack. So, yeah, a couple of tools. There's a few more tools. There's a few more. I talked about Code Interpreter. One of my favorite

Starting point is 00:30:19 Claude features is, again, in the future from a few weeks ago, called Artifacts, which is this thing where Claude can now write HTML and CSS and JavaScript, and then it can show you that in like a little secure eye frame.

Starting point is 00:30:30 And so it can build you tools and one of interfaces and prototypes on demand. And it's quite limited. They can't make API calls from in there. It can't actually see the results, so it doesn't have that debug loop that code interpreter has. But still, it's amazing.

Starting point is 00:30:45 Like, I've been, I've redesigned pages on my blog by pasting in a screenshot of my blog and then saying, suggest better color scheme for this and show me a prototype in an artifact. And it did. So, so cool. So I'm doing a lot more front-end stuff now because I can get clawed to build me little interactive prototypes along the way to help speed that up. So I'm spending a lot of time with that. I have my command line tool, LLM, lets you run prompts from the command line.

Starting point is 00:31:12 And the key feature of that is that you can pipe things into it. So I can like cat a file into that and say LLM, write the tests, and it will output the tests for that. And then just to understand, you just build like it's a command line. Are you running a local model or somewhere, a model on a server? LLM, the tool, it's based around plugins and it can talk to over a hundred different models. So is this an open source tool? Yes, it's my big open source. my open source language model command line project.

Starting point is 00:31:40 We'll link it in the show notes as well. And yeah, so it's plugin based. Originally, it could just do open AI. And then I added plugins. And now it can run local models and it can talk to other models too. So I mainly use it with Claude because that's the best available model. But I've also run like Microsoft's Phi3 and Lama and mistral and things. I can run those locally, which to be honest, I don't use on a day-to-day basis because they're just not as good.

Starting point is 00:32:05 The local models are very impressive, but the really high-end, the best of the best models run circles around them. So when I'm trying to be productive, I'm mostly working with the best available models. I love running the local models for sort of research and for playing around. And also, they're a great way to learn more about how language models actually work and what they can do. Because when you, like, people talk about hallucination a lot. I think it's really useful to have a model hallucinate at you early because it helps you get that better mental model of what it can do. And the local models hallucinate wildly.

Starting point is 00:32:41 So if you really want to learn more about language models, running a tiny little, like some of them are like two or three gigabyte files that you can run on a laptop. I've got one that runs on my phone. It's actually surprising. Yeah. There's an app called MLC, MLC Chat, and it can run Microsoft 53 and Google.

Starting point is 00:33:01 Google's Gemma and it's got mistral 7B instruction. These are very good models. Like if you ask them, like if you say who is Simon Willison, they will make up things. That's a great. I love, I use like ego searches to basically see how much they hallucinate. They'll say he was the CTO of GitHub. And I'm like, well, I really wasn't, but I do use GitHub. But they, like, I've used these on planes.

Starting point is 00:33:23 They're good enough of Python that I can use them to like look up little bits of API documentation. They can't remember and things like that. And it runs on your phone. It's really fun. Yeah, awesome. So, like, looking back, you've now been coding for, like, more than 20 years, right? Yeah. I mean, depending on how we count.

Starting point is 00:33:41 Coating professionally, people have been paying me for 20 years at this point. People are paying for 20 years. So, like, throughout the time, you know, we have seen some increases in productivity. May that be firebook coming out for developers or other things. Like, if you, could you talk through, like, what were kind of, like, bumps when you became more productive as a developer, and then when we get to LLMs, compare to how this bump compares to those ones. I love that you mentioned Fybug, because that was a big bump, right?

Starting point is 00:34:09 Yeah, Fybug was the Chrome Dev tools before browsers had them built in. It was an extension for Firefox that added essentially what you recognize as the developer rules now. And that was an absolute revelation when it came out, especially for me because I've spent most of my career as a Python program. my favorite feature of Python is the interactive prompt. I love being able to code by writing a line of code and hitting enter and seeing what it does. And then you end up copying and pasting a bunch of those explorations into a file,

Starting point is 00:34:39 but you know that it's going to work because you worked on it interactively. Firebug instantly brought that to JavaScript. Like suddenly you could interactively code against a live web page and figure things out that way. So that was a big one. I think the biggest... Yeah, I think just as a reminder, because some listeners were not necessarily around, but before Firebug, I was doing web development, and the way you debugged your JavaScript applications,

Starting point is 00:35:01 which were pretty simple at the time, but you did alerts to show things. We didn't even have console.org. Console.com. console.com. Was invented by Firebug. Yeah. So it was just really painful and really hard to debug.

Starting point is 00:35:13 And you also couldn't really inspect the elements. So you were changing. It was like doing it in the dark. And as you say, it was a game changer. And now these days, Chrome developer tools is better than what Firebug used to be. But Firebug was almost as good. as the Chrome developer tools today, in my memory, at least. So it was this huge leap.

Starting point is 00:35:31 And, like, I think for Fronton developers, like, it's hard to tell how much more. But I'm sure at least, you know, twice of productivity. I'll just say something because it took so much longer to fix things or to understand why things were happening. So, yeah, like, that was a big jump. So, five bucks a good one. The biggest productivity boost for my entire career is just open source generally. Like, so it turns out 25 years ago, you had to really fight to use anything open source at all. A lot of companies had blanket bans on open source anything.

Starting point is 00:36:00 Like Microsoft were making the case that this is a very risky thing for you to even try. That's completely gone out of the window. I don't think there's a company left on Earth that can have that policy now, because how are you going to write any front-end code without NPN? No. That's all. But that, so it was open source as a concept. And I was very early on in open source, you know, Django was a, we,

Starting point is 00:36:24 Django open source in 2005, Python and PHP and so forth all came out of the open source community. And that was huge because prior to open source, the way you wrote software is you sat down and you implemented the same thing that everyone else had already built. Or if you had the money, you bought something from a vendor, but good luck buying a decent thing. And then, of course, you can't customize it because it's proprietary. And that, the open source, and then on top of open source as a concept, it really was GitHub, coming along massively accelerated open source because prior to that it was source forge and mailing lists and cvS and subversion and just starting a new project you had like i started open source project

Starting point is 00:37:06 where i had to start by installing track which meant i needed to run a virtual private server and then get Linux secured and then install like the open source all terms of what gethub became it was great software but it was not exactly a one-click experience um so open source was absolutely huge and then you had GitHub, making open source way more productive and accessible and massively accelerating. Then the package managers. So PiPi for Python and NPM for JavaScript. I mean, the OG of that was CPAN for Pearl, which was up and running in the late 90s. And it's where we owe so much to CPAN and sort of how it made that kind of thing happen.

Starting point is 00:37:46 But, you know, today, the productivity boost you get from just being able to PIP install or NPM install a thing that solves your problem, I think my hunch is that developers who grew up with that already in place have no idea how much of a difference that makes. When I did my software engineering degree 20 years ago, one of the big challenges everyone talked about was software reuseability. Like, why are we writing the same software over and over again? And at the time, people thought OOP was the answer. They're like, oh, if we do everything as classes in Java, then we can subclass those classes and that's how we'll solve reusable software. With hindsight, that wasn't the fix. The fix was open source. The fix was having a diverse and vibrant open source community releasing software that's documented and you can package and install

Starting point is 00:38:32 and all of those kinds of things. That's been incredible. The cost of building software today is a fraction of what it was 20 years ago, purely thanks to open source. It's interesting because when we talk about developer productivity, it's a topic that will come back. And obviously it's very popular, very important for people in leadership positions, you know, who are hiring a certain number of people and their, their CEOs and we'll ask how are these people used. And right now there is a big, you know, push to say that Gen AI is adding this and this much productivity. But it's two things are interesting. One is that we don't really talk about how much just having open source or not having to do it ads. We just, I guess we just take it for

Starting point is 00:39:17 granted. And the other thing that I want to ask you, I want to ask you, like, how much more productive do you think with this current workflow you have, which is pretty advanced. It sounds like it. You're using a bunch of different tools. You spend a lot of time tweaking it. So I'm going to assume you're one of the software engineers who are using it more efficiently to your own personal productivity. How do you feel like how much more productive this makes you? And there's a KVit here. Obviously, it's hard to, you know, like be honest about yourself. But right now, the good thing is we don't have any like any polls. Vendors will, obviously have a bias to say AI vendors that it's helping them more. And, you know, people who

Starting point is 00:39:55 might not like these tools, they might have a bias to say like, ah, it's not even helping me. So I think the best answer we can probably get right now is just from like people like you looking honestly at yourself and like, eh, getting a side. So I think I've got two answers to this. It's difficult to like quantify this. But my guess for a while has been that I have had a giant productivity boost in the portion of my job, which is typing code at a computer. And I would estimate I am two to three times more productive, like faster at turning thoughts into working code than I was before. But that's only 10% of my job. Like as a software engineer, once you're sort of more senior software engineer, the typing in the code bit is not near, you spend way more time researching and figuring out

Starting point is 00:40:40 what the requirements for the thing are and all of those other activities. So huge boost for typing, for typing code. The other thing that's, and it does speed up a lot of the other activities, the research activity in particular. Like, if I need a little JavaScript library to solve a particular problem,

Starting point is 00:41:01 because I have a bias towards boring technology anyway, if I ask Claude or GPD4, it always asks for options. I always say, give me options for solving this problem, and it spits out three or four, and then I can go and look at those. And it's effectively using it as a slightly better, faster and more productive Google search because you can say things to it like,

Starting point is 00:41:20 okay, now show me an example code that uses that option. If you're using Claude Soney, you can say, show me the interactive prototype of that option. All of that. So that research stuff happens more quickly for me. There's a whole bunch of those sort of smaller productivity boosts. The bigger one, the more interesting one for me, is I can take on much more ambitious project because I'm no longer limited to the things that I already know. all of the trivia about. And I feel like this is one of the most important aspects of all of this

Starting point is 00:41:52 is if you want to program in Python or JavaScript or Go or Bash or whatever, there's a baseline of trivia that you need to have at the front of your mind. You need to know how four loops work and how conditionals work and all of that kind of stuff. And so I think there is a limit on the number of programming languages most people can work in. Like I've found personally, I cap out about four or five programming languages. And if I want to start using another one, there's a lot. So like a month, potentially a month long spin-up for me to start, get, get, and that means I won't do it, right? Why would I use go to solve a problem if I'd have to spend a month spinning up on Go when

Starting point is 00:42:27 I could solve it with Python today? That is gone. Like, I am using a much wider a range of programming languages and tools right now, because I don't need to know how four loops and Go work. I need to understand the sort of higher level concept of Go, like memory management and go routines and all of that kind of stuff, but I don't have to memorize the trivia. So given that, I've actually shipped Go code to production, despite not being a Go programmer just sort of six months ago that's been running happily every day.

Starting point is 00:42:56 And it has unit tests and it has continuous integration and continuous deployment and all of the stuff that I think is important for code. And I could do that because the language model could fill in all of those little sort of trivia bits for me. This episode is sponsored by TLDR. TLDR is a free daily newsletter covering the most interesting source in tech, startups and programming. join more than 1 million readers and sign up at tlDR.Tech. That is tldr.com. I sometimes dread going back to certain side projects where it takes me a while to spin up and remember

Starting point is 00:43:27 and it's in a language or an outdated framework that I just don't want to touch. And it's like what you said, the confidence is higher. And I can actually just paste parts in the chat, GPC or turn on GitHub copiled. And I know what good looks like. So I think when you know that, even in a different language. to have that experience. Like, if I was a brand new programmer, I don't think I'd be using it to write Go despite not knowing Go, but I've got 20 years of experience. I can look, I can read

Starting point is 00:43:57 code that it's written in a language that I don't know very well, and I can still make a pretty good, like, evaluation of if that's doing what I needed to do and if that looks like it's good. Well, and I guess there's an important disclaimer, right, that the more you look at languages, as long as it's an imperative language, like, you can read it, right? I think it will, will be a different if you. We don't really use, some languages are not as popular, like prolog and XML and some of these really declarative. I would trust myself.

Starting point is 00:44:25 Yeah, I would not trust myself to just look at prolog code that it had written me and make a judgment as to whether that was good prolog code. But I feel like I can do that with languages like Go and Rust, you know? Yeah. So with that, I think it's good. By the way, thanks for sharing. I think it's great to see that you are getting productivity, but it also took a lot of work. I think, like, a big takeaway for me would be anyone who's trying out is, like,

Starting point is 00:44:50 put in the work and experiment to figure out what workflow works for yourself and that there's just no answers. I mean, you've been, I think you've been experimenting a lot more than most people have and still sounds like it's a working progress. But with this, I really want to touch on misconceptions and doubts, they might not be misconception. There are doubts and questions that a lot of people have about these tools. Let's talk about resistance. Let's talk about resistance a little bit because I feel like the resistance, lots of people, I see so much resistance to this, and it's a very natural and very understandable thing. This stuff is really weird. You know, it's weird and it is uncomfortable and the ethics around it are so murky. Like, these models were

Starting point is 00:45:28 trained on vast quantities of unlicensed copyrighted data. And whether or not that's legal and I'm not a lawyer, I'm not going to go into that, the morality, the ethics of that, like especially when you look at things like image models, like stable diffusion, which are next, now being used when you would have commissioned an artist instead. And they were trained on that artist's work. Like that's, I don't go if that's legal. That's blatantly unfair, right? If something trained on your work and is now...

Starting point is 00:45:55 There's a person who wrote just this, that they tried it out, didn't work that well. Plus, they don't want to use it because they disagree fundamentally with this. And honestly, I respect that position. I think that it's... I've compared it to being vegan in the past, right? The veganism, I think there's a very strong argument for why you should. should be a vegan. And I understand that argument, and I'm not a vegan. So I have made that sort of personal ethical choice. And all of this stuff does don't come down to personal ethical choices.

Starting point is 00:46:23 If you say, I am not going to use these models until somebody produces one that was trained on entirely, like, licensed data, I absolutely respect that. I think that's a very, like, I've not made that decision myself. And, you know, for the code stuff, it's all, it's basically trained on on every piece of open source code they could get on. But it is ignoring the license terms, you know, the GPN licenses that say attribution is important. You can't attribute what comes out of a model because it's been scrambled with everything else.

Starting point is 00:46:52 So yeah, there are the ethical concerns I completely respect. But then there's also, it's scary, right? It is scary when you think, okay, I earn a very good salary because I have worked through the trivia of understanding Python and JavaScript and I'm better at that trivia than most other people. And now you've got this machine. comes along and it's better at the trivia than I am. Like it knows the things that I know, I mean, knows in scare quotes.

Starting point is 00:47:18 That is disconcerting. And I feel like there's a pessimistic and an optimistic way of taking some. The pessimistic way is saying, okay, I better learn to be, I need to go into the trades. I need to learn plumbing because my job is not going to exist in a few years' time. The optimistic version, the version I take on is I can use these tools better than anyone else for programming. I know, I can take my existing programming knowledge and when I combine it with these tools, I will run circles around somebody who's never written a line of code in their life and is trying to build an iPhone app using chat GPT. I can just do this stuff better. So we've essentially

Starting point is 00:47:56 got these tools that are, they're actually power user tools, right? You have to put a lot of work into mastering them. And when you've got that, when you combine expertise in using tools with expertise in a subject matter, you can operate so far above other people. And the competitive advantage you get is enormous. That's something that actually does worry me most about the resistance is I like people who are resisting this stuff, right? I like that they're not falling for the hype. I like that they care about the ethics of it.

Starting point is 00:48:25 I like that they're questioning it. I don't, it would upset me if that put them in a serious professional advantage over the next few years as other people who don't share their ethics, start being able to churn out more stuff because they've got this additional. It's like if you were to say, I don't like, I don't like search engines. I'm never going to search for an answer to my programming problem. That would set you back enormously right now. And it's, I feel like it's in a similar kind of space to that. Yeah. And so another, I guess, opinion I hear a lot is, well, it seems like this whole technology is plateauing. Like if we look at the past 18 months, chat GPT for,

Starting point is 00:49:07 is, okay, Cloud might be a little bit better, Sonnet. Okay, cool, but like, you know, let's ignore that for just a second. GitHub co-pilot hasn't changed all that much. So I do see a sense, especially for people who are managing engineers, and they're also playing with this tool saying, like, well, it sounds like this is what is going to be, you know, like we just use it as is, is this all? Like, you're more in the weeds. Do you see improvements or drastic improvements or little improvements?

Starting point is 00:49:34 That's a really interesting question. I mean, from my perspective, I'd kind of welcome a plateau at this point. It's been a bit exhausting keeping up with the stuff over the last two years. I feel like if there were no improvement, if what we have today is what we're stuck with for the next two years, it would still get better because we'd all figure out better ways to use it. You know, a lot of the, one of the most, one of my favorite advances in language models is this thing called chain of thought prompting, right?

Starting point is 00:50:01 This is this thing where if you say to a language model, solve this puzzle, it'll often get it wrong. And if you say, solve this puzzle, think step by step by step. And it'll then say, okay, step one, this, step two, step three. And often it'll get it right. And the wild thing about chain of thought prompting is that it was discovered against GPT3, about a year after GPT3 came out, was an independent research paper that was put out saying, hey, it turns out, take this model to say, think step by step. And it gets better at all of this stuff. Nobody knew that, right? The people who built GPT3 didn't know that. It was an independent discovery. We've had quite a few examples like this. And so if we are on a plateau, then I think

Starting point is 00:50:42 we'll still get lots of advances from just people figuring out better ways to use the tooling. A lot of this also comes down to whether or not you buy into the whole AGI thing. Right. Like so much of the mainstream. Right. And so so much like it's kind of like Tesla self-driving cars, right? You've got these, the CEOs of these companies go and say, we're going to have AGI in two. in two years time. It's coming. Nobody will ever work again, which helps you raise a lot of money. But it's also, it scares, I mean, it scares me. Like, I'm, I'm not convinced that human economies will work if all knowledge work is replaced by AI. And it also gives a very unrealistic idea of what these things can do. Because, well, don't forget, it's also happening with software engineers,

Starting point is 00:51:26 right? There are companies out there whose pitches, we will replace software engineers with AI engineers, which is a very direct, although I'm now starting to see a pattern of how this is really good for fundraising, because it means a lot of potential market. And don't forget that, that's who they're talking to. And once they raise the money, you know, they have that money. They can then operate. And often, like in this case, you know, with cognition AI, their claims are toned down to the point of it's pretty much a CIPO pilot. So there, but you see it in a main, it is scary because you see it in the mainstream media everywhere. This claim that software, like we are, I think someone said, we are replacing our own

Starting point is 00:52:06 jobs as software engineers. And as you said, I think it's the first time I've seen that written in the press. Maybe this happened like before I was born, but not recently. It's funny, isn't it? It's like, who would have thought that AI would come for the lawyers and the software engineers and the illustrators and all of these things that normally you don't think of as being automatable? But yeah, so the AGI thing, that leads lots of disappointment.

Starting point is 00:52:31 People are like, yeah, well, I asked it as this dumb logic puzzle and he got it wrong. How is this? But it also ties into science fiction. Everyone thinks about the Matrix and Terminator and all of that kind of stuff. Especially, honestly, the key problem here is these things can talk now, right? They can imitate human speech. And throughout human society, being able to write well and convincingly has always been how we evaluate intelligence, but these things are not intelligent at all, but they can write

Starting point is 00:53:02 really well. They can produce very convincing text, which kind of throws everyone off. So, so yeah, if you're in, if you're captured by the AGI hype, you're going to then I think, yeah, I think we're going to have a plateau. I'd be very surprised if we had anything that was AGI like. I'd also be, like I said, I have not been sold that this is a net win for humanity. I don't know how, how society you would cope with that. But if we, what we are seeing is incremental improvements. Like Claude 3.5 Sonet is a substantial incremental improvement over GPT40 and Claude 3 opus. The anthropic, the interesting thing about Claude 3.5 sonnet is that it's named Sonnet

Starting point is 00:53:44 because their previous Claude 3 had three levels, those haiku, sonnet and opus. Haiku was the cheap one, sonnet in the middle, opus was the really fancy one. They have said they're going to release Haiku 3.5. which will be cheap and amazing, and Opus 3.5, which is going to be a step up from Sonnet. I try to ignore the, it's coming soon. Those ones I am excited about in terms of it's coming soon. But yeah, so if you're buying into the AGI stuff, then I don't buy into it. I don't think you get to AGI from auto-completing sentences, no matter how good you are at auto-computing sentences. And then, yeah, if it's in terms of the plateau, I'm just, like, incrementally, improvement is enough for me. Like I want

Starting point is 00:54:29 well and the models I mean right now I want to be fair faster yeah. If we look through back through history like I'm I'm a little bit skeptical to believe that suddenly like fundamental things would change in in the software industry

Starting point is 00:54:44 you know there's always this people sometimes you know project that this time it will be very different and again there's always innovation but looking back we've always had innovation we've had some new technologies and then incremental improvement. So like pattern matching, that will be logical.

Starting point is 00:55:03 Obviously there's black swan events, right? Like who would have seen COVID come or this is also a breakthrough. But I think there's a part of like we're not just in a vacuum. There's not just this one event. And AI has been predicted to be around the corner by different people since the start of computing, really, to be fair. But I think the other something I think about a lot. is the impact of TikTok and YouTube on professional video creation, right? Like the iPhone is a, this is a really great video camera,

Starting point is 00:55:37 and TikTok and YouTube have meant that you can now publish videos to the entire world. And that has not killed off professional video. Like people who work professionally in that industry, they're doing fine, you know? Yeah. What's happened is millions of people who would never have even dreamed of trying to learn to stand in front of a camera or to operate that equipment. are now publishing different kinds of content online. And that's kind of my ideal version of the sort of AI programming thing is,

Starting point is 00:56:06 I want the number of people who can do basic programming to go up by an order of magnitude. I think every human being deserves to be able to automate dull things in their lives with a computer. And today, you almost need a computer science degree just to automate a dull thing in your life with a computer. That's the thing which language models, I think, are taking a huge bite out of. And then maybe, so there is a version of that where the demand for professional software engineers goes down because the more basic stuff can be done by other things. The alternative version of that is the thing where because a professional software engineer can now do five times the work they used to do, maybe two times, five times, whatever it is. That means that companies that wouldn't have built custom software now do, which means that the number of jobs of software engineers goes up, right? a company that would never have built its own custom CRM for their industry because you'd have to hire 20 people and wait six months, can now do it with five people and two months.

Starting point is 00:57:02 And that means that that's now feasible for them. And those five people are still getting paid very well. It's just that the value that they provide to companies has gone up. So that's the demand curve that I'd like to see. Well, and also don't forget, like one thing that we do talk about, or I think it's kind of a common knowledge, correct me, if it's wrong, but code equals liability. The more code you have, the more liability you have. And one thing, just what we're seeing is more code will be generated.

Starting point is 00:57:32 And at some point, I just think about this thing. Have you worked at a company or a team where you just had less experienced developers, one or two years experience? And you leave them for a while. Or you might have seen. Oh, yeah. And then what happens, right? Like, fast forward to two years.

Starting point is 00:57:47 You don't add anyone experience. You know, like usually, like my observation is like, it's, You get spaghetti code. It's a mess. It's hard to do. And then you pull in some people with more experience who look around. They point out some seemingly simple changes that are, you know, not that simple for the people. They simplify things.

Starting point is 00:58:06 You might delete a lot of code. And then all will be good in the world or those people get more experienced. But I do think about this part where, you know, a year in, everything still seems to be fine, right? Like the CEO of the company is like, oh, this team is shipping quickly. people are enthusiastic. And my sense is that there will be, there should be a demand. And again, like, I'm curious to hear your thoughts on this. But engineers who can go into the generate code and, for example, explain reason,

Starting point is 00:58:35 even when the machine fails to explain this complicated mumble, jumble, or just say, we're going to delete all of this and it makes sense. I'm confident I can tell you why I'm doing this. Right. And that's what I expect. That's the skill that you need. Like, it turns out the typing code, and remembering how four loops work, that's the piece of our jobs that has been devalued,

Starting point is 00:58:55 remembering that sort of trivia and typing really quickly. Nobody cares if you can type faster than anyone else anymore. That's not a thing. But the systems thinking and evaluating a skill that I think is really important right now is QA, like in terms of just the old fact, like manual testing, being able to take some code and really hammered it and make sure that it does exactly what needs to do, combined with automated testing, combined with system design and prioritization, there's so much to what we do

Starting point is 00:59:24 that isn't just typing code on a keyboard. And those are the skills which... The thing is language models can do a lot of this stuff, but only if you ask the right questions of them. Right. Like, if you ask a language model to write five paragraphs on how you should refactor your microservices, maybe it'll do an okay job,

Starting point is 00:59:42 but who's going to know to even pose that question and who's going to know how to evaluate what it says? So those decisions, these things, I don't think you should ever have them make decisions for you. I think you should use them as supporting tools to support the decisions that you're making. It's one of the reasons I love saying, give me options for X. And that's what we become. Software engineers, we are the people making the high-level design decisions, the people evaluating what's going on. I don't think you should ever commit a line of code that a language model wrote if you don't understand it yourself.

Starting point is 01:00:12 That's sort of my personal line that I draw. And yeah, so I do not feel threatened as a software engineer. And honestly, partly as a software engineer who's got good with this stuff, I really don't feel threatened by it. But just generally, I think the bits of my job that these tools will accelerate, there are a whole bunch of jobs, bits of the job that will accelerate, some of which are a bit tedious, some of which are kind of interesting. But it gives me so much more scope to take on more exciting problems overall. I love it. And if you can offer advice to two different groups of people, so two separate pieces, but experience engineers like yourself in terms of like, you know, like put in the years

Starting point is 01:00:54 work across different stacks, and also to less experience engineers who are like coming into, they're already working inside the industry, but obviously they're not at the level. What would you suggest to them to make the most out of these tools or to make themselves more future proof, if you will? I mean, my universal advice is always to have side projects on the go, which doesn't necessarily work for everyone. You know, if you've got like a family and a demanding job and so forth, it can be difficult to carve those out.

Starting point is 01:01:25 A trick I've used at companies in the past, I love advocating for internal hack days, you know, saying let's once a quarter have everyone to spend a day working or two days working on their own projects. That kind of stuff can be great. Good employers should always be able to leave a little. little bit of wiggle room for, you know, for that sort of exploratory programming, but some employers don't. But if you can get that, that's amazing. If you're earlier in a career, like, people in their 20s can normally get away with a lot of site projects because they have a lot

Starting point is 01:01:54 less going on. It's like when I'm managing people, I don't like people working super long hours and all of that. It's hard to talk a 20, like a 22 year old out of that. That's just sort of how people are wired earlier in their careers. So take advantage of that while you can. But yeah, I feel like, I'm doing my personal weblog. I'm using all sorts of weird AI tools to hack on that, because the stakes could not be lower, right? A bug in that, it'll break a page and I'll fix it. So that's where I've been using this thing called GitHub co-pilot workspaces

Starting point is 01:02:26 that they've just started trial. Yeah, it's a beta. You're in the beta. Yeah, and I've added four or five features to my blog using that. Some of them in live, like in meetings with people as a demo. I'm like, oh, let's show you this tool. I'm going to add auto-complete to the tags on my blog. And I did that last week.

Starting point is 01:02:42 And so I'm using my blog as a sort of fun exploration space for some of that kind of thing. But yeah, so if you can afford to do a side project with these tools, and like set yourself to challenge to write every line of code with these, have these tools write that code for you. I think that's a great thing you can do. If you can't afford side projects, just use them, like, get an account with, I mean, both of the best models are now free, like GPT4, with Open AI, Claude 3.5 Sonnet.

Starting point is 01:03:10 Now, you have to log in, you might have to give them a phone number, but you can use a free account with them. Use those and just throw questions at them. Sometimes have a question where you think it definitely won't get this and throw that in, because that's useful information, throw in basic things. Just work with them that way. I think that's definitely worthwhile. And play with the Claude 3.5 artefacts thing is just so much fun.

Starting point is 01:03:34 Like the other day, I wanted to add a box shadow to a thing on a page. And I'm like, what I really need is I need a sort of very light sort of subtle box shadow. And then I was halfway through prompting Claude for that and said, actually, you know, build me a tool. Build me a little tool with where I think I said where I can twiddle with the settings. That was my prompt. Let me twiddle with the settings on a box shadow. And it built me this little interactive thing with a box shadow and sliders with the different settings and a copy and paste CSS thing. and if I'd spent an extra 15 seconds on it,

Starting point is 01:04:04 I could have found a tool that existed on Google, but it was faster to get clawed to build me a custom tool on demand because if you're on a Google search, you have to evaluate the answers. You get back and then you click through and all of that. I'm like, no, I know what I want. So do that, right?

Starting point is 01:04:18 That's just wildly entertaining. I feel this of what you're saying is like, yeah, I mean, it's easier said than done, but experimenting. And I think your blog, which we're going to link in the show notes, is just a really good example. I did find myself a little bit re-energized reading how much weird stuff you're doing. Sorry for the...

Starting point is 01:04:37 That's the other thing. It's got to be fun, right? What is the things people miss in this space? I can see that you're having fun with it. Right. And again, thanks for sharing because you put it out there. I think, you know, that's another thing. But honestly, with these tools, it's a bit easier to write it up as well.

Starting point is 01:04:51 So I think that's helpful advice. Like, this is a crucial thing. These things are absolutely hilarious. And it's not like they can... Sometimes they can write it. joke that's good, but that's not what makes them funny. It's trying out weird dystopian things, trying something you didn't think we work and having it work. I get them to do, I use the voice mode. I use to do prank phone calls to my dog. So I'll be like, hey, chat, GPT, I need to give my

Starting point is 01:05:17 dog a pill covered in peanut butter. I need you to pretend to be from the government department of peanut butter and make up an elaborate story about why she has to have it. Now go, and it does it. And I hold the speak trip to my dog. It's just really, really amusing. So stuff like that is so much fun. For a while I was always trying to throw a twist into my prompts. I'm like, answer this. And at the bottom would say, oh, and pretend you a golden eagle and use golden eagle analogies. And they would say, well, if you're soaring above the competition, stupid things like that, right?

Starting point is 01:05:47 Just you can get it to rap, kind of. And it's awful. Like really absolutely appalling. But with the voice mode, you can say, now do a rap about that answer. And just, it's cringe-worthy. It is kind of wild how I don't really remember having a tool that we're talking about programming here, but you can get it to do all these things within a, you know, potentially even in the work context. Just throw it in there.

Starting point is 01:06:13 It's kind of, as you said, it is fun. So I like to look at that part of it. So thank you for the inside. And let's end with some rapid questions in the end if you're up for it. So these are questions I'm just going to ask and you just throw out whatever comes up. could you recommend two or three books to people that you enjoyed reading? Martin Kleppman's book, Designing Data Intensive Applications is... It's on my shelf, actually.

Starting point is 01:06:39 Absolutely incredible. The Blue Sky team told me, Martin Kleptom advises them that this is the book they have all on their shelf because this describes everything you need to know to build Blue Sky. It's kind of amazing. At Eventbrite, we had a book club, and one of the things we did with the book club is because nobody reads the book for book clubs, right? It turns out that just doesn't work. So what you do instead is you assign.

Starting point is 01:07:00 chapters to different people and they have to provide a summary of the chapter at the book club. So it's almost like you parallelize the act of reading the book. That worked so well. And I think that was the best book that we did for that one. And is there maybe a fiction book that you can recommend? So my favorite genre of fiction is British wizards tangled up in old school British bureaucracy. So I like, there's Charles Stross does the laundry files. series, which is about sort of secret like MI5 style wizards.

Starting point is 01:07:34 There's the River of London series by Ben Aronovich, which are a metropolitan police officer who gets tangle up in magic. I really enjoy those. Oh, nice. What's your favorite programming language and framework, and you cannot say Django and Python? Yeah, really putting me on the spot with this one. Oh, yeah. Okay.

Starting point is 01:07:50 JavaScript and no framework at all. I love doing the vanilla JavaScript thing. Basically, so I used to love JQuery, and now, document. dot query select at all and array. dot map and stuff. JQuery is built into browsers now. You don't need an extra library. It is kind of wild, yeah.

Starting point is 01:08:06 I remember that one, when I used to use it. I'm surprised. Nice. What's an exciting company that you, that you're interested in and why? So I'm going to plug fly.io here,

Starting point is 01:08:19 the hosting company, because partly because they sponsor some of my work, but no, actually, completely independently of their sponsorship, I picked them to build my dataset cloud SaaS platform. on because they're a hosting company that makes it incredibly easy to spin up secure containers for as part of your infrastructure.

Starting point is 01:08:37 Basically, I was trying to build this stuff on top of Kubernetes, which is not easy to use. And then I realized that fly.comio, their machines layer, is effectively what you can do with Kubernetes, but with an API that actually makes sense and pricing that makes sense. So I'm able to build out this SaaS platform where every one of my paying customers gets a private separate container running my software with its own encrypted volumes and all of that kind of thing. And so I don't have to worry about data leaking from one container to another. And it scales to zero. But it scales to zero in between the requests and all of that kind of stuff. So yeah, I'm really excited about Fly as a platform for specifically building that thing where

Starting point is 01:09:12 you've got an open source project and you want to run it for your customers, like paid hosting of open source. I feel like Fly is a really great platform for that. Awesome. Well, thanks very much. It was great having you. Cool. This has been really fun. Thanks a lot. Thanks a lot to Simon for this. If you'd like to find Simon online, you can do so on his blog, Simon Willison.net, and on social media like X and Macedon, all in the show notes below.

Starting point is 01:09:36 You can also check out his open source projects, dataset, and LLM, which are also in the notes. As closing, here are my top three takeaways from this episode. Takeaway number one. If you're not using LLMs for your software engineering workflow, you are falling behind, so use them. Simon outlined a bunch of reasons that hold back many deaths from using these tools from ethical concerns to energy concerns.

Starting point is 01:09:59 But LLM tools are here to stay and those who use them get more productive. So give yourself a chance with these. Takeaway number two, it takes a ton of effort to learn how to use these tools efficiently. As Simon put it, you have to put in so much effort to learn, explore and experiment on how to use them, and there's no guidance. So you really need to put in the time and experimentation. By the way, in a survey ran in the pragmatic engine about AI tools with about 200,000, software engineers responding, we saw some similar evidence.

Starting point is 01:10:30 Those who have not used AI tools for six months were more likely to be negative about the perception of these. In fact, the very common feedback from engineers not using these tools was that they use it a few times, but it just didn't live up their expectations and they just stopped using them. I asked Simon how long it took him to get good at these tools and he told me it just took a lot of time. He couldn't put an exact number of months on it, but it just took a bunch of time and experimentation and figuring out if it works.

Starting point is 01:10:56 My third and final takeaway is that using local models to learn more about large language models is a smart strategy. Running local models has two bigger benefits. Number one, you figure out how to just do these, how to run models locally. It's actually less complicated than one would think thanks to tools like Hugging Face

Starting point is 01:11:15 that make downloading and running models a lot easier. So just go and play around with them and see how smaller model feels like. The second benefit is that you learn learn a lot more about how large language models works because local models are just less capable, so they feel less magical. Simon said how it's really useful to have a model hallucinate at you early because it helps you get better at the mental model of what it can do, and the local models do hallucinate wildly. You'll also find some additional resources in their pragmatic engineer.

Starting point is 01:11:45 One of them is about rag, retrieval augmented generation. This is an approach that Simon talks about in this episode, it's a common building block for AI applications. We did a deep dive into Pragmatic Engineer about this approach, and this is linked in the show notes below. Also in the Pragmatic Engineer, we did a three-part series on AI tooling for software engineers, reality check. We looked at how engineers are using these tools, what their perception is, what advice they have to use these tools more efficiently.

Starting point is 01:12:14 Personally, I cannot remember any developer tool or development approach that has been adopted so quickly by the majority of back and in frontend developers in the first two years of its release, like large language model tools have done so since 2022. So it's a good idea to not sleep on this topic. And this marks the end of the first episode on our Pragmatic Injury and podcast. Thanks a lot for listening and watching. If you enjoyed the episode, I'd great to appreciate if you subscribed and left a review. Thanks and see you in the next one.

The Pragmatic Engineer - AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.