Software Misadventures - From "AI mid-life crisis" to the "time of my life" | Steve Yegge (Sourcegraph)

Episode Date: August 27, 2024

A Silicon Valley veteran and known for his writings like "The Death of the Junior Developer", Steve Yegge joins the show to chat about his "AI Midlife Crisis", the unique writing process he employs, and building the future of coding assistants.   Segments: (00:00:00) The AI Midlife Crisis (00:04:53) The power of rants (00:09:55) “You gotta be able to make yourself laugh” (00:11:46) Steve's writing process (00:14:10) “I published them… and nothing happened for six months” (00:17:30) Key to perseverance in writing? Get pissed. (00:23:24) Writing in one sitting (00:29:05) The AI Midlife Crisis (00:35:04) Management to IC (00:38:35) The acceleration and evolution of programming (00:41:43) Picking up new skills in a new domain (00:43:40) The power of prompt engineering (00:47:27) Secondary hashing (00:50:47) The importance of context in coding assistants (00:53:56) “The future of coding assistants is chat” (00:57:15) The importance of platforms in coding assistants (01:02:30) The nefarious T-word in AI (01:06:32) The death of the junior developer and its consequences (01:09:35) The future of code understanding and semantic indexing (01:13:15) The power of context in AI platforms (01:16:21) Surprising capabilities of LLMs (01:21:04) Transferable skills in AI product development (01:23:53) Mental health and the innovator's dilemma   Show Notes The Death of the Junior Developer: https://sourcegraph.com/blog/the-death-of-the-junior-developer Steve’s blog rants: https://steve-yegge.blogspot.com/ Steve’s medium posts: https://steve-yegge.medium.com/ Sourcegraph’s blog: https://sourcegraph.com/blog Steve on twitter: https://x.com/steve_yegge   Stay in touch: 👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! [e-mail]   Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en

And it's funny to watch people melt down over this. I guess I had my AI midlife crisis just a little before they did. Like I had mine about a year ago where I just had an existential crisis where it hit me hard. I was stunned. I was like reeling because I realized that many of the things that I thought that I was, that I kind of like my value, my sense of self-worth, my identity was tied up in my ability to do things like memorize long things or generate long things or whatever. It's all just BS now. Computers do it way better, right? So that was tough. I think a lot of other people haven't gone through that yet. So my blog really touched the

nerve. Welcome to the Software Misadventures podcast. We are your hosts, Ronak and Gwan. As engineers, we are interested in not just the technologies, but the people and the stories behind them. So on this show, we try to scratch our own edge by sitting down with engineers, founders, and investors to chat about their path, lessons they have learned, and of course, the misadventures along the way. Awesome. Steve, you're very well known for your rants, the Google Plus rant, I think being the most popular one. And I will categorize it as very insightful and suitable for the chili emoji, for the spiciness of it. So me and Ranec we also rant, but we do not have like the near, which we'll call it skill level.

So maybe some pro tips to get us started. Like what makes a really good rant in your opinion? A really good rant. Well, you got to start off by being really frustrated, ideally at a coworker or a team that's not giving you an API. And then you get really, really hammered one night. And then, yeah, the rest usually just comes naturally.

Love that. And when did this sort of start? If you can take us back a little bit. Yeah. So, I mean, like I kind of started it before there was like anything like blogging or anything like that i was doing it at amazon okay because i when i started amazon there was only like 250 people in the whole company like including customer service and stuff it was very small and so uh you know the engineering team was only like i don't know like maybe 50 or 80 people

and and so as they grew like i knew everybody personally right and as we grew like there was a large contingent of people that was using java but they were just like having to build really comical really dumb stuff back in the early days of java and uh and i started getting i started getting kind of mad that we weren't using better languages we were using like pearl and php and you know just a bunch of, right? So I started this kind of like soapboxing internally, right? To like 150, 200 people, it eventually grew to like 700, 800 people. Eventually I left, I went to Google, I brought them with me, the CTO, Werner Vogels, he like called me up and said that I needed to remove one of

them because it gave away a little too much confidential information. So fine, I took that one out and I published the rest of them and nothing happened nothing happened for six months so my actually my biggest piece of advice to you is like if you're gonna write something big like write it but then don't expect anything to happen for a few months it takes time for people to like absorb right six months later like on the nose all of a sudden people start walking up to me at google going yo i read your blog i'm like which? And they'd like name one. Oh, okay. Weird. And then somebody else the same day be like, I read your blog. And I'm like, yeah, which one? And they named a different one. And I'm like, what's happening? Right. And it was, it was that

somebody had found them and they made Hacker News and people were reading all of them, all the old Amazon ads. Right. So like at that point, I was point i was like all right well i guess i'll keep going so um that that part's pretty cool and i would say typically a lot of things i think most engineers rant but most people rant in private on calls which are either not recorded or people don't write about it in many of the posts slash rants that you have and sometimes your youtube uh tick tock series too you're unafraid to share your opinions and sometimes name names as well and typically at least when i was starting out in tech i was told silicon valley is a or tech community in general is a very small world so just just make sure you don't piss anyone off even if you don't like them.

I was like, okay, that's a fair piece of advice. Now, in your case, when you're sharing these opinions, a lot of people tend to agree with the opinions you have, but the people you're sharing the opinion of may not necessarily feel that way. So I'm curious, was this a concern ever in terms of when you share your thoughts or are you like, you know what, who cares?

Let's just share it. I mean, yeah, almost every single post I ever did at Google pissed off one VP or another, right? And they'd grumble at me about it, but they wouldn't censor it but you know i'm gonna i'm gonna say something that i've never said before but i'm becoming more and more convinced of it as i get like old old old which is that companies they really are people okay or they're very person-like and they can have a personality right that's kind of a

reflection of their leader plus their culture plus plus maybe their domain. And I think that as we move forward in this world where oligarchs are having more and more and more power and sway over governments, the only people that are powerful enough that we can turn to are corporations, right? And most of them, they don't talk. They're secretive secretive right they're all like a bunch of cloak and dagger stealing around alleyways because they got you know gun shy from you know from lawsuits or whatever right defamation or what whatever right you know plus plus your brand is always in free fall i don't know if you guys know this but the google's brand when it was peaked it was like early 2000s and i was talking to the public

relations folks and they were like yeah you know brands always decline basically over time i don't know amazon might be an exception to that right they've done really really well with their brand but you know yeah google google you know people decided they were evil like you know 10 years in they weren't trying to be i don't think so so yeah i mean like uh i think that companies need to talk more about what they do and what happens inside of them and the learnings that we can all get from each other that we don't share with each other i think it's absolutely ridiculous that people are so afraid to talk about the stuff that we all experience every single day in our corporate jobs we

don't share and you know what the time is it is time for sharing because tech is under attack because they're considered to have a monopoly on information you know like you know unbiased true stuff and that attack is gonna catch us real off guard unless like tech gets together. And right now it's all backward. I'm serious. You guys know this is happening, right? So like, I'm proud of the fact that I speak up when people are doing stupid stuff, you know, and I think more people need to do it. And you just need to just, if you've got a voice, like get it out there, you know, and if you've got something meaningful to say, you know, make it funny.

And more people listen. So you really got me ranting. That's one new brand new thought for me. Very well said. And in terms of the making it funny part. So like, I think this is unique to your writing style plus speaking style too. And I think that's what, at least in my opinion, makes a lot of difference. Where if I or someone else just goes online and says a bunch of angry things at people or at things that they are disagreeing with,

a lot of people wouldn't be happy, even the ones who would agree with it, because they would just see the anger. But then when someone reads your posts, they tend to always not, at least I do. And I'm also laughing while reading it or listening to you. How did you develop that unique style look you gotta be able to make yourself laugh right i mean like genuine you gotta think of something that's so funny that you were burst out laughing out loud and you're like oh i could never say that and that's the germ of your idea right there okay it's like okay how can say this and get away with it, right? And it's, you know, it's a bit of art, but I mean, like, you've got to practice it all day long.

You've got to, right, just see the humorous side of things and sort of remember it. And then when it comes time, you know, be bold, right? Say the funny thing. It's rough these days, right? But I don't know, for some reason, there's a lot of low hanging fruit in the corporate world, right? So when I have those ideas where in my head, right, I think of something and I'm like, oh my gosh, I'm like the funniest person ever. And then, you know, 30 minutes later, I go and I write it down. And then I try to, you know, like position and I'm like, okay, you know, okay, maybe not the funniest person I know. Is there a process you use in terms of...

So you said practice that all day. But more concretely... You get a bottle of wine. I actually quit drinking in 2016. But no, it's... I guess there's the Mozarts out there who like already have it well formed in their head and they just write it down and it's nice and clean. And then there's the Beethoven's that just like scratch it out over and over and over and over again until it's right. Okay.

I think most of us are like the Beethoven's, right? You know, you sit down and it's not right. You got a sketch of it. You know, some jokes in there, but some aren't landing or they're in the wrong order or whatever and so like it's you know the the writing process is pure generation and you're just trying to enjoy yourself and make a bunch of interesting points the editing process can take months i have blogs that i'm sitting on that i started writing last year that i still haven't published yet because they're not right yet you know and you can tell when I'm rushed and when people are rushing me out and they got to get a blog post out, it's not going to be as high quality when I can take my time and I can think it through

and research it and make sure the jokes still make me laugh. Two, three months later, I'll pick it up and go, ha, this is good. Why didn't I publish this? Oh yeah, because I said that. And so, you know, you go through this process, you know, right? Look, if you're not saying something that's, like, controversial, if you're not saying something that divides people actually fairly evenly, then you're not interesting. Right? Unless you're teaching them something, right, which is okay.

But, like, if you want to be interesting, you got to say stuff that's like, kind of like, you're not even sure if you a hundred percent agree with it. Right. But you know, you gotta be, you know, kind of, kind of thinking outside the box and, and, and also, you know, I don't know. Yeah. But that's the important thing, right. Is if you can go, if you can go back and reread it later and laugh then you know you got gold i like that do you have um these like archetypes in your head or like say proxies to friends or something such that you know that oh when they read this they

would disagree or like when they read this you would agree interesting that's how you form sort of like a 50 50 split sort of um yeah yeah yeah yeah and i actually learned that trick from like you know writers like stephen king and neil steffensen from their books like stephen king's on writing is really good kirk vonnegut did a really good one on it uh neil steffensen came to google and talked to us for like an hour and a half right that was a treat we were there like i don't know 40 or 50 of us in the room you know and you know we got tips from all of them of course the number one tip from every writer is write right get up in the morning and write and write and write and that's how you get good at it and there's a lot of truth to that but there's also a lot of there's a lot of

value in picking a person and writing the post for them one person and maybe having like a peanut gallery of other people that you know disagree and kind of just like think about how they're going to take it too right i absolutely do that i mean like i can even name the people that i wrote the last like five blog posts for i won't but i could like how do you find those people like um how do you yeah like out of all your friends or like you know people that you know like how do you find the archetypes uh you know i'll uh i talked to a lot of people through the industry and a lot of a lot of my old friends are like super super high up there now you know like they're they've got you know 20 or 30 000 people in their org you know massive you know fortune 10 companies or whatever and it's funny because the woes and

their their trials and tribulations are pretty much identical to the couple of hundred people companies and the series of startups and everybody's in the same boat right now right so listening within talk i think wow i could tell a story about something I saw here. And I know it applies to their org too. So I'll just kind of poke the bear a little bit. I'm going to do this on stage in Las Vegas in two weeks. I'm going to stand up and poke the bear a little bit. That would be fun. I'm giving a keynote at this conference.

So, well, yeah. I don't know. Where were we? We were talking about the art of writing. So you mentioned some of these books, like Stephen King's On Writing. I actually have it in my bookshelf. I haven't read it, but I will. Oh, it's great.

It's really good. Are there any other resources that you've come across that people could refer to? I mean, you know, I've tried, right? But I mean, no. I mean, like, ultimately, it's, it's like that, it's like that Twitter exchange with the guy who wrote the worst movie of all time the room right what's his name and somebody asked hey i want to get i want to start writing the screenplay what should i do and he replied start right i mean this is like fundamentally like well you know everybody says that everybody who's successful in any sort of a entertainment or whatever kind of business has been doing it for a long time look at you guys you guys enjoy immense success with this podcast.

And you've been doing it for a long time now, as long as high school. Right. Time really flies by. Well, I've been blogging now for, I mean, like, I guess, gosh, probably 20, 25, close to 25 years. And right. And so, like, you just everybody tells you this persistence, patience, just keep it up. And Neil Stephenson said this, too. He told us, he persistence, patience, just keep it up because, and Neil Stephenson said this too. He told us, he said, look,

your first novel is going to be a failure and your second novel and your third one, but eventually you're going to ride a hit and people are going to want more of it. And if you've got nothing, it's your first novel, they're going to be like, Oh, later. Right. But if you've got like an oomph, then they'll go and they'll, they'll make their way through it and they'll find value and stuff that maybe they didn't see before. It's really encouraging. But I know it's also discouraging because you're like, oh, man, I can't become an overnight

success. And nobody on YouTube, nobody out there is, right? That you've got to build it up little by little. What kept you going when, you know, in that six months of like publishing all these blog posts, right? Which I assume after you publish it, you're, in that six month of like publishing all these blog posts, right. Which I assume after you publish it, you're like, this is pretty fucking great. And then just crickets of like no sort of, you know, validation or no feedback from, you know, anywhere, like what kept you like keep at it?

Oh, you mean when I published all the Amazon ones, like when I joined Google or just in general, in like when there is not a lot of feedback on your writings. You know, I've gone through a lot of dry spells. Writing one of my blogs is so much work. It really is. I mean, you have no idea. The funny thing is I need to be able to, one of my rules is that I have to be able to basically write the entire blog in one sitting. And if I fail, I will go back and rewrite the whole thing in one sitting. So it has to flow. Now, during the editing process, they usually expand by 40% or 50%.

But – because I'm cutting stuff, but then other people who are helping me review it are adding stuff. Believe it or not, it's not me uh usually but yeah um uh i'm such a i'm just like i feel like everything that i say has to be new which is stupid i'm doing it wrong like to be really successful you're supposed to say the same thing over and over again right that's how you know uh demagogues work right but um i prefer to like not say the same thing over and over again because you can always go back and look at it. Instead, I try to find new angles and nuances to things and that can take a long time. So I maybe only publish

once or twice a year of anything serious. But what keeps me going is I eventually get pissed off about something. Time to... Right? I'm just r'm riled and that's what i know i'm gonna write that it's gonna be good right because i'm heated right but i'm also

kind of snarky so then jokes are gonna start flying right because i've seen him and so that's that's the formula right is you you think about it you let it rattle around you can let it bake and then at some point you get yourself worked up into you're in LLM generation mode, and you whip the whole thing out at once. So if we took an example of, let's say one of your recent posts, death of a junior developer, for example, which is an amazing post, we'll link that in the show notes, not that we need to, but we'll still do and people should check it out if we had to take that one like how what did it look like to write that one how long did you sit on it how long did it

take to write it completely and then what does editing look like yeah i started that one um probably and i started thinking i started realizing what i wanted to write about maybe in february and it didn't come out till. So it was a good three months on that one. And I am so glad that I researched this one a little more than usual, like in the sense that I like, I went, I went to a bunch of colleagues that I have a lot of respect for and found actually that some of them were like vehemently in favor and some of them were just like, nope. Right. So that was cool. That was cool. Right. cool right because i was like oh like i think both of these people are really smart and they disagree on this so we're on to something here right but i mean like it really made people mad like there's some youtuber right you know that like you know painstakingly read out all the words very slowly

and then got about halfway through and started judging it but the funny thing was his like followers like got into this heated argument about whether we matter or not as human beings i mean that's how fundamental the point is that i touched on i mean like come on seriously and they're like no we matter which is really funny because i am the definitely and that we don't matter right so that could actually be part of what's the resistance. But I'm basically saying, yeah, the old way of doing stuff, it's gone. Just let it go. Let the LLM do the work for you now. You're staring at a higher level.

You're not swimming. You're driving a boat. And people are like, no, that just means we don't matter. And it's funny to watch people melt down over this. I guess I had my ai midlife crisis just a little before they did like i had mine about a year ago where i just had an existential crisis where it hit me hard i was stunned i was like reeling because i realized that many of the things that i thought that i was that i kind of like my my value my sense of self-worth my identity

was tied up in my ability to do things like memorize long things or generate long things or whatever it's all just bs now like computers do way better right so like that was tough i think a lot of other people haven't gone through that yet so my blog really touched the nerve but i went through a lot of editing on that one we even took votes on the title we weren't sure if the title was going to be too harsh. Gene Kim, right, who's kindly offered to let me talk at the Enterprise Technology Leadership Summit in Vegas in two weeks, which is going to be awesome. He's the one that contributed, like, it was chatting with him that I realized this is hitting the whole industry. And that's when I realized, oh, wow. Okay. So this is big. And you know what? Like the funny thing is a lot of people came out, people were calling me

friends, like old friends, like retired friends were calling me. And I figured out the pattern. It was their kids are graduating from college and they were getting discouraged by my blog post. Right. And they're mad. Right. But the funny thing is like a bunch of other people called too all right uh we got calls from some very large very very large companies where you'd hear their name and go oh that's money and they have you know thousands and thousands of developers and a couple of them came to us and said yeah they've been seeing signs of this as well right we got we got confirmation so there's a huge change of foot and i was able to kind of like get in just at the right time and show people's nose in it. I think that blog definitely had an impact on a lot of people.

In fact, I forwarded that blog to a few of my friends who have their kids or someone they know who's about to enter undergrad and they were essentially trying to figure out or at least let's just say until last year their plan was obviously my kids gonna do computer science or the kid thought they would do computer science and now when they saw something like this or they just see lms in general like well does it still make sense should i still go on this route that i thought for the past four years that I should be doing? So when you get some of these calls, at least for the camp of people who have, let's say, their kids trying to go to school, what do you tell them? Well, I'm never going to discourage someone from going into computer science. I mean, it's an amazing discipline, and it'll give you the foundation that you need to be a great, you know, CHOP engineer, prompt engineer, you know, doing chat programming.

But, I mean, look at the... You saw the graph in there from Indeed.com, right, of software engineering jobs. And basically, they happen... Jobs come when there's money, and there's no money right yeah okay like there was that there was an anomaly i'm going to be talking about this in vegas but the zero interest rate policy and the trillion dollar stimulus package during cobus injected the covid injected

so much money into the economy that everybody had money for engineers and it caused all kinds of weird weird knock-on effects in the industry that i'm going to talk about. But, you know, like, there were jobs aplenty, yeah? And now that that's over, and moreover, right? Like, yeah, AI provided a bit of a bubble, a bit of a boost, but except in coding assistants, where we're seeing tremendous gains from AI, where else is it? No, it's just... I know that's a silly thing to say, but we know it's right around the corner. But still, investors are asking

this question. Where's the money? Because it's been a year, a year and a half. On these lines, so when it comes to publishing posts at let's say when you were at Google, companies that size typically have a policy for what you can write on their company's website.

You can publish on your own blog and you say, opinions are my own. The last few blogs that you have published are on Sourcegraph's website. And there's a marked difference. Like if I'm reading your post versus a post someone else wrote, all of them are good. Yours is way more entertaining. No offense to anyone else. Is there a policy at Sourcegraph that lets Steve publish the post when it comes out? Or do your posts go through the same review policy that others might go through as well?

Well, if I just said, I'm going to post this, post it, they would post it. But Sourcegraph are my friends, and I want to make sure my friends aren't caught blindsided. In fact, we've got one right now that was so, it's a really, really interesting blog. I mean, my God, it's probably really really interesting blog i mean my god it's probably one of my most interesting of all time but it's so sensitive that i basically have to write rewrite about three and a half or four pages out of 12 it's a big one because we want to make sure that you know we're sensitive i want to spend a couple more minutes on the writing part and we can move on to a lot of other topics. You mentioned when you want to, when you write a post, you want to write that in almost one sitting, at least the major ideas. Oh man, I was going to make a joke about, you know,

getting handcuffs. Like I was like, okay, you know, where are you getting the handcuffs? Like to cuff yourself to the chair? Or he's like like do you lock yourself in like a room and then throw away the keys like oh man still my joke running now cool i'm sorry continue please are there handcuffs involved let's just ask that first sorry um many times it happens to me where something gets stuck in my head where unless i do that thing i don't want to do anything else. Makes my family unhappy at times. Is that a similar situation here where you have this blog post or you're just so riled up that you need to write it down? So unless you do this, everything else takes a backseat. I kind of work myself into a frenzy, if that makes any sense. So like I'll set aside time and be like, I think I'm going to try to do the blog post this week.

I've usually got three or four at any given time that need to be written. Like right now I have two drafts that need to be rewritten and finished, right? And plus I've got two more ideas. And so I'll pick one that seems promising. I'll go back and read it, see if any of the jokes kind of make me laugh if I'm into it. Right. And eventually, you know, maybe maybe I won't and I'll just go program or whatever. But if I can kind of catch that feel of, oh, yeah, yeah, I remember now those jerks. And then at that point, I'm hooked. Right. And so, like, when I say one sitting, you know, I mean, like the reason I do it in one sitting, it's the whole what is it, the Edgar Allan Poe quote, you know, where he basically said, like, a good story has to be readable in one sitting. Yeah, you know, I actually and so if I type pretty fast and so if I write it in one sitting, then it's usually winds up being readable in one sitting for most people. And I find that that's a good. I don't know, it's a good forcing function, right? So that I don't write

forever. Sanyam Bhutani. And also that you actually write and finish. Yeah. And one last thing on this bit about just writing and rants. Have you ever experienced any backlash? Well, not people getting pissed off, but getting in your way, in terms of you want to do something, work somewhere, convince people, but something's getting in your way because of some of the things you've said. Sure. I mean, yeah. I mean, there are going to be a lot of people who, in this industry, who just don't like me. And that's going to be pretty normal if your name is known, and there's going to be some then yeah, they can get, you know, who knows, right?

But I mean, like by and large, you know, people are pretty respectful in our industry. And also I would say that my blogs have done more to open doors for me. I mean, like I've kind of developed a thick skin over the hate mail and the angry letters and all that. I mean, that was something that bothered me a lot more like 15, 20 years ago, right? But now I just realized, like, you could put out the best video game of all time and get a bunch of hate mail, and I'm like, oh, right? I mean, like, I saw that happen with things like Tears of the Kingdom

or whatever. It's like, oh, I get it. There's just a lot of angry people out there, right? So you don't let that part bother you. But yeah, I don't know. It's, um... Yeah. That's, yeah. That's fair. I had a follow-up for the, you know,

I know I'm not in the group being able to vote on the titles of the posts, but I love the AI midlife crisis as the title. Would you be interested in like diving a little bit more into that? Like what? Into my AI midlife crisis for real uh yeah i mean i guess sure i mean yeah loms are terrifying right i mean like i just like, I just, I don't think people, I don't think people have really,

most people have pieced it together yet. But I remember hearing, I don't know, four years ago that like, there were half a billion users of this chatbot in China. And this is before LLMs, right? This is with good models, but definitely not transformers. And I heard it's quite addictive, or it was, right? You got half a billion people addicted to a chatbot. That's going to happen here within a year, right? Because opening eyes, even warning people. You know, and it's like, it's so it's a combination of like, I worry very, very, very much about the social consequences, people's bank accounts are going to get drained. Right? Because you're somebody who's going to get a phone call, a banker is going to get a phone call from somebody

who claims to be you, and it sounds just like you and blah, blah, blah, right? Or you're going to get a phone call from a family member or whatever. this is terrifying because you could happen it could start happening like today yeah you know so i mean like the midlife crisis has more been like a kind of a rolling snowball where first i realized i don't matter right our brains are just neural nets you know uh and then i mean absolutely you can have beliefs beyond this of course and can can and do, right? But still, our brains aren't really particularly special. We'll be able to engineer them, right? So that's a tough hurdle for a lot of people to get over.

And then when you look at what's going to happen to our culture, to things that we cherish and hang on to, even hardened progressives, I think, are not totally prepared for how much that's going to change maybe i'm i mean like people call me a doomer though right so i don't know um was there any did you come up with any sort of solution you know including buying a motorbike to kind of counter react this midlife crisis? So like, I mean, like, just to give you one concrete example, I'm, I'm consider myself to be very, very good at arranging music for a guitar, whether it's electric guitar, acoustic guitar, whatever. I can play almost anything. Like I played, at one point I was playing all 24 Paganini's violin caprices on a steel string guitar, that kind of thing.

Hundreds and hundreds of pages of transcriptions, right? And as soon as LLMs came out, I was like, oh, well, that's done, right? No more of that. Like that's just stupid because a model will be able to do it. You just give them a bunch of examples and be like, give me a good transcription and give me a good arrangement. It's just a little bit of prompt engineering. That skill that I developed over all those years is just stupid now, right?

And it's like, so I don't do it at all. I throw it all away. I'd love to train a model. I'd love to keep doing it and use prompt engineering to make it my own music or arrangement or whatever. Ultimately, all the clever work of mapping fingerings and voices and doing inversions and stuff like that,

the model's going to do that. Basically, a lot of things that are pattern matching are almost tedious that I got really good at in my life. I wouldn't even do now. Like, why would you do it? You guys look really depressed,

so maybe I should talk about it. I think, so, we are expecting a baby in a couple months. Well, congrats. Thank you. Some of these, on similar lines, when I think about AI LLMs and all the pieces you said that we have come to cherish as part of our culture, I do think about that. And then I find myself without any answers. And I put my hands up and it's like, what am I going to do? Let's just roll with it. That's right. That's right. Roll with it. Big wave coming in, surf it. So eventually, just at least in my head right now, the way I'm making peace with it is humans somehow figured out a way to evolve with circumstances changing.

This is one of the circumstances, maybe a bigger one than others. And where some of the skills mattered till now, which may not be relevant anymore, like memorizing stuff, for example. Like this is, I don't know about many people growing up, but at least back in India when we were growing up, the kid who remembered names, the kid who remembered numbers, the kid who could do some of these things,

like, oh, I can look at that car's number plate and know exactly whose car that is. Like these kind of stupid games used to matter in a way. But at this point, like, memory, do you care? Like, pick it up, make it general. I can recite pi to 50 decimal places because I learned it in fifth grade. Exactly. So, like, these kind of things may not matter anymore.

But I guess there will be different things that will matter, similar to not having to worry about swimming, like you said, but riding the board and what that looks like. That's right. We're leveling up. We're leveling up. That's what's happening. Like I said before, we were swimming and now we're driving a boat. So you're not just driving the boat, you're building one too. Talking about building AI products some time back. So you joined Sourcegraph, I think, towards the end of 2022, if I remember correctly. You joined as head of engineering, and then you transitioned to being an individual contributor

working on Kodi, which is Sourcegraph's AI code assistant. Can you tell us a little more about that transition? Sure. It's not the first time I've done transitions like that in my career. I worked my way up at Amazon at one point to a senior manager and then just handed it off to one of my directs because she was enjoying it more than I was. There's a lot of people side of management that a lot of people thrive on and they love it, you know, and they want to mentor people and they want to guide their careers. You know, for me, you know, I've done that for a long time. And for me, these days, it's more about what can I build and how fast can I build it? Right. And, um, uh, it would have been fine. I would have been fine to stay in the head of engineering role, except that AI was so new and so different.

And I was doing so much non-engineering stuff in my role that it was, I was starting to feel like I was never going to catch up. And I started getting worried, right. Because like, for example, I never could have written death of a junior developer that, that, um, for example, I never could have written Death of a Junior Developer, that post, had I not made that transition and been coding, like, the whole first half of this year as an actual engineer, like, helping us get, you know, get the product out. And I'm so happy I did, right? And now I can see, I can see where it's going in a way that I never would have been able to see that in a management leadership role. Now, will I go back into leadership someday that maybe possibly right but i mean with this ai stuff because like you said we're building the boat uh it is the time of my life and i'm not i'm not exaggerating the slightest i thought about this i've had some pretty darn good times in my life but this is it this is the best in terms of just my professional career.

Having a lot of fun, right? Because like there's a, I blogged about this a long, long time ago. I think it was called Dreaming in Browser Swamp or something like that. But it was basically like I was talking about this feedback loop that you get into when you're building an IDE or an editor or something that where as you build it, you're making yourself faster and it's so addictive. Yeah. And that's what we're, and we're doing it. We're doing it with Cody now and,

and everything that we do is wiring up this deeper and deeper, you know, integration with the LLM and it, there's nothing like it, right? You're just like, you're running. It's like you're, you're, you're, you're in a personal hovercraft or something. And, and you wonder where, where's it going to, where is it going to lead? Where is it going to go? Because not only are the engineering, the tools, and the protocols and stuff getting better,

the models themselves are also getting way, way better, really fast because of the competition. And so, you know, probably this time next year, it'll be commonplace to have models and workflows that span big graphs where you've got to get an application built and the tests and the CI CD and you've got to get it deployed and maybe even look at the logs. And we're finally at a point where you can expect a lot of that stuff to get done for you in a couple of passes with the tools that we'll have.

A couple of years after that, this is what I'm saying what happens to all the I mean, anybody who's doing old style programming? I become a chop engineer. So got to go be shot. There's a lot of things I want to dive a little deeper into. Sure. Before we go there. You said just buildingodi is like you having the time of your life. What's different about building AI products than it is to build something else?

And you've built a lot of things in your career. I mean, like, it's the, look, it's like driving. Like the number of red lights that you hit has a big impact on your sort of perception of the ride. And like when I was programming in the 90s, if you got stuck on something, you could expect that it might take you days to weeks to get it figured out because you'd have to go look in books or try things or work with colleagues or whatever. Right. And so you just like it was like we were in slow motion and it's gradually beginning faster. Google search made it a lot easier to find stuff. And then Stack Overflow made it even easier. You start asking people stuff and getting answers. Each one of those was a big accelerator, yeah?

And each one of them made programming more fun, didn't it? Oh, absolutely. I mean, if you went through any of those transformations. And cloud computing, I think think made programming more fun i know i mean it's a pain but it was a lot more fun than running i had my own computer in a data center in downtown seattle and when my game would go down sometimes it was because the computer like wouldn't reboot and i had to like go in the middle of the night and get a key card and reboot a server on the 13th floor of this horrible hot building. I mean, cloud competing was a real, real level up for us, right? And this is no different, right? This is like, wow, this is a huge level up. It's a toy. It's a fun toy. It's like, you can make it do things. And you're like, well, I've been doing this for

six months now. I still didn't know it could do these things, you know? So, so that's kind of like where I'm at right now. That's why that's why I think it's like time in my life. It's like, because programmers, we like to build things that people use, build things that have an impact. And, and that that gives us that dopamine cycle, right? And so if you're speeding that up, it's getting more addictive. Speaking of that history, so you were actually retired before coming back to Switchgraph. And you mentioned before you transitioned from management to IC. I'm curious, what are some of the skills that are really useful in just picking up a completely new domain, right?

Like all this stuff didn't exist, right? Like the last time that you were like, you know, like doing coding. So then like, yeah, like how do you go about when diving into like a completely new domain? Like what do you do? You know, there's an interesting question behind your question. I feel like lurking there, right? Please help me up if that a question. Well, I mean, like it's related to something like lurking there, right? Which is... Please help me up if that a question. Well, I mean, like, it's related to something we talked about earlier, right? Which is,

if I'm going into software engineering, should I be going into software engineering, right? So, I guess, let me just make absolutely sure, state your question one more time, make sure I'm going to answer the right one. Sorry. When you go into a new domain, like building an AI product now with Cody, like how do you go about picking up the new skills that are relevant to building it? Yeah. When AI came out, a lot of the folks, you know,

that I worked with didn't want to have anything to do with it. Like they just didn't, they weren't interested. This was people outside Sourcegraph or outside Sourcegraph? Both. Both. And they just, you know, because maybe it was a misconception. They didn't know that it was, it wasn't like a bunch of PyTorch anymore.

It was engineering involved now. But for whatever reason, you know, a lot of people feel like the entry into this new domain is actually so steep that they're scared of it or worried or they're just putting it off. And so, like, seriously, I mean I mean like getting into a new domain, I mean the best thing you can do is be young. I'll tell you that right now. It's harder for me. And so like when I got to learn a new programming language, I can't just like read the book in two hours and like know the programming language

like I could 20 years ago. So that helps. Like just spend the time while you're young. Learn as much as you can. But seriously, I mean, don't be afraid of new domains. I mean, we're all kind of reinventing our jobs kind of all the time anyway. And this is necessitating that we all reinvent our jobs as prompt engineers. Even if you're doing regular programming, you're going to have to do a fair amount of prompt engineering from now on.

Right? So how do you learn it? Well, I mean, gosh, there's so many more resources now than there used to be. Right? But, you know, what I would do, I'd use Chop to figure it out. I'd go into Emacs or ChatGPT or whatever, go into Kodi, and I'd start asking there. Because why bother with Google searches anymore?

You're going to get summaries from the AI, right? The AI knows how to even present it to you at your level. You can say, I'm a complete newbie, or I actually know a similar domain. Can you compare it to that? So the answer to your question is different than it was a year ago. How you ramp up in a new domain is you go ask the LLM about it.

That's my opinion. It'll make you a better engineer. Because there's a lot of nuance. There's a lot of tricks. You know what? There's a lot of people out there today who still possibly listen to this podcast. Yes, you, who think that CHOP is not a thing

because uh they asked the lom to write some tests for him and the tests were wrong and therefore lms are dumb which is just like saying you know a google i did a google search and it didn't come up the right answer so google's dumb right i'll tell you who's dumb there it's not google and and and it's it's because people don't realize that fundamentally, I mean, you have to still be patient and persistent and make it your own when you're doing shop. So you have multiple questions around trips with the LLM, right? People aren't doing that, right? They're not doing that. But that's, I mean, fundamentally, I mean, prompt engineering and I'm not as scared.

If you go back and look at my, seriously, you go back and look at my chat history in Gemini, in Cloud, and in ChatGPT, you'll see that I am ramping up on new domains right there in place as I'm coding. Haven't used WebSockets in 12 years. Let's go through the fundamentals again, right? Oh, man, I haven't used Windows in 15 years. Walk me through how I, like, install it. I mean, literally anything you want to know, you literally ask the LLM while you're coding. And so everything that you're doing becomes this like, it's in a blender, it's blurred together.

You're working and doing all the things that you need with the assistance of your coding assistant, your LLM. And you're cranking out a bunch of code, but you're not the one writing most of it. Do you guys do this? There's like a bit of a cycle for me as well where at some point i was kind of like you know that meme of like can't tell you the lm really bad or i'm just really bad at prompting the lm and usually i think it's the

latter so it took me a while i think to learn like oh yeah i need to you know really be strategic and then i think to a point, like in terms of reducing red lights, one thing I really realized it was like, oh, if my prompt is bad, I can just ask the LLM to improve the prompt. I think that was like the biggest unlock where I was like, oh shit, this is like way better than what I would have asked it.

So yeah, it's been pretty big. Yeah, I read papers on prompt engineering all the time So, yeah, it's been pretty big. Yeah, I read papers on prompt engineering all the time, and that's a new one, right? It's rephrase and expand. You tell it and you'll get a better prompt. But what I found, honestly, truthfully, okay, is that I get good results without resorting to special prompt engineering techniques. Models are good enough now that just a regular conversation with the right context gives it everything that it needs. And the context arguably is more important than the prompt.

This is my opinion because it's going to get – I mean, you want prompt engineering – you want it to be really, really, really good if it's doing some very specific task for you. Of course, you want the prompt to be really good. If it's the right unit test prompt, we're going to put a lot of work into that. But for solving any given random problem that you're doing during the day, it's a throwaway prompt. So you don't really need to put that much effort into it. And instead, what you've got to do is you've just got to recognize that some of the time it's going to just make stuff up. And you'd be like, no, no, no, that's made up. And they, oh, apologies for the confusion or whatever they tell it to say these days. Right. But you just you just you just steamroll past that.

You're like, nope, that didn't work. Nope, that didn't work until the thing does the thing you want. And if it starts to go into that circle, you mentioned this. Sometimes it'll start giving you the run around and you realize the model cannot make any forward progress. Just treat it like a chess match, right? If you hit the same position like three times, it's still going to go take the whole conversation and take it to the next LLM over, right? Just go in a circle and say, this LLM is having a lot of trouble. And the other LLM would be like, they sure are.

Here's your problem. And they'll like tell it out to you, right? It's so cool. It's kind of like the magic property of second hashing right second secondary hashing the second hash function almost never ever ever has a collision just because you know because the odds and i just it's just awesome that you can go to a second lom and pretty much always get the answer that you were missing the first time around you got a third lom interesting i need to try that oh very cool dude this is a chop technique there's a bunch of it i'm starting to do some videos on it and whatever it's like there's an art to it

should do a post on chop techniques i think that'll be super helpful for a lot of people out there that is one of my uh completed drafts that needs to be rewritten to be funnier but yeah it's ready to go awesome uh thanks so one thing that you mentioned there's a lot more engineering to this now to building air products i was curious generally when you talk to a lot of people who are not building air products or software engineers they also have this fomo of like hey kind of missing out but you know i'm not a machine learning engineer and they may not have the right opportunity either at their own job or they can't find one these days because it's generally harder.

But just to put things in context, building AI products at this point where you're not necessarily training an LLM yourself, but let's say you're building something like Kodi. How much engineering work is that versus how much researchy work is that? You get to decide where to turn that dial. I mean, it is a dial. And I mean, look, why is that the case? It's because there is sort of like

infinite low-hanging fruit in the space right now. I mean, really low-hanging fruit. Like we're like even just, I think two weeks ago, I was in Krakow in Poland for, we did a hackathon, a year of hackathon, right? And I realized one of our teammates, we were chatting and we realized that nobody has gone and fixed our chunking for embeddings and it's been a year and we still don't,

like an embedding will literally, like the chunk will be from the middle of one function to the middle of another function which is just totally useless for the lrm right there's a low low low hanging fruit right so of course we fixed that yes sorry you know but um and we have lots of other things besides embeddings that can get us indexes but you get the idea right it's just like the the road is so clearly set out for you it's so obvious what to do there's too much to do and so you got to decide okay all right we're how are we going to allocate our resources this is a strategic sort of battle early on in the game like last year i thought we were going to like really double down hard on you know on the ai side you know and we've got it we've got a great ai team

and they're doing great stuff you know fine tuning and eliminating bias side. And we've got a great AI team, and they're doing great stuff with fine tuning and eliminating bias from models. And they've got a trained context ranker. Because what we found ultimately was the biggest bang for the buck that you get in the coding assistance space, aside from the LLM itself, the biggest bang for the buck you're going to get is it's not the fine tuning models, right? It's it's and it's not the prompt engineering.

It's your context. Okay, it's because because you get a lot of tokens. You don't get anywhere near enough, but you get enough to put a lot of context in there, right? And even if you're doing raw chop without a coding assistant, if you can just shovel screenshots and paste in logs and stuff, right? That context means everything. Because every time you talk to the LLM, and I think people have trouble getting their heads around this, every time you're calling a phone like, you know, around the world, and the person's never met you before, you know, hi, I'm a programmer, and you got to explain your problem to them and give them all the context that they can give you the answer every single time you know which is why coding

assistants are actually so useful in this game yeah so you had this blog post cheating is all you need i think it came on sometime last year again one of the other amazing blog posts we'll link it in the show notes and recommend people check it out. You had this very simplified diagram of how Kodi works, which could be a diagram for any coding assistant for that matter. It's like you have a human developer on one side,

you have the assistant in the middle, and then you have an LLM and you have some source that is giving you embeddings, etc and you tie all of this to ensure it right it is the rag uh how much of that has changed since then uh how much of it has changed okay so like in in other words yeah go ahead rag is still big everybody's doing doing rag. Rag is well, well known and acknowledged to be the right way, right? Because

fundamentally, you got to pick the right context put into the that's, that's, that's the goal of rag, which is to use basically local local, local searches. Now, it's definitely expanded in terms of like of the techniques and the approaches and things that go into RAG. A customer of ours introduced me to a really, really interesting new kind of source code index, for example, that I think we can bring in. I've got to go write it up. It's just mind-bending. People are innovating like crazy in this space. Yeah. So fundamentally hasn't changed. So then you got to ask the

question, well, if they're all doing the same thing, like how is your coding assistant differentiated from the others? Right. And to some extent, you know, we thought we were going to maybe differentiate on like local AI, smaller models, you know, like we do with our content ranking. So to the extent that you can do that with RAG, yeah, we can differentiate and we are. But for the most part, the effect of the actual foundation models in each generation is so overwhelming compared to the other stuff that you're doing that you pretty much just got to decide like, okay, how am I going to ride this train to maximize value for ultimately myself? I'm building this for me, right? For programmers. And yeah,

what we've decided is that chat is the future. Chat is where it's at. Everybody's been so focused, so narrowly on completions. Oh, yeah. I never understood it. Completion rate.

Yeah, and car. Yeah, completion acceptance rate. It's just been the industry's been obsessed with it. And completions are like, they're okay. I mean, they're kind of gimmicky, right? I mean, you're coding along, all of a sudden it goes, and it finishes it for you, and you're like, dang, that's pretty clever, you know? But that's not, I mean, like, most of what you're doing.

Most of the time you're not doing code that's conducive to completions in a way that they would help. Whereas if you could, plus it's hard to guide the completions. You get into this mode where you're like putting comments into the code and hoping it picks up on the comments and right. So like there's clearly what you need is sort of like prompted completions, right? Which, which we have with inline edits, which is a really cool feature where you just type,

make these changes to the code and it'll do it, which is sort of like a directed completion. And we're even working on some modalities kind of between those two, like super completions, right? Things like that. But then completion, they're almost a niche compared to how often you use chat. Once you realize that chat covers all phases of programming, right? From the generation of the code to the fixing of the code and the diagnosis and the troubleshooting

and every part of the lifecycle, chat can help you with it and accelerate you. And at that point, completions just fade into this gimmicky background. And I think most people haven't made that transition yet, but they will over the next six months. And they're going to realize, oh, wow, the coding assistants that focused on chat are the ones that are going to help me the most. And that's what you're going to find. So you mentioned this in your blog post too, after Chad, GBD4 and Clark 3.5, Sonnet, we saw this huge jump in the performance of coding assistants. And just the understanding of the code base base you could ask them to do complex things

like at times i've asked it to refactor certain methods and before this you could try i don't know how many loops with how many other lamps and they would just not make progress but now it's been very different not exactly how amazing either my prompts are really bad or lms aren't that great uh one of the two but they're not exactly where you want it to be. But in terms of the improvements that we are seeing and some aspects that you mentioned, like focusing more on chat than auto-completion, there is an aspect of your foundational capabilities are just improving by so much. And there's a step function improvement with every new foundational model.

And on top of that, now you're differentiating with features because models in a way are commoditized to an extent. Everyone has access to it over an API. So then how do you leverage that model becomes a differentiator? So it goes back to regular product building fundamentals. How easy it is to use? Are you providing the right things at the right time to the user when they need it. So a lot of the improvements that we see today, apart from the models, what else do you think plays a role right now? And how do you envision this would be better?

I'm really glad you asked that. I almost could have prompted you to ask this. So I think that the piece that I'm most interested in, because obviously the models are going to continue getting better and better, and the tools are all going to continue sort of like copying each other and converging, as tools will do, right? So the piece that I think that's really critical that's missing that everybody needs really desperately and they're asking us for,

and we're making good progress, is the platform. you knew i was going to be the platforms at some point and so to me the real value in kodi is that we are building it as a platform i mean the internal the guts are very platformy so we've got a command line tool you can use to integrate with your workflows and we've got sort of like a set of curated platform APIs for connecting clients, but also for connecting your own context. We've got an open context protocol and like, there's so many surfaces, right? Because you've got to bring in all of your personal context, your wikis, your private code, whatever. You may have your own personal model. So K coding also becomes a clearinghouse

for you to mix and match what you want. Because we've got users and customers that really love one particular model or another for whatever reason. And a lot of coding assistants, they're pretty much a bundled deal. You don't have to think, and it's great. They give you a model, and that's the model you get.

And for people who just want to just start going, they're great. They're fine. But if you want a little bit of control over it, if you're the kind of person that builds your own PC, you want to build your own tools, your own personal. If you're the kind of person that has a bin directory with scripts in it, and you like to tinker with your tools to make yourself even faster, then I think you're really going to like Kodi

because we're building it for that kind of user. It's actually a necessity because enterprises have such whack requirements, right? So you kind of have to build it as a platform. Sanyam Bhutaniya- What are some of the use cases do you see that could be built on top of the platform that are probably not as obvious to many people? Well, I mean, that's a good question. I mean, like, there's some of them are kind of feel like science fiction, but I'm sure we're gonna we're gonna get there. Oh, sure. All of this was science fiction three years

ago. Right? Yeah, right. I think the Holy Grail is batch, right? You got to be able to basically like run the L and you got to be able to basically run the LLM. You've got to be able to drag the LLM's nose through all of your code. Yes. Right? So what we do now is we do it at ingress and egress. Okay, a PR just came in, have the LLM look at it and see if you can do the linting or whatever else.

And that's great. And you've stopped the bleeding, basically, like codes that are in your code base. But what about all that existing code, right? That's the holy grail. You know, our product actually has a really cool batch engine. It's similar to one that Google built internally

that Google really should have open sourced. It's called Rosie, but they didn't because they never do. And it's pretty cool to be able to like, like know everybody that you're about to break right at least in the open source world and be able to like even go and refactor them and send them a pull request saying i'm about to break you but if you accept this pr you'll see that it fixes you that's nuts okay being able to do that yeah now imagine being

able to do that with cody imagine being able to do that with AI, right? Because now, I mean, it can do, I mean, these things can do remarkable things. You can basically give it high-level instructions for a file, and it'll go ahead and port it or whatever it is that you need to do. So, yeah, that's the Holy Grail. You know, when do we get there? I don't know. I don't know. There's so much to focus. It's like I told you.

You've got to pick where to spend, where to turn that dial. We've got people chasing the AI side because that's important we've got people in the context we've got people making the clients really really you know slick and good we've got people you know working on the uh the mixing and matching of you know services and and and models in the back end we're spread thin everybody's spread thin and we're just we just hope that we're making choices that are going to make the most people the happiest. That's where it is right now. So one of the aspects you mentioned,

like the holy grail of being the batch, like today, if you think about LLMs or many of these coding assistants, they're essentially assistants. And I think you said it well, they are safer in the hands of a senior engineer. of these coding assistants they're essentially assistants and i think you said it well they're safer in the hands of a senior engineer i was probably going to go where you just already prompted it is like you don't know when it comes but the thing that i was thinking about is it

would be really nice to have not a coding assistant but a coding partner of sorts where it works on your instructions, but you trust its output as if you were an office unit engineer. So the part about that you said on the legacy code or the existing code base, and I think that's where a lot of value lies too. And I would say majority of the organizations struggle there. We know there's a lot of code which is dead in many cases no one goes and cleans it up sometimes you want to do a big refactor and just updating libraries of all the clients is extremely painful in a large code base being able to do such operations which require

chain of thought breaking tasks down into pieces but then executing over hours or days sometimes is this a direction where coding assistants would eventually go in your perspective but then executing over hours or days sometimes. Is this a direction where coding assistants would eventually go in your perspective? I mean, dude, your whole question was tainted with the T word. You said trust. There's no trust, right? And so when that happens,

because it will happen, somebody's going to have to go through every single step of it and validate it. Right. We're going to be very was it linen trusted verify? Or is it crucial? I don't know. That's where we're at. Right? Don't trust and verify. Reagan Reagan, trust by verify. So, so yeah, that verification, that validation, you know, it happens in the small when you're doing chop and you validate it yourself. Your IDE can provide you a very quick feedback loop because you dump it in and it goes that

doesn't compile or that API doesn't exist and you're like, no, right? But there's larger feedback loops, larger and larger ones that go all the way out to production and come all the way and via some logs or whatever. And those things they're going to get looked at by models and models that are trying to do those kinds of things, workflows. And it's going to be just like chop is right now, which I wish more of you were doing it and I'm doing some videos, but basically it is an assistant and you're still doing the work and you need to make sure

you're, you're the, like if you build guitars or cars or whatever in your garage, right? You can bring in assistants and you can train them up, but ultimately it's your guitar, it's your car, right? You got to make sure that the right things happen and assistants drop stuff. Right? So I would never get out of that mindset i know you want a partner i know you want someone you can trust you're not going to get it okay settle for an assistant for now helps me keep my job in a way so it is job security absolutely and you know what i mean

despite the doom and gloom you know nature of the obviously i had a provocative title in my blog but i mean like you can do that validation yourself as a junior engineer right because it'll spit out a bunch of stuff and it seems plausible but you can use other tools like other lms or compilers or whatever to go in and do the valid tests to do the validation that you need to see yeah it did work now you're going to miss things you know probably like you know it's inefficient or whatever, something that it did that it didn't need to, right, if you're a junior engineer. But if it meets the basic spec, right, so I mean, I don't think it's all doom and gloom for junior engineers at all. They're just going to have to get really, really good at validating.

And you're saying eventually you would have models which would also help you validate. So keep feeding. Yeah, right. a mix you pipe from one end to the other. I mean, as soon as you start doing a set of validations, that's the same, you know, like every time, then you need to train them. You need to embed that whole thing, right? Train a model that knows how to do that. So CHOP is definitely a useful skill

and not everyone is doing it. And even the people who are, some are better than others. Do you evaluate for this in interviews anymore or at all? Not anymore? Oh, my goodness. No, we haven't started asking it as an interview. It's too new, right? That's interesting.

And, you know, I mean, fundamentally, we would still be asking people. That would be like asking people, can you type? Like, we might ask people some prompt engineering questions, like, just to see if they have any common sense about it, I guess. But mostly, yeah, you're always still focused on fundamentals, right? Computer science, timeless stuff, ideally, in interviews. And in terms of hiring junior developers versus senior developers, this is at least a shift that I see at LinkedIn. And I think it's probably true across the industry right now, where overall there are fewer positions open.

And the ones that are, are for experienced engineers. And when I say experienced, I like how you described it, where a person who has an idea of what needs to be done has a framework of how they would do it, and they can control the LLM in this case. Do you see that affecting productivity of organizations that you speak with? You mean the shift towards more senior? Yeah, I mean, sure. Yeah, they're getting more productive. But, you know, that's short-term gains, right? What long-term consequences are we going to reap from not hiring enough junior engineers? We become dinosaurs.

Companies tend to get a little bit short-sighted. I mean, like, have you ever been in a company where everybody there was, you know, like retirement age? I've, you know, been around companies like that. You know, it happens. And, and it's really, really difficult. It's at that point, you've got Japan's problem, right? You know, it's, at some point, it's, it's kind of not reversible. You got to have, you got to have young people entering into the workforce and learning this stuff. So continue doubling down on that and hiring hiring junior people and then just i don't know i think i think we're just gonna have to figure out um like a new workflow like there's been a lot of people suggesting that maybe the junior folks can

crank out a lot of the work and senior folks can kind of like more do that second level of betting but uh i haven't seen anybody fall into that yet. Everybody's working as hard as they can and really planning for the future. It's just such a big race. But yeah, it's a huge problem. And there is this thing called Devin. There's a company called Cognition.ai. I'm curious.

If you have seen it, what's your take on it? I didn't see the demo. I heard people talking about it, but I don't think it comes down to the T word again, trust. I ain't going to trust agents until they're trustworthy. Not before, not a jot before then, which means I'm not going to be an early adopter of them unless they have a mode that allows me to use the agent as an assistant or it does part of the job and then i tell it where it went wrong with chat and then it can continues the job if they can do that like human in the loop you know driving it along and steering it then i think they'll land a lot faster and people will trust them a lot faster but my my my suspicion is that people are they're chasing investor dollars and so they're trying to do something that's hitting home runs, and they're not going to hit it. Not in the next year, 18

months, I don't think. But I've been wrong before. So similar to coding assistance, there is another aspect that engineers care about, which is understanding how the existing code works. And you mentioned batch in case of Kodi and Sourcegraph. Is it a direction which you're pursuing where one could ask questions of the code to understand how it works? And I recently saw a demo, I forget the name, but they had this thing called, which would basically generate a wiki out of an open source code base. So I could just go go and say how does react work or how does kubernetes work or aspects of it yeah yeah this is it this is an idea this is the idea i mentioned earlier in the talk that one of our customers pitched to us and they actually got it from google via another customer of google so this is spreading

what you're talking about is i've look it was hard for me to get my head around it, but I view it as a new kind of source code index that's complementary to a search index or a graph, right, that a compiler would build, or embeddings, any sort of index, right? They all have different, you know, strengths. There is a new kind of index emerging, which is generated by LLMs and it's a semantic index. And I think they're going to have incredible, they have incredible potential. I know some people who are building them now. We will be looking at them for sure. I'm excited about that direction, but that is, that is smoking hot new. That'd be amazing by the way. Like wikis and dogs get out of date. This would be pretty neat.

This customer spent a whole afternoon educating me about it. Why? Because they didn't want to build it, right? They wanted me to build it, which kind of makes sense, right? Because it's kind of rad. So, yeah, I'm getting chills. This is a new direction, right? It's so weird.

You're basically building this document store, this semantic index that doesn't have to be human readable. It just has all the semantics that are unpackable by LLMs. And then you can have business owners who can't look at the code necessarily. They can be going into legacy code bases of 20 million lines of COBOL or whatever, your company doesn't even understand anymore,

but you're dependent on it. And they can go in and actually start making like informed, not just questions, but potentially even start, the company could start doing refactorings now that are like tractable when they weren't tractable before. And trust me, I know a lot of you out there that have intractable migration problems, right? Trying to get your monolith split up or whatever. Like this is going to be a game changer for that. A teeny tiny benefit of this would be developers getting less pings from their managers about how does this thing work?

They could just go ask. It is one of many benefits that this structure has. And I'm just, it was weird. It's like a discovery. You start realizing this is an inevitability. Like people are all going to figure this out, you know, is use the LLM to explain all the code in great detail. And then now you know how it works in ways that you just couldn't before.

Now, I mean, like how to build that index is still a wide open question. What's the format of the index? What's the granularity? Do you run it at the function level, the file level, the module level, the graph level? We'll see, right? But it's really cool that a lot of people are innovating on this right now. And a lot of developments we see right now are on the left side of the developer's cycle, in a way, which is code assistance to help them develop code or the summarization in a way or the code understanding semantics of the system and then helping you understand how

it works. There is this entire world of things on the right side, which is when system is running in production. And you can't reason about a system, especially in the service-oriented architecture, where you have like 2,000 things talking to each other at any time with configurations and somewhere else, secrets somewhere else, and they are just connected via this network. Do you see any movements on that side, which would help one understand the production environment better? I mean, I'm sure there's like startups and stuff doing that, but we view that as a subset of the context problem. Companies came to us and immediately they were like,

oh, Cody can index all our stuff and then we can ask questions about it. Then can we throw our wiki in there? And we're like, yeah. And they're like, well, can we throw our issue tracker in there? And we're like, yeah. And all of a sudden they want everything in there, the logs and whatever. And so that's why I said we're building a platform with an open context that allows you very simply to basically allow other contexts to get put into your workflow.

It doesn't have to be a query. It can be a CICD workflow that you're building, right? Those platforms are huge. Okay, I completely missed then what you meant by platform because this is fascinating. So essentially, Kodi's, while the way you interact with it is through the coding assistant interface or the chat interface in your IDE,

but the context it has is beyond just the code in the repo. So you're saying... It is, it is. It's negotiating. You can mention URLs and stuff, so you can slurp down now in Kodi. It's negotiating. You can mention URLs and stuff. So you can slurp down now in Kodi. It's actually quite, it's funny. It's not the way we thought it was going to go. And our AI people are a little bit indignant, just a little bit, right? Because they're like, well, technically

you shouldn't have to figure this out. The model shouldn't be able to figure this out. And I'm like, well, there's your operative word word should. All right. So why don't we just go ahead and work with reality, which is that we're going to work. We need control over the context. We need to be able to sling it around like gun slingers. I mean, slice it, dice it, prune it, go back and rewrite it, edit history. Your conversation becomes the dynamic context, right? That's going to help you solve your problem, whatever it is. And so, yeah, you need full editing ability to go back and rewrite that thing, you know? And we decided to go down that path with our AI people kind of running after us to try to, you know, model it, right? And that's given us just tremendous flexibility. That's what I mean by platform.

We can integrate on many different surfaces. We can integrate on the model, the backend. We can integrate your context. We can integrate at the repo level or with your particular tools. That's a lot of work. It's like three times much work to build it that way. Then people can build stuff like what we just talked about. That magical document store, you know, and,

and integrate it into your tools. And this is why I'm telling people that the, the growth is exponential, that it's only been what, 18 months since GPT three, five came out, you know, or whatever,

three GPT four. And, and, and that's kind of garbage compared to what we have today. The next 18 months, isn't going to be, you know, like, as much as happened before, it's going to be significantly more than happened in the last 18 months, because we're growing exponentially. So it's a little bit dizzying,

actually, man, is it exciting? Now? Do you see why I'm saying this? Seriously, I think everybody should they should they should be jumping on this so you mentioned earlier that while building kodi and also using kodi it has surprised you in ways where it acted in a way you didn't anticipate it to act that way uh are there any specific instances that you can recall where it surprised you yeah i mean just sometimes it just has no, you feel like it has no business knowing what it figured out, right? And this happens a lot, especially because Kodi does a lot of

magical context fetching for you. You can see it. You can open it up and look at which ones it picked, right? It'll rank them. It pulls some out of the embed. It'll embed your query and see if it can come up with some relevant stuff that way. And it'll do a search and it'll check our graph and so on.

And sometimes that context is completely irrelevant we need to work on it and sometimes it's so relevant that even with like it's incomplete and you look at it you're like actually even like a senior engineer probably shouldn't have been able to figure that out and yet here the lm generated the right code maybe it's just really good at guessing, right? But I mean, in the end, if it's doing the right thing, I'm still regularly blown away by these things. So I don't really have products, so this question might be a dumb question, but I was just trying to visualize in my head in terms of how this thing functions. Visualizing things makes it easier for me to understand stuff. So you have this model in one place,

which is the core LLM, call it GPT-4 or Cloud 3.5 Sonar, something else. You have Kodi in the front and then context you said is where the magic happens or that's the differentiator. Yeah.

Kodi can pull context from different places. Now, context here, tell us a little more about what that looks like. So I'm just imagining in one case you have all of your source code, in one case you have your wikis. And this is something I see when I just search on Sourcegraph.

By the way, Sourcegraph search is outstanding. That's my go-to place to search code i've recommended to most of my teammates i work with um anyway so in that case it says something like search based it'll actually show that tag but in terms of fetching this context can you elaborate a little more on what that looks like? Yeah. I mean, it's not a one-size-fits-all problem, right? It's domain-specific.

So if you're generating unit tests, your context is going to be your company's testing style guide, whatever framework classes you're using, maybe some examples of tests that are particularly representative of how you want them to be written in your organization. So we actually allow for things like this by allowing you to put in pinned context and, you know, having administrators actually point in context that everybody's going to be using all the time for specific use cases. You can do custom prompts basically and build up a prompt library so that, because the prompt can help bring in some of the context as well or interpret it.

And then it's gonna be a completely different set of context for the use case that you brought up in your earlier question about how do you debug in production, right? And the answer is the LLM can use tools and it can read and interpret the output of tools. And so, our context there is, can we get a, what information can we get out of prod

that we can get in front of the LLM? And since they're multimodal, maybe that's as desperate as doing a screenshot, but hey, right, that may work too. So context can be anything. Context is context. It's literally like, it is defining the problem. It's the most fundamental thing. Sanyam Bhutani. So in the rag part of this equation, this is

the retrieval part. And how much of the classical search techniques apply to this part? Or is this part also very different nowadays? It's very similar, actually, like if you think about Google Search, right, it's not one back end, it's not one size fits all, just like this AI domain, right? If you're searching for baseball cards, there's going to be a very specific set of, you know, things in your query that are very different from if you're searching for, you know, a pet store. or even thousands of models that all report back on what they think the relevant answer, context for your query is, search results, say. And then somebody has to go and arbitrate, right, those answers and multiplex them.

And that's where some of our key differentiation is. We're not just a context fetching engine. We're a context assembly engine. And there's a lot of, there's an art to that as well, right? Cause it's a bin packing problem. So for someone who has some knowledge or experience with information retrieval systems or search in general, I can see them transitioning into the space

relatively easily. What are some of the other transferable skills in terms of software engineering to just building AI products? Typing. Right? Because if you're doing chat-oriented programming, that means you've got to chat. And so if you're really a slow typist, you're going to be the bottleneck. Although it would be great if we could... I think, I think this is, it's finally time for, you know, speech to text to, to really shine.

We need to get it wired up so that you can talk and type and whatever at the same time. That was slightly tongue in cheek. I get it, but you know but yeah, in terms of software engineering, I would say, look, the skills are the same, but for my entire career, the low level skills have been becoming obsolete one after another as we level up. Okay. So when I started, you know, in college, like we were literally like using put pixel, like to put a colored pixel on the screen and we would use our own Z buffering and whatever to build 3d graphics and by the time i graduated from college it had already gotten to animation courses it was accelerating that fast right and at some point we didn't need to know how to render

polygons and do brezhenham's algorithm anymore right and because it's down in the hardware and this steady march of pushing software down into the hardware layer over and over again has meant that we've been focusing on higher and higher and higher level things there are still computer steady march of pushing software down into the hardware layer over and over again has meant that we've been focusing on higher and higher and higher level things. There are still computer science-y things, and you still have to understand algorithms and data structures and big O complexity and all that. You have to, right? But the specifics of technologies that are staples, I don't know, do you need to know relational databases anymore? Maybe not for long.

If LLMs know everything about relational databases and all you need to do is tell them what you need, right? You're gonna need somebody to go in and fix it, one person on staff. Again, this is the problem, right? But yeah. So we're coming towards the end of our conversation. And so in one of your youtube posts uh you mentioned that having something to truly push up a mountain is fulfilling we

like struggle and struggle is necessary for life you've had a very successful career over this period what have been some of the struggles that you are the proud, most proud of? Am I the most proud of? Or most have been most fulfilling in a way? Yeah, I mean, like, I guess one that I would love to do is almost like a PSA is your mental health is really, really important resource that I think that maybe like, so I'm Gen X, and we don't talk about that stuff ever. Okay, it's dilemma, you know, head on. Very, very difficult problem. You know, my mental health really, really took a hit, you know, and I learned how important it is.

And that was a huge struggle, a huge struggle, right, to get better, you know what I mean? Like, and get back to my jolly old self, you know, it was not an easy thing. And so I just want people to, especially the older programmers, you know, to recognize that, yeah, stuff can get so hard that you need to go get help. So there's a struggle I'm proud of telling you about. Does that count? That totally counts. It totally counts. Steve, this has been an amazing conversation. We highly encourage people to check out all of your blog posts we'll link to them in the show notes and we learned a lot through this conversation and i'm sure all of our listeners

will too not only learn they'll find it entertaining so thank you so much again for joining the show well thank you so much for having me thank you steve awesome all right guys Awesome. All right, guys. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com. We would love to hear from you. Until next time, take care.