Signals and Threads - Learning Goals and the Goals of Learning: Teaching in the Age of AI with Aaron Bauer

Episode Date: September 23, 2026

Aaron Bauer is a software engineer and one of Jane Street's few developer educators—a role that splits his time between writing code and teaching other people how to write it. Before joining the firm, he taught computer science at Carleton College, including four straight terms online during the pandemic. In this episode, Ron and Aaron discuss what it takes to teach engineering inside a company with its own language, its own version control, and its own editors, and what changes when an LLM can do the exercise for you. Along the way, they consider the underrated power of a live lecture; why the text editor is still the software engineer's home base; using editor telemetry to find out how AI is actually changing developer workflows; and how Jane Street rebuilt its intern curriculum around testing, design, and code review now that producing the code is the easy part.You can find the transcript for this episode on our website.Some links to topics that came up in the discussion:Foldit and AlphaFoldCarleton College Computer ScienceFlipped classroom, active learning, mastery learning, and cognitive load"The Pen Is Mightier Than the Keyboard" — Mueller & Oppenheimer on handwritten vs. typed notesMerlin and ocaml-lspecaml — writing Emacs extensions in OCamlBonsai — Jane Street's OCaml web UI libraryOxCamlClaude Code output styles — the "learning" mode Aaron describes (note: Anthropic has since moved this to a plugin)Jane Street's tech internship

All right. Well, it's my great pleasure to have Aaron Bauer with me. Aaron is a software engineer and one of Jane Street's proud but few developer educators. We'll talk a little bit more about what that means in a bit. But thanks for joining me. I'm very excited to be here. So I like to always hear a little bit about how people started out. You've done a lot of work, which is kind of on the overlap of both software engineering and computer science and teaching. And I'm kind of curious, like, how did you get there? Sure. So my origin story, you might say, is I was sort of fascinated with computers from a child computer programming, was always felt kind of intimidating. And then I did this summer program before my senior year of high school, creatively named the summer science program, where went to New Mexico for six weeks and they taught us a bunch of math, astronomy, physics, and some Python. We took telescope observations of an asteroid. And then, wrote some Python code to take those measurements and compute an orbit for the asteroid. So really fun and interesting program, but I came out of it or went into it being like, all these things seem interesting. I came out of being like, wow, the programming was just way more fun and like really my thing. So at that time, my small town high school didn't offer AP computer science, but found a retired teacher on a beach in North Carolina who could take it from and then sort of

How do you find the retired teacher on the beach? So it's North Carolina had a like online course program that you didn't have to live in North Carolina to use. And it just turned out the guy teaching this course was not currently working in a school, but instead was on a beach. Oh, amazing. And then the teaching side comes in that before the summer before I go into grad school, I go back to the summer science program. I said TA. And that's sort of my first exposure to writing an exercise for students to do and giving a little lecture on binary search trees. And I was hooked and sort of from then knew that I wanted to do something with teaching.

It's funny, by the way, just like the kind of science and physics hook in programming, that's actually like a deep thing in the history of computer science, right? A lot of the early programming and programming languages and work was all about like, you know, high-end physics simulations. I always every now and then when I like run into like someone who's like you know gathering telescope data just like the hardcore like computer science of like you know weird FPGA exploits and stuff that people are doing there I always find very impressed there's just like a lot of interesting pushing of the state of the art and also a lot of messy not understanding a lot about computer science like comes out of that like scientific computing world yes something I didn't undergrad was work with. with a Fortran simulation of impacts, like kind of asteroids or comets hitting the surface. And this came out of code that was used to simulate atomic explosions in the early days competing because the physics of those two situations

are not that different, it turns out. Oh, wow. Yeah, and there's a bunch of like super cool stuff in the early Fortran work. Like, you know, we care a lot about automatic differentiation these days, right? Because we write all these machine, learning models and you need graded a descent, and for that, it's nice to have derivatives.

But automatic differentiation goes back to, like, the early Fortran days, where there was a bunch of Fortran compilers that could do this. It was messy. They could do first derivatives, but not second derivatives. Like, you would take the derivative and you'd get one program, you'd get another program, and if you tried to take the derivative of that second program, God help you. Nothing good would would happen there. Yeah, it's very much computer science is a means to an end. It's the I'm not programming because I love programming. I'm using it just as a tool because it beats a slide rule, I suppose. Okay, but you loved programming.

Yes, I certainly did. And then you went to grad school. Like, why did you go to grad school and what did you study there? So I didn't know I was going to grad school until I took a computer graphics course my last year of undergrad and got incredibly excited about rendering. And it was like, I would like to do, this is something I could do for the rest of my life. what's a way that you get to do one thing for the rest of your life you go get a PhD in that thing

so I applied to some programs, got into them and then as I was talking to people realized that people weren't so much doing research and rendering anymore a lot of the field had sort of moved on it was very mature the remaining work was largely an industry

so I kind of inertia carried me into grad school anyway and kind of did some game design research some computer science education research and then ended up writing a dissertation on what you might call human computer problem solving, studying how players of the scientific discovery game Foldit solve problems and collaborated and sort of worked with the automated tools available there.

Oh, right. Foldit was the early, like, here's how we're going to solve protein folding by having people go off and try and solve protein folding as a game. Exactly. It was humans have some like big picture geometrical. side of, oh, I sort of know how these amino acids tends to fold up and, oh, what if we took this part and flipped it over here? And at that time, computers were really good at the, I'm just going to wiggle the protein a lot to find the lowest energy kind of local optima. And those two things together were able to resolve proteins that had gone unresolved in the lab for many years. People playing this game often had no biochemistry background. So it was very successful and eventually moved on to

people actually designing novel proteins in the game. But I think Alpha Fold and other things have sort of eaten their lunch. Do you know if some of the data that went into AlphaFold and the like came out of Folded? I assume AlphaFold was trained on all known structures in which case, absolutely. So there's a little bit of an educational point in that, right? You're talking about like how do you get a bunch of people who do not know a domain to learn it? Like how does that process work and the kind of gamifying the education process essentially? Yeah, a lot of my work in grad school was, had this sort of cognitive psychology element to it of how, of trying to understand how people think or learn and then how they apply those skills.

Okay. And then where'd you go from grad school? So I knew that I wanted to go somewhere where teaching was my primary responsibility. And so that's how I ended up at a small liberal arts college, Carlton College in Minnesota. I went there and taught two quarters in person, and then it was spring 2020, and then I taught four quarters online. Oofus. So this is like a hard introduction to this teaching. Yes, it was challenging for me, very challenging for the students, but also like involved a lot of trying new things and experimenting and different formats. which I think is an important part of succeeding in education.

So how well do you feel like Carlton did at adapting to these new online constraints? And how well, you just even forget Carlton, I guess, like your own classes. Like how successful do you feel like you were actually able to be at this? I think I eventually got to the point where I was fairly successful. I started out taking the model of that online AP computer science course that I had taken in high school, which was entirely text-based. There were no video lectures, you just read some stuff and you did some exercises. So I started out on that in part due to guidance of like people were at home, not everyone had good Internet connections. I want to make this very accessible.

But just immediately got feedback that this really does not work for all. It worked great for me. It does not work for all students. accessible in some way, deeply inaccessible in others. Yes. So I experimented with sort of flipped classroom approaches with synchronous classes, splitting out into kind of small working groups. So there's a bit of jargon that I bet not everyone understands.

Can you say a little bit more about what a flipped classroom is? Yeah. So a flipped classroom is instead of the class time being when the professor would deliver a lecture, the professor prepares a lecture, a recorded video ahead of time. Students watch that before class, and then you use class time to do exercises or discussion, something more interactive. Right, and this is like not a new idea that came up in COVID. People have been talking about flipped classrooms for a long time. Yes, this is a longstanding idea in education that has been implemented in different ways,

but it gained a sudden relevance and I think widespread adoption when classes had to move online. Right. And I guess part of the point of the flipped classroom is the in-person time, right? Being able to leverage the in-person time to do things that are better than just giving a lecture. Yeah. So you take the time where students are, say, passively kind of receiving some content or seeing worked examples and say, that does not need to be done in person. and we're going to leverage the instructor's ability to tutor someone one-on-one or to lead a discussion, things that can only happen when you're, when you have time together in the classroom.

Out of my, where I sort of personally ended up was just trying to make, not doing the flip classroom, but trying to make the time we're together more of a mix of interactive things and some instruction. Yeah, like I'm curious how much you buy the kind of argument. behind the flipped classroom. Because, like, my instinct about this has always been that people underrate the power of lectures, that, like, lectures are kind of a form of performance that is actually, like, really compelling and, like, the really being in person with a person doing the lecture, even though in some sense it seems dumb and how could you possibly need it.

And surely, you should just, like, record the best lecture that has been given on the topic and watch it, that, like, actually with real human beings and real environments, this just kind of isn't true and there's just something more compelling about being there in person. Like, it's not like I think the flipped classroom idea is totally wrong, but I do feel like it under, like the whole thing of like, oh, there's just like the passive lecture is just like missing something about the human psychology there. Yes, I would tend to say that lines up with my experience, that there is something about like having a person, as you say, perform live in front of you, that I think has a sort of focusing of. effect of your attention. And because it is live, I think you're more likely to try and take notes, which I think is good for people understanding, which is not to say you couldn't do that when you're watching a video in your dorm room, but I think it's less likely. I think you and also, I think

a solid hour of someone just talking at you, that starts to lose the engagement and become counterproductive. So another bit of jargon, active learning is very important for these sort of in-person and things where throughout the lecture, I want to stop and pose a question and have people talk to their neighbors or have people stop and do a little problem on their own to check their understanding, things like this. By the way, note taking is like another great example

of like a super weird and also strangely effective learning technique where like just being in the classroom and taking notes. And then if you walk it, you just like light them on fire. You're like, you never look at them again. Like, there's actually, as I understand it, pretty good evidence that like this actually helps retention in a serious way, even though the notes themselves are like totally useless. Having to process things and then sort of distill them down to something that you write down and then I agree the evidence seems to be that the act of writing it down helps set it in your memory in some qualitatively different way than just hearing it, even if you understood it. Right. And apparently writing is better than typing for this.

Maybe I'm getting this. This is what I thought was true about the research. Although it's not like I remember the study or anything. That sounds familiar to me. I would say my own experience in the classroom is it's maybe less about the typing per se than someone having a screen with all the information and distraction that humanity has created. Sitting between them and the presenter tends to undermine once. ability to pay attention. That seems very believable. This is also why I have generally tried to avoid teaching in computer labs, which is something that people often do in CS courses. But I

just can tell as soon as there's a screen in front of people, like 50% of people, even if they want to pay attention. They just, they can't. Yeah, exactly, this echoes a little bit my own experience, just like teaching my kids how to program. Like, one of the things I did a lot of was have them think about programs on paper. Right? Just like, in fact, one of my, one of, one of, uh, one of things I conned my kids into thinking was like an acceptable distraction, like when waiting for food to arrive at a restaurant was like I would write a little scheme program on a piece of paper with like obfuscated names

and then they would have the puzzle trying to figure out like, what did flub do? Somehow I convinced this with an acceptable way to spend time. But I do think like the thing of like talking to them about what a program means and starting out by being like, you know, I'm going to write down on the board or on a piece of paper, like, here's an expression, like, what does this evaluate to? I think the moment you gave them, like, a computer to type it in and try to evaluate it, like, they were playing the game of getting the computer to do thing and not thinking about the question of, like, what is actually happening? And I think that was just like, there were ideas you could get across much more easily when you,

like, got the computer out of the way. Yeah, and there is definitely a thing that people just learning programming will tend to do, which is this is kind of arcane instructions for the machine. I will perturb it arbitrarily until it does what I want. And this is very easy to do when it's on a screen and you can run it. But if I ask you to write out a program on paper and then decide when you think it's correct, it's more likely to force you to think about what it's actually doing. For sure. Okay. So you were at Carlton during this like very, very, complicated and disruptive COVID pandemic. You did a bunch of experiments and learned things.

What happened when you got back to the classroom and people could be there? Like, are there things that you took away from that experience that you kept? Like, what was persistent that you learned there? Yeah, I think one of the persistent things was I just found it very useful to have a lecture recording. I think this varies by institution. I've heard from other instructors that if they record the lecture, No one shows up to class because they can watch it. This wasn't the case that Carlton students wanted to come to class,

and I tried to make being in class with this sort of interactive stuff actually useful. But I... It might be institution-dependent. It also might be lecturer-dependent. This is fair. This is fair. The classrooms of Carlton were not equipped generally with video equipment. So I was carrying around a webcam on a tripod to each of my classes

and plugging into my laptop and recording the lectures and posting them online. And because I would get questions throughout this, like, oh, I have this track meet or this obligation. I have to miss this class. What should I do? And if I have a recording, it's like, well, it's there. You can watch it. This is, or if someone asks a question, I can send them a link with the timestamp or the lecture.

Here's where I explained this thing. So that carried over. I think I think I may have arrived at this even without COVID lockdown. But during that, I experimented a lot with sort of automated feedback in various ways. So weekly quizzes that were multiple choice and auto-graded, or different sorts of like fill-in-the-blank kind of things, different auto-grading software where students could submit homework and get immediate feedback on whether it passed tests, that sort of thing, and carried that forward into my courses. and found it like because getting feedback on your work is an essential part of learning.

And I was always dissatisfied with, okay, you do the homework, you turn it in a week or two later, an instructor, TA has graded it and you give back some feedback. But at that point, you've moved on and you may not even read the feedback you get. Also, that incorporated a technique that is sometimes called mastery learning. I didn't do it in its fullest extent. but the idea is you have to demonstrate sort of mastery or competence at a thing before you can move on where you can attempt it any number of times and you keep getting feedback and you sort of reattempt until you actually have kind of gotten it right. And so some of this automated

assessment, let me take more of that approach. Interesting. My first instinct about the mastery thing is I would have done very poorly without in when I took math classes in college because I felt like the experience of taking high-end math classes was like, I was like holding on barely keeping up, trying to understand what happened. And then six months later, someone else was taking the class and asked me about it, be like, oh, yeah, I know how that works.

They're like, just somehow, like, I couldn't quite get my arms around at the time, but somehow just like the passage of time after. And then you look back, you're like, oh, yeah, I know how that fits together. But I'm probably under-imagining, like, probably I could have, like, demonstrated some local mastery.

this is like, I think the demonstration of mastery and also feeling really confused or like consistent experiences. Yeah, I think this is often applied in a sort of maximal extent with much younger students, like in elementary school, math class, you have to like, it's sort of self-paced in a way where the class might be at different points through the material and you have to like master, addition or kind of long, the long division algorithm before you move on to the next unit. I think as you get higher level, you start needing to apply this in more targeted ways and that the sort of the flexibility is not there to the same extent in a, in say, a college course.

Did you actually do the thing of like having people stay on older topics until they demonstrated mastery? It was more like, there's a thing that you should do, which is like keep on iterating until you pass. But that happens at a fixed time and you kind of have to like do it at a particular point in time in the semester. Yeah, it would not have worked for people to like. progress through different rates, not least to which because students can procrastinate. This is the thing people do. And I think it would have been a disservice to let people pile up all the work until the very end. So you do have to give people deadlines throughout for their own, for their own good and force people to move on. So it was a model of like, you have this short quiz to do this week.

you can attempt it any number of times, but you are just going to, like, get sort of the best result of the times that you take it. So, and the questions are set up in a way such that it's pretty annoying to brute force, like the order of them keeps changing, the order of the answers keeps changing. So it's just going to be faster if you actually, like, try and understand all of it and then get 100% that way. Have you thought much about whether in the modern world some of this feedback, the kind of instant feedback that you think would be more useful for students, can be generated by LLMs? Yes, I think there's quite a lot of potential for that, particularly because when you have,

say, a more structured thing to sort of narrow the focus of the LLM. So, for example, feedback on code style as opposed to an essay. I think I see a lot more promise in in addition to like some automated grading of test cases and maybe some linter to give students feedback of like you need to follow these rules, some additional kind of code style pass I can see being very useful. That being said, in my experience, there's also plenty of false positives of LLMs giving feedback that is misleading. And so I think at the very least you would want to frame it carefully to students, at least, or find ways to really reduce the rate at which it's of leading them astray. Sure.

Somehow you ended up here. How did that happen? Right. So I was living and teaching in Minnesota, and my brother and his wife had moved to Brooklyn. She's from Bay Ridge, and I decided that I wanted to live near them, be. my brother and I are close. So I decided to move to Brooklyn and was not going to be able to keep teaching in Minnesota

while living in Brooklyn. And it was very fortunate that at that exact time, Jane Street was trying to hire for this weird educator programmer combo. I had just assumed, like, there were some things I liked about academia, but I was ready to try something new and didn't really want to go through the academic job market.

So I figured I would. would go be a software engineer somewhere. But it's been amazing to continue doing education stuff in addition to doing some programming. Yeah, and it's maybe worth saying a little bit about, like, the idea that we had in creating this job, right? We really wanted, I mean, to take a step back. Teaching has been an important part of what we do,

at Change Street for a very long time. In part, this is because we do some weird stuff that we've not done so much in the outside world. Some of that is just subject matter. stuff of like nobody kind of comes in knowing much about trading. And so there's a big effort in teaching people to understand how to think about trading. And then also our tech stack is in a bunch of ways weird. And so there's a lot of things we want to educate people about to understand the specifics of the programming language and the environment and the tools, a lot of which is very

different from what you see on the outside. And so we've spent a long time, a lot, we've spent a lot of energy on education for a long time now. And at some point we're like, you know, it's great to have people who are on the ground involved in things diving in and doing the educational work, but maybe we could get some people who have some expertise in the area and have actually done it for a material period of time and try and leverage that effectively. Well, let's just actually create a job that's just for this.

And it seemed clear to us that we wanted the people who were doing the teaching to also be people who were practitioners, right? Again, in part because there's lots of things that are unique about our environment. And if the people involved don't directly experience and work in the same ecosystem, they're just kind of not going to be effective or compelling teachers there. And so we design this as a kind of 50-50 split where the idea is like you're going to spend half your time programming and half your time focusing on educational work, some of which is going to be directly teaching classes or creating curriculum, and some of which will be helping other people

grow as teachers, right? Because we actually don't want there to be like a small number of developer educators who do all of the teaching work, but to, like, help grow the practice and make people better at it. I guess the other thing that I find interesting and a little frustrating about your description is that, like, this putting up this job at did not at all work in the way I wanted it to, which to say, I just sort of thought this was like an obviously awesome job, and we will advertise it, and, like, great candidates will fall from the sky because, like, it's a cool and unique job opportunity. And it turns out like if we had put this out like three months differently or something, we would have not gotten you at all. And in fact, in fact, actually it's an ongoing thing.

It's actually been pretty hard to like find the right people for this spot. Like it's like, I think it is less the case that people, in general, people don't fall from the sky. I think hiring people is really hard. Yes. I think that this job is, as you say, very cool, but also very unusual. And the kind of person that would, like, be looking for this job is also, therefore, kind of unusual. In particular, most people with a lot of teaching experience are in academia, and most people chose academia because they are passionate about what they get to do there and often are not looking to leave academia and work at a place in an industry. The job is also a kind of unusual marriage of skills. Like, if I'm teaching courses at a university, I'm not necessarily doing a lot of

day-to-day software engineering. And I may or may not even be interested in that being a significant fraction of what I do. Right. And we really want, at least in the ideal case, to find people who are, you know, really good educators who, like, have a love for the craft and have spent a lot of time doing it and really think hard about, you know, what kind of things what you do and want to experiment in the space and learn how to be really better. And are also just like fully credible Jane Street Software Engineers. And like that's like the intersection of two things where like that intersection is relatively small. It's maybe we're saying that like we're really excited with how having people in this role has turned out. There's some amount of like

taking some internal people and kind of converting them into this role, which is like another path. But we're also excited to get more people from the outside to do this. And I think There are just like more and more domains where we want it. Maybe it's worth like talking through like some of the other spots that we are trying to find developer educators for. Yeah. One thing I would add about the role itself is that I think one reason it has been successful despite Jane Street having been doing a lot of education before this role existed is having someone where that is a sort of first class focus of their job is sort of a step change and kind of. how they're going to be able to engage in the sustained attention on education. But yes, somehow there's just more and more stuff we need to teach.

And not just to, you know, new people who need to learn O'Camel, but it turns out we're changing the O'Camel language and everyone at the company, no matter how long they've been here, we used to learn about it. There are all these new AI tools. We want to teach everyone how to engage with these effects. And it's sort of the case that the more we have kind of put into this and expanded and refined it, the more places we go like, oh, we should make that better too. And so the kind of appetite for this kind of focused educational expertise seems to have only grown. Right. And so like on the O camel side, we had, we've posted this like ox camel educator spot for people who kind of have like

a strong PL background and are also excited about teaching. And we've actually had some good luck on the higher there. We're still looking for great people to do a kind of machine learning educator role. We're kind of focusing on like, you know, we're doing all this exciting new, you know, generation and training of exciting new ML models and people who understand that space and can teach more people to do it. It's a very valuable thing. And just kind of the general version of this role that hits all sorts of things.

Like you mentioned, you know, having to like, you know, teach everyone anew how to, how to use the exciting new corners of OCamil as they come into existence is one thing. Also, we now do a bunch of Python, right? And we kind of want everyone to know something about Python. In fact, one of the things I worry about is people getting captured by their tools. One of the things that's nice about having a small number of programming languages is like you can kind of go anywhere into any code base and do stuff. And once you have like two or three or four programming languages, there's a kind of natural like striation where people kind of get good at the one thing.

And I'd really love to be more in a world where there are no like, O'Kamel programmers and Python programmers. They're just like software engineers. And they know how to use many tools and can figure out what's the right tool to use, given the particular circumstance. But for some of that, we actually have to teach people. There are people who've been here for a long time who've like,

I mean, everyone knows some Python. But like, there are lots of people who aren't any good at it. And there's real things to learn to be good at it. And so that's like another corner where I think there's more teaching. Plus, we also invent our own programming languages in spots and teaching people about that. Yeah, and like a million other things. So, yeah, I do think that there's just more surface area to cover more technology for people to understand. And therefore, more opportunities to, like, create real value by teaching people stuff.

Yeah. And to say another thing about this sort of like general dev educator role, I think a model that I feel like we're heading toward and that sort of mimics some of the ways that, say, designers have been incorporated into sort of different teams and spots throughout the firm. I think that may be where we're headed for educators as well. So I mentioned AI tools. I think we would love to have an educator like sitting on our AI assistants team to focus on the education work that they're doing.

And I think more and more places will want sort of. an educator embedded there for all the sort of teaching materials that need to come out of that part of the world. Right. And I actually feel like there's two benefits there. One is the thing you're saying of like there's education that's, there are education needs that are focused on what's happening in a particular area. And then also, I think you want the people who are doing the education work to have a variety of different technical experiences. So there's sort of more of a body of experience. If you like poke them all in the same corner of the firm, they'll like kind of have all the same context, and that's just like less of a robust foundation.

Yeah. Let's pivot for a second to talk about like the technical work that you do. So like where in kind of Jane Street's technical organization are you and what kind of engineering problems do you think about there? Yeah. So I'm on the editors team within the tools and compilers group. So our team maintains the text editors that folks at Jane Street used to write code, VS code, Emacs, NeoVim, as well as some related.

like the code indexing service that we have, the Ocamel language server. And I have sort of worked across various things in the kind of dev tools space, started out working on an extension to our feature management software iron to make it much easier to split up work into different pieces of code review and organize it in a way that was sort of awkward and manual before, but giving people a nice tool. It would be like, take this and split it over here and arrange it in this nice chain of relationships

that will get reviewed separately. It's maybe worth saying when we see the word feature at Jane Street, we mean something that's like roughly analogous to like, sometimes analogous to commit and sometimes analogous to PR in the other world, in like the outside kind of world's debt tooling. I think the thing that really strikes me about all of the, the tooling we've built in this kind of code review and feature management area is just like, it's actually made a very different set of choices.

Like we've been off on a like, you know, 25-year alternative universe adventure of building development tools that just like look pretty different. And if you look at Iron, which is the tool that's responsible for code review and releasing features and all of that, it's like way better than everything in the outside world and also way worse, depending on exactly which they. you want to look at. And the thing you're talking about this, like, having nice ways of carving up and breaking down

a larger collection change into a sequence of smaller changes is like a good microcosm of, like, first of all, it has really good support for what we sometimes call stacked PRs in the outside world. And that's like built in and like a really nice foundational piece. And also

the process of really editing diffs and moving things back and forth was awful. Like slow and painful and the tooling wasn't very good, and you would have been much happier using, like, you know, I don't know, like the Emacs maggot extensions or whatever in Git than using the stuff that we've had. It's just interesting that, like, there are lots of different technical paths you can take towards the same big picture goal and end up with really different, like, sets of choices and

really different functionality as a result. Yeah, and there's a whole engineering culture that flows from that, how you do code review, How you think about giving people feedback on their work is entirely sort of shaped by the underlying technical system. Right. And, actually, I think this comes to the question of, like, why does Jane Street have an editors team exactly, right? Like, you know, text editors, can't you just, like, use one of them? And I think it's just the case that, like, editors are much more central

to how we work and how we review code than you might imagine. I don't know, these were a little bit saying, like, Like how are editors here like different than maybe people have seen in the outside world in terms of the role that they play in our software development process? Yeah. So there are a few different things. One at a fundamental level, Jane Street has approached the text editor as the software engineer's home base. Like all the interactions with Jane Street's code review and version control and all these other systems live within the editor. You do everything there.

It's all easily keyboard, navigable, and you're not sort of jumping between different applications or pages. And so that means that because Jane Street does a lot of weird, different stuff, like not using Git, but using Iron, we actually need a team to build all this integration into the editors so that people can actually live there and not, you know, do some things in the command line and some in the browser and then write code in the editor

and sort of jumping between these different surfaces. Yeah. And despite the fact that we've built a lot of, I think, quite cool web UIs over the last few years and tooling for building web UIs, Jane Street is just like a much less web-focused place, certainly than like, you know, the big tech firms, which I think have all ended up leaning quite heavily

in terms of taking all of the interactions and sticking them into a web browser. And for us, yeah, it's just like you're in the editor, you make the changes, and the editor becomes both the place in which you interact with all these tools and also, like, the central location for understanding code and for communicating about code. Like, I think another interesting early design choice, which has, like, ups and downs, but I think has certainly cemented the importance of editors is that the way in which you give feedback on a change is in the editor, right? There's, like, specialized comments that you write with a

particular format to, like, mark a piece of feedback rather than, like, going off into some separate web UI and like clicking a button and leaving a note on a kind of PR that somebody else owns. And part of it is like this like formal feedback comes through the editor, but also because you're in the editor, some of what you do when working on a feature with someone is you just collaborate on the feature. Instead of leaving a CR, which is the name for these comments that we put in, you might just fix the thing. You might just make a change, which I think is like a great, it's like it's certainly I think it's a great part of how development works here it's also like weird and unsettling I think for people who are used to the other thing it's like

someone else is reaching in and modifying my PR it feels unsanitary but it's just like for good or ill or maybe as I want you to say for good and ill it's like just like a different modality a different way of interacting with him thinking about code and the editors like just have a really central piece in all of that yeah and as you're saying this really biases us to keep the editors as the home base, because if you're interacting with these systems in a surface where you can't just as easily jump in and edit the code, you then are sort of losing

that property of this code review system and property of a code review culture that I think we see is pretty positive. What is challenging about this at a technical level? Like, what are the things that make it hard to make the editors do the things that we want them to do? So one aspect is that the editors that we're using are things that exist in the outside world, VS code. Emacs.

And for example, one problem that we're at work solving in VS code right now is that in JASU's work, we sometimes have incredibly large diffs. So a giant configuration file or just a massive Okamil file. And it turns out that the diff algorithm in VS code has some serious performance problems once you have diffs of a certain size. And sort of there's furthermore an issue of if you try and open a large diff and it hangs and you close it, it's now like locked up the queue for viewing diffs at all. And you can't look at any others. And so there are just things like this where either there's. there's some functionality that we want internally that isn't accommodated by this external tool that we're building in. There are also challenges where, like, we have implemented some

integration with the iron code view system that messes with some assumptions that, say, VS code makes about what a buffer is going to be, what the URI for that buffer is, and that say, makes the built-in editor search misbehave in a strange way, and we have to track that down and fix it. I think that there are also challenging things about the kind of shared infrastructure between the editors, like the OCamal language server and the underlying system, Merlin, that provides type lookups and jump to definition for O'CAML. And there there's kind of lots of interesting performance problems and questions about what are the optimal data structures. And if we want to build a index of the entire mono repo,

someone has added extra information to do better to the files that the compiler produces to do better lookups of modules using this index. Well, now this index uses many gigabytes more of RAM. we can't actually build the analysis anymore. As I'm saying this, I realize a lot of the problems we're trying to solve relate to this kind of the scale of certain things that we're doing at Jane Street and making these general purpose tools like perform really well at these idiosyncratic use cases. Right.

And I think one thing that comes out of what you're saying, you're describing here is that the editor has this kind of central role where it's like the surface through which many other things come at the users. And so maybe the problems you run into are problems about like the type lookup system or this indexing system, things that are like not like

literally part of the editor, but the editor is the place where you see the issues. And so you're kind of driven to like fix things sometimes in the coder view system and sometimes often the kind of Ocama language stuff. Increasingly the Python world is like another place

or we're spending a bunch of time. Actually, maybe it's worth saying, like what has changed as we've needed to do more support for Python? I'm kind of curious what kind of pressures that puts on the editors. Because in some sense, it shall be easy, right? The whole world uses Python. Now we can just do what everybody else does. I think the main surface for sort of folks in kind of trading and research who are interacting

with Python is the Python notebook. and this has been, I think the area that's been very challenging. There are different notebook solutions. They all have different significant flaws or limitations. And I think it's been kind of one of the pressures is even understanding like what the different use cases are and why people use one over the other. and a sort of unfortunate trend of spinning up a new one that we hoped would replace the old one, except it has a non fully, like a, it has, it's missing some functionality that the old thing has, and so now we're maintaining multiple of them.

And one, other pressure is that for people who are writing a lot of code in Python notebooks, that code is not always easily shareable. and for example, you might want to take it and put it in a Python library that lives in the repo that can be imported and shared between multiple notebooks. And for that, you'd really want people in a service like VS code where it's sort of just as accessible to edit Python files as it is notebooks. But at least the kind of Python notebooks work that people do here, VS code has been very, challenging. It has had a lot of performance and usability problems to the point where we're now starting to work on kind of just building our own solution to this, which is, I would say, a theme of a lot of the tools work at Jane Street. Right. And I do worry a lot that we like

over lean in the direction of building our own things. At the same time, like, you do, there's a lot of power in having control and having everything built in an ecosystem that you kind of understand from top to bottom. So I actually feel like you see work happening, especially in the editor space, that's like shearing in both directions, right? There's some places where like in the notebook work, we're like, ah, yeah, now we're going to like, instead of like these multiple things on the outside world, none of which exactly work in the way that we want, none of them have quite the design goals that we want. Some of the ones that look closer aren't maintained anymore or whatever. There's like a whole bunch of problems with the kind of notebook stuff

in the outside world, we're like, we're going to write a nice little like, O-Camel thing, use our own web UI stuff, and like, it's just going to work smoothly, and we have a prototype that's pretty good. And like, I feel pretty optimistic about this, like, being both not that hard to land and also giving us a huge amount of control to be able to adapt the notebook to the very specific things that we want to do internally and have, like, a much better experience. At the same time, I feel like a lot of the other work that you guys have been doing is about, like de-weirdifying various aspects of our, of our setup. Like we have like, you know, 25 years of like weirdo special case emacs stuff that we

have built up over time. And we've just been spending a lot of time funding work on Emacs, you know, figuring out how can we can like take the weird custom thing that we do that we've done internally and replace it with the standard open source thing. So I feel like actually both of these directions are good. You just kind of have to pick your spot. If you can find a way to make the upstream. thing, do what you want, that's just so much nicer.

Other people improve it for you. Right, the code that you don't have to write is the best code of all. Not only you're not writing, but lots of other people are using it and reporting problems, and those are getting fixed. So you don't have to find, we don't have to find all the problems ourselves by running into them. And turns out the upstream Emacs people really understand Emacs well and they know how this system works and they have the design aesthetic down and like in many ways you just get better stuff

than you would get from our like custom creations. Yeah, all the different pieces work together better that some external package will play much more nicely with our internal extension. If our internal extension does it the sort of native Emacs way as opposed to some totally different approach that then breaks when it comes into contact with. the outside worlds. Right. Although a thing that pushes in the opposite direction

is like a bunch of the extensions that we've done, we've written in Ocamille, which is like super not the standard way. In fact, we do this across multiple editors for like Neovim and EMAX and VS code. We have Ocammels are like,

at least one of the major implementation language. You also have to use the kind of more native language to do some of the pieces. So how does that like play into the kind of technical tension? Yeah, it definitely complicates the story and in Emacs in particular we're still, we have some ideas about

we haven't sort of put the boundary between ELISP, the sort of native EMAX implementation language and E-Camel, the flavor of a camel that we use to write EMAX things. Like the boundary there, particularly for asynchronous things, is pretty awkward. And so I think So a long-term goal would be sort of re-architecting that to have sort of fewer potential for weird deadlocks when these two sides are waiting on each other or things like this. Is part of the issue here is that like just Emacs is like super ancient and does not have a great concurrency story at all? Yes. So I think to some extent that that's true, I think upstream EMAX is actually working on this. I don't think they, and in particular there were some changes to the garbage collector

that I think may play more nicely with asynchronous things, but unfortunately that kind of wasn't ready for the kind of current kind of of EMAX that were working on upgrading to Emacs and VEM are both sort of like tools that have been around for a while and like have some limitations in terms of the like kinds of interactions that they were. intended to support. Right. And even beyond that, just like marrying the concurrency permutives of two different languages is really hard. We actually have run into like very similar challenges around Python and O'Campel, where I think we have like now maybe started to work out a story that makes more sense. But it's, you know, but it's a tricky,

it's a tricky thing even when you have like pretty good concurrency stories on both sides. So to kind of pivot a little bit, like I think a thing that has affected kind of everything about developer tools is the emergence of like actual good agentic AI for for programming. And I'm curious how that has affected the way you think about work in the editor and like, you know, what you prioritize and what are the things that you want to build. Yeah, there are a few different pieces to this. One, I think it hasn't really affected our belief that the editor can still serve as an excellent home base for the software engineer and that we want to make the new workflows that are emerging

sort of just as natural and accessible via the editor as the ways people were working before. And then there are also some things that seem like they aren't going to necessarily change about what people are doing the editors. This is looking at diffs, doing code review, working to understand and read code, whereas kind of how people write code may be changing and is changing significantly. There are certainly plenty of engineers all over the place, but also at Jane Street, who just like don't type code very much anymore. And most of the typing of, you know, they type more in the way of prompts.

But they're still reading code and they're still like looking at the types of things and looking at the definitions of things. So they're doing lots of things you do in the editor, but like maybe less editing. Yes. And this, I think, has created quite a lot of uncertainty for people working on editors in terms of what is it that is useful to build now? What is it that is going to be useful a year from now? Will anything? And one thing that I've gotten quite interested in recently, particularly because the editors team has been investing in improving the sort of telemetry and observability of what people are doing in editors and how they're interacting with.

with the OCamill language server is trying to use this telemetry to understand how is it that people's work is actually changing. I looked the other day, I had a hypothesis that, well, if I look at the number of times per user that people jump to definition in OCam, if people are, if the LLMs are producing good code and people aren't sort of writing and having to look up APIs as much, they'll be jumping to definition less on a per user basis. But then in the telemetry, it seemed, if anything, they're doing it slightly more, which was quite surprising, but also maybe relates to people are actually needing to read and understand more code than they were before. And so I'm excited for this kind of data to guide, to help us understand, like, what is it that people are actually doing in editors because it's kind of been quite amorphous as things have been changing very quickly.

And using that to help us decide what is important to invest in and what are the sort of workflows to make better because questions like, should we try hard to make jump to definition faster? That depends if people are using it more or less. Yeah. No, it makes a ton of sense. And the thing you're proposing of like using the data his drive seems more right than my instinct, which is just like theorizing the air about it. But on the theorizing the air part, it does feel like understanding is like, at the moment,

the thing that's, like, most important, right? You know, what's the, you know, the cost of creating a plausible looking PR, I think this is, I'm stealing something from Charlie Marsh, I think, like, the cost of making a plausible PR has, like, dropped nearly to zero, and the cost of verifying it is about the same, or maybe worse, right?

I think notably, like, there's something, kind of uncanny about a lot of the features generated by LLMs in that they're like, they're very smooth. They like look very nice. A lot of like tells of things that you might think of as like things that are like indications of a problem just kind of aren't there. But they're often still super broken.

Right. And so the process of reading through an understanding is at a minimum different and in some ways certainly harder. And so the idea that we need to like spend more energy and engineering work making it easy for people to understand what is happening seems really important. I think some of that understanding should come from further leveraging of LLMs. I think LLMs are like a really powerful explanatory tool. But I think it's easy to underrate the value of kind of traditional, like, hard ways of extracting

data from code and surfacing it. And like, I think things like go to definition and inferred types and test results and things like that are all really exciting. Yeah, I've also been thinking more about people's actual sort of workflows when getting LLMs to write code and how that relates to sort of understanding it at the end result. So one thing that I have played around with in some of my education work is Claude Codd Codes learning output style, which changes the system prompt to tell it like stop sometimes and leave it. a to-do-human sort of comment in the code and ask the human to go fill in that piece. And in using this myself, I found that having it stop part way through, and then I had to read what it had written so far and understand it enough to fill in this piece.

I felt like I came out at the end, sort of both have had a sort of more pleasant experience of producing this code, but also a more pleasant experience and I think effective and actually understanding what it had produced. And I'm curious to explore not just kind of on the output end, adding more explanatory power and tools to the code that's already produced, but thinking about how do people interact with these models as they're producing the code to most effectively end up with a solid understanding of it at the end and not just be guided well,

it feels more efficient to have it write everything and then read it at the end as I become a little skeptical that that is actually sort of the optimal way to approach it. It's interesting. It reminds me a little bit of the thing we were talking about about the utility of taking notes while writing in the pencil. The idea that, like, you are involved in the production function

will cause you to better understand the thing at the end. And there's maybe a kind of infillicity in the current way the tools are set up in that they largely, like kind of everything is kind of tilted in the direction of, the model generates the code, and then you decide whether or not to accept it, and maybe that's wrong.

That's a fascinating idea. Yeah, I was giving a talk to a group of interns about code review recently and had a slide. You know, code review is hard. Writing code, you're sort of incrementally building your understanding, and when you're catching mistakes, you're sort of correcting your mental model. And when you're reviewing someone else's code,

they're just presenting you with the finished result, and you kind of have to awkwardly kind of build your own mental model of it. And it immediately struck me, wait, this is exactly what working with LLMs is like in the way that it makes it harder to have a mental model of the code you were authoring. That sort of started me more friendly down this path of, yeah, maybe we should be interacting with these things differently, at least some of the time. That's really interesting. And there's just a lot of space for innovation here

and new methods of approaching. Just like LLM suddenly make the things that you can do. It's like a very different cost structure and there's a very different opportunity set. And so like the thing you're talking about with like, you know, the cloud code learning mode thing is something where you can like integrate a kind of tutor-like behavior into the automation.

And like, that's just like dramatically cheaper everything to have done than, you know, before you have to like go off. and go and talk to another person to get that experience to happen. I also feel like in the editor space in general, there's a lot of room for custom integrated visualizations and ways of understanding the data

and that the models are really good at ginning these things up. And like the idea that we should be able to ship along with a feature, a custom visualization that you made for understanding that diff, like that just kind of shows up in the middle of review. Like, that's the thing that we could plausibly do now that would have made no sense three years ago. Yeah, and one thing that I'm excited to explore that connects to your point about these things

being very good at explaining is that you could also sort of bake into a feature some sort of pre-computed sort of segmentation of the code into sort of logical chunks and then a pre-baked kind of explanation of each of those chunks that you can just call up with a single keystroke as you. you're reviewing the code. I mean, as you're, you can have highlighted an ASCA model to explain the piece, but that's some additional friction. The response doesn't necessarily come back that quickly. And if like for any given piece, you could sort of call this up,

I think that that might also be a sort of powerful tool for helping people more easily understand the increasing volume of code they're being asked to review. Right. And I think there's just like the modern condition of a software engineer, and by modern, I mean for the last eight months or something, is that there's just a lot more review to do as a proportion of your time. You're just spending much more time reading and understanding. And so suddenly the value of making that better just like shoots up by a lot. I mean, it was already a lot of work, so it was already somewhat valuable to do it. But like, it's now much more clearly the rate limiting step in all of this work.

So we're talking a bunch about how AI affects the experience of developing code and understanding code. And this like resonates, I think, a lot with a question of like, how does AI affect the process of teaching people stuff? So I'm kind of curious what that has felt like as you've been kind of helping to run education programs here amidst the middle of this AI revolution. Yeah, so this is, I would say, quite a thorny problem. Because you're faced with this tension of, like, one of the things that I help work on, and maybe we can talk more about this, but this workshop for early career engineers at Jane Street focused on techniques for software testing. And we call these workshops teach-ins. So this testing, teach-in, teaches people how to interact with various libraries for,

property-based testing or for deterministic control of time in tests. And when you're teaching someone how to use an unfamiliar library, well, an LLM could just write that for them. So what is it about this library that someone actually needs to know in practice and what might you consider a sort of boilerplate that you don't need? The initial scaffolding, can be made really easy, right? The models can do a decent job of that. But what's the understanding that you have to get to people? Yes. And so I think so far we've come down on the side of people having to think through

these sorts of problems and implement them themselves, at least once, is actually pretty important for coming out of it with a sufficient understanding. And even if what they are doing subsequent, to that is reading applications of this technique initially authored by an LLM, building that understanding by having done it themselves once is sort of a critical input to then being able to evaluate those things. That is one piece of it, and I think we are actively thinking about like where to draw the line in terms of there's some stuff that we were probably asking people to do in the past because it was sort of stuff that they had to do, even though it was somewhat

toilsome that, yeah, we should probably not have kind of people try and learn how to do those pieces. But there's also, I think, the challenging question of how do we teach people and even what do we teach people about using these tools effectively, about how to integrate them with their workflow, about how to understand the ways in which they can be effective or lead you astray. And I think this is still in very early days. I don't think we have sort of settled answers to this, and the tools are changing very quickly.

But you can already see problems, right? There are definitely people who run into problems where they just, like, kind of vibe it up too much and, like, use the LLMs to generate stuff and do not think. And, like, that's a real problem. Like, you still need to think. I actually worry more generally

about what it means for the overall education system. Like, I think a kind of critical component of learning is struggling with the material. And the world where you always have a genie at your beck and call that you can just, like, summon up to solve the problem, is itself like a real problem. And the overlap of problems that are easy enough that they are suitable for someone who is learning to do and hard enough that an LLM can't just one shot it is like close to the empty set now. So it's like a really challenging problem.

I mean, it really does make me just think back to the like doing stuff on paper without a computer. Sometimes an important thing about making the educational process work is withholding certain affordances. And I worry that we have like a whole generation of people who are like learning in this new environment where we don't know what to do. We don't know what affordances to provide and which ones to withhold. And like, I worry that we're.

like not building a good educational system for people in this world. Yeah, I think you and pretty much everyone else involved in education is deeply worried about this. To your first point about like, how do you get something that's small enough to be tractable for students and large enough to not be trivial with an LLM? Like, we have run into this exact problem multiple times. With new curriculum we've built and we've sort of repeatedly ended up, the thing we're asking people to do is too large for the time we have. Like, we are asking them to do something where they do have to put a lot of work into the LLM code to make it good and actually do what they want. But that just takes too much time to do well, given sort of the amount of time we have on the education piece. And I think it's going to take more experimentation to figure out where that lands.

but yes, I think that figuring out where to slot using these tools into sort of traditional education of it feels inevitable that we're going to end up with some sort of structure where you do this piece or this set of activities without the aid of LMs, and then you do some with, and I think figuring out kind of how to make that, pairing and what you're trying to get out of each of them is something that a lot of people are going to be experimenting and figuring out. Yeah, and I think it'll be interesting over the next few years as we're doing this increasingly with cohorts of people who have not spent that much time typing code themselves. Right? They'll just be like, there's like a core learning mechanism that maybe they haven't had enough experience with. And yeah, I wonder,

I wonder if that will change what we have to teach people in the end. Yeah, there's been a lot of conversation about things along the lines of that cloud code learning mode, where what LLM tools, if any, should we provide to folks going through OCamill Boot Camp, learning OCamill for the first time? And if they're just having LMs write all the code, well, it doesn't seem like they're going to learn anything. Yep. but it also feels a bit odd of people are going to leave this and then LMs are going to be a core part of their day-to-day work.

And so not having those be at all a part of this also feels like maybe not a great match for what we want to teach them. And so can we come up with prompts something, a mode where the LM will just sort of ask you questions to check your understanding or will leave you large? pieces to do yourself, like we'll sort of set up a kind of skeleton, but actually ask you to do most of the implementation. I think it's going to be a lot of experimentation with that as well. And maybe the thing that's been talked about a lot in educational circles of setting at the LLMs to be effective tutors, right, so that you can like, there's a part you're supposed to write

yourself, and maybe you get stuck on something, and maybe you can get the LLM to give you a hint, but not to go off and write the code for you so that you do have the opportunity to, like, actually do the hard thinking of yourself. Yeah, this is actually a category of thing that I did research on in grad school, of an automated tutor that could walk you through an arbitrary example problem where you had some algorithm that you wanted to teach people to follow to solve this kind of problem. And with the idea that, sure, you had the one hour you could go talk to a TA and get one-on-one help, and that was the most effective thing you could do. But if you were at other times studying for an exam, it might be really useful to have this

help like not as effective but still helpful sort of tutor that is infinitely patient and can walk you through any example. Yeah. I think that that is likely going to become a large part of at least higher education. Makes sense. So we just talked about this testing teach-in, which is part of like the kind of formal educational program that we have for new engineers. Maybe it's worth walking through like what what do we do to kind of onboard people and teach them about our world? Right. So a new software engineer joins, and the first thing that they're likely to do is Okamo Boot Camp,

a sort of cumulative series of exercises, building up a little client server app in Okamel, and learning the different core libraries and getting their code reviewed by a mentor and starting to understand some of Jane Street's style. And from there, we have something called a production boot camp where go through a series of exercises to learn how applications are actually deployed and monitored and configured at Jane Street. And that forms, say, like, the first two weeks.

Someone is on the job. And to me, we were saying, like, that's really been refined a lot over the years. I think, like, the O'Kamel Boot Camp itself, like early versions of that are something like 20 years old. But there have been a lot of work. In fact, you've been involved in a bunch of the recent stuff to kind of refine and hone and make that better. And the fact that nowadays, like,

someone can arrive and get through that in about a week and then, like, a little more for the production boot camp. There's already, like, a really good improvement. Like, I think we've been able to make that process more efficient. And, I don't know, people who come in to Jane Street, there are good and bad things. People are, for example, always super enthused about, like, the overall state of all of our documentation.

which, you know, has its ups and downs. But I feel like I've gotten, you know, especially in the last few years, really positive reviews from people of, like, these early boot camps. Like, they really, I think, do a really good job of, like, setting people up with the basics that they need to do and doing it in a really efficient way.

So I think that's been a nice piece of progress. Yeah, one thing that we've worked hard on, which I think is important when teaching this kind of material, is introducing new concepts in context where it's motivated, It's motivated by some specific application, some feature you want to add to this system you're working on. And I always frame this as the mistake people make is they introduce a tool and then go in search of a problem that it solves. Rather than like, we're working on this thing. Now we want to do this.

We don't have a good way to do this. But wait, here's this thing. Why don't we use this? And yeah, I think that makes it smoother. That helps people like have sort of hooks that they can hang their understanding on. And so that's been a lot of the work to rework these intro materials. Cool. From there, in someone's first year or so at the firm, they'll go through this series of T-chan.

So I mentioned the testing one. There's one on sort of writing performant OCamal Code and the tools to analyze the performance of OCamil Code. There's one on systems debugging, all the different tools. we have for understand what might be going wrong with some system. There's one on advanced functional programming, a bunch of the advanced features of OCaml and cool ways to apply them. And then there are also some more specialized concepts. It's a teaching on Bonzai, the library for writing web pages and web UIs via OCam.

There's one on market data, how the systems for market data work at Janestrand. which very important to some people, but other desks not so much. And I think as we add more, we're going to be even more in the mode of it's sort of a course catalog. And you and your manager are folks on your team, sit down when you start and sort of think about which of these are relevant to you to do when and kind of almost devise a sort of course of study for your first year at the firm. And we also try and create these materials in a way that's a to self-study, and you could also sort of go and work through them on your own, sort of at the point where you actually need to learn and apply the material.

Yeah, one of the interesting things about the teachings and about how those have evolved over time, like they were initially built as a kind of reflection of a trading side education program, also called teach-ins, where that was mostly organized around basically going around from trading desk to trading desk learning the kind of core concepts that were important for that desk. And my mental model of it early on was like, you know, there's some specific technology you want to teach people about. But the more important thing is the kind of conceptual layer of like, you know, Bons has like a web framework, but it's also an introduction to incremental computing, right? And, you know, market data is like a particular domain you need, but it's also thinking

about, you know, handling like complex streaming data and building modular and efficient protocols for kind of consuming those in a good way. Like, there's a bunch of architectural lessons from each one of these. And I feel like that initial vision sounds good and has been less successful than I expected to be. And we've kind of over time gone in a direction that's been more about giving people like the practical tools that let them go do things. I'm kind of curious how, like, how do you think about that? Like, how does the conceptual piece, fit in? When and where should we be focusing on that? And to what degree should we just giving people? What would feel like the most practically useful tools? Yeah, this is a very interesting difference in the

Jane Street context compared to, say, college courses, which in a college degree, you know, maybe I'm teaching a databases course and there's some domain knowledge about databases I want to impart. but as part of their university education, I also want to teach them to be capable problem solvers and how to work through ambiguity or try out different solutions, kind of a lot of these general skills. And one of the great things about working in Jane Street is that my colleagues are very smart and capable, and I don't really need to use their time to teach them problem solving skills. They're already really good problems all, which it turned out.

There's some selection effects there. And so this, I think, has biased the education away from what you might see in university of, like, we're going to have you spend a bunch of time grappling with this tricky problem and figuring out a solution because actually when people are at their desk in their day job, that tends to be what they are doing. and it therefore can feel pretty, I guess, unsatisfying to do that on a kind of canned educational thing, as opposed to something that's more directed at here, like, useful tools that we're going to, like, have you work through and understand, which you would then go apply to, like, the truly tricky problems.

So I think that you're right that there are some core conceptual things, like the incremental computing, things like this, which we do want people to be exposed to and think about. But we tend not to do that by asking them to take on some particularly sort of thinky piece of it to spend several hours just kind of having to come up with a kind of novel solution to it. What you're saying here is even if a big part of your goal is to convey the conceptual lessons, it will work better if the way you do that is by teaching them like concrete useful tools and focusing on those. And like some lessons will sneak in that way, but not because you're shaped it explicitly about that, but because understanding the tool and being able to use it required the understanding of those lessons. Yeah, I think that's right.

and also teaching this conceptual knowledge just in a different way than you would in, say, university setting, and doing it often more in ways that look like a walkthrough of let me take you through this kind of problem and how you would solve it. And you'll do some pieces, but how you will sort of build the more complete understanding is when you actually go and apply this. to the problems to sort of solving on the desk. So this is a little bit about some ways in which teaching at Jane Street is different from the outside world. I feel like a thing I learned from you early on is about some of the ways in which lessons from the outside world apply here. So we talked a bunch about the testing teaching. This was actually like the first real project that you did here where the way I remember it is we were trying to organize this testing teaching before you started.

And it was going badly. like a little bit because it was this problem of like it was hard to quite get enough of anyone's time to like you know people were doing you know good work and focusing on various parts of it but there wasn't like a real effort to kind of bring the whole thing together and like make the whole thing good and land it and I remember you asking me early on because like the shape of it was a thing I had come up with and so you asked me so like what are the learning goals and I was like fascinating what are learning goals like it sounds like you know there's you know those you know those are like English words that kind of make sense together, but clearly you mean it in a more specific way. And I found that to be like in the end like a very useful and insightful way of thinking about teaching. Maybe you can like say a few more words about like what are learning goals and like how should that affect how you design curriculum and build courses? Yeah. So when starting off on like any larger project, it's good to write down what you were trying to accomplish. And in this case, learning goals are a specific way of writing down what you want to accomplish in terms of what is changing about the students as a result of the thing that you're doing.

And specifically, I find it very useful to write them in the form after completing this teach and say, students will be able to and then write down a bunch of active verb phrases. So, for example, like, understand property-based testing is not a great learning goal because that's very vague. It's not a specific thing that they're going to be able to do. And how would you measure it? And it's hard to know how you would measure it. But students will be able to, like, write a property-based test, like a sort of nested record-type input. So sort of some sort of concrete, like, scoping, what is the complexity and the sort of thing

they're going to be able to do. One that really forces you to structure your thinking about what am I actually trying to accomplish here? And importantly, what am I not trying to accomplish? Like, what is not a good use of students' time? Because it's not actually a goal to have them do this thing. But then in terms of the overall design of the educational program, once you have written down pretty specifically what you want people to do or once you want people to be able to do afterward, well, it follows pretty directly what you should have them do in the thing that you're running because they should practice the things that they're going to be able to do when they're at the end. And, okay, now I know the sort of exercises or activities I'm going to have them do.

Okay, what is the background that they need to know for this? So what lectures or kind of other materials or what parts of code would we want to provide them for which writing those is not relevant to these goals? And so the whole process of developing some new piece of curriculum sort of flows nicely out of having been sort of disciplined in writing down these goals. And I feel like to some extent, my education work at Chain Street has been going into different rooms with different people and saying the word learning goals. We should do this. Right. And the other thing that I found interesting about the learning goal idea is it tells you something about how.

how you should think about evaluation. And here, like, we mostly are not so interested in our teaching stuff to evaluate the students. But you really want to evaluate the program of, like, did the thing you do actually teach people anything useful? And so the idea that you could, like, see whether or not at the end that people are able to do the thing that you wanted to do is, like, a way of evaluating yourself and your own work as an educator. And I guess the other thing that I think striking about is I think that, like, learning goals has now showed up for me as like,

in some sense, two different things. One is there's this like somewhat more specific way of writing them out and thinking them through and using them to structure the program. And then there's also just the frame of mind of like, how are the students going to be different at the end, right? Like, what are you even doing here? Like, why did you show up to work today?

What are you trying to achieve? And I found actually that framing to be very useful in talking to, like, potential developer educator, Like talking to people who want to hire to do education work is like, you know, I think some people have really thoughtful answers of what they were trying to achieve in the classes that they taught. And sometimes it was like, oh, I just like this material. So I thought a course about it. And like, I don't want to complain too much.

I think a lot of great classes are just like someone is excited about material and they teach it. And like an enormous amount of good education is not about process, but is about like charisma and ideas and being able to like just carry the performance through and really. caring about the end results. But I do feel like the structure helps, and especially helps people who haven't done it before, right? People who are trying to figure out how to teach classes for the first time. I think it's, like, very helpful kind of way of organizing your work and your thought. And it forces you to focus on things that are sort of not just in the students' head to some extent.

of what am I trying to accomplish here that isn't just sort of like I tell the student a thing and now they know it. But yeah, kind of how are they changed in some sort of observable way? So are there other aspects of like the outside like learning curricula and approach?

Like, you know, the whole world of like education, the whole world of research on education is like complicated and, and also, like, includes lots of conclusions that, like, probably aren't true. And also lots of conclusions that we've known for a while that somehow don't get applied. Like, there's all this crazy stuff about, like, ways of teaching reading where, like, we had pretty good research telling us what to do, and then a lot of, like, not doing it for all sorts of messy reasons. And yet, I think, like, the learning goals is a nice example of there being real insight

that comes out of the kind of whole, like, intellectual stew of the educational world. Are there like other pieces that you think of as like really important and valuable that you try and leverage here? Yeah, one that I've tried very deliberately to apply to the Ocampo boot camp is what is sometimes called cognitive load, or you might call it working memory. But just when you are trying to learn something new, there's actually a limit to like how many new things you can sort of handle at the same time before it becomes harder to learn any of them. And so in an older version of the Ocamel Boot Camp, right away you were learning O'Camel, you were learning an editor that perhaps you hadn't used before, and you were learning a version control and code review system that you also hadn't used before, and you were doing all of these from the jump.

And one thing that I did was sort of split these apart. So you start off just learning O'CAML syntax in a sort of in U-TOP, the sort of command, line Ruppler and now we have nice kind of in-browser kind of text editor boxes where you can run little snippets of OCam. And then you go to a piece where you get introduced to your editor and you do some activities just with the version control system. And only after you've done each of these sort of introductory pieces do you get to the part where you're bringing them all together. So kind of applying this cognitive load idea in that way. And I think the other thing that comes to mind, which I mentioned before, is this idea of active learning.

of trying to make the educational activities not just someone telling you things, but as interactive as possible. And I think that some of the new material that we have developed for the internship this summer is really applying this in some interesting ways of lots of small group discussions or kind of interactive activities around design or practice. practicing talking to stakeholders, yeah, sort of applying this idea of people actually interacting with other students as an important part of the educational process. Right.

Again, leading into like, this isn't just about conveying information. There's a big psychological element to people learning and retaining. That's right. So maybe before that, you're talking for a second about this point about the internship. So you've been involved in what hasn't been a pretty serious rework of the kind of educational kind of intro that people get to the firm. Can you see a little bit more about what motivated that and what we've tried to do in that space?

Yeah. So the sort of precipitating motivation was, wow, AI tools are really good now. And maybe the four-week project we used to have interns start off by doing, like, that's one prompt and you're done. And so that's not a useful way to evaluate it. an intern or like productive work to ask an intern to do. And so, okay, we're going to change the nature of the projects that interns are going to do to be more open-ended, to maybe involve more elements of design or gathering requirements, things like this.

Basically making them like harder and more realistic. Yes. And more sort of going beyond like, can you produce a reasonable amount? of good code. It's just sort of just producing the code. Like that used to be maybe among the most important signals and it's less important now, I would say. And so aligning the projects with like, these are the things we sort of want to sort of see people do successfully. And, well, to set them up for success, we should try and teach them some more about how to approach this, how we think about this at Jane Street. And in addition, not just sort of dump them into the deep end of like,

you have access to LLM's good luck, but actually like try and train them some on what to keep an eye out for, what are effective ways of working with these tools, what tools even exist at Jane Street and how do people use them. So the sort of new materials have been a combination of giving people, and this is the sort of balancing act that I was talking earlier, tasks that are like sufficiently complicated or large of the LM doesn't just give you a perfectly fine version on the first try, but not too large, that, you know, it's going to take you a week to work through all the code that is produced. And also taking this as an opportunity to focus on some new software engineering topics that maybe didn't get a lot of formal training at the

starting internship, like testing, kind of writing a very good, test suite for this database backed application is kind of part of the new curriculum. And that's testing has always been important, but like in many ways more important now because the models thrive on feedback, right? You really need to give them ways of figuring out whether or not the thing they're doing is right. They're not just going to like, when you get beyond relatively simple things, they're just not going to one-shot good answers. For sure. Yeah, it's been important, but there's always a balance of like, well, we could spend half of the internship teaching people's things, but we actually want them to spend most of their time working on a project. So sort of

in this new world, we're like, okay, we're going to definitely invest more in education, but we still at some point need to get them starting to work on their projects. So there's also been new exercises around looking at design docs and judging what is a good design doc, what is looking at some bad ones and discussing the flaws and more on this sort of planning side of things. There's been, we developed a new activity where there are various full-timers, sort of role-playing different stakeholders for this app that the interns are working on, and they talk to them and sort of try and gather what are the requirements for what I would design. And then the final piece that I would mention is that we've talked about code review,

becoming a sort of more central part of the job. And so we traditionally have not had interns review other people's code. At least that would be pretty uncommon, but this summer, this is something that we're going to ask interns to do and that we want to actually understand how well they perform at it.

But to that end, we actually have three additional days of training in the middle of the internship on code review, where the interns will see pre-made features actually adding functionality to the

same application that they were doing the testing and design work on earlier in the system. And they are leaving code review comments and then they are seeing a reference of like, here is the set of kind of the answer key. Here's the set of code review comments that someone might have left in discussing as a group sort of what they saw. There are ones that they disagree with. And then also implementing their own features and interns reviewing each other's code to give them some reps doing this with the idea that then their mentors and their actual project will

send them some features for a review. So this is like a really big change, both to like the educational part of the internship and really to how the internship with the whole runs. How do you think it's gone? I think we're learning a lot. I think a lot of the challenges that have come up are a lot related to the scope of what we're asking people to do. So if we want people to generate with code with the LMs and really hammer the message in, you own this code, you need to review it carefully, you need to get it into a good state, you don't just stamp whatever the LLM produced. Well, if we ask people to produce much more code than they have time to polish,

in some sense, that doesn't quite line up, and we're not kind of communicating the lesson that we want. So I think we've sort of learned a lot about, like, what is the right scope for these different pieces? And similar with the code review, like sending people sort of too large a feature with too many problems, and if it's hard to sort of, for people to kind of spot the issues that we actually want them paying attention to.

And maybe it goes to like the cognitive load issues you were talking about before. Yes. So I think there's been some challenges there, but I also think it has been really great to see this sort of more interactive form of education compared to, say, the O'Caml boot camp that people, the interns went through because in that, they're going at their own pace. They're getting code reviewed by a full timer and sort of discussing with them, but they're not necessarily doing a lot of talking to them. other interns about the material. But this new curriculum, interns are in these pods of four people, and it's explicitly structured with these small group discussions frequently throughout it, both like, how are you using the LLM tools? Like, what problems are you seeing with the sort of tests that it's spitting out to you? And I think that those discussions, including kind of

full-time mentors, have been really productive. And I'm excited about kind of anchoring a lot of our, at least intern education in that sort of model. So it sounds like next time you run this, we run this, you'll want to, like, reduce the scope of some of the tasks people get. Are there any other changes you think you should make? So I think that reducing the scope, but also there are adjustments to, like, emphasis and pacing. So one thing that is that is always a challenge with programs like this is that people go through them at very different paces. And how do you accommodate someone who's like racing way ahead?

You want useful things for them to do. And if people are going through it more slowly, what do you actually have them focus on? So some of this is about being careful with the schedule and also kind of how we are messaging about what the expectations are. such that people are actually taking the time to learn from the things that we think are most important and not trying to rush ahead to sort of keep to some schedule. And I think there was, at least there are different groups of interns. And this has been, this whole curriculum has been evolving rapidly and significantly between each batch of interns as they start throughout the summer. But in an early one, we had an issue of starting out with,

emphasis on a lot of multitasking and sort of vibe coding stuff. I think we felt afterward that that was not sort of the right place to start. It sort of set a bad kind of tone for the rest of the exercise and there's been some rearranging to. We're actually going to start with like the testing focus piece and making this testing suite like really high quality and sort of framing it around that rather than oh, you can have an LM do three different things at once. and like, it's cool. It is cool. Yeah, and I would want, like I was talking earlier about thinking about different ways of working with these models

and having people sort of kind of checking in with what they're doing more often. And I think sort of by next summer, I think we might have like a lot more considered ideas about this or people have tried different things. I think it's very possible that we'll want to, like, build more sort of guidance into, like, here are specific workflows to follow, whereas I think now it was, yeah, we have some things to know about ways in which the systems can misbehave and things to keep an eye out for, but not a lot about, like, we think this is the, like, good way to prompt or interact with, these systems. And this is another reason why these small group discussions were great,

because people were sharing the sort of different techniques that they were playing around with and what was working well and what sort of wasn't serving them. Right. And I guess one of the challenges of this particular aspect of the world is it's just changing fast. And so knowing what advice to give, it's like we can give some advice now. Next year might need to be different advice. Not only is it changing fast, but two people using it in the same way might have it do different things. So when you give people an exercise that involves using an LM, it just introduces much more uncertainty into what is going to happen when they do it than you would have had in the past. Right. All right. Well, maybe that's a good place to stop. Thank you so much for joining me.

This is great. Thanks so much. You'll find a complete transcript of the episode along with show notes and links at Signals and Threads.com.