Future of Coding - Learning Programming At Scale: Philip Guo

Episode Date: May 2, 2018

Philip Guo researches ways to scale programming education beyond the classroom. He is the creator of Python Tutor (http://pythontutor.com/), a widely-used code visualization and collaborative learning platform, and an assistant professor at UC San Diego. In this episode, we discuss why diverse groups of people study CS, his various prototypes, and the differences between technological research and industry. http://futureofcoding.org/episodes/22Support us on Patreon: https://www.patreon.com/futureofcodingSee omnystudio.com/listener for privacy information.

Hello and welcome to the Future of Coding. This is Steve Krause. So first things first, I want to apologize for the radio silence for the last month or so with this podcast. I've been traveling a bit and it's been a little bit difficult nailing down guests, but today I'm excited to bring you Philip Guo. He's an assistant professor of cognitive science and an assistant professor of computer science at UC San Diego. He has, I think, a really fascinating research topic, summed up by the phrase, teaching programming at scale. His research spans human-computer interaction, online learning, and computing education.

He builds scalable systems that help people learn programming and he also studies from more of a sociological perspective how and why diverse crowds study computer science. One of his big claim to fames is the PythonTutor.com, a widely used visualization and collaborative learning platform. And his website pgbovine.net has over 500 articles, videos, and podcast episodes. And he gets 750 page views per year. And I think that's originally how I found Philip. It was either through Python Tutor or through his very prolific and fascinating website. And I think we had a really fun conversation with a little bit of different flavor than

normally, than I normally have on this podcast. We talked a lot about education and different demographics, why they learn computer science and how to help them. And then towards the end, we talked a little bit more about building prototypes that point towards the future in the way that we technologists do. So without any further ado, I bring you Philip Guo. Welcome, Philip. All righty. How's it going, Steve? Great. How are you? I'm doing good. I'm looking forward to this conversation because I have absolutely no context because something just showed up in my inbox and I went there. And this is actually fun because of that.

Great. Yeah. Thanks for being brave and being open to stuff. Yeah. So your work centers on this phrase, learning programming at scale. And it's a pretty self-explanatory phrase, but I was wondering if you could just unpack it for people who aren't familiar with your work. Yeah. So I came up with this kind of catchphrase, learning programming at scale, for some of the research talks I was giving a few years ago. And the basic idea is that, you know, learning programming or learning computer programming is now a pretty

widely accepted thing. And, you know, more and more schools are trying to teach coding to, at both the K-12 level and at the university level, you know, computer science departments are growing. A lot of people from various different majors are taking programming and computer science classes. So it's just a very blossoming thing. And there's things like Hour of Code and like trying to get computer science and programming in schools.

And I think these are all really great efforts. And my own focus is on something complimentary, which is the at scale part, which is, you know, let's think about the, you know, 99% of the people who don't have access to either a traditional school, you know, whether in K-12 with computing education access or to a college environment, you know, all these people who have to learn, who want to learn at all ages, all demographics, and all different life situations, how can we build tools and support to help them learn? So, you know, I think to really scale up to the world, we have to do stuff online. And, you know, it can probably, it probably won't be as

good as having in-person teachers. But, you know, the reality is 99% of the people in the world who want to learn this stuff don't have access. And we want to try to scale that up to the world. So that's kind of where the learning programming at scale, the at scale part came from. Interesting. So it sounds like you answered one of my questions already. Because my first one of my questions was, that is the ideal of learning programming at scale that one day we get rid of the classroom and then everyone kind of learns online. But it sounds like you think that the classroom is a wonderful thing to have if you can afford it. But your work is for people who can't for some reason. That's right. Yeah, i think it's largely complimentary that yeah i i think that it's it's still very hard to beat you know good in-person instruction um both from just from a

technical perspective of having someone helping you debug and helping you um along but also from motivational perspective right so it's like you know you could read all the stuff on stack overflow and read programming blogs you know day, but then having someone there coaching you along, motivating you, giving you some kind of an impetus to learn the stuff is just so much stronger than, you know, having to look through that stuff on your own. So I think the in-person stuff will not be really going away, in my view, but most people just don't have access to that. Yeah. So I think you mentioned in your work, there are a lot of different demographics that are like non-traditional programming learners that you focus on.

One in particular made me smile. You are focusing on why adults 60 and over are studying programming. And part of why I thought that was interesting is that two years ago, I was teaching a coding course for adults, which was strange for me because I normally teach children. And a woman in her 60s signed up. And her story is that her son was doing a startup in Silicon Valley and she was retired and she wanted to help him in her free time. And so she thought that she'd learn to code and maybe get to help him with his him with with his coding yeah that's really cool yeah i mean these these are really interesting stories so you know the the impetus for that project came about because we can talk a lot more about this later but um you know i have this programming education platform called the python tutor which

now is a bunch of languages so the name is kind of a misnomer but um we have a lot of users on the site so as you know it's a great platform for collecting So the name is kind of a misnomer. But we have a lot of users on the site. So it's a great platform for collecting research data because I can put these mini surveys on the site. So one of the surveys I put on a few years ago was trying to get people who are 60 and over because I think that's a very understudied demographic. So I collected a bunch of survey data and analyzed a bunch of survey, kind of more qualitative data, and trying to get at their motivations, like your anecdote before about the woman and her son, and really just trying to classify, you know, why are these

people learning programming? You know, most of them are probably not doing it professionally for their jobs, because they're maybe retired or semi-retired. But I just thought it was a really interesting population. There's also another population that I did a recent survey for that one of my more recent research papers about is about non-native English speakers. So like another understudy demographic are people who are not native English speakers. And, you know, when you think about programming, it's so English centric, right? These programming languages, tools, documentation, all the primary websites are in English. But again, if you programming languages, tools, documentation, all the primary websites are in English. But again, if you think about the at scale part, the 99% of the people

around the world, most of the people are not native English speakers, but yet they have this kind of double language barrier of having to learn English and having to learn the programming. So, you know, I think the non-native speakers and the older adults are two examples of populations that I'm interested in studying. And there are many others as well. Those are the two that I've kind of done concrete projects on so far. Yeah. So I'm curious if you could summarize what you found, like why were other older adults studying programming and also what you found with the non-English speakers? Yeah. So the older adult one, you know, the story you told about the woman or son was actually a really good one. Some of them were, you know, were about people who are retired, who wanted to understand technology and kind of empathize and keep in touch with the younger generation. Right. So, you know, some of these stories are about, you know, oh, my grandson or my granddaughter is in college learning computing, you know, programming stuff. I want to just take some online courses to see what they're doing because I have free time. There are other people who are actually still working. I mean, 60 is not that old nowadays. People are still working in the workforce. So there are people

who want to update their skill sets for work. So I think both between updating skill sets for work and the enrichment kind of aspect of trying to understand what technology really is about is both interesting. And then for the non-native speakers case, you know, the motivations, a lot of it were job related, right? Because especially people learning online is taking these online courses, trying to get certificates. A lot of it is job related. And I think for a lot of the people, it's really a way for economic mobility, right? If you're living in countries where there may be not that great paying job opportunities to do these remote programming jobs or be able to work as a software engineer, you know, in a remote office for a multinational company or even hoping to be able to immigrate to countries like the U.S. to work in the software industry. It's just like a tremendous boon in terms of just economics, right? So I think that's why people are really motivated, you know, especially from many countries where English is not their first language. Yeah, that makes a lot of sense. It makes sense to me when you said that one of the things you

found with the older adults is that they wanted to connect with the younger generation. I guess that's kind of the story that I personally found with the old lady in my class. Yeah. I'd be curious. I think you mentioned one of the other demographics that you mentioned in your research is conversational, people who want to be conversational with programming. And I've, and I definitely, you know, like, basically every like MBA student these days wants to be conversational, wants to be able to talk to their code monkeys. So I think that that's a very interesting demographic to study.

Yeah, so the conversational programmers, once this was led, these studies were led by one of my colleagues, Parmeet Chalana, who is currently at Simon Fraser University. So that's near Vancouver. And we did a bunch of things where it was mostly her and her students interviewing people, but they did a bunch of in-person interviews. And also we did some surveys as well at companies about basically our demographic was, you know, anybody who is trying to learn programming or, you know, taking these coding courses or self-teaching themselves, but their requirement is that they don't need to actually write code for their job, right? So this is, again, you know, this kind of going along with my research theme of thinking about the people who are learning programming, but not your typical demographic of what a lot of people study. So in this case, the conversational programmers is our requirement is you're trying to learn programming, but you don't actually have to write code. So then we're like, why are all

these people learning to code when they don't actually have to write code? And like you alluded to a big reason was because they are, you know, I think one typical case is like someone who's like a product manager, right? Like, you're kind of working closely with engineers, and you want to kind of know what, you know, how to speak their language and how to relate to them. But you don't necessarily need to be coding up all the all the implementation. Yeah, so is the point of this recent, like this motivational demographic research to like trying to expose problems with the coding environments we have now so that like kind of in the other half of your research, you could like build better environments. Is that? Yeah, that's actually, that's actually,

that's actually way more for thinking than, than I would love to say that that was a master plan all along. But, but, you know, in reality, it's these two threads of building interesting programming environments and studying demographics have been pretty independent. But yeah, I mean, one hope is that through a lot of this research, it will try to expose gaps in current environments. And then we can close the loop later on and say, you know, what are better ways to teach programming for people who are conversational or who are older adults or who don't speak English well or who are struggling with English as a language barrier? So far, I haven't really connected the dots yet. But I think that in the future, that would be a great thing to do.

I think right now, these two threads of research of just building programming tools, which we can talk about in a bit, and also studying these demographics are largely independent. And the stuff we just talked about so far with these few research papers have mostly been trying to add to the body of knowledge, right? you know, what are pedagogical best practices in classrooms for, you know, elementary schools, let's say, and then also what are pedagogical best practices for, you know, first year college students trying to take a CS1 intro course. So there's a lot of research in that. And, you know, what myself and my colleagues are trying to do is, you know, think about, let's think about people in completely different demographics who have not been studied and figure out their challenges and motivations. Yeah. So one question that I thought about when I saw the phrase learning programming at scale was like, if there are people like you studying learning like X at scale, now that we have, I guess MOOCs, like MOOCs is like the thing that kind of made learning at scale an interesting question.

Are there researchers? Like it almost feels like from some perspective, you're closer to like someone who's trying to teach learning math at scale than like to a regular programming field. Yeah, that's a great, yeah, I mean, that's like the other dimension, um yeah i mean that's like the other dimension right so so yeah the other way to parse the learning programming a scale sentence or phrase is uh is learning at scale and then you just shove programming in there right because they

can do learning math at scale or learning physics at scale yeah so the learning at scale community is you know that that that kind of keyword came up uh like you said, with the rise of these MOOCs in the early 2010s. And there's a very active community of researchers in the learning of scale community of which I'm part of. And we have a conference series, we have research papers, we have research meetings. It's a, you know, it's a burgeoning academic community. And you're right, a lot of those folks are trying to do kind of large scale data analytics, or, you know, basically applying, you know, A B testing data analytics, you know, kind of student progress tracking. And, and a lot of that work has been trying to be fairly independent or agnostic

of domain. But, but my own work is really just honing it on the programming domain. But you're right. There was a pretty big body of work on just learning a scale in general, I would say. And so I'm wondering, because I imagine there's a lot of overlap. So like, for example, your uh python tutor system um which i'll let you summarize it uh in a second here maybe why don't you just summarize it before i ask the question sure sure um so the python tutor system if you go to i mean you can link to it just python tutor.com um it started out as being for python but now it's actually for a lot of languages um but basically what it is it's a um it's a it is, it's a code visualization environment. So there's a lot of these environments where you can write code online and run code now, these online IDEs,

right? So you can just write code, maybe have a REPL, you can kind of run stuff to see what your code does. But what my system has that's really unique that I haven't really seen anywhere at this scale is the ability to do a single step visual debugging, right? So it's kind of, you write a piece of code, and you can actually step through it one step at a time, exactly like a single step debugger, except the twist is that it actually draws the state diagrams for you. So it kind of draws you all the stack frames, the global variables, what the types of the variables are, what the values are. And if they're pointing to complex objects, like an array or a list, or a dictionary, or, you know, some kind of compound object, it draws the pointers and the box and pointers for you.

So it allows you to step through execution, really inspect what your program is doing, in addition to just whatever it's supposed to do, like print to the console or something. So a lot of people have been using it to basically build up a better mental model of what their code is doing, because they can actually kind of peek inside of their code step-by-step as though a teacher were explaining it to them. Cool. Okay. Yeah. Thanks for the, the explanation. It's yeah, it's a wonderful, wonderful website. I,

I used it many years ago when I was first learning to code. Oh, wow. Thanks. So one of the things I think, at least because when I went back to look at it, I just now I saw that you have this feature that allows people to ask and receive anonymous help, which I thought was great. And it was like, even I was just like, you know, interested to like click on a link and see if I could help someone who, who requested it. And I was like, yeah, just, it was so simple the way the interface was built. But it, it seems like it could be a compelling feature. So anyways, what got me to this question is that I imagine that peer tutoring online is something that is common in the world of learning at scale. So I'd just be fascinated to hear about the challenges and successes in that world of

peer tutoring. Yeah, that's a great thing to bring up. And I'm glad you brought it up. So I'll talk about the feature in a bit. But this general idea of peer help is, yeah, I mean, I think that is the, one of the ways to scale up learning is that, you know, the reality is that there are very few instructors and there's a lot of peers, right?

Especially when you're at scale, there's even more peers because if you're in a MOOC and you have thousands of students, you only have one instructor, maybe a few volunteer TAs running the course. I mean, a lot of times with these MOOCs, the instructor isn't even actively running, right? It's been pre-recorded, and then they just run the course like a bunch of times every few months. And maybe there's some volunteer TAs or some kind of instructional staff who helps out, who mans or who kind of moderates the forum. But for the most part, it's the peers kind of trying to help each other. And I think the most common form of peer help in MOOCs and online settings is asynchronous, right, is via forums and discussions. So the forums and MOOCs are fair, well, good ones,

I guess, are fairly active. You have sites like Stack Overflow, you have other sorts of, you know, discussion forums, Facebook groups, you know, people who take MOOCs actually form these ad hoc Facebook groups, you know, where they just post messages and help each other. So researchers have kind of mined data from, you know, Stack Overflow is a good example, or forums from discussion from MOOCs are another source of, you know, public data that they combine. And I think there's less done in the synchronous space, which is, I can talk about that in terms of my project. But I think, you know, my own interest in this, again, is trying to push forward on where other people haven't done as much. So the synchronous real-time tutoring space is something I'm super interested in. So I'm happy to go into that in depth or answer your questions about that in general. Yeah, I'd be curious to hear how, I guess I'm also kind of, I was going to say I'd be curious to hear is how it's going, but I'm now wondering how you measure the effectiveness of a system like that. Like, do students rate at the end of a session if their question was answered?

Yeah, that's a great question. So yeah, just stepping back, the live tutoring mode, you know, I really do need a marketing person because, you know, this whole Python Tutor project, this whole thing is kind of a one- you know band at this point it's basically me holding it together with duct tape uh we can talk about open source project sustainability later too it's a another giant topic yeah that's a question on my list yeah that's a great question to cover later but um so so the idea is with this project is that um you know the the site originally was a visual debugger so that provided a service that was valuable to a lot of people. And it was, it was a single user mode, right? You go on, you paste in some code, or you write some code, and you kind of debug it. So it's like a, a scaffolded environment to help learners

understand how their code works. And then a few years ago, this was, I think, 2014, I basically added, there's this great library called Together.js that Mozilla Labs worked on. It's been discontinued. I mean, like many, I'm talking about open source, like many open source projects, it's been discontinued. But Together.js, what that lets you do is it lets you basically instrument a website, which is some JavaScript code, and it lets multiple people go on that website and browse together. And it works through WebSockets. So you start a server,

and then whatever website you're on, let's say Wikipedia, I'm just making up. You can have multiple people go on the site. You can see each other's cursors, and then you can chat on that site. So I thought that was a really cool library. So I integrated that into the Python Tutor site. So that actually turned the site into a multi-user mode. So what you can do is you can have multiple people go on a single session,

write code together, just like in Google Docs, and then run the code and play with the visualizations together. And you see the multiple cursors and then chat through text. So that feature was implemented about four years ago at 2014. And the catch with that feature, though, was that you have to send the link to somebody, right? So that feature, I called it, you know, live help mode or something. But the idea is that you start a live session, just like actually, just like this thing we're using to record the podcast, the Zencaster, exactly like it. It's like you start a special URL, you generate a garbled URL, then you send it to your friend or to your tutor, or maybe you post it on a discussion forum for your MOOC. And you say, help, I need help. I'm online right now. If someone can click on this link and join in, you can basically pair a program with me and guide me through these things. So that feature worked pretty well and we had a fair amount of users.

But the interesting thing about that feature was that a lot of people didn't realize that no one will come to help you, right? So they clicked on the start the chat button and they expected someone to come chat with them because they thought this was like a help service all right um so we got a bunch of this you know a bunch of the server logs just show people you know opening the chat and saying hi you know can you help me you know i really need help on this and they're just kind of typing into the void. That makes sense.

Yeah, so then, you know, after, you know, then the natural next feature to think about, I mean, this is how I design a lot of these things is actually by user demand, right? So by observing this, the natural thing to do is to think about how can you get the peers on the site to actually help each other. So the feature as it's currently implemented, which I implemented,

I think in October, so about six months ago, yeah, about six months ago. Weird story about that is I actually broke my foot. So I was immobile for about a month. So I just actually had a lot of time to code. It's not the best reason to code, but it's actually it was productive use of my time. I was basically at home hobbling around. But so the feature that I have now, which I guess I call live help mode, I don't know what better way to call it is, you can start a public help request. So basically, when you start a help session, you can either choose to be private or public. And if it's public, then it will post your request on a help queue, which is like displayed at the top of the screen, whenever people go to the site. And, and then it's just ordered by, you know, who goes on at first. And this was motivated

by kind of in a, again, a lot of these, this work is motivated by in-person classrooms. So in a lab, you know, imagine if you're in a lab class in college, you would probably either raise your hand and have the TA come help you, or you would write your name on the board because the TAs have to help too many people. So people write their name on the board as a public help queue. And they say, I need help. I'm sitting in seat whatever in the room. And then there's just a list. And the TAs or other student volunteers just go around and just grab people off the list and go and help them.

So this interface is very similar. You just put on a list and then anybody in the website, if they want, they can click and they can actually join in to your session and help. And that's the very basic idea of that. And I'm happy to take questions at this time or go into details. We haven't talked for a while on this. So that's the very basic idea. Yeah, it's very cool. And the progression of how you got there makes a lot of sense.

I have a project in a very similar space. It's an IDE environment for kids to code. And yeah, we provide a little chat interface, but they can only talk they can only talk to us. I'm just, you know, a little bit too scared of letting kids loose in a peer to peer way. Yeah. Yeah. I mean, we can talk about that. That is, that is one of the biggest concerns is that, right. Of like, if you just open this up to the world, you know, maybe all hell will break loose. Yeah. I guess for me in particular, my, I think one of the differences between our systems is that yours doesn't have users. So everyone's kind of anonymous, which at least in my head seems to like lower liability. But I guess another thing that I have that I imagine you don't is this system was built for use at a specific afterschool program that I was running at the time.

And, um, I just was, you know, terrified of a parent, you know, being upset that, you know, some kid or some person at some creeper on the internet was saying bad things to their child. Yeah. You know, man, these, these are very valid concerns. And I was, you know, I, I thought a lot about that in designing the system. And, you know, earlier on, you mentioned how the interface just looked really simple and, you know, and straightforward. It's like, oh, that's exactly what you expected to be. But actually, there was a lot of thought put into

how do you make this in a way that minimizes kind of bad behavior. And I'll go into a little bit. I mean, the first decision was to not have user accounts. Part of that was just an engineering thing, right? It's like, it's just a lot harder to run a site with user accounts than not. And so basically everyone has a pseudonym that is just assigned and stored in their local storage in their browser. So, you know, your user 2A5 or whatever. And the nice thing about that is you have some kind of persistent identity, but you

don't actually customize your username. And then the other thing about the thing is it's anonymous and it's also private. Right. So like a lot of the issues that comes up with with kind of trolling and other sorts of spamming and stuff. And it's because these forms are public. Right. If you have a public forum, you have to really moderate it. Otherwise, you know, all sorts of bad stuff can happen. So having a private thing, you know, lessens incentives for bad behavior, and also not having identity associated with it. And the other part is that there's you have the ability to just, you know, kick or block people, basically, if something bad's happening, you can just kick them out of your session, and then you can undo the changes they made. And with these lightweight things, I've looked at the chat logs and the server logs,

and so far, there's been very few instances of bad behavior. I mean, it does happen because it's internet, but it's fairly few. And also, you know, my site's not popular enough that like, it's not like Reddit, right? If it's on the order of Reddit size, then yes, you're going to have a lot of chaos because it's millions and millions of people, but it's still small enough that the people going on it are usually pretty honest and pretty earnest about learning and helping others. And they're not just on it to mess around. So I think that sort of combination helps it to maintain, you know, quality, you know, as much as possible right now. Yeah. Are there another, I guess,

I had a question, but I just want to say before I, I was going to, the question was are there other interesting experiments in the like leveraging humans in some creative way for scalability? Like one of yours that I thought was really fun was the Code Opticon project. So the way I think of that project is it scales a tutor. It allows one tutor to look at like the screens of a dozen students at the same time, which intuitively to me makes a lot of sense because as a teacher myself, you know, very, if you have a handful of students, it's so much more efficient use of your time than one-on-one

tutoring because chances are, you know, the only one student will need help at a given time. Yeah. I mean, that's exactly along the right line. So I mean, a lot of these projects have the same flavors of the CodeOpticon project a few years ago was prototyping, what does it mean to scale up one tutor, right? So the stuff I just described is still one on one, it's, it's scaling in the sense of you're taking advantage of the fact that there are maybe dozens of learners on the site at once. And, you know, rather than waiting for a tutor, which might not exist or might not come, you just post your question publicly and then let a volunteer crowd of people, whoever is available to join in, join

in. The CodeOpticon one requires a dedicated staff, right? So the idea behind that is you might use that in a course, like an online course or real in-person course, where you're like, I'm going to have a virtual office hours today at 3 to 4 p.m. Anybody who comes on the site during that time and asks questions, I'll be online and I can monitor a bunch of you because I have this special dashboard interface. And then I can kind of chat with you selectively. So that was kind of pushing the ideas research wise to how do you scale up tutors, the system itself isn't deployed,

it was a prototype, the code opting system is employed, just because I don't have the infrastructure currently to support tutors and accounts and all that stuff, right? Because then you have to like, create a tutor account. And there's just a whole other like workflow to, to get that in. But that, you know, but the proof of concept is there and you know given more given more resources that would be a great thing to have on the site yeah yeah it's to me it's a yeah it's a fascinating idea at the very least and i'd be curious to hear, like, from your knowledge of the greater learning at scale space, if there are other interesting experiments that you know of, even if they don't have to do with coding. Yeah, that's a great question. I mean, I think in the synchronous

space, the kind of real time space, there was not as much. And, you know, some of my colleagues from, it was this collaboration between UC San Diego, where I currently work, and you know, uh, some of my colleagues, um, from, uh, it was this collaboration between UC San Diego where I currently work and Carnegie Mellon. They have this thing called, uh, they have two systems, this thing called talk, talk about and peer studio. I can send you the links later. And the idea behind those systems is to, one of them tries to connect people on Google Hangouts to do video, synchronous video chats.

And the idea behind this talk about system, this was a few years ago, was that these discussion based courses, they wanted to emulate kind of a in-person kind of discussion based course, right? Because a lot of these MOOCs are very mechanical, you know, you have multiple choice questions, you might be reading, you might be doing some code. But what about for MOOCs in the social sciences or the humanities where, you know, you really want that experience of a small class of diverse viewpoints of people really interacting with each other. And, you know, their hypothesis is that if we structure these incentives, right, and make it bacon in the course, maybe we get better discussions because these people are taking the course from all over the world, right? It's not just a bunch of students at one university,

it's people from all over the world. So they did these experiments where they, they kind of made, I guess, optional assignments or course activities where they're like, you know, at one o'clock to two o'clock this, this hour, we're going to group these students, you know, we randomly pick students from different parts of the world. And you're, you know, the five of you are assigned to a discussion group. And then we'll give you some discussion prompts. And we're just, we'll record your discussion and see how that goes. And their other project, Peer Studio, was about peer grading. So there's quite a bit of work on having students and MOOCs grade each other's work

or give feedback on each other's work because again this this helps scale the the assessment or grading problem right because you can't have one instructor grade a thousand you know uh reports or you know writing prompts right if you imagine anything that's not auto graded right like if you have to do a design sketch for a design class or a write a paragraph for paragraph for some social sciences or humanities class, you want to try to have different students give feedback on them. And their whole system was structured around kind of doling out these bits of grading and distributing amongst the student population and then giving prompts and structured rubrics

so that they can do a better job in grading. So that's sort of the flavor of the work in terms of kind of multiple people connecting together in real time or near real time, but not as much as far as I know about the kind of one-on-one tutoring or the one-to-many tutoring that I've been doing. Interesting. Yeah yeah thanks for providing perspective that's interesting that it's less crowded than you then maybe i would think yeah and i think a part of it is um you know we can we can get into this later i mean part of it is just that you know i i kind of choose problems or areas to tackle where kind of my own technical expertise is at an advantage, right? So, I mean, the, the, the thing is no one,

not many people have done the stuff that I've, I've been investigating of this scaling up tutoring and, and especially with coding and stuff. And it's because the, you know, you do need quite a bit of kind of programming and technical expertise to build and scale these systems. So, you know, being able to do the engineering is actually, I think, a prerequisite for being able to explore a lot of these interesting research frontiers. So I've kind of picked problems kind of purposely where I had kind of a technical advantage in a way.

Yeah, I guess that makes a lot of sense. I guess now it's, I feel like it's easy to predict that learning programming at scale will definitely be like a more developed field than learning any other thing at scale, just because the people who want to teach programming at scale, like know how to scale. Yeah. And I think that what you bring up is an interesting blind spot too right so like you know i think a really fascinating problem is how do you scale up all these classes that require in-person discussions you know in the social sciences in the humanities in law you know these kind of case-based things i mean i am i don't think i'm personally well equipped to do that because i that's just not where my expertise lies but i think those have tremendous potential because you know a, a lot of the quote unquote criticism of learning a scale is that, oh, this is just for learning how to code or learning physics or learning very STEM kind of science and engineering topics that some people might find very dry. But

what about, you know, more humanistic things? What about, you know, the things where you need discussion and debate and all the very human to human messy human interactions i think scaling those are fascinating i just don't have a great sense of how i would personally do it yeah me neither um so uh to kind of jump to a separate topic uh one of uh your videos you talk about um the difference between research in i guess like the hard sciences versus research in technology. And I, yeah, I just like the way you described that in technology, research looks like building prototypes that point the way to the future. Yeah.

Yeah, I like that phrase a lot. So, and I guess like you kind of hinted that like one of the main differences between uh industry and academia because like it seems like they're similar because they're because in both we're building stuff we're building things unlike in science but it sounds like uh one of the distinctions that you pointed out is that in academia the time horizon is is a lot longer than in industry. Yeah. So I think that was a video I recorded once at the dog park, I think. It was a dog park one.

One of the few dog park vlogs. Yeah. So this is something that, you know, in a lot of the stuff we talked about, I mean, we started this hour talking about more traditional, I would say, traditional education and social sciences sort of research on studying people, right? So studying older adults, learning the code, doing surveys, doing interviews of conversational programmers in industry, doing surveys of non-Indian speakers. So those are more traditionally what you would think of as kind of social sciences research, right? Of understanding

how people work and why they work as such. But a lot of my work is on building technologies and prototyping technologies and the Python tutor system, this CodeOpticon thing, the live chat thing. Those things are all technology-based. And oftentimes, it's hard for people to kind of parse apart, you know, like, you know, what, where's the line between what it's meant by quote unquote research versus more developing a product, right? So a lot of these things you can imagine could go into products. And where is that line drawn? And one dimension to slice this, like you alluded to is via time dimension. So when I was at I think Microsoft Research, they had this, they have these papers or these blog posts. I mean, many companies think about this. But, you know, at Microsoft that we I worked there for a bit as an intern, as a visiting researcher, they had this chart somewhere. I mean, I'm sure there are many charts

like this in the business world. And it was like talking about how, you know, there was the, you know, six month horizon, three, six month, one year. That's like the product cycle, right? You have to ship something within the few months. We have to iterate. You know, the business, the pace of business is very fast, right? Especially in technology. So if you're a public company, you have quarterly earnings reports.

If you're a startup, you have to compete with other startups and technology moves fast. So things that you're thinking about three months, six months, one year out are more in the pure development. And then things that you think about two, three years out, people sometimes call this advanced development or advanced research or advanced R&D. They have different names. So basically, it's things that we want to productize in the next few years. And we know it's on the horizon, but we want to do more exploration and more prototyping for. And then there's the five years or plus out. And it's really hard to predict where the future will go in five years or more. But the hope is both in academia and universities and at companies that have invested in long-term research labs,

the idea is that we want to try to think about stuff that may be relevant five years or more in the future and try to prototype these sorts of possible futures, right? So kind of building possible futures and knowing that most of these possible futures might not come to fruition, but if we don't now seed the 10,000 possible futures, we won't actually have a future when we get to it. I think a criticism of the classic criticism of academia is that, you know, people work on projects that seemed like they were just very heady, not practical, but now they form the basis of all sorts of systems that are used every day by millions or billions of people. Yeah, that's a classic thing. And it's hard for people to see often because because it's, you know, if you were to pick a project at random, right? I mean, it's like the, you know, they do this in Congress all the time,

right? That, you know, they, they pick out these, you know, these science research grants, and they're like, Oh, why are they spending $300,000 on, on, you know, testing frog legs, or whatever, right? They pick out the most obscure thing. And it is true. I mean, if you pick out a project at random, then it might sound kind of ridiculous. And chances are that one project out of 10,000 is not going to affect the future in any noticeable way. But collectively, you know, some subset of those will move the field forward. And even the ones who don't might, the ideas might percolate into other ones that do so the effects are very indirect so i i totally see that so um if um not sure how to phrase this basically um clearly it works somehow like clearly enough papers uh enough research does good work that you know science moves forward

um but but you But there's some research that doesn't move any field forward. So I guess both from an internal perspective, like if I'm a researcher who wants to produce more meaningful work or if I'm someone who's working at the NSF and I want to write better grants, what are the strategies for like, you know,

just doing more meaningful research, finding better problems, I guess. Yeah. That's a, that's a big question. I mean, I, if I, I had the answer to that question, I'd be a lot, you know, I'd be a lot wealthier. I mean, if I answer that question, you would probably start a company off. I guess I kind of left off the context of that question. There are two projects that it seems like you were part of. One was the CDE project, which looks a lot like Docker.

And then this other project around opportunistic programming, which looks a lot like this this new startup called kite um and so it's like it's at least to me you definitely have uh like a lot of street cred in terms of being like ahead of the curve uh definitely ahead of industry so um but that's the context of why i'm asking you and not just like some random researcher yeah well thank you for that actually it's uh yeah it's it's funny because um you know one of the marks of it's interesting because, you know, one of the marks of, it's interesting because, you know, the life of a researcher is interesting because, you know, if you really are dedicated to it, that means that you don't really directly profit off of the inventions or the kind of discoveries you make, right? Because, I mean, you can because then what you do is you transition to being an entrepreneur, right? You start a company out of your PhD work or out of your research work, and people successfully navigate that. But if you kind of adopt a mindset of I want to always be trying to push forward on the next thing, then by its very nature, you are trying

to look for stuff five years out, and you're not doing the really hard work of commercializing it right so um i think i think a big part of how a lot of these projects came about was just kind of sniffing at real world demand and um and you know i think paul graham of y combinator a long time ago he was writing essays you know before y combinator about just if you hang out at, you know, if you're just hang out at, you know, places where people are, are inventing the future in a way, you know, so university research labs, you know, these, you know, in Silicon Valley and spaces where people are just on the forefront of doing stuff, you're just able to sniff out problems before, you know, most of the world does, right. And I think, you know,

how some of these early projects came about was because I observed, you know, graduate students, researchers, struggling with how to do programming, right, struggling with, you know, using the web, struggling with installing different environments on their computer to run their research projects and stuff, and then coming up with sort of prototype technical solutions for those kinds of things. But yeah, it's a really interesting progression. And also, I mean, the other meta point about this is that you don't really, you know, there's not really a, I forgot who did an interview about this, but kind of the impact of research is that it's oftentimes hard to prove a causal link, right?

I mean, obviously, a lot of ideas are in the air at the same time. There's this great book by Stephen Johnson, the science writer called Where Good Ideas Come From. It's a great book. And one of the theses in the book is that there are just good ideas in the air at any given time, and there will be people who latch onto them, right? So this idea of Docker or CD or, or, you know, reproducing a build environments is, you know, more than one person was coming up with this idea at that time. And some people like myself, we're thinking about it from an academic perspective, how do you help researchers? And then there are others who are more entrepreneurial in mind is how do we help sysadmins and DevOps folks do that? So yeah, I think that, I mean, I think in the end, it's very

hard to actually show that a certain project you pursue or certain papers you publish directly led to X. And I think that as researchers, we, you know, that bar is too high, right? We shouldn't really aim to say that, you know, whatever we do directly influences a product, because it's hard to prove that. And, and my kind of view of research, like I put in that dog park video is if you just seed the world with interesting ideas and openly share them, then, you know, that's just that's a net positive to society. And, you know, if other people pick it up, that's great. If they don't, then you just keep doing other stuff. Yeah. Interesting. I think that kind of describes how most researchers look at it.

Like, I guess a different, your Python tutor is kind of like different because you came up with the idea and you wrote papers about it. But then you also are pursuing like the actual idea. Yeah. It's a weird thing. Yeah. I mean, we can talk about it i mean it's that it's a i'm in this strange position of you know maintaining this project that's actually pretty popular and widely used and we have you know every day there's i would say there's almost i mean i can look at

the stats but on the order of around 10 000 users you know daily active users on the you know around 10 000 and they're executing you know each one is executing you know i don't know a few pieces of code 10 pieces so it's like on the order of 50 000 or 100 000 unique pieces of code that's being run per day so it's it's reasonably trafficked um you know it's not a giant production site or anything but um so yeah so on one hand it's python tutor is the site that's a service and on the other hand it's a platform for is the site that's a service. And on the other hand, it's a platform for trying to trying to explore different research ideas as well. So there's this dual role that I think is actually very rare in academia, because, you know, in academia, you're mostly building

one off prototypes or doing studies rather than actually maintaining a piece of software at the same time. So it is unique in that way. Yeah. The canonical example in my mind of someone who does research and then makes a company about it, it would be like the Google guys. They wrote the paper and then they left. But it sounds like that's not in your future. This Python tutor is kind of a one-off for you. Oh, yeah, yeah. This question of commercializing and everything. Yeah, I mean, I've gotten this question quite a few times. And one, I mean, partially, it's just, you know, where I put my mental efforts. And, you know, I'm kind of my head's more in the academic and the research and the academic game at this point. And also, the other thing is, I feel like, you know, ed tech or educational technology businesses are hard to build. And also, the other thing is, I feel like, you know, ed tech or educational technology

businesses are hard to build. I mean, there's a lot of challenges in building ed tech businesses I've kind of kept up with as a spectator. You know, you basically these kind of learn to code platforms, it's hard to get a lot of traction and also paid traction, right? I mean, I think the best quote, unquote, I don't know best, but the closest path for this, if I were to commercialize this technology, would be to actually create content around it. So, you know, creating mini courses, creating tutorials, creating screencasts.

You know, there are people who run these independent businesses. This guy, what's his name? Gary Bernhardt, who does the Ruby. He does a lot of Ruby stuff. Destroy All SoundCloud. He has a paid. Yeah, he has a Destroy All Soundplay. He has a paid, yeah, he has a Destroy All, so he has a paid section. And one of my colleagues, Michael Kennedy, who does Talk Python, he does a Python pod, very popular Python podcast.

And he has Python training courses and premium services on his site. So I can imagine running something very independent like that. But then, you know, that would just require me to go all in on that right and right now i'm not you know that's not really in the cards for me but you know if i could fork myself and you know have a parallel version of myself that would be great yeah um so a lot of your uh prototypes um i think like the word is scaffolding. They like, they take existing languages and then somehow provide this structure that make them, make them easier. So like Python tutor makes the existing Python environment easier.

You had another project that makes like version control, get kind of stuff easier. Like you have a lot of your projects take the existing environments that we're stuck with and makes them easier, which contrasts with my focus and, and like kind of the, the research that I follow, our focus is more understanding new languages and tools from the ground up.

So like to keep with the scaffolding metaphor, like if you, you take the building that exists as, and then you build scaffolding. And I think we're, we're like going off off into a new place and building new buildings from the foundation up. So I'd be curious to hear about why you chose scaffolding

versus foundation. Yeah, this is a great question. I think that one way to think about that is that I feel like that earlier on in my career, it's easier to get traction, I guess scaffolding, it's a very meta word, by thinking about how do we improve the world we currently have, right? It's like the world we currently have is kind of, you know, hacky, it's kind of put together with duct tape. So, so computer security research is a great example of this, or network security is a great example of this, you know, a lot of it is putting duct tape around our internet protocols, which are were put together with duct tape, you know, 40 years ago. So you know, a lot of the practical security applied security work is look, you know, our internet's so insecure, because these protocols were made so long ago. There's this, we need backwards compatibility.

We have to live with this. How do we make it better? And then there's the other end of people are doing, let's imagine designing Internet protocols from scratch. And how can we create a better Internet if, you know, if we could start over. And I think in the educational world, similar of, how do we give a better experience to people who are currently using languages? And then also, how do we make better languages or better user interfaces that are completely different and not constrained by what we currently have? So I think my current focus on the scaffolding, as you mentioned, is more of a pragmatic one in terms of, I just feel like it's easier to make kind of concrete traction that way. And also it doesn't require, I guess, as much upfront investment. So like a good example of

the other end that you're talking about is, you know, what Brett Victor and his lab are doing, with Dynamic Land and with, you know, where Alan Kay was involved in this before and the folks who are really trying to invent the future. Yeah, that's, that's who I was kind of alluding to. Yeah. Yeah, yeah. Yeah, we can talk about that. So, so, you know, that requires a tremendous amount of what one funding, right? Because you need people, I mean, you need to build a team to do that. And also, sustainability is a is a challenging thing, right? So, you know, the history of the Y Combinator Research Lab and, you know, and how Alan has gotten funding

from various companies to cobble together funding from SAP and from, you know, various other companies throughout the years. And, you know, the latest, I think, I feel like the latest status is

a lot of that has collapsed and Brett's lab is now, you know, he has his own lab and they're trying to get collapsed and brett's lab is now you know he has his own lab and they're trying to get fun they have some funding from you know wealthy silicon valley folks and it's it's hard to sustain those things and like i mean think about it right like you know you know alan k is a touring award winner and he's you know he's that you know he's one of the most famous people in the field and even he has to cobble together money from sources to do these sorts of really big, grand sort of things at scale and to be sustainable. And as someone like myself who just came out of graduate school, right, who no one knows who I am, it'd be very hard to imagine building that structure in that lab to do that.

And another fascination of mine is, you know, how do you build organizations to sustain research? And so, you know, the super long winded answer your question of this, the strategy I chose of building on scaffolding and enhancing existing tools is because that was the most practical path, at least for me now. And I hope, you know, if my career becomes successful and sustainable longer term to be able to support these bigger ambitions. So, you know, one direct path for academics to do that is if you can get more well known and you can get funding from a larger organization to build, you know, kind of a larger research group around this broader vision.

That's one way to do that. But as you can see, a lot of people when they're starting their careers, it's hard to get that traction, right? Because one of the only ways to raise money when you're younger is for entrepreneurship, right? So you see young people raising money in tech, but that money is going toward building a startup that is hopefully going to be profitable in some way, right? So and not toward this, you know, another example, there's another example with Chris Granger, right? Building the Eve project and the light table thing. That was, I think, the closest someone has gotten to, you know, take achieving liftoff on that, but that ultimately had to be shut down as well. Sorry to be so pessimistic, but.

No, I think that was a, I love following yeah your your line of your line of thinking there and i'm to be honest a little bit shocked to hear that you hope to get um to to the foundations one day but like you're you're kind of hoping to to start kind of with lower hanging fruit for practical reasons like i'm i'm like um kind of inspired by that you know i was i kind of figured that you'd say something like, ah, like programming is like never going to improve. Like this is all we can.

I thought you were going to say something even more pessimistic. Yeah, no, I mean, I think there are people who are doing the more greenfield sort of stuff, but they are all fairly supported by well-supported teams.

So one example is Scratch, right? The blocks-based environment for targeted for kids, right, the Scratch environment. That's very popular. That's run out of the MIT Media Lab by Mitch Resnick and folks. And they're relatively well-funded because he's very influential in the field.

And then there's other folks working on, I mean, I follow these sorts of future programming education projects, at least, you know, Microsoft has tried to do a few. And a lot of times the history of these is that some company, you know, some senior person at a company gets excited, right, about, you know, pick whatever company, like, oh, let's invest in this for a few years. And then we have some engineers, some researchers work on it, we build some prototypes, but ultimately, you know, attention kind of waxes and wanes. And then, you know, the project gets abandoned, or it gets open source, which is great. But then it's just hard to keep up the momentum. So these projects are few and far between their scratch, which really targets K 12. There's the racket series of programming languages in educational environments. So that was called Dr. Scheme. So it was a scheme-based kind of languages. There's a few other ones like, you know, Android App Inventor, which started at Google and now is being run at MIT as well. But I think everyone is facing the similar challenges of how do you find sustainable funding for this

sort of stuff? And we could talk a little about that toward the end as well but yeah that's i think everyone is fine because there's a lot of funding for you know venture-backed the traditional venture-backed startups where the prerogative is to grow and get customers and then you know hopefully get acquired or you know if you're really lucky go public as a company but there's not really a direct route for sustainable funding for trying to do this stuff that's five or ten years out or more yeah i i i'm totally with you i think yeah like a lot of a lot of people in in this world the people who um like brad and jonathan edwards and um alan k that i think they they complain about that a lot i think

have you talked to jonathan at all uh or for your podcast or just privately about have you talked to who did jonathan yeah yeah we've talked privately um and just like on the internet um he and brandon yeah right yeah and he has this like great gallery right he has this google slideshow of the you know history of programming interfaces yeah no it really is awesome that's i think that might be the first time i saw him um yeah jonathan was actually uh in san diego he was doing kind of a so after after he spent some time at alan k's lab he went down to san diego and he just you know i'm at uc san diego and he just came to visit our lab and come to our group meetings and chat with our grad students and he was really great and i actually first knew him when he was at MIT

many years ago as a research scientist. And he's been working, well, he was working on this thing called Subtext back then, but he has a new system he's working on. But he's been keeping this up for a while, but it is rare, right? And it's only him, right? It's only him.

It's kind of a one-man band, just like me. But it's rare, but it's possible. Yeah. Um, so I think we're kind of, kind of at the top of an hour. So I want to just check in if you have time for one or two more questions or if you have to run. Yeah. Yeah. I'm fine going for a, for a little bit more. Um, um, so just in the interest of time, I'll, I'll jump to a question that I'm particularly curious about. So a problem that I've been facing in my own research, I guess, to summarize without getting too much in depth, my research, it's similar in spirit to your research in that I'm interested in helping people learn programming, prototype things without having to be like

amazing programmers uh and i i build like prototypes and for me um like winning would be a prototype that i could eventually turn into if not a company then maybe a non-profit like something like i'll eventually build a ship that i'll then get to like you know right on um as opposed to just like abandoning each prototype for the next uh as yeah so like a con academy you know that sort of yeah you're like con academy or or like mitch resnick uh scratch like a project that turned or um like um like google but like less successful you know i don't expect to start the next people but like a project that turned into a business somehow without the billions of dollars yeah i imagine yeah anyways um so uh but one of the things so when i first started um my you could describe my

um attitude was was like i'm the first person to have any original ideas like i'm just gonna go sit with my computer and hack on things until i you know fix like figure it out and and now my attitude is is kind of the opposite like you know i like no ideas is original every idea i've ever thought was original there's like tons of papers that um that i that i could have read instead of wasting all this time on my own, like reinventing the wheel. And so now I probably, Aaron, have had to read too many papers. And I found your article on opportunistic paper reading, which gave a good strategy for staying in the middle somehow.

So I'd be curious to kind of go through that a little bit with you. Yeah, that's another hard problem. I mean, you bring up, I mean, one of the reasons why I was excited to do this podcast was that I would feel no matter what, it will give me new ideas for things to write about or to vlog about. So that is, you know, so we have one in this hour. I guess at least I have one. I hope you've gotten something useful out of this too.

But at least I've got something useful because, you know, I've gotten some great questions to think about. But yeah, this question of to what extent should you just start hacking and learning by personal experience? And to what extent should you step back and read what other people have done or study? Not just papers, right?

Study old systems. So I was listening to your podcast with Pete Hunt of React.js fame right before this one because I wanted to see how you were as an interviewer and everything. And it was interesting listening to him because I've heard his talks before, but I really like how he went into a lot of the history of programming systems and even before the web. He was saying a lot of these ideas from React, yes, they were around in older object-oriented

systems and Java, C++ and stuff, but we took them to the web and we innovated. So on one hand, studying old stuff is good, but on the other hand, you know, you shouldn't be discouraged, right? Because like the setting is so different, right? So like there's always this running joke that whenever people are doing something and, you know, a new programming language is research or programming tools or education tools, there's always going to be someone, you know, if you're giving a talk in the back of the room that raised their hand, like, Oh, we did this in the 1960s with Lisp machines.

Yep. Always going to be someone who's like, yeah, we did this with Lisp in the sixties or at Xerox PARC. There's always the Xerox PARC line. Yeah. Right. We're always doing this. Yeah. We did this in small talk at Xerox PARC. We did this in small talk at Xerox Park. We did this

in Lisp and MIT in the 60s. And while the seeds of the ideas might be there, the world is so different that you can make really interesting innovations. I think that my own personal, I guess my own hack on this is just to ask someone who is a bit more knowledgeable in the field and use that as a way of filtering. And that's also why I think I probably mentioned in a vlog or blog somewhere is that that's why being in some kind of a community of practice in an institution, it doesn't have to be a university. It could be things like, you know, the recurse center, which used to be called hacker school, or being in a startup incubator or, you know, wherever you are, if you're just in a community where you have mentors who have been through this before, or, you know, in your case, maybe if there are startup incubators or these spaces where people have thought a lot about commercializing educational technologies, they would just have insights like, yeah,

you know, my hunch is that my spidey sense is that this way may be good, or you should stop reading so many of these papers, and you should start hacking. Or, yeah, maybe you should take a step back from hacking and go talk to potential users. And like, I just think that, I mean, this goes full circle at the beginning of, I just think that in-person mentorship from someone who has done this before is so powerful. And, uh, and it's something that's really hard to replicate online, right? I mean, if you're kind of reading into the void, um, have you seen my, uh, just, sorry, another side.

So I, I do this podcast hour with my friend, Robert Ikeda, who he's running this website called mathpapa.com mathpapa, is a um it's a it's basically it's funny how we did this arrived at this independently but his site is basically like a python tutor for math so you write math equations and it tutors you and it steps you through breaking down the equations into parts and helping you scaffold really and solve equations and he decided to go a commercial route. So he started a business off that that's actually very sustainable. It's, it's two people right now. But they make I don't know how much they make, but they make enough to like, you know, so that

they don't need a job, right. And on our podcast, we talk like every month, and he talks a decent amount about the challenges of starting an educational technology business. So like, you know, someone like himself might be a good person to connect with, I can connect you with him offline. But you know, just any personal connections to just help you filter out this large swath information, because otherwise, it's just, it's super overwhelming, right? You're sitting in front of a terminal saying, you know, what should I do? Should I start typing code? Or should I read a bunch of papers or look at a bunch of other people's projects? It's the answer isn't really clear. Yeah. That, that does a good job of summing it up. Um, and I, I think your advice of finding an in-person community is makes a lot of

sense. Um, I'm wondering what, what approaches there are, if that's like not an option um you know uh i kind of kind of like your a lot of your work is like enabling you know learning programming at scale i wonder um what what sorts of ways we can scale like research you know mentorship it's kind of like learning learning at scale right yeah well i guess yeah and i'm thinking more like specifically like scaling research like uh no that's right okay i see so scaling kind of how do you that's right so you know research has been by definition kind of a non-scalable thing right you're doing it in a certain kind of institution you might need close mentorship you know you have this uh there's a lot of structure behind it, you know, how do we actually enable, you know, discoveries, you know, you know, there are billions of people in the world, there are many, you know, many,

many of these people have the ability to create really interesting discoveries or inventions. But they're not able to right now, of institutional barriers or just lack of access to the right information. I think that is a really, really cool. It's an interesting problem. One way that some of my colleagues have dealt with this, it's actually called a crowd research project. This is some of my colleagues from Stanford did this and it's called crowd research. And what they've basically done is they've basically done remote mentoring. So it's back to this mentoring thing. So they've structured research projects. I don't know the details.

But basically, they have grad students. Let's say grad students at Stanford or something. And you're like, I want help with my research project. I want help brainstorming and ideating and implementing. So they actually put out a call online. And they have students like undergrad students or even people who are not in school from all around the world kind of sign up to be on these kind of project teams. And they kind of have weekly meetings and try to usher it

kind of usher the teams together to formulate ideas and implement them. And they've actually, you know, their marker of success is literally, can we get these teams together to say, participate in research projects with people from, you know, from the US. And some of them might actually want to go to graduate school in the US or go work in the US later. And they're motivated to do this as kind of a credentialing, right? Like, if I can participate in an interesting project that gets published, then I can get recognition from the publication and from a recommendation letter from, you know, a Stanford professor. So in fact, the long story short is kind of closing out the hours. One of those students actually, from a university in India, ended up coming to San Diego as a as a master's student. And I didn't know about him before. But one of the crowd research grad students referred him to me and he said, oh, yeah, you're you're joining the faculty being a professor at UC San Diego. There's this guy that I was working with and he got into the graduate program here and he would love to chat with you. And I actually chatted with him right when I started two years ago.

And we work really well together. So we've we've we published one paper together, and we submitted a second one, he's, he's well poised to, you know, going, he's going to industry right now, he wants to work in industry for a few years, he has a job lined up, but he wants to come back to do a PhD later. And with these research publications under his belt, he's well poised to get into a very good place. And that all started because of this crowd research program, because otherwise, he wouldn't have been really discovered. So that was a happy ending. Wow. Yeah. That's, that sounds amazing.

Yeah. Very cool. Maybe, yeah, maybe now it's, it's inspiring me. I feel like I need to go, go find myself a mentor of some sort. Yeah. And that's, it's, it's a challenge. It's, it's really a challenge. And that's why, I mean, one way, you know, one thing I tell students about is, is looking into alumni networks, right? So we know from your university, or from, you know, you know, I think just in person networks are so key. And, and you're right, you know, before and now, you're this kind of mentoring at scale, or really this kind of scaling of research is, you know, you can reduce it to a problem of mentoring at scale, right? How do you scale up mentoring and, and finding people, uh, mentors who can point them in the right direction? That's, that's really hard, right? Because even the work that I've talked about isn't really mentoring. It's kind of helping you do some programming and do some more rote and simple tasks, but not like,

how can we take someone who is you know very talented and very uh hard working and motivated and connect them to the proper resources and guide them and that's i think that's like the you know a trillion dollar question that's the holy grail um cool well anyways i don't want to take up too much of your time i just want to thank you again for for taking the time to speak with me from a cold email i really appreciate it and great this is fun