The Joe Walker Podcast - Behind the Scenes of My Interview Research Process — Andy Matuschak Crashes My Crib

Episode Date: January 25, 2025

This episode is a little different: I’m the one being interviewed—and my interlocutor is Andy Matuschak, an independent applied researcher focused on "tools for thought" (ways to augment human intelligence). Andy founded and led Khan Academy’s Research and Development Lab, and prior to that, he was a senior engineer at Apple where he helped build iOS. I first discovered Andy’s work in 2021, and it was a game changer for me and the podcast. We spoke on the show in 2022. In 2024, I recorded some podcast interviews in the US, and had the pleasure of hanging out with Andy while I was in San Francisco. In October, Andy dropped by my place in SF to go behind the scenes of my podcast research process and interview me while I prepared for a conversation with Larry Summers. This is an unvarnished, unfiltered look at my tech stack and how I prepare for my interviews. I'm very grateful to Andy for suggesting the idea and for so thoughtfully drawing out my current strategies, tactics and tools. I support Andy's research. If you'd like to do so too, go here. If you'd like to access my interview research notes for podcast interviews, you can support to access here. Watch this episode on YouTube here: https://www.youtube.com/watch?v=QTI69kKeaC4  See omnystudio.com/listener for privacy information.

Hey everybody, just a quick note to introduce this episode. This time the tables are turned. My friend and former podcast guest, Andy Matuszik, interviews me about my research and preparation process for podcast interviews. Last year in 2024, I spent about three months in San Francisco where I recorded a few different podcast interviews. While I was there, Andy dropped by my place to observe me behind the scenes as I prepared for an upcoming interview. At that point, I was just a few days away from sitting down with Larry Summers and Andy got an inside look at my research and preparation process. We filmed the whole thing, three different cameras, screen recordings, the works. So this episode is really best experienced on video.

If you want to watch the video, you can find it on my YouTube channel. Just search Joseph Noel Walker in YouTube, or you can find it on X, formerly known as Twitter. My handle is at Joseph N Walker, or just head to my website, jnwpod.com. I'll include a link to the video there. Thanks and enjoy. So I'm here with my friend and key intellectual influence, Andy Matuszik. Andy is a former guest of the podcast. He was an iOS engineer at Apple. He led research and development at Khan Academy, and he's now an independent applied researcher thinking about tools for thought. And we've had some really fabulous conversations in San Francisco over the past couple of months. And today, Andy is going to be

looking under the hood of my preparation process for my podcast. So Andy, welcome. And is there any, I guess, context or introduction you'd like to add? Yeah, for sure. I mean, so the reason why I'm here is that you have an extremely unusual process for preparing for your podcast. In some way, you are a kind of athlete of learning along some really unusual axis. And because my research is all about learning, I am very curious about your process. And I want to ask you many, many questions about how it is that you go about learning. But just so your viewers have some idea of what we're getting out here. Tell me about at a high level, what preparing for the Taleb interview was like for you. That one was, that one was sui generis in that I also did about two months of work with a

private tutor where we're working through the bliston and huang introduction to probability textbook and some of the the wasm and all the statistics textbook i also did nassim's two-week real world risk institute course so that was i think like five hours every night for two weeks 10 10 business days. So that had an enormous amount of, I guess, like background or lead up prep before what I would call the intensive prep process. So the intensive prep process- That was not intense. I just want to highlight, it's 50 hours of course, just for the last course.

That was not intensive. Yeah. Plus the tutoring, not intensive. So what's the intensive prep, Joe? Yeah, actually, actually well before we do that maybe let me try and add up how long the non-intensive prep would be so 50 hours for the real world risk institute course and then the tutoring every saturday we might do like five hours we maybe did that eight times so already we're at about 90 hours of work um there's also

a sense in which i've been preparing for that podcast for the last eight years, reading his work and being an interested reader. But then the intensive part is one to three weeks before the interview where, you know, eight to 14 hours a day, you're reading as much of their stuff or stuff that's relevant to them as possible. Note taking practice, spaced repetition memory prompts, so every morning I'm doing prompts on the previous day's material. And for Taleb, that intensive process would have been, because I'd just come off the back of the interview with Boyd and Richardson in San Francisco, I didn't have too much turnaround time, maybe nine days of intensive work for Taleb.

So that's every day. So call it another 100 hours. Another 100 hours. Right. So this is what I'm so interested in here. This is a kind of extremely athletic learning. There's like 200 hours of very focused learning happening for this particular interview. And then of course you're prepped for the interview, which is tied into the second 100

hours where you're preparing your questions and the structure and so on. That's part of the learning process and it's also part of what makes the learning interesting because it's not... There are some other super learners that we could talk to who are just very attracted to the idea of like, I want to check all the boxes. I want to know probability and I want to know statistics and I want to know economics and so I'm going to work through all the textbooks. But the learning that you did is motivated by this particular conversation. You have this extremely concrete thing that you are trying to be able to do with great excellence.

So I'm really interested in digging into what those 200 hours look like. And, of course, then it's so interesting that then, well, you finish the 11 review, and then you have Fukuyama, and then you have Larry Summers. And so, like, these things just continue and compound. And so I'm also very interested in the relationships between these various sprints. So to begin, tell me about how you orient. So you've just booked the interview, and now, well, perhaps you've booked the interview a while ago, and now that particular person, Summers, say, is the next one up, and you're getting started for your first 100 hours.

There's an enormous pile of material that you could read. There are many textbooks that you could brush up on. So you have to do a lot of satisficing. You only have 200 hours to spend learning this stuff. So how do you think about orienting to it? Right. So one of the reasons I was very excited to speak with you today was I haven't articulated a lot of this stuff

and even though I'm I'm kind of living it I haven't like codified or documented this process so I'm kind of thinking through answers to these questions or articulating them for the first time in this conversation great um so I think I think there are a couple of things one is if it's an episode like the Taleb episode or to an extent the Boyd and Richardson episode where I'm not only trying to understand the person's work, but I'm also somewhat as a prerequisite trying to master a sufficient amount of the field that that work sits within. Then I'll firstly be speaking to people who might be able to advise me about a syllabus. So I did that with Taleb. I spoke with a friend who's a professor of statistics.

I spoke with some other friends who work in finance and have studied statistics and probabilities. You tell them that you're trying to master this subject. You tell them, I'm interviewing T 11. I want to be competent. Like what do you, what do you ask them? Both. Okay. Yeah.

Do you give them some sense of the constraints? Yes. The time constraints. What do you tell them? I could even pull out my messages, but it would be something like, it would be something like, okay,

so I've got two months and I want to get to like, um, a pretty decent undergraduate understanding of probability and statistics what should i focus on and you want that so that you can what understand the primary works yeah i'm trying to furnish myself with context for um for the interview and the guest's body of work. Right. So the concern is that like, so I've read a couple of Taleb's books. And the concern is that if I were to try to be in your shoes and prepare for this interview with Taleb,

with like, let's say, you know, two years of undergraduate mathematics, but 14 years ago, then reading his books, I would somehow be deficient. I wouldn't understand the theories deeply enough. There's something like you really need to be able to engage in order to ask the good questions

and participate with Taleb in this interview the way that you want to show up. Yeah. Okay. Yeah. Very interesting. Sorry, exactly. Oh, okay, cool, great.

Great. So you ask these people, they give you like a syllabus, they give you a reading list. Yeah. So they send you textbooks. You're not going to read the whole textbook, right? So how do you decide what to read?

Yeah. So again, you're usually consulting the same people and asking questions like, which are the most important parts of this textbook? Which are the most important chapters? Which are the most important topics to focus on? Is there anything that you think I can kind of leave out or omit in my work, given these time constraints? Okay. And with Taleb, you were working with a tutor. Okay. So how does the tutor relationship relate to the syllabus that's provided by these people?

Yeah. So shout out to Tom. So Tom has a physics background and it just so we belong to like this math group in sydney there's a signal chat and i posted a message in the group asking whether anyone was competent in physics sorry in probability and statistics and whether i could pay them to be a private tutor for one to two months and i got really lucky with Tom because Tom was looking independently to learn statistics anyway so we just worked through the the two textbooks together with that context about Tom it was somewhat more of a peer relationship than a mentor mentee relationship but he he he still knew a lot more than me and I leaned on him more than, I don't think he leaned on me at all, but I was leaning on him a lot.

But this is not a tutor relationship where he's presenting the material. There's a textbook. The textbook is the primary source of explication. You're both reading the textbook independently and you're coming together? Yes. Okay. So then we're coming together and we're solving basically the practice problems at the end of chapters on a whiteboard at a university on the weekend.

Okay, right. Have you done problem solving separately? Yeah. I mean, if you were good and did your homework, yes. Okay. And then you're additionally doing it together. Are you working on the same problem at the same time? Are you working on the same problem individually and then comparing?

A little bit of both. So sometimes we would jointly work on the same problem on the whiteboard and you kind of alternate if you're coming up against an obstacle in the problem or something, maybe the other person takes over, takes the whiteboard marker and they try for a few minutes. Other times we would separately work on them and then compare notes so it was quite fluid. Right, was there a time where you found some of the material very difficult and you were having trouble making any progress and you really had to lean on Tom? Yep. Okay can you tell me about that time? This happened often often because I haven't I guess I always had an aptitude for maths, but I

haven't had to use that skill much in the last few years. I think there was just like a lot of basic mathematical stuff that got exposed in the process. And so then I would, I mean, it was always pretty, pretty apparent that I was lacking some kind of prerequisite if the problem is just like not making sense. And then I would, I think Tom and I were both good at calling that out and then going back, I guess what you might call prerequisite chaining, where it's like, okay, I'm missing some foundational piece of knowledge here. Let's go back. Does it make sense? Oh, it still doesn't make sense. I'm missing an even more foundational piece of knowledge.

Right. And those are probably often pieces of knowledge that are not in the textbook because they're like, oh, you need to know something about geometric series that you were supposed to remember from algebra two and you just don't. Exactly. So then Tom is doing a mini lesson on the spot. Exactly. Exactly. Which is one of the beauties of being able to work with someone who's so knowledgeable because if you're on your own, then you're kind of just floundering around on the internet or using LLMs.

Right. Right. Okay. Okay. Great. So in these sessions, not only are you building some problem solving fluency by doing practice problems together, but you're also like acquiring net new information because you're getting some mini lessons or Tom is revealing some things to you.

And so how do you go about absorbing that? Are you taking notes while you're in these sessions with Tom? Are you doing a brain dump when you get home? So a couple of things. I'm taking notes on, I'm taking handwritten notes on a notepad. i've actually got some of my math tutoring notepads in the other room which i can grab if if we think it's worthwhile but then when i get home or sometimes in the tutoring session but more often when i get home i might add some memory prompts as well okay i should actually say i also separate this is another part of the Taleb story I worked with a second tutor which

was my aunt who is a really good high school math teacher and so I was kind of coming at it from both directions like Tom and I were doing some of the more advanced stuff and then I was like rebuilding from the foundations with my aunt oh very cool yeah okay um for the moment I'm going to jump ahead um so you did all this this preparatory work that's not about economics. It's not about Taleb's work. And then you turned to Taleb's work. You also had a textbook reading in economics as well. Did you have a textbook reading in economics?

Is that what you said as well? A textbook reading in economics? Was there part of your syllabus in economics or was it just probability statistics? Oh, just the probability. Do you have a background otherwise in economics? No. Okay. Because it's a lot of discussion about options trading and stuff like this. It kind of lost me. I don't really know about this stuff. Now, let's say we jump forward. You're engaging with Taleb's books and you run into this stuff about futures trading and why obviously you'd want to do this in this circumstance and he often doesn't really explain this stuff very thoroughly.

So are you doing kind of just-in-time learning while you're reading his books and what does that look like? Yeah, I do a lot of just-in-time learning. So, I think partly that looks like just Googling stuff or using LLMs to like, for example, quickly look up definitions or clarify some kind of concept. I also have the benefit of a bunch of friends and people I could message and pester with different questions. How do you decide when to do that versus using the alarm? Yeah, I think in the first instance, you should always look it up so you're not wasting someone's time. But if you're still not getting it or you feel like, yeah, maybe the question

you're asking yourself is ill-posed or not even wrong, and you're not sure why, then that's a good time to message a friend. Okay. So you have this fixed amount of time, these 200-ish hours. You have this kind of preparatory learning to do and probability and statistics, but also some earlier mathematical topics. And then also you want to engage with Taleb's primary works, as well as probably a bunch of primary works that surround that, like perhaps criticisms or other things that are affiliated. So how do you compose this reading list?

And then I'll be curious to learn about how you structure your time attacking it. You're constantly prioritizing, right? Right. I don't know how I compose it. I mean, I guess maybe this isn't a satisfying answer, but I feel like you just need to have a lot of taste about what are the most important topics? What are the topics that I need to spend the most time on in my prep? Because maybe they're not, maybe they're not as important, but they're more difficult to understand or comprehend.

What has he not spoken about much before? So I can create some kind of counterfactual value by focusing on these particular questions. Yeah. And then my reading list evolves constantly on a day-by-day basis as i'm in that kind of intensive research sprint because you're every day you're getting every hour you're getting more context and that is leading you to reconfigure what you think the priorities are yeah you can't make a serious plan in advance

no um okay it has to unfold so i'm very curious what it looks like to engage with these books the priorities are. Yeah, you can't make a serious plan in advance. No. Okay. It has to unfold. So I'm very curious what it looks like to engage with these books for you. It's a purposeful reading. You're generally interested in Taleb's ideas, but also there's this very specific thing that you have to do at a very specific time. So you're opening the first book. You've studied the probability and statistics, but you don't yet have a lot of background in Taleb. Book one, are you going in with a lot of questions? Are you

trying to get a lay of the land? How do you approach this? My approach to reading books could be a lot better. I should go back and- Okay, so tell me what you do. Yeah, okay. Okay. So it's not super systematic. So I guess there are always questions in one's head that are kind of floating around. Can you think of a a question you had like when you were opening book one of taled yeah so okay so because i'd read his stuff a lot over the years so what was interesting about this prep process was that for the the inserto like his popular books it was more me it was more me returning to those and trying to refresh my memory.

So maybe one question was, okay, here's something that is a category of thing that I often think. So I go in to the intensive research sprint with these kind of preconceptions and I'll have a note where I'm just like dumping thoughts and vague ideas for questions, links to different interviews the person's given or critiques of their work or other sources I think might be interesting. And so, and this is, this is, this is maybe a, I mean, for some guests, I've been accumulating these notes files for years. Right.

And probably for guests that are like wishlist guests. Do you have one of these notes for Taleb? Yeah, sure. You pulled up right now. For Taleb? Yeah. If you, if you, that are like wishlist guests. Exactly. Do you have one of these notes for Taleb? Yeah, sure. You pulled up right now. For Taleb? Yeah, if you got it. Yeah, yeah.

I mean, we could do... Okay, so let's... Yeah, if it's better to do some other guests, that's fine too. I just want to orient to what this super messy getting started... Okay, so there's actually also a... This is the third note app that you're opening right now. And I feel like there's something very true about this. There's something very true about this. It's also a testament to the fact that he's been on the wish list since I started the podcast.

Right. Back when I was using Apple Notes. You have Strata. Exactly. Exactly. Okay. Teleb questions. This is a long note.

Wow. Yeah. I mean, this note... This note... Okay. So this note is years old. Screenshots. I mean, tell me if you want me to slow down. No, this is great. This is great. Okay. So there's this one.

Stop. I'm just going to read aloud a little bit. Okay. But yeah, people should know I'm probably going to be embarrassed by about 95% of the stuff in here because it's just like the ramblings of a madman. Yeah. Let me pause and express gratitude for opening up your creative process. This takes vulnerability and I think if I can brag on you a minute, I think, I hope that you feel confident in showing me this behind the scenes vulnerable stuff because the interview with Taleb was so competent. That's the finished work. This is the messy work that went into it that we're seeing.

Absolutely. Okay. What do you think of Eugene Fama's arguments on bubbles? That seems like a very representative example of just like a very scrappy thing. I assume that like you ran into Eugene Fama's argument on bubbles at some point. And then what you just add this to this note? Yep. And I would have added that six years ago years ago at the time I would have thought that was a

good question but that is not the kind of question I would ask in an interview today why is that a bad question uh it's it's two a few reasons for I mean firstly I don't think the question is well phrased it's a bit too I mean what are you gin farmers arguments that should be you know that should be elaborated on in the preface to the question secondly it's just no longer interesting to me yeah fair enough okay so you're collecting these questions for like six years or whatever it is and um you're coming to these these books um do you refresh yourself on the questions? You just kind of hope they're in the back of your mind. So I will consult the questions every sort of other day or when I get to the kind of tail end of the intensive research sprint

and I'm trying to now compose the sheet of interview questions. You're consulting the questions every other day. That's really interesting. So what does that look like? You're scrolling through this? Yeah, I'm trying to now like compose the sheet of interview questions. You're consulting the questions every other day. That's really interesting. So like what does that look like? You're scrolling through this? Yeah, I'm skimming through it and I'm not going through it in detail, but it's a more serendipitous process. I'm just letting stuff kind of jump out.

Yeah, yeah. That makes a lot of sense. Okay. So you open book one. What does this look like? Are you reading linearly like front to back? Are you jumping around? You've read this book before, but maybe it's been a few years.

What is book one, by the way? Yeah, so Fit to Lead book one is filled by randomness. Okay. Okay, so the thing I really wanted to focus on, actually, let me step back and give some more context for the Taleb interview. So I'd kind of read his work over the course of several years. And what I wanted out of the interview was to focus on some of the stuff that I thought was more civilizationally important, but not well understood there's a real i guess like self-development kind of cult that's embraced to led's work um especially anti-fragile is his third popular book but i didn't really want to go for that angle i wanted to do an interview that was a bit more technical and so the best book for that was his technical inserto which is is called Statistical Consequences of Fat Tales.

And he actually gave me a copy of the book we caught up in April in New York prior to recording the interview in August. And one of the things that was really helpful about that meeting was he was directing me to the to what he thought was the most important stuff in his work right so firstly the non-technical chapter in that book or the chapter that um the chapter that kind of is like the best i think the best distillation of his entire body of work is it's chapter three or four it's a kind of more fulsome version of a lecture he gave at darwin college or something like that um and that that chapter was the first thing i started with in that book okay so how did you know that that chapter was going to be the most useful thing because nothing talib told me he told you okay great great um which well you don't you don't get that privilege for every interview but Sure, but you have various signals.

Is there another document like this document that's keeping a queue of what you should be reading next and what is at the top? I do all of those on paper. Okay, great. Love it. Sorry, I'll also do them in notes. Yeah, right. Okay, so here's like, this might be like a reading list.

It's an absolute mess. Sure, sure. But I mean, this is real though. Yeah. But I do a lot of my, yeah, I do my reading lists on paper. Okay. I don't know why I prefer paper. You're capturing this.

It's a little hard to rearrange stuff on paper. So I'm guessing these notes where like the numbers read in order 1, 7, 6, 2, 3, 4, 5, and that's because you're prioritizing. Yeah, right. Okay. And I noticed that only one of these is checked off, and that's because this is very live for you right now, and you're just getting started diving into it.

Okay. When did you write this list? Sometime last week. Cool. Coming back to back to for a minute um while you're reading that third chapter what are you doing are you reading digitally you're reading physically physically okay um are you holding pens are you holding post-it notes is there a notebook next to you okay cool book tags book tags critical uh so we might use them should i grab should i grab the book yes okay i'll be back this episode is brought to you by math academy whether you're attempting to learn the math necessary for a career pivot to machine learning looking to repair poor math

experiences from your original schooling or an adult learner with math envy who wants to become the shape rotator you always could have been, Math Academy will meet you where you're at and help you grow your knowledge of math faster than any other system I've seen. Math Academy is a fully automated online learning math platform. Husband and wife co-founders Jason and Sandy Roberts are math geeks who successfully taught calculus to fifth graders and they've distilled their evidence-based approach to learning into Math Academy. I first heard about Math Academy in a conversation with this episode's guest, Andy Matuszik, in San Francisco last year in 2024, about a month or so before this episode was recorded.

Sitting on a bench above Dolores Park, Andy and I were discussing the challenges involved in individual learning, and that's when Andy told me about what makes Math Academy so intriguing as a tool. As Andy put it to me, and I quote with his permission, I like Math Academy because it lets me make steady, reliable progress in little bits of stolen time. I can take 15 minutes here and 20 minutes there and be confident that the work will stack up into something meaningful, end quote. Part of that stacking up effect comes from Math Academy's focus on active learning. Rather than passively watching videos, you're solving problems, answering questions, and receiving immediate feedback. So you know you've understood

something before moving on. But there's something else going on here, which is Math Academy's personalized knowledge graph, and this is one of the features that I appreciate most. My research for podcast interviews gives me a strong perspective on why this is a great feature. When I'm diving into a new topic for the podcast, it's very hard to commit to a one to two week sprint if I don't have complete confidence that the curriculum is optimal for me. Pouring weeks of time and effort into a set of topics that aren't quite right, maybe some of them are obsolete, or maybe there are some prerequisites missing, is a big fear of mine. Math Academy is solving that for people who want to learn math. Math Academy has an enormous interconnected structure

of topics from fourth grade through university level mathematics. They have thousands of these topics. After an initial test, Math Academy's technology creates a personalized curriculum for you, mapping out the gaps in your knowledge and showing you the best curriculum for where you're at in your journey. But this is just the tip of the iceberg when it comes to the amount of pedagogical thinking they've poured into their platform. Math Academy is one of the only startups I know that has a manifesto that runs to 456 pages and counting. It's called the Math Academy Way. It lays out the cognitive science-backed principles behind Math Academy's learning philosophy. I recommend checking it out. When I read it, one of the things that struck me was that many of Math Academy's key pedagogical insights are the same ones I've converged on for my podcast preparation process,

including especially spaced repetition. So if you'd like to learn math at whatever level of understanding as quickly and effectively as possible, I cannot recommend Math Academy more highly. For a limited time, new customers can get their first month free. Check out Math Academy at mathacademy.com and use the code JoeWalker at checkout to get your first month free. That's mathacademy.com

and enter JoeWalker, that's J-O-E-W-A-L-K-E-R to get your first month free. All right. So just retrieved some things. Yeah, I really enjoy the talismanic pile. I love the antifragile. Is this a toilet paper in it? That is the kind of kitchen paper towels. Okay. That's probably, I read this book in san francisco in late 2016 and this this will be san franciscan tissue paper okay um so it's it's come full circle uh-huh um so but you began with this one yeah so this is so so these are all things i've kind of read over the years but i've sort of forgotten things and I didn't have a very good memory practice back then.

Yeah, sure. And so I know I'm needing to kind of return to them. And one of the points I was going to make earlier was you're going with a lot of preconceptions. So like a lot of this stuff is just going to be like not even wrong. when I'm coming back to these books or opening them for the first time is, what are all the ways I'm just like obviously wrong about questions that I currently think are really good? And they're the kind of questions where you go into the interview and you look like a fool of you. So you're seeking to... Some of the questions I already have that I think I'm excited about, like maybe these are my juicy questions, I'm looking to refute them.

You're looking to falsify. I'm looking to falsify. Not to falsify what you think the answer is, but to falsify the hypothesis that that question is good. Exactly. And I find it really interesting that like what your preparation is for here is that there's really two things that are related but distinct. You talk about looking like an idiot.

Probably to your audience, you wouldn't look like an idiot but you would look like an idiot to to lab and that would ruin the interview right exactly he would no longer take you seriously yeah i mean with you there there's there is also like i guess like a maybe a gelman amnesia that you sure protected by but you don't even with the audience you don't want to but that that is the important thing because if you if you ask if you lose the guest's respect by asking stuff that's just silly or a waste of time, that means that the rest of the interview isn't going to go as well. Right. The second thing, of course, is preparing so that it's interesting for your audience

and you understand well enough that you can be a translator. You know when you need to supply this extra information while you're also kind of playing the foil, like you're sort of acting as the non-expert insert that can interpret and communicate the person's ideas. Exactly. Right. And I guess the converse of not looking like an idiot with your questions is ideally if you're asking really penetrating questions that they haven't been asked before, then

not only do you not lose the guest's respect, but in fact, they kind of lean forward and they're like, okay, this is real. We're doing this. Absolutely. And to add to that, they give you better answers. Yeah, of course. And maybe even some of the best answers they could give, things that they haven't shared publicly before, things that are haven't shared publicly before,

things that are truly valuable to elicit. You have to morally deserve those answers by putting in the work. Yeah, I remember when you interviewed me, you asked me some incredibly deep cut question about something that I'd said in some, I don't even know, it was like some tertiary material, maybe about Deutsch and the beginning of infinity How it influenced me

And even years since I said that but you dug it up and you asked about it And that was the moment for me when I was like, okay Like I'm switching into a higher gear like this person has done their homework I'm I'm engaging it at a higher level. Yeah. Okay. So back to the Post-it stickies. I do want to see what you've done. Right. So I also found this. This was a reading list for the Taleb intensive prep.

But this is the second one. I think I like eventually. So I had one piece of paper and it got so messy and crowded that I started a fresh one. Right. So this is at the very end. And actually, what's interesting about this is how much fresh one. Right. So this is at the very end. And actually, what's interesting about this is... How much is incomplete? How much is incomplete.

Right, right. And I never feel satisfied with how much I've been able to prepare for. Right. This is something you've doomed yourself to. Like, because you're doing fixed time flexible scope, like, always, always things will fall off the list. Yeah. Yeah.

It's a sad sad painful truth um okay so this these are the remnants of some book tags it's like falling off and stuff yeah exactly um so we're kind of like returning to the the battlefield okay show me how you use them yeah so okay so say i'll pick okay so which so it was chapter three is the kind of distillation of, for anyone interested in Nassim's work who wants like a somewhat more technical without being overly technical distillation of just what I think is the most important part. Chapter three, a non-technical overview of the Darwin college lecture. Yeah. It says non-technical, but I'm seeing like hyperboloid level plots and stuff. It's quite, I don't know it's quite it's quite technical okay it's

it's the most non-technical chapter in this book so so what so how i will read this is um okay so the book's on the desk in front of me i'm in the computer yeah ideally also i have a digital version of the book open in some kind of llm or llm wrapper simultaneously so i've got two versions yeah and uh okay so so the wrapper i use at the moment is called ai drive there's probably 200 of these cool you could use I mean you could even use Claude projects. Does OpenAI have an equivalent now? I think Canvas is I think, but I think Canvas is more like an artifacts analog. So I'm not sure that they do. I think it would be more like you would attach the PDF and talk.

But I could be wrong. It's very hard to keep up. Yeah. So I use AI Drive. I settled on this for three reasons. Firstly, as opposed to say, cloud projects where you're uploading text files, here I'm uploading PDFs. A lot of the files I deal with the pdfs in the first

instance so it's just a more frictionless process i'm uploading a paper or a version of the book that i've a pdf i've downloaded from libgen okay whatever you can't upload pdfs into cloud projects i thought it was only text files yeah i think you can upload pdfs okay okay okay i might be mistaken the second reason i like ai drive is I can organize things with folders. And the third reason is I can select any model that I want to use to interrogate the file. And that's important because when I was uploading books, a lot of those files are too large for either Claude or ChatGPT. And so I'm using Gemini Pro.

And this gives me all of those options. So unfortunately you're not going to be able to see my chat history because they only introduced that feature a few weeks ago. It may not exist for Talib. So here I've uploaded all of the PDFs. Right. So you have the physical book here, you have PDFs here, and so you're encountering some technical explanation that you don't understand, and so you ask a question.

Exactly. How do you localize the question, by the way? You presumably can't say on page N, like probably the PDF page doesn't correspond. Oh, so, okay, yeah. So I mean, it depends on the PDF, whether it's like a photocopy or not. In my experience, they basically never correspond. So actually you know what I remember when I was doing this I think I actually checked to see they correspond and they do.

Oh cool. So I might say something like, so in this case I would say something like on page 22 where he mentions that in Mediocre Stan the probability of sampling higher than X twice in a row is greater than sampling higher than 2X once. I don't understand what this means. And then I'd put in some kind of prompt. Yeah, right. So you get an explication.

Yeah. Okay. That's really interesting. So I've got a kind of study buddy with me as I read. So I noticed like you are, this requires sophisticated reading on your part. This requires reading comprehension practices. You are monitoring your understanding. Is that just natural for you?

Is there a systematic way that you do it? Are there kinds of questions that you ask yourself as you read? It's pretty natural and pretty chaotic. Cha chaotic in the sense that probably you do that sometimes but not all the time and sometimes you don't understand something and you just let it go right yeah right yeah and and it's like okay do i i'm do i have does this does this seem and i'm not going to be able to understand everything and the constraints i have does this seem important not really okay i'll let this slip. Right. So, sometimes like the way that some people will achieve this is by writing

like a or something of each section in order to check their comprehension. You're kind of doing it in your head. Yeah. Well, so I will write some kind of note at the end of, so usually when I'm working with a book like this, the unit or increment I'm dealing with is a chapter. So at the end of, so as I'm, as I'm reading, I'm asking for help with things or looking things up. I don't understand. You're also using these. I'm also using these posters and these are for things that I want to turn into memory prompts at the end of the chapter

I don't want to do it on the spot because I don't want to interrupt my reading it's distracting it becomes too much of a chore do the colors mean anything no okay so it's only for mark is needing a memory prompt did you have some way to mark like i don't understand this i need to follow up uh do i do that i i usually so my habit is all if i think it's important i'll look it up on the spot right you don't let yourself go on until you know because because i'm always afraid that if i don't it's going to affect how well i understand the subsequent material okay Okay, cool. But you're probably also... Are you writing questions into notes while you're reading? Interview questions or... Questions for yourself, questions that need further investigation. And so you're like typing into your notes file while you're reading?

Yeah. I'm jotting things that jot out as particularly important. I'm kind of jotting them down. And does that go into the Nassim Taleb research mega note? No, that goes into Obsidian. Notes program number three. Great. Love it.

This is very true. Okay, we have a per book notes. Right. So this is kind of commentary. I see excerpts. Everything that I'm seeing on the screen right now is an excerpt. Yeah. This is not so much... So this is probably a bad example. Scroll up. Scroll up. Yep. For this chapter, your note for 16. For this chapter, it's fine to focus on the actual not rescaled estimates of death as 12 and

so on. Okay. So that's not a note to yourself about... That's kind of a note to myself. Okay, okay. Those are your words. Yeah. Okay, okay. Interesting. As to Levin...

Okay, this is it. Yeah, right. I'm confused as to which correct inter-interval time for wars generating deaths of more than 10 billion... So this question is like buried in this massive note and a pervasive problem. I think that many of us have who try to do this kind of process is like, then what, you know, you have this, what, many thousand word note, which is merely one of several dozen, many, yeah, 4,000 word note,

which is one of many dozen, presumably many thousand word notes. And there's like a question buried in the middle of a whole bunch of excerpts. What happens to that question? So usually, okay. So as I'm reading the chapter, I'm jotting down things like that. This question obviously wasn't properly resolved or maybe... Actually, what was it? It kind of was. What I'm wondering is, is that an active decision not to resolve it or is it more like it just kind of falls through the cracks? I think for this one, it's a little bit of both.

I kind of got an answer that I was happy with, but I was only like 85% sure. So I'm happy leaving this in because I might come back to it and it'll force me to get to the 100%. Okay. And your pages look pretty clean here. So I take it you're not writing in the book with a pen? No, I usually don't. I usually don't. Right. So what's the function of these excerpts? You have a lot of like just pull quotes. Yeah, so these are a bit lazy. So I think the context here is I'm under time constraints.

Normally I would try to synthesize and express things in terms of my understanding, which some of the other notes will demonstrate. But these excerpts are things that I think are very neat distillations of an important idea. Cool. So it's almost like a resource for you. You can easily access these distillations now that they're in this mega note. There's another way I use these. So, okay,'m jumping so no let me okay let me let me talk about so so at the end of at the end of a chapter so i've finished a chapter and it's kind of peppered with these yeah um these book tags now you're writing memory prompts yeah and the reason

i like these is firstly as we discussed it i don't interrupt my flow as i'm reading but also things that at the beginning of a chapter things that like seem like important memory prompts are superseded by maybe even more atomic ideas further into the chapter exactly exactly or maybe they're expressed in an in a later part of the chapter in a much clearer way and so i can just kind of like when I'll go back through, maybe only 70% of the tags will actually become prompts. So I've got to the end of the chapter and firstly, actually, I'll put in a lot of the sections that I think are important or that I want to turn into prompts.

You'll begin by extracting excerpts that correspond to where you put stickies next to it. Exactly. Okay, before you write the prompts, you'll just extract the verbatim. Yep. There'll also be, and then also just for posterity, I might extract other excerpts that I think are generally important but that I don't intend to turn into prompts. Are you doing that while you read or how do you know which ones to extract? I guess I just use the tags.

So you're also marking things that you want to extract but which don't necessarily want to become prompts? Yes. Okay. And I can remember which of which because I've just read the chapter. Yeah. Although this chapter, do you read it in one sitting? No, this one might have taken...

This is a meaty chapter. This one might have taken a This is a meaty chapter. This one might have taken a couple of sittings. Yeah, it's like... I think it's about 40 plus pages. Right. I'm kind of picturing you engaging with this over the course of maybe two days or something. Right. Yeah. Okay.

So at this point, I actually... Oops, why don't I just do that? I open two windows of the same node, and at the bottom of the node, I have my Mochi prompt. Great. And so on the left I'm looking at the content that corresponds to the prompt I want to write. And then I'm writing it here. Right, so you're kind of just moving through it. And then I do that chapter by chapter.

And you put the mochi prompts at the bottom of the note as opposed to interleaved because? Because, I mean, if you look how many, for some notes I have. Oh yeah, this is great, look at you. A lot of, this is all Andy's doing, you did this to me. Well, okay, okay. So I have a lot of mochi prompts and I think it's just kind of unwieldy if I intersperse them through the main notes.

Yeah, I've had that problem too. Honestly, it's bad both ways, you know. The challenge with this way is like it's very difficult to... It's like you have the original excerpt plus possible commentary up there and you have the mochi prompt down here and sometimes you want to kind of like see them together and it's hard to reconcile. But then as you say if you intersperse them, it's dissatisfying for the other reason. Yeah, and then it's like, do you want maybe two columns with the note on one side and the prompts on the other? Sounds like a translation.

Yeah, but that doesn't always work because sometimes the prompts connect to multiple pieces. They're synthetic. Yeah. Yeah. Yeah, I feel you uh what is this so you're using like some kind of obsidian plugin that extracts these yeah which you uh recommended to me great i mean so i recommended it to you without having used it just because i i saw that it kind of well i should say

it's it's crazy how this this plugin made the whole experience like disproportionately enjoyable. Yes. Right. So let me just like narrate for your viewers, cause it's very non-obvious. A thesis that I staked in the ground like five years ago was that I think it's just really a bad idea to be like jumping back and forth between like the place where you're writing long form prose and about a thing you're studying and the place where you're writing long-form prose about a thing you're studying and the place where you're writing memory prompts about the thing you're studying. It adds a lot of friction and dissociation.

And so I made a system that lets you interleave those things. Other people have also made such systems. You're using one that I have not used, but which I think I suggested to you. It's a great one, by the way. That's great to hear. And so this format is like, those are some kind of markdown heading? Yep.

Okay, right. So you can set it however you want. I've chosen C- H5. Yeah, H6, actually. Okay, I see. Just to visually differentiate it from the main headings in my text. Got it.

Okay, and then the following block becomes my text. Got it. Okay. And then the following block becomes the answer. The answer. Okay. Cool. Should we demonstrate? Multiple blocks? How does this work?

This? Okay. So the plugin doesn't work for images. So for images, what I do is I separately place the image in Obsidian and then I have to go and do it in Mochi. In Mochi. Yeah. Okay. Got it.

So I think one of the questions I wanted to ask you is that I think it's pretty non-obvious why memory prompts should be a big part of your process. You're sort of cramming. You're doing this in two weeks before the big thing. And memory prompts, they're big advantages. Naively, one might see them come over months and years, but they seem to really help you with your process. And I want to understand that. I think it's really interesting and non-obvious how even just in the span of like 10 days or something,

the memory prompts are really helping you. So tell me about how. Yeah, I can answer that question in two ways. The first is I want to be in a place where my knowledge is compounding and prompts that I've created for interviews years ago, I'm leveraging for future interviews. So it makes your study more rewarding because it doesn't feel like a one-off thing. It feels like something where you're growing yourself in a durable way that's compounding over time absolutely so even if it didn't help with that just one interview absolutely it would still be worth

doing because it would make you more enthusiastic about the studying and help in the future yep and when as i'm as i'm writing them i'm writing them with an eye to the future so when i was doing the boyden richardson interview i know that a lot that whole deck it's like four to five hundred prompts i'm going to be using for a future interview I do with someone in their field. Right. But I think in these kind of intensive sprints, the prompts help in two ways. And this is the kind of second way to answer your question. And the two ways they help with the intensive sprints are, firstly, it makes, because I'm retaining more information,

it makes the next day of the sprint even easier. So I'm compounding the information. It's really interesting and non-obvious that you'd notice that day over day. Right. And the thing that... I wasn't convinced that that would be true. And I noticed that actually it depends a lot on what I'm reading, whether it is true. When I'm in a situation like the one I think you're in, where it's difficult material,

it's new to me, it's outside of my field, that's what tends to make it most likely to be true. And the thing that makes it most obvious to me that it's true is I like to, when I'm doing a session like this, write the prompts and then at the end of the day do the prompts, like the same day, at the end of the session even. And it's kind of humbling how often I will not be able to summon the answer to a prompt that I just wrote like an hour or two ago. I have that feeling sometimes. Often, I should say. When that happens, it increases my confidence that this practice is helpful. It's very humbling. Yeah. Okay, it is hard to write good and effective memory prompts. But actually, so I'm not sure if we were coming back to this, but I should say the second reason I find the prompts super helpful.

Please do, please do. Is with these, you don't, when you're doing, and you would be able to empathize with this, but when you're doing solo research you don't get a lot of feedback and for me the final output is oh the fuck the feedback is is firstly the guest themselves in the interview and how they respond and then ultimately the audience after the interview is published but it can be quite a tedious lonely difficult process doing this like one to three weeks of long you know days of research yeah for sure by yourself and having a little ritual every morning

where i'm reviewing the prompts from the previous day and you know getting 90 plus of them correct or whatever is incredibly rewarding so it adds adds some kind of shorter feedback cycle to my prep process. So it's been incredibly motivating. Right. You can also see you're producing something. Yes. There's a pile that's growing. There's an output.

Yeah. I'm making progress. Right. Whereas this book, it retains its shape as you work your way through. Yeah. It gets little stickies on the side. Yeah, I resonate with that. I want to ask you about prompt writing. This has been an interesting question in my research.

It's just a thing that a lot of people who try to do this deal with. It's hard to write prompts about just general material as opposed to memorizing vocabulary words or whatever. Tell me about your challenges with that or if you even have them. Yeah. So, I mean, firstly, I should say everything I've learned about prompt writing I've learned from you, your writings online. So I think for me personally, the way I would articulate the most important overarching meta skill is to have empathy for your future self and to know what kind of prompt is going to be

trivial, easy to guess, what kind of prompt is going to be too ambiguous or confuse you. Yeah. And yeah, what kind of prompts are going to be most durably useful great okay and uh sometimes it's probably like hard to write a prompt for a particular thing yeah so so i think a few things that help are i mean firstly okay and, I'm articulating a lot of these principles for the first time here. But one rule of thumb I have is I won't write a prompt on something that I don't understand. Yeah, yeah. This is actually, I learned a lot from Peter Wozniak's 20 rules of effective space repetition,

flashcard writing or something like this. And his number one suggestion is understand before you memorize. Right. Yeah. Having said, okay, maybe there's like one little exception to that though, which is for sort of like definitional things, I might write prompts even if I don't fully understand all of the contours of the underlying concept. And it's still helpful because even if I'm kind of just like slowly, stochastically learning

this concept, I'm still picking up the language and it's making the eventual understanding easier. It also creates some feedback, right? Like if you have this review come up and like you can parrot the answer, but you can kind of feel that you're parroting and that you don't know what these particular words in the middle mean. It's like, it's an extra signal for you. It's like, okay, better track down those words. I mean, should we review some of the prompts I've been doing recently for Larry Simon? Yeah, let's just flip through a few.

Cool. Okay. What are the two basic stories of catch-up growth? And when this prompt is imported into Mochi with the kind of plugin that you're using for Obsidian, can you name that plugin? Probably people are going to ask. Yeah. What's it called? Mochi Cards Pro. Mochi Cards Pro. Great. Love it. Okay. When this is imported into Mochi, do you get some kind of context about when you say what are the two basic

stories of catch-up growth? My concern as a prompt writing coach might be like, whose two basic stories? There's probably lots of basic stories of catch-up growth did you get context as like this is what Halperin said I do often use that as a caveat and and the reason that caveats like that are important is you don't want to assume that this is correct or consensus. So it's important to associate certain ideas with certain people. I think here I just felt sufficiently confident that this like is the kind of... That it is universal. Okay, okay. But there... So is your answer that the plugin in fact does not supply any kind of built-in... Oh I see what you're asking. No. Okay should I quickly demonstrate how the plugin

works or how I use it? Well let's just go look at Mochi. Like what does it look like to review these in Mochi? So firstly in Mochi, I'll create a deck which becomes a sub folder in Larry Summers. So there's like a Bazel chats folder. Bazel who I've been speaking with about this. Right. And then I write a new prompt. And then I will push it through to Mochi. This is so nice. Yeah, I see.

Okay, right. But you do get context, so open that up again. So I see at the bottom here, Larry Semerow's Basil Chats. Do you see that when you review? Yeah, but it doesn't help me. Because I know I'm reviewing Promise for Larry Summers. Is that because when you sit down to review, you're choosing what you review. You're not doing interleave review.

Yeah, so each morning I'm reviewing. So say I'm in the intensive sprint for Larry as I am at the moment, each morning I'm really just reviewing Larry. Yeah. Right. You're not, you're not reviewing your, um, to lab prompts. No. Right. Um, so, so I have to ask him, I promise you. I mean, it would be like that where I'm preparing for several interviews concurrently. So. Uh, like you're preparing right now for summer simultaneously with Fukuyama.

Fukuyama. So some mornings last week when I sat down, I'm getting a mixture of prompts for both interviews. Okay. Right. But so although you write the prompts because you want the long-term memory benefit, you aren't necessarily always doing the long-term review of all of the prompts.

No. Right. No judgment, I promise. I notice you have 1,200 cards today. Can you tell us a little bit about... How do you relate to that emotionally? It's like meaningless to me. Meaningless to you. Okay, cool, cool. That's great.

Because I'm using Mochi in a very specific way and also I haven't like... This is bad, but I haven't taken the time to kind of go through and learn how to change these settings and optimize. I probably should get your advice at some point on how to use it better. Okay, but it doesn't bother you. No, I just ignore it. This bothers a lot of people. Okay.

It's a big problem for a lot of people. People feel judged. People feel overwhelmed. I mean, like, how are you going to review 1,200 cards? And, you know, the answer is 50 at a time for a few weeks. Okay, so I am curious how you reconcile this desire to have the long-term accumulation of knowledge on the one hand, with on the other hand the practice of not actually doing the reviews for older cards. Or when you're between projects, do you do them?

Yes. Okay. So you switch into a different mode. Exactly. But you've been in intensive mode for a couple of months straight now. Yeah, exactly. It's been just back-to-back sprints because of this trip. Right.

Okay. When I get back to Sydney, it'll be a morning practice where I'm getting a kind of mixture of different prompts. Cool. Yeah. Okay. What rate do you find yourself, like a card comes up and you're like, this is just a bad card and you suspect you can get rid of it?

Interesting question. pretty rarely. The more, the more common thing that happens is I will think, I will decide, okay, so maybe like maybe five to maybe five to 10% of cards after I've first written them, I'll be like, Oh, I need to revise that. That, that phrasing is convoluted or unclear. And maybe 1%, I'm like, this just isn't necessary or helpful altogether. Delayed. Cool. Okay. And when you have that observation about revising, do you do it on the spot?

Yeah. Okay. I have to. I have like a zero inbox policy with that because otherwise you forget. Yeah. The only, yeah. So one of the benefits of this plugin is in theory, have like a zero inbox policy with that because otherwise you forget and yeah the only yeah so so and so one of the benefits of this plugin is in theory you can so if i edit the original prompt

in here yeah and by the way the reason i which i think i i may have got from you originally the reason i like having the prompts in in here is that they're embedded with the kind of original knowledge so the underlying knowledge changes then i can edit the prompts as well and then the way this plugin works is if i um if i edit one of the prompts and push it through again it should change that prompt rather than adding a new one really it's a little bit finicky how uh well we can test i'm not sure i believe you it doesn't seem like the information is there that would allow that to be possible.

Okay, so we're looking at some kind of fuzzy patching. I would expect it to be inconsistent, I guess. Yeah, sometimes it works, sometimes it doesn't. I see. Okay, so this is BazelChat, so let's put in a test here and see if it works. No, it's just... No. It's created a new set. Yeah, it didn't remove the old one, interestingly.

Has it created a new set? Up here. Oh, here. So it added a new card for the modification that you made. It didn't duplicate all the others. So actually, no, it did. I'm sorry. So you're going to have to clean this up now.

Yeah. That's unfortunate. Yeah. So it's funny. That's a feature that I really, really value. Me too. And it is pur, you know. That's a feature that I really, really value. Me too. And it is purportedly a feature of this, but sometimes it works and sometimes it doesn't. It needs to do something else.

I had this same conversation with Jason Ben a few weeks ago and I was like, oh, it never works. It's really finicky. And he was like, okay, show me demo. And it worked perfectly. Like what we did just then worked. Yeah. I mean, I can see how they could have some heuristics that would make it work sometimes, but having designed one of these plugins, there's limits to that. It really needs to

add some kind of identifier somehow to the question that allows it to track the identity of that question over time. It shouldn't just duplicate everything, though. That's silly and avoidable, and I'm willing to believe it's just a bug. I really value... My system doesn't have a manual push-through operation. It just goes in the background. And so I really like being able to just jump into an old node and delete three prompts and add two new ones or whatever,

and just have it show up. But yeah, it's difficult to arrange. Okay, so you have these prompts. You're reading, what is that notebook for? The Rhodia notebook? This might have been some Taleb math stuff. And you're doing math in the notebook rather than on the computer because you want to be able to draw. You don't use an iPad?

No. I probably should. I've just never been introduced to it. So this is just math. Cool. Labs and comms. So now kind of moving through the process, like you've done some of this initial study with the tutor, you're reading the books, the book list is kind of ever-growing as we've seen. It does not actually get fully consumed. At some point here, you are starting also to accumulate

like real questions for the guest, I've seen references to like a Trello board okay, okay can you share some about like how do the how do the questions start getting accumulated in the structure? Yeah, so this is also a recommendation from you

I don't remember which again, it's been fantastic uh so these are the two plugins i use both recommended by andy are the mochi plugin and the the obsidian kanban plugin great love it so i i mean i think i think the software is a constraint here but like ideally i would have questions embedded in so i don't know maybe I've got a note on this paper that I'm intending to maybe it's giving me some context for the Larry interview

and I can add like a comment or something that connects to a question and then aggregate all the questions your log seek andam and Tana users are foaming at the mouth right now

to tell you about the many ways that this is possible if only you were to see the likeness. Right. Use yet a fourth note-taking software. Yeah. But we're not doing that. So what I'll do is, as a question occurs to me, I'll come into this Kanban board.

Cool. So we should tell people to plug in is Kanban. All right. It's Markdown backed. Interesting. Yeah. I'm curious about that format.

This episode is sponsored by Vanta. Trust isn't just earned, it's demanded. Whether you're a startup founder navigating your first audit or a seasoned security professional scaling your GRC program, proving your commitment to security has never been more critical or more complex.

That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across 35 frameworks like SOC 2 and ISO 27001, centralized security workflows, complete questionnaires up to five times faster, and proactively manage vendor risk. I have personal experience here. Startups I've worked at in the past have used Vanta and I've always found it reliable. Vanta can help you start or scale your security program

by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Dovetail, and Fireant who use Vanta to manage risk and prove security in real time. For a limited time, get $1,000 off Vanta at vanta.com slash joe. That's V-A-N-T-A dot com slash J-O-E for $1,000 off. What it means is that the naive way to do a Kanban board plugin

would be that when you make a Kanban board, it's just some proprietary format. It's like a JSON file or something. And you wouldn't be able to just copy it anywhere. I mean, it would be something that only that plug-in knows how to read. But Obsidian's ethos is these plain text, human readable, interchangeable files on disk that they're just plain text files um and so it is uh it's lovely that the the this kanban plugin is following that ethos okay so you have this kanban

board you're throwing clearly tons and tons of stuff into this kanban board more than you can possibly ask uh how how do you use it yeah so i have columns based on the different topics I want to ask about, and then I'll have a two-categorized column. Where do these columns come from? How do you know you want to ask about precautionary principle in AI? Yeah, so, I mean, leading into the interview, during the prep process, I'm forming ideas about what are the most important kind of things to ask the person and organizing those into themes and topics. Okay. So a different process that it sounds like you don't use would be you come up with lots of questions that seem interesting and then looking at the questions you say like, well, these questions all seem to have the same theme.

I'm going to label that theme. Yeah, I do that too. You do that also. Yeah. So that's why this column is crucial to categorize. So most things I just dump in there. So let's take, okay, so if the Larry Summers. Yeah, it's interesting to contrast the in-process one to the finished one.

Right, so here the to categorize column is biggest. Right, and you have this challenge now where it doesn't fit on the screen. It's actually several screen heights. Yeah, so I can actually, I actually change the width of these columns. The reason I haven't is because I want to see my categories. Yeah, right. It's just a limitation of the digital canvas. Yeah, it's not ideal.

Okay, so this right now is mostly a dumping ground. It's in your gathering. Absolutely, and these won't be perfectly formed questions. This right now is mostly, it's like a dumping ground. It's in your gathering. Absolutely. And these won't be perfectly formed questions. Sometimes there'll be like two or three versions of the same question. And I've forgotten that I'd put it in and I've put it in in like slightly different words. Or I ultimately realize the questions can be merged or collapsed. So at this point, it's like very messy.

Got it. the questions can be merged or collapsed. So at this point, it's like very messy. Got it. And you start organizing it more aggressively, like while you're still in the studying process, probably. Yeah, but most aggressively, like, you know, 80% of the sort of work on this common board would happen a day or two before the interview. Right.

Yeah, that makes sense. You kind of want to save up until you've kind of accumulated all the material you're going to have, and then you're prioritizing and arranging. The thing I really like about doing this in a Kanban board is, for an interview, the sequencing of the questions is really important, and the kind of narrative arc, or maybe you need to establish a certain premise before you can ask another question so the audience can follow along and the kanban board enables me to like reorganize questions right very easily and then uh yeah so then the final

uh how do i oh yeah so i can change it to markdown and then I'll copy and paste it into Google Docs at that point when it's in Google Docs I may share it with a collaborator sorry I may share it with people who I'm sharing the questions with to be like do you think these are good questions am I missing anything but also in in google docs i'll tidy up the formatting and um then print it out and take in physical questions for the interview so i could i could actually if you're interested i could show you what the formatting looks like for a final interview yeah i i think maybe i want to stick for the moment to kind of the studying, planning phase. I noticed that there's this prompt near the top that says,

to practice, loading more context into the preface of a question to get a better answer. And so this is a note to yourself? This is a note to myself. Right. And you're keeping in the two categorized things, so it's just kind of like always visible and near the top. Exactly. You have it highlighted, so it's really standing out for you

these are pulled in so they're standing up these are kind of meh uh yeah so there's two meta things and that's just because they're relevant to the so the second meta thing which is in bold is about how many questions i think i'll be able to ask in this interview this is only a 50 minute interview yeah yeah it's tight okay i see really tight uh much tighter than i'm used to and these are so these things obviously very strongly pertain to the kinds of questions i'll be crafting how many in this sequence and so i want to be reminded of them as i'm going into the place where the questions are um and the only way i can do that is making them cards on the Kanban board themselves. What's interesting about this to me is that you're not only learning about the domain,

you're also learning about the art of interviewing. And you're weaving those two learning processes together. Yeah. I have decks in Mochi for... Yeah, you were telling me about this. Show me Mochi decks for the art of interviewing. You have videography ones, of course. That's a small one.

But yeah, maybe we can see a little bit of interview technique. You can show the secrets. Don't talk unless you can. What are the three words? Improve upon silence. Improve the silence. Yeah, that's great. I think improve upon silence is actually a little more elegant. Me too.

That's why I naturally gravitated towards it. What is a very easy way to transform potentially any question into a more interesting one? Flip the sign. Actually, I think I've got it. So instead of asking what would you do in X situation? What wouldn't you do in X situation? Right. Okay. I need I think I need a better example there But right so this is this is a challenging kind of prompt to write because what is a very easy way you have an answer? In mind, but there are many very easy ways potentially

So as you get a better and better at interviewing you will probably learn more Easy ways to transform any question into a more interesting one absolutely this is a very flawed prompt yeah it's not a durable hard to write a prompt like this well exactly exactly and as i was writing it i was conscious of all the you know prompt writing best practices that i was i was violating but I was also cognizant of the fact that at least for the next few months, I'm going to know what the answer is and it's going to serve me.

Yeah, I was just thinking like a modeling prompt writing coach, what would I do with this? What would you do? Really, really difficult. So one thing that I might do is, so you want the prompt to have like an unambiguous right answer or for it to be a generative thing so a different way to phrase

this prompt is like name three easy ways to transform any question into a more interesting one and it can be a different answer every time or often when that answer comes up flip the sign it's an epiphany that occurs at a particular place and time in a particular

context in response to something. And so if I'm having some very general epiphany, but it was at a dinner with my friend Laura and we were talking about X and her suggestion about how to handle X was this specific piece of advice. I have some like that. So it's like, okay, where is it? Okay so like when running a loop around the Berkeley campus. Yeah, this is perfect. Yeah, exactly. That's exactly how I would handle it. It's a cue, but it's not a cue that gives away what the answer is.

The other thing I would do is practice applying it. Prompts that have you practice transforming questions in that way and just thinking to yourself about how it makes the question feel. Okay. So, just so I'm clear, practice writing better prompts? No, no. Given the negation insight, a prompt which says, use the get a more interesting question by negating tactic, flipping the sign, in order to improve this.

Oh, that's a nice one. Or you could list a few to apply it to. Okay. Yeah, I like that. I like that as general advice, that practice. The reason it might not be so applicable here is because it's, I mean, it's trivially easy to do. Because you're literally just flipping the sign. Yeah.

But it does make it more salient. You have to, that's right. And salience is something that we don't understand well enough yet. And it's something I'd really like to understand better. And so it has to not just be the task, but it also has to be reflection. So what is the effect of flipping the sign on this question? Evaluate it. Do you like it better? Is it worse? How does it feel? And then you're kind of engaging with it. Right. And maybe do it for a question where it really does transform it into a more interesting question and one where it doesn't so much. Yeah, yeah.

A more challenging version of that would be come up with a question which benefits enormously from flipping the sign to make it more interesting and you must provide one that you haven't provided before. Yeah, that's a great one. That's a nice one. I should write that down. Come up with a question that benefits enormously from flipping the sign. Make it one you haven't created before.

And this, by the way, is a general technique. So we listed a few things so uh there's just recall like what is the technique that i the insight i had at that particular moment um there's application practice applying the technique um and then there's um uh if it's a technique that wants to get applied in a particular situation or that like it doesn't always, you want cueing, you want practice about the cueing of the technique, and so this kind of thing, you can create a question like this that interrogates or supports the trigger or the cue

for the salience. This pattern works generally. Okay, great. So we were looking at this because we were talking about the meta layer that you're learning about interviewing. I can see all the super interesting questions in here. I'd love to spend a lot of time with you. Honestly, I'd love to benefit from your learnings here. And you're doing that simultaneously with learning about the material. At some point, the time runs out and you're interviewing the guest. And so one question I have is, has there been a moment when you were in an interview and you felt some kind of a failure?

Like, you know, you didn't study adequately the thing that you needed to study so i can think of two examples but they're somewhat different with the interview i did with katia carrico that's when i was still working full-time alongside the podcast and i did that on a very short trip with three other interviews including the danny kahneman one with very short i mean another thing i've been learning is how to how to travel effectively handle jet lag scheduling all of the logistics that go behind it um and that one i felt i felt like i i was really disappointed because i felt like it was an incredibly important interview and i didn't feel like I'd done enough prep for that one. Interesting.

I love that interview. Okay, tell me about what disappointed you. I just didn't feel like I understood her field in any deep sense. I didn't feel like I put in the hours that would have made me feel better. Okay. So a naive person could ask here, well, Joe, Andy liked this interview. So you're doing this incredibly intensive study in prep. Can your viewers tell when you fail? Can your viewers tell? I don't know.

I think some viewers, particularly discerning viewers, probably could. I guess there's a question of what failure means here. Right. Right. The bar here is kind of interesting because when you did that Carco interview, it was so significant because she had not done other interviews. And so it was kind of like the first long form interview that anyone got to hear from her about. And that alone has value, kind of irrespective of anything else. I guess one thing I'm wrestling with here is like, Joe, of course you don't have expertise

in her field. Like even if you did have the full 200 hours to devote to it, which sounds extraordinary. It sounds like a huge amount of work to prepare for a podcast guest. It is obviously inadequate for understanding a deep niche expertise like synthetic biology. People go to PhD programs for that. And so how is it possible to understand the field well enough, even with the extraordinary 200 hours? Right. How is it ever possible? Yeah, well, I mean, I think that's part of the project of the podcast, to work out the extent to which those kind of endeavors are possible.

There's some kind of satisficing thing where it's like you get far enough. I can see this in your interview with Taleb. I can see like a difference in your preparation, maybe it's because you've been thinking about those questions for so many years, where it's almost like you get to pose as not a peer, but maybe it's a really interested rising senior undergrad or something who's talking talking to the professor and can like kind of hold up the conversation as opposed to somebody who showed up for the podcast and was like you know I just I have some questions for you and and the way that manifests for me as a listener is in the

unprepared stuff where like he says something that you don't expect and you come back immediately with like an improvised follow-up or reply that requires that you understand his response thoroughly as it relates to the domain, and that you can generate a thoughtful or curious follow-up. It does indicate a different degree of expertise. Right. And so how did it manifest, when you were talking with Karuko, the not preparation? Yeah, I think. So the way, I mean, I knew ahead of time that I hadn't done enough prep. So the strategy I evolved going into that interview, like the day before or whatever, to deal with that was to focus more on her story.

And it was a justifiable strategy on the grounds that you had to. It's super interesting. Super interesting. And it was her first long-form podcast interview. So I could, you know, comfortably say that that was going to be the focus for those reasons, not because I hadn't done the prep. I see. So, yeah, but I just remember feeling like a deep sense of guilt and

shame that i hadn't hadn't been able to with this like precious opportunity i hadn't been able to put in as much work as i had wanted to um there was a moment in the steven wolfram interview where towards the end i asked him about physics and i kind of i asked I ask him a question that I didn't really understand myself but it had kind of been something I thought of and someone else thought it was a good idea and I hadn't even I mean there was no work behind the question and he kind of it was probably one of the few questions where his his reaction was genuinely disdain. Okay. And I remember feeling a sense of like ick with myself. And I think one of the rules I have is if you don't understand the question, don't ask it. That was like, but that was one little moment in an otherwise very good interview. Good thing it was near the end. Yeah. Right, right.

And so I hadn't done the work to even have the right to ask that question. I was kind of just like parroting the words. And it didn't feel right. And I'm not sure. But I mean, you can still get interesting answers that other people can appraiseise even if you're not really appraising it in real time but I think as a general heuristic I tend I have a yeah I have a rule of not asking anything that I'm I'm not like genuinely interested uh to hear the answer for and able to comprehend the answer able to comprehend the answer I mean those two are very highly correlated if you can't comprehend the, then you can't be in real conversation with it.

All you can do is nod. Thanks for that answer. Hopefully, some of it will benefit. And then ask your next question. You can't react to it. You can't ask follow-ups. You can't dig in at all. Exactly.

Right. Yeah, so you have to be able to comprehend the answer. Otherwise, it's fundamentally no longer a conversation. Right. You can't dig in. You can't react. Okay, that makes sense. So I notice that in your note system and in your emoji, everything is sorted into folders by the interview guest. You also have this aspiration that you've expressed to build your understanding of these various topics over time and have these conversations, I imagine, also build on each other so that what you're learning from one leads to another. I'm curious, have there already been instances of kind of breaking down the lines between these

folders, this strict hierarchical structure where these guests are kind of in conversation with each other? Kind of, not yet. I mean, this system is reasonably new. I don't necessarily mean it has to be in the system, but like in your process of doing the podcast. Well, okay, so this folder is actually called Cultural Evolution. Right, okay. And I already know that I'm going to be building on this for... So I use this for Boyd and Richardson.

I know I'm going to be using it for Cecilia Hayes, cultural gadgets, the Oxford based. Is it relevant for Fukuyama as well? Absolutely. Yeah. Absolutely. Yeah. So now you'll have to evolve your structure somehow to support this kind of cross cutting in learnings. do you have a sense

of how you want to do that no i don't cool no i was expecting to cross that bridge as i came to it yeah but yeah i'll be thinking about that more and more right do you have i mean do you have any kind of tips or suggestions in mind i I think it's an open research question. Obviously, I've written a lot about one way to do that is to break everything down into really small pieces, and that way they can easily be reused in multiple places and discover the structure that wants to exist before you know what it is.

The evergreen notes are atomic. Sure, or zettelkasten stuff. Yeah. There's lots of downsides of that. There's lots of ways in which that would work poorly with your process. I think it's not at all clear. That's why I asked.

So we've looked at your Kanban board. You also showed me earlier a photo of a physical version of one of these preparation boards. Can you pull that up? Yeah. So this is my hotel room before the Wolfram. So my instinctive reaction is this looks like a million times better to work with than the Obsidian version. I'm curious how you take that reaction.

I liked this. I like this a lot because I like operating in three dimensions. It's superior in many ways. The limitation here is I can't easily transport this stuff into Mochi. Yeah. But these are questions for the guest, right? So these are... Or no, these are your notes from the readings. Okay, so the yellow cards are like ideas or points. And then the pink post-its are questions for the guest. Interesting. Okay, right, right.

And I can see there's like some correspondence between the yellow card, it's like a point, and the pink card, the question that follows from it. And then the red sections, I see MKS, new kind of science, probably. Yeah. And... Oh, but also above, right. It's continued. Okay, so that's like a...

Dynamics of complex system. Okay, right, right. Yeah. Cool. Right, so the book notes and the questions for the guest here are interleaved. What about, how does prioritization and ordering and structure happen of the questions for the guest? So that... Do you take all the pink notes off and put them somewhere else?

Yeah. So this is, these ones on this mirror. Oh, okay. They're actually questions. And this is almost ones on this mirror. Oh, okay. Questions. And this is almost like a Kanban board. These are the categories. Yeah, I see. Okay, right.

So are you rewriting the pink questions onto the yellow stiches there? Yes. And the yellow ones here, so now the color coding system has changed on the mirror. The yellow ones here are my kind of final version of the draft question yeah i'm happy with the pig is the category yeah exactly so there's like one two three four five six seven categories cool eight categories um okay so uh do you need to get better at this process somehow and if so why like if the Teleb interview seemed like pretty good enough, like as a guest, as a listener, it seemed like pretty well informed.

I don't have a strong sense of like, well, it would be 10 times better if you were 10 times more informed. It's unclear to me. Like, is there something you're satisfied with in your learning that's that's a fantastic question so one of the realizations i've come to on this this trip and and in large part thanks to conversations with you is there's this growing bifurcation between the prep process and the podcast itself and you almost overextend yourself in the prep process and then people just see the tip of the iceberg in the interview right that's why this is so interesting yeah but also but i also i mean i don't i don't mean that in a self-congratulatory sense right well not only in a self-congratulatory sense but i also mean it in the sense that um like a lot of the stuff i do in

prep isn't actually ultimately useful in the interview. Yeah, sure. It's just you can't know in advance, right? You can't know in advance. Maybe you could argue that it helps you avoid asking dumb questions or the wrong questions. So it is actually helpful, but just not in a way that's visible. But I do genuinely think there's a large portion of prep that doesn't end up being useful. So the prep becomes a thing in and of itself and the interview is just like a forcing function for doing the prep and that's part of the reason why i'm you know uh wanting to share elements of the prep process as well

because i think if i'm benefiting so much from those and the value isn't always cashed out in the podcast other people should be able to get that value by looking at the prep process right uh so so i think one answer to your question is is i can keep doing more prep and better prep because the prep is valuable in and of itself apart from the podcast. I think another answer to your question is that the more efficient I can be with prep and the more I can compound knowledge between interviews, the more interviews I can do. Right. If you can do 200 hours worth of now studying in 100 hours, you could do more interviews. I see. Have a bigger impact, reach more people.

Yeah, that makes sense. Because I was kind of thinking about, well, what are the archetypes of interviewers, right? So there's low effort podcast interviewers, right? And you've left them behind and Carrico doesn't want to talk to them. So that's great. And now we have far away from that. There's the Joe tier. And then I was thinking, well, what would it mean to be more prepared? What would it mean to understand the topics way better? Like, is there an archetype like that? And one easy example is to look at, in research disciplines, most of the professional societies have, like, memoir-style interviews that they'll conduct. So the Association of Physicists, something like this, I don't remember its name, and the Mathematician Association, both of them publish these, like, you know, when a mathematician or physicist gets

to be 70, then, like, someone from the society will, like, sit down and interview them, but they, like, they are a physicist or a mathematician, so it's like a colleague doing the interview, and that really does have a different character, because they can go into the paper and, like, poke at, you know, where did that insight come from? Did that come from this person? You know, they know things that you couldn't possibly ever know. One thing I'm wondering is it's almost a continuum hypothesis. Is there a spot between the level of expertise that you acquire now with your current level of prep and all the way far away, a colleague physicist deep in the domain who's published

a hundred papers interviewing their more senior colleague colleague is there something that's meaningfully different from what you're doing now between those spots that is an interesting question i think so i mean i'm not sure what you me i'm not sure what you're contemplating my meaningfully different but but um i mean not not every person who needs to be interviewed, um, has the benefit of one of those colleagues to interview them. That's true. Yeah. Right. Right. Right. So over dinner, we were talking about this idea of like making accessible, uh, uh, interviewees who, who, who might otherwise not be, I mean, that, of course is the mission of those societies. Right, so who is not served by that kind of

society is a very interesting question. Yeah, yeah because there are so many people who they're just not going to take the time to write a book and yeah, so there's value to digging the information out of them. It doesn't really say much about what it would mean to be

more knowledgeable as the interviewer of those people because you're still not going to be their colleague, right? Right. But one model is that there are some people who go, some podcast hosts, often because they once worked in the field, they go really, their podcast is focused on one topic and so maybe finance or something. Maybe they worked in

finance. And so they're interviewing a bunch of people who finance things and they do some prep for each interview. So they learn more about finance and there's like an accumulation that like really cashes out over time. And like, maybe they're not going to be, you know, like an economist, but they are like extremely expert for a non-economist. And so they get to kind of a different strata. And you've pursued a breadth-y approach that seems to make it harder to do that. Yeah, it means that the knowledge isn't going to compound. I mean, clearly, we saw some ways in which it does.

Yeah, no, no, no. It will, but it will take much longer until the exponential curve starts to take off. Yeah, that's take off. Yeah. Yeah, that's super interesting. Yeah. Okay. And you mentioned this aspiration to share more of the prep process.

We last talked about this, I don't know, six weeks ago or something. Any thoughts? Maybe you can just articulate what that aspiration is as it sits in your head right now. What do you want to do? Right. So I think I'll publish at least three elements. One is the prompts. So I'll publish my Mochi deck for the Larry Summers interview.

I'll publish my Mochi deck for the Fukuyama interview. They're obviously still, there's only 80 prompts in this at the moment. But I can see, you know to lab 400 yep so uh so i'll publish i don't know how i mean i'd be interested to hear what you think but i'm not sure how useful they'll be to people but people might enjoy peeking under the hood of the process. I'll publish my Obsidian notes and then I'll publish video recordings of me working and preparing. So to give people like a fly on the wall kind of point of view of what that looks like. I've already recorded some of those for both Fukuyama and Larry Summers.

And another kind of video that I'll publish is calls with private tutors. So I've done a few of those again, both for Fukuyama and Summers, which are interesting. One kind of like mini challenge I'm contemplating there is, so obviously before I record one of those calls, I ask the person if it's okay if we record, and this is how I intend to use it. But making sure that they're not thinking

about the future potential audience and that they're still just focused on providing like a private tutoring call. It can sometimes change the incentives a little bit in a way that maybe cuts across the efficacy of the tutoring call. So I'm going to have to to think about all these interesting ways in which I've balanced those things.

I mean, the other interesting thing is there are a lot of people who say with the Larry Summers preparation, I've had the privilege of speaking to a bunch of people who would want to remain confidential. So there's always gonna be stuff I'm not able to publish. But I wanna give people different glimpses of the process

um I I've asked people what they would be interested in uh seeing and I was kind of surprised in that a lot uh a lot of the feedback I got was more about how I um how I think about interviewing and prepare uh for like the interview itself as opposed to the kind of background. That's in part because they don't know what your prep is. Exactly. That's what's so interesting about this like right like you go to a podcast app and there's like a bunch of podcasts that are interviews like with scientists or academics or whatever right and like they kind of look the same on the surface like here's Joe talking to some academics and like here's this other person talking to some academics.

But there's this totally different character underneath in the nature of the prep. And they both just look like podcasts with lists of episodes. It's understandable why people have that belief. You asked what might the use be of sharing the Mochi deck. My immediate high-level reaction

is that I think there's just a ton of value in the meta of this, of people having some sense of what it looks like for one person to seriously study and prepare in this topic where they're unfamiliar, because the nature of knowledge work and of studying, it's so invisible. It's people sitting alone with books and the things happening in their head, mostly. So I think that fact leads people to these beliefs about, well, this person's just really talented, or well, this person's just really brilliant. And like, you know, of course, maybe there's some elements of that. But I think those beliefs persist at the strength that they have, because it's invisible. So when I did, and we talked about this, like session with Dwarkesh, where we sat down and we read this quantum mechanics book together.

He had in the moment this great shock of like, wow, this is what it looks like to read a book and to ensure that one understands as one is reading what the things are saying to really interrogate the book, to be in conversation with the book. He found that really surprising and he reports post facto that it's really changed the way he prepares. And I like Dorkech, but I don't actually care that much about that one influence. I care way more about the fact that that video is very popular. And it's strange because it's a video of two guys reading a book together for two hours or something.

But there's tons of comments on the video saying things like, oh, I wish they'd taught this kind of thing in school. And to that's kind of sad this is so clearly not the right format for conveying yeah this kind of practice or knowledge it's two hours of just raw well your old live stream videos i think there might be three of them on youtube yeah something like that those were revolutionary when i found them. They like deeply informed some stuff I did at Forge in my last. Wow. And again, like what's interesting about that to me is that it's so low effort.

Like those were not special sessions. I did not think about like what could be especially clear, clarifying the show. And they're totally unedited. They're like three hours. So they're very hard to consume. And so it's easy to imagine improving upon this. So that's why I'm excited about you doing this.

Okay, I have one last question for you, Joe. You have these very charismatic sheets of paper on the desk that say, no semantics and concretize. Why are these here? What are these doing for you? These are reminders for the Fukuyama podcast. I mean, I could have just done these as prompts. Rem I think reminders for you, reminders for me. Yeah. Yeah. Um, so I've listened to a bunch of his interviews over the last few weeks. And I think like two of the main

failure modes is that interviewers tend to ask him about really abstract concepts like liberal democracy or whatever, without putting it in terms of like real world examples um and the second failure mode is a lot of the questions are of the sort what does this word mean to you or what's your definition of xyz and i find questions of that nature uh sort of you know of limited value i think for a couple of reasons one is like a philosophical reason it kind of um presupposes this like essentialism that definitions are somehow important and secondly almost by definition i guess pun intended you're not really creating any new knowledge.

You're just asking him for how he defines some word in a book and it's not a very useful question. So, I mean, I probably don't need to remind my, like I feel like I'm good enough as an interviewer that I'm not really going to fall into those traps. But I think it's just really important to me that I don't walk into those traps and that the interview is different

and that we talk about real stuff and specific stuff rather than just like, what's happening with liberal democracy? What do you mean by liberal democracy? Yeah. And you felt the need to write it out. Yeah. I'm reminded of Caro's Is There Desperation on this page. No. He has this wonderful practice of pinning index cards up next to his writing session that are like thematic reminders like this.

One of the ones that's most charismatic is about LBJ. Is there desperation on this page? He wants desperation on every page. That's awesome. You can see there's a whole collection. Okay, well thank you so much. This is super, super interesting, at least for me and hopefully for other people as well. Yeah, this has been a lot of fun. Thank you for coaxing these questions out of me and these answers out

of me and helping me to articulate all this. And really, I mean, all of this whole stack, this whole tech stack and these various practices have been influenced by you. So I'm very, very glad i discovered you and your work it's been a game changer for the podcast cool great thanks andy