Deep Questions with Cal Newport - Ep. 370: Deep Work in the Age of AI

Episode Date: September 15, 2025

A recent study called into question a core assumption about the generative AI revolution: that these tools, at the very least, will make us more productive. In this episode, Cal dives deep into the study and argues that when it comes to efforts that require deep work, AI can sometimes make things worse. He then answers listener questions and then takes a closer look at an article claiming that the lack of Wi-Fi in a West Virginia school is making their students dumber. Below are the questions covered in today's episode (with their timestamps). Get your questions answered by Cal! Here’s the link: bit.ly/3U3sTvoVideo from today’s episode:  youtube.com/calnewportmediaDeep Dive: Deep Work in the Age of AI [0:38]Can ChatGPT chats act as single purpose notebooks for projects? [26:16]When should I apply active recall? [29:34]Do you have insights into AI’s environmental impact? [31:16]What would you do if you left academia? [38:40]CASE STUDY: Using strategies to develop a Deep Life [43:04]CAL REACTS: Does lack of Wi-Fi make students dumber? [53:09W]Links:Buy Cal’s latest book, “Slow Productivity” at calnewport.com/slowGet a signed copy of Cal’s “Slow Productivity” at peoplesbooktakoma.com/event/cal-newport/Cal’s monthly book directory: bramses.notion.site/059db2641def4a88988b4d2cee4657ba?metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/theatlantic.com/economy/archive/2025/09/ai-bubble-us-economy/684128/marginalrevolution.com/marginalrevolution/2025/08/the-school-without-wifi.htmleducationrecoveryscorecard.org/wp-content/uploads/2025/02/report_WV_5401140_pocahontas-county-schools.pdfThanks to our Sponsors: This show is sponsored by BetterHelp:betterhelp.com/deepquestionscozyearth.com/deepshopify.com/deepmybodytutor.comThanks to Jesse Miller for production, Jay Kerstens for the intro music, and Mark Miles for mastering. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

In recent weeks, there has been a lot of turmoil surrounding AI technology. It increasingly seems now like those grand promises of superintelligence or AI systems automating most of the economy. These grand promises are probably not going to come true. But what about the more practical promise? The one that AI tools are going to make knowledge workers more productive, especially if you do something like computer programming for which AI is well suited. That's still true, right? Well, the answer turns out to be complicated. A recent study about AI productivity yield a completely

unexpected result. It created a glitch in the matrix, so to speak, that leads to some deeper truths about this technology and its potential role in our work. Today, I want to explore all of this. If you're at all interested in how AI might impact your job and then you're future. This is an episode you won't want to miss. As always, I'm Cal Newport, and this is Deep Questions. Today's episode, a glitch in the AI matrix. All right, so I preview there was a study that caught a lot of people off guard. I want to load this on the screen here for those who are watching instead of just listening. This study came out in July of 2025. It was produced by a nonprofit called METR. That's often pronounced meter.

It's a nonprofit that does evaluation of AI and its capabilities. They do high quality studies. They don't have any particular bias. AI companies often cite their results when they find them to be useful or impressive. So this is sort of a neutral body that does these reports. Here's the title of their July report that caused a bit of a surprise. Measuring the impact of early 2025 AI on experienced open source developer productivity. So what did they do? I'm actually going to read from the methodology section because I

think it's important to understand exactly what they were testing here. So let's read here together. This is from the paper. To directly measure the real world impact of AI tools on software development, we recruited 16 experienced developers from large open source repositories that they've contributed to for multiple years. Developers provide list of real issues that. that would be valuable to the repository, bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on that particular issue. When AI is allowed, developers can use any tool they choose, primarily Cursor Pro with Claude 3.5 or 3.7 Sonet, which were the frontier models of the time of the study. When disallowed, they work without generative AI assistance.

complete these tasks, which average about two hours each while recording their screens, then self-report total implementation time they need it. All right, so that's the setup. It's a very elegant experiment. Here's developers working on the normal stuff they develop and issue by issue. Hey, for this issue, I want you to use AI. For this issue, you don't. For this issue, you don't use AI, you do, right?

So you get this nice, randomized control of all the developers are both using AI and not using AI in a randomly selected manner. The simple thing they didn't want to figure out is how long did it take to do these tasks when you were using AI versus not. Now, here is where the conventional wisdom says computer programming is something that language models do well, so of course it makes productivity go up. I'm going to load up on the screen here the core chart of this whole paper. So if you're looking at this, what you're seeing right now is a bunch of green dots. So what these are showing, the things you can see, the data points you can see in. on my screen right now is predictions from various people about the increases of productivity

that they thought AI would give them. So if you'll notice there's a line in the middle here, that's neutral. So a data point on that line means AI makes no difference. If a data point is below that line, it means AI slows you down. If it's above it, it means AI speeds you up. So the first point is from economic experts. They ask economic experts, how much more productive will this make these programmers in this experiment? And on average, they said around a 40% speed up. They asked machine learning experts, the people who knew the technology.

They were similar. Yeah, it should be about a 40% speed up. They asked the developers themselves during the study. Like, hey, how much more productive do you think this is making you? And they were like, yeah, between 20 and 30% more productive. And when they asked them after the study was over, they were around the same place, like a little bit over 20. What was actually measured?

I'll scroll up to show you that result. The observed result is, on average, they were about 20% slower than the people not using AI. So the AI tasks were slower than the tasks in which the programmers did not use AI. This was an unexpected result. They thought they would just be measuring how much more productive AI made these programmers. they were unexpected to see that it makes them slower. So this is a bit of a mystery. And it's one we're going to look into because it gets to something core about AI and doing knowledge work.

Why did programmers, despite their predictions, become less productive when they used AI? All right. So if we want to solve this particular productivity paradox, I think what we need to do next is understand what type of work are these computer programmers doing? We've got to get specific here. Now, I'm going to use a term that most of my audience is familiar with. Computer programming handling these tasks,

which are a combination I looked into more deeply, but it's a combination of creating original code or fixing code. They require what I call deep work. Deep work, as you know, is a task that require you to focus without distraction on something that's cognitively demanding. So deep work is when you give your full attention to something that is demanding and you try to keep your attention on it as intensely as possible.

I sort of wrote the book and it sort of I wrote the book on it. I coined the term. It's been 10 years, Jesse. Not hard to believe? Yeah. In January, it's going to be the 10 year anniversary of deep work. And I guess I'm going to follow through with the promise I made my readers that if this book is still selling after 10 years, face tattoo. Boom.

Like Mike Tyson. Like Mike Tyson. Deep work right across the forehead. Ryan Holiday put his book title on his forearm. It's going to one up him right across the face. That's what a real committed author does. So anyways, here's the type of work we're talking about. We talk about computer programs, deep work, right?

It's something that requires focus. You're creating something complicated from scratch using your brain. It's cognitive demanding, and it benefits from being able to give it undistracted focus. In the modern knowledge economy, the core argument I make in my book on deep work is that this is the effort that creates most of the stuff that's actually valuable to the marketplace. All the other things we do, the emails, the meetings, the filling out the forms, the submitting the things, the tickets to IT, the making the PowerPoint presentations. That's basically all efforts that support deep work. The stuff we have to do to keep the lights on and tell people about what we did, but you can't run a profitable company just off of

email meetings and PowerPoints. Ultimately, someone has to actually create something valuable using their brain, and that requires deep work. So in this study, we had programmers doing deep work who used AI to help them, and they thought that would make their deep work more productive, but in reality, it made things worse. Great. So now we're narrowing in on the answer to this paradox. So we know what these programmers are doing deep work, and we know that AI wasn't helping them.

So let's narrow in this question even farther. How specifically were these programmers integrating the AI tools they were using into their deep work efforts? according to the study authors, the programmers were working interactively with these tools. So what they were doing was a lot of, can you produce me this code, I'll look at this code, hey, can you fix this about it, or this is not working? So kind of back and forth like that, or here's some code, you see any mistakes, great, then the programmer would run it. And be like, it's still not quite working right. Can you try to fix it here?

So it was a very interactive loop. They were going back and forth, having code produced, checking the code, having the code, having the, AI check the code, looking at what it sent back. So think of it as like a back and forth interaction. Here is how, let me see, or here is how the paper itself describes this. Let's see here. Okay, here we go. Here's text from the actual paper about this collaboration.

When allowed to use AI, developers spend a smaller proportion of their time actively coding and reading, searching for information. Instead, they spend time reviewing AI outputs, prompting AI systems. and waiting for AI generations. Interestingly, they also spend a somewhat higher proportion of their time idle where their screen recording doesn't show any activity. So the non-AI group, when they're working on a task without AI is more time makes sense, like actually writing code. And the AI group is spending more time prompting, asking for code, asking for it the check code. So it's more of this like interactive loop.

I'm going to give a name to this. Let's call this cybernetic collaboration, because these programmers, are collaborating on their deep work with a computer. So it's cybernetic and they're collaborating. Let's call it cybernetic collaboration. They're basically trying to split the cognitive effort of producing this code or fixing this code between them and this digital mind. Now intuitively, cybernetic collaboration should make you more productive. Why not, right?

Like you don't have to do as much deep work anymore because you're offloading some of it to a machine and machines are fast and machines are precise. and hey, this seems like you've just made things more productive. But, of course, that didn't happen. So now we've really narrowed in this paradox. They're doing deep work. They're collaborating interactively, doing cybernet collaboration with the AI while they're doing deep work. And that is not returning more productivity. It's still taking them longer to produce stuff than when they weren't using AI at all.

So let's narrow in even farther. And let's step back now and ask the question, what role does collaboration play? in deep work? Is that something that you can do with multiple minds? Now, this is actually an important question because a lot of people who had a glancing encounter with my book on deep work assumed that it had to be solitary. But it doesn't actually. I know a lot about collaborative deep work because this was one of the core skills I had

mastered as a theoretical computer scientist. Like this is, if anyone knows how to do collaborative deep work, it's scientists that do math theory like me. I learned how to do collaborative deep work first at MIT when I was doing my doctoral work and then refine these skills as faculty, computer science faculty at Georgetown. I write in my book Deep Work about how to do it successfully collaboratively. So yes, deep work can be done collaboratively. But how is it done successfully? If you want to do deep work with someone else, if you want to collaborate in practice, if you're hanging out with mathematicians, how do we do collaborative deep work?

successfully, well, here's the thing. Talk to any professional thinker who does this, they'll say the same. The reason why collaboration, I guess I just say, the way you make collaboration help with deep work is that you use the presence of other people to increase the intensity and duration of your focus. So the underlying formula, focus produces results, deeper focus produces better results. that formula reigns supreme. So when I would sit down with my frequent collaborators to work on some sort of theorem or some sort of paper working on, the whole game was trying to get even more intense focus. Here's how this works.

If I'm sitting at a whiteboard with two other mathematicians, a couple things happen. One, I'm going to maintain my focus on this problem much longer than if I'm just alone looking at a notebook. Why? Because if I'm alone, there's very little penalty for me allowing my attention to wander. Ah, this is hard. Let me just let my attention wander. Let me let some of the steam out of the metaphorical steam engine here.

Let me go check something else. Nothing bad happens if I do. It's just me trying to contain myself. If I'm at a whiteboard with two other mathematicians, however, and I let my attention wander, I lose a threat of thought. And then what do I have to do a couple minutes later? Something that's kind of embarrassing. I have to like, hold on, hold on.

Go back. I miss what you guys were just saying here because I let my mind wander. Everyone has to stop. everyone has to go back. So the social pressure of keeping up means that you keep your intensity longer. The other thing that happens when you're working with other people is it pushes your intensity of concentration deeper, right? Because you're sitting there trying to understand this. Someone has a breakthrough like, okay, hear me out. And they go up to the whiteboard and they start working on some sort of like complicated simplification of an equation or some sort

of graph construction. And you're like, man, I got to really lock in the follow this. Right? Someone is trying to download something complicated. They just had an insight about to my brain and the only way to get there is to like really lock in and focus. So it's a focus accelerator. That is what you get when you're doing deep work. I think it's off there now, but Jesse, that whiteboard we had here in the HQ for, I used to bring, like during the pandemic, starting the pandemic, he used to bring my collaborators from Hopkins and Georgetown, they'd come here and we'd work on the whiteboard. Yeah. I don't know if it's still up there or not, but like the last thing that there was, so we would sit at that literal whiteboard and the last thing that was up there for a couple years, we actually won an award for that paper.

So maybe I should have kept that. It's a good paper. But yeah, that's what we do. So there's other advantages to work with other people, but this is like one of the big things you get out of deep work, if you collaborate right,

is that it makes you focus harder and focus longer. So again, there's this underlying equation. Intensity of focus times time is how much you produce when you're doing cognitively demanding work. That still reigns.

So collaboration, all you're doing, if you're doing it right on hard things is getting more focus. You're squeezing more focus out of it. I call that the whiteboard effect my book. Go back to Cybernetic collaboration. That's not what these programmers are doing.

They are using collaboration with AI to reduce the amount of intensity of focus they have to experience. To get those breaks, you produce it, I'll look at it. It's easier for me to try to debug what you did that was broken than it is for me to have to produce it from scratch. It's easier for me to have these nice breaks while I'm waiting for the AI model to generate the code I'm going to look at, which can take. a little while.

We know this, Jesse, because we get this question all the time now. What should I do while waiting for the AI model to produce my code? Like, we only put it on the show like once, but we get this question a lot. So that gives you a break. So it makes the experience more pleasant. There's actually the authors of this say this somewhere in the study that it was a more pleasant experience for the programmers because you get all these breaks. You don't have to be locked in. When you're just looking at the blank coding page, no word, no piece of code is going to get there until you come.

come up with it and write it. It's really hard. There's no real break until you tell yourself to take a break because you're trying to get all this code together. Compiling for small programs as fast, too. You don't get much of a break from it. And so it's much more pleasant. Cybernet collaboration means much less, much less intensity of focus, much less duration of focus. It takes less energy. It feels nicer. But that's why they're slower. Because intensity of focus is what tells you how fast you're going to go. Intensity of focus tells you how good the stuff you're going to produce is when it comes to deep work. So the whiteboard effect says, yeah, come work with me so I can focus harder.

Good stuff gets produced. Cybernet collaboration, by contrast, says, I want a computer to offload some of the work, so I get a lot more breaks. But that means my brain is producing a lot less, slower, less quality stuff. And that is why the time required to get the work done begins to go down. Now, look, if the machine was actually able to take over all of the deep work, that would be different. If you could literally just say vibe code this program and then commit it, that'd be great because now you're like, I don't do any deep work at all.

But, of course, the machine can't do that. In cybernet collaboration, they can't do that yet. So now you're doing this back and forth dance where you ultimately still have to come up with the questions and edit the code and get everything to work, but you're cutting down your intensity of focus. So the whole operation inside your brain is going slower. I want to bring up another quote here. this is from an article from the Atlantic that was written by Roger Karma. And it talked about this study, among other things. The title of this article is just how bad would an AI bubble be?

And it really is about AI and productivity. I'm quoted towards the end of this article. It's a good article. But here's what he heard from the meter developers about what was actually happening as they were doing this sort of cybernetic collaboration. So let me quote here. Even the most advanced systems make small mistakes or slightly misunderstand directions requiring a human to carefully review their work and make changes where needed.

This appears to be what happened during the meter study. Developers ended up spending a lot of time checking and redoing the code that AI systems had produced, often more time than it would have taken simply to write it themselves, right? So they were just in this loop of you, right, I'll look at it, you debug, let me look at what you're doing, but because they were avoiding the full intensity of focus that they're capable of, because they avoided maintaining their focus without having context shifts or their distraction. The work they were doing was just like not at their peak.

So now you're doing all this work of cleaning stuff up and you're not operating at your peak. This is the potential danger of cybernetic collaboration that when you downshift your mind, let me downshift my focus intensity, it just doesn't work as well. It might feel nice, but deep work doesn't really have a lot to do with nice.

that's what seems to be going on there. So some combination of this rhythm of going back and forth creates like a lot of work you might not have had to do before. And more importantly, it reduces the gear at which your brain is operating. These programmers were just saying it was pleasant, but they were producing, you know, stuff slower. It was taking them longer to figure things out. That's one of my understandings from what is going on there.

But let's summarize this all. The end of our deep types, we like to do some takeaways. We've been experimenting with music. Our last takeaway theme music, which I thought sounded like the beginning. of a cable show about bass fishing. A little aggressive. So we're going to go a different way today. For our takeaway theme music,

we're going to do something a little more, you say, Jesse, intellectual, calming? All right, so let's do our takeaways with a little bit of calming background music. All right, so what are our takeaways? Deep work rewards intensity of focus. And if you add anything into your workflow that's going to reduce this intensity,

you'll probably get less productive. This seems to be the trap that a lot of knowledge workers experimenting with AI right now are falling into. Focus is hard. It doesn't feel pleasant. It's tempting to try to make it go away, but that doesn't make your work better. Now, I'm absolutely convinced that there will be upcoming applications of AI that will help our productivity. I think they will focus more on automating shallow tasks or speeding up things that don't require you to focus. but cybernetic collaboration, that is not the key to the AI future.

Focus remains absolutely essential to doing deep work, and for now it remains something that has to be hard to do. I feel like we should slowly high-five or something. We did it. There we go. There's a lot of other things going on in that study, but I mean, I think that's a lot of what's going on to it. It's like, look, deep work is hard.

It's tempting to make deep work easier. AI can help you do it, but doesn't mean you're going to produce more work. I love the takeaway portion of the deep dive segments now. I'm working on it. I think we're still working out the kinks. Because while you're going through it, it's just so the audience knows, I don't necessarily know what you're going to say. And I was like, well, what do I do?

Yeah. So I mean, I think it's good. Like sometimes it's advice and sometimes like here is the core message. Like here's the thing I'm taking away from it. So anyways, that's a cool study. Man, there's so much controversy around it. Like a lot of programmers are like, well, they don't know how to use the AI tools. It's not true, actually.

These are tools. They've got to choose which tools they use, and these are developers that largely were already using these tools. So, you know, there's always that argument of like, well, they're just using it wrong. Another argument, it's a small study. Yeah, it is small. But it's a good signal. It was a pretty clear signal. It's pretty well designed.

Another argument, which I think is more important, is some people say, yes, that use of AI, or you have a generating code from scratch is slower. Other uses where you're just automating, like looking up information, making that faster. I believe that would make a programmer more productive. So it does kind of, that's not trying to get in the way of the deep work. It's trying to speed up the things that's not deep work. So you can spend more time doing deep work. But basically, I'm sort of like a humanist intellectual chauvinist here. The brain focusing hard is an incredibly powerful tool.

Be wary of things that gets in the way of that. I mean, unless you really can just like outsource to work completely, it might be some fool's school. All right. We got good questions coming up. We got a good final segment. This one will be interesting to Jesse. I did a little bit of original data journalism. There's a claim out there that I'm looking into and the scene like, is this claim true?

I actually looked up some data from something, something. It'll be kind of fun. Mm-hmm. Just investigate journalism. But first, before we get to the questions, we have to do the thing that everyone is most interested in, which is hearing from a sponsor. Jesse, we have to talk about cozy earth. As listeners know, I'm a huge fan of their sheets that the most comfortable sheets we own. I think we have something like three different sets to rotate through.

We travel with them when we're away in our house rentals in the summer. I will say this, Jesse, if I was invited to stay in the Lincoln bedroom at the White House, I think my first question would be, what type of sheets do you have? Second question would be, why are you inviting me? But first question would be, what type of sheets do you have? But Cozy Earth, here's what I'm excited about. They've got a new product, one that I have been waiting for. Right? So I'm a professor. I'm a writer. This means I'm not wearing a suit and going to an office every day. And so I often wear pretty casual clothes. And as the weather cools down, I tend to wear sort of like athletic type pants. You know, I'm often on camera from the waist up. It doesn't matter. It doesn't matter what I'm wearing. I mean, I'm wearing bikini briefs right now because, you know, it's hot. You got to get people to want. Cozy Earth has a new product called bamboo joggers. So that sort of like casual pant in their super comfortable style fabric.

that we already know and love. You can wear their irresistibly soft jogger pants just about anywhere. They're made from that viscous bamboo fabric. These joggers are the softest staple wardrobe. One of the softest staples you're going to have in your wardrobe, whether you're staying in or stepping out. And what about if you're going to like a slightly more formal affair? So like, you know, me going to the Lincoln bedroom to investigate their sheets.

Well, they also have you covered with their everyday pant, which I have a pair of and I also love, they look snazzy, but they're also super comfortable with that fabric. So Cozy Earth nailed comfort with their bamboo joggers and their everywhere pant. That's next level. Everyone I know who is trying these things out says this is really comfortable.

So I'm a big Cozy Earth booster. You should be too. Go to CozyEarth.com slash deep and use my code deep for 40% off the best pants, jogger, shirts, and everything. If you get a post-purch survey, tell them you heard about Cozy Earth right here, built for real life, made to keep up with yours,

Cozy Earth. That's cozyearth.com slash deep and used to code deep. This show is also sponsored by BetterHelp. We talk a lot about the different types of advice that might help you be more organized or make more progress on the things that might matter. But you know what advice is probably the most important of all? You need to cultivate a healthy relationship with your own mind. Everything else rest on that foundation. If you want to improve this relationship, therapy might be exact.

what you need, a professional who can help you understand your own mind and improve that relationship. BetterHelp makes therapy easier. With over 30,000 therapists, BetterHelp is the world's largest online therapy platform. They serve more than 5 million people globally. It works. They have an average client rating of 4.9 out of 5 stars for a live session based on over 1.7 million client reviews. It's convenient too. You can join a session with a therapist at the click of a button helping you fit therapy in your busy life. Plus, you can switch therapist at any time. As the largest online therapy provider in the world,

BetterHelp can provide access to mental health professionals with a diverse variety of expertise. Talk it out with BetterHelp. Our listeners can get 10% off their first month at BetterHelp.com if they go to betterhelp.com slash deep questions. That's BetterHelp, H-E-L-P.com slash deep questions. All right. Speaking of questions, let's move on to our own.

All right. First questions from Sam. I found that using chat GPT questions can act as project diaries. I now keep multiple chats for projects I'm working on and enables a shutdown and a place marker for where I need to restart work. Can chat GPT chats act as single purpose notebooks for projects? I mean, sure. A couple questions here.

what is a project notebook, like this idea of like I need notebook to keep track of the progress on my project. Was that something you're doing before? I guess it's useful if you don't know where you left off and you want to have external notes on it. But here's my concern. My concern is that you're doing something like cybernetic collaboration with chat GPT. We talked about this in the deep dive. When you have this running list of notes and conversations with chat GPT, I'm concerned what you're doing is transforming your work into something where you go back and forth.

forth with the chat bot, you hand off more, you know, have them do some stuff and you look at what they did and you ask questions, you go back and forth. This is more pleasant. This feels nicer. It makes work seem better. But just as the meter study found with programmers, if it's reducing your peak intensity of focus or the sustained duration of your focus and you're doing anything just non-trivial, it also could be slowing you down and hurting the quality of your results.

The key with any type of effort that requires deep work is not reducing the difficulty of the deep work. It's not reducing the intensity of focus. It's setting things up so that you can reach high intensity of focus. The focus mine is what produces value. So that's my only issue. I would be wary if you're trying to make work in the Cybernet collaboration that might not be making you as productive as you think, even if it feels nicer. Last week you talked about leaving a narrative for personal projects. And originally, I think this question was in relation to that. But now that you talk about the Cybernet collaboration, it kind of makes sense. But if you could touch on the narrative part,

too. Yeah. So what I meant by that is like understanding how to hit the ground running. So, you know, if I'm writing something, I will end a writing session. It's like, okay, I've gotten up to here. And then I have a few notes. I'm like, okay, here's what comes next. So if I could like, I'm missing this example. But if I could get this key example, you can, we could finish this section and try to get to the end or something. Right. So it's a little note I left for myself. So that when I load up that writing project again, I know what I'm doing. I would do this a lot with mathematics papers. as well. Right? I would put notes right into the document where I was, you know, typing in my results or

I would email these to my collaborators and be like, okay, well, here's where I'm stuck, but I think here are like three ways I was thinking might be useful for getting unstuck. It's like a little narrative, a little note so that when I get back to that work, I'm like, okay, here's where I was or here's where I was going to think. So I do think that's useful, but I usually would just leave these notes right where I'm doing my work. It's in the document where I'm writing. It's in the document where I'm putting out my math equations.

And I think that's useful so you know where to start again and it's easier to get into it. The thing that makes me more nervous is this idea of this long ongoing interaction, I don't actually think that type of cybernet collaboration in most cases is a useful new form of knowledge work. I think it's just escaping strain. It's a fancier version of like I'm looking at my phone a lot. It sort of makes the work feel less hard, but hard is sometimes what you need. Who do we got? Next up is Mark. When using active recall to study a large volume of material, how soon and how often should I revisit material I've already mastered?

Right now I find that when I return to something a few weeks after learning it, I've forgotten many of the details. Well, I mean, it depends what you're trying to do here. Like, what are you learning and why are you trying to hold on to it? If you learn something that you then use, it will stay sort of active in your memory and eventually will really dig in. If you're memorizing some sort of information that you just basically don't come back to again, you're going to need to retouch on that. within a two-week window. Like, if you let about two weeks or more go, your mileage will vary depending on the information in your brain.

But if you let more than two weeks go without using information at all, it's going to start to lose its location. That is just like informal based on my experience as a student and professor. So if it's like this is arbitrary information I want to remember, every week or so you want to keep going back and doing that active recall. Eventually, if you do active recall enough, it will lock in. But again, be sure that you need to, if you're not using this stuff, be sure you need to remember it.

It's the best way to submit something in your memories to use it, to do it. You know, you teach the concept a couple times. You really remember it. You explain it to someone else because you're using it for a project. You know, you really remember it. Active recall, of course, simulates that. By recalling the information from scratch, you were firing up all the circuits you would use if you were actually applying the technique.

So it's using your brain's own memory apparatus. But, yeah, I would find if I memorize something using active recall and just never touched it again, within about two weeks, I might start to forget it. So, I don't, two weeks-ish, but what are you trying to remember and why and why aren't you using it? So maybe it doesn't have to be. All right, who do we got? Next up is Beth. I've seen articles ranging from AI as terrible for the environment to study suggesting its carbon emissions can actually be lower than traditional methods.

Can you help put this in perspective, for example, by comparing the emissions from one person's chat GPT use to something more familiar like a plane trip. and by clarifying whether AI's footprint is unique to platforms like chat GPT or similar to energy use and everyday tools like Google search. All right. I mean, I do have a hot take here. If you want like the actual answer, I don't know. I don't have the numbers in front of me, but something like a Google search is sufficiently significantly more energy efficient than like a chat chPD query. I mean, a chat chit query, you have to have this frontier model loaded up in memory. Now, this thing could have hundreds of billions, if not a trillion parameters defining it, right?

So you're not going to fit into the memory of a single GPU chip. You're going to have to shard this thing over like three, four, five, maybe six GPUs that are then going to have to be running all out just to generate tokens for your answers. I compare this to a Google search where they use commodity intel chips. Most of this stuff is cached. They can be dynamic of like, oh, there's a little downtime on this chip. Hey, can you like go look something up in a hyper-efficient sort of cached search index? and the amount of computation they've got, it's down, you know, minuscule.

Probably the answer you're looking for is in some CDN server that's like 10 miles away. So, you know, it's a lot of computational power. But here's my hot take. I might get yelled out for this one, Jesse. You ready for it? Yeah.

I think right now, the focus on the environmental impact of AI is in part a way for people who are critical of technology or big tech to get in on like I don't like AI in a territory that they're much more comfortable with than the technology. So if I'm just like, I don't know, like a typical like left-leaning critique of the tech industry and AI, I'm super comfortable for talking environmental stuff.

Like you don't care about the environment. I do. I saw the Al Gore documentary. You don't know science. The environment's important. Like I feel like I'm smart there. When it's me talking to other people, like that's where I'm smart. Like I know more about this than you.

You don't know science. global warming is a real thing. So that's like a very secure place of critique. A less secure place of critique is, you know, the type of stuff I've been doing in my writing recently. Like coming in and be like, here, I don't think that reinforcement-based learning, post-training refinements is giving you the sort of lost reductions that you would have expected on a more elongated power law curve. Like, people, getting into the technical details is very, like, it's hard. And you're like, I don't really know this details well. But I don't like this.

and I don't like the people involved and Sam Altman's kind of creepy and Mark Zuckerberg still talks like a robot whose a motion circuit board has shorted out. We've covered this on the show. And this is a territory that I'm much more comfortable in.

But just like also really leaning into the specific AI safety concerns over the particular words that a particular chatbot will or will not say. Well, I'm comfortable there. This is inappropriate speech. But I think if we're really thinking about

AI, these are kind of these are kind of transient concerns because I don't think a future of these massive frontier models being queried for everything is a future that makes any economic sense anyways. Here's the right way to think about it. If querying one of these models is bad for the environment, that means the amount of computation it's using can't possibly be profitable. So they can't be bad for the environment too long because it correlates also with expense and people aren't willing to pay for the massive computational expense.

I mean, stuff, it takes money to create heat because that requires electricity and electricity costs money. So, like, my thought about this is, yes, I would be concerned if we had chat GPT used all the time. But I just don't think it's profitable. I think the future of AI is going to be, it has to be systems that have much smaller models, machine native models of possible, meaning like this thing is running on my iPad, not in the cloud somewhere on a bunch of GPUs. all specialized language models combined with other specialized components, like policy networks for evaluating options, future simulators for trying to understand which actions to take next, control logic that's specific to the task at hand,

put together into a program that you can get your arms around that can run on existing hardware and does really good at being intelligent about a very specific thing. That I'm convinced as to future of AI. It doesn't have an environmental footprint that's different than, you know, other things that we're doing right now with our computer. So there probably is, it does use too much electricity, but I just don't think these massive frontier models are going to, it just doesn't make sense. I cannot be querying a trillion parameter model that requires six H100 GPUs to even produce a token for me for everything I'm doing in my life. That would probably be an environmental catastrophe, but that's not sustainable. I think frontier models like F1 cars.

It's a way of showing off your technology and proving that your company has the best technology, but it's not the car we're buying on the Ford lot. you know, a year from now. That's going to be a much simpler car. But we'll buy it from the company who had the best F1 car because that convinced us. So I don't know. That's right. So that's my hot take is,

of all the issues we have with AI, the focus on environment right now, we're not in some steady state yet. I just think it makes people more comfortable because that's territory where they're familiar being the sort of higher level of the hierarchy and the argument.

And they're uncomfortable getting the tech because, like, let's be honest, my people are nerds and are kind of uncomfortable to be around. And we don't talk very normal. and we know too much about algorithmic nonsense or whatever. So I don't know. That's what I think is going on.

So I'm not as worried about the environment because I don't think this industry can survive in a mode that is super bad for the environment. It just costs too much money. People aren't going to pay $1,000 a month to pay for all their queries or things. I think smaller models can work. Did you know, for example, the model, Noah Brown's model, Pluribus,

which was the first AI system to beat actual tournament players in Texas Holden Poker. The original way they built it was with very large neural networks, and they used a supercomputer system center at U. Pittsburgh to train it. It was like, okay. And then they figured out like, oh, we could have a couple different networks. And what matters is like the logic of how we connect them together and the logic of our AI. And the system they built pleuribus that can beat the professional players, you can run on a laptop.

Because it didn't have to, it wasn't just like we have 100 billion parameter network that we're just like learn poker. Like, no, like we're, we have a future simulator. that simulates the possible mind states and cards of the other players. Then we have a smaller neural network train just to like understand different poker positions and say what would be good or bad. And then we have some logic that we symbolic logic we hand programmed in about like, okay, let's calculate the value, the expected value. If we do this and this was the case, here's what would happen. Let's give that a number. What about this?

It's just like straightforward mix of neural network unsupervised learning with like symbolic old fashioned hard coded AI. And the whole thing fits on a laptop and it can beat professional poker players. So hopefully the future of AI I think is most likely. Environmental concerns are not going to be, I think, substantially different than sort of the environmental footprint of the computing we have right now. But we'll see. All right. What we got next? Next up is David.

If you left academia, would you spend your days as a writer and podcaster? Would you enjoy this or become antsy? Yeah, I would continue to write. and I would continue to podcast. I don't think I would become antsy. I've kind of two reactions to this. No, I want to become antsy. Writing and podcasting actually takes up a lot of time.

And it's not like I'm hurting for other things to do. You know how many teams I'm coaching and my kids like schools and sports leagues right now? Let me guess. Three to six. It's three. I'm coaching three teams right now. There's been a lot of time doing them.

I have a lot of things to do. That's fine. But here's the bigger point, though, is writing in podcasting is like a lot of what I do as an academic right now. So I don't know how different it would be. So, you know, for those who know my trajectory, I've been a writer of my entire adult life, right? I started writing in college. I trained as a theoretical computer scientist focusing on the theory of distributed systems. So I studied at Nancy Lynch's Theory of Distributed Systems Group at MIT and did my postdoc under Hari Balakrishna and his network and mobile systems group.

So I had a computer science specialty that was on the math behind distributed systems. Went to Georgetown. This is where like my NSF funding was all about this. My papers were all about this. My grad students. We did a bunch of, I was pretty good distributed theoretician, published a bunch of papers, we're on a bunch of steering committees. Somewhere around the time I was, you know, I got 10 year, then I got full professor.

Around the time I was getting full professor, Georgetown, where I work, they began making a real move for like, we want to be one of the places, grappling with technology and impacts. They call the field digital ethics. Like, we want to be at the core of this because we're in Washington, D.C. We have a big ethics background here. We were the university that, like, figured out bioethics in the 20th century, the Kennedy Center for Ethics. We want to do the same thing for tech ethics.

We're in the, we're here in D.C. We have all these policy centers. We have one of the biggest tech law faculties at our law school. Like, it makes a lot of sense. And I was like, this is kind of what I'm doing already with my writing. I write about technology and how it impacts us. I want it. So, like, I've really been focusing on that.

So I was one of the founding faculty members of Georgetown Center for Digital Ethics. I'm their inaugural director of our computer science, ethics, and society academic program. It's the first major in the country to integrate computer science and ethics in a combined major. There's places where there's majors where, like, you're a computer science major and you throw on a little bit of ethics. First integrated one in the country. So, like, this is actually what I'm doing largely right now as an academic is technology, how to impacts us what to do about it. And I do it in a lot of different forms. And I think, I think public outreach is important. I think the podcast is very pragmatic, but I reach a lot of

people this way. My writing for the New Yorker. Now we're talking to a little bit more of a, of a, like, a rarefied crowd. This gets me in front of policymakers. This gets me in front of like the Senate and on NPR or whatever. But it's a way, again, to work on tech and its impact. I do some academic papers on tech and its impact, which is more for like an academic crowd. And then my books fall somewhere in between. So I don't really know. My life wouldn't be that much different. If I left academia to write in podcasts, writing podcasting is what I'm doing in academia.

The main difference would be there'd be less time around really smart people and less time around students, both of which I like. Also teaching, but teaching increasingly I can teach things that like helps me think about these thoughts anyway. So, you know, no, I want to be antsy. And also I don't know how different it would be. Do you still do any math? I haven't very recently. I haven't. Yeah, been a year or so.

So do you still teach undergrads like traditional computer science classes? I do a mix of traditional computer science and stuff that's relevant for the computer science and ethics program. Yeah. And I teach less than I did before, too, because I'm running these things or whatever. So that's a good question, though. I think we have a case study. Yeah, before we do the case study, if folks have calls, then just go to the deeplife.com slash listen and submit some updated calls. The Deep Life or The Deep Life?

The Deeplife.com slash listen. There's a link. And you can record the call right there from the browser. Yeah, do that. We have a lot of calls, but they're getting kind of old, right? Yeah. You got a good shot. A good pithy call on a topic I've been talking a lot recently.

You got a good shot now. I would go record the calls. We do have a case study. However, I love case studies. This is kind of a long one, but it's great case study on my theories of lifestyle-centric planning. I'm thinking so much about this because of my new books. I'm happy to have this. We have, what, theme music for this, right?

Yep. Do we, I don't remember. Is it just to introduce this segment or do we play it the whole time? Oh, the two episodes we did one, but then last week you wanted to hear both. We'll play it now and see your mood at the end. All right, let's see what we go. Let's hear it. I just want to do a deep thoughts with Jack Handy.

All right. So today's case study comes from then. I hold a bachelor's degree in interface design and in my 20s first worked as a web developer and later as a U.X designer for several digital issues. agencies. I enjoyed my work, but I felt stuck in big cities and kept dreaming about a life closer to nature. So, in my mid-20s, I did something that you would surely advise against. I threw my career into the trash and moved to Norway to live and work on an organic farm. I had a great time there and enjoyed every aspect of it. However, I didn't have a sustainable long-term plan.

I realized I needed and wanted to move back to my home country where my girlfriend now White, lived. Back home, I decided that my next big move would be to find a job that allowed me to work in nature. So I signed up for a nature guide program and actually managed to land a job as a guide at the very place where I had completed that course. For the last six years, I worked there running workshops and similar programs. Sounds like a happy ending, right? Not quite. Life changed. I became a father of two children and we moved closer to my parents so we could get support from the grandparents with family life. This meant I was now living almost two hours away from my workplace, constantly torn between

family and professional life. On top of that, since most workshops take place on weekends, I was away from my wife and our two little ones at the very time they needed me most. It turned out that the dream of working in nature definitely had its flaws. I'm going to stop there for a second. Let's take stock of where we are. In the story so far, we're seeing a common issue when people think about. the deep life. So a common issue that people have is that you fixate on a single change that you

begin to believe will make everything better. It's simpler to think about a single change. We can sort of inhabit that change. The idea of doing something radical itself makes us feel excited in anticipation. And we can sort of take the imagined feeling of like the best parts of that change and sort of expand that in her mind. You'd be like, my life is going to be better. So, Sven did this with nature. It's like I like nature. So I'm just going to focus like a laser on if I could just be in nature all the time, I would be really happy. The problem, as we often say, when we talk about lifestyle-centric planning, is that your daily subjective mood is not the result of a single decision or change, but on all of the relevant aspects of your life.

Call this your lifestyle. All the different things that are relevant during the day add up to give you your subjective experience of the day. So you might like the part where I'm giving a workshop and it's in nature and that's nice, but you have other parts of your life too, like the fact that you had to drive two hours to get there and that you're kind of stressed out about what's going on with your parents and like the kids are, you're not around enough to help and it's creating tension with your wife. All of your houses may be nowhere near nature and you're actually spending most of your time in a car. All of these other aspects of your lifestyle matter too.

And this is how you can make one radical change that you're excited about. and yet end up less happy than you were before. Because by making this thing better, you accidentally maybe made other things worse. So lifestyle-sitric planning says you've got to construct your ideal vision for an entire lifestyle, all the parts of your life. How would I reorient my life so that all of the parts were something that resonated? And then how do I make progress towards that? You're going to have, you're going to enjoy your life more if you explicitly construct it so that all of the things that contribute to your enjoyment are being pushed in ways that are better. Let's return the Finn because he recognized that, and let's see how he made some changes.

Around this time, I came across your book's Deep Work, So Good They Can Ignore You, and Slow Productivity. They gave me a framework to better understand my professional situation. I had to take a hard look at myself and realize that for the last five to six years, I hadn't built transferable career capital that could help me in a different role. After some reflection, I decided to revive my previously built career capital as a web developer. However, I had it been coding seriously for years and looking at job descriptions, I realized how much had changed in the web development space. And I had a lot of catching up to do. All right.

So he's doing something here called evidence-based planning. So he makes this decision. He's looking at his life. And it's like, actually, I need a job that it's going to, whatever, the criteria here, but probably like more money, more autonomy. I don't have to commute so much. Whatever. Web development is like, I have career capital there. Let me start there.

but he does evidence-based planning. He actually looked at real job listings like, oh, skill, A, B, and C, I don't have. So I'm going to summarize a little bit here. But basically, he uses, like, techniques from deep work and slow productivity, as well as from Scott Young's excellent book, Ultra Learning, to begin a education process of learning the specific skills that his evidence-based planning said were important. Not what he wants to be true, but, like, I need to know A, B, and C. These employers want exactly those skills. and how do I get there as effectively as possible? I'll jump back to regular speed here.

Thanks to your techniques and tools, it took me about a year to get out of my difficult job situation and back into my old career. The side effect, I now earn more than twice as much as before. My next move won't be as bold as the ones I made in the past. I want to become a valuable asset for my company, get so good they can't ignore me, and then use my career capital to reduce my working hours

so I can have Fridays off to spend in nature without having to work there. So see what happened there is when he did lifestyle-centric planning. He's like, I like nature, but by reorienting my life around my career takes place in nature, a lot of other things got worse.

And actually being in nature is not as great as you think when you're also like working. They realize like, oh, if I could take my programming expertise, make it up to date, make myself really valuable, I can go back to programming,

get good enough that I can cut my hours, still make more than my old salary, only work three or four days a week, and then spend time in nature very intentionally, but now without the commute, now being home with my family, now things being more stable,

man, that's a better life. So lifestyle-centric planning worked out better than just laying back to me like, what's a radical thing I can do? I'll move to Norway. I'll get a job as like a nature guide. Lifestyle-centric planning was not as exciting,

but it led them to a better place. All right, so we got coming up, my investigative journalism, and beware. I do a little bit of data reporting. But first, before we get there, what you've really been waiting for to hear about another one of our sponsors. But here's the thing. If you run a small business, you know there's nothing small about it.

Every day there is a new decision to make it. Even the smallest decisions can feel massive. Jesse knows this. I get upset whenever anyone refers to our business here as small. I know how hard it is to run this. Just the other day, for example, the nice woman who runs the gift shop, down the on the next block here in Tacoma Park, asked me how life as a small business owner was going.

So naturally, Jesse and I rolled her car into a swamp. I can put up with that nonsense. So here's what I'm trying to say. It's hard to run a small business. So when you can find decisions that are no-brainers, you should take them. And when it comes to selling thing, using Shopify is exactly one of those no-brainers. Shopify's point-of-sale system is a unified command center for your retail business. It brings together in-store and online operations across up to a thousand locations.

It has very impressive features like Inless Isle, ship the customer, and buy online pick up in store. And with Shopify POS, you can get personalized experiences to help shoppers keep coming back. And they will come back. Based on report from EY, businesses on Shopify POS see real results like 22% better total cost to ownership and benefits equivalent to an 8.9% uplift and sales on average relevant to the market set survey. So get all the big stuff for your small business right with Shopify. Sign up for your $1 per month trial and start selling today at Shopify.com slash deep. Go to shopify.com slash deep shopify.com slash deep.

I also want to talk about our friends at My Body Tutor. I've known Adam Gilbert, My Body Tutor's founder for many years, and he is my go-to guy for fitness advice. His company, My Body Tutor, is 100% online coaching program that solves. the biggest problem in health and fitness, which is lack of consistency. They do this by simplifying the process into practical, sustainable behaviors and then giving you daily accountability and support, what you need to actually keep taking action on your plan. You have an app. You check it every day.

Like, hey, here's what I ate. Here's how I did my exercise. And your coach checks it. Hey, how to go. If you have a question, like, you know, I'm really struggling with X. Like, great, let's fix this right away. Knowing there's someone checking on what you do every day and is there to help you if anything's not working, that's how you get consistency and getting healthier.

It's not the information, the consistency. And having this online coach, it is a really, really good way to get healthy. All right. So if you're like, I want to get in better shape, this is the way to do it. Go to my bodytutor.com. Mention you heard about it on the Deep Questions podcast when you sign up and they will give you $50 off your first month. That's My Body Tudor, T-U-T-O-R.com and mention deep questions.

All right, Jesse, let's move on to our final segment. So I want to talk about an interesting article about technology and students, and I want to look a little bit deeper at it and have a bigger point to make. This article actually appeared recently in the Washington Post. It came to my attention, however, when Tyler Cowan wrote about it on his excellent blog, The Marginal Revolution. I'll pull it up here on the screen, the blog post because the Washington Post is behind a paywall, but Marginal Revolution is not.

All right. So here is Tyler's setup, just summarizing this article. He says, until recently, a nearby radio telescope meant the local school could not use Wi-Fi and how did that go? All right. So this article that he's citing here is about Green Bank, West Virginia. Have you heard about this place, Jesse? No. It comes up a lot in technology circles because it's rural West Virginia and the world's largest steerable radio telescope is there. Oh, really? Yeah. And there's a town next to it. And radio interference could mess with the telescope. So there's no cell phones and there's no Wi-Fi.

So it's like a town that's kind of like off the grid because of this telescope that's in the middle of the town. So what this Washington Post article that Tyler Cowan is citing, what it was about, it was actually an op-ed and was written by someone who has a book he's working on about Green Bank. what it was arguing is the fact that there is no Wi-Fi in the town has been hurting the students in the school there. Greenbank has an elementary middle school, combined school, about 200-something students. He's arguing without Wi-Fi, they can't use like the modern Chromebook, online curriculums, do online testing, and this is hurting the students. So let me read here. This is a Tyler Cowan citing the Washington Post. So what I'm about to read here is from the Washington Post.

Article. While the rest of the country rushed to bring tech in the classrooms, Green Bank remained stuck in 1999. Without Wi-Fi, the school's 200 students couldn't use Chromebooks or digital textbooks or do research online. Teachers couldn't access individualized education programs online or use Google Docs for staff meetings. Even routine tasks such as state mandated standardized testing became challenging with students rotating through a small hardwareed computer lab where they took the exams. Some of the teachers say this has been a problem. Here's a quote here. The ability to individualize learning with an iPad or a laptop, that's basically impossible.

That was teacher Darla Huddle. Here's another teacher being quoted in the op-ed. Without the online component of our curriculum fully working, it's really detrimental to our instruction. That was Sarah Brown. So this is the issue here. They can't do this sort of modern ed tech stuff. And the argument of this article is like that's held this school. back.

And how do we know this? Well, there isn't data on this in the article itself, like specific data, but here's what the author says. While these discussions dragged on, so discussions about, like, is there some way we can get, like, low-powered Wi-Fi or some other ways to get Wi-Fi in the school? While these discussions dragged on, students fell further behind in math and reading with GreenBank consistently posting the lowest test scores in the county. All right.

This is an interesting argument here. that without modern ed tech, students are falling behind. And that's like a natural experiment. And look, this school has the lowest scores in the county. This is like a relevant time to be arguing this because there's so much discussion about phones in schools and technology in schools. And it's mainly negative. Like technology in schools is often negative.

But here's this op-ed in the WAPO that's being contrary. And Tyler, who also has a real affinity for technology, is like, yeah, come on, let's be careful about it. Maybe some of this technology is really helping. So I want to look into this. You know, I'm not an expert data journalist, but I'm like, this is interesting. And I want to look a little bit closer. So what did I find? Well, first, is it true that Green Bank consistently posts the lowest test scores in the county?

This is, turns out to be called Pocahontas County. It's kind of interesting. I'm like in West Virginia. Okay. So it's Pocahontas County, West Virginia. Yes, the Green Bank Elementary Middle School has the lowest scores. in the county. But there's a problem here.

It's a very small county. I mean, it's a lot of geographical area, but, you know, this is not Montgomery County here in like Washington, D.C. The county, you know, there's the Green Bank middle school and elementary school. The county has one other middle school and a shared high school. And then there's two smaller elementary schools that are in like another town over there are much smaller, one of which, which has the counties like gifted and talented program.

There's only 70 kids there. half of them are gifted and talented kids. Like we're not talking to big county. It's a handful of schools of which like two of them of the six are in Green Bank. One is shared. And then there's like one other middle school and two other elementary schools. They do. So yes, like the elementary school in Green Bank has lower scores than those two other elementary school.

The middle school has lower scores than the other middle school. All right. So it's a small county. But that doesn't necessarily tell us much. Like maybe Green Bank, you know, for whatever reasons, that part of the county just is worse off. And they just get worse scores. I don't know what's going on there. Do we know this is from not having Wi-Fi or not?

Well, what we really would need to test this is we need time series data. So we need to see test performance over time, right? Because until somewhat recently, it wouldn't matter if you had Wi-Fi or not, right? So iPads were introduced in 2010. We see the rise of Chromebooks and classrooms picks up in the 2010. That's really where this thing, this happens. So before the 2010s, there would be a lot more technological parity between this particular school and the other school in the county, right? Because no one was using technology like that in school.

So what we really need is like a time series that shows somewhere in the 2010s, the Wi-Fi Free School begins to separate. that the impact on not having Wi-Fi makes them worse because they might have always been worse. We want to see them get worse. All right. So I went looking for this data. It's hard to find time series data for the particular schools in Pocahontas County. But because Pocahontas County is so small, there's two middle schools.

One of them is with Wi-Fi. One of them's not. There's three elementary schools, one with, one without. And the other two add up basically to the size of the one that didn't have Wi-Fi. It's such a small county. What we can do is compare, Pocahontas County to other counties in West Virginia. Because presumably if like half the students in your county didn't have Wi-Fi and that affected

them, that should bring down the whole county around the time that Chromebooks, etc, became big in school. So there is data. I can get data on any AP, you know, these are state mandated test scores. I can get data on this county by county. So let's see what's going on here. So let me load up some charts. This is from education recovery scorecard, which aggregates a lot of this data.

It's really interested in what happened after COVID. But they have data that goes back to as far as 2009. So it's perfect for us. What I'm loading on the screen here for people who are watching instead of just listening is math performance grades three to eight. Because again, going past middle school, it's a shared high school. So like we don't learn much from that. From 2009 to 2024. All right.

So as we see here, going from 2009, We see an increase. There's a bit of a gap in a data, but we see roughly actually math scores were getting better. Go Pocahontas County. Math scores were getting better from 2009 until about 2017. In 2017, oh, this is interesting. The score started going down. They went down, went down.

Then we get the pandemic where we don't have data, but they were falling multiple years before the pandemic. And then we see after the pandemic, we begin to get a recovery, which we've seen nationwide. as things fell so low during the pandemic that there's a recovery right after. So that's the chart we're seeing. If we look at reading scores, it's something kind of similar. Here, the peak is around 2015, and then it begins to go down after that as well. The timing here is compatible with the Wi-Fi hypothesis, right? That kind of makes sense.

Chromebooks and all these other things that require Internet begin spreading in the 2010s. It might make sense that where would you first start seeing the impact of not having this technology, you like midway through the 2010s as these things are gaining traction, that might be where you start seeing performance go down. So this data from within Pocahontas County itself is compatible, roughly speaking, is compatible with the Washington Post hypothesis of without Wi-Fi, things started to get worse. But if we want to do a controlled experiment, we have to compare this, the similar counties. So what we need to do, because Pocahontas is half the non-W-Fi and we can't break that out.

So let's find other counties in West Virginia that are similar in terms of size and demographics and socioeconomic status, like very similar other counties nearby in West Virginia where there were no radio telescopes and everyone is allowed to use Wi-Fi. What we would expect to find if the Wi-Fi hypothesis was correct is that Pocahontas County should have a much more notable drop in performance starting in the mid-2010. in these other counties that didn't have any Wi-Fi restrictions. We have that data. That's what's interesting. So let me scroll here a little bit. All right.

So we're going back to math performance here. We have three stack charts. So if you're listening, I'll explain to you what we see. Each of these charts has a downwards arrow and an upwards arrow. The downwards arrow, which is purple, I'll put this up here on the screen for Pocahontas County Schools first. That is the decline in math scores, during that period up to 2019.

Let me get the exact dates here. 2019 to 2020, where we had the steepest sort of losses we saw in those curves before. And then the green arrow is the improvements that they've seen since the pandemic. So this chart is showing here for Pocahontas County Schools. It's just quantifying what we saw on that chart. In that downhill period from 2019 through 2022, there was a 0.6, negative 0.6.

drop in their performance compared to, I think this is like state average. So yeah, they fell. And then we see the green. There was like this 0.36 increase. So don't worry about the numbers, but there was an increase. This is the recovery after the pandemic. All right. So I'm just quantifying what we saw on that chart.

Here's what's interesting, though. They did the same thing for all of West Virginia counties. That's the next one. And for counties that had similar social, economic, demographic, and size. So similar population. So if you're wondering, this is Nicholas County, Hampshire County, Barbara County, Tucker County, and Pendleton County.

Here's what's interesting. The similar West Virginia districts that were the same in terms of demographics, but differed mainly and they didn't have the Wi-Fi restrictions. They saw a bigger drop from 2019 to 2022. And they saw a smaller recovery after the pandemic. So the similar counties that had Wi-Fi and never had that take. in a way did worse than the county in which like half the students are in this Wi-Fi Free Zone.

And in fact, if you look at the West Virginia as a whole, that also was worse than what Pocahontas County did. So look, I don't have school-by-school data. I hear this a lot. I found a lot of informal reporting online that did blame poor performance in Green Bank by them not being able to use Internet in the school. So that might be true. And these teachers seem to think it's a problem.

But if we look at the data without having time series data from specific, the handful of schools within Pocahontas County, we do not see here a data story that looks at, that looks like the lack of internet, once Chromebooks became a thing and these online curriculums became a thing, began to make Greenbank much worse. It seems like that's just a bad school. Those schools are bad, and they've been bad. And I don't know why you'd have to know about, like, you know, it's just like what this town is. I don't know what particularly is bad about that town. It could be little things, by the way, because these numbers are so small. Not to get too much into the data, but I was looking at proficiency on math and reading test scores broken down by the individual school. So I can get those numbers for the most recent times they've measured them.

And they're like, you know, 50% better, for example, in Hillsborough Elementary versus the Greenbank Elementary. But Hillsborough Elementary in Pocahontas County is the gifted and talented program. There's only 70 kids in that school. 70 kids total. So all it takes is like, oh, we have 30 of our gifted and talented kids are there. You're going to get 50% better, you know, number of people who are math proficient, right? So these are small numbers. So we have to be careful.

So, yeah, there's not a lot of schools there and the Green Bank one doesn't do well. But we don't have evidence that it is because they couldn't use Internet connected Chromebooks. Again, the only way for this to be possible would be. Somehow the lack of Chromebooks at Internet-connected tech really caused the Greenbank schools to fall really hard. But for unrelated reasons, coincidentally, the other two or three schools in Pocahontas County did unusually well during this period. And they sort of offset the fall that the Green Bank was having. And that's why that county, even though other counties with the same demographics as Pocahontas County fell farther, that there was something special happening with the non-green bank schools in Pocahontas County where they offset the Green Bank losses.

Maybe that's true, but we would have to hear a plausible reason why that's true and actually see school-by-school data. So I don't know. The author of that op-ed is writing a book on Green Bacon, he might have really good data. But knowing what we know now, I think this is a nice cautionary tale. And it's a good reminder for myself or anyone else who talks about technology trends. When there's an answer that we like, it's often easy to jump to it. Find any point of data that seems to imply that and expand it beyond what the data says. And I think that seems to be what was happening here.

There's a nice subtle leap from the schools without Wi-Fi are worse to the schools without Wi-Fi are worse because they don't have Wi-Fi. It's an easy leap to make. But the picture gets much more murkier once you pull even a little bit on the data story. So anyways, I guess this is just a back-to-school note of we all have to be careful and take with grains of salt, claims that are made that sound intuitive and there's a little bit of data to support. doesn't necessarily mean it's true. Like, it's like when it comes to cell phones in schools, I have read exhaustively, not just the research, but the debates that researchers are having about the research and the complaints and then how the complaints are answered. I, you know, there's an hour-long talk I give about the evolution of the research literature on harms from phones for kids. I feel like I know that data very well.

And it makes me confident to say these are often a problem. The benefits aren't worth it. You shouldn't have phones before high school if you're a kid, right? that's really different than like, hey, that school struggles and they don't have Wi-Fi. Internet's good. So anyways, there we go. I'm not investigative journalist data. Who knows?

Maybe this professor has the school-by-school data. But I think the picture in West Virginia, Pocahontas County is more complicated than if only I could synchronize my Eureka math curriculum with like online resources. Our kids would suddenly be much better. Harder reality. All right. Speaking of hard realities, that's all the time we have for today. So thank you for listening. We'll be back next week with another.

episode and until then, as always, stay deep. Hi, it's Cal here. One more thing before you go. If you like the Deep Questions Podcast, you will love my email newsletter, which you can sign up for at calnewport.com. Each week, I send out a new essay about the theory or practice of living deeply. I've been writing this newsletter since 2007 and over 70,000 subscribers. get it sent to their inboxes each week. So if you are serious about resisting the forces of distraction and shallowness that afflict our world, you got to sign up for my newsletter at calduport.com

and get some deep wisdom delivered to your inbox each week.