3 Takeaways - Learning from the Best: A Nobel Laureate's Journey to Understand the Secrets of School Quality (#153)

Episode Date: July 11, 2023

Why do some schools routinely produce high-performing students? How truly important is class size? Why do charter schools typically outperform regular schools — is it the schools themselves, the students they attract, or a combination of both? Get ready to be educated on crucial issues of education by Nobel Prize laureate Joshua Angrist.

Welcome to the Three Takeaways podcast, which features short, memorable conversations with the world's best thinkers, business leaders, writers, politicians, scientists, and other newsmakers. Each episode ends with the three key takeaways that person has learned over their lives and their careers. And now your host and board member of schools at Harvard, Princeton, and Columbia, Lynn Thoman. Hi, everyone. It's Lynn Thoman. Welcome to another

Three Takeaways episode. Today, I'm excited to be with Nobel Laureate Joshua Angrist. Josh is a professor at MIT and co-founder and co-director of MIT's Blueprint Labs. I'm excited to learn about his surprising and completely counterintuitive findings on education, including what he calls the elite illusion. Welcome, Josh, and thanks so much for joining Three Takeaways today. Thank you. It's great to be with you. It is my pleasure. Josh, your results on what you call the elite illusion or eye-opening, can you tell us about them? Sure.

That's a name we gave to a phenomenon, I would say. It's kind of something that's apparent in our research on schools. Many people, parents and families and schoolchildren, of course, are interested in the question of where the best schools are. And people are naturally drawn to schools that have very good outcomes. And now I'm thinking about K through 12 American education and public schools. And the most eye-catching schools in the public K through 12 sector are what are known as exam schools. Those are selective schools. This includes, for example, the Boston Latin School in Boston, near where I live and work.

That's the oldest high school in the country. It's a very selective public school. A lot of famous people went there, including some Nobel laureates and well-known public figures of all stripes. And those kids who go there have very good outcomes. Other schools that your audience might be familiar with are the legendary three of New York City, Brooklyn Tech, Bronx Science, Stuyvesant. But not just in Boston and New York, around the country, there are stratospherically selected public schools where kids have very impressive outcomes, careers in public life and science and so on. And so,

naturally, families observe those outcomes and they're keen for their kids to go to those schools. Now, we've actually studied the causal effect of going to exam schools. This was one of the first series of studies done at the lab. So we had data on people who applied to the schools and were able to follow the applicants who do and don't get offered a seat at those schools. And we can use the magic of econometrics using a methodology called regression discontinuity research design, which simply essentially produces something like a randomized trial for schools that admit kids based on a cutoff because very near the cutoff, people just above and just below are

similar. And what that research shows is that, yes, it's true that kids who go to, say, Boston Latin or Stuyvesant have very good outcomes, but it's not because of the school. People who come close to the cutoff but don't get offered a seat have similar outcomes. And so we call the fact that there isn't really a causal effect of going to those schools, yet parents see the very good outcomes and are seduced by that. We call that the elite illusion. They desire to take advantage of what looked like very good outcomes. But really, what you're seeing there is a result of a phenomenon we call more technically selection bias. The kids who go to Boston Latin or Stuyvesant have good outcomes, meaning higher test scores, and they're more

likely to go to college and so on, because that's the type of kid that gets in. But it's not actually a causal effect of going there. Can you explain in simple terms what selection bias is and why you struggle against it? Selection bias is any force or forces that confound simple comparisons and make them a misleading guide to causal effect. There's a lot of variation in the world. People do different things. They take different drugs. They eat different things and so on and so forth.

And selection bias is a shorthand technical name for the fact that people who do and don't do various things, whether it's medical or educational, are different going in. And so we look for empirical strategies in economics to try to fix that problem. In the case of schools, selection bias arises from the fact that different sorts of families are drawn to different sorts of schools, or maybe only particular types of families or children are allowed to attend certain types of schools. And that makes any kind of causal inference about the effects of those choices hard to obtain an unbiased gauge of causal effects. And so our lab and people like me, you know, we make a living trying to find empirical strategies. Sometimes we actually run randomized trials, but mostly we try to use the data that are available to come up with something that's close. So to put it in very simple terms, if there were two schools in a neighborhood and one is a public school and one is some kind of a school that people need to apply to, and if that school where the families need to apply, if those children have better outcomes than the children in public schools, what you're saying is that might be selection bias because the families that have children in those schools have preselected

themselves or they care about education? They were special. And so they're not representative because they're more aware, say, of schooling options or more involved with their children's education. And so that predisposes some comparisons to look good. There are other cases where those sort of forces predispose some comparisons to look good. There are other cases where those sort of forces predispose some comparisons to look bad. One of the early questions that our lab tackled was the controversial question in the U.S. of the effects of going to a charter school relative to a traditional public school. Charter schools, as you may know, Len, are essentially

publicly funded private schools. You can set up a charter under state law in most states now. And somebody who is not a traditional public school district can open a charter school provided they meet various requirements. And there's a vigorous debate about whether those schools are better or worse than the traditional public alternatives. And the early evidence on charters was based on simple comparisons. And it showed, for example, that kids who go to KIPP, which is an important charter school operator in the country, it's a charter management organization that runs many schools around the country, it was one of the first big charter operators. Kids who go to KIPP schools tend to have higher test scores later than kids who don't. And charter critics

were quick to point out, and rightfully so, that that's probably misleading because the people who are aware of KIPP and so on and so forth and choose to go to KIPP and, you know, the KIPP is also more demanding. So there's some selection there into your willingness to being able to put up with a long school day and a long school year. That biases that comparison. And so our lab used the fact that oversubscribed charters, meaning they have more applicants than seats in Massachusetts, must pick their students by lottery. So we took advantage of that.

And that produces a very clean, natural experiment because it is literally a lottery. It's a random number that determines whether you're offered a seat at a charter school like a KIPP school. And as it turns out, the lottery-based analysis of schools like KIPP shows that they do have very large benefits. So that's a case where there might have been selection bias, but it doesn't turn out to be true. Josh, let's talk more about your findings in education. How important is class size? Does the size of classes matter? Well, many people who have kids in schools worry about class size, and families that see that their kids are in large classes

mostly don't like that. And teachers in general would prefer to have smaller classes. I'm thinking again about K through 12 and elementary school. Personally, I'm a teacher and I like to have a nice big class because it's kind of more fun, but it's a different story. It's college and everything. Anyway, here, too, you have to think about selection bias and what you can infer from the data that are just lying around. So if you just look at the data that are just lying around, which doesn't necessarily mean it's lying around on the sidewalk, but it's data that you could get if you're an academic, you would see that, in general, kids in smaller classes do worse and kids in bigger classes have higher test scores. So you might think, oh, well, that

means that big classes are good. But of course, that's another example of selection bias, because where are the classes largest? Well, they tend to be largest in big cities where incomes are higher and people are more educated. And inside schools, they tend to be larger than smaller special education classes or classes for non-native English speakers. And those kids are going to have lower test scores anyway. So comparing kids in big and small classes is not an apples to apples comparison. There are a lot of options in education. There's class size. There are some schools you talked about, no excuses schools, which have longer school

days, longer school years, national standardized testing and a lot of teacher feedback. But if I asked you in very simple terms for non-economists, what are the most important findings on improving education for students? What works? One of the things that seems to work very well in this context is a particular type of charter school. You mentioned No Excuses. Nowadays, it's called High Expectations. And it provides a kind of menu of features that seem beneficial. One is a long day. Another is a long year. I should say, in the interest of full disclosure, my daughter, Adi, teaches in a Boston charter school. So I know the charter experience both in the data and personally. We can't, for example, have a family vacation in August because Adi starts teaching

typically on August 1st. She has to start work, which is not true in the traditional public sector. And other things these schools do is they emphasize traditional reading and math. They tend to have a little bit more emphasis on discipline and comportment, though that's been changing now. They sometimes have incentives. Kids wear uniforms and various things. So I can't say in particular what's the most important feature there. But as a collection of features, those schools tend to produce very good educational outcomes, very good test score gains, and kids are more likely to go to college, and they're more likely to go to four-year colleges. And we know that because we can use the lotteries to generate apples-to-apples comparisons, kids who are and

are not offered seats. So that particular sector, the high expectations charter sector, has proved itself. I'm sorry to say it's now, it's kind of come under attack. It's fallen out of fashion. It used to be that there was a fairly broad consensus in favor of that type of school in the setting where it's found, mostly large urban districts. And that seems to have eroded somewhat. I'm skeptical of what many people believe, which is that the school districts with the best outcomes have the highest quality schools. So this goes back to the question of the elite illusion. When you come to MIT and you join the faculty at MIT, you'll have lunch with the faculty and people will welcome you and so on. And then almost always there'll be a discussion about where are the good schools if you have school-age children.

You'll ask the colleagues. And they'll say, oh, you know, you should live in Newton. The schools are good in Newton. Newton is a high-income suburb. You should live in Lexington or Weston or Wellesley. Well, it's true that Newton, yeah, they have great outcomes there. Those kids all go to college and they have high test scores. But I mean, who lives in Newton? Doron Asimoglu lives in Newton and Andre Schleifer lives in Newton and Glenn Ellison lives in Newton.

And these are all very educated people with stellar careers and scholarships. So their kids, of course, tend to do very well. But that is not a reflection of Newton's school quality. Newton may have good schools, but you don't learn that from simply observing the fact that Newton has good outcomes. It's a case of the elite illusion, perhaps. What have you learned about college and college completion? Unconditionally, people who go to expensive private universities have much higher wages

later in life. But then once I know that you got into an expensive private university, it doesn't actually matter whether you go there. There's no relationship between where you go and your earnings later in life. So that's another example of the elite illusion. Josh, what are the three takeaways you'd like to leave the audience with today? Well, the most important takeaway from my research, I think, is that selection bias is everywhere. And simple correlations and associations and descriptive facts, while almost always interesting, including to me, are often misleading when it comes to the causal effect of a choice you might make about school or health or whether to serve in the

military, career-related things. So I would like people to think twice about that. And if it's an important decision, to take the time to dig a little deeper and that you might learn something that you didn't initially think based on sort of the simple facts of the matter. In our world, I give advice about careers and what it takes to be a successful researcher. I talk to a lot of prospective graduate students. And one of the things I try to leave them with when they come to me for advice is how hard it is to succeed in the world of scholarship. I know that sounds funny coming from me since I've been reasonably successful. But most people who aspire to careers in scholarship do not succeed. So it's important to kind of understand that and to adjust your expectations. The world of scholarship is a lot like the world of major league sports or the performing arts, that there's a small number of people who have very successful careers. But most of the people who aspire to careers in sports or performing arts fail at that.

And I don't think aspiring scholars are sufficiently aware of that. The third thing is to take some risks. It's very hard to succeed if you're so risk-averse that you can't tolerate failure. I think that's a handicap for lots of people, not only in academia, that people are too afraid of failure and look for something where there's more guarantees. It's nice to have guarantees and have a secure life economically and so on. But if you want to accomplish something and leave your mark on other people, you'll need to take some risks. That's great. Thank you so much, Joshua. Sure.

If you enjoyed today's episode and would like to receive the show notes or get new fresh weekly episodes, be sure to sign up for our newsletter at 3takeaways.com or follow us on Instagram, Twitter, and Facebook. Note that 3takeaways.com is with the number three. Three is not spelled out. See you soon at 3takeaways.com.