Revisionist History - Puzzle Rush

Episode Date: June 20, 2019

Malcolm challenges his assistant Camille to the Law School Admissions Test. He gets halfway through, panics, runs out of time, and wonders: why does the legal world want him to rush? Learn more about your ad-choices at https://www.iheartpodcastnetwork.comSee omnystudio.com/listener for privacy information.

Pushkin. It's 8 o'clock in the morning on a Saturday. A little chilly. Deep in downtown Manhattan. Streets are empty. I'm standing on the sidewalk with my assistant, Camille Baptista. First of all, Camille, I need to know, did you sleep well last night? I did sleep well, yes.

I did not. Really? Oh, no. I could almost no sleep, and I had a nightmare about the LSAT that I left before it was over. I had an exam nightmare from high school, essentially a high school exam. You left before it was over? I had a nightmare about the LSAT that I walked out before the test was over. Oh, but my nightmare was just beginning. This is Malcolm Gladwell.

I'm back with Season 4 of Revisionist History, my podcast about things overlooked and misunderstood. stood. This episode is about what happened when Camille and I took the law school admissions test. We had three number two pencils each, a small pack of Kleenex, a package of trail mix, all in clear plastic bags. We lined up outside Pace University with hundreds of other nervous people clutching clear plastic bags. Everyone wanted someday to become lawyers. Except us. We were there in the name of science.

We have to go. This has all been very traumatic and and stressful but i wanted to get one last thing i would like to get your handicap on your chances of beating just tell me what you think your chances of beating me are um my chances as a as a your mom i would point out was quite confident you would win my mom is confident in me no matter what which is very nice but you know maybe unrealistic uh you know i don't know 50 50 camille i really i how old are you i am 24 okay um i can't even i won't even tell you how old i am but it's it is a it is a large multiple of your age i know but i don't think it has a i don't think it has to do with age. I think it has to do with reading ability.

And you read nonstop 24 hours a day. I've been in intellectual, I've been in cognitive decline for 30 years. It's all going to be exposed. All right, we're off. We've got to go. I got the idea of taking the LSAT from a man named William Henderson. I read a paper that he wrote. It was on my favorite website, SSRN,

which is where academics from around the world post their papers and they get ranked. If you're a regular listener to this podcast, you'll know how genius I think SSRN is. Anyway, this paper was called The LSAT, Law School Exams and Meritocracy, The Surprising and Under-Theorized Role of Test-Taking Speed. Anything with the words surprising and under-theorized in the title, of course, is going to be catnip for me. So I read it.

Then I read it again. Then I had to meet him. So just a little bit of background here. I didn't go to law school until I was 35. And so I had had a whole career before then. What were you doing before you became a lawyer? I was a firefighter paramedic. I was a union rep for a suburban Cleveland fire department. I did that for nine years. And as I kid people, and remember, I'm kidding, is I went to law school because of those bastard management attorneys. Henderson goes to law school. After that, he gets a clerkship with a judge. And one day, he's in the shower and a thought occurs to him. What was the aha in the shower? The specific aha was what?

Was, my God, the two most time pressure things I've ever done in my life is taking the LSAT and these damn law school exams. He had been a firefighter paramedic. He had raced to the streets of Cleveland to save people on the brink of death. And the most time pressure he'd ever felt was taking the LSAT to get into law school and then taking the exams once he got there. I can remember taking Cass Sunstein's elements exam my first year of law school. It was a two-hour exam. Cass Sunstein used to teach at the University of Chicago, which is where Henderson went to law school. He's a genius. He asks the kind of questions that you can spend months, years thinking about. But his exam? Two hours.

That's all you got. The proctor said time, and everybody dropped their pencils, and there was this huge ugh. You know, everybody was like groaning that they didn't get more time to keep on working on Cass Sunstein's element exam. It's like, you know, 90 people who all dropped their pencil and they're all, you know, exclaiming the desire for more time. That was a memorable experience. Now, I'm sure this is obvious to you, particularly if you're an American. You've been tested a thousand times in your life. SAT, ACT, GRE, on and on. And every one of those

standardized tests doesn't just test whether you can answer the question correctly. They test how quickly you can answer the question correctly. But I'm a Canadian. I've never taken a standardized test in my life. And as an outsider, I have to say the whole system seems really weird. Why is quicker better? The LSAT is Exhibit A. It's the single most important thing that determines where you go to law school. Nothing else comes close. And what is it? It's around 125 questions divided among five sections. Analytical reasoning, reading comprehension, two sections on dissecting an argument, and one experimental section. Everything's multiple choice. You mark your answers with a pencil by filling in little bubbles on an answer sheet, like an IBM punch card. A perfect score

is 180. And if you're a super go-getter, you cannot score below 175. Because then you can't get into Harvard. And if you can't get into Harvard, you're never going to get an offer from a big law firm or get a Supreme Court clerkship. Your life is over. Everything, the country club, the BMW, the multiple cases of 2000 Chateau Lafitte Rothschild. It hinges on those five sections. And how long do you get to spend on each of those five sections? 35 minutes. Not 40, not 50, not 55.

And if you finish up one section early, you don't get to add that time to the next section. When we decide who is smart enough to be a lawyer, we use a stopwatch. So that was William Henderson's question. What does putting that 35-minute time limit on a cognitive test like the LSAT do? So I said to Camille, let's find out. And because we're both super competitive, the whole thing got very involved. The first thing I would like to do is to understand on a very, very granular level what time pressure means for the way people take the test, the strategies they use, the kinds of mistakes they make.

Everyone told me that there was no way I could do well on the LSAT without getting some coaching. So I went to one of the top new educational technology startups in America, a company in New York called Noodle, started by John Katzmann, who before this was one of the founders of the famous Princeton Review test prep company. I sat down with Katzmann and two of his top people in a big loft building on Union Square in Manhattan. Katzmann, Fritz Stewart, and Dan Edmonds. No, but I am very anxious.

Other than pride. I am very anxious not to lose to Camille. So I thought I would go. Was she born in the U.S. or in the U.K.? Oh, she went to a fancy school. Like, this is a seriously intelligent woman. But I'm saying, U.S. or U.K the U.S. or in the U.K.? Oh, she went to a fancy school. Like, this is a seriously intelligent woman. But I'm saying, U.S. or U.K.? U.S.

Okay, you have a problem. Oh, great. I'm too old, and I'm from the wrong country. It was not an encouraging start. I've done things under time pressure before, of course. I've written newspaper stories on deadline. I wrote exams in college. In those cases, though, the task required me to be me, only with a sense of urgency.

But the first thing the noodle guys told me was, the LSAT didn't require me to be me. It required me to be someone else. A lot of it is about helping people understand they don't get to do this at a comfortable speed. That's Dan Edmonds. Right. In order to finish these sections, you have to do it at a speed that's a little uncomfortable, which means you have to hone your instincts by the test rules, not by your rules. What does it mean to read it at an uncomfortable speed? For me, it means that if there's something, like in a reading comprehension passage, if I hit a paragraph that I don't fully understand, I don't get to go back and reread it. I just kind of have to accept the parts that I understand and move on.

The noodle guys approach the LSAT like pathologists approach a cadaver. For me, I'm like, eh, all right, I didn't get that, but if a question asks about it, I will go back. But it doesn't bother me, as long as I understand the topic sentence of the paragraph and the overall thrust of the author's argument. If I miss a few details here and there, or even a chunk of the argument,

I'm like, all right, fine, who cares? I'll go dig it out exactly when I need to. Wait, it's okay to miss a chunk of the argument? So I don't even aim for a level of what we would normally call comprehension in my first read. I am out to process the information, not understand it. I don't get any points for understanding it. I get points for bubbling in the right question. I imagine that when you read, there are lots of moments where you sort of,

oh, that's an interesting point. And you sort of pause and you think and you let your mind meander a little bit. There's no meander time on the LSAT. There's no digression time. Meander and digression are my whole MO. That's what I do for a living. Where do you think this podcast comes from?

So then John Katzman gave me a sample question. What I would have you do is read this and then for each of these questions, tell me the two stupidest answers. Two stupidest? Okay. The two answer choices that you know are wrong. Whoa, this is one? That's what I'm saying. You can't read this on a podcast. No, no, I can't. The passage was 600 words. It seemed longer. I've just been told that I don't need to understand it. I don't need to comprehend it. It's okay if I miss chunks of it.

In other words, I'm supposed to read it without reading it. And as I sit there puzzling about this, the three of them debate right in front of me how long this reading without reading should take. John said a minute. Fritz said three tops. Dan said it depends on the person who's reading. The only thing they all agreed on was that I had to hurry. Fly, get the bones of the argument, and now feel free to spend the time on the question itself. And again, all you're trying to do is tell me the two worst answers. Two worst answers. Can one of you time me on your question?

Sure. Just so this is on page 142. Ready? Go. Okay. I read it out loud because I thought maybe that would help. Researcher, people who participate in opinion surveys often give answers they believe the opinion surveyor wants to hear. And it's for this reason that some opinion surveys do not reflect

the actual views of those being surveyed. However, in well-constructed surveys, the questions are worded so as to provide respondents with no indication of which answers the surveyor might expect. After the passage, there were multiple sets of questions to test my understanding of what I had just not read. There were pages of them. So if a survey is well constructed, survey respondents' desire to meet surveyors' expectations has no effect on the survey's results. The reasoning in the researcher's argument is questionable in that the argument overlooks the possibility that... I'm going to say A and B are... Just cross-map and move on to the next one.

Okay. Those are the worst. Not they're wrong. They're the worst answers. The clock is ticking. This is such a brutal passage, Johnny. Cruel. Jesus. Bananas.

What am I at, time-wise? This is what they meant when they talked about uncomfortable reading. I was being forced into a kind of altered, frenzied state. The word they used was breathless. You should be a little breathless, they said. I was breathless. I kept asking, how much time has passed? They'd tell me.

My heart would accelerate. I started to panic. Okay. Okay. I have no idea what I did. Great. Timing on that, 7 minutes 20. So this section is how long?

8 minutes? 35 minutes. 35? 35 minutes. 35 minutes. And we just did 1, 2, 3, 4, 5, 6, 7 questions. You would have had, say, 10 minutes. Yeah. But I haven't picked any right answers yet. But you've done all the hard work.

And that was 4 and a half? Oh, 7 and a half. 7 and 20. So, you're tight. Oh man, I'm toast. You're not toast. Oh, I'm toast. We did another question. I was convinced the answer was B.

It was C. I could feel Harvard slipping away. Great. Thank you. What do you guys think my odds are of beating Camille? Do you want the honest answer? I want the honest answer. I think you're in real trouble, buddy.

She's got the advantage. On the day of the LSAT, I sat in a little classroom in one of those fixed half-desks that I last had in grade school. We got our test packets. I raced through reading comprehension with time to spare. Sections two and three, I was right up against the deadline. Then came analytical reasoning, logic games, and I looked at the questions and what I had to figure out and realized that there was just no way I was going to finish in 35 minutes. I needed to slow down, and I wasn't allowed to slow down. I glanced behind me, and there was

Camille with like a death stare in her eyes, ruthlessly dispatching question after question. Then I looked around some more and saw all these kids half my age, beavering away, full of purpose, because all you Americans apparently accept as gospel this idea that the smart person is not the person who gets the right answer. The smart person is the person who gets the right answer the quickest. Mercifully, it ended. At 1.30, Camille and I stumbled outside into the sunshine, clutching our plastic baggies. Our producer, Jacob Smith, was waiting.

We had no idea that some people wanted to be lawyers, first of all. That was a shock. Camille cheated because she packed my... We had to have these plastic bags. She packed my plastic bag for me, but she gave herself like apples and like really, really nutritious snacks. And I, I got it. Gave you trail mix, which is the same thing.

So you had fresh fruit and Malcolm? Malcolm had trail mix from CVS, but, but, but who sharpened all your pencils, Malcolm? And packed you little tissues in case your nose runs? Now after both taking it, what's your honest opinion? Who do you think did better? Oh, I think Camille did.

I did so poorly. I was fine and I got really cocky. And then I hit the logic games and basically I think I got zero right. I had no idea what to do. I sat there and I was fine and I got really cocky. And then I hit the logic games and basically I think I got zero right. I had no idea what to do. I sat there and I was like in a state of complete panic. I was untested until this moment. Now I have been tested.

America has taken its measure of me. And it's a pretty humbling experience. Why do Americans do this to themselves? Do they play Scrabble with a stopwatch? In literature class, do they get extra points for reading Tolstoy's War and Peace overnight? Is there an Oscar that goes out every year to the movie that got shot the quickest? I really don't get it. Okay, time to meander and digress. don't get it. Okay.

Time to meander and digress. I became a Grandmaster at 15, so I think I became an I Am Right after I turned 13, and I was a Grandmaster at 15, so that was very, very good at the time. That's Hikaru Nakamura.

Grew up in Westchester County, outside New York City. He's in his early 30s, although he looks about half that age. He's one of the best chess players in the world. How much chess were you playing as a kid? Pretty much all the time. So I was playing at the Marshall at least, I would say, two or three days every single week. And then I would also be playing Blitz on the Internet Chess Club, I would say, at least five to six hours every single week. And then I would also be playing Blitz on the Internet Chess Club, I would say at least five to six hours every single day. Years ago, I met Magnus Carlsen, who is the greatest chess player in the world. And I asked

him how much he used a chess board versus how much he just worked through chess positions in his mind. And he said, oh, I practice mostly in my mind. In fact, I'm working on an opening right now. Hikaru Nakamura is a little like that. He's incredibly gracious and humble, but even when he's giving you what seems like his full attention, you get the feeling that there's a whole separate part of his brain breaking down a Bobby Fischer match from the early 1970s. The first thing I did after my disaster with the LSAT was call up Heikaru. But I didn't want to talk to him about chess. I wanted to talk to him about time. So my first question about this would be,

what would happen within reason if there was no clock? How does the way that you approach a game of chess change if I remove the time constraint entirely? All right. So if you remove time entirely approach a game of chess change if i remove the time constraint entirely all right so if you remove time entirely from the game of chess um every game of chess would be drawn because without without time if you have an endless amount of time to think about any given move if i could think for 30 minutes on every single move i do not think i would ever lose a game of chess to a human to a computer i would I would still lose, but to a human... Even Magnus. Yeah, if I had a half hour for every move, I don't think I would ever lose to Magnus. This is why chess games have a time limit.

Otherwise, it's not a game. Tournaments would go on for months, and everyone would end up tied for first. It would be like Little League. Everyone would get a participation trophy. So there's classical chess, which is the kind played at the World Championships. Classical allots 90 minutes for the first 40 moves, then 30 minutes for the rest of the game. Winning at classical chess involves calculation, working through many possible scenarios

before deciding on a move. Then there's blitz chess. That's what Hikaru was playing for five or six hours a day growing up. In blitz, each player gets five minutes for the whole game. When you play blitz chess, it very much becomes about finding moves that look good, that are not blunders, that you can play almost instantly where you use a couple of seconds. In classical, how many moves would you go deep? It normally would be about five to six moves

in about three or four branches. It's a fantastically complex mental exercise. Yes, very much. I mean, it's probably at least, I would say, close to a hundred different permutations of moves or sequences that you're looking at for every single move. Yeah, yeah. In Blitz, we've truncated that process.

Right. Everything about Blitz and classical is the same. Same pieces, same board, same players, same choreographed openings. But the time limits are different. And what happens when you tinker with the time limit? You get a completely different set of results. At classical chess, Magnus Carlsen is number one.

He's also number one at blitz, because he's a genius. Hikaru, right now, is 11th in the classical rankings. But at blitz, he's number two in the world. Why? Because he's really, really good at the rapid pattern recognition that's necessary for blitz, and he's not quite as good at the complex calculation that's necessary for classical. Now, who's an example, a good example of the opposite? Fabiano. Fabiano Caruana is, actually, he's probably the best example. I can't even think of anyone else who is that much better at classical chess than they are at blitz and rapid.

Why do you think he's not good at blitz and rapid? I think with Fabiano, it's the other way. He's very, very good at calculations. So when he gets positions, he's very good at calculating and understanding what the possibilities are with more time. Whereas when he doesn't have time

to calculate the long sequences all the way through, his intuition has to take over and his intuition and natural feel are not as good as everyone else. Fabiano is a tortoise, slow and steady. Hikaru is a hare,

ears back, speedy. You construct your chess that favors the tortoise or you construct your chess so that it favors hares. I think you can see where I'm going with this. If we had a Blitz tournament for the World Championship, you know, in two months, where would you put your odds of winning?

I would put my odds of winning probably around 20%. I would say I'd put Magnus at about 60%. I'd put myself at 20%. So pretty good odds. Yes. So your world changes if, just arbitrarily, we decided that the standard for international tournament chess ought to be blitz. Yes, it would change, yeah. You would make a lot more money.

Probably, yes. Wait, there's a third variation, bullet chess. In bullet, you get one minute for the whole game. And Hikaru is the king of bullet. Are you better than Magnus at bullet? At bullet, yes. At bullet, yes. Why wouldn't Magnus, if he's so serenely superior at all the other kinds of chess, why can't he beat you a bullet? In terms of the calculation, what he does is like the, well, it's not an algorithm,

but sort of the way that he fine-tunes it or the way he thinks about it, I think it takes a little bit more time to come to the conclusions. What strikes me is that the chess hierarchy, the formal chess hierarchy, is an arbitrary function of the amount of time we have decided to spend on a chess match. Right? Yes. Like I said, I think you can see where I'm going with this. Hikaru is a hare, not a tortoise.

And what is the LSAT? It is a test that rewards hares over tortoises, which means that if the LSAT ran the chess world, they would consider Hikaru the greatest chess player in the world. They would crown him champion over Magnus. LSAT logic is that the best player is the one who solves the hardest chess puzzles quickest, and that's Hikaru. But that's insane. Not even Hikaru thinks he's better at chess than Magnus. What Hikaru would say is that he's a different kind of chess player than Magnus.

And I haven't even mentioned Puzzle Rush. It's an online game where you get a series of endgame chess positions, and you have to get to checkmate as often as possible in five minutes. Puzzle Rush is insanely popular. Do you play a lot of Puzzle Rush? I have the highest score at it. Do you really? Yes, I do.

So what's your score? My highest score is 55. And what's the second highest? I think there are two people with 54, and then their couple was 53, and it falls off from there. Yeah, yeah. um have most of the uh top players in the world played puzzle rush most of them have yes has magnus played i mean not officially not officially you think you might be playing and hiding he probably has yeah yeah

yeah do you get any award for being the number one on puzzleuzzle Rush? No, no, no. They don't like Crown You King of Puzzle Rush? There's a three-hour video of Hikaru crushing Puzzle Rush on YouTube. It has 130,000 views. This actually might be the most hotly contested form of chess you've ever played. You've never played something where millions of people are in the same tournament. Yeah, that's true. You're right. You could make an argument this is your greatest accomplishment. I guess. I wouldn't think of it that way, but yeah,

you're right. The only reason we don't consider that your greatest accomplishment is we've arbitrarily decided only to honor chess played under these archaic rules in a tournament atmosphere that puzzle rush is a i mean if we decided to say that's what chess is right yeah if you arbitrarily decide yes that's true yeah it has many advantages chiefly is it allows the entire world to compete on an equal right right yeah so you went against

the entire world and you won i don't know yeah so you went against the entire world and you won i don't know yeah yeah i've never thought about that but yes that that is true yeah but if one thing comes out of this it is i hope you you uh give yourself a pat on the back the order in which people finish in any cognitive task is an arbitrary function of how much time is given to complete that task. You can make it fast, or you can make it slow. The chess world has chosen to reward the tortoise. The LSAT has chosen to reward the hare. They've decided to play Puzzle Rush and reward the Hikarus of the world and not have us play classical and reward Fabiano. What does the legal world have against Fabiano?

As I mentioned earlier, there's going to be a second part to this examination of the LSAT. Of course there is. Why would I rush? In the next episode, I'm going to visit with the folks who administer the LSAT, the Law School Admissions Council of Newtown, Pennsylvania. But that can wait. Let's go back to William Henderson, who started me wondering about time and tests. The great justification for the LSAT is your score is supposed to be a useful indication of how well you will do in your first year of law school. It's a predictor. But Henderson's great question was, what if the LSAT only predicts law school grades

because law schools make the same mistake that the LSAT does? In most law schools, grades are based in large part on how well students do on exams where they are deliberately not given enough time. You take Cass Sunstein's Elements exam, where you engage with the ideas of a legal genius, and you have two hours. That's it. Henderson wondered if a law school doesn't rely so heavily on its students doing things quickly, if the school relies instead on take-home exams and essays, what happens to the usefulness of the LSAT as a predictor? And he found that its usefulness declines.

I'm quoting, The data showed that the LSAT was a relatively robust predictor of in-class exams and a relatively weak predictor of take-home exams and papers. In other words, once you stop racing against the clock, then the people who do well on the LSAT no longer are the best at law school. Which is exactly what every chess player in the world would have told you would happen. So what did William Henderson do when he became a law professor at Indiana University?

He changed the way he evaluated his students. He started placing more emphasis on take-home exams. And when he gives an exam in class, he makes it four hours, not two. It's a completely open book, and I give out one of the questions in advance. I say, here's three questions. I'm going to test you on one. So you've seen one of the questions already, and then four hours is plenty of time to do the issue spotter, and I give a word limit. as a direct result of that LSAT study that I did because I was acutely aware that you change the ordinal ranking when you pick a test method. And I say, you know, I don't think anybody will need all four hours,

but if you want to take four hours, you're free to take four hours. Not long after he started grading that way, a student came to see him. This kid got an A on the exam. And so he comes in, he wants to talk about exam. He said, I've been here for three years. I've never done this well on the exam before, and I want to know why I did this well. And I pulled out his exam, and it was an eight-hour take home. And I go, look at your first paragraph.

You hit every single issue here in the first paragraph. It's just a well-organized pearl. The light bulb went on for him. It's just like if I had had more time, all my exams would have been disorganized. You have a student who the system declared was of average ability. The student believed it. Why wouldn't he? But then someone came along, someone, by the way, with the great benefit of being a firefighter from the suburbs of Cleveland, who had the freedom to think a little differently.

And he said, wait, maybe you aren't of average ability. Maybe you only think you're average because we have chosen an arbitrary system to evaluate your ability that makes you look average. You are Fabiano, and we have been making you play Puzzle Rush. I know what you're thinking. Yeah, yeah, yeah. But who won between you and Camille? Who got the higher score in the LSAT? No!

Stop racing ahead. You are engaged in uncomfortable listening. You'll have to wait until the next episode. What I want you to think about is that student of William Henderson's and his miraculous revelation that he really was a good student after all. There are a ton of tortoises like him out there, not just taking the LSAT and sitting in law school classrooms, but competing for places at any number of schools and professions that have decided to tell their applicants whether they are or are not any good.

And what I don't understand is how the hares got to set the rules. I thought the whole point of the story of the tortoise and the hare was that the tortoise won. Clearly, the kind of person who is most disadvantaged by this system is the tortoise. I realized after my experience with the noodle guys and my time with Hikaru that I'm on the side of the tortoise. I feel for the tortoise. I might be a tortoise. I come from tortoises. And we've all met tortoises in our lives. Yeah, yeah. Who will not, my mother is a little bit of a tortoise. She will not be rushed under any circumstances. She will not, she does

not make mistakes. She goes over things five times to make sure that they're perfect. Yeah. She is ideal for a certain kind of work. She'd have a problem with the LSAT. She would say, why are you rushing me? And she would, you know, and she wouldn't finish. And she would never work. She'd have a problem with the LSAT. She would say, why are you rushing me? And she would, you know, and she wouldn't finish. And she would never guess. She can't guess. She's incapable of guessing. So my mom could never, the profession is putting up a barrier to my mom. You put a stopwatch on thinking, whatever. You rigged a system so Camille wins. Fine. But when you go after the Gladwells, then it's personal. Revisionist History is produced by Mia Lobel and Jacob Smith with Camille Baptista. Our editor is Julia Barton.

Flan Williams is our engineer. Fact-checking by Beth Johnson. Original music by Luis Guerra. Special thanks to Carly Migliore, Heather Fane, Maggie Taylor, Maya Koenig, and Jacob Weisberg. Revisionist History is brought to you by Pushkin Industries. I'm Malcolm Gladwell. But Camille, Camille is like a smarty pants. She's going to do, don't you think? She's going to do amazingly well. Yes, yes. Well, Camille has a natural,

she naturally prepares and is naturally persistent and determined. So I think those things will really help her a lot. Should I be worried? I think you should be worried. I think you should be worried. Yep.