Revisionist History - Lord of the Rankings

Episode Date: July 1, 2021

For 30 years, US News & World Report has been using a secret formula to rank the best colleges and universities in the United States. As a public service to our listeners, we hack the algorithm and discover the dirty little secret of the rankings game. Learn more about your ad-choices at https://www.iheartpodcastnetwork.comSee omnystudio.com/listener for privacy information.

Once upon a time, in the middle of the 1980s, when the Cold War still raged and a man named Ronald Reagan occupied the White House, there were three major weekly news magazines in the United States. Time magazine was number one, with millions upon millions of copies in circulation. Newsweek was number two. And what news magazine came in third place? A sleepy little publication based in Washington, D.C., known as U.S. News and World Report. One of the magazine's managing editors in those years, Peter Bernstein, remembers the day

when everything changed. It was a very exciting time because Mort had just taken over. There were a lot of people buzzing around. Mort being Mortimer B. Zuckerman, a Canadian billionaire who bought U.S. News and World Report

with dreams of becoming a media baron. There were a ton of really good ideas sort of in the magazine, and I think it's fair to say our feeling was that the execution could have been a lot better. And one of the things we wanted to do was distinguish ourselves from Time and Newsweek, because we were the third. And how do you distinguish yourself if you're always coming in third? If a Canadian billionaire with visions of grandeur

is breathing down your neck, you do something audacious. You create something that no educated parent could possibly ignore. A scheme to examine every American college and university and rank them from top to bottom. We were looking to reinvent the magazine, and the college rankings became part of that. So Bernstein and his colleagues began collecting a mountain of statistics on the universities and colleges of the United States. To make sense of those numbers, they came up with an algorithm, a secret algorithm that could assign every school

a score of 1 through 100, regardless of its size or mission or complexity, an algorithm to capture the elusive quality of academic excellence. Finding those points of data and putting them into a ranking, as you can appreciate, is a complicated task. Yes. Full of value judgments, full of problems, challenges, dangers, but we were undaunted and and preceded i love that you're laughing about this it was a very heady time and and we thought and we did really challenge time and newsweek at their game the rankings became an institution u.s news surged careers were forged, fortunes made. I mean, what's fascinating today is that Time and Newsweek have basically disappeared. The rest of U.S. News has basically disappeared, but the rankings are as strong as ever. Well, it's nice to know in this world of ephemera that something can

last. And as a journalist, you know, when your work is used for old fish after a day, it's nice to see that something has an impact after a while. Peter left U.S. News long ago, but he and his colleagues had added something new to the American academic calendar. Beyond freshman week, finals, commencement, there's now the day the new rankings come out. The new 30th edition of Best Colleges was just released this morning, and we've got it. You're seeing the 2015...

The list for 2019 was Princeton University, followed by Harvard. All right, attention, scholars. America's best colleges have been revealed. The U.S. News and World Report is out with its annual list, and we've got the inside scoop on how... The algorithm that's developed is undergoing a constant state of evolution over the years. How often are you tweaking it?

I thought you might ask about this. I mean, I did go online and look at it, and I was glad to see they are tweaking it on a regular basis, at least if you look at their methodology. And, of course, they've also been reacting to competition. I mean, you know, a lot of people have been critical of the rankings and whether this methodology really—this methodology, excuse me. That's it. Peter, that is like the textbook definition of a Freudian slip.

You're talking about your brainchild, and you refer to it as mythology. That's true. I couldn't have come up with that one consciously. At what point did the U.S. news methodology become a mythology? Let's find out. My name is Malcolm Gladwell. You're listening to my podcast, Revisionist History, about things overlooked and misunderstood. In this episode, I ask, how is it that a promotional gimmick from the 1980s turned into a fetish object for American higher education. How do these rankings work? And why does everyone take them so seriously? We're going to launch a revisionist history investigation, one that will take us to all kinds of strange places, to the bond markets, to Portland, Oregon, to New Orleans. We'll meet a mysterious college administrator

who asked to be identified only as Dean, as well as a band of hackers who agreed to join our cause. All right, all right, so we're ready to go. This is very exciting. Now, before we start, can both of you guys introduce yourselves? So I'm Kelly McConville, and I'm an assistant professor of statistics at Reed College. I'm Hua Ying Qiu, and I'm an international student who is currently studying at UNC's grad school. So these are some of my accomplices, and here's why I needed them. U.S. News will tell you what variables they use to determine whether a school is good or mediocre, but they won't disclose exactly how they combine those variables into a score. I mean, why would they? It's their secret sauce.

They charge $40 a year for premium access to their findings. This is a big business. So I figured my only chance at really getting under the hood of the ranking algorithm was to find out if anyone had, in effect, hacked the algorithm. And it turns out someone had. It was a group of students at Reed, a small liberal arts college in Portland, Oregon, under the leadership of their professor, Kelly McConville. Tell me how on earth this whole project of looking at U.S. News rankings, how did it begin? So I teach a course called Statistics Practicum. And in that course, the students complete a semester-long empirical research

project. And I solicit those projects and the data from other Reed faculty and staff. Every year, U.S. News asks for a lot of data from schools, but the Reed administration had stopped supplying that data. They decided they didn't believe in the rankings anymore. But still, the magazine kept generating a ranking for Reed anyway, and someone in the Reed administration came to Kelly McConville and said, ever since we dropped out, our U.S. news ranking has plummeted. We think they've been punishing us. Can you double check their math? That was our task. We were tasked with recreating the model and then seeing what our ranking would be if we did supply our data.

Yeah. So tell me, the two of you, walk me through, how does one go about conceptually trying to answer this question? They use a model that's called a weight and sum model. That means they take about 20 or so different factors about a school, like class size and graduation rates. They weigh each of those factors based on how important they think the factor is. And then they sum up all those pieces and then calibrate it so that the highest score is at 100. And then based on those scores, they rank a school. The Reed team had to build their own version of the U.S. News Algorithm from scratch, trying to get as close to it as possible. We grabbed all the variables we could from open government sources, and we built a model that isn't trying to recreate the U.S. News weight and sum model. Instead, it's just trying to predict the U.S. News score of a school with a high degree

of accuracy. Huaiying, how close did you get? I would say fairly close. The only school that's outside of a reasonable range of predicted score in the test set is Reed. Oh, really? And all other schools are within a reasonable prediction interval of their scores. Meaning, the Reed students'

identical twin algorithm is so good that when they run their rankings of American colleges, their list looks almost exactly like the U.S. News list. With one exception. Reed College. Wait, so they were messing with you?

Based on our finding, it seems reasonable they did something. In 2019, U.S. News put Reid in 90th place. Where do you think you should be ranked? 36th? Oh, wait, what? That much of a difference? Yeah.

That's bananas. Oh, my. Oh, they really of a difference? Yeah. That's bananas. Oh my. Oh, they really did a number on Reed. Now, we don't need to go much further into the ensuing beef between Reed and U.S. News. The magazine calls the undergraduate's analysis, quote, inaccurate. But just imagine my joy when I ran across Hua Ying's analysis. Here I was searching for some way to understand the secret algorithm cooked up by Peter Bernstein and his pals back in the 1980s, and lo and behold, a bunch of undergraduates at Reed College had cracked the code. And were they motivated to help me? Absolutely. Because U.S. News dissed their school.

My first question to Kelly and her team was, where should I start? And they said, start with a little something called the peer assessment score. Can you give me a more precise sort of statistical sense of just how important is it? Okay, so we find that the peer assessment score has the largest effect. So it has a coefficient of like 6.58. In other

words, if you hold everything else constant and you increase the peer assessment score by one unit, your overall score will increase by 6.58. The peer assessment score was at the heart of the U.S. News secret sauce, a measure of reputation, one of the most powerful elements in the whole ranking system. So if I'm a school and I'm anxious to move up the U.S. News rankings, the thing I should really be focused on is

trying to boost my reputation score. Yeah. I had my marching orders. And I knew where I had to go next. To the guardian of the U.S. News gates, the dragon at the mouth of the cave,

Robert Morse. Robert Morse, the man who runs the U.S. News rankings, is an older man, white hair, very preppy, nice, if a bit lugubrious. Heavy hangs the head that wears the crown. I called him up. I wanted to understand the mysterious variable that the college hackers had told me was so important. Describe again, how is the reputation score generated? So we send out three surveys per school and then they respond.

They rate the schools on a scale of five to one or no response. So who are the three people at each school who are asked to fill out the survey? The president, the provost, and the enrollment manager or the head of admissions. The surveys arrive via email once a year to everyone who matters in American higher ed. If you're at a liberal arts college, you're given a list of all 222 other liberal arts colleges in your category. If you're part of a big university, you get a list of the 388 other big universities in the United States. Your job is to rank the schools on your list on a scale of one to five. Five being amazing, one being there's a serious problem here. And on what basis are people making those judgments? Well, I mean, that's a good question. I mean, we asked them to base it

on their view of the school's reputation for undergraduate academic quality. But I mean, how would you know? I mean, we believe that to aggregate some of all the raters who are leaders in higher education, you know, they're presidents and provosts and mission deans, so they're not rookies. When you aggregate their views at any given time, they're representing where those schools stand in the marketplace. I mean, so it's an assessment of their feelings about other schools. Well, they're leaders in higher education. So it's based on their knowledge of the other schools

that they've gathered or through meetings or for exposure. Do they necessarily know a lot about all the schools they're rating? I don't know. If I rate a restaurant on Yelp, it's because I've eaten there. Right? Yes. But here, when I rate a university

on the U.S. News Ranking, it's not because I've attended that university or taught there. It's just... Well, you may have. We shouldn't say that you didn't, but if you're saying, have the people who rated the schools been on every campus or taught at that school or know in depth their course catalog? No. Do some of them have greater degrees of knowledge about the schools they're rating? Yes. Do some of them have lesser degrees of knowledge? Yes. So I think it varies from having substantial knowledge to far less knowledge.

In my naivete, I had imagined that there would be some level of rigor to this crucial variable in the U.S. News weight and sum model, a methodology that I could study, analyze, learn from. But Morse wasn't being very helpful. So I tried to draw him out. I mean, I was fascinated that there's a little clump where BYU, Brigham Young, Yeshiva, and Gonzaga are all ranked right in the same, roughly equally in the national universities rank. So a Mormon school, conservative Jewish school, a Jesuit school. And I was trying to think, what does the rabbi who runs Yeshiva know about Brigham Young? And what does the Jesuit priest who runs Gonzaga know about Yeshiva?

No, I mean, it's a good point. Like, is Gonzaga giving Georgetown a five and giving Boston College a five? We're not looking at how people are rating other schools. We are doing a trim mean, so we take off a few of the highest ratings and a few of the lowest ratings in an effort to eliminate some of the strategic voting that you're alluding to? Well, I don't know if that's strategic voting. I'm wondering if it's just that if I'm president of Gonzaga, I'm powerfully motivated to believe that the Jesuit education is the best education out there. So when I look at Georgetown, I think they're doing the right thing.

And when I look at Yeshiva, I say, where's the New Testament? Why are they not using the New Testament? I mean, this is just natural human biases. I just don't know what they have to do with the quality of the institution. Well, I don't have a different answer. Devoted listeners of Revisionist History will remember how years ago I fell for Rowan University in New Jersey. I love Rowan. But Rowan is ranked 187th in the National Universities list.

Dismal. Mediocre. And why? Because its peer assessment score is a forgettable 2.4. How on earth have the university presidents of America come to the collective decision that they hate Rowan University way, way down in South Jersey? I call up Ali Houshmand, the school's president. How many university presidents have visited Rowan in the last five years from other colleges? Two or three. One from Canada, one came who was part of an accreditation, and I believe another one was a friend. That's it? Yes.

So when they're giving you a score, on what basis are they using for giving you? How do they know anything at all about Rowan? Again, I don't know. I can only speak for myself. I can assure you that I really don't know much about the institutions that I'm asked to rank or score. I just don't have enough tools. Now, it's not like university presidents don't know anything about grading. They were all teachers themselves once. But when it comes to grading schools, it seems like they might just be mailing it in. Next, I called up the head of admissions at a major university.

You've heard of it, trust me. But I can't tell you what it is. And we altered the voice to protect this person's identity. Of course we did. We had to. Look how U.S. News treated Reed College. My source told me to address them only as Dean. I am certain that I am far more qualified to be a member of the Baseball Writers Association of America and vote on the Hall of Fame than I am

to assess other institutions for the U.S. News and World Report. Yeah, yeah. I proposed to Dean that we do an exercise. I'd name a school at random, and then the two of us would try to peer assess it. Florida State. Never been there. Have you been there?

I've not been there. I think. 3.2. Syracuse. Yes. I stopped for coffee there once on the way, driving back to my mom's house in Canada. I don't know if that counts. I don't think it does.

I think it counts. You're going to count that? You experienced the weather. But there were no students there. It was Christmas. But I will say this. My cousin went to Syracuse and had a good time. Uh-huh. And so you would rate it based on that?

Let's give Syracuse some love. 3.9. No, 4.0. University of Washington in Seattle? Yes. You've been there? Yes, I have. What did you do when you were there? Um, I walked around. I think that counts. It counts. I said to Dean, you can see the mountains from there, the Space Needle. Let's go big. University of Washington at Seattle, 4.2. Have you ever been to William & Mary? I have.

You have. I love history. I love Colonial Williamsburg. Do you remember what score you gave William & Mary? I probably gave them a 4 or a 5. Really? You went for 5? Yeah.

It's the second oldest college in the U.S. Do you think that affected your judgment on that? I'm sure it did. Dean and I went on for quite some time. We had a long chat about Southern Methodist University in Dallas and whether or not the great football team the school had in the 1980s should count in their favor. Dean said no. I said yes. Now, was all this a helpful exercise? Not really. But then again, now there was my conversation with the wizard of the rankings, Robert Morse. One suggestion that has been made to me by people who do the rankings is that the thing they rely on to make their reputation scores for other universities is the only source they have of reputation scores for the other universities, which is the previous year's U.S. News rankings.

Is this circular? If they're telling you that, then that's not our expectation of how people are doing their ratings versus looking at the ranking table and looking at the score. We believe that there's more thought going into it than that. But we haven't done any kind of social science research to prove or disprove that point. You haven't done any. Right.

Yeah. They haven't done any social science research. Only one person gave me much insight into the mechanics of the peer assessment score. And that was Ali Houshmand, president of Rowan. His advice was, you have to take matters into your own hands. So what does he do? He makes his own hot sauce. And he told me that Rowan sends out free samples to every college president in the country.

That's not true, really? He did. He did? He did. At least the federal president does not know that I make hot sauce. Whether that will help me in terms of scoring, I hope so. Yes. Wait, I want to know about your hot sauce. What kind of hot sauce do you have?

Oh, I make serious, serious hot sauce. But I actually grow from seeds all the way to the packaging. And there are three levels of heat. The lowest one is called Ali's Nasty, which is kind of mild. And then the one next to it with habaneros as a heat is called Nasty Delicious. And then the one with Carolina Reaper is called Nastylicious. And then the one with Carolina Reaper is called Nasty Vicious. And I will send you a package. The Nasty Vicious can get you.

But this isn't going to help your cause. This is going to hurt your cause. If you're giving some university president in the middle of the Midwest happens to have a little bit of vicious, they're going to curse your name. They probably will. But I figured that no publicity is bad publicity.

Did I try the nasty vicious? Oh yes, I did. On a scale of 1 to 5, 5.0. In the middle of my investigation of the peer assessment score and its many implications, I had a conversation with a professor at Duke named Bill Mayhew. Mayhew was friendly with some bond traders, and he was talking with some of them one day when they mentioned that they were having difficulty with bonds issued by historically black colleges, HBCUs as they're known. If you're a bond trader, you worry about moving the bonds that you have in inventory. And

they just mentioned, you know, hey, we're having a really hard time placing these HBCU schools because people don't want to hold them. The way the bond market works is that an institution needs money. Let's say a university wants to build a dormitory that costs $50 million. The school can't raise all that money on its own, so it goes to a Wall Street bank, which purchases a bond, a promise of repayment from the university. Then the bank breaks that bond into many hundreds and hundreds of pieces and sells those pieces to wealthy investors or insurance companies or hedge funds. So when Mayhew hears that banks are struggling to sell bonds for historically black colleges, he wonders, is this just some apocryphal

Wall Street story or is it true? Luckily, there's a simple way to test this. The price the bond trader charges is called the gross spread. If it's super easy to find buyers for the bonds, the gross spread is likely to be low. If the trader has difficulty finding buyers, the spread can get quite pricey. So if the bonds from black schools are harder to move than bonds from white schools, the spread should be higher. And are they? Yes, they are. Mayu and his colleagues compared black schools with non-HBCUs that were otherwise identical. The spreads for the historically black schools were 20% higher. In the bond market, a 20% difference in spreads is huge. Mayu and his co-researchers then went one step further. They zeroed in on three states in the Deep South.

And those three states were? So those are Louisiana, Mississippi, and Alabama. And what's the difference in spread in those states? Not 20%. 37%. Mayu and his colleagues have given us a scientifically rigorous measurement of just how much of a burden black institutions face in the real world.

You've put a number on reputation. What is the reputational cost of being a school that overwhelmingly accepts and educates black kids? In Louisiana, the reputation cost is 30%. That's one way to think about it, but I would say it's the racial component of reputation. A spread is a reputation score, isn't it? Investors don't know the precise ins and outs of all the institutions they're asked to buy. How could they? There are literally hundreds of thousands of bonds on the market at

any given time. So the investors proceed on a feeling, a feeling about how worthy and how safe and how legitimate the institution getting the loan is. You see a black school and you say to yourself, I don't value that HBCU quite as highly as the white school across town. So you give the black school a discount, which is what prejudice is, by the way, a discount. The black life does not matter as much as the white life. And who are these people imposing a discount on black schools? They're wealthy, successful people who are simply checking a box. I like this bond. I like this one a lot. I don't like this one. They are, I suspect, not a whole lot different from the college administrators

who assign a number to their peers for the U.S. News rankings every year, motivated by the same unconscious biases. I like this school. 4.2. I like this one a lot. 4.S. news rankings every year, motivated by the same unconscious biases. I like this school, 4.2. I like this one a lot, 4.5. I don't like this one, 2.4, even though I don't really know anything about it. So if you were the president of a historically black college in one of those states where the reputational discount for being black is 37%, how do you think you'd feel about your peer assessment score? I thought I'd call Walter Kimbrough, president of Dillard University, a historically black college in Louisiana. And so when I start to look at the formulas and dig into it,

I was like, this isn't a real good measurement of what we do. In fact, it's a measurement of privilege. Dillard's reputation score is 2.6. That's not good. You can't ever get to the top of the U.S. news rankings with a peer assessment score of 2.6. Maybe that sounds defeatist, but I mean, it's... It's not defeatist, it's realistic. It's realistic, yeah, it is. It's the world we're living in. Nobody's going to just go and say, this school with all these black people is great. In their haste to compete with Time and Newsweek,

way back when, maybe all U.S. News did was create a system that allowed the presidents and deans of every college in the country to assign a number to their prejudice, to disguise mythology as methodology. Next week on Revisionist History, we take up the cause of Dillard University. And I enlist my old friends at Reed College to help me out.

So that is how we can get Dillard into the tippy-top tier of the liberal arts colleges. So it's Williams, Amherst, Dillard, Swarthmore. Yes. Revisionist History is produced by Mia LaBelle, Leman Gistu, and Jacob Smith, with Eloise Linton and Ananai. Our editor is Julia Barton. Original scoring by Luis Guerra, mastering by Flan Williams, and engineering by Martine Gonzalez. Fact-checking by Amy Gaines. Special thanks to the Pushkin crew, Hedda Fane, Carly Migliore,

Maya Koenig, Maggie Taylor, Eric Sandler, Nicole Morano, Jason Gambrell, and of course, Jacob Weisberg. I'm Malcolm Glabaugh. If you love this show and others from Pushkin Industries, consider becoming a Pushnik. Pushnik is a podcast subscription that offers bonus content and uninterrupted listening for $4.99 a month. Look for Pushnik exclusively on Apple Podcasts subscriptions. You have three flavors. All these nasty, nasty-licious, and nasty-vicious. Okay, guys. Hold on. We have to do this simultaneously.

This is it? Yeah. All right. Okay. One, two, three. Go. Oh, I like that. That one's hotter.

It comes on glad and steady. Oh, it does, doesn't it? It's spicy. Go Ali! Go Ali! This is really good. It's good. Hold on.

Are you okay? It's very spicy for a very long time. Guys, am I... It's not eating me as hard as it is. I'm taking a second bite it's spicy though

Martine's struggling by the way where's my Harvard hot sauce I don't see any Harvard hot sauce no one's sending me hot sauce from New Haven are they no