The Secret Sauce Behind College Rankings

NN Social Media Blog Post the 2021 Best Colleges Rankings 28 Sep 2020 2 825x510

In a recent segment of his Revisionist History podcast, Malcom Gladwell takes on the Lord of the Rankings. Also known as the US News & World Report annual list of the “best” colleges and universities in the United States.1

The ranking was first published in 1983 as a tool to raise their profile (and sell magazines) in the pre-internet era when they were a distant third to Time and Newsweek in the category of weekly news magazines.

Fast forward to 2021.

The rankings became an institution… Time and Newsweek have basically disappeared. The rest of US News has basically disappeared. But the rankings are as strong as ever.

Gladwell was curious to know why so many people take the rankings seriously and, more importantly, how they are compiled in the first place. US News, of course, wasn’t going to disclose their list-making methods.

Instead he found a professor from Reed College in Oregon, one of the institutions who refuses to play the ranking game, who was curious to discover the secret sauce behind the lists.

She and some of her undergrad students were able to very accurately duplicate the algorithm being used by the editors. And they discovered that one particular factor, something called the Peer Assessment Score, carried a great deal of weight in determining a college’s place on the list.

That particular component is determined using a survey sent to a few officials2 at every school, asking how they feel about the other schools in their peer group.

The surveys arrive by email, once a year, to everyone who matters in American higher ed. If you’re at a liberal arts college, you’re given a list of all 222 other liberal arts colleges in your category. If you’re part of a big university, you get a list of the 388 other big universities in the United States.

Your job is to rank the schools on your list on a scale of one to five. Five being “amazing”, one being “there’s a serious problem here”.

The recipients are supposed to base their assessment on how they view the other school’s “reputation for undergraduate academic quality”. Which sounds a little squishy for what’s supposed to be a mathematical formula.

Anyway, the podcast episode is an interesting, fun story, one that plays a little like a geeky detective story, and worth forty minutes of your time. 

However, there are many larger questions that arise from this discussion, starting with why does anyone (other than university marketing departments) buy into this game? An experience as complex as college (and education in general) involves too many components to produce a simple list of the “best”.

But they’re profitable, so I’m sure US News and others will continue to assemble them, regardless of how subjective they really are.

Finally, while an intelligent defense of this kind of college ranking system might appear to be a very difficult task, there’s always someone who will make the attempt.

And standing on this particular bulwark we find none other than long-time Washington Post education columnist, and lord of the high school ranking, Jay Mathews.

Writing on the About the Grade blog on the PDK website, Mathews challenges the conclusions of Gladwell and many other critics that “school ratings and rankings increase segregation and inequality”, saying he has “never seen any evidence of that”.

While he admits there are flaws in the US News ranking, Mathews claims it remains a very useful tool for parents and students in choosing a college to attend. He also flatly states (offering no evidence) that attending one of the colleges at the top of the lists “rarely has any effect on success in life”.3

Then, not surprisingly, he spends the remainder of the post promoting his own work and defending his equally flawed “challenge” index of “best” US high schools.

Some things never change.

The photo is from a page about the 2021 rankings on the Princeton Review website, a company that makes lots of money from selling test prep services to parents and students. Not sure why the kid is so happy.

1. Beginning in 2014, US News also publishes a global version of their rankings. Four US universities, Harvard, MIT, Stanford, and U Cal Berkeley, are always in the first four places.

2. The president, the provost, and the enrollment manager or the head of admissions at the school.

3. Which seems like a pretty odd defense of the rankings themselves.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.