The Challenge, One Last Time

This week Jay Mathews once again unleashed on the world his annual ranking of the “top 9 percent” of American high schools in a list he calls the “challenge” index.

And once again I’ve been thinking of how to write about this far too high profile and extremely trivial approach to discussing school quality without repeating myself.

Looking over my past posts on the subject (really? that many?), I got nothin’ new.

Mathews thinks he has something novel by adding a “sampling of private schools as a way to compare private to public schools”, but, other than complaining about private schools hiding data from him, nothing about this tired exercise has changed.

His index is still based on computing a simplistic ratio of the number of AP and other college level tests taken at a school divided by the number of graduating seniors. Just ignore the number of students who actually pass those tests or any other factor of school quality you care to name.

In this week of challenge overload on his blog, Mathews also grumbles about similar competing “best high schools” lists published this month in Newsweek (former home of his index) and US News and World Report (are they still publishing on paper?), while still taking some credit for both.  But it doesn’t sound as either of the alternatives is any more substantive.

Ok, I’ve already given Mathews too many links concerning his overly-hyped ranking, both in this post and all the others.

I’m done.

This Index Just Won’t Die

When the Post company sold Newsweek for a buck last year, I was hoping it was the last we’d see of their annual cover story proclaiming the “best” high schools in the US based, a statistical exercise based on Jay Mathews’ “challenge” index. And that I could drop this as a topic to rant about.

Unfortunately, that was wishful thinking as I found the 2011 edition of this incredibly simplistic and misleading list stuck in the middle of my Sunday paper. For those who don’t get the Post, here’s the web version.

Other than the fact that the Post has rebranded the package since inheriting it from Newsweek, now calling it The High School Challenge, nothing here is new.

As always, the index is based on a simple ratio of the number of AP (and other college-level programs) tests taken to the number of graduating seniors and Mathews’ still believes this is a mechanism to improve high schools, by guilting them into challenging their students (which means pushing more kids into AP classes).

Doesn’t matter if the students are prepared or if such courses are appropriate for their needs. And how they score on the tests certainly doesn’t matter, only that they were taken.

Beyond the shaky conceptual and mathematical foundation for the index, is how the listing is interpreted. Although Mathews’ says he doesn’t intend this to be a measure of school quality, that is exactly how readers interpret it.

The simple numbers will be splashed uncritically across local papers and school web sites, ignoring the many other factors that go into a making a good high school experience.

And, in order to boost their numbers next year, even more schools will narrow the educational options of their students to only those prescribed by the AP people at the College Board.

However, one thing is different this year: Mathews finally has someone in the Post organization who is willing to challenge the validity of his index.

It would just be nice if Strauss’ pushback was given even half the exposure of Mathews’ high profile sloppy love letter to the AP program.

A Very Weak Challenge Defense

In his Class Struggle column today, Jay Mathews is promoting Newsweek’s annual ranking of “best” high schools and also attempts to defend his “challenge” index that was used to compile the bogus list.

Many people prefer rating schools by average test scores, but I consider that a measure of the student family incomes, not the quality of the schools,…

So, instead of using one narrow, inadequate measure of school quality, use mine.

… I get many messages from principals, teachers and parents who like this way of assessing schools.

My index is popular so the results must be valid.

The list gets about 7 million page views a year.

And we all know popularity on the web equals quality information.

An extremely weak defense for this simplistic, misleading system.

Challenging Credibility

I guess I didn’t stay away long enough to avoid Newsweek’s annual cover story defining America’s “Best” High Schools.

That “best” ranking, of course, is based on the tenuous (and that’s being generous) assessment tool known as the “challenge” index, which assigns each school a number based solely on the ratio between numbers of AP/IB/Cambridge tests taken and the numbers of graduating seniors.

No factoring in how well students actually did on those tests (or any other academic criteria). Ignore the quality of arts programs. Dropout rates are irrelevant. And forget completely about students in vocational or any other programs that don’t involve college prep.

Schools rise to the top of this pile if they get kids to take tests.  Lots and lots of tests.

Which results in totally meaningless scores that often produce headlines in local papers, sometimes for very strange (and somewhat amusing) reasons.  Such as this dichotomy in Houston:

Newsweek has come out with its latest ranking of the nation’s best high schools, and the Houston school district is crowing that a record number of HISD highs made it.

The usual suspects are there — DeBakey, Carnegie, Bellaire and Lamar — but joining the list this year are 11 others, including Waltrip, Chavez, Sharpstown, Milby and — WTF? Sharpstown?

The same Sharpstown that is on quite another list — HISD superintendent Terry Grier’s “Apollo 20” list of failing schools? (Lee HS, too!)

Newsweek says Sharpstown HS is among the best in the country while the superintendent says it’s one of the worst in his district.

Who’s right?

And I wonder how many other schools racked up enough tests given to make this farcical “best” list while still failing to educate the majority of their students.

New Decade, Same Lame Challenge

Front page of this morning’s Post, above the masthead, in space normally reserved for major, earth shattering events, comes the news…

headline.jpg

The 2010 “challenge” index for DC-area schools has been unleashed on the unsuspecting, and largely statistically clueless, public!

The method for computing this highly-publicized ranking of high schools hasn’t changed.

Divide the number of Advanced Placement, International Baccalaureate or other college-level tests a school gave in 2009 by the number of graduating seniors. Tests taken by all students, not just seniors, are counted.

Also not changed is the glorification of the taking of tests, while factoring in nothing about how student actually score on them.

As with the 2009 release, the list includes something called the Equity and Excellence rate, defined as “the percentage of all seniors who have had at least one score on an AP, IB or Cambridge test that would qualify them for college credit”.

Which is also not an entirely accurate number since colleges make their own decisions as to what score on an AP test will earn credit. Or whether the student will get a pass on taking a similar level prerequisite course instead of credit.

So, what exactly is the purpose of the assembling the “challenge” index in the first place?

The rating is not a measure of the overall quality of the school but illuminates the one quantifiable factor that seems to reveal best the level of a high school’s commitment to preparing average students for college. [my emphasis]

The ONE quantifiable factor. Love to see the study supporting that contention, much less the concept that college is the best goal for every student.

While the Post seems to be avoid a “best” tag, it remains to be seen if Newsweek (owned by the Post), when they likely publish the the national version of the index in May, will refrain from billing Mathews’ list as the “nation’s best high schools” as they have in the past.

Ok, I know it’s probably a hopeless cause to continue ranting about this incredibly shallow assessment of high school quality year, after year.

Especially since both politicians and the press seem to be obsessed with reducing everything done in school to simple, headline-friendly numbers, something for which the “challenge” index is tailor made.

However, it would be great if more people would take a critical look at this and other hyper-simple schemes for assessing the complex process of teaching and learning.


By the way, I thought you added the possessive to a name ending in ‘s’ by simply adding an apostrophe. Or am I wrong that the proper punctuation is supposed to be Mathews’ list not Mathews’s list? I’m sure I make plenty of grammatical errors around this place, but I have an excuse. There are no highly trained and paid copy editors around here.