Someone else doesn’t think much of Jay Mathews’ Challenge Index. An organization called Education Sector has released a report calling the Index a “seriously flawed measure of overall quality”.
The Index is Mathews’ ranking of American high schools based solely on a ratio of the number AP and IB tests taken by students to the number of graduates. His annual list is reported in Newsweek (often making the cover) and in many other news outlets.
So what’s wrong with that? As I’ve ranted before, this is a far too simplistic approach to judging school quality, one which completely ignores anything but the attempt.
Creating a list like this wouldn’t be so bad if it wasn’t given such high credibility by the news media. School quality is a complex issue and the flurry of publicity that surrounds the Challenge Index masks many other factors that need to be addressed.
Using publicly available student performance data, we found that many schools on Newsweek’s 2005 ranking have glaring achievement gaps and high dropout rates. By presenting them as America’s best, Newsweek is misleading readers and slighting other schools that may in fact be better than those on Mathews’ list.
While some schools on Newsweek’s list may be among the best in the nation, a closer look at the data reveals that many do not meet a reasonable definition of a good high school. Indeed, some of the schools on the list have such significant achievement gaps that they should be on a list of schools needing improvement rather than on one for best schools. And it is not merely the case of a few outliers. In fact, so many of the schools on the list have such significant gaps in achievement among their student subgroups that it calls into question the entire Newsweek enterprise.
There is much more to the report, with their findings backed up by large amounts of data on student achievement. Far more than is behind the Challenge Index, which contains none at all.
It would be nice if this kind of information was presented on the front pages of national magazines and newspapers. It probably won’t happen, however, since that would require readers to understand more than simple arithmetic.
Update (3/8): Not being the swiftest blogger on the net, I completely spaced the fact that Education Sector is the organization behind Eduwonk, one of the best sources of education news and commentary (mostly right :-) on the web.