Look Behind the Graph

According to many excited retweets in my stream today, the number of females and “underrepresented” minorities taking AP Computer Science tests is way up. Like double up according to USA Today.


Now, I don’t want to rain on anyone’s parade (although a little rain today might be nice :-), but I’m also bothered by the unquestioned acceptance of statistics in the form of dramatic bar charts. So let’s take a closer look at that chart.

Notice that the bar to the far right combines two AP exams, the standard AP CS A exam, first offered in 1984, and the new AP CS Principles exam which was first administered in May of this year.

If you remove that new program, there is still a growth in both females and “underrepresented” minorities1 in the CS A class, just not nearly as dramatic as reported in the headlines. Even so, a very positive sign. It’s also positive that so many students are enrolling in the Principles course, which is far more accessible to those who are not necessarily looking at CS as a career path.

However, also missing from the analysis, both in the article and tweets pointing to it, is any information about how many of those new students actually did well on those AP exams. Five is the best score but three or four would also be respectable. We could discuss some value in earning a two.

I call this the Jay Mathews syndrome: attributing all success to just taking an AP exam, regardless of any measure of actual learning demonstrated by it.

Anyway, I mean absolutely no disparagement of the efforts to encourage more female and minority students to at least sample the study of computer science. And hopefully we’ll see this kind of steady increase when AP statistics are released next summer.

But anytime someone reports huge statistical increases, or decreases, especially in anything dealing with education, be skeptical and take a closer look. The story is likely much more complicated than the graph out front.

Challenging Clickbait

Last week the education RSS feed from the Washington Post was spammed with at least seven stories about Jay Mathews’ “challenge” index. Of course, they were all written by Mathews, who never misses an opportunity to tell you how he created this annual list of the “most challenging” high schools in the US.

So, these posts were not so much news as general clickbait.

In one of the articles, Mathews lets us know that this year is the 30 anniversary of the day this idea first popped into his head. Next year will be the 20th year since the Post and a then paper-based Newsweek magazine first published his list.

And I’ve been ranting about it in this space for almost three-quarters of that time. So I’m not sure what’s left to be said about this simplistic, headline-grabbing, mess. But I’ll say it anyway.

For those not familiar with the “challenge” index, here’s how it works: for each high school that will send him the stats,2 Mathews adds up the number of Advanced Placement (AP) and International Baccalaureate (IB) tests taken and divides it by the number of seniors who graduate. Any school with a score of 1 or higher goes on the list.

What? You were expecting more? Maybe like incorporating the number of students who actually passed the exams? Or other factors that go into making a successful high school beyond pushing kids to take more tests?

Mathews seems to think that his index has improved American education by pushing more schools into adopting the AP curriculum (after complaints a few years back, he grudingly included the IB program). Which assumes that those very limited programs, largely dictated by colleges and framed around the idea that college is the only goal of learning in K12, are appropriate for every student. It also ties nicely into Mathews’ love of charter schools, especially KIPP, many of which tightly embrace AP.

Then there’s the general idea in the public mind that this is a ranked list of the “best” high schools. I know, both Mathews and the Post will say that’s not the intent. They simply want to spotlight the schools that are “working hardest to challenge students from all backgrounds”.

However, that’s not how it works in the real world. Since the start, schools, especially those in the upper levels of the list, local media, public school critics, and others have trumpeted this “challenge” index as THE list of top US high schools.

For the Post, that also helps sell newspapers and magazines, and in the internet age, generates clicks.

A Very Simplistic Challenge

In this morning’s paper, the Washington Post wasted valuable space in the Metro section on Jay Mathew’s annual promotion of the fraud known as his “challenge” index. If you’re not familiar with this artificial ranking of school quality, take a look at some of my regular rants on the subject.

His premise in the column is that schools need to be more transparent, mostly by giving him their data on how many students take AP tests each year. Mathews is pissed that many private schools still don’t want to play along with his index and this piece is little more than him explaining why they are evading the noble purpose of his list.

However, I do agree with Mathews’ overall idea: schools do need to be more transparent. But with more than just numbers. We need to open schools by involving our communities in some basic discussions around what school is and should be.

Is the basic assumption, made by Mathews and many others, that every student should attend college a valid one? If so, is the AP program, created and marketed by a non-profit organization run by the colleges themselves, the best way to prepare them for that goal?

Are AP courses the best way to challenge students in high school, which is the core concept of Mathews’ index? And is publishing a list based only on the number of students who take a standardized test, ignoring completely their scores, a valid way to judge school quality?2

There are many more questions that need to be asked, as well as including other people who are not currently part of the conversation. Like students, who are most impacted by the decisions made by politicians, administrators, and teachers.

Bottom line is that preparing students for their future after high school graduation is a very complex issue. One that requires more options for students than just college. An issue that is far more complex than the simplistic approach promoted by Mathews, a columnist who gave up being a journalist many years ago.

Jay Demands the Data

I haven’t ranted about Jay Mathews’ writing in a while, in part because he’s no longer the primary education columnist for the Washington Post (fortunately, they chose Valerie Strauss to fill that role), but mostly because there’s really nothing new to discuss.

He loves the Advanced Placement program and is responsible for the “challenge” index, possibly the worst measure of high school quality ever created.2 What else is there to know about Mathews? He certainly doesn’t address many other education-related topics in his writing.

Anyway, his most recent column continues this AP romance by chastising a prominent local private school for not releasing to him data on how many students take Advanced Placement tests. So he can flatter them by including the numbers in the next round of the index scores, of course.

And that’s about it. Most of the piece is just Mathews complaining about not being able to include private schools like this one in his lame ranking of mostly public schools.

If he had only accepted the headmaster’s very rational justification for their policy, he would have had nothing to write about.

“We believe that parents and students are not helped by rating systems which purport to evaluate school quality based on test data,” Sidwell head of school Tom Farquhar told me. Independent Education, the local private school association, encourages all of its members to keep their numbers to themselves.

Good for them.

Two other minor points.

First, calling Sidwell Friends “our most famous private high school”2 because the president’s children and those of other politicians attend is a prime example of inside-the-beltway pretentiousness. In keeping with Quaker tradition, Sidwell likes to keep a low profile and I’m not sure they would consider themselves the most famous private school in the DC area, much less the whole US.

Second, the fact that his daughter graduated from Sidwell also lends a bias, and not a small dose of privileged attitude, to this column. In addition to letting us know that the Post must pay him very well since the school has always been one of the most expensive in a very pricy market.

Ok, I’ll now return to ignoring Jay Mathews.

I’m Right… Get Over It

Also in yesterday’s Post, Jay Mathews informed a group of parents here in the overly-large school district that he’s right, they’re wrong, end of story.

They want schools to preserve choices for their kids by maintaining the “the three-track system–basic, honors and AP/IB– in the county’s high schools” while Mathews proclaims “honors courses for all”.

However, as always, he is working from several flawed assumptions.

One is that it is a “well-researched fact that these days every student needs at least a college-prep curriculum” – with his “well-researched” link going to another of his columns about one report from Education Week backing his side of the discussion.

Certainly almost every high school graduate needs some kind of post-K12 education, but for many students there are better options than traditional four-year college program, and they need to understand those alternatives and the best way to prepare for them.

Another of Mathews beliefs is that every student will benefit from simply taking AP (or other college-level courses) in high school.  Never mind how they actually do in the class.  Just disregard whether they have the background, capability, or interest in the subject.

But does the talented writer really need Calculus, when a good understanding of basic mathematical concepts, including statistics, would serve them better? Would some students be better served with practical classes in mechanical engineering, rather than four years of laboratory sciences?

Finally, Mathews continues to assume that the AP program, an inflexible and unrelated set of courses designed to fit the traditional college model, offers the only possible solution to provide students with a good high school education.

And there’s no possibility that he could be wrong.