Questioning Dubious Statistics

BBC More or Less Postcard

More or Less is a radio programme1 and podcast produced by the BBC World Service. The weekly show tries to make sense of the statistics presented in popular media (including the broadcasts of their own organisation1) in a way the average educated listener can understand.

As you might expect, a common thread in the podcast is whether the numbers reported in stories about studies, polls, and surveys are accurate and used appropriately. Spoiler alert: they often are not.

In a recent “bonus” podcast, the host offers a short debunking guide that would fit on a post card2 from his holiday at the shore. “How to question dubious statistics in just a few short steps.”

The whole thing is worth ten minutes of your time. If you teach math to high school students, you may even want to play it for them.

However, if you’re very short of time, the final step is, for me, the most important idea presented.

Number 6: Be Curious.

If a statistic is worth sharing, isn’t it worth understanding first?

Forget that nagging feeling that says you might just be spoiling a good story. Facts matter… but facts are also fascinating.

Treat them as puzzles. Treat surprising or counterintuitive claims, not with suspicion nor open arms but as mysteries to be solved. It’s fun.

And they close with this reminder.

Hopefully, with this postcard as your guide, you can step into a world of statistical adventure. Because it’s not just about winning arguments, it’s about being curious. The world, after all, is a fascinating place.

Whether you consider statistics “fun” or an “adventure”, the advice is solid. Be curious, some would say skeptical, about the numbers constantly being thrown at you in the news and your social media feed. Very often the story behind them is far more interesting, and different, from what has been presented in the headline.


If you listen to podcasts, More or Less is a good one to add to your playlist. I would have embedded a player here for the episode but the BBC doesn’t allow those of us outside the UK to do that kind of thing.

1. British show, British spelling. :)

2. For you kids out there, postcards were something your parents (maybe grandparents) sent from locations where they were on vacation in the days before Facebook, Instagram, Twitter, and the rest. It was a slower method of trying to impress their friends and relatives. Or maybe make them jealous.

Look Behind the Graph

According to many excited retweets in my stream today, the number of females and “underrepresented” minorities taking AP Computer Science tests is way up. Like double up according to USA Today.

 

Now, I don’t want to rain on anyone’s parade (although a little rain today might be nice :-), but I’m also bothered by the unquestioned acceptance of statistics in the form of dramatic bar charts. So let’s take a closer look at that chart.

Notice that the bar to the far right combines two AP exams, the standard AP CS A exam, first offered in 1984, and the new AP CS Principles exam which was first administered in May of this year.

If you remove that new program, there is still a growth in both females and “underrepresented” minorities1 in the CS A class, just not nearly as dramatic as reported in the headlines. Even so, a very positive sign. It’s also positive that so many students are enrolling in the Principles course, which is far more accessible to those who are not necessarily looking at CS as a career path.

However, also missing from the analysis, both in the article and tweets pointing to it, is any information about how many of those new students actually did well on those AP exams. Five is the best score but three or four would also be respectable. We could discuss some value in earning a two.

I call this the Jay Mathews syndrome: attributing all success to just taking an AP exam, regardless of any measure of actual learning demonstrated by it.

Anyway, I mean absolutely no disparagement of the efforts to encourage more female and minority students to at least sample the study of computer science. And hopefully we’ll see this kind of steady increase when AP statistics are released next summer.

But anytime someone reports huge statistical increases, or decreases, especially in anything dealing with education, be skeptical and take a closer look. The story is likely much more complicated than the graph out front.

More or Less Accurate

Did you know that 65% of the jobs that will be available to our elementary students when they graduate have not even been invented yet?

cartoon of school as an assembly lineOr maybe you’ve heard the statement that 60% of children currently in kindergarten will be employed in jobs that don’t exist today?

Or possibly you know for sure that some similar large percentage of jobs our students will be doing at some vague date in the future (2035 seems to be very popular) are yet to be created?

More or Less, a wonderful podcast from the BBC World Service, is all about investigating statistical claims like this, and in a recent segment, they tried to track down the source of this particular number. They weren’t very successful.

So, will our children leave school into a world in which 65% of the jobs are brand new inventions? I doubt it. Nobody can prove this claim for sure, and we’ve not found any explanation of where the number came from or what the logic was behind it. Sources lead from the UK out to America and Australia and then hit dead ends.

One of the primary reasons that pundits and politicians toss around statistical myths like this is to reinforce their particular efforts to reform the education system. Scaring people with visions of millions of unemployable students might just work, right?

However, this discussion about future jobs dictating what students learn in their K12 years is totally wrong.

Certainly the world is changing in very unpredictable ways. And our education system, which in many ways is stuck in the 1950’s, needs to be restructured to reflect that unpredictability.

But school should be about much more than job training. The emphasis should be on kids gaining some basic life skills, and spending most of their valuable time exploring a variety of interests and ideas. If they discover a career in the process, terrific. But making that the primary goal of school is a crappy idea.

So, what does all that look like? Good question, and one that we should be discussing, without all the bogus statistics. And the More or Less people have a great suggestion for something to include.

As for what it tells us about what children should be taught, that’s far from obvious too. But here’s a suggestion. What about trying to teach some basic statistical common sense? It’s a useful skill, and our children can’t possibly be worse at it than than the grownups.

Excellent!

Bad Questions

I don’t often agree with Jay Mathews.

Mathews was the long time education reporter for the Washington Post and now writes a weekly column for the paper. I’ve written many posts about his adoration of KIPP charter schools, the unrequited love of the Advanced Placement program, and especially his Challenge Index, his annual ranking of high schools based solely on the number of AP tests taken by students.

However, his post from last week, which is not really about education, is one I can get behind. In it, he highlights two popular questions used by polling companies that produce little to no useful information, calling them dumb and deserving of ridicule.

One comes from the annual poll done by Gallup for Phi Delta Kappa, a professional association for educators. It asks people “What grade would you give your local public schools, the nation’s public schools and, if you have children at home, their schools?” The results are pretty much the same every year.

Since 1985, the results have been consistent. Respondents award their children’s specific schools the highest grades, with about three-quarters giving A’s and B’s. About half of them give their local schools A’s and B’s. About a fifth give A’s and B’s to the nation’s public schools.

My kid’s school is great. But those other schools, and the national education system, are crap. Of course PDK asks other questions about many aspects of education (in 2015, 64% of respondents said there was too much emphasis on standardized testing) but it’s that top number that is most widely reported by the headline driven media. And it’s too often used to perpetuate the “failing public schools” narrative. If people tell a survey taker something, it must be true.

The other poll question Mathews ridicules is even worse.

The right-track-wrong-track question is even more aggravating because it is so often extolled as a mark of voters’ desires. The latest Washington Post-ABC News poll uses this question: “Do you think in this country things are generally going in the right direction or do you feel things have gotten pretty seriously off on the wrong track?”

Over the 44 years that polling companies have asked that question “results have been on the negative side nearly 90 percent of the time”. Again, the surveys include other related questions but it’s the right-track-wrong-track number that gets put into the headlines and analyzed by pundits who probably haven’t bothered to read any of the supporting data or metrics used to select the sample.

Both of these examples are why K12 math instruction needs to include a whole lot more statistics. Maybe if more people reading these headlines actually understood something about polling methodology they might push back and question both the results and the reporting of them. Some of those better informed graduates might even become presenters on those talking heads channels and bring those questions to their jobs.

Powerball Math

It has been sorta fun watching all the hype surrounding the billion dollar Powerball lottery. Lots of stories about what could be done with that much money (greatly reduced by taxes and other factors), with reporters and others trying to make sense of the odds of winning, too often comparing them to those of being struck by lightning.

[326/365] Lottery Money2However, the best piece in the flood of coverage came from Wired, explaining The Fascinating Math Behind Why You Won’t Win Powerball. Including how the non-profit organization behind Powerball recently changed the format to make it much harder to win the top prize, and easier to win the pocket change prizes.

As I’ve previously ranted about, this is the kind of math that should be at the foundation of our K12 school curriculum. Real world, interesting, practical,2 stimulating lots of questions with connections to other areas of study. Instead of sending every student through the school theoretical math tunnel heading straight for Calculus, a subject used by very few adults.