I guess the problem with international assessments, the ones that show US kids doing poorly compared to their peers in other countries, is that we’re using the wrong ones.

At least according to the expert Jay Mathews interviewed.

He says the PISA (Programme for International Student Assessment (PISA)) is a bad one because it doesn’t fit “the way U.S. students are taught” and aligns to the “losing” side in the debate over how to teach math.

Which is to say that the designers of the PISA expect that schools are using curriculums that “make math instruction more relevant to the real world, and emphasize mathematical reasoning more than calculation”.

How dreadful to expect that kids should actually be able to understand and apply math concepts!

PISA includes questions like this one:

For a rock concert a rectangular field of size 100 m by 50 m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?

A. 2000

B. 5000

C. 20000

D. 50000

E. 100000

Mathews thinks this is a bad question because it involves too many variables, such as the fact that “some people don’t like to get close to at concerts”.

Certainly it’s a lousy item when students are expected to locate the one and only “right” answer, according to the test writers who have been programmed to create just the right kinds of distracters (been there, done that :-).

However, it’s an excellent question when you want them to consider all those different, and sometimes messy, factors that clutter up problems here in the real world.

And if we also expect students to justify their answer, explain the logic they used to arrive at it, and use that interpretation as part of their assessment.

So, is the problem that we’re giving kids the wrong test?

Or that we’re not teaching to the right assessment?

I’m reminded of a portion of Neal Stephenson’s book, CRYPTONOMICON, where one of the characters gets sucked into this kind of problem on the Navy entrance exam, and becomes so absorbed by solving this kind of problem that he forgets to take the rest of the test.

The Navy puts this brilliant mathematician to work scrubbing floors.

These estimation problems always irk me. I just spent two days at ACL, a massive outdoor music festival, and 1 person/m^2 and 4 people/m^2 are both within the satisfactory range for the word “full”, depending on a variety of factors not addressed in the prompt. It seems like that’s the way estimation problems usually go: if you have real world experience, and it’s not the real world experience that the test-writers think you should have, then you suffer for it.

In the real world, when would this situation ever come up? Anyone who needs an estimation of crowd size is either going to have access to better data (a journalist could look at the crowd, ask an employee what capacity is or how many tickets were sold) or have specific skills in this direction (a fire marshal or roadie would have more experience to draw on when making the estimation).

I digress — this doesn’t really address your main point. I think we’d need a broader assessment of the assessment to answer those questions.