Interpreting the Data

This past week the owner of the Tesla electric car company got into a fight with a reporter for the New York Times over a somewhat negative article about his road test of the vehicle. To prove his point that the reporter had not conducted a fair test, the owner released all the telemetry data the car had collected during the trip.

Which might have been the end of things except that a writer for the Atlantic looked at the same data and came up with a different interpretation. And the Times own public editor weighed in with analysis looking at both sides and not necessarily supporting either of them.

Although I saw a little of this story pass by in my info stream, the larger point of all this didn’t really register until reading David Weinberger’s post yesterday.

But the data are not going to settle the hash. In fact, we already have the relevant numbers (er, probably) and yet we’re still arguing. Musk [Tesla owner] produced the numbers thinking that they’d bring us to accept his account. Greenfield [the Atlantic reporter] went through those numbers and gave us a different account. The commenters on Greenfield’s post are arguing yet more, sometimes casting new light on what the data mean. We’re not even close to done with this, because it turns out that facts mean less than we’d thought and do a far worse job of settling matters than we’d hoped.

Electronic data tracking on a car – where it went, how fast it got there – yields very straightforward numbers and, in this case, still produces different interpretations of the meaning of that information.

Now I’m sure the Tesla is a very complex piece of technology. But it’s not nearly as complicated as understanding and managing the growth and learning processes of a human being, especially kids in K12 schools.

However, using much less precise measuring systems than those in the car, we collect far fewer data points on each student here in the overly-large school district during each year.

We then accept those numbers as a complete and accurate representation of what a student has learned and where they need to go. That very narrow information stream also leads to even more narrow judgements on schools (success/failure) and now we’re starting to use the same flawed data to assess the quality of teachers.

In his post, Weinberger is celebrating the open and public way in which the dispute between Tesla and the Times is being played out, with many different parties lending their voice to the discussion of how to interpret the data.

How often do we ask even the subjects of our testing to analyze the data we’ve gathered from them? Why are then not included in the development of the assessment instruments? When do we include at least a few of the thousands of other factors that affect student learning in our interpretations?

I’ve ranted before in this space about the increasing amount of resources being poured into data collection and analysis here in the overly-large school district (and elsewhere). But it’s the absolutist approach to the analysis of those numbers that may be an even larger disservice to our students than wasting their time.

Coming Soon: Super School!

A new charter school is applying to open in DC and Jay Mathews is all excited because it combines two of his favorite education reform concepts: charters and AP.

According to Mathews, the original version of this model in Tucson, Arizona “has become by one measure the sixth most challenging high school in the country”.

What is that “one measure”?

Why it’s Mathews’ own creation, the “challenge” index, by which he compiles an annual list of “best” high schools based solely on a ratio of number of AP tests (and other college-level exams) taken to graduates.”

It’s one reason why he loves the DC area.

This region has the highest concentration of AP and International Baccalaureate courses and tests in the country. Some local schools, like Richard Montgomery High School in Rockville, do almost nothing but AP and IB testing every May.

Can’t imagine a better way for kids to spend one month out of the year than doing “almost nothing” but testing.

Anyway, the point of Mathews’ column, beyond taking yet another opportunity to express his adoration for charter schools and AP tests (a two-fer!), is to speculate on whether a charter high school based on the AP program will succeed in Washington, DC.

I’m pretty sure it will.  In the same way that KIPP and other high profile charter programs have succeeded in the city.

By attracting a relatively small, highly select community of students with very motivated parents and siphoning off money from the public schools, while supplementing those funds with large pots of corporate donations and grants (which in this case they’ll need to pay actual living wages for AP-trained teachers).

It’s how all charters demonstrate that they can do a better job than public schools for the same cost.

Except that most don’t.

An Obsession With Testing

For all the talk about change during the 2008 presidential campaign, one policy area in which the Obama administration differs very little from that of his predecessor is education.

In this morning’s Post, Dana Milbank discusses the similarities between the two.

Unfortunately, his focus is almost entirely on the political consequences, which, as we all know, is far more important than any impact of the policy itself.

Easier to write too since most political analysis these days seems to be based on personal opinion, the louder the better.

Anyway, Milbank does manage to make a few relevant points.

But in education, the Bush-Obama comparison is spot on. If anything, Obama has taken the worst aspect of Bush’s No Child Left Behind education law — an obsession with testing — and amplified it.

Obama has expanded the importance of standardized testing to determine how much teachers will be paid, which educators will be fired and which schools will be closed — despite evidence that such practices are harmful. In the process, he’s offended just about all the liberals involved in or advocating for education without gaining much support from conservatives. (emphasis mine)


There’s nothing wrong with testing*, but when you use tests to determine pay and job security, you inevitably induce teachers to turn children into test-taking automatons, not the creative thinkers that have been the most valuable product of American schools. Test obsession won’t help the bad schools, and it will wreck the good ones. (emphasis mine)

“The curriculum will be narrowed even more than under George W. Bush’s No Child Left Behind,” New York University education professor Diane Ravitch, an education official in George H.W. Bush’s administration, wrote of Obama’s education policy in a piece for the Huffington Post. “There will be even less time available for the arts, science, history, civics, foreign language, even physical education. Teachers will teach to the test. There will be more cheating, more gaming the system.” The tests, she said, are “simply not adequate” to separate good teachers and schools from bad.

We can only hope that “Obama’s erstwhile allies” who Milbank claims are now pushing back on his Bush-like education policies are able to alter that all-consuming effort to graduate “test-taking automatons”.

*A more accurate statement would be there’s nothing wrong with assessment.