Posts Tagged → value add

More Statistical Crap

More on the statistical crap known as Value-Added Measure (VAM), the teacher evaluation system that’s supposed to incorporate the improvement in learning (aka standardized test scores) experienced by students in their classes.

This time some teachers and their local union are suing the Houston Independent School District over the way this process has been applied.

There’s not much new in their criticism of this “badly flawed method of evaluating teacher effectiveness”, one that’s already been challenged by people who actually understand the analysis of data, the American Statistical Association.

However, one highly inaccurate statement posted to the district website in support of using VAM jumped out at me:

Nothing matters more to student success than teachers.

While good teachers certainly can make a difference, there are many studies showing that a student’s socioeconomic status and parents are far more important factors of their success in school.

And, which is important in this case, there’s very little credible research1 supporting VAM as a teacher evaluation tool or as a means of improving student learning (again, aka standardized test scores).

If It’s Crappy Enough for Bill…

Vanity Fair this month has a long look at how Microsoft has managed to screw up a lot of things about it’s business over the past decade or more. Having been deeply involved with personal computing since the time the company began1, I found the whole thing fascinating. Your mileage may vary.

However, there is one section of the article that I think should interest anyone involved with public education.

In it the author discusses what she sees as a major cause of Microsoft’s lack of innovation and subsequent decline, a personnel evaluation system used within the company called “stack ranking”.

Every current and former Microsoft employee I interviewed–every one–cited stack ranking as the most destructive process inside of Microsoft, something that drove out untold numbers of employees. The system–also referred to as “the performance model,” “the bell curve,” or just “the employee review”–has, with certain variations over the years, worked like this: every unit was forced to declare a certain percentage of employees as top performers, then good performers, then average, then below average, then poor.

“If you were on a team of 10 people, you walked in the first day knowing that, no matter how good everyone was, two people were going to get a great review, seven were going to get mediocre reviews, and one was going to get a terrible review,” said a former software developer. “It leads to employees focusing on competing with each other rather than competing with other companies.”

Now the CEO who signed off on using this “destructive process” (that would be Bill Gates) is considered a “visionary” leader in the education reform movement. And he and his billion dollar foundation are advocating for several other similarly adversarial assessment programs for teachers such as merit pay and “value added” rankings.

Assessment programs which continue to envision classrooms as discreet spaces sealed off from the rest of the world, and teachers as independent contractors whose work is the only influence on student achievement (aka test scores).

I’m certainly not the first person to make the connection between Microsoft’s stack ranking and Gates pushing the idea that teacher assessment should be a more competitive process. But that point needs to be repeated as often as possible.

The bottom line that Gates and others miss entirely in their efforts to “reform” education is that schools are not businesses and those corporate practices cannot, and should not, be applied to the process of teaching and learning.

Especially an evaluation system that has been a major contributing factor to screwing up what was at one time the most valuable company in the US.

1 Although, in all those many years, I’ve never actually bought a Microsoft product or anything containing one. I do have Windows and Office on my MacBook Pro but those licenses belong to my school system.

Misusing Math

We do a crappy job of teaching mathematics in this country.

How else do you explain the proliferation of lotteries and casinos offering worse than lousy odds or the complete lack of understanding, by both the media and the general public, of anything involving probability and statistics?

Even worse, according to a mathematician in a call to speak out directed at his colleagues, is the misuse of math when it comes to making public policy.

But the most common misuse of mathematics is simpler, more pervasive, and (alas) more insidious: mathematics employed as a rhetorical weapon–an intellectual credential to convince the public that an idea or a process is “objective” and hence better than other competing ideas or processes. This is mathematical intimidation. It is especially persuasive because so many people are awed by mathematics and yet do not understand it–a dangerous combination.

He goes on to explain that one of the more recent and egregious examples of this “mathematical intimidation” shows up in the debate over the “value-add” concept of teacher evaluation.

Value-added modeling pops up everywhere today, from newspapers to television to political campaigns. VAM is heavily promoted with unbridled and uncritical enthusiasm by the press, by politicians, and even by (some) educational experts, and it is touted as the modern, “scientific” way to measure educational success in everything from charter schools to individual teachers.

Yet most of those promoting value-added modeling are ill-equipped to judge either its effectiveness or its limitations. Some of those who are equipped make extravagant claims without much detail, reassuring us that someone has checked into our concerns and we shouldn’t worry. Value-added modeling is promoted because it has the right pedigree – because it is based on “sophisticated mathematics.” As a consequence, mathematics that ought to be used to illuminate ends up being used to intimidate. When that happens, mathematicians have a responsibility to speak out.

At the top of the list of many problems with this concept, mathematical and otherwise, is the fact that the value-added model is based on changes in test scores, which themselves are unreliable predictors of student learning and subject to many factors other than the quality of teaching.

There is much more to the article and it’s worth taking the time to read (dredge up some memories from that stats class you took once upon a time :-).

Especially since this value-added scheme is not going away, with the LA Times going back for seconds on publishing “value” ratings on teachers in their elementary schools, and my own state of Virginia pushing a new teacher evaluation system based in part on how their students score on the state SOL tests.

All of this educational malpractice by politicians and other “experts” put mathematics in the same league with evolutionary biology, climate science, and physics when it comes to making public policy.

If you don’t understand the science, just make up your own facts to fit what you already believe.

Short Term Thinking, Long Term Failure

Any other public school educators getting tired of being told you’re overpaid?

State and local politicians are loudly claiming teacher pay and benefits are bankrupting their governments, and a long parade of “experts” on the talking heads channels have declared us to be wildly over compensated (at least compared to the divine private sector), greedy, and don’t forget lazy.

Many of those same people make lots of noise when international comparisons of test scores show our students ranking in the middle of the pack but ignore other statistics showing teacher pay in the US also ranking low when compared to Finland, Korea and other industrialized countries.

They also want to change teacher compensation formulas to pay them based on changes in test scores, the so-called “value add” system, despite there being no evidence that such formulas (or the tests themselves for that matter) have any validity when it comes to student learning.

On the other side of things, one lone voice, New York Times columnist Nicholas Kristof, tries to make the case for raising teacher salaries, starting with the common reasoning that, if we want to improve schools, we need to attract more talented, better educated people to the profession.*

However, his better argument, one that needs to be made more frequently, is the long term benefits that comes from investing in good teachers and good schools.

Recent scholarship suggests that good teachers, even kindergarten teachers, increase their students’ earnings many years later. Eric A. Hanushek of Stanford University found that an excellent teacher (one a standard deviation better than average, or better than 84 percent of teachers) raises each student’s lifetime earnings by $20,000. If there are 20 students in the class, that is an extra $400,000 generated, compared with a teacher who is merely average.

During the economic mess of the past three years, we were told that the million dollar bonuses being paid by the likes of AIG, Bank of America, and other financial service companies were necessary so that they wouldn’t lose talented people.

The same “talent”, of course, that created the incredible risky investments that cause the crash in the first place.

Anyway, you rarely hear anyone saying anything about paying to retain talented teachers and principals, despite all the talk about them being the key factor to building a “world class” school system.

That’s because most people running business and government in this country are totally fixated on short term gains – the next quarter’s profits, the next election – rather than paying for long term public investments, ones that pay off in 10, 20, 30 years.

It’s why our roads, bridges, and public transportation systems are falling apart, why we continue to rely on increasingly expensive and scarce fossil fuels for energy, and why our school system is stuck with a 1950’s industrial model in this hyperlinked information age.

Considering that too many people in this country are worried about their day-to-day issues, I’m not optimistic about the short term thinking of our “leaders” changing anytime soon.

But something about our national attitude about planning for the future does need to change.

*Kristof weakens his case somewhat with the line “as measured by SAT scores” but that’s another rant.

Still Not Much Value Added

Well, that didn’t take long.

Last month preliminary results from a study sponsored by the Gates Foundation were released that seemed to support the validity of using a “value added” system to evaluate teacher quality.

This month, someone else looking at the same data came to a very different conclusion.

But Economics Professor Jesse Rothstein at the University of California at Berkeley reviewed the Kane-Cantrell report and said that the analyses in it served to “undermine rather than validate” value-added-based measures of teacher evaluation.

“In other words,” he said in a statement, “teacher evaluations based on observed state test outcomes are only slightly better than coin tosses at identifying teachers whose students perform unusually well or badly on assessments of conceptual understanding. This result, underplayed in the MET report, reinforces a number of serious concerns that have been raised about the use of VAMs for teacher evaluations.”

“A teacher who focuses on important, demanding skills and knowledge that are not tested may be misidentified as ineffective, while a fairly weak teacher who narrows her focus to the state test may be erroneously praised as effective.”

So, who’s right? This being education research, it probably means neither and both.

However, there’s one question that never seems to be addressed in all this research about teacher quality.

Are constant waves of standardized, mostly multiple choice exams – accompanied by a narrow school focus on test prep – the best way to improve student learning?

I’m gonna vote no.

Â