Any book with a title like that is worth looking into, right?
In a recent interview, the author, Cathy O’Neil, says she started looking into how data and math was being applied to a variety of human processes following the financial crisis of 2008, in which she had a “front-row seat”. Her research found that the “very worst manifestation” of those applications was “kind of a weaponized mathematical algorithm”.
One of the first of those algorithms she investigated was the “value-add” model for assessing teachers being used in New York City and other districts. This is the process where student test scores are mixed with other data to determine which teachers are given raises, and which should be fired.
However, O’Neil, a data scientist and “former Wall Street quant”, someone who might actually understand the mathematics (and explain it to the rest of us), was denied access.
It’s opaque, and it’s unaccountable. You cannot appeal it because it is opaque. Not only is it opaque, but I actually filed a Freedom of Information Act request to get the source code. And I was told I couldn’t get the source code and not only that, but I was told the reason why was that New York City had signed a contract with this place called VARK in Madison, Wisconsin. Which was an agreement that they wouldn’t get access to the source code either. The Department of Education, the city of New York City but nobody in the city, in other words, could truly explain the scores of the teachers.
It was like an alien had come down to earth and said, “ Here are some scores, we’re not gonna explain them to you, but you should trust them. And by the way you can’t appeal them and you will not be given explanations for how to get better.”
O’Neil says similar secret formulas are used by financial institutions to determine who can borrow money, by courts to decide who goes to jail and how long they will stay there, by Google in presenting the results of your last search, and many more. Some of this activity is benign and some is extremely manipulative.
But which is which? And what do we do about it?
The very first answer is that people need to stop trusting mathematics and they need to stop trusting black box algorithms. They need to start thinking to themselves. You know: Who owns this algorithm? What is their goal and is it aligned with mine? If they’re trying to profit off of me, probably the answer is no.
I’m doubt many people actually “trust” math, certainly not after the classes they took in school. But I’m pretty sure most of us don’t understand or even know about those “black box algorithms” being used by companies and governments to analyze our information.
However, just like the data manipulation behind polling and studies that drive public policy, its clear we all need to start asking questions.
Anyway, I’ve just started reading the book. Based on just the introduction and the first chapter, I’ll probably have much more to rant about later.