Cathy O’Neil is the author of a great book with the wonderfully provocative title, Weapons of Math Destruction. But it’s not hyperbole. In it she clearly and compellingly explains how algorithms, fueled by big data, are being used everyday to make decisions about important points in our lives. This includes whether or not we qualify for a loan, get a job, or go to prison.
Or create the “value add” rating a teacher receives, the one that mixes student test scores with data other points to determine if they keep their job.
The companies who sell those evaluation systems want us to believe that their mathematics is completely objective and a neutral judge of people. O’Neil, however, presents a variety of examples, starting with the teacher evaluation programs sold by Pearson and others, to explain why that’s just not true. She explains how the data used to build the algorithms usually comes with biases based on how and why it was collected. And additional bias is built into the systems by the people who ultimately decide what the resulting numbers mean.
While I highly recommend reading the book, you can get the TL;DR version of O’Neil’s warning in her TED talk from last spring, recently posted to their site and embedded below.