Equal parts mathematician and political activist, Cathy O’Neil has calculated the impact of algorithms on society. For the most part, she says, big data adds up to big trouble.
When it comes to human activities, algorithms are expected to be models of objectivity, owing to their basis in mathematical formulae and reliance on enormous quantities of measured facts about a given general population, whether students or teachers, job applicants or criminal defendants.
Cathy O’Neil makes the case that real-world mathematical models are anything but objective. In her new book, Weapons of Math Destruction, she asserts that big data WMDs are opaque, unaccountable and destructive and that they essentially act as unwritten and unpublished secret laws. Weapons of Math Destruction was long-listed for a National Book Award in nonfiction and was published in September, 2016, to enthusiastic reviews from the likes of Clay Shirky and Cory Doctorow.
“It’s really not a book about math,” she tells CCC’s Chris Kenneally. “I know a lot of people worry about that, but the way I describe it is [that] it’s a book about power.
“In particular, it’s a book about the way that people with power are building tools of social control and shielding those tools from scrutiny by saying, ‘this is mathematics, you’re not an expert in mathematics, so you wouldn’t understand it.’ So in other words, they’re flashing the ‘math ID,’ just as you might see a policeman flash a badge, and saying, ‘this is something that you can’t ask questions about.’
“My book is about looking past that shield,” she continues, “and saying, ‘yeah, we actually have every right as human beings to ask questions about things that affect us very deeply, that are secret, and [that] are possibly quite unfair and destructive.”
To those concerned for the future of publishing, the arrival of Big Data requires carefully considered choices about best practices.
“Big data is sort of taking over the publishing world… But as in every other way big data is taking things over, there’s always this kind of choice of how you define success,” she explains.
“And probably for publishing – I’m just guessing – success looks like number of copies sold, right? But if you’ve defined that as success and you optimize to the number of copies sold or whatever the book – or the number of subscribers or something along those lines, then you actually lose, over time, if you’re optimizing only to that and focusing only on that, then you’re actually probably losing value in other ways.
“Publishers probably also care about how many awards that book was nominated for, or whether the book actually was high quality,” O’Neil adds. “But if you’re focusing only on the number of books sold, then you’re going to be blind to those other kinds of things that you do find valuable but which are harder to measure.”
Podcast: Play in new window | Download | Embed