Pasquale, The Black Box Society

For some time I believed my life was pretty much an open book and therefore, like Scott McNealy (“You have zero privacy anyway. Get over it.”), had a casual attitude toward privacy. As for secrecy and obfuscation, I must confess I didn’t think much about them. I walked in the light, or so I thought, and prided myself on clarifying the confused or confusing. But even as I, like most people, eschewed the worlds of shadows and darkness, governments and businesses embraced them as sources of power and profits.

Frank Pasquale, a law professor, takes us deep into these dark, often creepy worlds in The Black Box Society: The Secret Algorithms That Control Money and Information (Harvard University Press, 2015). And offers some ways out, “toward an intelligible society.”

To understand the problem of dark money, data brokers, proprietary methods, supercookies, trade secrecy, and the like, Pasquale uses the metaphor of the black box. It is particularly apt, “given its own dual meaning. It can refer to a recording device, like the data-monitoring systems in planes, trains, and cars. Or it can mean a system whose workings are mysterious; we can observe its inputs and outputs, but we cannot tell how one becomes the other. We face these two meanings daily: tracked ever more closely by firms and government, we have no clear idea of just how far much of this information can travel, how it is used, or its consequences.” (p. 3)

We are constantly being profiled, and targeted. “It’s a cinch,” the author writes, “for companies to compile lists of chronic dieters, or people with hay fever.” The vice president of a company in the health sector went even further: “Based on your credit-card history, and whether you drive an American automobile and several other lifestyle factors, we can get a very, very close bead on whether or not you have the disease state we’re looking at.” (p. 30) Are you going to marriage counseling? Well, at least one credit card company pays attention to this fact. Counseling is a tip-off that “marital discord may be about to spill over into financial distress” and you thereby become less creditworthy. “Once one piece of software has inferred that a person is a bad credit risk, a shirking worker, or a marginal consumer, that attribute may appear with decision-making clout in other systems all over the economy.” (p. 32) On the other side of the financial spectrum, splurge on a pair of headphones and you can see higher prices on sneakers in a later online search.

Black box scoring misapplies natural science methods to the social realm. “A civil engineer might use data from a thousand bridges to estimate which one might next collapse; now financial engineers scrutinize millions of transactions to predict consumer defaults. But unlike the engineer, whose studies do nothing to the bridges she examines, a credit scoring system increases the chance of a consumer defaulting once it labels him a risk and prices a loan accordingly. Moreover, the ‘science’ of secret scoring does not adopt a key safeguard of the scientific method: publicly testable generalizations and observations. As long as the analytics are secret, they will remain an opaque and troubling form of social sorting.” (p. 41)

And then there are the big banks, which exploit “two levels of black box finance: obfuscation in the service of illegality, and opacity resulting from complexity.” (p. 103) Pasquale focuses on risk modeling in the subprime crisis, another misapplication of natural science methods, and high frequency trading. In the former case, to cite but a single problem, “the models had to assume the stability of certain kinds of human behavior, which could change in response to widespread adoption of the models themselves.” (p. 114)

The Black Box Society is a frightening portrait of the ever more powerful shadowy world that blocks light from reaching our everyday lives. It is also a call to action, with a range of suggestions that inevitably pale in comparison to the gargantuan task at hand. But small steps sometimes have outsize consequences. Just ask the folks who control what we see, influence what we buy, and determine whether we can pay for it.