Muller, The Tyranny of Metrics

We’re fixated on metrics, in part because we have come to embrace two dictums: “If you cannot measure it, you cannot improve it” (Lord Kelvin) and “What gets measured gets done” (Tom Peters).

Jerry Z. Muller, in The Tyranny of Metrics (Princeton University Press, 2018), sets out to show “the unintended negative consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics, but metric fixation.”

Metric dysfunction manifests itself in multiple ways. Problems that fall under the general heading of distortion of information include: (1) measuring the most easily measurable, (2) measuring the simple when the desired outcome is complex, (3) measuring inputs rather than outcomes, and (4) degrading information quality through standardization. Then there are the inevitable attempts to game the metrics. This gaming can manifest itself in (1) creaming, (2) improving numbers by lowering standards, (3) improving numbers through omission or distortion of data, and, when all else fails, (4) cheating.

We are deluged with quantitative data that are viewed as the answer; we just have to come up with the right question. But these data rarely give the full answer to a meaningful question. For instance, back when spreadsheets were becoming “a worldview—reality by the numbers,” Seth Klarman warned (in 1991) that “spreadsheets created the illusion of depth of analysis.” By now, at least in certain quarters, that illusion has been transformed into accepted dogma.

In a chapter titled “The Mismeasure of All Things?” Muller analyzes case studies from the fields of education, medicine, policing, the military, business and finance (especially pay-for-performance schemes and short-termism), and philanthropy and foreign aid.

The Tyranny of Metrics may not break a lot of new ground, but it shows how metric fixation permeates, and often creates hazards for, so many aspects of our society. And it does so in a thoroughly convincing way. As Muller concludes, “Ultimately, the issue is not one of metrics versus judgment, but metrics as informing judgment, which includes knowing how much weight to give to metrics, recognizing their characteristic distortions, and appreciating what can’t be measured.”