Gaming excellence
According to Goodhart’s Law, as soon as a measure becomes a target, gaming ensues, which undermines its function as a measure. Charles Goodhart, an economist, was referring to the gaming of economic indicators, but his law applies equally well to all sorts of regimes of evaluation, including the metrics that command so much authority in today’s higher education. Universities are investing ever more heavily in curating and occasionally faking figures that enhance their national and global rankings, while simultaneously keeping those metrics in mind when deciding anything from campus development projects to class size. [1] (Architecturally ambitious campuses attract alumni giving, which is a positive factor in the U.S. News & World Report rankings of universities, as are classes capped at 19 students.) Now in full swing, this trend started inconspicuously a few decades ago. Already in 1996, Northeastern University’s president, Richard Freeland, observed that “schools ranked highly received increased visibility and prestige, stronger applicants, more alumni giving, and, most important, greater revenue potential. A low rank left a university scrambling for money. This single list […] had the power to make or break a school.” [2] Freeland quickly figured out which numbers Northeastern needed to privilege. Ranked 162nd in 1996, Northeastern jumped to 98th in 2006 and, ten years after his departure, 47th in 2016.
An exceptional article by Mario Biagioli on fraud and gaming in the area of research metrics, drawing on his co-edited collection Gaming the Metrics: Misconduct and Manipulation in Academic Research (MIT Press, 2020). Also podcast conversation between Mario Biagioli and John Markoff on fraud and gaming in scholarly publishing. https://human-centered.simplecast.com…
Responses to the modern university’s embrace of “audit culture” — the shift from traditional qualitative forms of evaluation to quantitative indicators of excellence or impact — have been polarized, especially when metrics are deployed to evaluate specific publications or individual authors through the citations they receive. [5] Advocates praise the transparency and accountability that metrics are claimed to provide, which they contrast with the traditional, opaque, and discipline-specific protocols of peer review. Critics see metrics instead as the predictable outcome of the contemporary university’s managerial culture. Hiding behind the rhetorical shield of objectivity, metrics function, these critics argue, as disciplinary techniques while failing to measure anything worth measuring. What has gone unnoticed in these debates, however, is that regardless of whether citation-based indexes are a paragon of objectivity or mere numerical simulacra of imaginary value, they have an uncanny ability to spawn new and expanding forms of research misconduct. It’s worth carefully analyzing such practices to understand the ways in which concepts like value and quality, and “publication” are being radically reshaped in the age of metrics.
[1] Cathy O’Neil, Weapons of Math Destruction (Crown, 2016), pp. 50–67.
[2] Max Kutner, “How to Game the College Rankings,” Boston Magazine, August 2014.
[3] Bill Readings, The University in Ruins (Harvard University Press, 1996), pp. 23-43.
[4] Mario Biagioli, “Quality to Impact, Text to Metadata,” KNOW 2 (2018).
[5] Marilyn Strathern (ed), Audit Cultures (Routledge, 2000); Michael Power, The Audit Society (Oxford University Press, 1997).