Paperclip Morality

The Madness of Tunnel Vision Utilitarianism

Mary Harrington has published a Substack piece entitled “The Sorcerer's Apprentice,” in which she reckons with the unyielding march of technological innovation. It reminded me of a section of our conversation that I’d cut because it felt too convoluted for a discussion that already pushed the limits of coherence. Her piece inspired me to revive the clip and prop it up with some text.

For newcomers to The Process, I sat down with Mary Harrington to discuss her concept of the Cyborg Theocracy, a fascinating frame to describe the amorphous technocratic regime evolving through our institutions.

Before the clip, Mary and I were about contemporary social movements and their moral structures. We touched on Effective Altruism, a movement committed to using evidence and reason to 'optimise’ the world for well-being. EA is disembodied by design and tends toward utilitarian approaches to moral conduct.

It’s popular among New Atheists and Silicon Valley autists, perhaps due to its rationalist approach and laser focus on quantifiable metrics.

An interesting case study for the problems embedded within EA comes from one of its most prominent figures, Sam Bankman-Fried. He stands accused of one of the biggest financial frauds in U.S. history after building a successful cryptocurrency exchange and stealing billions of dollars of customer deposits to make charitable donations and fund high-risk investments.

Sam's endlessly professed commitment to EA has led many, including myself, to speculate that he’d been morally offsetting his criminal behaviour against philanthropic endeavours. It’s fair to call this presumptive because fraud can be found in all moral and religious circles. However, if you listen to interviews with Sam before his arrest, it’s clear he thinks obsessively about these matters. A rationale of 'the ends justify the means' aligns with the disembodied EA framework, which neglects the fuzzier aspects of ethics like duty and virtue.

Even the brains trust behind the EA movement, a group of philosophers who spend their days contemplating moral trade-offs, have been accused of nonfeasance. They turned a blind eye to Sam’s dodgy business dealings while stewarding more than 160 million dollars from his Future Fund into their EA causes.

At the beginning of the clip at the bottom of this piece, I propose that a similar species of machine-like calculus was applied to the complex moral problem of COVID-19 in my home city of Melbourne, Victoria. There I saw state power fully mobilised to the dogged pursuit of simplistic metrics and a bizarre deferral of all ground-level sensemaking to a dissociated central authority.

While science can inform decision-making by quantifying the risks involved in different courses of action and evaluating the trade-offs between them it doesn’t actually lead anywhere. Once you decide to do something with the facts offered up by the scientific method you’ve stepped out of the scientific domain. When a leader claims their political decisions are scientific, they’re either confused about the scientific method or preemptively skirting responsibility for the outcome of their choices.

The Victorian leadership did just this by leaning into the authority of experts who took on the role of being the representatives for science. These public health officials presented us with a dizzying array of statistical arrangements to justify thier actions that frequently contradicted facts uncovered by the real scientific method that was going on quietly behind the scenes.

Opinions took on the authority of holy writ through appeals to “The Science” and our attempts to make sense of the trade-offs were suppressed by partisan zealots, ‘disinformation' suppression techniques, and in some cases, the police.

As weeks turned into months, the people around me became confused and listless, and even those who’d been enthusiastically supportive of the measures were plagued by dissonance. For all their apparent quantitative success, the Victorian leadership had created a qualitative hellscape.

In the end, we were held for 262 days under the most extreme lockdown measures in the Western world.

An over-the-top faith in science can transform what is essentially a methodological approach into the ideology of Scientism. Reason too, when not adequately balanced against humility, can turn into a form of madness. In both cases, human existence is oversimplified and aspects of reality that can’t be easily measured or quantified are dismissed.

Similar to the accusations against Bankman-Fried, the Victorian government engaged in tunnel vision utilitarianism. They defined zero community transmission as their metric, which was effectively a simplistic stand-in for the impossibly complex notion of ‘well-being’, and then mechanically pursued their goal.

In the clip below, I call this mindset ‘paperclip morality’, after the paperclip maximiser thought experiment.

The point here is not to attack science and reason, but to illuminate how they are neither scientific nor reasoned when pushed beyond their limits.

Perhaps the Melbourne lockdowns and the tragic tale of Bankman-Fried can join The Sorcerer’s Apprentice as allegories for what Mary identifies in her piece as the core struggle of modernity.

She steps with trepidation into a potential answer and I strongly recommend the read.

The Sorcerer's Apprentice