Computers are an inherently oppressive technology

This may seem a strange heading for someone whose career is in computers, yet I feel that this article has been a lifetime in the making. It is the product of intuitive observations and things that have stood out to me, even as a child, who even then could sense the sinister side of the most banal of technologies.

When I was about 5 years old, I had a dream about a chocolate bar vending machine. There was a certain particular model of Cadbury's chocolate vending machine which UK railway stations were lousy with at the time, in the mid 90s. The real vending machines had a flap which you lifted up to take the dispensed chocolate, but the flap would only unlock and let you lift it up once the item was dropped.

In my dream, however, the lifting flap was replaced with a sliding metal door; the door would slide open quickly to let you take the chocolate inside, and would then slide closed again. But the door would open only for one or two seconds; and it would slide closed with such speed, and the edge was so sharp, that if you timed it wrong, your hand would be amputated. Taking the chocolate, you had to time it exactly right; I wanted the chocolate, but was scared to put my hand inside.

I'm certain I know what theme my unconscious mind was trying to explore with this dream: the ruthlessness of machinery. I know this because the ruthlessness of machinery is always something that played at the periphery of my mind as a child, something I noticed the sinister edge of where most people would notice nothing at all.

It's probably partly for this reason I long had a fascination with train doors as a child. I was always profoundly entranced and fascinated by acts of mechanical synchronicity; machines with everything moving as though part of an orchestra of machinery conducted with perfect timing, which I found supremely beautiful. For this reason, watching all of the doors open and close with perfect synchronisation, was naturally appealing. Yet this fascination was coloured by an intuitive sense of something more sinister; a kind of agency implicit in that motion. Too fleeting and weak a feeling to be called an emotion, and without particular words to describe it, whenever I watched train doors open and close, I would also sense the undercurrent of something more unsettling. Probably this is what made my fascination more than a passing one.

To describe it properly, what I was perceiving then even as a child, and what I am articulating now, is the ruthlessness of machines. The doors move with ruthless synchronicity. They care not whom they close against, or whom they separate. They can (and do) divide families, parents from their children. The sinister undercurrent I detected every time I watched a train's doors close, was the complete, absolute and total ruthlessness of machines.

I never watched soaps on TV, but as a child, one once came on when the TV was on in the background. A man was running for a train, which he sorely did not want to miss, a great deal counting on it. At the very moment he gets to the door, it closes in his face. He breaks down, crying, and the episode ends. This was many decades ago, but surely lodged in my memory because it spoke to a highly peculiar sense of horror (as in the genre) tuned to my hyper-awareness of that disconcerting quality: the sinister ruthlessness of machines.

Yet the ruthlessness of machines does not imply or require malevolence on the part of their designers. Instead, the ruthlessness of machines is an intrinsic consequence of their nature, perhaps even an inevitable one. Suppose, for example, that a computer system is setup to accept some kind of bureaucratic filing, and to enforce a certain deadline. Were a human to accept delivery of such a filing, it is unlikely they would be bothered by the filing being late by a second. Not so with a computer; the computer was given a deadline of noon, and so a filing one nanosecond later is rejected. It does not matter if the filing not being accepted ruins someone's life utterly, or leads to the destruction of one's whole family, or for that matter to nuclear war; it was a nanosecond late, so it was late. There is no magnitude or scale of human misery that a computer's decision might cause that will naturally overturn it.

Occasionally, of course one will encounter a kind of person who endeavours to be as inhumane and as ruthless as a computer, the jobsworth, who truly would reject something for being a second late and turn a blind eye to whatever scale of misery it causes. However even this person still responds to human sentiment, if only in the form of superior authority. If the refusal to accept a filing for being a second late will lead to, say, nuclear war, organizational and political pressure will be brought to bear on the jobsworth by humans with greater authority; the jobsworth, whose ultimate ends are their own self-preservation, can smell authority and will be responsive. The computer, however, remains unmoved.

What computers, and machines in general, therefore possess, is this: the power of unlimited, total ruthlessness, ruthlessness greater than even the most warped and terrible human is capable of, or can even comprehend. May the consequence of a computer's decision be even the extinction of humanity, still there is only the cold indifference of a non-sentient machine. As mammals we have evolved to perceive threats and malice in the actions of animate things arrayed against us, animals and people, but in machines we create our own sinister antagonist from the inanimate, not despite the lack of sentience but precisely because of it.

While this ruthlessness of machines may often be a purely accidental byproduct of their inherent nature, more and more this intrinsic property is noticed and in fact, deliberately leveraged by the designer to ruthless ends. Train doors are themselves an easy example, probably unnoticed to most by their mundane nature: whereas once train doors were manual, slam-door affairs, the adoption of electric, centrally controlled doors was naturally motivated in major part by timeliness. In other words, their principal advantage is that very property: the absolute ruthlessness of the centrally synchronised guillotine. The doors are preferred not despite their ruthlessness but because of it. Children may be divided from their parents, but the schedule will be kept.

Moreover, the machines offer no face to talk to, and therefore brook no argument. A railway employee who insists on closing a slam-door at the exact time prescribed, separating parent and child and in full view of the passengers on the train and the platform, purely to ensure the rules are upheld and punctuality is maintained, is likely to become extremely unpopular, in the immediate case from said passengers, and after causing negative PR for the railway, in the eyes of the company's management as well. By the power of machine-assisted ruthlessness, such events are now routine; yet no human being visibly at fault, people are led to view the consequence of the ruthlessness of the machine as a kind of force of nature, and rendered powerless to object. Yet it is not a force of nature; in reality it is a system deliberately designed by humans to advance a ruthless end, without admitting to it. Behind the door hides the human controlling it. In this way the machine becomes an abstraction and a disguise for human ruthlessness.

In 2014 the Silicon Valley startup “Juicero” was launched to general laughter, and is now a canonical example of the absurd excess of dubious business ideas (and VC capital) to be found in the Silicon Valley community. The product was a lineup of juice bags containing fresh fruit and vegetables which needed to be squeezed using an expensive juice-squeezing machine also made by the company. Predictably, the juice-squeezing machine was designed to only accept genuine Juicero bags — but not only that, would only accept bags which hadn't expired. Though perhaps unlikely, my hyper-awareness of the sinister ruthlessness of machines in the ordinary things around us led me immediately to note the pathological outcome: Someone stuck on a desert island (or equivalent) who starves to death, because the only food they have is expired Juicero juice bags and a machine which refuses to serve them. While in Juicero's case this might seem like a comical example, I fully believe it's only a matter of time until something like this happens. The deliberate utilisation of machine-assisted ruthlessness to malevolent ends becomes more and more common with every passing day, as companies realise they can escape from accountability by hiding behind the facelessness of the machine, which disempowers the individual to fight or object to it. The now memetic expression, “computer says no”, pithily expresses this phenomenon.

The topic of machine ruthlessness has been touched on substantially in fiction. One example is found in the second Hitchhiker's Guide to the Galaxy novel. The protagonist, while exploring an abandoned civilisation, finds a derelict spacecraft in a hangar which, against all odds, appears to still be powered.

Inside, the flight crew, who are all humanoid androids, explain that the flight has been delayed as they are still awaiting the delivery of certain supplies:

“Transtellar Cruise Lines would like to apologize to passengers for the continuing delay to this flight. We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey. Meanwhile we thank you for your patience. The cabin crew will shortly be serving coffee and biscuits again.”

In actuality, as civilisation collapsed long ago, no lemon-soaked paper napkins are ever going to be delivered, and the robotic flight crew have been awaiting the arrival of lemon-soaked paper napkins for over 900 years. To this end, the passengers have all been physically strapped into their seats and placed in suspended animation by the robotic crew.

“Yes,” said the voice again, “there has been a delay. Passengers are to be kept temporarily in suspended animation, for their comfort and convenience. Coffee and biscuits are being served every year, after which passengers are returned to suspended animation for their continued comfort and convenience. Departure will take place when the flight stores are complete. We apologize for the delay.”

Each year, the passengers are briefly revived and spend the time desperately trying to get free of their restraints and escape the flight, to no avail, and to the indifference of the crew. This passage disturbed me a great deal as a child and stuck with me for a long time, as while intended to be humorous, it is also a direct portrayal of the terrifying banality of the inherent inhumanity of machines.

Probably the most memorable example in fiction however is The Terminator, where a sapient machine spends every ounce of its effort trying to do, exactly what it was programmed to do, to the horror and suspense of the audience. However, these examples are sufficiently fantastic that they perhaps cause the average viewer to fail to notice the more banal and mundane ways in which we are menaced by machine-assisted ruthlessness even today. No strong AI is necessary.

What is particularly interesting about The Terminator however is its immediate sequel, which pits two Terminators against one another. The first is programmed to destroy the protagonist, and the second to protect him. Just as the first spends every last ounce of energy trying to fulfill its programmed purpose, so to does the second. Evaluated purely as machines, rather than according to any lens of human morality, the two Terminators are of equal virtue, being equally dedicated to their function as machines. If the T-850 is of greater virtue, it is because he was ultimately successful in his mission, whereas the T-1000 was not.

The film itself briefly notes the extreme dedication of the T-850 to its programmed parameters — in other words, to machine-assisted ruthlessness:

“Watching John with the machine, it was suddenly so clear. The terminator, would never stop. It would never leave him, and it would never hurt him, never shout at him, or get drunk and hit him, or say it was too busy to spend time with him. It would always be there. And it would die, to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice.”

The two Terminator films thus form an interesting contrast to one another, showing two different angles to the same question of machine ruthlessness; to the question of the moral content of the utter dedication of a machine to its programmed course. What this tells us is that machine-assisted ruthlessness is not even necessarily always negative. While I use the word “ruthlessness‟, one could equally call it “absolute incorruptibility”. Whereas machines have the power for ruthlessness infinitely greater than that of any human, they also possess the capacity for integrity infinitely greater than that of any human. They cannot be bribed; for better, or worse. In actuality, it seems like the only line dividing ruthlessness and absolute integrity is the moral content we assign. For every degree to which we are disturbed and horrified by the T-800's dedication to its end in The Terminator, we are equally moved by the T-850's dedication to its own end in the sequel. The “ruthlessness” of machines, it seems, has within it the capacity for infinite good and for infinite evil.

I once watched a video of automated manufacturing equipment in a bottle factory. Bottles are sorted this way and that, checked, packed. A video demonstrated one piece of equipment which checked that all the bottles going down a particular line were the right colour. Only blue bottles should be passing down that line, so if any orange bottles got down there, they'd gone the wrong way. Hundreds of bottles passed a minute, and the machine instantly detected the colour of each as it rolled past. Most would be allowed to continue past, but if a bottle was orange, it would instead be punted out of the line, dropped through a trapdoor precisely to remove that bottle — and only that bottle. The bottles being vaguely human-shaped, it didn't take much imagination to conceive of the terrible possibilities for our own oppression.

Interestingly, cryptoanarchism is arguably nothing more than an articulation of this realisation. As I see it, it is the belief that, using cryptography, systems — institutions — can be created which possess absolute integrity, where all past efforts to create such institutions have failed, having been comprised of humans who are infinitely more corruptible. It seems hard to dispute at this point that Bitcoin is the canonical cryptoanarchist success story, being the most successful cryptoanarchist technology by a wide margin. What Bitcoin represents at its core, as any cryptoanarchist technology must, is a kind of inviolable and absolute integrity. It is the effort to create a T-850 in kind, a Terminator of money. It cannot be bargained with, it cannot be reasoned with, and will not stop, ever. It cannot be bribed; its tokens cannot be forged.

While Bitcoin clearly demonstrates that the infinitely greater power of machines for ruthlessness has as much capacity to do good in the world as evil, it's worth noting that the distinction between integrity and ruthlessness is purely one of perception; and arguably, infinite integrity implies, and requires, infinite ruthlessness of a kind in turn. All transactions are final; none can be reversed. It does not matter if your coins were stolen, or if your family will come to ruin because of it; the system does not care. There can be no exceptions. If we desire a system of absolute integrity, we must accept these outcomes as the cost. Such a system is not to most people's taste, as the popularity of payment systems which make payment reversals easy or at least possible demonstrates. The infinite ruthlessness of machines, even when employed to noble ends, is alien and unsettling to our human sentiments.

In this regard one wonders if there has ever been a human who truly desired infinite integrity in full and prescient knowledge of its consequences. Humans are creatures of exceptions, and strong, overpowering emotions which bend rules which previously seemed unbreakable. For any principle there is some hypothetical scenario which would have any man begging for the exception. A computer will do exactly what you ask it to; but as in the case of an evil genie, be careful what you wish for. ¹

The capacity of computers to embody total integrity at least remains interesting. What should concern us is on the one hand the deliberate use of machine-assisted ruthlessness to malevolent ends, which runs rampant largely unchecked, and on the other hand, the intrinsically ruthless nature of all machines, whether intended or not, which has arguably equal power to oppress; all that is required is the thoughtlessness of its maker.