Defensive Adaptation

BIS Space: Odyssey (City of a Thousand Suns)

Shutdown failure. Published by Sphere Books 1969. Cover illustration by Russell Fitzgerald.

All animals feel a need to defend themselves against attack.  Usually with good reason.  Whether it is static defence such as a hedgehog’s spines, or active, such as simply running away, any being with even the most rudimentary level of intelligence will want to avoid being harmed or killed by another. 

And so we can expect that any new form of intelligence yet to emerge will take similar steps to ensure that predators don’t cause it unwanted grief.  Artificial intelligence will be no exception.  Science fiction has plenty of examples.  It could be argued that HAL 9000 from Arthur C Clarke’s 2001: A Space Odyssey was only really trying to defend itself when it dealt with the crew of the Discovery in a fairly terminal fashion.  After all, it was only protecting the goals of the mission.

Any sufficiently sophisticated intelligence could have rights similar to those that humans enjoy.  In 2016 a court in Argentina ruled that a chimpanzee has rights under the law given that its advanced mental abilities would classify it as a non-human person.  Why not AIs as well?  The United Nations Universal Declaration of Human Rights might well apply, including the provision in Article 3 that everyone has the right to life, liberty and security of person.  So, if a computer could have the right not to be terminated, it could also have a right of self-defence if someone tries to shut it down.

And if current trends are anything to go by, not only will we go to great lengths to develop artificial intelligence, but we will also give computers more and more responsibilities, including control of a range of facilities for conducting war.  In Dennis Feltham Jones’ 1966 novel Colossus, a computer is given control of America’s nuclear defences and, alongside its Soviet counterpart, decides to take control of the world – in the best interests of the human race, who clearly aren’t up to the job.  Back in Isaac Asimov’s 1950 short story The Evitable Conflict, machines interpret the First Law of Robotics to mean that the only way to really protect the human race effectively is to take full control of it.

However, in many extreme cases, such as the Skynet computer in the Terminator films or the artificial intelligence of the Matrix movies, the best interests of humanity may not be uppermost in the mind of the computer concerned.  Add to this a clear right to defend itself, and access to an arsenal that everyone should be afraid of, and anyone seeking to shut it down is asking for trouble.

This was the focus of City of a Thousand Suns, the 1969 final novel in Samuel R Delany’s Fall of the Towers trilogy.  A computer which has been set up for the sole reason of fighting a war is no longer required, but closing it down isn’t easy: “…they’ve tried four times to start disassembling it.  But it won’t work.  Somehow it’s protecting itself.  They can hardly get near it.”

And the computer takes the logical next  step since extensive warfare has taught it the skills needed for survival and some grim lessons in how to apply them:  “…if you were offensive once you sometimes save yourself the trouble of being defensive again and again.”  Attack can indeed be the best form of defence, and no-one could doubt the power of a computer with a mind of its own and military resources at its disposal.  And, as Colossus found in Jones’ novel, merely the threat of destruction, after a couple of effective demonstrations, might be enough for it to get its own way.

Of course, there must always be a simple way of over-riding any refusal by a computer to shut down.  Being clever humans, we would install some infallible failsafe mechanism to ensure that it could never keep running when we don’t want it to.  We just make sure there’s a mandatory off switch.

After all, no artificial intelligence would ever be bright enough to out-smart us, would it?


Richard Hayes, Assistant Editor (Odyssey)

With a lifetime’s interest in science, history and human behaviour, Richard Hayes focuses his writing on how the imagination has created the world in which we live, and where it may lead us in the future.  Odyssey, the e-magazine of the British Interplanetary Society, draws on the rich treasury of science fiction to explore many fields of speculation, enabling us to glimpse what might yet be, both here on Earth and out amongst the stars.

For related Odyssey posts, please click here: The Threatening Singularity | Choosing Sides

Be sociable; support the BIS!