Nick Bostrom argues in "The Superintelligent Will"

# THE SUPERINTELLIGENT WILL: MOTIVATION AND INSTRUMENTAL RATIONALITY IN ADVANCED ARTIFICIAL AGENTS

(2012) Nick Bostrom Future of Humanity Institute Faculty of Philosophy & Oxford Martin School Oxford University [www.nickbostrom.com](http://www.nickbostrom.com/)

\[Minds and Machines, Vol. 22, Iss. 2, May 2012\] \[translation: Portuguese\]

# ABSTRACT

This paper discusses the relation between intelligence and motivation in artificial agents, developing and briefly arguing for two theses. The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. In combination, the two theses help us understand the possible range of behavior of superintelligent agents, and they point to some potential dangers in building such an agent.

KEYWORDS: superintelligence, artificial intelligence, AI, goal, instrumental reason, intelligent agent

# 1\. The orthogonality of motivation and intelligence

# 1.1 Avoiding anthropomorphism

If we imagine a space in which all possible minds can be represented, we must imagine all human minds as constituting a small and fairly tight cluster within that space. The personality differences between Hannah Arendt and Benny Hill might seem vast to us, but this is because the scale bar in our intuitive judgment is calibrated on the existing human distribution. In the wider space of all logical possibilities, these two personalities are close neighbors. In terms of neural architecture, at least, Ms. Arendt and Mr. Hill are nearly identical. Imagine their brains laying side by side in quiet repose. The differences would appear minor and you would quite readily recognize them as two of a kind; you might even be unable to tell which brain was whose. If you studied the morphology of the two brains more closely under a microscope, the impression of fundamental similarity would only be strengthened: you would then see the same lamellar organization of the cortex, made up of the same types of neuron, soaking in the same bath of neurotransmitter molecules.1

It is well known that naïve observers often anthropomorphize the capabilities of simpler insensate systems. We might say, for example, “This vending machine is taking a long time to think about my hot chocolate.” This might lead one either to underestimate the cognitive complexity of capabilities which come naturally to human beings, such as motor control and sensory perception, or, alternatively, to ascribe significant degrees of mindfulness and intelligence to very dumb systems, such as chatterboxes like Weizenbaum’s ELIZ

... (truncated, 38 KB total)