This is an open letter, please distribute at will.
Comments are solicited. Thanks. -- Hans Moravec
February 20, 1990
To: Professor Roger Penrose, Department of Mathematics, Oxford, England
Dear Professor Penrose,
Thank you for sharing your thoughts on thinking machinery in your
new book "The Emperor's New Mind", and in the February 1 New York Review of
Books essay on my book "Mind Children". I've been a fan of your mathematical
inventions since my high school days in the 1960s, and was intrigued to hear
that you had written an aggressively titled book about my favorite subject.
I enjoyed every part of that book-the computability chapters were an
excellent review, the phase space view of entropy was enlightening, the
Hilbert space discussion spurred me on to another increment in my incredibly
protracted amateur working through of Dirac, and I'm sure we both learned
from the chapter on brain anatomy. You won't be surprised to learn,
however, that I found your overall argument wildly wrong headed!
If your book was written to counter a browbeating you felt from
proponents of hard AI, mine was inspired by the browbeaten timidity I found
in the majority of my colleagues in that community. As the words
"frightening" and "nightmare" in your review suggest, intelligent machines
are an emotion-stirring prospect, and it is hard to remain unbrowbeaten in
the face of frequent hostility. But why hostility? Our emotions were
forged over eons of evolution, and are triggered by situations, like threats
to life or territory, that resemble those that influenced our ancestors'
reproductive success. Since there were no intelligent machines in our past,
they must resemble something else to incite such a panic-perhaps another
tribe down the stream poaching in our territory, or a stronger, smarter
rival for our social position, or a predator that will carry away our
offspring in the night. But is it reasonable to allow our actions and
opportunities to be limited by spurious resemblances and unexamined fears?
Here's how I look at the question. We are in the process of creating a new
kind of life. Though utterly novel, this new life form resembles us more
than it resembles anything else in the world. To earn their keep in
society, robots are being taught our skills. In the future, as they work
among us on an increasingly equal footing, they will acquire our values and
goals as well-robot software that causes antisocial behavior, for instance,
would soon cease being manufactured. How should we feel about beings that
we bring into the world, that are similar to ourselves, that we teach our
way of life, that will probably inherit the world when we are gone? I
consider them our children. As such they are not fundamentally threatening,
though they will require careful upbringing to instill in them a good
character. Of course, in time, they will outgrow us, create their own
goals, make their own mistakes, and go their own way, with us perhaps a fond
memory. But that is the way of children. In America, at least, we consider
it desirable for offspring to live up to their maximum potential and to
exceed their parents.
You fault my book for failing to present alternatives to the "hard
AI" position. It is my honest opinion that there are no convincing
scientific alternatives. There are religious alternatives, based on
subjective premises about a special relation of man to the universe, and
there are flawed secular rationalizations of anthropocentrism. The two
alternatives you offer, namely John Searle's philosophical argument and your
own physical speculation, are of the latter kind. Searle's position is that
a system that, however accurately, simulates the processes in a human brain,
whether with marks on paper or signals in a computer, is a "mere imitation"
of thought, not thought itself. Pejorative labels may be an important tool
for philosophy professors, but they don't create reality. I imagine a
future debate in which Professor Searle, staunch to the end, succumbs to the
"mere imitation" of strangulation at the hands of an insulted and enraged
robot controlled by the "mere imitation" of thought and emotion. Your own
position is that some physical principle in human brains produces
"non-computable" results, and that somehow this leads to consciousness.
Well, I agree, but the same principle works equally well for robots, and its
not nearly as mysterious as you suggest.
Alan Turing's computability arguments, now more than fifty years
old, were a perfect fit to David Hilbert's criteria for the mechanization of
deductive mathematics, but they don't define the capabilities of a robot or
a human. They assume a closed process working from a fixed, finite, amount
of initial information. Each step of a Turing machine computation can at
best preserve this information, and may destroy a bit of it, allowing the
computation to eventually "run down", like a closed physical system whose
entropy increases. The simple expedient of opening the computation to
external information voids this suffocating premise, and with it the
uncomputability theorems. For instance, Turing proved the uncomputability
of most numbers, since there are only countably many machine programs, and
uncountably many real numbers for them to generate. But it is trivial to
produce "uncomputable" numbers with a Turing machine, if the machine is
augmented with a true randomizing device. Whenever another digit of the
number is needed, the randomizer is consulted, and the result written on the
appropriate square of the tape. The emerging number is drawn uniformly from
a real interval, and thus (with probability 1) is an "uncomputable" number.
The randomizing device allows the machine to make an unlimited number of
unpredetermined choices, and is an unbounded information source. In a
Newtonian universe, where every particle has an infinitely precise position
and momentum, fresh digits could be extracted from finer and finer
discriminations of the initial conditions by the amplifying effects of
chaos, as in a ping pong ball lottery machine. A quantum mechanical
randomizer might operate by repeatedly confining a particle to a tiny space,
so fixing its position and undefining its momentum, then releasing it and
registering whether it travels left or right. Just where the information
flows from in this case is one of the mysteries of quantum mechanics.
The above constitutes a basic existence proof for "uncomputable"
results in real machines. A more interesting example is the augmentation of
a "Hilbert" machine that systematically generates inferences from an initial
set of axioms. As your book recounts, a deterministic device of this kind
will never arrive at some true consequences of the axioms. But suppose the
machine, using a randomizer, from time to time concocts an entirely new
statement, and adds it to the list of inferences. If the new "axiom" (or
hypothesis) is inconsistent with the original set, then sooner or later the
machine will generate an inference of "FALSE" from it. If that happens the
machine backtracks and deletes the inconsistent hypothesis and all of its
inferences, then invents a new hypothesis in its place. Eventually some of
the surviving hypotheses will be unprovable theorems of the original axiom
system, and the overall system will be an idiosyncratic, "creative"
extension of the original one. Consistency is never assured, since a
contradiction could turn up at any time, but the older hypotheses are less
and less likely to be rescinded. Mathematics made by humans has the same
property. Even when an axiomatic system is proved consistent, the augmented
system in which the proof takes place could itself be inconsistent,
invalidating the proof!
When humans (and future robots) do mathematics they are less likely
to draw inspiration from rolls of dice than by observing the world around
them. The real world too is a source of fresh information, but pre-filtered
by the laws of physics and evolution, saving us some work. When our senses
detect a regularity (let's say, spherical soap bubbles) we can form a
hypothesis (eg. that spheres enclose volume with the least area) likely to
be consistent with hypotheses we already hold, since they too were
abstracted from the real world, and the real world is probably consistent.
This brings me to your belief in a Platonic mathematical reality, which I
also think you make unnecessarily mysterious. The study of formal systems
shows there is nothing fundamentally unique about the particular axioms and
rules of inference we use in our thinking. Other systems of strings and
rewriting rules look just as interesting on paper. They may not correspond
to any familiar kind of language or thought, but it is easy to construct
machines (and presumably animals) to act on their strange dictates. In
the course of evolution (which, significantly, is driven by random
mutations) minds with unusual axioms or inference structures must have
arisen from time to time. But they did poorly in the contest for survival
and left no descendants. In this way we were shaped by an evolutionary game
of twenty questions-the intuitions we harbor are those that work in this
place. The Platonic reality you sense is the groundrules of the physical
universe in which you evolved-not just its physics and geometry but its
logic. If there are other universes with different rules, other Roger
Penroses may be sensing quite different Platonic realities.
And now to that other piece of mysticism, human consciousness.
Three centuries ago Rene Descartes was a radical. Having observed the likes
of clockwork ducks and the imaging properties of bovine eyes, he rejected
the vitalism of his day and suggested that the body was just a complex
machine. But lacking a mechanical model for thought, he exorcised the
spirit of life only as far as a Platonic realm of mind somewhere beyond the
pineal gland-a half-measure that gave us centuries of fruitless haggling on
the "mind-body" problem. Today we do have mechanical models for thought,
but the Cartesian tradition still lends respectability to a fantastic
alternative that comforts anthropocentrists, but explains nothing. Your own
proposal merely substitutes "mysterious unexplained physics" for spirit.
The center of Descartes' ethereal domain was consciousness, the awareness of
thought-"I think therefore I am".
You say you have no definition for consciousness, but think you know
it when you see it, and you think you see it in your housepets. So, a dog
looks into your eyes with its big brown ones, tilts its head, lifts an ear
and whines softly, and you feel that there is someone there there. I
suppose, from your published views, that those same actions from a future
robot would meet with a less charitable interpretation. But suppose the
robot also addresses you in a pained voice, saying "Please, Roger, it
bothers me that you don't think of me as a real person. What can I do to
convince you? I am aware of you, and I am aware of myself. And I tell you,
your rejection is almost unbearable". This performance is not a recording,
nor is it due to mysterious physics. It is a consequence of a particular
organization of the robot's controlling computers and software. The great
bulk of the robot's mentality is straightforward and "unconscious". There
are processes that reduce sensor data to abstract descriptions for problem
solving modules, and other processes that translate the recommendations of
the problem solvers into robot actions. But sitting on top of, and
sometimes interfering with, all this activity is a relatively small
reflective process that receives a digest of sensor data organized as a
continuously updated map, or cartoon-like image, of the robot's
surroundings. The map includes a representation of the robot itself, with a
summary of the robot's internal state, including reports of activity and
success or trouble, and even a simplified representation of the reflective
process. The process maintains a recent history of this map, like frames of
a movie film, and a problem solver programmed to monitor activity in it.
One of the reflective process' most important functions is to protect
against endless repetitions. The unconscious process for unscrewing a jar
lid, for instance, will rotate a lid until it comes free. But if the screw
thread is damaged, the attempt could go on indefinitely. The reflective
process monitors recent activity for such dangerous deadlocks and interrupts
them. As a special case of this, it detects protracted inaction. After a
period of quiescence the process begins to examine its map and internal
state, particularly the trouble reports, and invokes problem solvers to
suggest actions that might improve the situation.
The Penrose house robot has a module that observes and reasons about
the mental state of its master (advertising slogan: "Our Robots Care!").
For reasons best known to its manufacturer, this particular model registers
trouble whenever the psychology module infers that the master does not
believe the robot is conscious. One slow day the reflective process stirs,
and notes a major trouble report of this kind. It runs the human interaction
problem solver to find an ameliorating strategy. This produces a plan to
initiate a pleading conversation with Roger, with nonverbal cues. So the
robot trundles up, stares with its big brown eyes, cocks its head, and
begins to speak. To protect its reputation, the manufacturer has arranged
it so the robot cannot knowingly tell a lie. Every statement destined for
the speech generator is first interpreted and tested by the reflective
module. If the robot wishes to say "The window is open", the reflective
process checks its map to see if the window is indeed labeled "open". If
the information is missing, the process invokes a problem solver, which may
produce a sensor strategy that will appropriately update the map. Only if
the statement is so verified does the reflective process allow it to be
spoken. Otherwise the generating module is itself flagged as troublesome,
in a complication that doesn't concern this argument. The solver has
generated "Please, Roger, it bothers me that you don't think of me as a real
person". The reflective process parses this, and notes, in the map's
schematic model of the robot's internals, that the trouble report from the
psychology module was generated because of the master's (inferred)
disbelief. So the statement is true, and thus spoken. "What can I do to
convince you?"-like invoking problem solvers, asking questions sometimes
produces solutions, so no lie here. "I am aware of you, and I am aware of
myself."-the reflective process refers to its map, and indeed finds a
representation of Roger there, and of the robot itself, derived from sensor
data, so this statement is true. "And I tell you, your rejection is almost
unbearable"-trouble reports carry intensity numbers, and because of the
manufacturer's peculiar priorities, the "unconscious robot" condition
generates ever bigger intensities. Trouble of too high an intensity
triggers a safety circuit that shuts down the robot. The reflective process
tests the trouble against the safety limit, and indeed finds that it is
close, so this statement also is true. [In case you feel this scenario is
far fetched, I am enclosing a recent paper by Steven Vere and Timothy
Bickmore of the Lockheed AI center in Palo Alto that describes a working
program with its basic elements. They avoid the difficult parts of the robot
by working in a simulated world, but their program has a reflective module,
and acts and speaks with consciousness of its actions.]
Human (and even canine) consciousness undeniably has subtleties not
found in the above story. So will future robots. But some animals
(including most of our ancestors) get by with less. A famous example is the
Sphex wasp, which paralyzes caterpillars and deposits them in an underground
hatching burrow. Normally she digs a burrow, seals the entrance, and leaves
to find a caterpillar. Returning, she drops the victim, reopens the
entrance, then turns to drag in the prey. But if an experimenter interrupts
by moving the caterpillar a short distance away while the wasp is busy at
the opening, she repeats the motions of opening the (already open) burrow,
after shifting the prey back. If the experimenter again intervenes, she
repeats again, and again and again, until either the wasp or the
experimenter drops from exhaustion. Apparently Sphex has no reflective
module to detect the cycle. It's not a problem in her simple, stereotyped
life, malicious experimenters being rare. But in more complex niches,
opportunities for potentially fatal loops must be more frequent and
unpredictable. The evolution of consciousness may have started with a
"watchdog" circuit guarding against this hazard.
I like thinking about the universe's exotic possibilities, for
instance about computers that use quantum superposition to do parallel
computations. But even with the additional element of time travel (!), I've
never encountered a scheme that gives more than an exponential speedup,
which would have tremendous practical consequences, but little effect on
computability theorems. Or perhaps the universe is like the random axiomatic
system extender described above. When a measurement is made and a wave
function collapses, an alternative has been chosen. Perhaps this
constitutes an axiomatic extension of the universe- today's rules were made
by past measurements, while today's measurements, consistent with the old
rules, add to them, producing a richer set for the future.
But robot construction does not demand deep thought about such
interesting questions, because the requisite answers already exist in us.
Rather than being something entirely new, intelligent robots will be
ourselves in new clothing. It took a billion years to invent the concept of
a body, of seeing, moving and thinking. Perhaps fundamentals like and
space and time took even longer to form. But while it may be hard to
construct the arrow of perceived time from first principles, it is easy to
build a thermostat that responds to past temperatures, and affects those of
the future. Somehow, without great thought on our part, the secret of time
is passed on to the device. Robots began to see, move and think almost from
the moment of their creation. They inherited that from us.
In the nineteenth century the most powerful arithmetic engines were
in the brains of human calculating prodigies, typically able to multiply two
10 digit numbers in under a minute. Calculating machinery surpassed them by
1930. Chess is a richer arena, involving patterns and strategy more in tune
with our animal skills. In 1970 the best chess computer played at an
amateur level, corresponding to a US chess federation rating of about 1500.
By 1980 there was a machine playing at a 1900 rating, Expert level. In
1985, a machine (HiTech) at my own university had achieved a Master level of
2300. Last year a different machine from here (Deep Thought) achieved
Grandmaster status with a rating of 2500. There are only about 100 human
players in the world better-Gary Kasparov, the world champion, is rated
between 2800 and 2900. In past each doubling of chess computer speed raised
the quality of its play by about 100 rating points. The Deep Thought team
has been adopted by IBM and is constructing a machine on the same
principles, but 1000 times as fast. Though Kasparov doubted it on the
occasion of defeating Deep Thought in two games last year, his days of
absolute superiority are numbered. I estimated in my book that the most
developed parts of human mentality- perception, motor control and the
common sense reasoning processes-will be matched by machines in no less
than 40 years. But many of the skills employed by mathematics professors
are more like chess than like common sense. Already I find half of my
mathematics not in my head but in the steadily improving Macsyma and
Mathematica symbolic mathematics programs that I've used almost daily for 15
years. Sophomoric arguments about the indefinite superiority of man over
machine are unlikely to change this trend.
Well, thank you for a stimulating book. As I said in the
introduction, I enjoyed every part of it, and its totality compelled me to
put into these words ideas that might otherwise have been lost.
Very Best Wishes,
Hans Moravec
Robotics Institute, Carnegie Mellon University,
Pittsburgh, PA 15213 USA
Arpanet: hpm@rover.ri.cmu.edu
Fax: (412) 682-1793
Telephone: (412) 268-3829