The Mind's Eye: Research

In your collection of essays, you have a biographical note which explains your odyssey. Some of your major work has been on the eye. You write, "As a young neurologist trying to make sense of disorder of motor coordination so often seen in a variety of neurological syndromes, I found classical neurophysiology without adequate explanations to offer and turned to engineering science, cybernetics. Our approach to neurological systems was to define, measure, quantify, analyze, and form concepts having the same hard mathematical structure found in the physical sciences." Is that the story of how you wound up working on the eye?

As I mentioned, the idea of feedback was very, very important in engineering and had been invented in recent times by a man on a ferry going from New Jersey to the Bell Labs in Manhattan, a man named Black. And he patented it for Bell Labs.

Now, the idea of feedback is very important. And I was looking for a brain mechanism, like motion of the hand, but it was too complicated for me to handle at that time. In later years, I did go into control of hand movement. But I hit upon the pupil of the eye, the iris -- this beautiful blue or brown or yellow muscle in our eye, behind the transparent cornea. This muscle changes the size of the hole to the back of the eye called the pupil. And if you shine light, it gets smaller. If you take the light away, it gets larger. So it's an ideal system to study feedback control with. And so I started working on the pupil, and that led me into later studies on vision. But my main interest in the pupil was as a motor control feedback system.

So you started with hand motion, but then moved to the eye.

I was interested in hand motion, but it was too complicated and I was looking for a simple example of the feedback principle.

Some of your major work has been with showing how the eye sees, or the mind's eye sees, and you've done that through what you call "scan path theory." Explain to us what that is.

One of the things I eventually got into were eye movements. There are very fast, jumping eye movements, we make about a million a day, called "saccades," which
direct our glance to many parts of what is called the visual field. The area
you're looking at. And these saccades have very many interesting control properties.
I was starting to think about how people would use these eye movements to dissect
an image. I worked with a very brilliant student, an electrical engineer here
at Berkeley named David Newton. We measured eye movements of people looking at
different pictures. Now, people had done that before, but nobody had had our
equipment; we could measure the eye movements as a time function in a sequence.
So we could see, to our amazement, that people would look at five or six important
things in a room, or at a face or in a scene, an outdoor scene. And then they
would return in almost the same order to those same points. We called the sequence
of glimpses carried by the fast eye movements, or saccades, the "scan path eye movement." And so that was a very interesting experimental finding.

But it made us think even more deeply about how the scan path was generated, and we moved on to what we called the scan path theory, which says that the internal brain is generating a picture that we see and that we naively think we're seeing through our eyes, but we're actually seeing it through our brain. That doesn't mean that the eye doesn't jump around and collect very important checking information, but it's all in the mind's eye. Or most of it is in the mind's eye. The schema, this internal spatial cognitive, which means "knowledge model," is what drives the eye movements. And in the last thirty years we've done a lot of experiments to try to prove that beyond doubt.

Let's talk a little about this scan path. It's repetitive. It's ... you use the word "idiosyncratic." What do you mean by that? It has an "idiosyncratic nature"?

That means that even though people may share what they think are the important points in the scene, the regions of interest or the points of interest, they move about them in very different sequences. So it's almost like a fingerprint. So your scan path and my scan path on the same picture of the Mona Lisa might look at the same parts of the Mona Lisa -- her eyes and her enigmatic smile. But we would travel across those features in a different sequence. And, of course, we would have different regions for different pictures. And we would have different sequences for different pictures.

The eye is looking at this, if one can say the eye, in a top-down modality [i.e., directed from the brain, not by the visual image itself], right?

Well, it's driven in a top-down modality. See, the way vision works is we have a retina in the back of our eye that picks up signal information, like a television camera. We also have a very high resolution part of the retina called the fovea, F-O-V-E-A, if I can be a professor. And that has very high resolution. If we want to read a word, we have to do direct our fovea to that particular word. So if I held up a newspaper here, we would look at it and we would think we see every word clearly. But that's an illusion, an illusion of clarity and completeness, I call it. There should be a better name, but I haven't thought of it. And this illusion of clarity and completeness makes us think we see every word in the newspaper clearly. But, in fact, if we're going to read it, our eye muscles have to direct our eye so that the fovea falls on a particular word so we can actually read the word, we can actually bring it into the brain with high resolution. So it's important to bring some information into the brain through the retina and fovea to the visual cortex with high resolution. But at the same time, the general shape of the world is given by our mind's eye and by low resolution peripheral information.