A way round the problem is to postulate that experience is
not something we feel, but something we do: a kind of give-and-take
with the environment, analogous to the "feel" of driving
a car. One consequence of such a "sensorimotor" theory
of experience is that it provides a way of explaining the differences
between seeing, hearing, touch, etc., which is more principled
and has more explanatory power than Müller's notion of "specific
nerve energy" or its modern counterpart, the notion of sensory
pathways or cortical areas. The feasibility of sensory substitution
is an empirically verifiable implication of this approach.

As applied to visual perception, a consequence of the sensorimotor
approach is the idea that seeing does not consist in the creation
of a "re-"presentation of the world inside the brain,
but rather in knowledge that the outside world is immediately
accessible through a flick of the eye or of attention, like an
"outside memory". The world-as-an-outside-memory idea
has empirically verifiable consequences in the phenomenon of Change
Blindness, among others.

It's quite interesting that sometimes
you can be looking directly at the change and still not see it.
So here if I tell you to look at the man's nose, you'll be within
a few pixels of the change, and yet often you wont see it: it's
the bar in the background going up and down.

Some changes are easier to see than others.
But even a change which occupies a very large portion of the visual
field may not be noticed if it's not part of what the picture
would be said to be about. Here for example,

you probably consider the picture to be
about a glass of milk, so its easy for you to see that that's
what's changing, even though it's much smaller than the reflection
in the lake was.

This phenomenon has been called change
blindness, and has attracted quite a lot of attention over the
last few years. There are a number of variations of the phenomenon.
I've shown you the flicker paradigm, but the phenomenon has also
been obtained with eye saccades, blinks, film cuts, and even in
real-life situations.

A particularly interesting variant of
the CB paradigm is the mudsplash paradigm.

The reason this is interesting is that
the mudsplashes are positioned in such a way so as to not cover
the change location. That way it can't be argued that the reason
you miss the change is that it is somehow masked or wiped out
by any kind of superposition with flicker.

It's clear that there is something very
shocking about all these experiments: they seem to suggest that
our internal representations of the outside world, instead of
being very detailed and rich...

are actually rather sparse...So what's
going on here? How can we have the impression of richness in the
world if there is no richness in the head?

I suggest that a possible solution is
the idea of what I call the World as Outside Memory. The idea
is that to get the impression of richness, there's actually no
need for the richness to be in the head. What has to be in the
head is merely algorithms or recipes for getting at the
information in the world.

Such algorithms we have, in the form of
movements of the eyes or shifts of attention. If we're interested
in some detail of the visual scence, we simply need to move our
eyes or our attention to that detail, and it is immediately available.

Instead of storing all the information
about the outside world in the brain, we use the outside world
as an external memory storage. Thus, we get the impression that
we're seeing everything there is to see in the visual field,
because if we so much as faintly wonder whether we're actually
seeing something, we turn our eye (and our attention) to that
thing, and it becomes available for processing.

Perhaps you've played the children's game
in which someone puts a household object like a cork, or a potato,
or a harmonica into a bag, and you put your hand into the bag
and try to figure out what the object is.

At first you feel this or that texture
on the tips of your fingers. You have no idea what the object
is. But suddenly you have a kind of "Aha" experience.
Suddenly you feel you're no longer touching bits of texture on
the ends of your fingers, but you're holding a whole object: it's
a harmonica. And it's ALL there at once, even though you're in
fact only touching a few parts of it. It's not just that you know
it's a whole harmonica, you actually feel it's a whole
harmonica

The reason you have the feeling of touching
the whole harmonica is that you KNOW that IF you were to
move your fingers this way, then you would get THIS feeling, and
if you moved them that way, you would get THAT feeling. You feel
familiar with all the things you can do with your fingers at this
moment. It is the knowledge of feeling at home with the possible
things you can do, and with the resulting changes in the feelings
on your fingertips, that gives you the impression of having the
whole harmonica in your hand.

Extended to the domain of vision, this
analogy suggests how it could be possible to have the distinct
feeling of seeing whole objects and whole scenes, even though
only a minute part of the scene is actually available for visual
processing at any moment. The impression of seeing everything
there is to see in the visual field is therefore a sort of
illusion...

It's a bit like the light in the refrigerator:

The light always seems to be on: you open
the door, the light is on. You close it. You quickly open the
door again to check: yup, the light is still on. You have the
illusion that the light is on all the time, but you have to keep
checking to convince yourself.

Let me go on to show you some of the consequences
of this way of thinking for scene perception.

This picture shows the path that the eye
of one observer took while searching for large changes which occurred
every time he blinked.

Here ís an example of a scan path
for another picture.

This scan path corresponds to a few seconds
of exploration, and you would think that since the observer was
actively looking for a change, he would look around in the picture
in a rather systematic way, covering all the picture elements.
But if you look at what the observer did over the next few tens
of seconds, this is what you find.

It looks like the observer is just going
round and round in circles. In fact this kind of behavior is typical
of what people do when they look at pictures. Only a fairly limited
number of positions are directly fixated by the eyes, and they
are fixated repetitively. Why is this?

Under the WOM point of view I'm sketching
out here, this can be explained. It could be that seeing a picture
is not: accumulating information into an internal representation,
but rather: checking that you have access to the things that the
picture is about. If you think the picture is about a couple having
dinner, then seeing the picture involves making sure that those
things that you think the picture is about, are really there.
The eye will therefore go round and round checking.

Ongoingness/Continuousness

In summary up to now, I've suggested the
possibility that the feeling we get of seeing everything in the
visual field doesn't require us having an internal representation
of everything. It suffices to have immediate access to the information
in the external world, which acts somewhat like an external memory
store.

But you might object that there's still
a problem. Consider again the comparison with the refrigerator
light. When I look at the world, I have quite a different impression
than I do with respect to the refrigerator light:

I have to keep opening the refrigerator
door surrepticiously, and still there remains a bit of doubt about
whether the light really stays on all the time. I think it's
on continuously, but I dont see it on continously. This
differs from the real world, which I have the impression of seeing
in an "ongoing" or continuous fashion. Why is that?

I think the answer is to do with two things.
I call one "bodiliness", and the other "grabbiness".

Bodiliness is the fact that in vision,
the things you do in order to get information are very closely
linked to minute and even unconscious bodily actions: The slightest
twitch of an eye muscle allows you to change from one point of
the scene to another. A small head movement or body movement modifies
what you see.

By virtue of this bodiliness, the outside
world is intimately linked to you, almost as though it was part
of your own body. I suggest that this makes seeing more real,
more felt and ongoing than the refrigerator light, which doesn't
budge when you move around.

Now let me talk about grabbiness.

We all know that if there's a sudden flicker
in the visual field, we can't help but immediately look at it.
This is because there exist mechanisms in the first stages of
the visual system designed to detect fast transitions in local
luminance, and which incontrovertibly grabs your attention. Movement
detectors are examples of such transient detectors.

It could be that this grabbiness of sudden
events constitutes a second factor which contributes to the feeling
of continual presence, and ongoingness of visual stimulation.
Grabbiness makes it seem like we have continual "tabs"
on everything that's going on in the visual field, and gives us
the illusion of seeing things continuously, because if anything
should change we're immediately informed.

Developments

Let me now stand back and make some remarks
about the way of thinking that I've sketched here.

Under the standard view, seeing consists
in making an internal representation of the outside world. Under
the new view, seeing consists in knowing different things you
can do, and knowing the changes that these things will produce
in your sensory input.

Though unnatural at first sight, the new
view has an interesting advantage:

In neuroscience today, one of the problems
people are grappling with is to try to understand how a physical
entity like a brain can give rise to something like the feeling
of seeing, which is patently not physical.

Some as yet unknown mysterious, possibly
even nonphysical mechanism has to be postulated to instill experience
into the brain. But under the new view, the problem disappears,
because experience is not in the brain at all.

It's in the doing of the exploration,
and in the knowledge of the things that will change as you explore.
Instead of the role of the brain being to generate the experience
of seeing, the role of the brain simply becomes that of generating
the exploratory activity which underlies the seeing, and
that of holding the knowledge of current possibilities
for action that underlies seeing.

Thus, the problem of finding a mechanism
to generate experience in the brain disappears.

Still, there is an objection that may
be disturbing you. You could say, ok, seeing is a thing we do...
I see the red cup when I'm going about checking I have access
to it by the flicks of my eye movements. But what about when I
finally am actually looking directly at the red cup. I
now have red stimulation on my retina. Now surely there must be
something that takes that takes red stimulation on my retina and
causes me to experience the redness. We seem to be back to the
situation of having to explain how brain activation can engender
experience.

But it seems to me that it's possible
to escape from the difficulty even for the raw sensation of redness.

Consider looking at a piece of red paper.
Depending on whether you turn the paper so that it's yellowish
sunlight or bluish skylight, or reddish incandescent light that
is reflected off the paper, the spectrum of light being sensed
by the eye is quite different. I suggest you see the paper as
red when the laws that are obeyed by the changes in incoming spectrum
are typical of redness. Thus, red is not a pattern of excitation
caused by incoming light, but knowledge about the laws that the
excitation obeys when you move the paper around.

Another fact about red has to do with
the way the eye samples color. At the center of the retina, color
information is readily available, being sampled by retinal cones
sensitive to long, medium and short-wavelength light, symbolized
by colored dots in the slide. But the density of the cones falls
off quite rapidly, so that the nature of the neural stimulation
that arises from looking straight at a red surface is quite different
from that obtained by looking at the surface in peripheral vision.
here, there are many more rod photoreceptors, symbolized by black
dots, not sensitive to different colors. I suggest that the quality
of red is NOT just the particular combination of long, medium
and short wavelength stimulation, but also the way in which the
stimulation changes as you move your eye on and off the red object.

Like the experience of seeing everything,
the experience of seeing red then is also a kind of knowledge:
knowledge that the appropriate contingencies between sensory input
and motor actions are currently applicable.

The notion of sensorimotor contingency
can be generalized to cover not just the sensation of red, but,
I suspect, all aspects of vision, both general and particular.
For example the fact that the retinal image essentially goes blank
when we blink, or shifts in lawful ways when we move our eyes,
or has an expanding or contracting flow field when we move our
heads backwards and forwards, are facts about vision in general.

A fact that's more specific, characteristic
of straight lines, for example, is the fact that when you move
your eyes along them, nothing much happens to sensory input, whereas
when you move your eyes across them, sensory input changes more
drastically.

In sum then, it could be that the experience
of seeing derives from being familiar (in the sense one is familiar
with practical know-how) with a wide variety of sensorimotor contingencies
related to the way the visual apparatus samples the environment.
It could be that we feel we are seeing at this moment, when we
know (in a practical way) that all these contingencies are currently
applicable. The experience of seeing would then not be generated
by activation of a brain mechanism. It would be constituted by
knowledge that if you do certain things, certain things will happen
to sensory input.

You dont always see what you're looking
at

These ideas have some interesting consequences.

Let me go back to the experiment where
we measured eye movements while people were looking for changes
in pictures.

We looked at the probability of detecting
the change as a function of the position of the eye. We found,
as you might expect, that the further your eye was from the change
location, the smaller the probability of detecting the change
was. You can see this from the fact that the graphs drop lower
and lower as we go to the right, corresponding to greater eccentricities.
Don't worry about the fact there are two curves, these just correspond
to two different kinds of changes we used.

But one very surprising thing is visible
in this graph. The probability of detecting the change when you
are looking directly at the change, that is the leftmost point
of the graph, is less than 60%.

That is, in almost 50% of the cases when
the eye was looking directly at the change, it is not seen!

This fact is coherent with the approach
I've been putting forward. According to this, when something falls
on your retina, or when your visual system processes something,
that does not necessarily mean you see it. Seeing only occurs
when you are currently exercising your mastery of the sensorimotor
contingencies associated with that thing, sort of "manipulating"
it with your eyes. When I look at an object, I can be conscious
of any number of its aspects: its color, its identity, itís
background, its position, etc. I would say that only the aspect
that Iím currently checking on is actually being seen.
So other aspects, even if they are being directly looked at, will
not be seen.

Ambiguous figures and figure-ground competition
actually provide examples illustrating this. Here, you can be
fixating on the white nose and not see the black nose, even though
it ís in the same location.

A similar rather shocking finding was
reported by Haines, at the NASA Ames Research Center in California.
He had commercial airline pilots land a 727 in a flight simulator,
using a heads-up display of certain instruments on the windscreen.
On certain landing approaches, Haines suddenly superimposed a
stationary small aircraft right in the middle of the runway. He
expecte pilots to immediately abort their landing approach. However
2 out of 8 pilots simply blithely landed through the obstructing
airplane. When shown a video of what they had done, the pilots
were shocked and incredulous and noted that they should perhaps
resign from commercial flying.

Here is yet another example: you can stare
at this for minutes and still think it says "The illusion
of seeing". But actually it doesnt; it says: The illusion
of of seeing.

Here is another example of where you can
be looking directly at the change and not see it. This picture
is changing. Except it's changing very slowly. The change is rather
large: see if you can find it. Dan Simons at Harvard has also
been doing experiments with slow changes like this.

The point is, seeing is mentally manipulating
some aspect of the scene. If no attention-grabbing visual transient
brings your eye or your attention on to some area of the picture,
you wont see it at all, let alone see it change.

Studies like these are part of a growing
literature on what's called inattentional blindness: Ulrich Neisser
was one of the first to look at this, but Mack & Rock have
just published a book on the question. Dan Simons has recently
done some some other beautiful experiments showing your eye can
be very close to some totally obvious thing in a picture and yet
not see it.

Müller's specific nerve energy and
sensory substitution

Now I'd like to discuss another consequence
of what I've been saying, which concerns the quality of different
sensory modalities.

Everyone agrees that the qualitative nature
of experience in one sensory modality is quite different from
the experience in another modality: hearing feels quite different
from seeing, which feels different from taste and touch... The
explanation for this has remained problematical ever since Johannes
Müller at the end of the last century had suggested that
different neural pathways might have what he called different
"nerve energies".

On the other hand a natural and principled
approach to the problem might be available if we adopt the view
that seeing is a kind of knowledge about what happens when you
do certain things. Driving a car feels different from driving
a truck or riding a bicycle, because it involves doing different
things. Similarly, seeing feels different from hearing, tasting,
touching, because it too involves doing different things.

For example, we know we're seeing when
we know that:if we blink, the sensory
input changes drastically;if we move
forward, there is an expanding flow field; if
we move our eyes, there is a translating flow field; if we block our eyes with our hands the visual
field is obscured; on the other hand,
if we block our ears with our hands, nothing much happens.

On the other hand we know we're hearing
if:when we blink or move our eyes nothing much happens;
if we move forward the intensity of incoming
stimulation obeys an inverse square law; if
we move our head the asynchrony and spectrum of the input changes
in certain characteristic ways; if we block our eyes with our
hands, nothing much happens; if we block our ears with our hands,
the intensity changes in a certain way.

Degrees of raw feel

I've been suggesting that, contrary to
our intuitions, memory and visual experience might actually be
one and the same kind of thing: both involve knowing ways of getting
at information: in one case the information is in the brain, in
the other case it's in the outside world.

In the case of memory of latin verbs,
for example, I know I can recover the conjugation of a particular
verb by attending to that verb. Analogously, in the case of seeing,
I know I can recover information about some object in the scene
by paying attention to it.

The explanation for the difference in
the quality of the experience we get from our memory for latin
verbs and the experience we get from seeing, could be due to the
amount of bodiliness and grabbiness that was involved.

Memory for latin verbs has no bodiliness
and no grabbiness: no bodiliness because my body motions dont
affect the availability of latin verbs in my memory; and no grabbiness
because changes in my memory dont attract my attention -- for
example, if a word dropped out of my memory overnight, no bell
whistles in my mind to tell me.

On the other hand seeing involves a lot
of bodiliness and a lot of grabbiness: the slightest twitch of
an eye muscle changes my visual input, and any change of the input
attracts my attention.

So we see that memory and seeing are situated
at two ends of a continuum of bodiliness and grabbiness. As expected
therefore, memory has nothing like what one would want to call
a "raw feel". Seeing on the other hand has lots of raw
feel.

It's interesting now to reflect on whether
there are intermediate cases.

Consider for example the experience of
being wealthy. Like seeing, wealthiness is a form of knowledge
about accessibility. It has more bodiliness than memory of latin
verbs however because wealthiness consists in the expectations
that when I do certain things with my body, I expect certain results.
For example, I ask the bank manager to give me the money, and
he does. However the things you do to get the results are things
which are not very intimately linked to your slightest body motion.
Thus I dont want to give wealthiness full marks on bodiliness.

Wealthiness on the other hand is not grabby:
unless I have a particularly conscientious bank manager or stock
broker, when my account is depleting or the market is down, nothing
advises me of this fact. No bell rings in my mind. But we see
that on the scale of raw feel, because it has a bit of bodiliness,
wealthiness gets better grades than latin verbs. This explains
why people do say, sometimes: I feel wealthy.

Consider now driving a car. Here bodiliness
is a bit more intimate than for wealthiness: the slightest twitch
of my foot on the accelerator or my hand on the steering wheel
have effects on the car and thereby on the sensory input. But
what about grabbiness. It's true that as I drive my attention
can be automatically attracted to various things happening, but
it's not ever attracted to the driving experience itself. Thus
I would say that driving has no grabbiness. Comparing with wealthiness
and memory of latin verbs we see that because of the extra bodiliness
there might be a bit more "feel" to driving. I think
again that this corresponds to people's intuitions: there is such
a thing as a "feel" to driving a car.

I must say that I find very promising
this use of the concepts of bodiliness and grabiness to make a
classification of the amount of "feel" that a mental
state or activity possesses. I think there may be a simple way
to extend this approach to include emotions and pain.

Conclusion

In conclusion then, the approach consisting
of taking the rather counternintuitive stance of saying that seeing
is not something ongoing, but rather a form of knowledge, like
memory, has at first been hard to swallow.

It led us to postulate that the impression
we have of seeing everything in the visual field is actually a
sort of illusion, generated by the immediate availability, by
a mere flick of the eye or of attention, of visual information.

It also led us to postulate that the impression
of continuousness or ongoingness of vision was also an illusion.
I suggested that the concepts of bodiliness and grabbiness might
account for why we have this illusion.

Now this approach seems at first rather
hard to swallow, it has some very interesting advantages.

The approach puts experience in the doing
of exploration, rather than in the brain. That way we escape from
the problem of having to find a brain mechanism that generates
experience.

The approach explains in a principled
way the differences in sensory qualities of the different sense
modalities. I havent had time to mention how it accounts for the
ineffability of sensations.

Finally the approach provides a neat classification
of the phenomenology of certain mental states like memory, wealthiness,
and sensation.

If you've been interested in this way
of thinking about vision, you may like to look at the paper we
have under review in BBS, where with Alva Noë we've more
carefully laid out the relation of this work to the problem of
consciousness and qualia, and where we've tried to show how the
view allows us to bring conveniently together strands of empirical
research not only from Change Blindness, but also from a variety
of other domains: among them sensory substitution, synesthesia,
and sensory adaptation.