VRR: What do you think is the greatest obstacle facing the VR
industry and why? Latta: Lack of basic research. The issues of having the most
intimate form of human computer interface in Virtual Reality
necessitate a thorough understanding of human perceptual, muscle and
psychological systems. Yet that research foundation does not exist.
Dr John Latta interviewed in Virtual Reality Report,
2 (7) p 4.
Taken from: [195, page 43,]

In real life, humans use machines, e.g. a car or a machine tool, and
interact with them in a multimodal way, i.e. by means of perception
(vision/hearing/haptic perception), and action (knob/wheel/lever/button
control). In the case of virtual reality, the purpose is simply to
substitute the sensory-motor flow elicited by a physical ambient
environment with a computer generated sensory-motor flow which
imitates/emulates (a) physical reality to a sufficient degree. Flight
simulators are a good example. The purpose is not to improve or optimize
man-machine interaction but to reproduce the standard one, with the goal of
training the user or simply entertain him. Imitation/emulation is a
possible paradigm of man-machine interaction, but it is not the only one.
Due to the decoupling of the sensory-motor flow from the physical world,
new environments can be generated based on environments which are physical
in itself, but of a completely different scale (molecular surface
landscapes vs galactic environments based on astronomical data), and which
would normally never elicit an ambient sensory-motor flow.

More generally, one can define paradigms in which multimodality is used for
improving the interaction between the user and reality via a computer:

Figure 4.9
: The computer as a mediator (or agent,
see 5.2 ) between the user and the world

It can be seen from this list that VR will have a great impact in the
future in many different domains. The techniques and devices which are
needed in order to let the user immerse in the virtual world created by a
computer are currently under development. In this sense, and because of the
money that will be spent for VR research, it may be called one of the
driving forces of multimodal technology.

Ellis (in [89]) and Null and Jenkins(in [247]) from NASA
labs on Virtual Reality (VR) argue that Virtual Environments (they use this
name instead of VR), presented via head-mounted computer-driven displays,
provide a new medium for man-machine interaction. Like other media, they
have both physical and abstract components. Paper, for example, as a
medium for communication, is itself one possible physical embodiment of the
abstraction of a two-dimensional surface onto which marks may be made. The
corresponding abstraction for head-coupled, virtual image, stereoscopic
displays that synthesize a coordinated sensory experience is an
environment. It is important to collocate VR (or VEs, following NASA) with
respect to man-machine interaction.

In summary, we can understand multimodal interaction as a process
characterized as follows: (i) the computer is able to capture the largest
possible part of the human motor outflow, (ii) human movements are as
little constrained as possible, (iii) the human receives from the computer
a perceptual inflow which maximally utilizes the different available
channels, (iv) the inflow and the outflow are optimally tuned, in relation
with the specific task.