KARMA

Knowledge-based Augmented Reality for Maintenance Assistance

Steven Feiner, Blair MacIntyre, Dorée Seligmann

We believe that one of the most powerful uses of virtual worlds will not
be to replace the real world, but rather to augment the user's view of the real
world with additional information.
This idea, introduced by Ivan Sutherland's pioneering work on
head-mounted displays, is often referred to as augmented reality.
For example, graphics and text overlaid on the surrounding world could
explain how to operate, maintain, or repair equipment, without
requiring that the user refer to a separate paper or electronic manual.
Similarly, participants in a business meeting could interact with a dynamic
shared financial or organizational model represented in 3D and
selectively supplemented with each user's personal (and private) annotations.
Generating such material by hand, however, will require
a tremendous amount of expertise and effort, far greater than that currently
needed to design ``hand-crafted'' hypermedia and multimedia
presentations.
We have begun to explore how overlaid explanatory graphics can instead be
designed on the fly automatically using AI techniques.
This work builds on our current research on the knowledge-based generation of
3D graphics
[Seligmann & Feiner 91]
and multimedia presentations
[Feiner & McKeown 91]
that explain physical tasks.

KARMA

We are developing KARMA (Knowledge-based Augmented Reality for Maintenance
Assistance)
[Feiner, MacIntyre, and Seligmann 93],
a prototype system that that uses our see-through head-mounted display to
explain simple end-user maintenance for a laser printer. We
attached several Logitech 3D trackers (the small triangles in the figure
shown above) to key components of the printer, allowing the system to
monitor their position and orientation.
A modified version of the IBIS rule-based illustration
generation system
[Seligmann & Feiner 91]
interactively designs graphics and
simple textual callouts that fulfill a set of goals that are input to KARMA.
This virtual world is intended to complement the real world on which it
is overlaid.
For example, one rule states that if a goal is to show the user where
an object is located, the system must determine if the object is blocked by
other objects. If it is blocked, it will be displayed so that it appears
to be seen through the blocking objects; if it is already visible in the
real world, it need not be drawn at all.

Overlaid graphics

This figure shows a virtual world designed by KARMA, viewed ``live''
through our see-through head-mounted display. It was generated to show the
user how to remove the paper tray from the laser printer. The virtual world
contains a highlighted representation of the paper tray that tracks the
tray as it moves, a dotted line ``ghosted'' view of the tray's desired
destination state, and an arrow that pulses outward to represent the
action of pulling out the tray.

This research was supported in part by the Office of Naval Research
under Contracts N00014-91-J-1872 and N00014-94-1-0564, a gift from NYNEX
Science & Technology, the New York State Center for Advanced Technology
in Computers and Information Systems under Contract NYSSTF-CAT-92-053,
the Columbia Center for Telecommunications Research under NSF Grant
ECD-88-11111, and NSF Grant CDA-92-23009.