Application histories have been used for a variety of purposes
including error recovery, browsing past activities, macro
recording and demonstrational interfaces. However, in most
systems the history is kept as a simple list of primitive
commands, which poorly reflects the user task structure. In this
paper we first present Command Trees, an alternate
representation of command histories that offers a richer
representation and provides better support for undo/redo
mechanisms and programming by demonstration. We then introduce a
new model to support incremental construction of command
trees and an object-oriented application framework that
implements this model. An important property of this model is
that it is independent of the interaction modality, thus
extending its purpose.

When interacting with the computer, users often reuse previous
activities either to consult them, to recover from an error or to
reduce repetitive tasks [Greenberg 93].
The system usually supports such reuse by recording user commands
in a history list. Unfortunately, lists only provide the user and
the system with a linear view of the past interaction, which
poorly maps the task structure.

In fact, some systems record a sequence of low-level events,
such as keystrokes and mouse actions, as is the case in certain
macro recorders (Tempo II [Affinity 91],
QuicKeys [CE Software 93]) and
certain predictive interfaces (Reactive Keyboard
[Darragh 92], Predictive Calculator
[Witten 81]). However, low-level events
convey few semantics, making it difficult for the system to
reliably replay them or to extract meaningful information.
Low-level events also make it difficult for the user to picture
the whole story: for example, the action of selecting the trash
on the desktop is easier to replay and to understand when
expressed as "Select Trash" rather than as "Mouse
Click at (623,383)". Nevertheless, such histories can be
implemented easily -- even across applications -- and can prove
very effective [Potter 93].

Another approach records high-level commands, like in some
undo/redo facilities (Chimera [Kurlander 90],
AIDE [Piernot 93], WeMet
[Rhyne 92]), macro recorders (AppleScript
[Apple 93a and Apple 93b],
HP NewWave [Fuller 89]), predictive
interfaces (Edward [Bos 92], Eager
[Cypher 91], Metamouse [Maulsby 89])
and programming by example systems (SmallStar
[Halbert 84], Chimera, Mondrian
[Lieberman 93], Pursuit
[Modugno 94], Mike [Olsen 88, AIDE).
Some systems (Chimera, Mondrian, Pursuit) take
advantage of the fact that high-level commands are recorded to
display their history using storyboards; they also use rules to
group commands together so that one pane can show more than one
command.

Observing that no system has attempted to represent
application histories in a more structured way than lists, Kosbie
[Kosbie 93] introduced the notion of
aggregate events -- events made of other events -- leading to a
tree-like representation of the history that we call Command
Tree. In this model, low-level events (e.g. "Type
Character") are grouped together to form abstract events
(e.g. "Type Text") which in turn can be merged and can
lead to higher-level events (e.g. "Replace Text") etc.
Such aggregate events better reflect the task structure than a
stream of low-level or high-level events because they correspond
to the decomposition of tasks into sub-tasks: events towards the
top of the tree are more general and represent high-level tasks,
whereas events towards the bottom of the tree are more specific
and represent low-level tasks. Moreover they allow the user and
the system to choose the granularity at which they want to
manipulate the history. As a consequence, command trees provide a
good basis for multiple-level undo/redo operations; for
demonstrational interfaces (by recording high-level events,
pattern matching algorithms are less sensitive to variations in
the way the task is performed); for capturing user intent (by
still keeping low-level events, the system knows the details of
how a high level event was performed: in some cases, the
low-level event appropriately represents the user's intent); and
for better visualization (by displaying a feedback closer to the
user task).

An example of such a command tree for a text editor is given
in Figure 1, the bottom of the figure
shows miniature screen snapshots representing the effects of
performed primitive commands. The task represented here
("Put Bullets") is replacing the first two characters
of line one and two by "* ". The top of the tree is the
root command, one of its children is "Edit Text" and
consists of two "Replace Text" commands.

Now that one approach to represent histories has been proposed
(i.e. aggregate events), we raise in this paper a new problem :
how to incrementally build command trees from the user's
commands? This issue is important because each time a new command
is carried out, histories should be updated accordingly and for
the sake of efficiency they should not be totally rebuilt. To
solve this problem, we introduce a new model where a command has
a prototypical parent which represents its natural
abstraction; where the history construction is performed by
keeping adopting commands; and where the eventual parent
ambiguity is solved by creating partial commands.

The remainder of this paper discusses how we implemented this
model in an extremely extensible way in AIDE, an
application framework assisting developers in integrating
undo/redo and programming by demonstration capabilities in
Smalltalk-based applications. We first present the AIDE
architecture and its main components: an extensible
incremental command parser and command trees. We then
present the framework from an implementation point of view. As a
proof of concept, examples are drawn from a text editor widget
which has been developed using AIDE. Finally, we conclude with
some future research directions.

2 THE AIDE ARCHITECTURE

2.1 Overview

The AIDE architecture (Figure 2)
represents application commands as command trees, objects
that understand the do and undo messages and
that can contain other commands. These commands are recorded in a
history -- itself a command tree -- constructed by the command
parser. Thus, when interacting with an AIDE-based
application, the user generates a stream of low-level events,
which the application transforms into a (primitive) command
object and sends to the command parser. The latter adds
the event to the history.

The architecture supports either a system-wide history (one
command manager for all running applications) or
application-specific histories (one command manager per
application). Details on how the architecture also supports
demonstrational techniques are given in [Piernot 93].
Our focus here is to show how the incremental
construction of command trees is achieved. To illustrate this
architecture and its associated mechanisms, we consider as an
example a text editor widget developed using AIDE that keeps a
structured history of the user commands and provides multi-level
undo/redo facilities. We describe in the next two sections each
of the components in more detail.

2.2 The Command Parser

The command parser is in charge of grouping commands together
into more general commands. The parser we developed is incremental,
i.e. adding a new command to the history does not force the
system to parse everything again -- which would be very
inefficient -- and assumes that each command has a prototypical
parent. Our parser tries to insert an incoming command into
the history by asking the latter to adopt the command. If the
command pre-defined parent already exists it is adopted;
otherwise, since the command knows its natural abstraction, the
parser generates the command parent and tries to include it in
the history the same way. For some commands, there may be an
ambiguity regarding their prototypical parent; in those cases,
the identity of the command parent is only known with the arrival
of subsequent commands. There are two approaches to solve this
problem: the first one is to pick one of the possible parents as
a guess and backtrack if wrong, the other one is to leave the
uncertainty with a partial command, whose identification
will be assigned later. We have chosen the second option so that
the command tree only reflects what the user has done so far.

As an example, in Figure 3 we give a
step-by-step description of how the text editor widget commands
have been added to the history to form the right side of the tree
depicted in Figure 1. To add a new command
(C) to the history, the parser first looks for the rightmost
branch of the history and tries to determine if any command along
this branch could adopt C, going upwards. If the parser finds a
command willing to adopt C, it performs the adoption. Two cases
can occur at that point: if the adoption generates a new command
(i.e. a partial command adopted a new child as in Figure 3b),
then the old command is removed from the tree and replaced by the
new one; otherwise, C is simply added to the command children. As
a result, one by one, each of the command's ancestors updates its
arguments to reflect the change that affects one of its children
(Figure 3c). If C is not adopted, the parser generates its parent
and repeats the process with its parent until an adoption occurs
(e.g. in Figure 3d "Type Character" could not be
adopted by "Delete Text", however its parent,
"Type Text" was adopted by "Replace Text").

2.3 Command Trees

As mentioned in the architecture overview section 2.1, command
trees represent application commands: as in MacApp
[Wilson 90], they are objects that
understand the do and undo messages, but they
can also contain other commands as well.

In Table 1 we illustrate the set of
commands we implemented in the text editor widget. The first
column represents the command class, the second one gives the
grammar governing how the command can adopt a new child and the
last one gives the command prototypical parent (parent command
class). For example, the "Select Text" command can have
as a child the following commands: "Select Insertion
Point" or "Select Range" or "Move
Cursor", and as a parent "Replace Text". The
"Select Text" parent class is "Partial
Command". This command is a temporary one, waiting to adopt
another command: this is due to the fact that the potential
parent for "Select Text" is either "Replace
Text" or "Format Text". If "Partial
Command" adopts "Type Text", it generates the
"Replace Text" command. If it adopts "Set
Font" or "Set Color" or "Set Style" or
"Set Size" it generates a "Format Text"
command. When a new command is generated by the adoption, the old
one (i.e. "Partial Command") gets removed from the tree
and the new method is added. To define the command grammar the
developers has to define the parentClass and canAdopt methods.

Command trees facilitate the work of programming by
demonstration engines because inference is less sensitive to
variations in the way the task is performed. In the "Put
Bullets" task (Figure 1), the user
replaces the first two characters of line one and two by invoking
different primitive commands, but both ways lead to the same
high-level command ("Replace Text").

Command

Rule governing children adoption

Parent Command

Select Ins. Point
Select Range
Move Cursor

No children (primitive command)

Select Text

Select Text

(Select Insertion Point | Select Range | Move Cursor)+

Partial Cmd

Partial Command

Select Text. Can adopt a command whose grand
parent can be
Edit Text. If so, adoption result is the parent of the
newly adopted command (i.e. adoption result is Replace
Text or Format Text)

Command trees offer better support for undo/redo mechanisms by
reducing the time it takes to navigate through the history
(either by undoing or by redoing commands). Since we have
intermediate commands, we do not need to undo/redo all the
primitive commands but instead we can undo/redo the intermediate
ones. For example to undo the last four commands of the tree in
Figure 1, it is enough to undo the
"Replace Text" command. We redefined undo and do
methods for all the commands except "Edit Text" (these
methods use the arguments of the commands) and are faster than
asking their children to undo or do themselves.

3 IMPLEMENTATION

3.1 Command Parser Algorithm

First, the parser keeps a pointer to the last executed
command. When commands are undone the command tree is not
modified, instead the pointer is updated. When a new command
appears, the undone commands are removed from the tree before the
new command is added (truncate/redo in the Archer, Conway,
Schneider script model [Thimbleby 91]).
The parser is encapsulated in the class Command -- as a
set of methods-- and its entry point is the add method (algorithm
outlined in Figure 4). This means that to
add a new command to the history, one sends the add message to
the root of the tree with the command to add as an argument.

Due to the recursive nature of our algorithm and to the fact
that the parser is encapsulated in the command class, an
application can extend the parser by overriding the try to
adopt method in its own commands. Thus, the default behavior we
described in section 2.2 could be
modified to fit the special needs of an application in case they
would not be fully addressed by the current parser. This point is
specially important with a system-wide history (one history for
all the running applications), where an application might want to
modify the parser behavior for its own commands, without breaking
the rest of the system.

To addaCommand
While I try to addaCommand and no adoption occurs,
I ask aCommand to create its parent and put it into aCommand.
If the adoption generated a new command,
I remove the old command and add the new one.
To try to addaCommand
If I am not a leaf, I ask my last child to try to addaCommand.
If an adoption occurred,
If it did not generate a new command, I update my Arguments.
I return the adoption result.
Otherwise,
If I can adoptaCommand, I adoptaCommand and return the result.
Otherwise, I return that no adoption occurred.

Figure 4. Part of the incremental command parser

3.2 Description of class Command

The class Command (Table 2) is an
abstract subclass of class Tree. As such, it inherits all the
behavior of trees (it can have children -- sub-commands -- and
provides a means to access its descendants) and it adds some
methods of its own. Among these methods are do, undo
and name which govern the command's execution and return
the command's name. To implement a new kind of command, the
developer creates a subclass of Command, adds the
command arguments as object variables, and redefines some of the
methods mentioned above. Moreover, the developer has to decide
how commands should be grouped together to form higher level
commands. This is done by defining the command's parent class (parentClass
method), defining how the command should update its arguments
when one child is added to it, removed from it or itself too
changes some of its arguments (updateArgs method), and
defining the commands it can adopt (canAdopt method).
Some methods such as do, undo and canAdopt
may not have to be redefined when the default behavior is
satisfactory. Thus by default, the do and undo
methods go through the command children and send the same message
to them. This means that do and undo can be
implemented only in primitive commands (commands with no
children, i.e. leaves of the command tree). Similarly, the canAdopt(self,cmd)
method checks if the receiver can have cmd as child by
checking if cmd parent class is the same as the receiver
class and thus does not need to be overridden in most cases
unless the arguments need to be taken into account.

One major concern related to the implementation of histories
is that the command history can be very large. Our solution is to
discard children of old commands. For intermediate commands (e.g.
"Type Text"), removing their children does not prevent
them from being done and undone if they define their own
arguments (e.g. text which has been typed), do and undo
methods. We implemented this mechanism for all the non primitive
commands to provide a faster navigation in the command tree.

Class Command
subclass of Tree

name(self)

Returns the name of self.
Should be overridden

do(self)

Executes self.
Must be overridden at least by primitive commands (leaves).

undo(self)

Undoes self.
Must be overridden at least by primitive commands (leaves).

parentClass(self)

Answers the class self
parent should be member of. Must be overridden.

updateArgs(self)

Updates self's
arguments (using its children). Must be overridden.
Called when self children change.

canAdopt(self,cmd)

Returns whether self
can adopt cmd (uses parentClass).

adopt(self,cmd)

Determines the command
resulting from the adoption (the receiver or a new
command), then adds cmd to the children
of the resulting command (uses updateArgs) and
finally returns it.

In this paper, we first presented command trees as a better
history representation than lists, offering improved support for
error recovery mechanisms and demonstrational techniques. We then
introduced our model (prototypical command parent, adoption
mechanism and partial commands) and showed that it is a practical
and an elegant solution to the problem of incrementally
constructing command trees. We also presented how we successfully
implemented this model in an object-oriented application
framework, and we illustrated the framework use through a text
editor widget developed with it.

Kosbie [Kosbie 94] adopts a more
lightweight approach than ours, by letting the programmer define
the command grammar declaratively. To do so the user creates a
hieractor which describes under which conditions a high-level
event is to be generated. Such conditions include a trigger
event, a running event and a stop event. As an example, the
hieractor for a button as "Mouse Down" for trigger
event, "Mouse Move" for running event, and "Mouse
Up" for stopping event. Unfortunately this approach does not
take command arguments into account: indeed, in a drawing
application it is impossible to state that the "Rotate
Object" command can be made of other "Rotate
Object" commands, provided that they apply to the same
object, and that they have same rotation center. This kind of
relation can be expressed in our framework. Also our approach
requires more work from the developer, this problem can be
alleviated if the developer properly structures the command class
hierarchy. An advantage of our approach is that it is not limited
to widget-based interfaces and would work for speech or command
line interfaces as well.

We are currently using the framework to include programming by
demonstration capabilities to a telecommunication application
[Yvon 94]. We plan to extend the framework to
allow users to retrieve information about past sessions according
to certain criteria, such as date, time, etc. We also intend to
explore two ideas: the use of storyboards to present the history
tree to the user and the use of the model in the context of
robotics applications.

ACKNOWLEDGMENTS

We would like to thank Norbert Cot, Tom Gruber and Walter
Sujansky for help with this paper. This research is partially
sponsored by a grant from the Centre National de la Recherche
Scientifique and the Centre National d'Etudes des
Télécommunications (project CNET/CNRS/COGNISCIENCE with the
LIAP-5 laboratory).

Witten I., "Programming by Example for the Casual User: A Case
Study," Proceedings of the Seventh Conference of the
Canadian Man-Computer Communication Society, CMCCS, Waterloo,
June 1981, pp. 105-113.