2. Example-tracing Tutors

This section will walk you through the main phases of authoring an
Example-tracing Tutor with CTAT. For a full, example-based tutorial, see
the tutorials
section of the CTAT web site.

There are four main phases for authoring an Example-tracing
tutor:

Demonstrate: in the student interface
(typically Java or Flash), demonstrate examples of correct and
incorrect behavior for each problem for which students will be
tutored. Alternate ways of solving a problem are recorded as
separate paths in the graph.

Generalize: in the Behavior Recorder, define
ways in which the tutor can recognize a broader range of behavior.
Methods for this include specifying ordering constraints so that
students can solve a problem in orders other than the one in which
the problem was demonstrated; enhancing 'matching' to generalize the
conditions under which a demonstrated step is matched; and marking
certain steps (links) in the graph as optional.

Test: use the tutor, playing the role of
student attempting to solve the problem. If possible, field the
tutor with a user or number of users to get feedback early and
often.

2.1. Demonstrate

Authoring a single graph (problem) is done by demonstrating
(performing) problem-solving actions. During this phase of authoring,
you will commonly have the Behavior Recorder window showing in CTAT
along with a student interface with which you'll interact.

To record your problem-solving demonstration, set Author Mode to
Demonstrate. Now, when you interact with the student interface, new
action links and interface state nodes will appear in the behavior
graph displayed in the Behavior Recorder.

Generally, you will demonstrate a preferred, correct solution
path, and revisit points in the graph to add common student
misconceptions in the form of both incorrect and suboptimal actions,
and alternative correct paths. To jump to a state in the graph, click
the state node.

When in Demonstrate mode, all steps are initially created as
Correct Action links. To mark an action as incorrect or suboptimal,
right-click the link's label and select an option from the menu Change
Action Type.

The last step of a demonstration should be a done step, that is,
a step where the student signals completion of the problem by pressing
a Done button.

Save your graph by selecting File >Save Graph As.

2.2. Generalize

A complete graph can be generalized to account for various
problem-solving strategies, as well as specific student input and
ordering constraints.

2.2.1. Input Matching

In an Example-tracing Tutor, each action that a student
performs is compared by the tutor against the set of potential
actions described in the graph. Specifically, the tutor compares the
student's 'selection' (widget), 'action' (interaction type), and
'input' (entered value) with those on the links of the graph.
Whether this comparison is against all links in the graph or a
specific subset of links depends on the graph's ordering constraints
(see Ordering Constraints).

When CTAT compares a student action with that defined on a
link, it can do so precisely (an exact match), or by another method
(eg, a regular expression, range, wildcard, or any value). The type
of comparison is specified on each link in the graph. By default,
the type of comparison is an 'exact' match. Other types of input
[action?] matching are available in CTAT and can be specified for a
link.

Exact Match

The default matching type. Requires that the student's
selection, action, and input match exactly with that specified
on the link.

Any Match

Any input value for the specified selection and action
is considered a match.

Range Match

An input value within the specified numeric range is
considered a match. Includes the minimum and maximum values
(eg, 0-100 would match inputs of 0, 2, 10.12, or 100). A range
cannot be specified for selection or action—these are exact
matches.

Wildcard Match

A value that matches the wildcard pattern is considered
a match. A wildcard * denotes any number of characters.
Special characters such as {, \, or & should be avoided. A
wildcard expression can be used for selection, action, and/or
input.

A value that matches the regular expression is
considered a match. Uses the Java regular expression engine. A
regular expression can be used for selection, action, and/or
input. Regular expression matching is the most powerful type
of matching in CTAT.

Example 1: The regular expression ^cat$
would match 'cat', but not 'a cat', 'cat one', or 'Cat'; the
'^' character specifies beginning of a line, and '$' specifies
end of a line.

Example 2: The regular expression animal.
would match 'animals' or animales', but not 'animal'; the '.'
character represents any single character.

Example 3: The regular expression [THK]im
would match 'Tim', 'Him', or 'Kim', but not 'Jim'; any single
character from the set of characters between the square
brackets can be used in the match.

Example 4: The regular expression a{3}
would match 'aaa' or 'aaaaaa' (it would match this twice), but
not 'aa'; the number between the braces specifies the number
of times the preceding character must occur to match.

Example 5: The regular expression .*hello
would match 'hi hello', 'xhello', or 'hello'; the '*'
character represents zero or more occurrences of the preceding
character, and '.' represents any character.

Example 6: The regular expression \d\w\W
would match '0z!' or '09@', but not 'b1+' or '4!!'; \d
represents a digit (0-9), \w represents a word character
(a-zA-Z_0-9), and \W represents a non-word character.

2.2.2. Student- and Tool-Performed Actions

Steps in a behavior graph can be either
student-performed (steps the student should
perform) or tool-performed (steps that the
tutor itself should perform). While a student-performed step
specifies behavior the student should perform while using the tutor,
a tool-performed step specifies an action that the tutor software
will perform on the interface. Tool-performed steps are therefore
powerful links that can be used to complete problem-solving steps,
make changes to the student interface such as hiding or showing a
widget, or perform other actions that you specify.

By default, demonstrated steps are stored in the behavior
graph as correct steps that a student should perform when using the
tutor. Steps such as these have an actor of
student; that is, they are performed by the student. For steps where
the tutor performs the step, the actor is the tool itself—a
tool-performed action.

To specify a tool-performed action:

Create a new link (optional) - while
in Demonstrate mode, demonstrate a step in the student
interface. For steps that cannot be demonstrated (e.g., hiding
an interface widget), create a blank step in the graph by
right-clicking a node in the graph and choosing Add
Blank Step.

Right-click (Windows) or
Control-Click (Mac) on the action label of the link in the graph
for which you'd like to specify a tool-performed action.

Note

Tool-performed actions can only be specified for links where
the matching type is Exact Match.

Timing of tool-performed actions

A tool-performed action link in a graph is activated by the
tutor whenever the state before the tool-performed link becomes
the current state. In an ordered group of links, this process is
intuitive: when the state preceding the tool-performed action link
becomes the current state, the tool-performed action link is then
activated. Slightly less intuitive is that this criterion—that a
state with a single outgoing tutor-performed action link becomes
the current state—applies for unordered groups of links as well
(see Section 2.2.3, “Link Groups and Ordering Constraints”
below).

2.2.3. Link Groups and Ordering Constraints

In an Example-tracing tutor, student problem-solving steps are
represented in a behavior graph (BRD) as links in a graph. Also
represented in the graph is the notion of order: in what order (if
any) should the correct steps be performed?

In earlier versions of CTAT, a Behavior Recorder graph could
be one of two types: ordered or unordered. In an ordered graph, the
student had to perform the entire series of steps (links) in the
graph in order, on a single path from the start state node to the
final node, for the problem-solving behavior to be considered
correct. In an unordered graph, the student could perform the steps
in the graph in any order for the behavior to be considered correct;
for sections where the graph branched, the student had to perform
all steps on the branch they chose and could do so in any order.
Later, the ordered graph type was extended to include the notion of
an unordered group of links—a set of links that, once encountered,
could be performed in any order. Groups could only be of the
unordered type, and could not be nested.

In CTAT 2.0, the concept of ordering constraints has been
extended to allow for an arbitrary number of link groups, each being
either ordered or unordered. The root, or universal, group of the
graph must also be either ordered or unordered; therefore, a graph
with a single group of links emulates one of the simpler graph types
(ordered or unordered) of the past.

See the CTAT tutorial on Fraction Addition to see how ordering
constraints might be used in a real-world tutoring scenario.

To define link groups:

Note

This task requires editing an XML file in a text editor.
As of CTAT 2.0, only a user interface for adding unordered link
groups to a single ordered group exists. Although arbitrary
nesting of groups is possible, creating such groups can only be
accomplished by editing the BRD file in a text editor.

In a text editor, open the desired behavior graph (BRD
file) for which you'd like to create link groups.

Identify the link IDs of links that you'd like to group.
To do so, locate <edge> elements in the file.
Each <edge> element describes a link in the
graph. Examine the <edge> element for
identifying characteristics such as selection, action, and input
values, or hint messages. When you've found an edge (link) that
you'd like to group, note the value of its
<uniqueID> element—this will be used later
when defining groups.

When you've noted the <uniqueID>s of a
set of links, you can define a group using syntax similar to
that shown in the code example below.

At the bottom of the BRD file, add an
<EdgesGroups> element, including its closing
tag, </EdgesGroups>. Within this element, add
a <group> element, giving it attributes
name and unordered. Name is a
descriptive name of your group; unordered is a boolean variable
(true or false) distinguishing the group as unordered (true) or
ordered (false). Be sure to close to the group element with
</group>.

Within the group element, add <link>
elements for each link that you'd like to make a member of that
group. Identify the link using the id
attribute—this is the value of <uniqueID>
that you noted when browsing the <edge>
elements. Note that link elements are empty elements
(self-closing elements).

Save your BRD file and open it with CTAT (File
>Open Graph).

Test the tutor in Test Tutor mode. If
the tutor does not work as expected, examine the group syntax in
the BRD file to verify that it conforms to the example below.
Also verify that the link IDs map to valid (and expected) edge
uniqueIDs.

Editing unordered link groups in an ordered graph

Using the Edit Groups dialog, you can
define multiple unordered groups of links within a single ordered
group. That is, at the highest level, the behavior graph is
ordered; and series of links can be grouped to form unordered
groups.

In an unordered group, the student can visit the states in
that group in any order. The groups themselves, however, must be
visited in the order they are shown in the graph. States that are
not part of any group must also be visited in the order shown in
the graph.

Note

Although arbitrary nesting of groups is possible in CTAT
2.0, the Edit Groups dialog only supports a
single level of nested groups: unordered groups within the
graph's root ordered group. Also, a path from one node to
another can be used in an unordered group only if there are no
other paths coming off of it (other than incorrect action or
untraceable error links).

To define a new unordered link group:

Select Graph> Edit
Groups ...

Click New.

Enter a Group name.

Select a First link description
from the first combobox. This is the first link of the group
of unordered links.

Select a Last link description from
the second combobox. This is the last link of the group of
unordered links.

Click Close.

To edit an unordered link group:

Select Graph> Edit
Groups ...

Select the name of the group that you'd like to
edit.

Make any changes to the Group name,
First link, or Last
link options.

Click Close.

To delete an unordered link group:

Select Graph> Edit
Groups ...

Select the name of the group that you'd like to
delete.

Click Delete.

Click Close.

2.3. Annotate

2.3.1. Hints

When a student requests a hint, the first (top-level) hint
from the relevant link is displayed. A link can have multiple levels
of hints, where each successive hint request by the student displays
the next level of hint.

Tip

The last hint is commonly a 'bottom-out' hint, a hint that
informs the student of the correct answer so that they can
continue. This approach, however, can facilitate students 'gaming'
of the tutoring system by clicking through to the final hint for
each step of the problem.

Note

Only correct action links can have hints associated with
them. If you switch the action type of a correct action link to
one of the other types, you will be prompted to choose what to do
with the existing hints—you can convert them to buggy (error)
messages or remove them from the link.

To define hints for a correct action link:

Click a correct action link in the Behavior
Recorder.

Select Edit Hint and Success
Messages.

Enter hints into the hint message text areas.
Alternatively, you can copy hints from another link or rule by
selecting the link or rule name from the combo boxes at the
bottom of the Edit Hint and Success Messages dialog box.

(Optional) Click Add Hint Level to
define more than three levels of hints.

Click Done.

Figure 2.2. Edit Hint and Success Messages dialog

2.3.2. Feedback

Feedback for a student action can be specified on all types of
links. It is given after a student action is evaluated by the
tutor.

For correct action links, feedback is a success message. For
an incorrect or suboptimal action, feedback is an error ("buggy")
message.

To define a success message:

Click a correct action link in the Behavior
Recorder.

Select Edit Hint and Success
Messages.

Click More Options.

Enter a success message in the Please Edit
Success Message text area.

(Optional) Choose whether or not the success message
should be copied to links with a) the same
selection/action/input, and/or b) the same production
rule.

Click Done.

To define a 'buggy' (error) message:

Click an incorrect or suboptimal action link in the
Behavior Recorder.

Select Edit Bug Message.

Enter a buggy (error) message in the text area of the
dialog box. Alternatively, you can copy the buggy message from
another link in the graph by selecting the link's description
from the combo box.

(Optional) Choose whether or not to copy the buggy message
to all links with the same selection/action/input.

Click OK.

Figure 2.3. Edit buggy message dialog

2.3.3. Skills [Knowledge Components/Labels?]

Each link on a behavior graph can have any number of skill
labels associated with it. A skill label is a tag used to describe a
piece of acquired knowledge that is associated with a step (link) in
the graph. When applied consistently to steps in a problem, skill
labels facilitate many types of analysis on the data generated by
student use of the tutor. Such analysis based on knowledge
components can be accomplished using the PSLC
DataShop, a web application that facilitates learning science
research through advanced reports and queries. Using the DataShop,
you can easily produce learning curves and other reports from log
data coded with knowledge component labels.

To collect this data for later analysis, logging must be
turned on for the tutor. See Logging for
instructions on configuring and enabling logging, including logging
to the PSLC DataShop.

Note

If Knowledge Component labels are not visible in the graph,
click Graph >Show Skill
Labels.

To associate a knowledge component with a step:

Single-click the default skill ('unnamed') or another
existing skill that you'd like to change.

Select Edit Skill Name.

In the Edit Skill Name dialog (figure below), enter a new
skill name in the combo box or select an existing skill name
from the drop-down menu.

(Optional) Enter a new skill set name or select an
existing one from the drop-down menu.

Click OK.

Figure 2.4. Edit Skill Name

To disassociate a knowledge component from a step:

Single-click the knowledge component that you'd like to
disassociate from the step.

Select Delete Skill Name from
Link. You'll be asked whether or not you'd like to
delete the skill name.

Click Yes to remove the skill name
from the link; to leave the skill association as is, click
No.

With knowledge component labels applied, you can view a matrix
of knowledge component usage by problem.

To view a knowledge component matrix:

Select Windows >Skill
Matrix.

Using the Problems Organizer, select a subset of problems
(or a single problem) by selecting course, unit, section, and
problem name from the drop-down boxes.

Click OK to view the skill matrix
for the problems you've selected.

The Skill Matrix (pictured below) shows problem names
vertically along the left-hand side of the table and skill names
horizontally across the top. As skill names are often long,
shortened identifiers are used (S1, S2 ...) to represent skills;
these are defined below the table. The numbers in the table are
counts of the number of times each skill appears in each
problem.

In this mode, you can use the tutor, playing the role of a
student attempting to solve a problem. Feedback, hints, and other
messages will be enabled.

To jump to a state of the problem—partially solved, for
example—click a node in the behavior graph. The student interface will
update to reflect that state. You can then continue testing the tutor
from that state.

To reset the problem, click on the start state of the behavior
graph (the first node in the graph).

Changing the tutor's feedback and hint messages can be done
while in Test Tutor mode. To demonstrate new behavior for the problem,
you will need to switch to Demonstrate mode