at-a-glance information on problem coverage—the number of students exposed to a
particular step or knowledge component, the knowledge components associated with each step, and
the the problems that contain each knowledge component.

Viewing by problem

When viewing by problem, you can view only one problem at a time. Change the problem by clicking a
problem's name from the scrollable list in the navigation sidebar.

The current problem is broken down by step, and lists the various attempts students made.

At the top of the page, the full problem name is shown.

Steps

A problem is commonly broken down into multiple steps. For each step, a list of details about
that step are displayed.

Step Name

The first line of step information is the step name. The format of the step name varies
across tutor technologies and logging implementations. CTAT tutors, for example, characterize
step name as the widget the student interacted with (ie, the 'selection') and the action the
student performed on that widget. (Student input—the third component of the
selection-action-input format—is not displayed in this portion of the table since it
varies by student response. Instead, if it exists, it appears under the Answer
column of each row in the Attempts table.)

Number of Students

The total number of students that attempted this step of the problem.

Knowledge Component(s)

The knowledge component(s) that are associated with this step of the problem.

Sample

The name of the sample(s) from which the shown data comes. The sample name here will
one of the selected samples shown in the upper-left corner of the sidebar.

Attempts

Figure 1: Evaluation Types

Evaluation

Student attempts are classified into three evaluation types: correct, incorrect, and
hint. Each evaluation type corresponds with a background color (see Figure 1).

Number of Students

The number of students that attempted this step. Expressed in the form: count
(percentage of students that attempted this step).

Answer

The answer that the student provided on his or her first attempt for this step

Feedback

The feedback message provided by the tutor to the student.

Classification

Used to further describe the student-problem interaction (e.g., some tutors employ
multiple hint and error levels).

Viewing by knowledge component

When viewing by knowledge component, you can select multiple
knowledge components, but each will be shown in its own table row.

Values shown are aggregates by knowledge component. From these
you can tell, for instance, the more difficult knowledge components
by examining the percentages on corrects, errors, and hints.

Knowledge Component

Knowledge Component (KC) Name

Displayed in bold type.

Number of Observations

The total number of times a student took an opportunity to demonstrate the knowledge
component.

Problem(s)

The problem(s) that contain steps with this knowledge component.

Sample

The name of the sample(s) from which the shown data comes. The sample name here will
one of the selected samples shown in the upper-left corner of the sidebar.

Aggregate Values

Evaluation

Student attempts are classified into three evaluation types: correct, incorrect, and
hint. Each evaluation type corresponds with a background color (see Figure 1).

Number of Observations

The number of times a student took an opportunity to demonstrate the knowledge
component, broken down by evaluation. Expressed in the form: count (percentage of observations
for this KC).

Sample Selector

Sample Selector is a tool for creating and editing
samples, or groups of data you compare across—they're
not "samples" in the statistical sense, but more like filters.

By default, a single sample exists: "All Data". With the Sample
Selector, you can create new samples to organize your data.

You can use samples to:

Compare across conditions

Narrow the scope of data analysis to a specific time range,
set of students, problem category, or unit of a curriculum (for example)

A sample is composed of one or more filters, specific
conditions that narrow down your sample.

Creating a sample

The general process for creating a sample is to:

Add a filter from the categories at the left to the composition
area at the right

Modify the filter to select the subset of data you're interested
in, saving it when done

View the sample preview table to see the effect of adding your filter,
making sure you don't have an empty set (ie, a filter or combination
of filters that exclude all transactions).

Name and describe the sample

Decide whether to share the sample with others who can view the
dataset

Save the sample

The effect of multiple filters

DataShop interprets each filter after the first as an additional
restriction on the data that is included in the sample. This is also known
as a logical "AND". You can see the results of multiple filters in the
sample preview as soon as all filters are "saved".