lated unmanned aerial vehicle (UAV) to conduct various missions. At any time, these users chose whether
to teach by demonstration, by feedback, or by providing an example of a concept. They could also test
the agent to see what it had learned. The authors
found that users never taught exclusively by feedback, instead generally using it after teaching by the
other available means. Together, these two studies
provide insight into the design of natural interfaces
for teaching agents.

People Naturally Want to
Provide More Than Just Data Labels
Labeling data remains the most popular method for
end-user input to interactive machine-learning systems because of its simplicity and ease of use. However, as demonstrated in previous case studies, label-based input can have drawbacks (for example,
negative attitudes toward being treated as an oracle).
In addition, emerging research suggests that in some
scenarios users may desire richer control over
machine-learning systems than simply labeling data.

For example, Stumpf and colleagues (2007) conducted a study to understand the types of input end
users might provide to machine-learning systems if
unrestricted by the interface. The authors generated
three types of explanations for predictions from a
text classification system operating over email messages. These explanations were presented to people
in the form of paper-based mockups to avoid the
impression of a finished system and encourage people to provide more feedback. People were then asked
to give free-form feedback on the paper prototypes
with the goal of trying to correct the classifier’s mistakes. This experiment generated approximately 500
feedback instances from participants, which were
then annotated and categorized. The authors found
that people naturally provided a wide variety of input
types to improve the classifier’s performance, including suggesting alternative features to use, adjusting
the importance or weight given to different features,
and modifying the information extracted from the
text. These results present an opportunity to develop
new machine-learning algorithms that might better
support the natural feedback people want to provide
to learners, rather than force users to interact in limited, learner-centered ways.

People Value Transparency inLearning SystemsIn addition to wanting richer controls, people some-times desire more transparency about how theirmachine-learning systems work. Kulesza and col-leagues (2012) provided users of a content-basedmusic recommender with a 15-minute tutorial dis-cussing how the recommender worked and how var-ious feedback controls (for example, rating songs,steering toward specific feature values, and so on)would affect the learner. Surprisingly, participantsresponded positively to learning these details aboutthe system. In addition, the researchers found thatthe more participants learned about the recom-mender while interacting with it, the more satisfiedthey were with the recommender’s output. This casestudy provides evidence that users are not always sat-isfied by “black box” learning systems — sometimesthey want to provide nuanced feedback to steer thesystem, and they are willing and able to learn detailsabout the system to do so.

Examining transparency at a more social level,
Rashid and colleagues (2006) examined the effect of
showing users the value of their potential movie ratings to a broader community in the MovieLens recommendation system. Users who were given information about the value of their contribution to the
entire MovieLens community provided more ratings
than those who were not given such information,
and those given information about value to a group
of users with similar tastes gave more ratings than
those given information regarding the full MovieLens community.

Transparency Can Help People
Provide Better Labels
Sometimes users make mistakes while labeling, thus
providing false information to the learner. Although
most learning systems are robust to the occasional
human error, Rosenthal and Dey set out to solve this
problem at the source. They sought to reduce user mistakes by providing targeted information when a label
is requested in an active learning setting. The information provided to the user included a combination
of contextual features of the sample to be labeled,
explanations of those features, the learner’s own prediction of the label for the sample, and its uncertainty
in this prediction (Rosenthal and Dey 2010).

They conducted two studies to determine the
subset of such information that is most effective in
improving the labeling accuracy of users. The first
involved people labeling strangers’ emails into categories, as well as labeling the interruptability of
strangers’ activities; the second involved people
labeling sensory recordings of their own physical
activity. Both studies found that the highest labeling accuracy occurred when the system provided
sufficient contextual features and current predictions without uncertainty information. This line of
research demonstrates that the way in which information is presented (for example, with or without
context) can greatly affect the quality of the
response elicited from the user. This case study also
shows that not all types of transparency improve
the performance of interactive machine-learning
systems, and user studies can help determine what
information is most helpful to the intended audience.