Interaction

We designed this category to contain the most interactive part of the data set. Human-human as well as
human-robot interaction scenarios are inside the category. Each scenario consists of a natural scene in which
several human beings interact with each other and with the robot. In some cases one of the actors and/or the
robot act as a passive observer. This category contains six different scenarios.

Download

In the following you should find the preview video as well as
the download folder for each one of the characters. In the download folder you may find:

The sequence of the concatenation of the left and right images "sceneXX.mov".

The left, right, front and rear audio recordings in a four-channel file: "sceneXX.wav".

The synchronization data: frame number and time instant ("sceneXX.sync").

Annotation: some of the sequences are annotated. The annotation consists on the image and 3D position of the speakers as well as their speaking state. The annotated sequences are:

Asking for directions

In this scenario an actor asks the robot for directions to the toilets. The robot recognizes the question,
performs gender identification and gives the actor the right directions to the appropriate toilets. Six
different trials (four male and two female) were performed.

Scene 1

Scene 2

Scene 3

Scene 4

Scene 5

Scene 6

Chatting

We designed this scenario to study the robot as a passive observer in a dialog. The scenario consists of
two people coming into the scene and chatting for some undetermined time, before leaving. There is no fixed
script and the sequences contain several actions, e.g. hand shaking, cheering, ... Five different trials
were recorded.

Scene 1

Scene 2

Scene 3

Scene 4

Scene 5

Cocktail Party Problem

The Cocktail Party Problem has been matter of study for more than fifty years. In this scenario we
simulated the cocktail party effect: five actors freely interact with each other, move around, appear/disappear
from the camera field of view, occlude each other and speak. There is also background music and outdoors noise.
In summary this is one of the most challenging scenarios in terms of audio-visual scene analysis,
action recognition, speech recognition and dialog engaging.

Scene 1

Where is Mr. Smith?

The scenario was designed to test skills such as face recognition, speech recognition and continuous dialog.
An actor comes into the scene and asks for Mr. Smith. The robot forwards the actor to Mr. Smith’s office.
However, he is not there and when he arrives he asks the robot if someone was looking for him. The robot replies
according to what happened. Seven trials (five male and two female) were recorded to provide for
gender variability.

Scene 1

Scene 2

Scene 3

Scene 4

Scene 5

Scene 6

Scene 7

Introducing people

This scenario involves a robot interacting with 3 people in the scene. There are two versions of this
scenario: passive and active. In the passive version the camera is static, while in the active version the
camera is moving to look directly at faces of people. Together with the Cocktail Party Problem
scenario, they are the only exception where the robot is not static in this database.

In the passive version of the scenario, Actor 1 and Actor 2 interact together with Robot and each other; Actor 3:
only interacts with Actor 1 and Actor 2. In the active version, Actor 1 and Actor 2 interact with Robot and
each other; Actor 3 enters and leaves room, walking somewhere behind Actor 1 and Actor 2, not looking at robot.