ABSTRACT Several emerging computer devices read bio-electrical signals (e.g., electro-corticographic signals, skin biopotential or facial muscle tension) and translate them into computer- understandable input. We investigated how one low-cost commercially-available device could be used to control a domestic robot. First, we used the device to issue direct motion commands; while we could control the device somewhat, it proved difficult to do reliably. Second, we interpreted one class of signals as suggestive of emotional stress, and used that as an emotional parameter to influence (but not directly control) robot behaviour. In this case, the robot would react to human stress by staying out of the person's way. Our work suggests that affecting behaviour may be a reasonable way to leverage such devices.

ABSTRACT Several emerging computer devices read bio-electrical signals (e.g., electro-corticographic signals, skin biopotential or facial muscle tension) and translate them into computer- understandable input. We investigated how one low-cost commercially-available device could be used to control a domestic robot. First, we used the device to issue direct motion commands; while we could control the device somewhat, it proved difficult to do reliably. Second, we interpreted one class of signals as suggestive of emotional stress, and used that as an emotional parameter to influence (but not directly control) robot behaviour. In this case, the robot would react to human stress by staying out of the person’s way. Our work suggests that affecting behaviour may be a reasonable way to leverage such devices. Categories and Subject Descriptors I.2.9 Robotics. Operator interfaces General Terms Experimentation, Human Factors Keywords Robots, bio-electric signals, emotional control, OCZ NIA™, iRobot® Roomba®, emotional instrument 1. INTRODUCTION The concept of ‘brain-computer interaction’ involves one or two-way communication between a human brain, and an external computer device. Our particular interest is in how such a system could be used to direct robot behaviour. This is not the realm of pure science fiction: Millan et. al. [1] demonstrated how two people could successfully move a robot between rooms using an EEG-based interface that recognized three mental states. Yet we wondered if direct control is the best (and only) way to exploit such an interface, especially given the foreseeable limitations inherent in such devices (e.g., accuracy, number of input states, difficulty of controlling such a system). Our research focused on using a very limited type of brain-computer interaction to control an iRobot® Roomba® vacuum cleaner robot. In particular, we used the OCZ NIA™ neural impulse actuator [2], an off-the-shelf, low-cost commercial interface designed for video game use that reads bioelectrical signals. We linked this input device to the iRobot® Roomba®, a robot vacuum cleaner, and constructed two methods allowing people to control the Roomba®. The first was direct control, where a person controlled their bioelectrical signals to directly affect robot speed. The second was behavioural control, where a person’s emotional state was inferred based on their bio-electrical signal state, and the robot adjusted its behaviour to fit that person’s emotional state. 2. DESCRIPTION 2.1 The System The OCZ NIA™ consists of a headband worn by the person on their forehead (Figure 1). The manufacturer claims that the device reads “bioelectric signals that are amplified, digitized and further de-convoluted into computer commands”, where these bioelectric signals are collectively generated by facial muscles, eyes, and the brain [2]. It is somewhat unclear what signals are actually being read, but one reviewer [3] suggests that its sensors read skin biopotentials, i.e., small electrical changes on the surface of the skin. Software calibration must be done before every use, and the software displays live measurements for muscle tension, eye glancing as well as alpha and beta waves. The device’ software convolutes its sensor reading into keystrokes mapped to various computer games. We were able to customize this convolution to our own uses, i.e., to control the iRobot® Roomba® through its API (Figure 1, right side). However (and in spite of manufacturer claims inherent in

Figure 1. A person wears the NIA™ device. Our software infers the person’s stress level from it (also displayed graphically) and uses that to influence Roomba® behaviour. Copyright is held by the author/owner(s). HRI’09, March 11–13, 2009, La Jolla, California, USA. ACM 978-1-60558-404-1/09/03.

Page 2

the software display), we found the OCZ NIA™ quite limited, where we could only capture muscle tension data reliably. In spite of these shortcomings, we were able to use muscle tension input as commands to directly control robot speed, and for inferring human emotions such as stress to influence robot behaviour. 2.2 Direct control We mapped the OCZ NIA™’s muscle tension to robot speed. With minimal training and practice, we were able to increase and relax muscle tension as desired to control the robot’s speed in real time using the control software we developed. We did this by converting the continuous bioelectric signal input into 4 discrete values, where each value corresponds to a specific robot speed in the forward direction. Thus a person would consciously change their muscle tension to change robot speed, using methods such as jaw clenching, or eyebrow tensing as we did. While this mapping worked, its utility is limited. First, not enough input parameters are available to truly control the robot. Speed is just one aspect; other critical control factors include direction, rotation, operation (e.g., clean / don’t clean), and so on. Second, using muscle tension alone is not sufficient. For example, there is currently no way to turn off the link between robot and human action except manually, which limits the utility of the device. Except for extremely specialized circumstances (e.g., quadriplegic hands-free robot control), direct control does not yet seem particularly fruitful with such limited technology. 2.3 Behavioural control Our next strategy was to infer one’s emotional state from muscle tension reading, and then use that state to influence (rather than directly control) robot behaviour. Our control software reinterprets natural muscle tension as estimating one’s stress level: the more muscle tension, the more stress is inferred. We average muscle tension readings over 5 seconds and map these onto 4 stress levels, as displayed graphically on a GUI (Figure 1, top). Two distinct robotic behaviours corresponding to two extreme emotional states, either relaxed or stressed, are triggered when the stress reading reach a threshold. Robot actions are then influenced by these stress readings. When a person shows high stress (~levels 3 & 4), the robot enters its cleaning mode but moves away from the user so as not annoy them. When a person is relaxed (~level 1), the robot (if cleaning) approaches the person and then stops, simulating a pet sitting next to its owner. If the reading is in between these two levels, the robot continues operating in its current mode until the stress reading reaches a threshold. While a simple mapping, this strategy reveals several important properties. 1. A crude emotional level can be inferred from bio-electrical signals. While far from a brain interface, it roughly simulates the kinds of input we may be able to capture in the future. 2. Control of a robot is implicit, where the robot reacts to emotions rather than direct control. The person does not have to do anything (except wear the input device). 3. It is the robot’s behaviour – not low level actions – that is altered based on its perception of human emotion. 4. Appropriate robot behaviour can be provided as simple, and perhaps anthropomorphic, heuristics. 5. The heuristics can be robust in terms of input errors. That is, in this example none of the actions of the robot are ‘bad ones’. At its worst, if the robot incorrectly assumes a relaxed state when the person is in fact stressed, the consequences are small. 3. DISCUSSION While the idea of a bioelectric signal interface is typically marketed as a way to explicitly direct commands to devices, perhaps a better approach is to use this (often inaccurate input) as a way to mediate the robots behaviour, where it adjusts its behaviour to respond to people’s moods in socially appropriate ways. This opens the idea of a bio-electrical signal-based human robot interaction as emotional instruments. Our example illustrated how the Roomba® would work away from stressed people so as not to annoy them. Even if the Roomba® were wrong, this would not be considered bad behaviour. Another example could be robot companions – such as Sony AIBO robots – that approach people when they are perceived to be emotionally down. As far as we know we are amongst the very first to pursue bioelectric signal interfaces in human-robot interaction, where we program a robot to react to the user’s emotional state rather than just direct control. Of course, this is just work in progress. We will continue to explore the limits of current bioelectric signal and brain interaction technology, particularly when it comes to inferring user emotions. We also plan to explore using emotions drive other autonomous or ambient behaviour, such as the type of music being played by a computer, or room lighting levels. We believe the lessons, failures, and successes of this ongoing research will provide a road map for future efforts in using emotions to influence computational behaviour. 4. CONCLUSION Based on the results our research, we conclude that mapping emotional state of a user to the emotional state of a robot is not only possible with today’s technology, but that it opens up many exciting possibilities that have yet to be explored entirely. Acknowledgements. We thank James Young for his thoughtful input, and Cheng Guo for his help with the iRobot® Roomba® control software. REFERENCES 1. José Millán, Frédéric Renkensb, Josep Mouriñoc, and Wulfram Gerstnerb. Non-Invasive Brain-Actuated Control of a Mobile Robot by Human EEG. IEEE Trans. on Biomedical Engineering, Vol 51, June 2004. 2. OCZ Peripherals. Neural Impulse Actuator. http://www.ocztechnology.com/products/ocz_peripherals/nia-neural_impulse_actuator . Retrieved December, 2008. 3. TechRadar.com. OCZ Neural Impulse Actuator Review. Reviewed on July 3rd, (2008) and retrieved December, 2008. http://www.techradar.com/products/computing/peripherals/input-devices/other/ocz-neural-impulse-actuator--269721/review

Page 3

Interactions Lab, University of CalgaryPaul Saulnier, Ehud Sharlin, and Saul GreenbergUsing Bio-electrical Signals to Influence the SocialBehaviours of Domesticated RobotsWhat is a person’s stress level?We translate bio-electrical signals (read from person’s OCZ NIA™ headband) into a crude stress readingHow do we use stress to control a robot?Robot motion and actions depend on stress level readings.Why is this different?Robot behaviour reacts implicitly to emotions (stress) rather than direct explicit control. Screenshot of controller softwareNow I’m stressed!I’m happy!On the night before a deadline, the computer crashes!I like it when he’s relaxed. I’m going to sit next to him.Uh oh. I better get out of here and start cleaning!

Data provided are for informational purposes only. Although carefully collected, accuracy cannot be guaranteed. The impact factor represents a rough estimation of the journal's impact factor and does not reflect the actual current impact factor. Publisher conditions are provided by RoMEO. Differing provisions from the publisher's actual policy or licence agreement may be applicable.

[Show abstract][Hide abstract]ABSTRACT: This article shows the experiences carried out in the context of human/robot communication, on the basis of brain bio-electrical signals, with the application of the available technologies and interfaces which have facilitated the reading of the user's brain bio-electrical signals and their association to explicit commands that have allowed the control of biped and mobile robots through the adaptation of communication devices. Our work presents an engineering solution, with the application of technological bases, the development of a high-and low-level communication framework, the description of experiments and the discussion of the results achieved in field tests.

[Show abstract][Hide abstract]ABSTRACT: Emotional intelligence is the ability to process information about one’s own emotions and the emotions of others. It involves perceiving emotions, understanding emotions, managing emotions and using emotions in thought processes and in other activities. Emotion understanding is the cognitive activity of using emotions to infer why an agent is in an emotional state and which actions are associated with the emotional state. For humans, knowledge about emotions includes, in part, emotional experiences (episodic memory) and abstract knowledge about emotions (semantic memory). In accordance with the need for more sophisticated agents, the current research aims to increase the emotional intelligence of software agents by introducing and evaluating an emotion understanding framework for intelligent agents. The framework organizes the knowledge about emotions using episodic memory and semantic memory. Its episodic memory learns by storing specific details of emotional events experienced firsthand or observed. Its semantic memory is a lookup table of emotion-related facts combined with semantic graphs that learn through abstraction of additional relationships among emotions and actions from episodic memory. The framework is simulated in a multi-agent system in which agents attempt to elicit target emotions in other agents. They learn what events elicit emotions in other agents through interaction and observation. To evaluate the importance of different memory components, we run simulations with components “lesioned”. We show that our framework outperformed Q-learning, a standard method for machine learning.

[Show abstract][Hide abstract]ABSTRACT: The widespread adoption of personal service robots will likely depend on how well they interact with users. This chapter was motivated by a desire to facilitate the design of usable personal service robots. Toward that end, this chapter reviews the literature concerning people interacting with personal service robots. First, ongoing research related to the design of personal service robots is discussed. This material is organized around generic activities that would take place when a user initiates interaction with a future personal service robot, for example, understanding the robot’s affordances or its cognitive capabilities, as well as when a personal service robot initiates interaction with a user, for example, understanding the user’s intent or engaging and communicating with the user. Second, research areas that deserve more attention from the human-robot interaction community are discussed, for example, understanding when people do and do not treat robots as if they were people. Throughout the chapter, recommendations for the design of future personal service robots are offered along with recommendations for future research.