Empowering the therapists in intensive training programs. Emilia I. Barakova ... scenarios to the best of the technical capabilities of the robot,. i.e. through co-creation. ..... âExploring emotional and imitational android-based interactions in.

and shopping arcade located by the entrance to Universal. Studios Japan, a major theme park. We operated the robot within a 20 m section ..... of the map represents the side of the corridor, where people tend to walk slowly when the arcade is ...

behaviors produced by populations with ASD and with typical development (TD), ... designs that support the application of social robots to the specific domain of speech ...... blue line shows cumulative error versus number of trials in the tutorial .

1 Department of English, IBAIS University, Uttara, Dhaka, Bangladesh .... revolt against the artificial, bastard form of drama. He wanted to bring a .... It is a beautiful ..... shodhganga.inflibnet.ac.in/bitstream/10603/709/6/06_chapter%202.pdf.

Realistic Humanlike Robots for Treatment of ASD, Social Training, and ...

Training, and Research; Shown to Appeal to Youths with ... development of humanlike robot technologies as possible social training tools or ..... with the android.

Realistic Humanlike Robots for Treatment of ASD, Social Training, and Research; Shown to Appeal to Youths with ASD, Cause Physiological Arousal, and Increase Humanto-Human Social Engagement David Hanson, Ph.D.

[email protected] ABSTRACT This paper describes results from preliminary psychology experiments, which indicate that people (both children and adults) with autism or ASD accept realistic human-like robots, are not afraid of such robots, find such robots appealing and engaging, and may be more likely to increase social awareness as a result of interacting with such robots. These results support the further development of humanlike robot technologies as possible social training tools or treatment options for ASD, and as enabling new directions in research into the nature of ASD as a social disorder. The paper describes how the authors adapted ABA into robotdelivered scripts and protocols, with focus on labeling and joint attention tasks, and tested the results using observation of behavior, input with iPad interface, questionaires, and direct physiological measurement of biosignals including ECG and derived LF/HF ratios. Additional research is required to validate these preliminary results, to determine if such realistic humanlike robots are effective in long-term training and/or therapy. However, given the strength of the early results, and the potential value of such tools for treatment and research, such further research is warranted.

1.

Introduction

The research described in this paper explores the use of highlyrealistic and nearly-realistic robots with several groups with autism spectrum disorders (ASD), under the premise that such realism may be useful for high-fidelity training of social skills, and/or treatment using ABA techniques. The preliminary results appear promising and encourage further research. Most robots used in autism research are not very humanlike, but instead are intentionally far from realistically humanlike in appearance, often under the untested assumption that realism will be disconcerting to individuals with ASD [1]. Many studies over recent years, using non-realistic robots, find that children with autism/ASD respond positively to non-realistic humanoid robots, with noted increases in affect [1], language, and imitation learning [2], and that such robots promise new treatment options [3]. As demonstrated by recent studies, people with ASD perceive robots as artificial partners instead machines [4]. These observations inspired various researchers to develop robots aiming to engage social response in people with ASD (Robota, Infanoid, Keepon, etc. [5], [6], [7]. Most such systems have been

Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

1

proposed as innovative devices for analysis of Autism related behaviors but there are also example where has been used as tool for increase social capabilities of clients with ASD [8]. The research in this paper tested prototype autism treatments using realistically humanlike robots developed by Hanson Robotics and Hanson RoboKind, which display humanlike facial expressions using novel Frubber [9] [10] and expression mechanisms, gestural bodies and eyes [11], and interactivity with face tracking, adaptive expressions, and conversational A.I. [12] plus software for controlled therapy and experiments. The research explores the use of such realistically humanlike robots in service as a tool for practicing social skills, empathy training, language training, and experiments in behavioral therapy. We placed these robots into studies at the Autism Treatment Center in Dallas Texas, and at the University of Pisa and the Stella Maris Neurological Hospital in Tirrenia Italy, performing experiments wherein autism therapists and researchers studied encounters of RoboKind robots with both children and adults with ASD. The key premise is that realistic robots can provide a highly accurate simulation of a human-to-human social encounter, with special benefits of controlled repeatability, tireless repetition, and absence of therapist-frustration or other uncontrolled negative affect towards a patient or client. Relative to other, less-realistic robots or virtual simulations, realistic robots may provide a more accurate simulation of social interaction with a real human. If so, then interactions with an extremely human-like robot (versus lessrealistic robots) may be more effective for social training, transferring more effectively to real human-human interaction (the real goal of such treatments). This premise is supported by studies about the higher efficacy of high-realism simulations for training in other applications, such as flight simulation and military training [13]. It may also be that realistic, facially-expressive, humanlike robots may be better tools to transmit emotions to ASDs because they better simulate human-to-human interaction than non-human-like robots—a critical difference due to the fact that many people with autism struggle to interpret emotions during social interactions.Note however, that this effect needs to be validated with humanlike robots studies contrasting the efficacy of realistic and non-realistic robots in social training for subjects with ASD in addition to the work described herein. To start with, it is important to establish whether such robots can be appealing and engaging to individuals with ASD, which is the focus of the described research.

1.1

Hypotheses: 1. 2.

3.

4.

5.

Nearly-realistic humanlike robots can be not-frightening to individuals with autism spectrum disorders. Robots with nearly realistic humanlike appearance, movements, and interactions, can be appealing to individuals with ASD. Robots with realistic humanlike appearance, movements, and interactions can be engaging to individuals, holding attention throughout a session, resulting in a desire for additional interactions with such robots. A robot with realistic humanlike appearance, movements, and interactions, can arouse in YASDs physiological conditions which are positively correlated with learning and social engagement. Following interactions with robots with realistic humanlike appearance, movements, and interactions,

Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

6.

1.2

individuals with ASD display increased social engagement with people. Therapies using a robot endowed with the ability to adapt and change its demeanor or expressiveness according to the inferred emotional and mental states of a subject will train YASDs to gradually enhance and refine their social skills.

for training social skills in people with ASD, especially children at young, formative ages. Evaluate a new training program utilizing realistic robots for possible inclusion into the treatment of autism.

4. Improve the social and non-verbal communication skills 5.

2.

of adolescents and young adults with ASD, using a human-like expressive robot. Improve social competency firstly by learning social skills, and then gradually reinforcing emotional reciprocity and empathy, allowing YASDs to better integrate and interact with their peers.

Background

An estimated 1.5 million people in the USA have Autism Spectrum Disorder (ASD) (http://www.autismspeaks.org/whatisit/faq.php), impacting their ability to communicate, socialize, and interact with the world. These disorders have few treatment options, and in some cases are profoundly disabling. New, effective treatments are strongly needed. In July 1976, the Autism Treatment Center (ATC) was established in Dallas, Texas as a 501(c)(3) non-profit corporation designed to provide services to children and adults with autism and other related developmental disorders. Over the previous 34 years, ATC has continually met the challenge of an ever-increasing rate of autism and has expanded services and capacity to remain current with the needs of the community, as well as treatment and trends. The ATC strives to meet the tremendous need in the community by making promising new innovations available to the Dallas and surrounding communities in treating persons with ASD. The rate of children diagnosed with autism has increased dramatically. Texas has witnessed a four-fold increase in autism in the last decade. Currently, 1 in 110 children are affected with autism (Centers for Disease Control, 2009). In the 1980s, the rate was 1 in 1,000 [14]. A new case is diagnosed every twenty minutes and 67 children are diagnosed per day in the United States. This is not only a problem in the US, but a global pandemic affecting people all over the world. To explore ASD reactions to realistic humanlike robots, the University of Pisa contracted David Hanson and Hanson Robotics to develop the Mia Alice robot in 2009, and together began human-participant experiments with the robot at the University of Messina medical school in May 2009. Followup experiments began in Pisa in 2011. To explore addressing this need with smaller humanlike robots called RoboKind, the authors created a program titled “Testing 2

RoboKind in Experimental Autism Treatment” (TREAT). Under the TREAT program, the authors organized two feasibility studies using Hanson RoboKind expressive robots in autism treatment at the Autism Treatment Center at Dallas in February and June of 2011. The program targeted four main objectives: place robots into autism treatment centers for research, develop new curricula and protocol for such therapy, scientifically evaluate the efficacy of the robots in these uses, and raise and manage funds in pursuit of these goals.

3.

Material and Methods

Experiments were conducted with Alice in Pisa Italy, and with Zeno at the Autism Treatment Center in Dallas Texas. In this project we used two Hanson robots--one realisitic, human-sized, female robot with an expressive face named Alice, and one small, childlike, boy robot, with gestural walking body and an expressive face, named Zeno RoboKind. Hanson robots are built using proprietary Frubber material, mechatronics, and artistry combined systematically to realize humanlike facial expressions. Frubber is a porous silicone elastomer, comprising bio-inspired lipid bilayers in silicone achieving hierarchical porosity extending from macro-porosity to the 40 nm scale [15]. This technique results in an extremely soft, supple and strong silicone [16]. The physical characteristics of Frubber closely correlate with those of living facial tissues [16], and therefore achieve more humanlike facial expressions than other materials [16], and with orders of magnitude greater energy efficiency [16]. The biomimetic Frubber material results in lightweight, low-power facial robotics, a full range of expressions on biped robot bodies. The Frubber material is combined with mechanisms of facial actuation, derived from human anatomy and physiology, and actuated using servomotors controlled by a PC by means of usb-to-serial-to-PWM signal conversions achieved with Atmega microcontrollers [17]. The Pisa experiments utilize the generation-1 Alice robot, a human-size adult female robot with highly realistic facial form and expressions (see figure 1). She contains 30 DOF in the face and neck, consisting of electric gearhead servomotors with potentiometers for position feedback. The motors end in linkages that connect with anchors embedded in the Frubber and strategically designed to simulate the stress distribution on natural facial muscles on the human skin surface. Because the motors are bidirectional (versus human muscle which can only pull) each servomotor can simulate 2 muscle groups. A total of 14 muscles are simulated in the neck, and 48 muscles are simulated in the face--the natural number of human facial muscles. Each motor has 512 discrete possible positions, which results in 1.93 x 10 756 possible configurations or expressions that the robot can affect-effectively the full-range of human facial gestures. In the gen-1 Alice robot, only the face and neck move; there are no body motions.

Figure 1. Alice robot by Hanson Robotics. The Dallas experiments utilized the Zeno RoboKind 2010 model (see figure 2), a child-sized boy robot, with somewhat less realistic form and expressions than Alice, but more realistic than other robots used for autism research (such as iCub, Kaspar, Bandit, or Nao). Zeno has a total of 37 DOF, 10 of which are in the face and neck, and 27 of which are used in the gestural body, with walking capabilities and grasping/gestural hands and arms. The Zeno robot wirelessly connects to a remote computer for operation, but has an on-board PC computer for autonomous functionality in the event of loss of network connection. Figure 2. Zeno by Hanson RoboKind. Both robots display naturalistic expressions including happy, sad, angry, disgust, fear, surprise, and neutral--the Ekman seven transcultural expressions [18]. The robots have separate DOF corresponding to human facial muscles including the zygomaticus major, depressor labii, mentalis, nasalis, orbicularis oris, orbicularis occuli, bucinator, frontalis, proceris, and palbebrus muscles. This allows very fine control of expressions, and enabling billions of possible microexpression configurations. The robot sensors include cameras embedded in the eyes, microphone,s potentiometers, and load sensors. The goal with Hanson robots is to achieve human-level behaviors and interactions in the robots, while removing non-humanlike artifacts that may distract from social engagement. This goal is partially completed, and the robots used represent incremental advances in the development of technique in this quest.

Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

3

3.1

Pisa’s protocol

Pisa’s test panel was composed of 10 subjects: 7 YASDs with ages in the range 15-22 years and 3 controls with ages in the range 15-17 years. The intervention consisted of a therapist driven interaction between FACE and subjects of about 30 minutes divided in five main phases. Each phase of the protocol was tailored in order to gradually increase subject-robot interaction. A baseline acquisition of about 2 minutes was first recorded in the absence of FACE (phase A). The youth then entered in the room where the android was placed (see Figure 3); this first phase (Phase B) was oriented to famialiarize with the therapy room and with the android, in this phase the therapist investigate also spontaneous interaction with FACE and the comfort of the youth was evaluated. In the third phase (phase C), FACE starts to express the six basic emotions along with a neutral face and the youth was asked to label and imitate these expressions [19], [20].

Figure 3. Alice robot (the middle figure) interacting with a NT child and a nurse. Courtesy of “L'androide che si emoziona”, Abitare 49b-12/2009, Italy. In the fourth phase (phase C) the youths are involved in a conversation with the therapist that questioning them on how the interaction with the android was and how they was feeling during the facial interpretation task. During the sessions an observation schedule was filled for each participant. Pisa’s team r attention was focused on the analysis of heart rate and his variability derived from ECG signals extracting for each phase the mean Heart Rate value and the LF/HF ratio [21] .

3.2

Dallas Autism Treatment Center’s protocol

In the pilot program of the TREAT studies, the Zeno RoboKind robot was used with children and adults with ASD in two sessions. The protocol for each session was based on ABA principles, targeting both cognitive training and practice of social skills. The first session involved 3 children, mid-functioning YASDs, ages 7 to 17 (two boys and one girl), and 2 adults, one woman aged 23, and one male, age 21. The protocol involved an encounter of the client with the Zeno RoboKind robot, mediated by a therapist. One additional therapist-observer was present in the room, taking notes, and the robot operator was present in the

Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

room at a computer terminal. The robot was introduced to the participant by the therapist, and the robot spoke the child’s name in a greeting. Then the therapist directed the interaction, which was mostly scripted, but for part of each session did involve a phase of open interaction with Zeno, fully automated using Character Engine NLP chatbot technology. Otherwise the interaction was a robot-mediated spoken conversation (telerobotics). Several statements and questions were asked by the therapist, and several questions were asked by the robot. In the second session, Zeno interacted with 2 mid-functioning YASDs, ages 9 and 10, in a protocol involving a touch screen and an ABA-based receptive labeling task. The participants were introduced to the robot, and the robot responded by greeting the children by name. The therapist told the children they were to play a game with Zeno using an iPad touch screen interface. When the participants agreed, then they were asked to touch the image on the screen that Zeno requested. Two images were simultaneously displayed on the screen, one correctly corresponding to that which the robot mentioned, in addition to one distractor. A total of 18 trials per subject were conducted during this session. Images were displayed in the following order (parentheses indicate the correct response for that trial): 1. ball/block (ball) 2. ball/block (block) 3. apple/orange (apple) 4. apple/orange (orange) 5. house/tent (house) 6. house/tent (tent) 7. hammer/screwdriver (hammer) 8. hammer/screwdriver (screwdriver) 9. Spiderman/Mickey House (Spiderman) 10. Spiderman/Mickey Mouse (Mickey Mouse) 11. cat/dog (cat) 12. cat/dog (dog) 13. monkey/elephant (monkey) 14. monkey/elephant (elephant) 15. car/truck (car) 16. car/truck (truck) 17. robot/person (robot) 18. robot/person (person). While each set of images was displayed, Zeno would say, "Touch the (image)." If the participant selected the correct response, Zeno would deliver praise ("Good job!") and the iPad would display the next set of images, whereas Zeno would have responded "Oops-try again." in the event that an incorrect response was selected (in this case, the participants responded with 100% accuracy).

4.

Results

Overall, the results show that the subjects responded favorably and enthusiastically in response to the robots, in all three studies.

4.1

Results from Pisa studies

Preliminary data and general discussion on the robot acceptability demonstrated the following [23]: ■ no fear, 20 minutes alone with therapist and the robot, usually children don’t want to come out ■ Activation differences between baseline and robot interaction phases means that children are “triggered” by the human robot interaction task ■ Areas of variability in the results among individuals is interesting. Possibly the robots will be more useful for some individuals than for others, or different protocols and scripts may be needed among individuals. These questions require additional research investigation. The robot interaction was well accepted by the youths and 4 of 7 children stated that they did not want to leave the session with the robot, asking to stay longer and play with it. The heart rate variability (HRV) frequency-domain parameters revealed differences in VLF (very low frequency, calculated for the entire session) and LF (low frequency, calculated for each phase) with an increase of these values in ASD participants

4

compared with controls while no significant differences were found in HF (high frequency). In particular, the VLF and LF parameters are known to be related to the sympathetic nervous system while HF has parasympathetic origin and the ratio LF/HF consequently indicates the balance between sympathetic and parasympathetic systems [22], [23]. The analysis of HR timedomain parameters showed a decrease of the mean HR values in YASDs with respect to control youths during the entire session, showing that YASDs were not disturbed by the presence of the android. However it was interesting to notice that the highest HF value, i.e. the highest parasympathetic activity, corresponded to the phase (D) in which the participant establishes a conversation with the android. This indicates that social interaction tasks seem to activate the parasympathetic system [24]. An higher LF trend in participants with ASD was also confirmed by the LF/HF ratio which was higher in participants with ASD and it is consistent with several studies reporting an increased cardiac sympathetic activity in ASD individuals [25] [26]. Although the number of samples was limited, these experiments suggest that the platform can be used as innovative tool for studies on participants with ASD and that the intervention protocol is able to involve the active participation of ASDs, despite their well-known difficulties in attention focusing and in performing relationship oriented tasks.

different versions of humanlike robot protocols would be better suited for different individuals or sub-populations.

4.2

Results from the Dallas Autism Treatment Center ■ ■ ■

No fear shown in 9 of 10 subjects. Minor fear or discomfort exhibited by one subject. Tremendous affect and multiple dialogue turns, great curiosity about the robots, Desire for additional interactions with the robot.

In the first session, all children and adults with ASD responded with interest to the robot, answering the robot’s questions and stating that they liked the robot when asked. All subjects but one stated that they would like to interact with the Zeno again. One participant asked Zeno to come home, and another asked Zeno to go to a party. All but one subject interacted with Zeno spontaneously through multiple dialogue turns. All subjects showed atypically high facial affect, and verbalization. Four subjects asked over following days when Zeno was going to return. Four subjects spontaneously tried to make physical contact with Zeno, usually by touching Zeno’s hand or face. During the second session, a number of observations were made including that the children showed great interest in the robot, looking directly at the robot throughout the session, verbalizing in repeated dialogue turns with the robot, and one of the children made a positive comment about another child who was not even in the room (“Jeffrey would like this!”), which was incredible due to the fact that the client in question never exhibits empathy in therapy sessions, which is typical as many children with autism do not think about other's perspectives especially when they are absent. The other observation was that the children appeared to want to please Zeno and had rapt attention during the trials. Most of the children stated that they liked Zeno and wanted him to come back, and several repeatedly asked in the weeks and months following the session: when would Zeno be back at the Center.

5. Table 1: LF/HF percentage of variation between the familiarization and the questioning phase, ASD5* was not interested in the robot and didn’t show interest also in describe to the therapist is feeling. Control2** was very young (6 years) and was not comfortable with the therapy room. [20], [23]. In table 1 is reported the percentage variation of the LF/HF parameter between the familiarization and the questioning phase. These preliminary results shown that most of the participants with ASD had a higher sympathetic activity during the familiarization phase than during the robot facial expression interpretation phase. This difference can be correlated with a stress condition due to the first approach with the FACE robot highlighted by the sympathetic activity. This stress is reduced and resolved after the familiarization phase and there is an increase of the parasympathetic activity due to the expression interpretation analysis task [20], [23]. The range of physiological reactions is unexpected and warrants closer attention and discussion. The differing reactions may indicate that treatment using humanlike robots may be more effective on sub-populations or on particular individuals, or that

Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

Interpretation ■

Interaction with both human-scale and small-scale realistic humanlike robots resulted in positive results

■

Cross-cultural positive results, both in Italy and in Texas. Similarities included:

● ● ● ●

■

lack of fear interest desire to continue interaction display of unusually high levels of affect, interaction, shared attention, eye contact, and concern for feelings of the robot and others during the interactions.

Support or Refutation of Hypotheses

○

Hypothesis 1 was confirmed. Nearly-realistic humanlike robots were found to be generally not-frightening to individuals with autism spectrum disorders.

5

○

○

Hypothesis 3 was confirmed. Robots with realistic humanlike appearance, movements, and interactions, can be engaging to individuals, holding attention throughout a session, and resulting in a desire for additional interactions with such robots.

○

Hypothesis 4 showed mixed results. In several participant YASDs, a robot with realistic humanlike appearance, movements, and interactions, aroused physiological conditions which are positively correlated with learning and social engagement. However, in others, encounters with the highly realistic robot lowered bio-signs associated with arousal.

○

Hypothesis 5 was confirmed. Following interactions with robots with realistic humanlike appearance, movements, and interactions, participants with ASD displayed increased social engagement with people. The behaviors exhibited included increased affect, eye contact, verbalization, and several instances of theory of mind by showing concern for the feeling of others.

○

6.

Hypothesis 2 was confirmed. Robots with nearly realistic humanlike appearance, movements, and interactions, can be appealing to individuals with ASD.

Hypothesis 5 was inconclusive. Not enough data was gathered to interpret whether therapies using a robot endowed with the ability to adapt and change its demeanor or expressiveness according to the inferred emotional and mental states of a subject will train YASDs to gradually enhance and refine their social skills.

social encounters over multiple sessions. The complexity of interaction may also be lowered or increased to custom-suit the needs of an individual, and then adjusted upwards over time. The data and behavioral observations showed that realistic and nearly realistic humanlike robots can well accepted by individuals with autism spectrum disorders at the mid-to-high end of the spectrum (no low-end-spectrum participants were included in the studies). Moreover the results show that such robots result in physiological changes associated with learning, and generally result in a desire for continued and repeated interactions. The variability in the physiological and behavioral reactions indicates that the mechanisms of response are not well understood and may vary widely among individuals and subpopulations of the spectrum. The positive results indicate that realistic humanlike robots may become a valuable new tool to transmit to this people emotions and for social training, including learning how to interpret emotions and people moods, and for practicing social scenarios. Additional studies are warranted.

Future Work Further experiments can help to validate the results with larger populations of human participants. To find cultural variations or transcultural universalities, it would be useful to replicate the experiments with other cultures. It would be useful to vary the ABA protocols, to explore the uses as a treatment tool. We would also like to test across a wider ASD spectrum, to investigate parameters for tuning different treatments to differences in subpopulations or individuals. We would like to explore the effects of changing and expanding the technology and aesthetics. Using addional AI: computer vision, speech affect recognition, VSLAM. We intend to explore the effects of other faces on the robots: comparing reactions to male, female, more or less cartoonlike faces, aesthetic variations of faces, and faces at different scales. Furthermore, we seek to explore the effects of inclusion of walking body, gestures, and other non-face related anthropomorphic stimuli. We also seek to develop additional treatment protocols, exploring empathy training, face reading, shared attention, and collaborative tasks.

Conclusions 7.

Expected Results We expect that social emotional training through the adaptive training platform can help YASDs to develop appropriate mechanisms to infer the robot’s “intentions” or emotional state as a step towards more appropriate interactions with social peers. Increased scores in social competence measures are therefore expected, particularly in recognition and labeling of emotions and theory of mind. In addition, by recording and processing physiological signals during assessment, training and at rest, we expect to observe autonomic state changes and identify those which may make YASDs more receptive to external stimuli [24] and more susceptible to social interactions [23]. YASDs appeared to respond to the robots more favorably, with more emotional reaction, than to humans around the robots, and this may be because robots are more predictable and mechanistic, and so less confusing or overwhelming than human stimuli. With such a robot, the complexity of the human-like interaction can be increased over time, for incremental training of more naturalistic Realistic Humanlike Robots for Treatment of Autism, PETRA 2012

References

[1] B. Robins, K. Dautenhahn, R. teBoekhorst, and A. Billard, “Robotic Assistants in Therapy and Education of Children with Autism: Can a Small Humanoid Robot Help Encourage Social Interaction Skills?” Special issue "Design for a more inclusive world" of the international journal Universal Access in the Information Society (UAIS), Springer-Verlag, Vol. 4(2), (2005), pp. 105- 120. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10. 1.1.66.9326&rep=rep1&type=pdf [2] M. A. Goodrich, M. Colton, B. Brinton, and M. Fujiki. A Case for Low-Dose Robotics in Autism Therapy. To appear in Proceedings of the ACM/IEEE RAS International Conference on Human-Robot Interaction . 2011.http://portal.acm.org/citation.cfm?id=1957702 [3] E. Kim, E. Newland, R. Paul & B. Scassellati. Robotic tools for prosodic training for children with ASD: A 6