Transcription

1 Free-Hand Interaction with Leap Motion Controller for Stroke Rehabilitation Maryam Khademi 1 University of California, Donald Bren School of Information and Computer Science, Irvine, USA Hossein Mousavi Hondori 1 University of California, Department of Neurology, School of Medicine, Irvine, USA Alison McKenzie Chapman University Department of Physical Therapy, Crean School of Health and Life Sciences, Orange, USA Lucy Dodakian University of California, Department of Neurology, School of Medicine, Irvine, USA Cristina V. Lopes University of California, Donald Bren School of Information and Computer Science, Irvine, USA Steven C. Cramer University of California, Department of Neurology, School of Medicine, Irvine, Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI 2014, Apr 26 - May 01, 2014, Toronto, ON, Canada ACM /14/04. Abstract 1 In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller s hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our freehand Fruit Ninja s score is a good indicator of the patient s hand function and therefore will be informative if used in their rehabilitation. Author Keywords Stroke Rehabilitation; Natural Free-Hand Interaction; Leap Motion Controller; Hand Tracking; ACM Classification Keywords H.5.2 [Information Systems]: User Interfaces; K.8.0 [General]: Games; J.3 [Life and Medical Sciences]. 1 The first two authors contributed equally to this paper.

2 Figure 1. Playing Fruit Ninja using Leap Motion controller Introduction In Human Computer Interaction, free-hand interaction has been explored for various applications such as games [1], immersive 3D modeling [2], mobile augmented reality applications [3], air painting [4], writing and sketching [5], etc. While using free-hand interaction systems for an ablebodied user may be considered a luxury, it might serve as a great assistive technology for patients with physical disability. One of the patient communities who may benefit largely from hand interaction are individuals with stroke. Stroke causes hemiplegia which affects motor functions in one side of the body; typically, patients lose part or full control of one of their hands. Considering that stroke is a leading cause of serious long-term disability in adults with more than 795,000 people in the United States each year [6], [7], helping this community to retain their hand motor control, could have a significant impact on their quality of life. Background and Related Work The human-computer interaction community has made several attempts to enhance stroke rehabilitation with HCI technologies. Hallam et al. [8] proposed an interactive glove to foster acceptance of partners after one of them is affected by stroke and help them reunite. Boulanger et al. [9] used Microsoft Surface s hand position as input of a tabletop game environment for stroke rehabilitation. The study involved exercises such as curling and uncurling fingers as well as a wrist flexion and extension of the hand back and forth about the wrist. Another example of fine motor rehab systems is PointAssist [10] which is a mouse interface that adapts to a user's level of impairment in reaching and clicking. The system could improve click success rates of individuals with fine motor impairments. Also [11] evaluated a novel click assistance technique to help users with motor impairments to click more accurately using a mouse. More specific studies focusing on finger rehabilitation include but are not limited to HandTutor [12], a glove-based treatment system which provides intensive flexion and extension movement of finger(s) and the wrist. Also MusicGlove by Reinkensmeyer et al. [13] motivates finger movements by pressing fingers against each other to play GuitarHero. The above research shows example of touch, mouse, glove-based technologies that have been applied to stroke rehabilitation. Yet, using free-hand interaction as the most intuitive option to stroke rehabilitation has to be investigated. A Gaming System for Finger Individuation We combined the game of Fruit Ninja with the Leap sensor, using open source JavaScript version of Fruit Ninja [14]. The desktop version of the game is supposed to be played with mouse events such as click and drag. Since our focus is finger individuation, we modified the game so that mouse events can be replaced by hand movements. Using the hand tracking information of the Leap sensor, we were able to trigger equivalent mouse events. We have also previously used Fruit Ninja in other stroke rehabilitation settings [15]. The setup includes a PC/notebook and a Leap sensor that is plugged in via its USB (Fig. 1). The game runs on a browser (e.g. Google Chrome) while the hand is monitored using our java API code which is written on top of the Leap sensor's SDK.

3 Figure 2. A snapshot from Fruit Ninja game Figure 3. Strapping a stick to the dorsum of the hand Study To determine the feasibility of free-hand interaction using Leap Motion controller for stroke rehabilitation exercises, we conducted a pilot study with 14 patients with stroke. The participants were asked to play our free-hand Fruit Ninja game. We also simplified the original Fruit Ninja so that patients with stroke with different levels of hand deficits are able to play the game for one minute. Game To avoid turning the rehabilitation exercises into boring and repetitive tasks, exercises are often gamified. For this purpose, we used Fruit Ninja, a top-ranked game in itunes and Google Play [16]. Fruit Ninja is not only an engaging game but also requires goal directed hand movements, which makes it appropriate for hand rehabilitation. In the original version, as the game starts, different kinds of fruits appear on the screen while the player has to slice them as fast as possible with mouse click and drag events (in the desktop version) versus touch and drag events (in the mobile version) (See Fig. 2). If the player misses slicing three fruits, the game is over. Also, in the course of the game, several bombs appear on the screen time to time that the player has to avoid; otherwise the game is over. We modified the Fruit Ninja game so that the patients with stroke were able to play. We changed the gameover restriction by removing the three missing fruits condition and disabling the bombs; instead we introduced a time constraint of 1 minute. This gave all the patients a similar game condition which was required for conducting a fair study and comparing their results. Participants 14 patients with stroke (7 males and 7 females), aged (M = 58.1, SD = 11) participated in the study. They were at least 6 months post stroke with different level of disability ranging from Fugl-Meyer [17] score of (M = 48.86, SD = 17.79) and Box and Blocks Test [18] score of 0-58 (M = 30.29, SD = 20.27). Experimental Procedure Four participants had difficulty with finger movements; we strapped a stick (one of a pair of chop sticks) to the dorsum of their hand (Fig. 3); this was to enable them to play the game using their wrist movements (e.g., wrist supination and pronation). Since our experiment was part of a bigger study for which these four patients passed the inclusion criteria, we also accepted them to try the Fruit Ninja game. However, we took note of their special condition and did not compare their results with those of the rest of the patients. After providing informed consent, the patients were briefed on how to play the game. They also had a warm-up round to get accustomed to the game. All the patients were asked to use their stroke-affected hand. They had three, oneminute rounds of playing the game while they were given break time in-between to rest. At the end, we asked some Likert questions on a scale of 1-4 to learn about the patients' overall satisfaction. These questions included: (i) whether the patient found the game engaging on a scale of 1 (not engaging) to 4 (very engaging); (ii) whether the game addressed a need (yes/no); and (iii) whether the patient would use the game at home on a scale of 1 (wouldn t use) to 4 (would use).

4 FM BBT Fruit Ninja Fruit Ninja Figure 4. Performance in Fruit Ninja versus FM and BBT clinical tests Results Fig. 4 shows the game scores of our participants against their FM and BBT scores. To make an accurate comparison between patients' scores, we removed the score of four out of 14 patients who used the stick instead of their fingers. As shown in Fig. 4, a similar trend is observed between the patients game (FN) scores versus clinical scores. The clinical score of FM and BBT are highly correlated (r = , p<0.01). We observed, the FN and FM scores are significantly correlated (r=0.72, p<0.05), as were the FN and BBT scores (r=0.86, p<0.01). Note that BBT's tasks require finer hand and fingers movements while FM tasks focus more on gross arm movements. Perhaps because the Fruit Ninja game involves fine hand/fingers control, FN- BBT scores show stronger correlation than FN-FM scores. In response to whether the game was engaging for the patients, 11 out of 14 subjects gave 4/4 (very engaging). Also 10 of them agreed that the game was addressing a need, i.e., the game provided practicing the movements that can be generalized to daily functions. In response to the question whether the patients would be willing to play this game if provided at home, 12 of them picked 4/4 (would use) while the other two participants gave 1/4 (wouldn t use). Two of the patients (P6-male and P1-female) who enjoyed playing the Fruit Ninja commented that the game provided hand-eye coordination which is very important in activities of daily living (ADLs). Also, three participants (P8-female, P13-male, and P7-male) mentioned that the game was not very responsive in terms of tracking their hand. Discussion Following up the tracking issue mentioned in previous section, we revisited the data of those particular patients (P8-female, P13-male, and P7-male) and found out that they had difficulty individuating their index finger to make a pointing gesture. Instead, they played the game with all the fingers extended. The Leap sensor assigns temporary ID s (1-5) to different fingers, for example, moving from frame n to n+1, the index finger may or may not be assigned to ID = 2. This creates a subtle confusion while showing the trace of slice on the screen (the trace may appear with jittering). Note that this event may happen in few percents of the frames (around 1%). To solve this problem, we propose a 3D model of the hand that includes specific ID s for each finger and contains all skeletal parameters of the hand. Given that Leap s data is limited to fingertip position and direction vector, palm center position, normal, and direction vector, we used inverse kinematics to extract finger joint angles. The human hand has 27 Degrees of Freedom (DoF) [19] where each joint contributes to 1-3 DoF. The wrist has 3 Cartesian DoF (x, y, and z) and 3 rotational DoF (roll, pitch, and yaw). Finger joints include the metacarpophalangeal or MP, proximal interphalangeal or PIP, and distal interphalangeal or DIP joints. The thumb has 5 DoF including 1 at the DIP, 2 at the PIP, and 2 at the MP. Each other finger (i.e., index, middle, ring, and fifth) has 4 DoF, including 1 at the DIP, 1 at the PIP, and 2 at the MP. Of these 27 DoF, Leap SDK directly provides only the 3 from wrist rotations but it provides enough data to

5 z (mm) y (mm) 0 0 x (mm) - Figure 5. Fingertip position and directions provided by Leap sensor Figure 6. Full hand skeleton made by applying inverse kinematics on hand's anatomical model and Leap's data compute the rest. We used inverse kinematics to calculate the MP, PIP, and DIP angles by going through fingertip position and direction vector of each finger to the palm's center. Fig. 6 shows the completed hand skeleton based on the Leap s data shown in Fig. 5. The hand skeletal model not only increases the robustness, but also enables us to record and reconstruct hand movements in a realistic way for medical assessments. Besides, the model can be used for simulating patients hand movements in a virtual environment (i.e., games or computerized therapeutic exercises) where patients manipulate objects. Conclusion The HCI advents have not been fully explored in the field of patient-computer interaction. In this paper, we used the cutting-edge technology of free-hand interaction with Leap Motion controller for stroke rehabilitation. We modified the Fruit Ninja game to use the Leap sensor s hand tracking data. The combination was prepared for patients with stroke to practice their fine motor control. We conducted a pilot study with 14 patients with stroke to evaluate feasibility of using this system for rehabilitation of upper extremity. The results demonstrated significant correlations between scores generated from the Fruit Ninja game and standard clinical outcome measures, such as the Fugl-Meyer Arm assessment and Box-and-Blocks Test. The qualitative evaluation of the system was also proved successful. To make the system more accurate and responsive, we also proposed a kinematic model of the hand. Using this model allows us to incorporate 3D parameters of the hand in tracking which in turn makes tracking more robust. Acknowledgement We thank our volunteers in the study. This research is funded by the American Heart Association. References [1] D. Kim, O. Hilliges, S. Izadi, A. D. Butler, J. Chen, I. Oikonomidis, and P. Olivier, Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor, in Proceedings of the 25th annual ACM symposium on User interface software and technology, New York, NY, USA, 2012, pp [2] H. Kim, G. Albuquerque, S. Havemann, and D. W. Fellner, Tangible 3D: hand gesture interaction for immersive 3D modeling, in Proceedings of the 11th Eurographics conference on Virtual Environments, Aire-la-Ville, Switzerland, Switzerland, 2005, pp [3] D. Datcu and S. Lukosch, Free-hands interaction in augmented reality, in Proceedings of the 1st symposium on Spatial user interaction, New York, NY, USA, 2013, pp [4] J. Sutton, Air painting with Corel Painter Freestyle and the leap motion controller: a revolutionary new way to paint! in ACM SIGGRAPH 2013 Studio Talks, New York, NY, USA, 2013, pp. 21:1 21:1. [5] S. Vikram, L. Li, and S. Russell, Writing and sketching in the air, recognizing and controlling on the fly, in CHI 13 Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2013, pp [6] S. C. Cramer et al., Harnessing neuroplasticity for clinical applications, Brain J. Neurol., vol. 134, no. 6, pp , Jun

playful therapy for finger, hand and arm rehabilitation Intensive and motivating training For hospitals, clinics and patients at home Award-winning Swiss therapy system swiss technology Our patients see

Fine Motor Development Manipulation Once a object is grasped one either manipulate the object or release it. If the object is to be manipulated stabilization of the fingertip forces must be sustained for

Text Entry with the Apple iphone and the Nintendo Wii ABSTRACT We present an empirical study with 16 participants comparing text entry speeds and error rates on the Apple iphone and the Nintendo Wii, both

To Virtualize or Not? The Importance of Physical and Virtual Components in Augmented Reality Board Games Jessica Ip and Jeremy Cooperstock, Centre for Intelligent Machines, McGill University, Montreal,

UOM Robotics and Industrial Automation Laboratory Department of Industrial and Manufacturing Engineering, RIAL Mission o to be a Centre of Excellence in Malta for research, development, teaching and outreach

REHABILITATION CONCEPT FOR THE LOWER LIMBS Life by motion LAMBDA CONCEPT The Lambda is the only rehabilitation device that offers the freedom of a plan workspace in which the leg can be manipulated throughout

Next Generation Natural User Interface with Kinect Ben Lower Developer Community Manager Microsoft Corporation Key Takeaways Kinect has evolved: Whether you did it -> How you did it One or two people ->

GRASP Graded Repetitive Arm Supplementary Program Exercise manual Level 1 This research project is funded by UBC and the Heart and Stroke Foundation 1 Welcome to the Grasp study! Thank you for volunteering

Computer Games and Virtual Worlds: New Modalities of Rehabilitation and Therapy Walt Scacchi Institute for Software Research and Center for Computer Games and Virtual Worlds University of California, Irvine

Patient and Family Education Brachial Plexus Palsy A Therapy Guide for Your Baby Brachial Plexus Palsy is caused by trauma to the nerves of the arm. This trauma may have caused muscle weakness, lack of

June 15, 2010 A New Vision of Rehabilitation Recovering cognitive abilities with Dynavision Occupational therapists are constantly striving to find the evidence-based interventions that will help clients

MEDICAL POLICY SUBJECT: COGNITIVE REHABILITATION PAGE: 1 OF: 5 If the member's subscriber contract excludes coverage for a specific service it is not covered under that contract. In such cases, medical

The ergonomics of tablet use in business applications An Ergonomic Solutions white paper by Melanie Swann MSc(Hons), Grad IOSH For more information and contact details, please go to ergonomic-solutions.net

INFLUENCE OF VIRTUAL REALITY ON ACTIVITIES OF DAILY LIVING IN HEMIPARETIC CEREBRAL PALSY Thesis Submitted in Partial Fulfillment for the Requirement for Master Degree of Physical Therapy BY HANAA MOHSEN

1 Rivermead Motor Assessment General instructions: Go through the items in order of difficulty. '1' if patient can perform activity, ' ' if he cannot. Three tries are allowed for each item. You may stop

General Principles: This protocol was designed to exercise the major muscles necessary for throwing. All exercises included are specific to the throwing athlete and are designed to improve strength, power,

, March 12-14, 2014, Hong Kong Work Performance Comparison of Holding and Using Tablet Naphat Yongbunthanaphat and Phairoat Ladavichitkul Abstract Tablet using is increasingly utilized in various working

Scooter, 3 wheeled cobot North Western University A cobot is a robot for direct physical interaction with a human operator, within a shared workspace PERCRO Exoskeleton Unicycle cobot the simplest possible

Robot Task-Level Programming Language and Simulation M. Samaka Abstract This paper presents the development of a software application for Off-line robot task programming and simulation. Such application

Self-Range of Motion Exercises for Shoulders, Arms, Wrists, Fingers These exercises will help keep your muscles strong and mobile, and your joints flexible. Other benefits of these exercises include: C

Basic Anatomy The arm and hand are divided into volar or palmar, and also dorsal, aspects. Distal to the elbow, structures are termed radial or ulnar to the middle finger axis rather than lateral and medial,

Movements of the Upper Limb - Page 1 of 18 Movements of the Upper Limb - Introduction This module presents the nomenclature of movement at the joints of the upper limb. When you first approach the upper

The Computer Mouse and Related Input Devices Shumin Zhai IBM Almaden Research Center Human input to computer systems is a critical and integral part of any human-computer interaction system. Input should

THROWER S TEN EXERCISE PROGRAM Diagonal Pattern D2 Extension Involved hand will grip tubing handle overhead and out to the side. Pull tubing down and across your body to the opposite side of leg. During

To ensure the functioning of the site, we use cookies. We share information about your activities on the site with our partners and Google partners: social networks and companies engaged in advertising and web analytics. For more information, see the Privacy Policy and Google Privacy &amp Terms.
Your consent to our cookies if you continue to use this website.