Transcription

1 Medical Imaging Specialists and 3D: A Domain Perspective on Mobile 3D Interactions Teddy Seyed Frank Maurer Francisco Marinho Rodrigues Anthony Tang Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s). CHI 2014, April 26 May 1, 2014, Toronto, Ontario, Canada. ACM /14/04. Figure 1. A medical imaging specialist exploring a 3D volumetric image of a human chest using a tablet. Abstract 3D volumetric medical images, such as MRIs, are commonly explored and interacted with by medical imaging experts using systems that require keyboard and mouse-based techniques. These techniques have presented challenges for medical imaging specialists: 3D spatial navigation is difficult, in addition to the detailed selection and analysis of 3D medical images being difficult due to depth perception and occlusion issues. In this work, we explore a potential solution to these challenges by using tangible interaction techniques with a mobile device to simplify 3D

2 interactions for medical imaging specialists. We discuss preliminary observations from our design sessions with medical imaging specialists and argue that tangible 3D interactions using mobile devices are viable solution for the medical imaging domain, as well as highlight that domain plays an important role in 3D interaction techniques. Author Keywords Mobile devices, medical imaging, 3D navigation, volumetric imaging. ACM Classification Keywords H.5. 2 [Information interfaces and presentation]: User Interfaces Input Devices and Strategies; Interaction Styles. Introduction Magnetic Resonance imaging (MRI) and computed tomography (CT) are standard imaging techniques used by medical imaging specialists when conducting medical investigations that lead to important decisions about the healthcare of a patient (e.g. choice of treatments or surgery options). These imaging techniques produce volumetric slices of scanned tissue (i.e. multiple 2D cross-sections) and are examined in abstract 2D by medical imaging specialists [1]. These 2D crosssections are considered individually, before medical imaging specialists mentally reconstruct them to fit a scanned organ or other anatomical structure [2]. Historically, 2D visualization approaches utilizing keyboard and mouse based interaction techniques have been employed in the medical imaging domain for 3D volumetric medical images instead of 3D interaction techniques. This is because medical imaging specialists consider 3D interaction techniques with keyboard and mouse based interfaces to be less practical than 2D, due to issues such as 3D images being hard to interpret on flat, 2D screens [2]. As a result, medical imaging specialists still face numerous challenges that are directly related to the interaction of 3D volumetric imaging data in a 2D manner. For example, an essential task for a medical imaging specialist is a detailed analysis of specific parts of scanned tissue, looking for irregularities that span multiple 2D crosssections. This requires an efficient means of navigation and selection in 3D, but due to issues such as the lack of depth perception and occlusion, traditional keyboard and mouse interaction techniques makes this difficult. The goal of our work is to improve interaction with 3D volumetric medical images for medical imaging specialists. Our approach is to leverage tangible objects as an interaction mechanism, taking advantage of existing spatial and physical reasoning skills [3]. As shown in Figure 1, we apply these concepts to the medical imaging domain, where a free-moving digital tablet displays reconstituted 2D slices of the volumetric 3D dataset (i.e. these slices are composed of multiple CT or MRI scan slices). In this paper, we discuss our explorations of tangible 3D interactions with mobile devices in the medical imaging domain and present our preliminary results from design sessions with medical imaging specialists. Our explorations present new requirements 3D interaction techniques with mobile devices for 3D volumetric medical images. Related Work Medical imaging specialists present a challenge in interaction design because of the variation in how 3D volumetric images are used. For example, when looking

3 Figure 2. Orthogonal views in 3D imaging. (a) Sagittal plane represents the left/right view (b) Coronal plane represents the front/back view (c) Traverse plane represents the top/down view [9]. for anomalies in structures or tissues, medical imaging specialists look at images in 2D ( slice by slice ) before examining the full volumetric 3D model. This means that 3D volumetric images are rotated, moved and cut, with guidance from their intuition and insights about the data they are able to read, as well as input from other collaborating specialists or physicians [3]. When software tools are used for viewing 3D medical images and performing tasks, medical imaging specialists typically explore a single anatomical structure. Keyboard and mouse-based inputs are mapped to navigate a camera around the anatomical structure, as well as to perform selection and manipulation. In the context of 3D interaction research, this is a common task [4] and illustrates the viewbased technique, in which different simultaneous orthogonal views (See Figure 1) are manipulated in 2D interface. While this technique is common for 3D software tools in the medical domain, they do not consider the challenges that exist if a medical imaging specialist wants to examine a 3D volumetric image along non-orthogonal planes. For example, if they were to examine a non-orthogonal cut of a tissue or organ, or follow a specific path within a 3D medical dataset, these techniques may be cumbersome. Interaction techniques in the research literature to address the challenges of non-orthogonal planes have been explored in the context of the medical domain, but they have been found to be obtrusive (e.g. virtual reality and its associated hardware) or require additional training to be properly utilized [5]. More recently, tangible techniques using mobile devices [6] have been explored for 3D interactions and have highlighted how leveraging intuitive spatial and physical relationships of users can be beneficial. Two common methods of mapping input devices to output are position control, which strictly maps input to a position (e.g. a mouse), and rate-control, where the input force and resistance is mapped to an output velocity that is used as a means of manipulation (e.g. airplane joystick). Prior research has shown that position-control mappings are generally more usable than rate-control mappings for most users [7]. These types of 3D interaction techniques and mappings have been applied to tablets previously [8], but as we show in this paper, the medical imaging domain introduces new challenges and considerations. Mobile 3D Interactions Working with a local medical imaging company, we developed two implementations based on the positionbased and rate-based techniques (see Figure 3 and 4). In both implementations, the interactions specifically manipulate the view around a fixed center point of a 3D volume and a user-selected orthogonal plane. Touchbased input (e.g. pinch-to-zoom and panning) manipulates the scale of the 3D volume and gyroscopic and accelerometer from the ipad provide the necessary information for non-orthogonal slicing of the 3D volume. Early Design Sessions To explore some of the challenges the medical imaging domain introduces to 3D interactions, we utilized a design session approach, allowing for an early examination of different 3D interaction types, in addition to providing a means of stimulating discussions with medical imaging specialists. Our design sessions primarily focused on qualitative feedback with three medical imaging specialists, who each performed

4 simple navigation tasks with both implementions, on varying sizes of 3D volumetric medical images. A B C Figure 3. The position-based implementation. To navigate through a 3D volumetric medical dataset, a user tilts an ipad in the direction in which they choose to navigate. As the navigation is done by position, the movements are discrete. In this example, the ipad is initially in a top down (a) orientation of a human skull in 3D space and as the device is tilted to the right (b), the skull appears but from a view on the left side in 3D space (c). Figure 4. The rate-based implementation. To navigate through a 3D volumetric medical dataset, a user tilts an ipad in the direction in which they choose to navigate. As the navigation is controlled by rate, the movements are continuous, much like an airplane joystick. In this example, as the device begins tilted to the left in 3D space, a human skull appears from a left view before eventually appearing in the right-view. Observations and Feedback From our design sessions, we clearly observed medical imaging specialists having less difficulty and easier navigation with the position-based implementation when compared to the rate-based implementation, confirming prior work by Zhai [7]. As one of the goals of the design session at this stage of the research was to gather feedback and inspiration on the usage of mobile 3D interactions for the medical domain, we synthesized the feedback from our discussions with the medical imaging specialists into four general themes. Preference. As a whole, both implementations received positive feedback and all medical imaging specialists repeatedly stated these interactions are definitely far more feasible than using a keyboard and mouse. Despite these comments, there was an immediate response when they were asked which implementation they preferred to use or found most feasible. Universally, they chose positionbased over the rate-based, indicating that it was easier to use. One of the medical imaging specialists noted: I preferred the [position-based] method because I feel I have more control and its more usable it s hard to look at structures precisely [in rate-based]. This sentiment was also echoed by the other medical imaging specialists, indicating that for tangible 3D interaction techniques, having

5 control is important for tasks such as finding important structures during information exploring or diagnosis or for decisions made that determine if another imaging procedure is needed. Learnability. Two of the medical imaging specialists preferred the position-based over rate-based, for usability reasons, and because they felt it was easier to learn. One medical imaging specialist noted: it would take time to get used to [rate-based] because it s not common in how we use or navigate medical images now. All medical imaging specialists stated, regardless of implementation or preference, that these types of interactions would take time to learn, but the amount of time to learn them would not be a hindrance if future adoption of such techniques were to occur. Alternatives. Despite the universal preference for position-based, the medical imaging specialists surprisingly found more value in the rate-based implementation for tasks that weren t a part of the study but a part of their domain. For instance, one medical imaging specialist noted: The [rate-based] one would be ideal for giving presentations that are more interactive or for teaching students and novice doctors and specialists about structures [positionbased] just wouldn t work there in that kind of setting, which needs to be dynamic. Another medical imaging specialist noted that: In fact, having both might actually work better. The [rate-based] can be used to quickly and roughly find an area and the [position-based] can then be used to refine the navigation and search ]. This was also echoed by the other medical imaging specialists, highlighting that in the medical imaging domain, not all 3D interactions are necessarily equal and their value is dependent on the different tasks that need to be performed. Also indicated is a possibility for hybrid interactions utilizing both rate-based and position-based techniques. Future Interactions. An interesting discrepancy in feedback occurred when the medical imaging specialists were asked about future interaction techniques. One noted I can definitely see this being used within 5 years and I myself would definitely use it at my desk right now. In contrast, another medical imaging specialist provided significantly different feedback, stating: I still see keyboard and mouse being used, despite this being better. Even years from now, a tablet won t be used but might be properly discussed by then. By introducing these types of [techniques], you re not just changing how imaging-specialists work, you re also changing the entire imaging chain, from image retrieval, to hardware, to software, to even how images are thought of. We still use keyboard and mouse because that s how we think. Discussion In our preliminary design sessions, we confirmed prior work by Zhai [7] but we also revealed several insights into mobile 3D interactions for the medical imaging domain that haven t been discussed in prior research literature. Firstly, we saw that rate-based interactions can provide value, and as suggested by medical imaging specialists, hybrid techniques that utilize a combination of rate-based and position-based can provide value for tasks involving quick exploration, followed by detailed exploration. Secondly, we saw that medical imaging specialists performed 3D navigation tasks in discrete steps, which may arise from familiarity with 2D interaction techniques and interfaces. Maintaining some familiarity to these techniques may

6 also impact the learnability of future mobile 3D interaction techniques and interfaces. This differs from the significant amount of research creating new 3D interactions, as the medical domain implies that 2D interactions need to still be taken into account. Lastly, one interesting backdrop provided by our discussions with the imaging specialists, is design implications for the medical imaging domain. A majority of interactions, techniques or prototypes for medical imaging in the research literature haven t fully considered the effects on the entire chain. For example, some work (such as our own) focuses solely on discrete medical disciplines without consideration for others disciplines. Having a disconnected chain when researching and developing 3D interaction techniques can hinder their adoption, not only in the medical imaging domain, but also other domains that require 3D interactions. Conclusion and Future Work In this paper, we discussed the challenges that imaging specialists currently face with 3D volumetric images, providing a backdrop for the introduction of tangible techniques for exploration of volumetric medical images in the medical domain. We implemented two different mobile 3D interaction techniques, position-based and rate-based and through our initial design sessions with medical imaging specialists, we saw that value is provided with both techniques, which differs from prior work in 3D interactions. We also saw that other aspects from the domain, such as familiarity with 2D inputs and techniques still need to be considered. This paper presents an initial starting point for research and design for mobile 3D interactions in the medical imaging domain, and moving forward, we to further explore the, in addition to further exploring hybrid interaction techniques. References [1] H. Cline, W. Lorensen, S. Souza, R. Kikinis, G. Gerig and T. Kennedy, "3D surface rendered MR images of the brain and its vasculature.," Computer Assisted Tomography, vol. 15, no. 2, pp , [2] L. Gallo, A. Minutolo and G. De Pietro, "A user interface for VR-ready 3D medical imaging by off-theshelf input devices," Computers in Biology and Medicine, vol. 40, no. 3, pp , [3] J. J. Robert, "Reality-based interaction: a framework for post-wimp interfaces," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'08), Florence, Italy, [4] J. Jankowski and M. Hachet, "A Survey of Interaction Techniques for Interactive 3D Environments," in Eurographics State of the Art Reports (STAR), Girona, Spain, [5] K. Montgomery, M. Stephanides, S. Schendel and M. Ross, "User interface paradigms for patient-specific surgical planning: lessons learned over a decade of research." Computerized Medical Imaging and Graphics, vol. 29, no. 3, pp , [6] M. Spindler, M. Martsch and R. Dachselt, "Going beyond the surface: studying multi-layer interaction above the tabletop," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'12), Austin, Texas, USA, [7] S. Zhai, "User performance in relation to 3D input device design," ACM SIGGRAPH Computer Graphics, vol. 32, no. 4, pp , November [8] B. Jones, R. Sodhi, D. Forsyth, B. Bailey and G. Maciocci, "Around device interaction for multiscale navigation," in Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services (MobileHCI '12), San Francisco, California, USA, [9] Image extracted from: on January 4, 2014.

To Virtualize or Not? The Importance of Physical and Virtual Components in Augmented Reality Board Games Jessica Ip and Jeremy Cooperstock, Centre for Intelligent Machines, McGill University, Montreal,

CS-525V: Building Effective Virtual Worlds Input Devices Robert W. Lindeman Worcester Polytechnic Institute Department of Computer Science gogo@wpi.edu Motivation The mouse and keyboard are good for general

Surgery Support System as a Surgeon s Advanced Hand and Eye 8 Surgery Support System as a Surgeon s Advanced Hand and Eye Kazutoshi Kan Michio Oikawa Takashi Azuma Shio Miyamoto OVERVIEW: A surgery support

Image Area View Point Medical Imaging Advanced Imaging Solutions for Diagnosis, Localization, Treatment Planning and Monitoring www.infosys.com Over the years, medical imaging has become vital in the early

Sectra Medical Education and Training Suite Sectra Table Medical Education and Clinical Training Needs and challenges Current educational tools do not allow students and doctors to interact with real patients

Voxar 3D TM A suite of advanced visualization and analysis software tools The power to deliver advanced visualization throughout the enterprise To effectively manage the rapid growth of large volumetric

Electronic Medical Record Integration and Controversy: Visualizing the Adoption of EMRs in the U.S. Lindsay A. Carrabine Drexel University, College of Information Science and Technology Abstract The integration

Classroom Management, part of Intel Education User s Guide Available in the U.S. only for the K-2 market starting fall 20. Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS.

Hello, I m Ryan Huber. I m currently the lead web developer and designer for a small team at Vanderbilt University. I value creative, thoughtful, elegant, and accessible solutions to challenges and I m

1 11.1 Definitions and Motivation Lot of research and papers in this emerging field: Visual Analytics: Scope and Challenges of Keim et al. Illuminating the path of Thomas and Cook 2 11.1 Definitions and

Information for Patients with an Abnormal Lung X-Ray Some of the materials in this resource were adapted, with permission, from: Understanding Lung Cancer, Juravinski Cancer Centre, Hamilton Health Sciences,

Visualisation in the Google Cloud by Kieran Barker, 1 School of Computing, Faculty of Engineering ABSTRACT Providing software as a service is an emerging trend in the computing world. This paper explores

3 Testing Websites with Users 3 TESTING WEBSITES WITH USERS Better Practice Checklist Practical guides for effective use of new technologies in Government www.agimo.gov.au/checklists version 3, 2004 Introduction

Sixth LACCEI International Latin American and Caribbean Conference for Engineering and Technology (LACCEI 2008) Partnering to Success: Engineering, Education, Research and Development June 4 June 6 2008,

1 An evaluation of a Wizard approach to Web design Karl W. Sandberg Joel Palmius Yan Pan Mid Sweden University and Luleå University of Technology, Sweden Mid Sweden University Luleå University of Technology

Accuracy of using a tablet device for the use of digital radiology manipulation and measurements Dr)Yasith)Edirisinghe)MBBS 1,)Mr)Marcus)Crossette)BaAppSc 1 1 SouthernHealth,Clayton,Australia CorrespondingAuthor:yasith_e@yahoo.co.nz

Twelve The 3D Curved MPR Viewer This Chapter describes how to visualize and reformat a 3D dataset in a Curved MPR plane: Curved Planar Reformation (CPR). The 3D Curved MPR Viewer is a window opened from

A Comparison of Real and Virtual 3D Construction Tools with Novice Users Hooman Baradaran, Wolfgang Stuerzlinger Interactive Systems Research Group Department of Computer Science York University Toronto,

Release Your App on Sunday Eve: Finding the Best Time to Deploy Apps Niels Henze University of Oldenburg Escherweg 2 Oldenburg, Germany Niels.Henze@Uni-Oldenburg.de Susanne Boll University of Oldenburg

Computer-Assisted Spine Surgery Computer-assisted spine surgery enables surgeons to view the position of their instruments against three-dimensional images of a patient s spine, increasing the accuracy

University of Edinburgh College of Science and Engineering School of Informatics Informatics Research Proposal supervised by Dr. Sethu Vijayakumar Optimized bandwidth usage for real-time remote surveillance

Growing the Digital Business: Spotlight on Mobile Apps Accenture Mobility Research 2015 Introduction 2 In the past five years, companies have spent considerable time, money and attention developing mobile

Human-Computer Interaction: Input Devices Robert J.K. Jacob Department of Electrical Engineering and Computer Science Tufts University Medford, Mass. All aspects of human-computer interaction, from the

The Complete Guide to CUSTOM FIELD SERVICE APPLICATIONS Copyright 2014 Published by Art & Logic All rights reserved. Except as permitted under U.S. Copyright Act of 1976, no part of this publication may

ENTERPISE CONTENT MANAGEMENT: A USABILITY STUDY Robert L. Totterdale, Robert Morris University, robert.totterdale@accenture.com ABSTRACT Over 90% of business records today are available in electronic form.

MRI: Magnetic resonance imaging: Like CT, MRI produces detailed cross sectional images of body regions. However the physical principles of image acquisition are different and very complicated. The following

CHAPTER 7 SYSTEM DESIGN Expected Outcomes Able to know database design Able to understand designing form and report Able to know designing interfaces System Design A process of transforming from logical

Tracking devices Virtual Reality Technology and Programming TNM053: Lecture 4: Tracking and I/O devices Referred to head-tracking many times Needed to get good stereo effect with parallax Essential for

Department of Veterans Affairs Health Services Research & Development Service Electronic Health Record-based Interventions for Reducing Inappropriate Imaging in the Clinical Setting: A Systematic Review

Creating a Jointly Sponsored Master of Science in Software Engineering Gregory W. Hislop, College of Information Science and Technology Spiros Mancoridis, College of Arts and Science P. M. Shankar, College

Chapter 14: Usability testing and field studies The aims: Explain how to do usability testing through examples. Outline the basics of experimental design. Discuss the methods used in usability testing.

From the: U.S. Bureau of Labor Statistics Radiology Tech What They Do Radiologic technologists (RTs) perform diagnostic imaging examinations, such as x rays, on patients. Duties RTs typically do the following:

FUJITSU Education Solution K-12 School Business Support Services for Elementary and Junior High Schools Toru Toguchi Under the measures and policies of the Ministry of Education, Culture, Sports, Science

ANALYSIS TECHNOLOGY er-oriented Electric Field Analysis System for Power Cable Accessories Yuuichi NAKAMURA*, Tomohiro KEISHI and Miki USUI Sumitomo Electric has worked on electric for more than 30 years.