Evaluation

During the product development process, effective design evaluation is central to user interface design and the drive to ensure a great user experience.

There are many ways to evaluate a product and the best way can depend on the stage of product development as well as the available resources (e.g., time, funding, prototype or finished products). Our evaluation methods include:

Early Stage Evaluation

We employ many techniques to collect feedback on early design concepts. One technique is called a cognitive walkthrough. The approach may call upon respondents to study product sketches, simple physical models, or a series of static computer screens and think through and explain how they might perform a task.

Examples:

Evaluate the workflow associated with operating an insulin infusion pump by viewing multiple sample screens and handling a “block model.”

Middle Stage Evaluation

Frequently, we are called upon to critique a product, also referred to as an expert review or heuristic analysis. In these cases, we apply established design principles and our judgment to identify user interface design strengths and opportunities for improvement. Such evaluations can usually be done quickly and at low cost yet produce valuable findings.

We are also asked to conduct formative usability tests of products. Such tests can be quick to deliver findings for a design team that might be in the middle of development sprint. Alternatively, a formative test might be a more formal activity involving detailed test planning and reporting.

Examples:

Perform an expert review of the screens associated with a fitness app to be used on a smart phone.

Conduct an 8-participant formative usability test of an exoskeleton used to help people suffering from stroke regain their stride.

Conduct a 15-participant “pre-summative” usability test of a dialysis machine, following a test plan (i.e., protocol) intended for use in the subsequent summative (i.e., validation) usability test.

Product evaluations expose design strengths and opportunities, such as those shown in the following table for a hypothetical home dialysis machine.

Sample Findings

Positive

Negative

Visual alarms are displayed in a conspicuous (i.e., attention-getting) manner.

[Source: Critique]

Smaller and weaker users are likely to have difficulty lifting the 40 lb. machine.

[Source: Critique]

Users considered the set-up workflow to be intuitive.

[Source: Cognitive walkthrough]

Several users were confused about the meaning of the symbol on the backlight control.

[Source: Formative usability test]

Test participants with small hands and those with large hands felt that the handle provide a secure, comfortable grip.

[Source: Formative usability test]

Four out of 12 test participants were unable to install the tubing set correctly.

[Source: Formative usability test]

14 out of 15 users practiced aseptic technique when connecting the patient line to the machine.

[Source: Summative usability test]

Two participants kinked the drain tube when placing it in its holder.

[Source: Summative usability test]

Late Stage Evaluation

When a product is nearly final, we can conduct a summative (i.e., validation) usability test. For medical products, this is a pivotal activity that can produce proof that a product can be used safely and effectively by the intended users. Such testing is also a regulatory requirement for products seeking FDA clearance. Such testing might take place in a conventional usability test laboratory or a more advanced setting, such as a high-end medical simulator.

We have conducted such tests with small (n=15) and very large (n > 200) population samples. Our test plans are routinely reviewed and tacitly accepted by the FDA. We conduct tests in our own laboratories, in facilities worldwide, or in rented focus group facilities. As required, our test reports thoroughly describe the usability testing approach, test findings, and root causes of any persisting interaction problems. We base our testing and reporting approaches on available guidance from multiple sources (e.g., FDA, AAMI, IEC), as well as best practices

Examples:

Conduct a 75-participant, summative usability test of an insulin pen injector to generate data proving that the product can be used safely and effectively by adolescents, adults, seniors, caregivers, and healthcare providers. (Note: Sample size is dictated by FDA requirements pertaining to validation studies. See: Applying Human Factors and Usability Engineering to Medical Devices, issued by FDA on February 3, 2016)

Conduct a 15-participant, summative usability test of a heart-lung machine used by perfusionists to sustain patients undergoing open-heart surgery.

Conduct a 12-participant, summative usability test of a tablet application used by automobile mechanics to diagnose and, where possible, fix electrical and mechanical problems.

Post-market stage evaluation

We frequently conduct product assessments, including design reviews and usability tests, of marketed products. Such reviews may serve the following purposes:

Provide benchmark performance data that forms the foundation for developing a next-generation product or competing product.

Provide user interaction-oriented performance data upon which to base a marketing claim.

Gain insight into reported interaction problems that might have led to product complaints, returns, and other types of failures.

Examples:

Conduct a usability test comparing the perceived ease of setting up a dialysis machine to deliver a treatment.

Conduct a study with people who have diabetes to determine which lancing device is easiest to use and causes the least pain.

Perform an expert review of the user interface of a recalled surgical device to determine the root cause of use errors that led to harm, and suggest design changes as part of a CAPA effort.

Coming soon: Designing for Safe Use (CRC Press, late 2018)

We – this book’s authors/designers – are members of the Human Factors Research and Design (HFR&D) at EMERGO by UL. In this book, we have consolidated the lessons we have learned about designing for safe use, that is, designing products that shield people from harm to the extent possible.

We settled on a target of 100 principles on how to make products safer. The principles pertain to hardware, software, document, and document design. Yes, settling on an even one hundred principles was a bit arbitrary and cliché. The myriad ways to design for safe use do not stop sharply at one hundred. But, we think we covered many of the key ones.

We elected to use the term “product” broadly to cover things one might consider to be systems, machines, equipment, instruments, tools, applications, manuals, and instructions. These are all things that need to be designed properly to eliminate or reduce the chance of harm due to normal use and foreseeable misuse.

Most of the design principles could be addressed in an expanded form; even an entire book of its own. We choose brevity for the sake of communicating core concepts with some fun facts to spice things up.

As you read the book, be mindful that the science and art of making things safe is ever changing and that some of the content we present is sure to age. So, complement our guidance with insights you may gain from other sources, ranging from books to technical articles to standards and more.

Usability Testing of Medical Devices covers the nitty-gritty of usability test planning, conducting, and results reporting. The book also discusses the government regulations and industry standards that motivate many medical device manufacturers to conduct usability tests.

Since publication of the first edition, the FDA and other regulatory groups have modified their regulations and expectations regarding how medical device manufacturers should approach usability testing. Reflecting these changes, this Second Edition provides updated guidance to readers with an interest or direct role in conducting a usability test of a medical device or system. Key updates involve the 2011 FDA guidance on human factors engineering, requirements set forth by the third edition of IEC 60601 and closely related IEC 62366-1:2015, linking usability test tasks to risk analysis results, and analyzing root causes of use errors that occur during usability tests.

Written by seasoned human factors specialists, Usability Testing of Medical Devices, Second Edition is an informative, practical, and up-to-date handbook for conducting usability tests of medical devices. The book helps ensure a smooth and painless development process―and thus, safe and effective medical devices. Buy the book.

By Michael Wiklund, Laura Birmingham, and Stephanie Larsen

This book provides the foundation for developing specific human factors engineering (HFE) work products that are needed to meet the FDA's human factors engineering (HFE) guidance. The authors have created a fictitious company and product to generate concrete examples of the plans and reports developed during various stages of HFE. The book includes an HFE project plan, a formative usability test plan and report, a summative (i.e., validation) usability test plan and report, and an HFE report. These work products and additional content outline the activities necessary to develop safe and effective medical devices, making this book an ideal resource for anyone interested in the medical technology field. Buy the book.

This book offers practical guidance on how to methodically discover and explain the root cause of a use error―a mistake―that occurs when someone uses a medical device. Covering medical devices used in the home and those used in clinical environments, the book presents informative case studies about the use errors (mistakes) that people make when using a medical device, the potential consequences, and design-based preventions.

Using clear illustrations and simple narrative explanations, the text:

Covers the fundamentals and language of root cause analysis and regulators’ expectations regarding the thorough analysis of use errors

Describes how to identify use errors, interview users about use errors, and fix user interface design flaws that could induce use errors

Reinforces the application of best practices in human factors engineering, including conducting both formative and summative usability tests

Developed to promote the design of safe, effective, and usable medical devices, Handbook of Human Factors in Medical Device Design provides a single convenient source of authoritative information to support evidence-based design and evaluation of medical device user interfaces using rigorous human factors engineering principles. It offers guidance on user-centric design supported by discussions of design issues, case studies, and examples. The book sets the foundation with coverage of fundamental topics such as aligning the interactive nature of medical devices to the expected use environments ranging from hospitals and ambulances to patients’ homes, drawing on anthropometric and biomechanical data to ensure that designs match the intended users’ bodies and physical abilities, and conducting usability tests and other evaluations to ensure that devices perform as intended. It then focuses on applied design issues, offering guidance on the design of specific types of devices and designing devices for particular use environments. Adapted in part from established design standards and conventions, the design guidance presented in this work distills professional judgment extracted from the contributing authors’ years of experience in applied analysis and design. Written in true handbook style, each chapter stands alone and includes tables, illustrations, and cross references, allowing you to quickly find the exact information you need. Most chapters begin with a general introduction to the selected topic, followed by the presentation of general and special design considerations and then specific, numbered design guidelines. The book also presents a listing of resources, literature, and website references. It not only focuses on the human factors issues that arise when developing medical devices, it supplies the necessary guidance to resolve them. Buy the book.

Advocating a user-centered approach to medical technology design, Designing Usability into Medical Products covers the essential processes and specific techniques necessary to produce safe, effective, usable, and appealing medical systems and products. Written by experts on user-centered research, design, and evaluation, the book provides a range of alternative approaches to the subject. Wiklund and Wilcox explore how to make medical devices safe and effective by involving users in the design process. They discuss specific design and evaluation methods and tools, present case studies of user-friendly medical technologies and corporate human factors programs, and supply related resources for medical design professionals.

The book conveys an in-depth understanding of the user-centered design process, covers design methods for FDA compliance, and offers guidance on performing a variety of hands-on user research, user interface design, and user interface evaluation. The authors make a compelling case for treating the user's needs and preferences as a top design priority, rather than an afterthought. They demonstrate that high-quality customer interactions with systems and products leads to effective medical diagnosis and treatment, increases the physical and mental well being of patients and caregivers, and leads to commercial success in a crowded marketplace. Buy the book.

Usability in Practice

Editor: Michael Wiklund

This volume investigates how major corporations, such as Microsoft, Borland, Apple, Eastman Kodak, and Silicon Graphics, address usability issues. It presents case studies of each organization, outlining their program structures, program goals, and team members' responsibilities and resources. The book also addresses how usability is marketed inside the organization and to customers, as well as the lessons learned during the course of product development efforts. Each illustrated study includes advice that should help readers establish and manage their own program.

The Beauty of Unity-in-Variety

by Ruben Post

This thesis embarks from the idea that aesthetic appreciation of product designs is determined by simultaneously perceiving the two partially opposing dimensions of unity and variety. People actively avoid boredom by searching for variety because it challenges the senses and offers the potential of learning new information. Hence, people browse through thick catalogues, are attracted to colourful bouquets and let their eyes and hands explore a novel car interior. In doing so, these products offer stimulation to the senses. However, too much variety leads to confusion, as people fail to make sense of what they perceive. It is therefore that they appreciate perceiving unity at the same time, as it brings structure to variety; items in a catalogue are precisely ordered, flowers are neatly arranged and components of a car interior are carefully picked and organized. The above idea is captured in an age-old aesthetic principle, aptly named Unity-in-Variety (UiV). The principle states that perceiving a balance between the opposing forces of unity and variety is aesthetically preferred. While this principle has been argued to explain aesthetic appreciation for works of art, music and landscapes, little empirical research existed on this principle and, to our knowledge, none for product designs.

Available at Institutional Repository, Delft University in Delft, The Netherlands. Contact Ruben Post at ruben.post@ul.com.