abstract

The number of design studies using statistical testing has increased dramatically over the past decade. While this has benefits, statistical testing requires scrutiny to protect against common errors and misconceptions. To illuminate how these issues affect design, this paper provides a comprehensive analysis of the past decade of studies within the DTM community. Specifically, the paper 1) reviews the background of statistical testing across multiple fields, highlighting recommended practices, 2) discusses its use in the Design community, and 3) provides concrete methods for authors and reviewers to evaluate statistical tests employed in Design Cognition studies.

The analysis identifies recurring issues with: ignoring multiple comparisons; deficiencies in study and result reporting; inadequate defense of modeling assumptions; unavailable plots, data, and analysis files for replication; and lack of interpretation of statistical results with respect to practical outcomes or alternate forms of scientific inquiry. Based upon practices already adopted in other research communities, we put forth: 1) checklists that help authors and reviewers verify data reporting, analysis, and statistical assumptions; and 2) design guidelines for creating more reproducible design experiments. Ultimately, we argue that design researchers, reviewers, and editors should view statistical testing less like a sword and more like a scalpel — a specialized tool best used in concert with other techniques — to gain a more complete picture of Design Cognition.

Fuge M. A Scalpel Not a Sword: On the Role of Statistical Tests in Design Cognition. ASME. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Volume 7: 27th International Conference on Design Theory and Methodology ():V007T06A008. doi:10.1115/DETC2015-46840.

Shibboleth is an access management service that provides single sign-on protected resources.
It replaces the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session.
It operates independently of a user's location or IP address.
If your institution uses Shibboleth authentication, please contact your site administrator to receive your user name and password.