Testing PatternFly Designs

When adding new features to PatternFly patterns and widgets healthy debate sometimes arises among the PatternFly team about the best way to design the features. Other times, the team might be in agreement about the best design approach but still wants external validation before the design gets implemented. When either of these situations arises, we try to make decisions based on data provided by users through empirical testing. Recently, questions arose around three issues: (1) the best way to communicate pagination behaviors for table, list, and card views, (2) the best way to visualize links in aggregate status cards, and (3) the best way to provide syntax hints for form fields. To answer these questions the team conducted some “cafeteria tests” with users. Cafeteria tests are short usability tests conducted in a high traffic area (like a cafeteria) that typically focus on a single question or a small number of questions and can be completed in a few minutes. Participants can be recruited beforehand or just grabbed on the spot when they wander by the testing location. The rest of this blog describes the three studies we conducted and the findings and recommendations for each.

Study 1 – Pagination Behavior

The goal of this study was to gauge participants’ understanding and usage of the pagination controls in PatternFly table, list, and card views. The controls include a column selector for selecting all items currently visible, a Select All button for selecting all items (including those not currently visible on the page), previous and next buttons, and a drop down for selecting the numbers of items to view per page.

Participants were shown a table view and asked the following questions:

How would you select the items on the current page?

How would you select all of the items in the table.

If a single item in the table were selected, what would you expect to see when you clicked to the next page – would the current selection persist?

Results showed that the meaning of the Column Selector control is unclear. Some participants thought it selected all items on the current page; others thought it selected all items in the table; and some (about 1 in 3) either didn’t notice the Column Selector or didn’t realize it was a control. (Interestingly, about 1/4 of participants also noted that they never noticed the Select All control in the upper right corner of the table.) When asked about selection persistence, there was no consensus among participants. About 2/3 correctly believed that selections persist when moving among pages, while the rest thought they would not.

Based on these findings we made the following recommendations:

Add flyover help text to the Column Selector explaining its function.

Change the appearance of the “Selected: x of y” when paging to help users understand selection persistence.

Study 2 – Visualizing Links on Aggregate Status Cards

The goal of this study was to understand users’ preferences for the way links are visualized in PatternFly aggregate status cards. Three options were considered:

Text change on hover – By default, link text text is black and icon color is determined by status (green, yellow, or red). On mouse over, text changes to blue and underlined.

All change on hover – By default, link text text is black and icon color is determined by status (green, yellow, or red). On mouse over, text and icon change to blue and text is underlined.

Static

Text Change on Hover

All Change on Hover

Participants were shown three identical web pages that included aggregate status cards. The only difference among the pages was how the links on the cards were visualized. Participants were not told what the difference was. Participants were allowed to view each page for about a minute and were told they could move the mouse around the page but not click, because none of the links were active. After viewing all three pages participants were asked if they noticed any differences among the pages. Then the different link visualization conditions were pointed out and participants were asked which they preferred and why.

Results showed that participants had difficulty identifying the subtle differences among the pages. Only three of 16 participants were able to identify the differences among the three pages without being shown. In addition, 12 of 16 participants did not notice a difference between the “All change” and “Text change” conditions until it was pointed out. Regarding preference, slightly more than half of the users preferred the “All change” condition and about 3/4 preferred either the “All change” or “Text change” conditions. The three users who stated a preference for the “Static” condition all said it was the most obvious.

Based on these findings we recommended that the “All change” condition be implemented. That is, by default text should be black and icons colored according to the status they’re indicating. On mouse over, icons and text should change to blue, provided the icon is clickable.

Study 3 – Syntax Hints

The goal of this study was to validate a proposed design for the way syntax hints are displayed in PatternFly form fields. The proposed design is to display hints in a gray, italicized font inside the form field itself. The hints disappear when the user begins to type in the field. We wanted to know if users are OK with this method of displaying hints and whether them disappearing would impact performance. Participants were given a card with various pieces of information (credit card number, date, IP address, etc.) and asked to input the information into a website form using the formats shown by the syntax hints.

Results showed that users had little difficulty correctly entering the information from the cards into the form fields, with one exception. No users input the Outgoing Mail Server correctly. They all neglected to include the “smtp.” before the server name. Whether this is because they did not notice it before the syntax hint disappeared or is an artifact of the test, in that users did not understand mail server formatting and didn’t think the “smtp.” to be necessary, is not clear. Despite having little trouble using the form with the syntax hints displayed in the form fields, user comments indicated that displaying hints in this manner is not universally preferred. Half of the participants commented (unprompted) that they want hints to be visible at all times and suggested moving hints outside the fields.

Based on these findings we recommended that PatternFly continue exploring options for the syntax hints design pattern.