Aristotle, Gestalt, and Usability Testing

Aristotle, Gestalt, and Usability Testing

Sokol Zace, Ergoline

In September 2004, I was assigned the task of identifying and recruiting ten potential users of a new model of digital camera for usability testing in the Milan area in Italy. My direct client was a US-based usability research organization. The final client, the digital camera manufacturer, was a big, high-tech corporation with headquarters in the US. I received information about required user profiles, such as percentages of users per gender, age, and experience with digital cameras. In addition, the client sent me a questionnaire consisting of eight groups of statements about the level of expertise with digital and traditional photography, camera accessories purchased, and photo print preferences. During the screening interviews, the potential users had to select the statements that best represented them. Each selected statement corresponded to a numeric score that had to be entered in an Excel spreadsheet. A formula calculated the scores of each candidate and generated a final one-digit score representing the user profile. The formula was so long and complex that the size of the Excel file containing it was 16.5 megabytes! My task was to recruit only users whose answers generated a score of five.

I started to identify and interview the candidates two weeks prior to the beginning of the tests, a time frame that had always worked well for me. I usually select one user among four candidates interviewed, so I expected to make around forty calls to recruit ten users. Unfortunately, after twenty calls I wasn’t able to find any person scoring a five. I was wondering if we had to postpone the tests and so was the client. Either the user profile the client was looking for was an extreme rarity or there was something wrong with my recruitment process.

I sent this e-mail to the client’s project manager: “What kind of candidates tend to score a five on your Excel spreadsheet? Which criteria should I use to pre-screen the candidates? If I do not pre-screen them, all I can do is call a great number of them, crossing my fingers in the hope of getting five as a score.”

The client’s reply was:

“If you haven’t found many “fives” by end of day (Italy local time) Friday (24 Sept.), let me know and I’ll ask our client about using “twos” and hopefully you can schedule any “twos” you have found.

A general description of the user profile is:

Owns a computer at home

Needs to or takes a lot of photographs each month

Most likely has children (the children are probably what the person is taking a lot of pictures of each month)

Most likely a female (mothers like to take pictures of their children)

Probably isn’t interested in using a lot of the technical aspects of a camera, so the person probably wants to own a camera that is easy to use

I hope this helps.”

It helped. I noticed that I had been automatically following the recruitment criteria I had used in two previous usability-testing projects. The most recent products tested by the research team in which I took part were cell phones and household appliances. For those products, the clients needed expert users. I had taken it for granted that the digital camera designers needed expert users too. This incorrect assumption had polluted my pre-screening process resulting in users that never scored a five. The “fives,” as they were called in the client’s jargon, were users with no expertise in digital photography.

I went through the database of potential users one more time. The database contained all candidates excluded from previous usability tests. I knew some of the candidates personally. What I did while reading about those candidates was visualize them in situations in which I had already observed them or in situations that I thought they were likely to be involved. If that mental image gave me the feeling that the candidates would fit the user profile I was looking for, I called them. For example, Elisabetta is an Ayurveda body worker, has
children, spends a lot of time with them, and is a new age kind of person with a lot of interest in natural things and with no interest in the technical aspects of the products she uses. She looked perfect for the digital camera usability test. I called her, went through the screening statements, and she scored a five. I followed the same approach with the other candidates and by the end of the day, I had already found four “fives.” During the following two days I completed the testing schedule.

The first thought that I had when I saw the drastic improvement was that the client’s mathematical model was very accurate. I have to admit that 16.5 MB of formula had given me the opposite impression. I assumed that very complex systems tend to be inaccurate. This wasn’t the case.

My next thought was that the recruitment process would have been more productive if I had personas and scenarios about the fifth category. Personas and scenarios are generated for marketing and design purposes, but they are precious tools in usability testing and technical publications too. High-tech companies should divulge personas and scenarios across the organization. If that were the case, I would have received personas and scenarios along with the mathematical model, and I wouldn’t have wasted the first twenty recruiting calls. I would have used personas and scenarios to pre-screen the candidates and use the Excel calculation sheet to make sure my pre-screening was correct.

I compared the digital camera usability testing with other usability testing projects in which I had participated in Italy. The previous tests were thoroughly developed by local teams. The digital camera usability testing process, including the recruitment criteria, was developed in the US. The camera manufacturer used the same usability testing process in different cities in the US and overseas. They wanted to make sure that their camera would give a positive user experience all over the world, in spite of cultural differences among countries. Before coming to Italy, they had already assessed the camera usability in the US and the UK. In both countries, the recruitment was carried out by agencies that had received the same screening information I had. Why had the US and UK recruiters not asked the same question that I asked about the meaning of five? Even if they had conducted the screening interviews without my wrong assumptions, they should have noticed that the number of calls needed to select ten “fives” was unusually high-around seven calls for one recruited person-based on the theory of probability and on the fact that the digital camera market segmentation contained seven categories. I think the answer to my question has to be found in the different work styles between Anglo-Saxon and southern European countries.

Italian usability researchers tend to use a lot of intuition, imagination, and multi-sensory participation in the recruitment and testing process. I have observed that Anglo-Saxon researchers tend to use less direct personal participation and more scientific, impersonal tools between themselves and the users. The Italian way is not particularly accurate but it is very fast and interesting. The Anglo-Saxon way is very accurate, supposing that mathematical models are accurate, but it generated some harmful misinterpretations in Italy. In fact, I was being influenced by the intuitive methodology while trying to complete the information at my disposal, assuming that the users should have a high technical profile. When conducting international research projects, it’s important to know the work culture of each country and adapt the methodology to those cultures. The localization of methodologies may seem expensive and time consuming, unjustified for small international projects, but even in these cases, there is a solution. The entire research effort can be delegated to a local team, giving them high-level information about the research goals and letting them implement the middle-level and low-level tasks, such as user-screening methodologies. I have seen this approach work well in various usability research projects in which the clients were located in the US and the UK and the usability testing team was Italian.

My reflections on this project reminded me of a series of analogous observations made by researchers of different fields on two distinct methodologies we use to gather and process information. Here are two examples.

Edwin C. Nevis, in his classic book Organizational Consulting—A Gestalt Approach, analyzing two different approaches in organizational assessment, writes: “If Sherlock Holmes is a metaphor for the directed awareness approach, a metaphor for the undirected mode is Detective Columbo, hero of the TV series. Unlike Holmes, who is well organized, precise, knowing, superior in perception and logical reasoning, rational and deductively oriented, Columbo is naive, rambling, slow moving, seemingly unfocused in his perceptiveness, and fuzzy, if not downright illogical. … Columbo may be said to act like a sponge, immersing himself in his milieu and waiting for important clues to be drawn to him. … Columbo teaches or coaxes the people and environment involved to ‘give up’ data as he makes contact with them. Holmes rarely makes close, personal contact with the villain; Columbo’s method rests largely upon repeated personal contacts. The Holmes approach is an application of 19th-century science as it combined technical discoveries with logical analysis. The method of Columbo is an application of 20th-century existentialism and its emphasis upon uncertainty, being, and here-and-now phenomena.”

Fritjof Capra in The Tao of Physics, writes: “A mystical experience… is not any more unique than a modern experiment in physics. On the other hand, it is not less sophisticated either, although its sophistication is of a very different kind. The complexity and efficiency of the physicist’s technical apparatus is matched, if not surpassed, by that of the mystic’s consciousness. … The scientists and the mystics, then, have developed highly sophisticated methods of observing nature, which are inaccessible to the layperson. A page from a journal of modern experimental physics will be as mysterious to the uninitiated as a Tibetan mandala.”

The two different paths towards knowledge emerge in every field. Aristotle versus Lao Tzu, west versus east, scientific method versus direct sensory experience, tools versus intuition, left brain analysis versus right brain global perception. But are those paths really distant from each other? No, they complete each other. The coordinated use of both methods increases effectiveness and makes the work experience more aesthetically rewarding. It’s time we give up thinking in terms of dichotomies. Holistic methodologies are making their way into all fields of knowledge, far beyond the human potential movement that pioneered the holistic vision in the sixties and seventies.