Wolters Kluwer Health may email you for journal alerts and information, but is committed
to maintaining your privacy and will not share your personal information without
your express consent. For more information, please refer to our Privacy Policy.

Abstract

There are many reasons why online self-management education is attractive to both patients and providers. AsthmaWise, an online self-management program, was developed using a Moodle platform, to enable older adults to learn asthma self-management skills. This study aimed to improve AsthmaWise through conducting: usability testing with a sample of end users; a cognitive walk-through undertaken by an independent health researcher; and assessment of content readability. A Perceived Health Web Site Usability Questionnaire score of 67% was achieved, indicating that there were usability issues that needed to be addressed. The cognitive walk-through and readability assessment identified unique issues that were not identified through usability testing with end users. The testing process allowed issues to be identified and rectified before piloting AsthmaWise, creating a more accessible and refined end product. The involvement of the site designer in the testing process was valuable and is highly recommended. This study shows that usability testing involving both end users and experts is an essential part of the design process that is relatively easy and inexpensive to undertake and can be effectively conducted by a nonexpert.

The potential of e-health as a source of consumer health information and support is wide ranging and undisputed. Consumer e-health applications have been classified into five categories: self-management applications, peer-to-peer support groups, decision aids, personal health records, and Internet use.1 Although all of these categories have strengths and weaknesses, this article focuses on the development of a self-management application for older adults with a current diagnosis of asthma. Some of the benefits of online self-management education include the potential reach of such applications, their ready availability, and ability to overcome barriers of time and geography.2 Such sites are relatively inexpensive to set up and launch, once initial development costs have been met.3 Furthermore, information can be readily updated in a timely manner.

As the biggest users of healthcare resources, older adults potentially have the most to gain from online self-management education. Although, traditionally, older adults have not been considered “tech-savvy,” they are moving online in increasing numbers. This has been documented in the United States, where the online population of people aged between 70 and 75 years was reported as 34% in 2012.4 Furthermore, a recent exploratory study found that many older Australians are online (62%) and most of those who are connected feel comfortable using the Internet (93%).5 Once online, adults aged between 51 and 59 years and those older than 70 years were most likely to use e-mail and search for health information.6

Asthma

Asthma is a common disease of the airways, which is characterized by episodes of wheezing, breathlessness, chest tightness, and persistent cough.7–9 Asthma is often thought of as a childhood illness, mortality is greatest in those older than 60 years.10 Although there is currently no cure for asthma, it can be effectively controlled through self-management, which minimizes the impact of the disease on a patient’s day-to-day activities.11–13 However, research suggests that primary care workers have insufficient time during appointments to teach patients new self-management skills.14–16 A recent survey of UK general practitioners found that 46% felt that there was not enough time to educate their patients about asthma.17

AsthmaWise

AsthmaWise was an asthma education site, designed specifically to help adults older than 55 years to learn about asthma self-management. It was produced by the University of Wollongong in collaboration with Asthma Foundation New South Wales after extensive formative research with the target population.5,18,19 AsthmaWise was developed using Moodle (Moodle, Perth, Australia), an open source, e-learning software platform that allows educators to create online courses.

It was envisaged that AsthmaWise would have a total of six modules that would be completed by participants in their own time. At the time of usability testing, only three modules were complete and visible to participants because of technical issues that had occurred. These modules were Learn More About Asthma, Managing Your Asthma, and Asthma Attacks.

Usability Testing

Usability testing is the process of “observing and learning from your users, who are working with your product to perform tasks that are real and meaningful to them.”20 It is often part of a larger user-centered design process, which has an iterative nature (ie, the product is tested and modified a number of times to enhance the user’s experience), resulting in a well-refined product. This is an important process because it has been shown that even experienced Web users can easily become confused on an unfamiliar site by the information architecture or small usability problems.20 Testing can be conducted either during development (formative testing) or once the site is complete (summative testing)20; it is seen as an essential process in Web site development that ensures the product meets the needs of the end users, rather than of the developer. As such, usability testing needs to be both broad and inclusive and include input from both end-users and experts.21 Although usability guidelines are available to guide design around general user behavior as well as for specific site genres, it is recommended that individual usability tests be conducted to obtain detailed information about specific sites.22 Usability testing is particularly important with older adults because the aging process is known to complicate computer and Internet use.23–25 Possible issues include vision and hearing loss, which may impact users’ contrast sensitivity and result in problems reading small fonts and hearing sounds embedded on sites.20,26,27 Furthermore, older adults may also experience cognitive decline (both short-term memory and speed of processing), which creates difficulty for users in recalling previously viewed information; and motor limitations due to physical decline may complicate mouse use.20,24–27 Despite these known challenges, there is a paucity of usability testing research reported in this population with regard to health Web sites.28

The purpose of this study was to assess and subsequently improve the usability of AsthmaWise through the combined use of usability testing, conducted with a sample of end users; a cognitive walk-through of the proposed site undertaken by an independent health researcher; and assessment of readability using Flesch-Kincaid Grade Level and Flesch Reading Ease statistics. The cost of usability testing is also discussed, as this process is often overlooked because of perceived expense. The results were used to inform refinement of AsthmaWise before the site went live.

METHODS

Design

Usability testing was undertaken using a think-aloud process and Morae Recorder 3.2.1 (TechSmith, Okemos, MI). The think-aloud process involves participants explaining what they are doing or thinking during testing. Morae is a usability testing software that allows user interactions in the form of visual, audio, and mouse movements to be recorded and analyzed. Testing involved an initial short demographic survey, followed by five set tasks, before the administration of the Perceived Health Web Site Usability Questionnaire (PHWSUQ) for Older Adults.29 Approval for this study was granted through the university’s human research ethics committee.

Sample

A convenience sample of 13 adults, who had previously been involved in formative research that informed the design of AsthmaWise, were recruited. To be considered for inclusion, participants were required to be 55 years or older, have received a diagnosis of asthma from a health professional, have used the Internet, and be willing to be recorded during the usability testing process.

Questionnaire

A modified version of the PHWSUQ for Older Adults was used to assess participants’ overall opinion of the AsthmaWise site. The questions are categorized under three domains: satisfaction, ease of use, and usefulness. Participants were asked to indicate their responses to items using a numeric seven-point Likert scale, where 1 indicated very unsatisfied and 7 indicated very satisfied. Question 3, which asks about ease of listening to audio-information, was excluded because it was not relevant to AsthmaWise. Question 10, which asks how the Web site helped people understand their health problem(s), was reworded to ask specifically about asthma. An additional open-ended question was included at the end of the survey to allow participants to note any further comments they had about AsthmaWise. The usability and reliability of this tool have been reported elsewhere.29

Procedure

Participants were asked to register for AsthmaWise at home, before attending the usability testing session. The registration process involved answering seven questions: age, asthma diagnosis by a health professional, current asthma, postcode, town, first name, and e-mail address. Some of these questions had previously been asked during the recruitment process, over the telephone, but were included to ensure that they would be functional in the final version of AsthmaWise, where they would be used to screen for eligibility and obtain a method of contacting participants.

The testing was carried out individually in an office on the university campus. On arrival, participants read a participant information sheet and completed a consent form. They were then asked to complete a short background demographic survey using Morae. Morae worked as a pop-up that displayed over the site being tested and administered the questions and tasks. Participants were subsequently presented with five sequential tasks that they were asked to complete (Figure 1). The first four tasks were based on the AsthmaWise site, whereas task 5 required participants to view and provide feedback on a video showing how to use a metered-dose-inhaler (puffer). Finally, participants were asked to complete the PHWSUQ. Participants were encouraged to use a think-aloud process to voice their thoughts on AsthmaWise during the testing procedure; these comments were recorded using Morae.

Usability testing was undertaken during a 2-week period in December 2011. The researcher (P.B.) conducting the testing refrained from engaging in conversation or helping participants during the testing process. However, help was given if it became apparent that the participant was very frustrated or if an individual task was taking longer than 10 minutes. The researcher kept a record of her own observations during the testing process. At the completion of the test, participants were given a $30 voucher to acknowledge their time and assistance. Data from both the initial demographic survey and the PHWSUQ were analyzed using descriptive statistics; the recorded data and researcher’s notes were explored using content analysis.30

Cognitive Walk-Through and Readability Assessment

An independent health researcher worked through the three AsthmaWise modules sequentially. She was asked to check that the content was clear, and concise and used simple English; that formatting was consistent; and that the images related to the text and had alternative text (IMG ALT) attributes (ie, text describing the image that is visible when you mouse-over an image and allows people using screen readers to interact with the images).

In a parallel procedure, the text of each page was copied into Microsoft Word (Microsoft, Redmond, WA), and the Flesch-Kincaid Grade Level and Flesch Reading Ease Scale were used to establish readability using the grammar check function. The Flesch-Kincaid Grade Level analyzes the average number of syllables per word and words per sentence to produce a reading grade based on US school grades.31 We aimed for a target score of 8, which indicates that the content can be understood by an eighth grade student. Flesch Reading Ease is a 100-point scale, with documents written in plain English scoring between 60 and 70 and documents that are harder to read scoring lower.32

RESULTS

Participant Demographics

The mean age of the 13 participants was 65.8 years (range, 57–86 years), seven participants were women (54%), most participants were born in Australia (85%), and only one spoke a language other than English at home. Six participants reported a household income of below AUD $40000 per annum (46%), whereas four reported a household income higher than AUD $80 000 (31%).

Nearly half the participants were classified as having late-onset asthma, having been diagnosed with asthma after the age of 45 years (46%).33 Four participants had been diagnosed in their mid-30s (31%) and three had had asthma since childhood (23%). All participants reported having used the Internet for more than 5 years, and seven participants stated that they were very comfortable using the Internet (64%). Most participants reported using the Internet for more than 10 hours each week (69%), and almost all (85%) had previously used the Internet to find health information.

The mean (SD) time taken to complete all five tasks was 32.73 (4.95) minutes (range, 19.63–48.15 minutes). The mean times taken to complete each task were as follows: task 1, 3.47 minutes; task 2, 14.21 minutes; task 3, 8.40 minutes; task 4, 2.20 minutes; and task 5, 4.44 minutes. On eight occasions, participants took more than 10 minutes to complete a task; however, all of these occasions were a result of the participant moving off task. Because these off-task activities were relevant to other task(s), the researcher allowed them to complete what they were doing.

Registration

Three people reported that the registration process was “moderately complicated.” However, only six participants (46%) successfully created a profile, suggesting that the other participants encountered problems registering, which they did not report.

It’s a little complicated. Participant 3

The button at the bottom left hand corner of page was very difficult to find. Participant 8

Perceived Health Web Site Usability Questionnaire

The PHWSUQ for Older Adults has three domains: satisfaction, ease of use, and usefulness. The total mean converted score for the PHWSUQ was 67%, indicating that there was scope for improvement in the overall usability of AsthmaWise (Table 1). Satisfaction was the highest scoring domain (70%), whereas the usefulness domain received the lowest score (61%).

Satisfaction

There were five questions in the satisfaction domain. These asked about the ease of finding specific information, the ease of reading the information, the appearance of the site, the quality of the graphics, and the quality of the video information presented.

The first five participants all commented on the small font size; because of this feedback, the font size was increased before further usability testing occurred. This resulted in no further comments about font size being recorded. All participants encountered problems navigating from the log-on page to the homepage because of the many options available. Most did not understand what the available options were (the labels included “forum,” “blog,” and “tags”); the lack of content on these pages further compounded their confusion.

Once participants reached the homepage, most found the use of toggles (arrows that could be clicked on to display the modules available) confusing to operate. Navigation between pages in a module was undertaken using forward and back buttons on the screen. However, the button size and their placement on the page varied, which also created confusion. The time taken for pages to load was often lengthy (greater than 10 seconds) which created confusion as participants were often not sure if they had clicked the link, and became frustrated with the untimely response.

Although the site content was generally well understood, many participants commented on the amount of text and suggested that greater use of subheadings and color would facilitate the reading process. Participants also felt that there could be more graphics and that the ones that were on the site could be bigger. Although participants liked the bar chart showing asthma prevalence, the axis labels were not clear, and many had problems interpreting the meaning. The addition of extra graphics may also improve recall, as the testing process indicated that many participants did not remember visiting pages they had read just minutes earlier.

Site had too much words [sic], maybe more use of pictures or icons. Too hard to see where you were going... Participant 8

More detail is needed in certain areas, better explanations and introduction of advice brought in earlier. Participant 3

Participants offered feedback around the use of language on AsthmaWise. The meanings of many words, used every day by health professionals (jargon), were questioned. These included asthma-specific words such as reliever, preventer, and spacer, as well as generic terms such as cure. In addition, although participants liked the quick quizzes, the use of negative wording was confusing for many, for example, “Which of the following is NOT a common asthma symptom?” and “You CAN’T get asthma for the first time as an adult.” Problems with some of the titles of the modules were also identified, with participants consistently looking for information on asthma attacks under “Managing your asthma” rather than “Asthma attack.”

Participants provided feedback on content that they felt was missing from AsthmaWise. This included the lack of information provided for white-collar workers on the page called “Work,” specifically in relation to office-based triggers and stress management. Participants also suggested that a link providing emergency advice on how to get help during an asthma attack should be included on AsthmaWise.

I would like information about bronchitis (how is it different from asthma, for example). I’d also like advice on how to relieve congestion from the chest (I suffer from severe chest infections which trigger asthma attacks). Participant 2

Ease of Use

The second domain, ease of use, contained three questions: I found this Web site easy to learn; finding information requires a lot of mental effort; and overall, this Web site is easy to use. AsthmaWise was seen as easy to learn and easy to use, with modal values being 6 for both categories. The category “requires a lot of mental effort” was reverse scored and had a modal value of 4, suggesting that the site navigation could be more intuitive and was reflected in participants’ comments.

Redesign the Website—make it easier to use and understand. Participant 3

Usefulness

The domain of usefulness also had three questions: using this Web site will help me understand my asthma (modal value = 6); using this Web site will improve my knowledge of health (modal value = 3); and using this Web site will help me maintain better health habits (modal value = 4). Overall, this domain scored lowest of the three domains, despite most of participants reporting that AsthmaWise helped them understand more about their asthma.

Knowledge base was excellent as were the examples... Overall a good site. Participant 12

Cognitive Walk-Through and Readability Assessment

The reviewer was able to identify inconsistencies across the site. These included screens that did not appear in a logical order (ie, they could be reached only by using the back button), inconsistencies in the presentation of information, grammatical errors, and formatting problems. The results of the readability checks are presented in Table 2. The target readability scores were a Flesch-Kincaid Grade Level of 8 or below and a Flesch Reading Ease score between 60 and 70. It was found that 14 pages had a Flesch-Kincaid Grade Level above nine and 11 pages had a Flesch Reading Ease Score below 60. The pages with scores outside the target range were rewritten; the Flesch-Kincaid Grade Level for these pages was reduced from 10.24 ± 0.41 to 9.74 ± 0.76, whereas the Flesch Reading Ease Score increased from 54.92 ± 3.81 to 59.81 ± 2.43.

DISCUSSION

The usability testing undertaken was part of the iterative, user-centered design of AsthmaWise. The findings from this testing experience are important, as no previous published work on the suitability of Moodle as a platform for delivering health education modules for older adults, or the use of Morae, with this population was found. Although current design and usability guidelines were used to inform the development of AsthmaWise, there was a discrepancy between the recommendations and what the site developers thought would work. Allowing the site designer to witness the usability testing was a powerful process because it allowed the problems that users encountered to be seen and demonstrated that anticipated user behavior often did not occur. Although site designers often have extensive experience developing a generic site targeting whole populations, this usability testing process has demonstrated that the nuances associated within designing and developing usable sites for specific segments of the population, such as older adults, can be easily learned by watching the target population undertake tasks with the application. The testing process also allowed issues such as the slow page load time to be investigated, which resulted in image optimization being undertaken, along with other backend changes.

Whereas previous research suggests that the think-aloud process can be hard for many participants,34,35 this was not our experience; only one participant struggled with the process. This person revealed during testing that he/she had an acquired brain injury. We found that numerous variables affected task completion time, independent of the site’s usability. These factors included participants who skimmed content versus those who read every word aloud, participants who forgot their reading glasses, and those who were side tracked by recounting personal stories. This latter point has been noted previously in research with older adults.36

The use of Morae with older adults was efficient; however, many participants found the pop-up windows distracting and were confused between the program being tested (AsthmaWise in Moodle) and the program conducting the testing (Morae). In the future, this could be overcome by improving the introduction of participants to the testing setting. Although the researcher took time to explain to each participant what to expect and how long testing would take, the inclusion of a short task to complete before commencement of testing may have been effective in overcoming this problem.

Font size was the issue most commented upon; this is consistent with findings from other studies.24,34,35 Hudson26 found text size problematic with 50% of respondents older than 60 years, although further qualitative research suggested that this number was much higher. None of the participants in this study tried to resize the text by changing the browser settings or searching for other controls; this again is consistent with other research and reinforces the necessity of having font optimally sized.26,37

Hudson’s finding that older adults frequently blame themselves rather than the program or equipment they are using might explain the discrepancy between the reported ease of registering for AsthmaWise and the actual number of profiles created.26 Creating a profile involved navigating to the registration page and answering seven questions to ensure eligibility for the study and contact details. None of the participants expressed any privacy concerns around these questions.

In keeping with the findings of others, some participants questioned the credibility of the information they read.35 A review by van den Haak and van Hooijdonk34 suggests that this problem is heightened when people are looking for information for themselves as opposed to taking part in a usability test, suggesting that this issue is likely to be more problematic when people view the content in their own home. A number of strategies have been suggested to the site designer to circumvent this problem, including adding the date last updated on each page, displaying appropriate logos on each page, and providing the source(s) of information.38,39

The PHWSUQ allowed us to obtain quantitative data reflecting participants’ satisfaction and perceived ease of use and usefulness of AsthmaWise. The ease of use category received the second lowest score, which reflects the problems encountered with the navigation. Researchers planning to use PHWSUQ in the future should consider the relevance of the generic health questions when using the tool to assess disease-specific Web sites. Participant feedback indicated that few understood the concept “health hygiene,” and many felt that asking if their knowledge of health had improved was too large a target for a site focused on asthma.

Both the cognitive walk-through and the readability assessments were quick and inexpensive processes that yielded valuable information, above made by the usability testing alone; reinforcing the need for multiple testing methodologies.20 This allowed mistakes to be corrected and pages that ranked poorly to be rewritten before piloting, creating an improved final product. There is scope for the readability scores of AsthmaWise to be improved further in the future. Ideally, the cognitive walk-through would have been conducted before usability testing. However, because of time constraints, this was undertaken as a parallel process.

In an ideal world, a site designer with extensive usability knowledge would be employed. Standards for the site would be set at the start of the project and built into a quality improvement process that concludes with the final revisions to the site after piloting. In addition, the site would be reviewed before usability testing to ensure that basic usability standards are met. However, our experience shows that in a real-world setting, with a multiagency approach and tight deadlines, the inclusion of usability testing, a cognitive walk-through by an independent researcher, and assessment of readability were of significant value to the site development process.

Usability testing was relatively cheap, costing less than AUD $2000, including the purchase of the software license and incentives for participants. It is notable that although the researcher who undertook the testing is not an information technology professional, she was able to set up and run the usability testing at a level that allowed in significant usability issues to be identified and addressed, resulting in a refined end product.

Strengths and Limitations

The usability testing process undertaken had many strengths, including guidance by a usability specialist and use of dedicated usability software, which facilitated recording and analysis of the sessions. It has been shown that usability testing with five participants identifies 85% of problems40; thus, usability testing with 13 participants, coupled with the cognitive walk-through and assessment of readability, is likely to have provided a robust testing of AsthmaWise and identified most usability issues.

All of the participants had previously been involved in formative research for this project either through completion of a paper-based survey and/or participation in focus groups facilitated by the same researcher. Their continued involvement in the project may represent an emotional investment, which is possibly reflected in more positive responses. Second, only one participant was older than 75 years; in light of findings by other researchers, it is likely that older participants may find using AsthmaWise more challenging.41 Third, all the participants were long-term Web users, and Web experience has been identified as a predictor of task performance.42 This suggests that Internet-naive participants may have taken longer to complete the tasks set and may have encountered more usability problems. Finally, most respondents spoke English at home, which would make understanding the content of the site more achievable than for someone from a non–English-speaking background.

CONCLUSIONS

This usability testing process has shown the importance of involving both end users and experts during testing. The involvement of the site designer in testing was incredibly valuable and is strongly recommended. The issues identified through usability testing were addressed and significantly improved the functionality of AsthmaWise, despite testing being undertaken at minimal cost. Although the importance of usability testing with unique target populations is well accepted, this study shows that a nonexpert can effectively conduct usability testing with an older population.

Enter and submit the email address you registered with. An email with instructions to reset your password will be sent to that address.

Email:

Password Sent

Link to reset your password has been sent to specified email address.

Remember me

What does "Remember me" mean?
By checking this box, you'll stay logged in until you logout. You'll get easier access to your articles, collections,
media, and all your other content, even if you close your browser or shut down your
computer.

To protect your most sensitive data and activities (like changing your password),
we'll ask you to re-enter your password when you access these services.

What if I'm on a computer that I share with others?
If you're using a public computer or you share this computer with others, we recommend
that you uncheck the "Remember me" box.