Conclusion

The iPad is the clear winner, offering the best accessibility for blind people. The Nexus still shows significant accessibility issues and gaps which put a question mark on the day-to-day utility of Android-powered devices for blind users. Windows 8 comes in third place. The touch interaction model is still error-prone and the implementation of accessibility of the Modern interface and core Windows apps via the built-in screen reader Narrator still fees half-baked.

Interpretation of of results

The result is not really surprising and confirms experiences by blind users who tried to use Android on a day-to-day basis and gave up after a while - compare Marco Zehe's or Chris Hofstader's reports. Windows 8 tablets have a special role as they are usually sold as a hybrid / convertible with a keyboard, and are presumably mostly used in desktop mode with external screen readers.

Our tests have shown that blind users without prior experience with touch devices need some time before they can use a tablet confidently and independently. There are significant individual differences. Some users approached the tasks systematically and progressively understood the device and the interaction mode, others were more spontaneous and exploratory, which often led to discoveries but also caused frustration. Some users were quickly confident in applying the basic gestures (swiping, double-tapping), others struggled for a prolonged period, again and again running into problems such as focus loss through unintended touches. The interpretation of speech output (such as interpreting the meaning of a control) equally differed a lot between users. General results are provided in the articles Blind people using touchscreens: The issues and Interface issues when using tablets with screen readers.

With all users it was apparent that the use of touch devices became easier during testing. Still, hints by the test facilitator were often needed when users got stuck, involuntarily opened apps, or happened to call up states that were unrelated to the task at hand. Some of the user actions were easy to carry out, others proved very difficult or even impossible to accomplish with the screen reader. More about that in our post Tablet test with blind users: A comparison of common tasks.

Why test tablets with blind users?

Tablet devices are increasingly used alongside and in addition to traditional desktop workplaces – and this includes blind and low vision users. They are also taken to off-workplace locations (meetings in other rooms or at clients’ premises) for a variety of purposes: navigation and orientation, simultaneous communication, note-taking or browsing, or the capturing/scanning of flipcharts or documents.

This is why we wanted to find out to what extent tablets can be used by blind people without prior experience of touch devices. We have compared the most common tablet operating systems: iOS, Android and – currently a distant third – Windows 8. Further tests will focus on the accessibility of tablets for visually impared users.

A two-pronged approach: Initial set-up and common tasks

The user test was split in two parts: the Initial set-up test, and the common tasks test.

Initial set-up test

In the initial set-up test, we wanted to find out to what extent blind users without previous exposure to touch devices would be able to start using a tablet from scratch – from finding the button to turn on the device, to activating speech output, to registration and the use of tutorials and documentation. The test started with a minimal introduction and explanation of physical device layout, screen reader activation and the main touch input gestures. To separate the principal operability for blind users from the problems occurring in first-time use, the installation process itself was then investigated in more detail in an additional expert test.

The initial set-up of tablets was tested only for the iPad and the Nexus 7. The ThinkPad Helix comes with a keyboard which makes installation with touch a rather theoretical exercise. Also, Windows 8 is a full-blown desktop operating system and is in any case hard to compare to the mobile operating systems iOS and Android. Detailed results are provided in the article Tablet test with blind users: initial set-up comparison.

Common tasks test

For the common tasks test, we defined five generic tasks: Unlocking the device, making a new entry in the default calendar, calling up a page in a browser (URL entry), revisiting the calendar entry, and changing the default speaking rate setting. The tests showed to what extent blind users would be able to carry out these tasks without sighted help. We recorded the accessibility problems users encountered. When users got stuck, we provided help and took note of this fact. Detailed results are provided in the article Tablet test with blind users: A comparison of common tasks.

Additional technical tests

The common tasks tests revealed a number of deficiencies in the apps used. We took a closer look at the three calendar apps (iOS, Android, Windows 8) and listed the issues found there in our technical tests. Examples of issues are not sufficiently descriptive names of controls, missing output of element role or preset value, problems with focus management, or plain software bugs. Detailed results can be found here:

The test users

As test users for both parts, we selected blind users with experience in using desktop computers with screen readers in a professional context, but without prior knowledge of touch devices.

The basic tasks tests brought up a range of problems that were also partly related to the different dispositions and usage strategies of our test users. Due to the low number of test users, our results cannot claim statistical significance. Nevertheless, the tests revealed many accessibility problems and gaps, especially where input in screen reader mode was difficult or impossible, but also in terms of the naming/labelling of functions and the provision of user instructions and adequate feedback after user input.

Devices tested

Our choice of devices spanned the three most common operating systems. Our choice of actual devices / form factor was influenced by what we had at hand:

Transferability of results

We assume that most of our iOS and Android testing results can be transferred to its larger siblings, the 10 inch iPad / iPad Air and the Nexus 10 tablet (as far as these run under the same OS version). The results for the ThinkPad Helix are probably transferable to other tablets running Windows 8 / 8.1 (apart from the position of physical buttons on the device). We were unable to verify to what extent our Windows test results can be transferred to Windows RT. Here we would be grateful for hints and comments.

The Nexus test results, however, cannot simply be transferred to other tablets running Android since these often use a custom interface preinstalled by device makers such as Samsung, HTC, or Huawei. These interfaces, also called ‘skin’ – differ from Google’s default Android operating system and are often less well integrated with the built-in screen reader, TalkBack.

Since Google’s ‘Vanilla Android’ is usually most accessible compared to other skins, we therefore picked Google’s Nexus device for our tests. This choice also ensured frequent OS updates, meaning that the test was most likely to reflect recent accessibility improvements of the Android operating system, the Google apps used in tasks, and the built-in screen reader, TalkBack.

The problem of updates

The frequent updates of mobile operating systems indicate an important caveat. For a testing scheme like ours that involved the definition of a test instrument with tasks, the acquisition of test users, the scheduling of tests spread over several weeks and finally, a period of aggregating and analyzing results, mobile OS updates present a problem: they are just too frequent.

For our results to be most relevant for users considering the purchase of a mobile device, we have decided to check our tests results against the latest OS version that was available to us (which is the one buyers will get when purchasing a suitable new device – and with regard to Android, most suitable usually means a device running Vanilla Android). This meant that some earlier test results with users may needed to be qualified and partially revised.

Share

Comments

Thank you very much for this comprehensive study. Even though you do not claim statistical significance, the fact of someone taking the time and conducting the study is huge, given how little user research is performed in this field.
Thanks again!

Comment by sara lee | 04.08.2014

Great Post! I have found another related article too in the link below:

http://technology-vibe.com/blog/popular-apps-blind-iphone-users/

Thanks

Comment by Alex | 29.10.2014

Thanks SO MUCH for this excellent study and article! My wife is legally blind and also owner of an Android cellphone, a Windows laptop (with Jaws ScreenReader) and and iPad mini and agrees with you - Android is still behind the others. We were hesitating whether to change the iPad for a Windows tablet but you've answered our question. Thanks again.

Comment by Merc | 01.11.2014

I would like to thank you for this article. Lately my grandfather started to loose his vision. I am searching any technologies that could help him stay active. One of the ideas is to purchase big screen tablet with the best accessibility features available. You made the research much easier :)

Comment by bea | 28.01.2015

Thanks for the tests. I am not blind--yet and am in the market for a mobile device. The article is really helpful.