The evaluation of mobile, multi-modal, accessible applications via traditional observation techniques (e.g. video recording) is a challenge because of mainly three related problems: the context problem (i.e. users behave different in various contexts), the invisibility of multi-modal feedback (i.e. vibration patterns cannot be seen), and the context-sensitivity of these applications (i.e. is a user scanning or not).

As part of HaptiMap we further investigated and understood these problems and are developing our own observation framework: the Virtual Observer. This framework is based on the well-known logging observation method. Logging describes the process of recording arbitrary information, which is subsequently saved into a file. There rather exist ultimate logging frameworks. Moreover, the to-be-logged events are identified and implemented by the developers themselves. The Virtual Observer is designed for location-based applications and therefore comes with a selection of logged events (e.g. latitude, longitude, speed, etc.). We also investigated more specific events, which are of interest for designers and experimenters of location-based applications, e.g. disorientation, navigation errors, and how the device is held in the user’s hand.

For a developers/experimenters convenience we are also developing a tool, named ContextPlayer, which is capable to display and interact with the recorded context information in a convenient way. Beside our logged values the ContextPlayer offers support for displaying images from the Microsoft SenseCam, a tiny camera worn around the neck.

With respect to the three identified problems for the observation of mobile applications, the Virtual Observer and ContextPlayer help as follows. To address the context problem the Virtual Observer provides lots of additional context information. Thus, the subjective context perception through the experimenter can be replaced by the objective sensor measures. That helps to make the separation of cause and effect more accurate and reliable. Further, the Virtual Observer also addresses the invisibility of multi-modal feedback and context-sensitive application problems. The Virtual Observer records whether e.g. tactile feedback is enabled and if the device is held parallel to the ground or not (exemplary designed for the PocketNavigator, one of the HaptiMap demonstrators). Thus, it can be accurately determined if either the scanning mode or pocket mode is active. In addition, unfiltered compass values are logged. Together with the exact user location and the next waypoint, the exact multimodal feedback (e.g. tactile feedback) displayed to the user in this situation can be reconstructed.

A native C implementation of the Virtual Observer logging framework is part of the HaptiMap toolkit mantle. An Android wrapper can be found in the toolkit crust. A more sophisticated Java/Android-only version of the Virtual Observer will be added to the toolkit soon. At this point also the ContextPlayer will be released as a side-project to the actual toolkit.

When creating evaluation tasks, one should take care to include some contexts that are not optimal – because it will never be optimal in the real usage context. A list of possible conditions that might limit the use of the respective application is given below:

Optimal conditions:

Optimal lighting and the user can easily look at the screen

Quiet environment without sound restrictions

The user holds the device in the hands (no gloves or similar)

The user has nothing else in the hands and can use both two handed and one handed grips

Noisy environment (e.g. in a crowd, by a busy street, at a train station, at a festival or fair, etc.) or an environment where sounds are not suitable (e.g. in a meeting, concert, theatre)

Situation which limits the user’s ability to touch the device - cold hands, using gloves (cold weather) or keeping the device in a pocket or a bag

Situation which limits the user’s ability to manipulate the device like having to hold something else in one or both hands (e.g. white cane, umbrella, bag, take away coffee, ice cream, pram, child etc.)

Heuristic evaluation

A Heuristic evaluation of the UI is part of the design process. You should do a heuristic evaluation, using an appropriate to the end-user set of rules, before field-testing. To do a heuristic evaluation, the tester should know about the application. However, it would be better if the tester is not the developer of the application him or herself.

Universal Design Principles

Multiple perspectives and techniques can be used to conduct the heuristic evaluation. One of them is to evaluate your application following the Universal Design Principles.

The principle 7 concerning "size and space for approach and use" might need adaptation for a mobile context. Instead of ability to reach the device, more relevant might be to check if the grip is stable enough to for use while moving. Instead of providing adequate space for assistive devices, checking if it can be used with busy hands (i.e. using a cane or with a wheelchair) and the body is moving.

Diverse conditions of use

Go back to the list of conditions in the introduction. For each of those, please state if your application will or will not work. You might need to specify if this answer is conditional to specific events or contexts.

About Accessibility Standards

In the field of accessibility, national, European and international standards play an important role. These standards form a commonly agreed base of knowledge helping to understand the consequences of the reduced abilities and the characteristics of older persons and persons with disabilities, their needs and the resulting recommendations for products and services. Moreover, these standards provide ergonomic data for this specific user group as well as design methods and evaluation criteria. Finally, these standards promote compatibility of products and services, systems, environments and facilities and support the interoperability of mainstream solutions with assistive technology.

Real standards are developed by national, European or international standardisation organisations in accordance to a well defined process. Besides these official standards there are a lot of de-facto-standards which are developed and maintained by interest groups (e.g. the World Wide Web Consortium), industry consortia, single companies, etc. Also public authorities like the US Access Board provide rules for accessibility, which are called standards, but which are - from a formal point of view - no official standards. Another example is the "Technical Specification for Interoperability: Persons with reduce Mobility" for Trans European Railways developed by the European Railway Association (ERA).

Types of Accessibility Standards

ISO/IEC Guide 71:2001 = CEN/CLC Guide 6:2002, "Guidelines for standards developers to address the needs of older persons and persons with disabilities" covers general qualitative requirements whereas ISO/TR 22411:2008, "Ergonomics data and guidelines for the application of ISO/IEC Guide 71 to products and services to address the needs of older persons and persons with disabilities" complements quantitative requirements following the same structure. This pair of documents can be seen as an answer to the "UN Convention on the Rights of Persons with Disabilities" article 9 (point 2.a). These top level documents are also bridging the gap between ergonomic approaches like ISO/FDIS 26800 "Ergonomics - General approach, principles and concepts" and sector or product or technology specific accessibility standards like ISO 9241-20:2008 "Ergonomics of human-system interaction Part 20: Accessibility guidelines for information/communication technology (ICT) equipment and services" or ISO 9241-171:2008 "Ergonomics of human-system interaction Part 171: Guidance on software accessibility" or the Web Content Accessibility Guidelines (WCAG) 2.0 or "PDF Accessibility guidelines" and many other documents (see Figure 5.1).

Further general information on user needs, a standards inventory, and guidance on user needs mapping are available in ISO/IEC TR 29138:2009, "Information Technology - Accessibility considerations for people with disabilities", consisting of three parts:

ISO/IEC TR 29138-1: "User needs summary"

ISO/IEC TR 29138-2: "Standards inventory"

ISO/IEC TR 29138-3: "Guidance on user needs mapping"

The HaptiMap project has developed a long list on accessibility related standards, guidelines, documents, training materials and background information.

Regulation and the Role of Accessibility Standards

As soon as accessibility is subject to regulation, as it is in most EU countries, standards are required to assess the fulfillment of regulated requirements. Where no standards are available which are officially published by a national standards body, regulators tend to "invent" their own accessibility rules. This bears the danger of reinventing the wheel, of creating incompatible and proprietary national approaches and of weakening well established (non standard) rules with the consequence of the fragmentation of the ICT market.

Prominent examples are the proposed new US Section 508 standards including rules for Web Accessibility or the German national web accessibility rules which are both mainly derived from the Web Content Accessibility Guidelines (WCAG) 2.0 published by the World Wide Web Consortium in 2008, but differ by adding new rules, dropping others and setting other priorities. This, in principle, requires as a consequence adaption in accessibility assessment and evaluation tools and methods. This is the reason why in European countries some twenty web accessibility schemes, methods and labels are known.

"Buy Accessible" is an approach, implemented in the US by the US "Rehabilitation Act, Section 508" (1998) and the respective US Access Board 508 standards (2008, under revision) first in the world. The idea is simple: All public authorities are forced to buy accessible ICT products and services only. The buying power of public authorities has an important impact on the accessibility of ICT products and services offered by industry.

Today all ICT companies, active on a global market, are familiar with the self declaration of the accessibility features of their products and services by using the "Voluntary Product Accessibility Template" (VPAT).

The European Approach to Improved Accessibility

The inclusion of the criterion "Accessibility" in public procurement of ICT products and services in Europe is the goal of the EU Mandate 376 "Standardisation mandate to CEN, CENELEC and ETSI in support of European accessibility requirements for public procurement of products and services in the ICT domain". Mandate 420 is a simular activity for the built environment. Both mandates are part of the "European Disability Strategy 2010-2020: A Renewed Commitment to a Barrier-Free Europe ". Phase II of mandate 376 started in Januar 2011. The three European Standardisation Organisations are currently working on the required accessibility standard, the modification of the European public procurement rules and criteria, an online toolkit and training materials for procuring authorities. Information on the status of mandate 376 can be found at http://www.mandate376.eu.

In August 2011 the following 6 draft documents were published for public comments:D1: Draft EN 301549 "European accessibility requirements for public procurement of ICT products and services", specifying the functional accessibility requirements applicable to ICT products and services D2: Draft TR 101550 "Documents relevant to European accessibility requirements for public procurement of products and services", listing the standards and specifications used for the requirements and tests in the EN D3: Draft TR 101551 "Guidelines on accessibility award criteria for ICT products and services", providing guidelines for contract award criteria D4: Draft TR "Guidance for the application of conformity assessment to European accessibility requirements for public procurement of ICT products and services" D5: "Online Procurement Toolkit for accessible ICT products and services" D6: "Additional guidance and support material for the procurement of accessible ICT products and services"

After an extensive discussion with all stakeholders the final documents are expected for October 2013.

The implementation of accessibility as a criterion in European public procurement by using one single set of European functional accessibility requirements will help to improve accessibility across Europe by a harmonized approach. With the national implementation this harmonized approach will substitute the diverse national rules and will avoid a fragmentation of the ICT market which will be beneficial for all parties: public authorities buying ICT products, companies selling the ICT products and, last but not least, end users using the products.

Ongoing International Standardization Work for Accessibility

Both documents ISO/IEC Guide 71 and ISO TR22411 are under revision. New versions are expected end 2012.

Siemens ACC is actively contributing to the ISO work.

ISO 9241-20, ISO 9241-171, and ISO/IEC TR 29138 are also under revision.

There are currently numerous accessibility related New Work Items on the way at ISO, ITU-T, and IEC. For example: IEC TC100 started the development of the document "Text-to-Speech Functionality for Television - General requirements" in May 2011 and ISO/TC 173/WG8 "Assistive products for persons with disability / Provisions and means for orientation of visually impaired persons in pedestrian areas" is working on wayfinding for blind persons.