Introduction: Individuals with mobility impairments associated with lower limb disabilities often face enormous challenges to participate in routine activities and to move around various environments. For many, the use of wheelchairs is paramount to provide mobility and social inclusion. Nevertheless, they still face a number of challenges to properly function in our society. Among the many difficulties, one in particular stands out: navigating in complex internal environments (indoors). The main objective of this work is to propose an architecture based on Mobile Augmented Reality to support the development of indoor navigation systems dedicated to wheelchair users, that is also capable of recording CAD drawings of the buildings and dealing with accessibility issues for that population.

Methods

Overall, five main functional requirements are proposed: the ability to allow for indoor navigation by means of Mobile Augmented Reality techniques; the capacity to register and configure building CAD drawings and the position of fiducial markers, points of interest and obstacles to be avoided by the wheelchair user; the capacity to find the best route for wheelchair indoor navigation, taking stairs and other obstacles into account; allow for the visualization of virtual directional arrows in the smartphone displays; and incorporate touch or voice commands to interact with the application. The architecture is proposed as a combination of four layers: User interface; Control; Service; and Infrastructure. A proof-of-concept application was developed and tests were performed with disable volunteers operating manual and electric wheelchairs.

Results

The application was implemented in Java for the Android operational system. A local database was used to store the test building CAD drawings and the position of fiducial markers and points of interest. The Android Augmented Reality library was used to implement Augmented Reality and the Blender open source library handled the basis for implementing directional navigation arrows. OpenGL ES provided support for various graphics and mathematical transformations for embedded systems, such as smartphones. Experiments were performed in an academic building with various labs, classrooms and male and female bathrooms. Two disable volunteers using wheelchairs showed no difficulties to interact with the application, either by entering touch or voice commands, and to navigate within the testing environment with the help of the navigational arrows implemented by the augmented reality modules.

Conclusion

The novel features implemented in the proposed architecture, with special emphasis on the use of Mobile Augmented Reality and the ability to identify the best routes free of potential hazards for wheelchair users, were capable of providing significant benefits for wheelchair indoor navigation when compared to current techniques described in the literature.

In spite of the technological advances of the last decades, a large percentage of people with disabilities still face enormous challenges to perform basic activities of daily living. Assistive Technology resources can be used to provide those people with various means to improve their quality of life and social inclusion (Santarosa et al., 2012). Mobility impairments are among the most common problems experienced by individuals with disabilities, affecting a wide range of the world population (Alm et al., 1998; Kouroupetroglou, 2013; Mirza et al., 2012; Sanchez et al., 2007; Tsetsos et al., 2006). Wheelchairs are commonly used assistive devices and, beyond providing mobility, allow for social inclusion and improved quality of life. Nevertheless, wheelchair users still face a number of challenges to navigate within indoor environments (Kouroupetroglou, 2013). Finding the most appropriate route to navigate through large and complex areas, such as hospitals, bus terminals, supermarkets and shopping centers is not an easy task (Tsetsos et al., 2005). The best route may refer to the shortest, the easiest trajectory, such as a path without stairs, or one which offers appropriate dropped curbs for wheelchair access. On the other hand, the best route may also be the one that passes by points of interest or still, a safer route for the users to reach their destination.

With the proliferation of wireless technology, a number of solutions for intelligent navigation have become common over recent years, as in the context of driving cars and other vehicles (Chakraborty and Hashimoto, 2010; Tsetsos et al., 2005). However, when such techniques are applied to wheelchair users, the systems fail in many aspects, such as the precision to indicate the best route to be followed or the best route for those with mobility difficulties (Alm et al., 1998; Deruwe and Wall, 2008; Park et al., 2006; Postolache et al., 2011; Stefanov et al., 2004). Limitations of this type prevent wheelchair users from safely moving around with agility. As a matter of fact, proper navigation systems for this class of users must also take into account the various impairments and physical limitations. On many occasions, the task of steering the wheelchair may be already too demanding for some users, while for others, reaching the elevator call button or manipulating a portable device may not be possible at all. Despite the various efforts, current systems still lack proper features to fully overcome those limitations (Kouroupetroglou, 2013; Mirza et al., 2012; Tsetsos et al., 2006). Furthermore, the majority of the solutions described in the literature do not provide crucial environmental information to wheelchair users, such as information concerning access ramps, counters, shelves, furniture, stairs and elevators that define the internal space of a building. Mobile augmented reality has emerged as an alternative technology that can be used to fulfill those requirements (Blum et al., 2011; Höllerer et al., 1999; Marston et al., 2006; Mulloni et al., 2012; Rovadosky et al., 2012).

Augmented Reality (AR) has the potential to provide a natural interface for large-scale environments along with the use of Ubiquitous Computing (Denning and Metcalfe, 1998). Ubiquitous Computing, more commonly known as Pervasive Computing, describes the ways in which actual technological models, based on intelligent devices (mobile phones and wireless devices), intelligent environments (embedded device systems) and intelligent interaction (between devices), relate and support computational vision for a wide range of devices used in a variety of environments and human activities (Poslad, 2009; Xing et al., 2009). While Pervasive Computing, along with mobile devices, could allow for the development of new and useful applications for daily activities, which can also be directed towards the improvement of the quality of life of wheelchair users (Ahluwalia et al., 2014; Chiara et al., 2010), Augmented Reality technologies can ideally be used for indoor navigation to provide environmental information and navigation routes (Delail et al., 2012; Hervas et al., 2014; Kotsakos et al., 2013; Matuszka et al., 2013; Rehman and Cao, 2015).

Positioning systems for outdoor environments have reached a high degree of success and can now be found commercially (Chakraborty and Hashimoto, 2010; Hansen and Thomsen, 2007; Tsetsos et al., 2006). Indoor navigation, on the other hand, still has to achieve the same level of success (Kouroupetroglou, 2013; Ran et al., 2004; Tsetsos et al., 2005; Tsetsos et al., 2006; Yohan et al., 2012). As for example, a number of navigation systems are based on GPS data. However, GPS signals can be weak or imprecise when operating indoors, resulting in insufficient resolution and can even render navigation impossible. Furthermore, most of those systems do not allow for the incorporation of new navigational data, being limited to only the information received from locations where GPS signals are present and do not incorporate data for precise localization obtained through possible feedback sources concerning the actual location (Hub, 2008).

Recent works have provided important evolutions to facilitate indoor navigation for wheelchair users (Cheein et al., 2011; De La Cruz et al., 2010; De La Cruz et al., 2011). Nevertheless, these new technologies do not yet incorporate the many advantages of Augmented Reality as a means of facilitating indoor navigation for wheelchair users, while taking into account their needs and abilities (Golfarelli et al., 2001; Kouroupetroglou, 2013; Parker and Tomitsch, 2014; Tsetsos et al., 2005). In fact, to this date, the authors have not found in the literature, systems that use Mobile Augmented Reality for indoor navigation and for visualization of information in real time, while also addressing accessibility issues. Besides, the lack of data to support indoor navigation also poses a difficult challenge, primarily when it comes to the design of a generalized navigation system for wheelchair users. In other words, a more flexible navigation system that is omnipresent and capable of incorporating user preferences while supplying a clear set of choices is necessary, in order to satisfy a more ample spectrum of society (Chakraborty and Hashimoto, 2010).

Table 1 shows a number of studies published in the literature over the past eighteen years, dealing with indoor and outdoor navigation for individuals with physical disabilities, describing the various technologies used and the main features that each system incorporates. No studies were found covering the main features we consider paramount for the successful generalization of navigation systems for wheelchair users: indoor navigation; information visualization (interactive visual representation of data to reinforce human cognition) endorsed by Augmented Reality techniques; specific navigation tools (such as those that consider dropped curbs and ramps); specific functionalities for different user profiles; capacity of switching visualizations and algorithms according to user feedback; and voice commands and gesture recognition.

The scarcity of research using mobile AR for navigating in indoor environments, associated with techniques that identify best routes by taking physically disabilities into account and yet directed to wheelchair users was, in fact, the main motivation of this research. It is believed that such inclusion will improve accessibility and facilitate navigation in various environments, such as shopping centers, hospitals and workplaces.

The authors propose an architecture to support the development of indoor navigation systems dedicated to wheelchair users that is also capable of recording CAD drawings of the buildings alongside accessibility issues. This overall strategy is also based on recording the location of special areas (such as rooms, bathrooms etc.) and fiducial markers positioned in points of interest. By means of voice or taping commands in a smartphone interface, the user can request a specific location. In turn, the system provides an optimized route, considering possible user’s physical limitations (a path without ramps and other obstacles, for instance). Also, when moving along the proposed route, the user will be able to spot fiducial markers. Positioning the smartphone’s camera in front of those markers will allow for the visualization of directional arrows in an Augmented Reality environment, facilitating continuous navigation along the correct path and to the next marker, until the desired location is reached.

Methods

We propose an architecture for the development of applications where a wheelchair user should be able to visualize routes that provide better and safer indoor navigation, avoiding rumps, stairs and other kind of obstacles. To support indoor navigation, Augmented Reality fiducial markers are employed to present direction arrows. Besides, the system should allow for the registering of different environments/locations. The configuration of places of interests (such as bathrooms and emergency exits) and the repositioning of fiducial markers should also be permitted. It is also important to take into consideration the motor and cognitive abilities of possible users (Jia et al., 2007). In other words, the navigation algorithm must consider the needs and capacities of each user. Those could influence a number of aspects associated to the design of the indoor navigation system, as well as the algorithms used for navigation and interface design. For example, the positioning of markers need to take into consideration the height of the user sat on the chair. Furthermore, the application should be designed to respond to both vocal and written (typed on a keyboard or selected from a form) instructions to define, for instance, the desired target for which the best route should be calculated and presented to the user.

In this manner, we propose the following main functional requirements (FR) for a system dedicated to indoor navigation for wheelchair users:

FR02: register and configure building CAD drawing in a smartphone display with fiducial markers, locating points of interest or obstacles to be avoided by the wheelchair;

FR03: find the best route for wheelchair indoor navigation, taking stairs and other obstacles into account;

FR04: allow for the visualization of virtual direction arrows in the smartphone display, according to AR fiducial markers and arrival point position. The placing of fiducial markers shall consider wheelchair user limitations (such as eye height level) and mobile device specification (such as the reach of the digital camera focus);

FR05: incorporate voice commands to call for specific locations (such as bathrooms and emergency exits) in case the wheelchair user presents limitations to manipulate the smartphone.

Considering those Functional Requirements, the authors propose an architecture for the design of applications dedicated to indoor navigation for wheelchair users composed of four layers, as shown in Figure 1:

Figure 1 Proposed architecture for an indoor navigation system for wheelchair users based on fiducial markers and Augmented Reality.

User interface layer: responsible for creating specific events for each client, depending on the user. This layer contains two modules - A Data Acquisition module that is responsible for interpreting the commands (touch or voice), when the user is selecting a point of arrival, and the Event Capture module that handles this feature.

Control layer: responsible for accepting or rejecting data or events that arrive from the interface layer. Two modules comprise this layer - The Data Processing module that receives and interprets the option selected by the user in Data Acquisition module (previous layer); and the Event Controls module, required when events such as voice/gesture commands are identified in the Interface Layer.

Service layer: this layer is responsible for managing all services that are requested to ensure wheelchair indoor navigation. For instance, the Service Location module is responsible to regulate the position of fiducial markers within the CAD drawing. This is important to generate arrow directions according to user´s selection of points of arrival, performed by the Routing Algorithm module. The Indoor Navigation module is responsible for controlling user navigation presenting direction arrows at the smartphone display, through Augmented Reality techniques. The User Profile Identification module identifies the type of wheelchair user (able to use both hands and voice, only voice, etc.). The Outdoor Navigation module is designed to allow for outdoor navigation with the help of GPS data. When GPS signals can no longer be identified, the Navigation Service module switches the control for the Indoor Navigation module.

Infrastructure layer: responsible for accessing information (such as CAD drawing, position of markers in this drawing etc.) stored in a database.

In order to develop an application based on this architecture, one must first consider registering the CAD drawing of the building of interest into the smartphone storage place. Next, the CAD drawing is divided in a matrix of cells. Each cell represents a 2 meters side square labeled according to its position (line, column). It is also possible to configure cells with different dimensions. Next, fiducial markers are scattered along the corridors of the building and labeled according to their position. In Figure 2, for example, fiducial marker ‘4’ is labeled as position (5, 5). Likewise, the system labels fiducial marker ‘6’ as position (5, 8). The Service Location module of the Service Layer in the proposed architecture handles the registration of cells and fiducial markers.

Figure 2 CAD drawing of the building used in the proof-of-concept experiment. In the zoomed area are shown the start and arrival points, along with the position of fiducial markers (small squares placed on the walls) and a possible path (dashed lines).

For operation, the wheelchair user can access the application and request a point of arrival by selecting it in the smartphone form by touch (handled by the Data Acquisition module) or by voice, managed by the Event Capture, in the User Interface Layer of the proposed architecture. For example, considering Figure 2, suppose the user is at the building´s main entrance (starting point) and selects to go to a room referenced by marker ‘4’. The Data Processing module of the architecture´s Control Layer process the requisition and register marker ‘4’ position as the point of arrival. At this point, the Routing Algorithm module of the Service Layer is activated and remains in a ‘listening’ mode, expecting a flag from the Indoor Navigation module saying that a marker has been found and a directional arrow is requested. After the selection, the user starts navigating in the building corridors, looking for markers.

When the user finds a marker, he/she must position the smartphone display, with its digital camera activated, in front of the marker to identify navigation directions. For example, consider the user reaches fiducial marker ‘9’, in Figure 2. When he positions the smartphone camera in front of this marker, the Routing Algorithm module proceeds with the arrow direction calculation. This is mathematically achieved by projecting the cell position of both markers in the plane of the digital camera. A normal vector of this plane goes from the camera to the fiducial marker. Then, the direction of the arrow to be presented at the user smartphone display is given from the current marker position to the point of arrival position. Once this calculation is complete, the Indoor Navigation module is requested to present the directional arrow onto the user’s smartphone display. In this example, the calculated direction arrow would be pointing to the right. However, if the fiducial marker ‘9’ was positioned at the opposite wall (see Figure 2), the calculated direction arrow would be pointing to the left. It is important to note that when the markers’ projections in the camera display plane are coincident, the direction arrow would point forward, unless the current fiducial marker is the point of arrival.

As a proof of concept, the authors developed an application for smartphone, based on the proposed architecture, to allow for indoor navigation based on the use of fiducial markers and Augmented Reality. Navigation tests were performed with two disable volunteers. Volunteer 1 used a manual wheelchair, while Volunteer 2 used a motorized/electrical wheelchair. Also, since the system relies on the smartphone’s camera to capture the images of the markers for further processing, we also tested the application with two different smartphones: Samsung™ Galaxy S2 with Android 2.3 Operational System and Samsung™ Galaxy S5 with Android 5.0 Operational System.

This study was approved by the Ethics Committee of the Federal University of Uberlândia, Brazil (CAAE 37874014.0.0000.5152). The volunteers were made aware of the objectives of the experiments and signed the free informed consent forms agreeing to participate in the trials, as well as with the use of their images in scientific publications.

Results

In order to evaluate the proposed architecture, a proof-of-concept application has been developed. The computer platform and tools used to implement this application are as follows:

Android operational system, version 2.3 (Burnette, 2009). Android is a worldwide open-source platform, based on Java programming language, for the development of mobile applications;

The building CAD drawings were stored in a local database – SQLite (Aditya and Vikash, 2014). This database is integrated with the Android platform.

AndAR (Android Augmented Reality) (Chen et al., 2016), which is a software library to create Augmented Reality applications. It contains a set of modules that are responsible for registering fiducial markers and associate virtual objects with them. When, the smartphone digital camera captures a real image, AndAR provides all mathematical transformations to position the generated direction arrows onto the fiducial marker. AndAR is based on ARToolKit that is one of the very first platforms to develop desktop AR applications.

Blender: an open-source 3D modeler that was used to create the directional arrows (Brito, 2008).

OpenGL ES (Open Graphics Library for Embedded Systems): this library is necessary because AndAR only places the virtual objects on fiducial marker. However, a set of rotations and translations are necessary to position the arrows in right navigation direction. OpenGL ES performs all these mathematical rigid transformations (Mithwick et al., 2012).

Experiments were performed in an academic building with various labs, classrooms and male and female bathrooms. For the experiment, the CAD drawing of the building (Figure 2) was inserted into the system and the position of markers (placed along possible paths) was registered in the application. It is also important to note that the height of the fiducial markers was defined previously so that it was according to the height of shoulder of the average user sitting on the wheelchair.

Navigation tests were performed with two disable volunteers using a wheelchair. The starting point was established at the entrance of the building. After starting the application, the user can select the desired environment and then the desired arrival point, as shown in Figure 3. Navigation can also be controlled using a standard voice interface of the smartphone, just by vocalizing commands and names of specific locations, such as “environment one” followed by “bathroom”. Synonyms are not handled in the current version of the system.

Figure 3 User accessing the application to (a) select the desired environment (left) and (b) the target location within, prior to starting navigation. If necessary, the user can also input commands to the application via a standard voice interface (c).

Once the final destination is selected, the interface presents the direction that the wheelchair user should follow to reach the destination via the best obstacle-free route. When passing by a marker, the user could capture an image of that marker, using the camera of the mobile device, which would prompt the presentation of the direction to be followed for the best route from that point on. Note that, the use of fiducial markers would allow for precise and swift navigation even in areas where traditional systems would not work at all, or would be imprecise or slow to respond, due to the lack of proper GPS signals, for instance. Figure 4 shows various stages of navigation using the proof-of-concept application.

Figure 4 Demonstration of the various steps associated with the use of the proof-of-concept application to guide navigation of wheelchair users in indoor environments. (a) Volunteer 1 starting navigation; (b) Volunteer 2 starting navigation; (c) User in front of the first fiducial marker with the system presenting the direction for the best obstacle-free route, from that point to the desired location; (d) User in from of another fiducial marker midway; (e) User at the arrival location.

We also tested the capacity of two different smartphones, as describe earlier, to correctly identify (focus on) those images. We have found that recognition could be easily achieved for marker-camera distances between 10 cm and 3.5 meters.

Finally, we also tested the use of voice commands to execute the same path as shown in Figure 3c. This feature was designed to help those users who have difficulties in handling the smartphone by hands. The results comply with the requirements, allowing for proper navigation.

Discussion

In this work, we propose an architecture based on Augmented Reality for the design of indoor navigation systems dedicated to wheelchair users. An algorithm that considers imminent physical difficulties (such as ramps, obstacles, etc.) has been provided to identify best routes and points of interest. In contrast to traditional approaches, the proposed technique also handles voice commands to provide support for different users. By integrating these three features (AR techniques, voice command and best route identification), advances in indoor navigation for wheelchair users have been acquired, as discussed hereafter.

In our experiments, the volunteers were able to navigate indoors and find points of interest, as stated by the first functional requirement (FR01). The user reached the point of arrival, without any additional help, by only following instructions of the system’s display. The ability to configure different locations was also achieved, as demanded by requirement FR02. Figure 2 shows the CAD drawing of the testing environment, which was registered in the portable/smartphone application, along with the position of markers scattered along the building corridors. When a best route was identified by the system (FR03), visual information was presented to conduct the user through the building. Our experiments suggest that the inclusion of fiducial markers, along the route identified by the algorithm, simplifies indoor navigation for wheelchair users. This is achieved through Augmented Reality techniques that dynamically update arrow directions, which, in turn, are displayed at the smartphone screen, to provide guides for navigation (FR04). The routing algorithm could successfully calculate the directional arrows during the experiments. Finally, by complying with requirement FR05, the system supported voice commands that were also helpful in identifying points of interests. This feature facilitates usability, especially for those who present difficulties in manual handling of smartphones.

Previous AR attempts where only designed for people with ability to walk and manipulate a smartphone touch screen (Koch et al., 2014; Neges et al., 2015; Ruta et al., 2015). However, due to their limitations, wheelchair users move around the environment in an imaginary line that is below healthy people’s eyes. That is why the location and the position of the fiducial markers were cautiously calculated to guarantee reachable information through the smartphone image capturing function. Experiments have shown that, when the users were sat on their wheelchairs, the camera was able to provide helpful information when positioned up to about 3 meters from the markers. Furthermore, the inclusion of voice commands has been shown to be a suitable option for standard users as well as those with more severe limitations (Jia at al., 2007). This is an important component and has not been observed in other works, such as those proposed by Sinyukov et al. (2014) and Takahashi and Kondo (2015), as shown in Table 1.

In addition, on many occasions traditional indoor navigation systems are fully integrated or on-board (Blum et al., 2011; Chiara et. al., 2010; Mulloni et al., 2012; Rehman and Cao, 2015; Sinyukov et al., 2014). Although this may seem as a more cohesive approach, we believe that this would also pose a number of problems when maintenance or substitution of modules is needed. In this sense, the proposed architecture has been designed in such way that the inclusion of new modules (such as outdoor navigation or gesture recognition) is allowed, without compromising the structure of the whole system, and with minimum code construction for the inclusion of new features. Such flexibility would be a major bonus when dealing with the development of indoor navigation systems for wheelchair users, since different particularities can arise from one user to another.

In summary, the authors believe that the various features implemented in the proposed architecture are capable of providing significant benefits for wheelchair user indoors navigation. In fact, none of the works showed in Table 1 addressed many aspects incorporated in our proposal, such as the need for deriving optimal routes considering the presence of rumps or stairs and the association of Augmented Reality techniques to improve the overall experience. Also, with the exception of the work developed by Sinyukov et al. (2014), where Augmented Reality techniques are handled by Google Glass™, no other work was found in the literature handling voice commands for wheel chair users. However, their solution does not handle concurrent presentation of information about the derived routes, possibly due to limitations of the Google Glass. Besides, the ability to generate directional arrows following users’ movements is not handled by their approach.

Although the proposed architecture presents a number of improvements over current strategies, the authors acknowledge the need of future research that would contribute to further improve indoors navigation. The integration of location devices known as the Bluetooth beacons (Chawathe, 2009), which can be captured by smartphones, will be considered in the near future. This technology is currently used in supermarkets, where push notifications are sent to users of smartphones and tablets with information about products closely stocked. This feature can be explored to help wheelchair users with additional information (such as safety navigation issues, close emergency exits, etc.), when a direction arrow provided by a fiducial marker is identified. And finally, we envision a future integration of this platform in fully automated wheelchairs to improved safety and indoor navigation for those with severe motor impairments, such as people with quadriplegia or those in advanced stages of neurodegenerative diseases, such as amyotrophic lateral sclerosis.

Acknowledgements

The authors thank CAPES (Coordination of Improvement of Higher Education Personnel) and FAPEMIG (Foundation of Research of the State of Minas Gerais) for the financial support that enabled this work.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.