We've all been there: just bought a flat-packed piece of furniture and the 2D instructions do not quite match the 3D version of bits and pieces of wood in front of you. In our case we can simply phone up a friend or even the shop from which the furniture was bought to get help. But what if you're an astronaut on the International Space Station or even further away as will be the case in future missions to the Moon or Mars?

Astronauts face operational problems almost on a daily basis and these difficulties make putting together your flat-packed wardrobe look like peanuts. Although they are of course trained extensively for almost any scenario, including the use of virtual reality as an enhancing technique, quite often the decisions they need to make would become much easier if they had some sort of help (especially in cases where contact with Earth is not possible or severely time-delayed).

ESA has been studying Augmented Reality (AR) and Virtual Reality (VR) as a possible aid for astronauts for several years now. A number of studies and projects have already shown the advantages of AR/VR for instance in the field of medical procedures.

In any six-membered crew it can be expected that one member has been trained in medical procedures. But on a mission to Mars or even on the Space Station it may eventually be needed to carry out an emergency medical procedure for which this crew member may not have the necessary knowledge or, worse, what happens if the doctor becomes the patient?

Several ESA astronauts have already been involved in testing AR/VR equipment, such as WEAR (Wearable augmented reality), EdcAR (Engineering Data in Cross-Platform AR) and 3DViT. WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as step-by-step instructions to guide them through lengthy procedures.

The EdcAR project developed a new AR system that considerably reduces the number of mistakes in the performance of tasks while speeding up and clarifying operations. As such the system could be used in future operations at the international space station, in particular for in-flight Maintenance tasks. The first practical tests carried out at ESA's Astronaut Centre (EAC) were successful.

Andreas Mogensen also used VR techniques during his iriss mission to perform maintenance tasks on the space station; the 3D visual training tool (3DViT) was successfully put through its paces. 3DViT provides efficient training on the spot where needed and can be expanded with a number of routine tasks.

ESA's Advanced Concepts Team (ACT) also researches the use of VR and AR to complement future space missions. In particular, the team is investigating the idea of using a VR headset such as the Oculus Rift to remotely inspect the space station via images streamed from the stereo-vision camera of the SPHERE drones currently on-board the station. However, for complicated operational procedures VR and AR systems have so far not been studied extensively. Especially in emergency situations, where every second counts, AR technology will be indispensable.

A new GSP study called "Augmented reality and Virtual reality techniques to enhance the execution of operational procedures by astronauts and spacecraft operators" has been started to assess whether and in which form AR and VR techniques can aid the performance of astronauts and spacecraft operators. The main objective is to try and root out human error in stress and emergency situations where often the main cause of problems is the discrepancy between the 2D and textual description of such procedure and the 3D reality.

Related Studies:

Related Documents:

We've all been there: just bought a flat-packed piece of furniture and the 2D instructions do not quite match the 3D version of bits and pieces of wood in front of you. In our case we can simply phone up a friend or even the shop from which the furniture was bought to get help. But what if you're an astronaut on the International Space Station or even further away as will be the case in future missions to the Moon or Mars?

Astronauts face operational problems almost on a daily basis and these difficulties make putting together your flat-packed wardrobe look like peanuts. Although they are of course trained extensively for almost any scenario, including the use of virtual reality as an enhancing technique, quite often the decisions they need to make would become much easier if they had some sort of help (especially in cases where contact with Earth is not possible or severely time-delayed).

ESA has been studying Augmented Reality (AR) and Virtual Reality (VR) as a possible aid for astronauts for several years now. A number of studies and projects have already shown the advantages of AR/VR for instance in the field of medical procedures.

In any six-membered crew it can be expected that one member has been trained in medical procedures. But on a mission to Mars or even on the Space Station it may eventually be needed to carry out an emergency medical procedure for which this crew member may not have the necessary knowledge or, worse, what happens if the doctor becomes the patient?

Several ESA astronauts have already been involved in testing AR/VR equipment, such as WEAR (Wearable augmented reality), EdcAR (Engineering Data in Cross-Platform AR) and 3DViT. WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as step-by-step instructions to guide them through lengthy procedures.

The EdcAR project developed a new AR system that considerably reduces the number of mistakes in the performance of tasks while speeding up and clarifying operations. As such the system could be used in future operations at the international space station, in particular for in-flight Maintenance tasks. The first practical tests carried out at ESA's Astronaut Centre (EAC) were successful.

Andreas Mogensen also used VR techniques during his iriss mission to perform maintenance tasks on the space station; the 3D visual training tool (3DViT) was successfully put through its paces. 3DViT provides efficient training on the spot where needed and can be expanded with a number of routine tasks.

ESA's Advanced Concepts Team (ACT) also researches the use of VR and AR to complement future space missions. In particular, the team is investigating the idea of using a VR headset such as the Oculus Rift to remotely inspect the space station via images streamed from the stereo-vision camera of the SPHERE drones currently on-board the station. However, for complicated operational procedures VR and AR systems have so far not been studied extensively. Especially in emergency situations, where every second counts, AR technology will be indispensable.

A new GSP study called "Augmented reality and Virtual reality techniques to enhance the execution of operational procedures by astronauts and spacecraft operators" has been started to assess whether and in which form AR and VR techniques can aid the performance of astronauts and spacecraft operators. The main objective is to try and root out human error in stress and emergency situations where often the main cause of problems is the discrepancy between the 2D and textual description of such procedure and the 3D reality.

As ESA’s ɸ-week draws to a close, it’s worth remembering that the science and technology involved in observing our planet from space isn’t entirely for the preserve of experts. With a growing interest in the environment and the climate, there are plenty of opportunities for the general public to take part.

Access to space was in the spotlight at this week's Φ event which followed an ESA-hosted workshop on Europe’s emerging microlaunch services held in Paris, France for industry, investors and institutions.

In mid-December, twin discs will begin glowing blue on the underside of a minibus-sized spacecraft in deep space. At that moment Europe and Japan’s BepiColombo mission will have just come a crucial step closer to Mercury.

It may not be your actual Superman, but participants to ESA’s ɸ-week are certainly embracing some ‘superhero’ ideas for the future of Earth observation, including high-flying platforms – something between a satellite and an aircraft.

As ESA’s ɸ-week draws to a close, it’s worth remembering that the science and technology involved in observing our planet from space isn’t entirely for the preserve of experts. With a growing interest in the environment and the climate, there are plenty of opportunities for the general public to take part.

Access to space was in the spotlight at this week's Φ event which followed an ESA-hosted workshop on Europe’s emerging microlaunch services held in Paris, France for industry, investors and institutions.

In mid-December, twin discs will begin glowing blue on the underside of a minibus-sized spacecraft in deep space. At that moment Europe and Japan’s BepiColombo mission will have just come a crucial step closer to Mercury.

It may not be your actual Superman, but participants to ESA’s ɸ-week are certainly embracing some ‘superhero’ ideas for the future of Earth observation, including high-flying platforms – something between a satellite and an aircraft.

Most-viewed images

Most-viewed videos

It is time to prepare to listen to space. To celebrate the launch of the European Student Earth Orbiter (ESEO), ESA’s Education Office challenges the amateur radio community to listen out to be first to hear the new spacecraft orbiting Earth.

ESA Education and the Raspberry Pi Foundation are delighted to announce that Phase 2 of the European Astro Pi Challenge: Mission Space Lab has begun. During Phase 1, we received a record-breaking 471 entries from 24 countries! Now, the 365 selected teams will have the chance to write computer programs for the scientific experiments they want to send to the Astro Pi computers aboard the International Space Station (ISS)!

Last week enthusiastic future engineers and scientists had the chance to participate in the ESA Academy’s inaugural Space Systems Engineering Training Course at the Academy’s Training and Learning Facility, ESEC-Galaxia, Belgium.

This pilot cycle brought together the 30 University students, representing 16 ESA Member States, with ESA current and retired staff, who, armed with a wealth of experience in Space Systems Engineering and Project Management, set about the task of divulging the rich and varied work of a System Engineer at ESA.