Blog

VR through ‘Charter-based’ exploratory testing

Virtual Reality applications – both those that are built for VR and those that only have VR features enabled – are a type of software that can be characterized by a high degree of interactivity, with many components and systems. From the ‘simple’ type of VR experience – such as consuming various types of media in VR – to complex VR games, the user is constantly choosing and receiving responses to his actions.

The best way to test these highly interactive products is through exploratory testing, focusing on the user experience, integration and system tests. The more systems and components a VR app/game uses, the higher the testing complexity becomes. As such, because not everything within the application can be 100% tested, the testing plan should focus on the ‘critical path’ and the interactions affecting it, prioritizing issues based on the risk to and impact on the user base.

But what about all the other components that are not on the critical path, but are part of user and systems interactions and have a major impact on user experience and their enjoyment of a VR experience? A smart and reliable way of testing this combination of systems and user experiences is through ‘session-based’ exploratory testing called ‘charter exploratory testing’ or ‘testing tours’. (See ‘James A. Whittaker – Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design’ for a more detailed explanation.) We recommend that charter-based exploratory testing is the best way to increase test coverage when testing a VR application.

How ‘charter exploratory testing’ works

This type of exploratory testing focuses on predetermined durations for testing, called ‘sessions’ or ‘tours’ depending on the Test Manager’s preference. Each session, which can be Short (30-60 minutes), Normal (60-90 minutes) or Long (90-120 minutes), has a specific goal or agenda (charter) that guides the testing within a particular session. The goal can be chosen by the tester based on a specification, ideas or heuristics, on a secondary system, or on analysing and acting on results from previous test sessions.

Examples of these types of goals could be:

considering only ‘pacifist’ (no destruction, no interaction, avoidance of the objectives) or ‘destructive’ (abusing the systems, destroying everything in your way, breaking objectives) user behaviours in testing different levels or systems;

backtracking or fast forwarding to completed/known objectives;

abusing/creatively using systems in areas where they are not supposed to be used.

During the session, new testing opportunities will arise, and they should be followed and noted down. But the tester has to try to return or at least continue with the chosen goal. Of course, the newly found opportunity can/should become the goal of a new test session later on. It is also important that the tester documents his actions during the session, and the act of collecting ‘bug’ data (screenshots, movies, logs etc.) should not interrupt the testing time and flow.

After the session is done, a debrief or even a test report should be written containing all of the important information regarding the test, such as the goal, duration, issues found and reported, ‘opportunity’ issues noticed, and observations.

How this works when testing VR experiences

Session-based testing is a documented approach to testing a system’s functionality within an architecture that is largely unknown or very complex. In the case of VR software, due to the major focus on ‘interactivity’, there will be situations when formal product’s requirements are incomplete, or rapidly changing the interaction between different systems (gameplay) with great impact on the ‘user experience’.

VR software is based on a higher level of immersion and interactivity. By using ‘charter-based’ exploratory testing, we can test and analyse the different connections between the user’s actions and the program’s response, or the different interactions that will be triggered between the game’s systems.

All of these will enhance the ‘emerging gameplay, experience and stories’. Yes, the developers probably never thought of or designed this originally when creating their VR concept. But when you build a huge castle out of the kitchenware that was only supposed to be ‘interactive’, and it works and reacts to physics and water so much that it almost mimics real life, then you really think: “hey, that’s cool!”