The Office of Naval Research takes great pride and interest in their sailors in many ways such as funding a massive training movement called Capable Manpower. UCF RETRO Lab has been subcontracted by Raytheon BBN Technologies to test, evaluate, and provide quantified research findings about the effectiveness of game-based training. RETRO was involved with both research and development as part of this multi-year effort.

Development

As a part of the Virtual Environments for Ship and Shore Experiential Learning (VESSEL) project, BBN and Alion developed the Damage Control Trainer (DCT), which is an award-winning serious game used to train US Navy recruits on communication protocols, shipboard navigation, and damage control procedures. This game is currently implemented at Recruit Training Command. The DCT is a story-driven training experience. The narrative is constructed using cinematics, in-game missions, and interactions with non-player characters (NPCs). Our 3D artist Julian Orrego created a number of cinematics for the start of each chapter to bridge in-game events between missions.

Opening Cinematic for the DCT

In addition to the cinematics, RETRO also developed Flash minigames to augment training, including Insignia Searcher and the Bullseye Trainer. Insignia Searcher is a quick game that helps recruits to recognize the different US Navy officer rank insignia, which provides information about rank and officer specialties. The Bullseye Trainer is used to help recruits understand the shipboard navigation system. Compartments in US Navy ships all have specific identifiers that are used to describe both the location and purpose of that space, sometimes known as bullseyes. Each ship conforms to the same system and the Bullseye Trainer is a drill exercise to help recruits memorize and understand the different parts of the bullseye as it relates to the ship itself.

Note: Games launch in a new window.

Modding games is a cheap and incredibly effective way to create serious games using existing game engines. Several commercial games are packaged with game-editing software and active modding communities often form. When discussing ways to improve our shipboard navigation training, we decided to create a 3D virtual training environment by modding Bethesda’s Fallout 3 (2008). One of the in-game locations is Rivet City, an aircraft carrier modeled after the USS Oriskany. Our programmer Daniel Brown modded Rivet City to have more levels and added to every room the appropriate compartment identifier. He then built a quest where players had to interact with Damage Control Central and check different compartments in the ship. This became both a testbed for shipboard navigation as well as a training environment.

Modded Rivet City Navigation Quest

RETRO Development Team

Lab Directors

Drs. Clint Bowers & Jan Cannon-Bowers

3D Artist for DCT Cinematics

Julian Orrego

IS Production Manger

Rachel Joyce

IS Designer

Peter Smith

IS Programmers

Jorge Salazar, Brian Stabile

IS Artists

Brent Hermanski, Jorge Salazar

Bullseye Production Manager

Rachel Joyce

Bullseye Designer

Lucas Blair

Bullseye Programmer

Stephen Nedley

Bullseye Artist

Julian Orrego

Rivet City Mod Manager

Katelyn Procci

Rivet City Mod Designers

Daniel Brown, Katelyn Procci

Rivet City Mod Programmer

Daniel Brown

Rivet City Mod Artist

Daniel Brown

Research

RETRO was responsible for conducting validation studies on the DCT, usability studies on the mission editing software VITAL, and a number of other studies examining different aspects of games and how they could be used to improve training.

Usability Studies

To support development efforts, several usability studies were undertaken. We conducted a usability study on the DCT in a sample of US Navy recruits. We found that brief, extremely inexpensive training (i.e., supplementary paper-based information about the compartment marking system) in conjunction with the game improved performance without making any edits to the software.

We also conducted usability studies for both Insignia Searcher and the Bullseye Trainer. Usability issues were identified and corrected. Additionally, we conducted a usability study on the VITAL Authoring Tool for the DCT. The tool was found to be useable, even by those with limited experience with computers. Suggestions for potential modifications were generated.

Validation Efforts

Several validation studies were also conducted on the DCT, Insignia Searcher, and the Bullseye Trainer.

To validate the Damage Control Trainer game, we initially conducted a pilot study using US Navy recruits in November 2008 which was followed-up with a complete study in April 2009. We also conducted a team validation study in late 2009. We found that those in the game condition consistently outperformed those in the no-game condition with respect to time to navigate to compartments and errors made. Decision making errors were reduced by 50%, communication errors were reduced by up to 80%, and situational awareness and navigation skills were improved by 50%.

In early 2008, we conducted a validation study on Insignia Searcher comparing groups of undergraduate and JROTC students who either played the game or viewed a PowerPoint. We found the game to be as effective as traditional methods and that it was more emotionally engaging.

In 2010, we conducted a validation study on the Bullseye Trainer. We were unable to improve our participants’ abilities to find bullseyes within the DCT with our 2D training intervention. We did find that gaming familiarity was the strongest predictor of performance.

Experiments

In order to further the science of serious games and articulate a set of guidelines for training game development, we conducted several experiments based on several different theoretical foundations to uncover why and how serious games are so effective. These studies included:

Brain lateralization as a predictor of team performance

The effectiveness of written narratives as pre-experiences for improving game-based training

The role of opening cinematics in improving game-based training

The use of avatars for training interpersonal skills

Engagement and flow as a construct and the effect these states have on training effectiveness

Achievements

Worked examples

Using synthetic voices for training communication skills

Avatars and personalization

Predictors of positive game-based training outcomes

Gaming preferences

For more details about these results, please check out our Publications.