Immersive Training Hasn’t Reached the Holodeck

Jun. 12, 2012 - 04:01PM
|

The U.S. Army is about to get a new standard in immersive virtual training. The Dismounted Soldier Training System allows a squad of nine to practice everything from room clearing and IED disposal to checkpoint and combat operations — all in a virtual world. Each soldier gets a head-mounted display, mock weapon, sensor system and wearable computer pack.

“The feedback [from soldiers] has been, ‘This is the kind of training that we needed,’” said Floyd West, director of strategic programs for Intelligent Decisions, developer of DSTS, which incorporates Quantum3D’s ExpeditonDI virtual environment training system. “It’s filling a gap in the training because the dismounted soldiers don’t have this kind of training.”

While DSTS was supposed to roll out to Fort Benning, Ga., for tests in January, it was delayed for a few weeks to get the newest upgrades in technology, such as the Intel quad-core processor and the latest Nvidia graphics processing units.

West still expects the systems to roll out in late June once user assessment is complete and the Army leadership green-lights the product. The Army is on contract to buy 102 suites and there are plans to field DSTS to more than 30 sites this year, including Forts Bliss and Hood in Texas and Joint Base Lewis-McChord, Wash.

Yet for all its leap-ahead technology, DSTS also shows how far sim makers have to go to truly replicate reality.

“Everyone wants the holodeck from ‘Star Trek,’ but they haven’t figured out how you fool the mind,” said John Foster, who oversees DSTS for the Army’s Program Executive Office for Simulation, Training and Instrumentation (PEO STRI).

In DSTS, soldiers can explore terrain, interact with civilians and enemy combatants, coordinate tactics and train much like they fight in the real world. Sensors on the body track whether the soldier is standing, kneeling or prone — and reflect it on his avatar in the virtual world viewed through the head-mounted displays.

Additional sensors in gloves track hand movements and can register if a soldier grabs a grenade or flare gun from his vest. DSTS will, many claim, change the way infantry soldiers can prepare for overseas deployments.

With all that in mind, however, researchers say better technology could add even more realism to the training.

It’s in the Way You Move

Currently, soldiers are connected to the virtual network wirelessly, allowing them to walk freely around the training room. However, to navigate in the virtual world, they still rely on a small joystick on their mock rifles.

West says that it hasn’t been a problem for soldiers trying the system.

“That was a question the general had for the squad leader there [at Fort Benning]. When asked how long it took him to get used to it, he said, ‘Four or five minutes, sir, and I’m not a gamer,’” West said.

As technology advances, however, the Army still seems likely to replace this component.

“We’re really looking at other technology for how we can make it more natural locomotion, so the soldier doesn’t have to use any sort of joystick or pressure pad,” said PEO STRI’s Foster. “It would replicate what he does in real life. But all those systems are very immature and very expensive, and take up a lot of space.”

One alternative is a 360-degree treadmill that would allow a soldier to walk virtually in any direction without moving in the training room. However, Foster estimates the cost of nine treadmills at $250,000 — half of what it costs to purchase an entire DSTS suite right now. In a similar vein, there are also “hamster balls” that allow the soldier to walk in place in a self-contained sphere.

Another option is using pressure points in shoes or on the floor, but this is still a few steps away from natural locomotion.

“The problem is how you distinguish between someone walking versus jogging versus running,” Foster said. “You still have those distinctions to deal with, and not everyone runs at the same pace.”

There also is the ability to use camera-based systems to track soldiers as they move through the training exercise. However, these training areas are then limited to the size of the building. Currently, DSTS can simulate an area far larger than its required 1,600 square feet, which houses a squad of soldiers, a staff control station, an after-action review space and storage.

Researchers at the Institute for Creative Technologies (ICT) at the University of Southern California are working on ways to create a path in the virtual world that differs from the path taken in the real world, opening up a world of space inside a confined training room.

“Through software techniques and trickery, they can manipulate the environment in the head-mounted display so that as you are turning and walking, you are seeing yourself go straight — but in reality, you are walking in circles in the room,” said John Hart, the program manager for ICT at the Army Research Laboratory. “They can trick your mind into thinking you went somewhere you didn’t.”

Making Faces

The head-mounted display that transports soldiers into the virtual world clips onto a soldier’s helmet where night-vision goggles normally sit and provides 60 degrees of high-resolution vision. Participants can see one another’s avatars and talk through their headsets, but there is still nothing in place to capture their facial expressions and render them on their avatars.

“As far as semi-automated forces, be it civilians or enemy combatants, the game engines can support those kinds of facial expressions and body movements,” West said. “But as far as soldier-to-soldier right now, it’s not a piece of technology that’s in there.”

While it is possible to capture facial expressions, it would require a set of cameras or some additional kind of technology. Without it, however, soldiers miss out on valuable nonverbal communications that humans exchange regularly.

Foster also said the facial expressions for civilian and enemy avatars could improve. Part of the problem stems from the fact that different facial expressions can have different meanings in the U.S., Europe and the Middle East.

“We’re trying to build a database of expressions for different cultures,” Foster said. “It’s a very intensive task to collect all this data to be able to portray it right. The one thing we don’t want to do is give any kind of negative training to the soldier that would possibly get him hurt or, heaven forbid, killed.”

Avatars can be problematic, particularly those controlled through game engine interfaces. Foster said that because game engines — particularly commercial, off-the-shelf ones — were designed to control individual avatars, they don’t always have good ways of giving commands to computer-generated units. Commanders want a cognitive, organized enemy that can react to soldiers’ actions, functioning in an expansive world that soldiers can explore.

“That’s the biggest challenge we have right now: providing a good environment that can replicate what the soldiers encounter in the real world,” Foster said.

The Fifth Sense

In a world that is created almost entirely in the individual soldier’s mind, stimulating as many senses as possible is key — and a prime one of these is touch.

“The military has done a great job in focusing on the visual and audio sensory channels,” Hart said. “What we’re trying to do is look at some of the other elements, whether it be olfactory or haptic, the physical touch.”

There are many ways to simulate the physical feeling, though few have been incorporated into the current model of DSTS. Hart envisions an augmented reality that combines real-world items with virtual sight — meaning a soldier could put his hands on a real desk in the real world and see a virtual desk beneath his virtual hands in his head-mounted display. Other groups are working on haptic gloves that provide pressure or vibration depending on soldier actions in the virtual world, such as banging on a door.

Intelligent Decisions plans to integrate further haptic feedback in newer releases of the system — particularly in response to soldiers getting shot. This could incorporate shock or vibration technology, something that the military has been recently interested in adapting for counter-improvised explosive device training scenarios.

“Our training is built to save lives,” West said. “As part of that, in making it more immersive, we want some kind of stimulus to make sure the soldier gets the appropriate negative feedback.”

As a result of the locomotion problem, it also is currently difficult for DSTS to replicate the fatigue that comes with having to physically walk and patrol an area. Simulation systems tend toward draining the mental rather than the physical capacity. Still, virtual reality offers many advantages and scenarios that can’t happen in the real world.

“You can’t drop shells around a soldier in live training,” Foster said. “They don’t like that.”

The Ultimate in Immersive

Blending real life into immersive training is on the horizon. For years, the Army has toyed with the idea of creating personal avatars for each soldier that would change based on their real-world performance, according to Maj. Michael Flatoff of the Army’s Training and Doctrine Command. If a soldier were a particularly fast runner or good marksman, that could be incorporated into his virtual persona and applied in different games and virtual training exercises.

If, on the other hand, a soldier failed a physical test, that disadvantage would also be replicated. The avatar would accompany the soldier throughout his career, but for now, soldiers can keep their real and gaming personas separate. The program is still in the middle stages of development.

With technology such as Google Glasses making recent news, it’s possible the military will adapt augmented reality sooner rather than later. West anticipates combining similar goggles with live training scenarios such as MOUT — military operations on urban terrain. Soldiers could then navigate through real buildings or streets but see enemy combatants or other features overlaid on the scene through their head-mounted displays.

“You can see a merging of the virtual and the physical,” West said.

That virtual display should experience increased realism and ease of use in the coming years. While graphics have gotten better and better, Foster says he won’t be satisfied until there are leaves and sand blowing realistically through the scene.

“That’s why we go to the commercial game industry,” Foster said. “The graphical part of it is the linchpin.”

And for the generation of digital natives who grew up on PlayStations and Xboxes, great graphics and intuitive technology are essential.

“They’ve come to expect that kind of technology,” West said. “If we have it for 10- and 12-year-olds in their houses, why wouldn’t we have these kinds of technologies for our soldiers to leverage and use?”