Turning Hollywood movie technology into the modern Sailor's reality

Standing in a dimly lit room in front of a television laid on its back, a pair of 3D glasses causes a robotic hand to rise up out of the screen. Like Tony Stark playing with his many gadgets in the newest superhero movie, the user turns the hand over with a stylus.

Produced by Mass Communication Specialist 2nd Class Jason Kofonow, Defense Media Activity

By touching four screws on the top of the virtual hand, he removes a cover to reveal its inner workings. The faulty part is identified and instructions tell the user how to replace the damaged piston and return the hand to normal operation.

"A lot of work is being done in holographic displays so you actually see something coming out of a display in front of you and you can put your hands around it in your real space," said Heidi Buck, program manager for the Battlespace Exploitation of Mixed Reality (BEMR) lab at Space and Naval Warfare Systems Center Pacific (SSC Pacific). "We want to take all that technology and figure out how the Navy can start using it."

The goal is to have that Sailor in an engineering space and rather than having to haul around large technical manuals, they can pull out a small tablet device and a pair of glasses and be shown step-by-step instructions on a 3D model of the complex systems they are maintaining.

Perched at the top of Naval Base Point Loma, Calif., surrounded by homes few military salaries can afford, with breathtaking views of San Diego Bay and downtown to one side, and a seemingly endless Pacific Ocean on the other, there sits a cluster of nondescript buildings that are forging the future of the Navy.

At any given time, the military and civilian personnel at SSC Pacific are working on thousands of projects, which aim to better our Navy and the Sailors who serve her.

BEMR lab is doing their part by utilizing new technology being developed in virtual reality and figuring out how the Navy can use it to transform the way the Sailors do their jobs. Using websites like kickstarter.com, the lab takes the newest technologies being developed and looks for ways to incorporate it in the fleet.

"In the BEMR lab we're looking to exploit commercial technology related to mixed reality and mixed reality is augmented reality and virtual reality," said Buck. "We're looking at the commercial space, primarily the computer gaming space, looking at low cost technology and how we can use that for Navy applications."

A large TV turned on its side sits against the wall with a six foot tall avatar of a Cmdr. Foster standing in the screen. As Buck walks across the room and moves in front of the television, a nearby camera senses her position and Cmdr. Foster snaps to attention. He then begins a brief introduction about the demonstration that's about to begin. 

Buck walks to a corner of the room that looks like a video game system on steroids. A display of about a half dozen televisions surrounds a chair on a small platform. Buck sits down and the screens come to life. The upper screens show the view from the bridge of a ship along with a representation of the current course.

In front of Buck are two large touchscreen monitors, which show a fully customizable helm. A swipe of the finger allows Buck to move to the controls wherever on the screen is most comfortable. Another slide of the finger lets her change course and speed with ease.

Cmdr. Foster moves across the pilot house and says, "Let's go fully immersive." Buck pulls on a pair of fingerless gloves. The gloves glow red from the implanted sensors and lights. She reaches for the Oculus headset and pulls it over her eyes. The headset resembles a tricked out set of ski goggles.

As soon as the headset is secured, Buck is instantly transported onto the bridge of the ship. A full 360-degree view allows her to turn and look wherever she wants. The sensors on the gloves translate her movement so she can see her arm wave around in this virtual space.

"When you look at kids today and the online games that they're playing and the immersiveness of those games and the reality of it; they're reaching out, they're ducking, and they're reacting," said Jeff Waters, a computer scientist at the SSC Pacific. "They really feel like they're there."

That sense of immersion is what the BEMR lab to trying to replicate with Navy scenarios with the goal of taking those demonstrations and building uses that real Sailors can use in the course of their day.

Cmdr. Foster announces there has been a distress call from a nearby ship, which is under attack by pirates. Buck sets a course and deploys an unmanned helicopter to help thwart the attack.

The view transitions from the bridge to a first person point of view from the aircraft. Buck sees the ocean rushing by and the pirate's boat turn and flee as the helicopter approaches. It feels like a video game but rather than just awesome graphics on a flat screen in front of her, she really feels like she is in the scene and a participant.

"I get the most enjoyment when I'm giving a demo and I see people's subconscious reactions," said Arne Odland, a computer scientist at SSC Pacific. "They can't control themselves. I suddenly almost get punched. It's happened many times. I see people reach out like you would see a crazy person reach out. It's uncontrollable, something kicks in in their mind that they can go out and pet the dog that's in their demonstration. They can go out and grab the object. They try and it's just uncontrollable. It just happens."

Flames are visible as smoke fills a crowded space on the ship. Cmdr. Foster says that there is an injured shipmate who can't escape. Buck presses a button on the touch screen and deploys a robot into the space. She's given a first-person view of the robot as it navigates the narrow passageways to close valves and fight the virtual fire.

While robots are becoming more common in the fleet, the folks at BEMR lab are ready to push the envelope and figure out additional ways to keep Sailors safe with virtual reality while they work.

"The people who are on the ships, the warfighters themselves, they grew up with a lot of this," said Waters. "They know how this can be used. These are things we can offer to the community and we can say to them, 'how do we visualize this best, how do we use this?'"

Many of today's Sailors are already familiar with a lot of the new technology that the BEMR lab is using. SSC Pacific is working so that the next time a Sailor sees some futuristic gadget, it's not in a Hollywood superhero movie ... it's in their workspace.