Modified Monitor Provides Glasses-Free 3D for Pilots, Gamers

NASA Technology

When flying the increasingly crowded skies, pilots need to have an arsenal of information at the ready: altitude, airspeed, fuel level, distance to their destination, the location of other planes in the sky. All of this information is presented in a series of two-dimensional instruments, panels and readouts, meaning the pilot has to mentally assemble the information and translate that into the three-dimensional world to better understand the relationship between air, ground, and traffic.

NASA has long been interested in making it as easy as possible for pilots and astronauts to have the best information available to ensure safe flights, knowing that humans are imperfect creatures.

Kyle Ellis is an aerospace research engineer in the Crew Systems and Aviation Operations Branch at Langley Research Center and spends a good deal of time analyzing pilot feedback to determine which tools and alarms best capture pilots’ attention during flight.

“We’ll look at their eye tracking and their brain activity to see what the state of the pilot currently is,” he explains. “Let’s say they haven’t looked at the airspeed indicator in the last five minutes and the autothrottle has been disconnected and the plane starts bleeding off speed. The plane is always monitoring what’s going on, but it also makes the assumption that the pilot knows what’s going on as well.”

Alarms and flashing lights are used to draw the pilots’ attention to gauges to signal when something’s amiss, but if it were possible to monitor what kind of alert worked best, new tools could be developed to further boost pilots’ awareness of their conditions and better ensure safe travel. “We can do a traditional blink, like we do for a warning, or we can do a 3D blink, have it pop out of the screen at them,” Ellis says.

Unfortunately, the ability to install a 3D monitor or other device in the flight deck has been limited. Fighter pilots have head-up displays that require the head be within in a certain space, also known as a head box. This is similar to what is traditionally required for 3D monitors and is easier for fighter pilots to maintain for the duration of their short flights because they’re strapped into place and don’t move around much, Ellis says. Commercial aviation pilots, however, who spend hours in their seats, move quite a bit more, and would often fall outside any head box into which a 3D image might be projected. The technical barrier of the limited head box has thus far made 3D displays non-viable in commercial operations.

Technology Transfer

For years, NASA has been working with Rochester, New York-based Dimension Technologies Inc. (DTI) to develop a 3D monitor that would not only provide a vivid image (Spinoff 1995, 2002) but could track the pilot’s eye movements, all without requiring glasses.

The ability to track and follow movement without losing resolution is DTI’s latest advancement in this technology and could be key to helping pilots. “Pilots need to be immersed in what they’re doing,” Ellis says. In the earlier iterations of the monitor, there were limits to the technology that prevented a pilot from having the kind of full immersion needed when attempting to use 3D displays. “They had a wide field of view with multiple little head box areas, so while it gave you a good image, the depth of field wasn’t as large.”

Earlier iterations of the technology couldn’t be considered for installation in airplanes because the images did not have the ability to move with the viewer, meaning pilots would have to stay perfectly still or risk losing a clear image.

Through a series of Small Business Innovation Research (SBIR) contracts, the most recent of which was awarded in early 2014, DTI has been working on monitors that can switch between 2D and 3D imaging. By modifying the backlighting system in any LCD monitor and incorporating a forward-facing camera, targeting the eyes of the person looking at the monitor, the display is able to follow the movement and position of the viewer in real time without compromising resolution or depth of field, explains DTI CEO Arnie Lagergren.

“Ideally, both the pilot and the copilot can see each other’s monitors,” he says. “They’ll be able to see with real depth.”

Benefits

A sample of the scenarios DTI has been testing is a loop of flight traffic data provided to the company, showing a swarm of red, yellow, and white airplanes massing along the Eastern Seaboard of the United States, with each color indicating whether a plane was landing, taking off, or in mid-flight. In standard display mode, the planes appear and blink off the map in real time.

“What you’ve got is an incredible beehive of data that doesn’t serve the pilot too well, nor the copilot, in terms of quick, intuitive understanding of what they are facing,” Lagergren says.

But switched into 3D mode, the clustered information is sorted and clarified. It’s possible to determine whether one yellow plane is flying at a higher altitude than another, and whether two planes might be using the same runway at a busy airport.

The 3D capability without glasses, thanks to eye-tracking technology, should help NASA ensure pilots have all the information they need to navigate increasingly busy airports, which are already struggling with higher demand for runway space, Ellis says.

“Traffic densities are going up by three or four times, projected, in terms of air travel, in the next 20 years or so. That causes a lot of problems. How do we take four times the amount of aircraft we have now flying into Chicago-O’Hare, Boston-Logan or Reagan National in Washington, DC,” he says. “You can’t build another Reagan because DC’s pretty densely populated. The only thing you can do is improve the efficiency of it.”

DTI isn’t only thinking about pilots with this glassless 3D technology.

Already the displays have been presented at two large digital visual arts conferences—Immersed in Toronto and SIGGRAPH in Vancouver—to rave reviews. At Immersed, DTI set up a console with the action role-playing game Skyrim, which features mountains, rivers, fields, and animals, including dragons and woolly mammoths. “We had these woolly mammoths walking over the terrain, up and down the mountain, into valleys, and it seemed like you could see forever,” Lagergren says. “People were awed by it.”

The successful demonstration earned the company an invite to several technology conferences to show off its monitors. DTI also took its displays to the NVIDIA GPU Technology Conference and Emerging Companies Summit, in 2015, which highlights applications that are breaking new ground and pushing the boundaries of visual computing.

Games built to be played or viewed in 3D rely on glasses, or players must remain in a “sweet spot” to get the full effect, but DTI has eliminated both those constraints, he says. “Now you can walk around and the monitor will track you and keep the 3D at your eyes. There’s no loss of resolution or brightness, no restriction in head movement, and it works for two players.”

DTI’s latest display, sold commercially as the DTI Mission Critical 2D/3D monitor, has also been met with enthusiasm from 3D animators, including James Stewart. His award-winning stop-motion short film, Foxed!, is among the clips used by DTI when demonstrating the monitor.

Finally, there’s been interest from car manufacturers as well, which could lead to 2D/3D displays as part of the center console, driver instrument cluster, and backseat entertainment packages.

“This thing is starting to take off,” Lagergren says. “The market’s ready to accept it.”