2D Analysis Software Clarifies Medical, Weather, Intelligence Images

NASA engineer Semion Kizhner spent an entire week trying to understand the Hilbert-Huang Transform with the colleague who had developed it, but to no avail. He hoped to develop a software that would get people using this powerful but complex new algorithm, he says, because he saw great potential in it that might require decades of development.

Fortunately, no one needs to fully understand a transform to use it. “I have no clue how intestines work, but that doesn’t prevent me from eating,” says Kizhner, now recently retired from Goddard Space Flight Center’s Instrument Electronics Development Branch.

A transform—essentially, a particular type of mathematical formula—helps scientists gather information from complex data. For example, Kizhner says, he might not be able to interpret the raw data from an electrocardiogram reading, but transforming the data into the frequency domain might show that something is happening about once a second, which is the normal respiration rate—a clue suggesting the phenomenon is probably associated with breathing.

“You try to apply some known transform, and if you’re lucky, it will give you something you understand,” Kizhner says.

One of the most commonly used transforms is the century-old Fourier Transform, originally devised to study heat transfer but now used in applications as diverse as quantum mechanics and digital image compression. The Fourier Transform, however, has severe limitations when it comes to analyzing natural phenomena. Kizhner had been looking for an alternative, and the Hilbert-Huang Transform (HHT), created by Goddard chief scientist for oceanography Norden Huang, seemed like a promising option.

It’s the product of the oceanographer’s nearly four decades of attempts to come up with a transform that could make more sense of data on ocean waves and atmospheric turbulence. It breaks down a one-dimensional signal into a series of what Huang called intrinsic mode functions, isolating various frequencies, and then applies the preexisting one-dimensional Hilbert spectral analysis to decompose frequency components.

Kizhner notes that the Royal Society of London published all of Huang’s 100-page paper on the subject despite the organization’s 10-page limit. “Obviously, someone was impressed.”

Although Kizhner still struggled to understand the mechanics of the transform, he and one of his students, Karin Blank, were able to write software to apply it, with initial funding from Goddard’s Internal Research and Development (IRAD) Program.

The resulting HHT Data Processing System software for one-dimensional data, which can be licensed from Goddard, works well beyond oceanography. “HHT1 works on data like nothing else before it, and people are using it all over the world,” Kizhner says, including to help monitor blood flow in the brain after brain injuries (Spinoff 2010). In 2006, the Federal Laboratory Consortium’s mid-Atlantic region named the software Best Technology of the Year.

But Kizhner—and many others, including Huang himself—were looking for a way to use the transform in two dimensions to analyze images. “There are certain things in image processing that cannot be done well with existing techniques, so we always strive to find something new,” he says.

Such a tool could help detect and eliminate stray light and other extraneous data in, say, long-exposure images of deep space. But it might also be able to extract even deeper, more specific information, Kizhner says. He likens it to using an image of illegible handwritten notes to make determinations about the writer’s character. “It’s very difficult to do.”

Following years of attempts, on a camping trip, Kizhner was finally struck by an idea that would let him apply the Hilbert Transform across two dimensions, paving the way for Huang’s technique to also analyze frequencies across a plane—a near real-time HHT2 algorithm.

It worked. With a little more IRAD funding and some help from coworkers, Kizhner developed and patented a prototype in the MATLAB programming language. To do so, he developed a way to also extend Huang’s empirical mode decomposition to two dimensions—and he sped up the whole thing by writing it as an algorithm that could run on many processing units simultaneously, in near real time. He also implemented a novel Hilbert Transform for two-dimensional spectral analysis.

One of his assignments for his day job was to help design a data processing system for a radiometer to fly on the Soil Moisture Active Passive (SMAP) satellite, which launched in 2015. The radiometer operates in the L1 microwave band, a wavelength that is unaffected by the atmosphere. That same characteristic, however, makes the L1 bandwidth popular for other uses, such as cell phones, which could interfere with the instrument’s signals. Goddard decided to try out the two-dimensional Hilbert-Huang software prototype, using it to remove radio interference from the satellite’s imagery and recover the data that had been contaminated, rather than simply deleting all contaminated data, as state-of-the-art systems did at the time.

Arlington, Virginia-based Syneren Technologies Corporation collaborated on the effort, and after a year of working together closely, the company expressed interest in commercializing the prototype HHT2 program. Syneren currently holds an exclusive field-of-use license.

Technology Transfer

It was clear the software’s image-processing capability could theoretically be applied to a wide swath of industries, but figuring out the details was better suited to the private sector, Kizhner says.

“I thought it was a great idea for us to continue the research and actually make a commercial product, because if we didn’t, this would just evaporate,” recalls Meg Vootukuru, Syneren’s owner and CEO.

Syneren signed a Space Act Agreement, and Kizhner has consulted with the company weekly ever since.

“They extended the software, enriched it, fixed bugs, and put in a lot of work, and they came up with SIETech,” Kizhner says, or Syneren Image Enhancement Technology.

To begin with, the company rewrote the software in C++, as MATLAB can be difficult to install and use. “We developed an algorithm and software that makes it easy to commercialize,” says Vootukuru, noting that the program can be installed on any hardware platform and leverages multiprocessing to run fast and efficiently.

By the end of 2016, Syneren was in talks with a number of companies and Government agencies about how the software could be tailored to each of their needs. Different clients would need different products, geared toward the solutions they’re looking for, says SIETech product manager Richard Williams. “The product is tailorable—we’ve got to determine what question you’re trying to answer. We’re open to whatever our clients need.”

Benefits

SIETech uses the Hilbert-Huang Transform to sift through different frequencies in an image, starting out by analyzing high frequencies from the entire image and moving to lower and lower frequencies, to identify different features, explains Mark Christman, a senior program manager at Syneren. For example, higher frequencies are often useful for edge detection, and lower frequencies might help distinguish healthy cells from unhealthy ones in a medical application.

“The whole process removes periodic noise and other obscurants and lets you see edges, features, and patterns you couldn’t see otherwise,” Christman says.

In one demonstration, the software was used to analyze radar images from a storm, and it identified more features and characteristics than the National Weather Service’s own image analysis system. Both identified the storm’s hook pattern, its debris ball, and some hail, but SIETech also found hail in areas where the weather service didn’t, and it was better able to determine the intensity of precipitation and debris throughout the images. “They certainly believed our product had the potential to improve forecast accuracy,” Williams says.

When Syneren turned the technology on an inflamed kidney, lower-frequency analysis was able to differentiate between normal and abnormal cells and color-code areas to assist with pattern recognition.

In defense applications, he says, the software can analyze aerial images to distinguish manmade features among foliage, spot footprints or tire tracks, and remove fog or other visual noise.

In the oil and gas industry, SIETech could be used to identify surface and subterranean patterns indicating features that might contain oil. And in oil spill crisis management, it could be used to pinpoint and prioritize areas where the oil is most concentrated, helping to assign resources for oil removal.

The Naval Oceanographic Office is considering using the program in support of its monitoring of sea surface temperatures to distinguish ocean currents, and Williams notes that forest services could use it for monitoring anything from forest fires to vegetation health.

“Just about every industry uses imagery in some way, and this software has the potential to make that more efficient for all those industries,” Williams says. “We’re just scratching the surface of what this software is capable of.”

Kizhner notes that the near real-time speed of his original software design has allowed improvements to the program to be verified quickly, enabling rapid development for new commercial applications.

He thinks the transforms have revolutionary potential that will require decades of work to fully realize—after all, no one even understands yet why they work the way they do. A lot of that work, he says, will come from others. “You have a breakthrough in engineering and science of very general proportions, and eventually people will find applications for it.”

Kizhner recalls his own motivation for struggling to apply the Hilbert-Huang Transform to two dimensions: “I tried, and it didn’t work. You know what happens with a NASA engineer when he tries something and cannot do it, and people tell him it cannot be done—the next day he starts trying to do it again,” he says. “That’s what NASA does. It finds things that weren’t known before.”