System-Health Monitor Predicts Failures before They Happen

NASA Technology

A young engineering and technical services company is betting that NASA teams at the Mission Control Center in Houston, aboard the International Space Station (ISS), and in Ground Support at Cape Canaveral—just about all of the Agency’s Hollywood roles—will set a trend that explodes across private-sector industries.

In an era of “smart” devices capable of interacting with one another and developing an awareness of and reacting to their environments, what’s still absent is any self-awareness in these technologies, says David Cirulli, engineering vice president and cofounder of Phoenix-based CEMSol LLC. The company name is shorthand for Comprehensive Engineering Management Solutions, which perhaps sounds a bit nebulous, but that may be due to the breadth of its software’s potential reach.

“In the future, there’s going to be an integrated system-health engine as part of every system out there, and it will be able to interface with other systems and components,” Cirulli says. “That’s missing today.”

He equates system-health monitoring programs with the human ability to verbalize symptoms to a doctor.

The first version of what became CEMSol’s Integrated System Health Management (ISHM) software was developed in 2003 at Ames Research Center to monitor an experimental hybrid rocket engine test bed that ran on both gas and solid fuel. Traditionally, this would have been accomplished by building models and running simulations.

“It’s hard to build models of some things, so we figured, let’s see if we can do it with the data automatically,” says David Iverson, computer engineer with the Intelligent Systems Division’s Data Sciences Group at Ames. To do that, he created what came to be called the Inductive Monitoring System (IMS).

Any system an engineer might want to monitor has sensors measuring factors such as temperature, pressure, fuel flow, voltage, and other vital signs, Iverson explains. The IMS program collects data from all those sensors and, by mining an archive of such data samples, determines relationships between these various factors and establishes a baseline and parameters for normal system behavior. Deviations from that norm suggest a problem and a possible failure in the near future.

“Our technology allows you to determine and detect known and unknown relationships and anomalies in any system,” as Cirulli puts it. He notes that a human can understand the interrelationship of five to seven entities at most, while the data-mining software can see how hundreds of systems relate to each other.

The engine test bed was just a chance to test this new method, Iverson says. “It works for just about anything where you have data collection and some sort of normal baseline.”

Around 2005, a group at the Mission Control Center responsible for regulating spacecraft orientation—known as attitude control—selected IMS as a technology that might help sort out problems with a failing gyroscope. Iverson’s group managed to enable the software to monitor the gyroscopes in real time, and the effort was a success.

By the time Mission Control started developing its own IMS models in 2012, Iverson’s team had helped to apply the program to about a dozen systems there.

Now, the software is being integrated into the next-generation launch control systems at Kennedy Space Center, and it monitored electrical systems on the Orion capsule during its December 2014 test launch. It’s also been applied to the F-18 fighter jet and the Black Hawk helicopter engine.

Also in 2014, a demonstration paired IMS with NASA’s Hybrid Diagnostic Engine to let astronauts aboard the ISS monitor the carbon analyzer that ensures the safety of their drinking water. The project is preparation for deep-space missions, Iverson says. “The goal is to help the crew so that when they’re a long ways away and have a long delay for communication, they can do more on their own of what’s done on the ground now.” He says the demonstration has so far been successful.

Technology Transfer

Seeing this sort of system-monitoring technology as the way of the future, in 2012 CEMSol licensed the program, which NASA had patented in 2008. Along with it, the company also licensed the Outlier Detection via Estimating Clusters software that Iverson developed as an extension to the IMS algorithm, along with the graphical user interface Ames had created.

CEMSol enhanced the program’s presentation and developed two software packages—one as a desktop application and another as a software developer’s kit. The desktop version uses a set of predefined functions to periodically import datasets and analyze them, charting data points and highlighting any that are outside the norm. The software developer’s kit, meanwhile, can monitor systems constantly in real time. “It’s a library of functions the programmers can choose from to perform whatever system-detection analysis functions they want to use,” Cirulli says.

Having laid the groundwork for a final product since 2008, the company first offered its ISHM software for sale in late 2014. By then it already had its first major contract, with DSI International preparing to integrate the software into its eXpress system-health management application.

The integration of eXpress and ISHM provides a combination of model-based diagnostics and empirical diagnostics. Model-based diagnostics are based on engineering data for a system’s design and are particularly good at identifying uncommon, first-time, or early failures that have not had sufficient empirical history. On the other hand, empirical diagnostics, because they’re based on observation of previous problems in an existing system, are especially good at identifying issues that fall outside model-based diagnostics or finding trends that fine-tune the engineering analysis.

Benefits

In 2012 CEMSol teamed up with Ames and Lockheed Martin to try ISHM on the Lockheed C-130 Hercules military transport plane, which tended to have problems with the bleed valve that switches air flow between engines during start-up. Datasets were taken from start sequences for 16 of the planes over four years, including starter-system failures. These were fed into the program, which was subsequently able to predict a start-up failure three starts before it occurred.

By this method, the manufacturer doesn’t have to redesign the part or introduce a new computer system, and the operator doesn’t have to schedule a backup plane or wait for the valve to fail and then find another plane, Cirulli points out.

Lockheed Martin invested $70,000 in the test and quickly recouped 10 times that amount in reduced maintenance costs and mission delays.

“The reason it’s so powerful is that it can detect subtle early-warning signs well in advance of a failure,”Cirulli says.

But he still sees ISHM as just a first step toward near-universal self-monitoring in everything from cell phones to medical devices, and from refrigerators to automobiles and snowmobiles. The company has even successfully used the software to analyze weather systems and predict snowstorms, thunderstorms, and dust storms. “So far, we haven’t found any system it hasn’t been able to work on,” he says. He plans to feed several years’ worth of data from his own blood work into the software to begin to determine if it could use patient data to predict health problems before they arise.

He compares the coming improvements in safety and reliability to the revolution in automobile quality that Toyota brought about in the 1980s, prior to which drivers were happy if a vehicle survived 70,000 miles.

“Historically, system-health management has been an afterthought,” he says. “In the future, things will be much more reliable, durable, and dependable, because they’ll have a much better understanding of their own behavior.”

He credits Ames with contributing to this shift not only through its invention of the IMS software but also through the assistance it provided his company in turning it into a commercial product. “I can’t tell you how good an experience this has been in terms of the support we received,” he says, commending the “significant contribution and cooperation of the NASA Ames team in deploying this technology.”