Lockheed Martin Study: NIST Computational Tool Improves Product Testing and Saves Money

Lockheed Martin, the global security and aerospace company, estimates that widely used software testing methods developed at the National Institute of Standards and Technology (NIST) can trim test planning and design costs by up to 20 percent, while greatly improving the thoroughness of product and system testing during development.

The projected benefits are based on the results of the company's two-year pilot study of combinatorial testing, as reported in an article in IEEE Computer. The study was carried out under a Cooperative Research and Development Agreement (CRADA) with NIST.

In contrast to creating test cases manually, combinatorial testing is a comprehensive "trouble shooting" tool for rooting out faulty code, calculation errors, unanticipated interactions and other causes of software failures. Used during the early stages of product development, the computational tool can detect faults that might otherwise linger until late-stage testing or, perhaps, slip through entirely.

Corrective rework to fix late-detected software errors can wreck project schedules and may be the costliest part of the development process, especially for complex products and systems.

Developed with collaborators from the University of Texas at Arlington, NIST's Advanced Combinatorial Testing System (ACTS) "uses proven mathematical techniques to greatly reduce the number of tests a company needs to perform to ensure the quality of a product or process," explains NIST computer scientist Richard Kuhn.

ACTS generates a plan for testing combinations of two to six variables that can interact and cause errors.

While studying software crashes of medical devices and Web browsers, Kuhn and colleagues determined that between 70 and 95 percent of software failures are triggered by interactions between only two variables. Nearly all software failures are triggered by no more than six variables.

"For example, all six-way combinations of 34 switches could be tested with only 522 tests instead of 17 billion for all possible combinations." said Raghu Kacker, a NIST mathematical statistician.

In one project, combinatorial testing was compared with conventional methods for testing commercial Web browser software. NIST used the method to find 100 percent of flaws using less than 5 percent of the original number of tests.

During Lockheed Martin's two-year study to assess the merits of the software testing approach, the company chose ACTS as its primary tool, which it supplemented with other combinatorial testing packages. In all, the tool's utility was evaluated in eight pilot development projects, including an electronic warfare system, the redesign of a fin on a fighter jet, and an evaluation of aircraft engine failure modes.

"Lockheed Martin's initial estimate is that combinatorial testing and supporting technology can save up to 20 percent in test planning and design costs if used early on in a program and can increase test coverage by 20 to 50 percent," company and NIST researchers wrote in the Computer article.

"Our experience showed that the combinatorial testing technique is maturing," says Thomas L. Wissink, director of integration, test, and evaluation at Lockheed Martin. "We are continuing to use combinatorial testing at Lockheed Martin and are planning to extend its use through the company."

NIST also is building on lessons learned from this collaboration. Knowledge and insights gained are guiding NIST research to further improve measurement and testing capabilities focused on ensuring the quality, safety and reliability of software and systems.

The ACTS tool and tutorial are publicly available.

Paper: J. Hagar, T. Wissink, D. Kuhn and R. Kacker. Introducing combinatorial testing in a large organization, Computer, April 2015.