XML-based Test Accelerator

Overview:  Many enterprises have become driven by digital data, including healthcare, e-commerce, and homeland security. Each of these application areas are developing standards that allow for the exchange and manipulation of this data.  As the plethora of standards is promulgated throughout the industry, the demand for metrics to measure conformance to these standards has increased.  Effective use of these metrics dictates that they be developed in a short amount of time, and be done so in a way that allows changes to the specification to be easily propagated through a given set of tests.

Industry Needs Addressed: It has become well-known that software is not adequately tested and is rushed to market full of bugs. A recent NIST study says that the annual cost to the country for inadequate testing is as high as 59 billion dollars.  One of the main reasons for inadequate testing is the time and expense of developing comprehensive test suites.  A test accelerator that provides core reusable components, yet allows input and output formats to be defined by the user, will facilitate building conformance tests.  Making the testing process faster, more flexible, and extensible provides the IT community with tools to quickly develop tests that will be used to identify and correct errors in their implementations and standards.

NIST/ITL Approach: NIST developed a prototype test accelerator that is flexible and extensible. Building upon our prior experience in XML-based testing, we developed a data-driven approach that consists of encapsulating the structure and semantics of a specification in a NIST developed XML test language.  The test accelerator takes the XML input and creates an input control graph, which is intelligently traversed, resulting in a set of test instances.  The test instances are transformed using NIST developed templates to generate specific tests for a given test suite.  Development of additional tests can then be accomplished via incremental changes to the pre- and post processors.  Work has begun on developing additional methods to trim the control graph and generate appropriate data values for specific data types.

This approach was applied to the Linux Standard Base stdlib and to XML Query expressions.  These standards were selected to scope the work and provide significant complexity to truly investigate this approach.  Each of these communities are developing standards that are large and complex, thus requiring an automated testing approach.

Impact: NIST developed conformance tests provide the information technology industry with the necessary tools to ensure that their standards are correctly implemented, which will lead to interoperability in many application areas.  The NIST test development system will allow us to focus on producing high quality tests while reusing capabilities from prior testing projects.  It will also allow us to develop methods for defining semantic capabilities for complex systems.