Estimating Error Discovery in Library Conformance Testing

Author(s)

Gordon E. Lyon

Abstract

Procedure libraries provide fresh insight into unit conformance testing. The library specification, as an ensemble, offers cost estimates for error discovery even prior to obtaining code to test. One extracts or decides counts of: (i) formal parameters for each procedure, (ii) test values for parameters, and (iii) procedures having the same number of parameters. A simplified, idealized formulation of this tally information employs randomization to nullify unwanted factors, normalization to control error variation, and weak ordering to schedule tests by return-on-investment. These techniques in turn support a novel strategy of truncated k-way testing. Truncated k-way requires no subtle combinatorial generation algorithms that often vex ordinary k-way testing. When test parameters take many values, cheaper truncated k-way testing becomes probabilistically equivalent to the more expensive k-way test. The approach addresses most conformance testing, but excludes high-assurance circumstances (e.g., flight-control software). Results underscore and quantify sometimes difficult tradeoffs faced in pursuing satisfactory test coverage, tolerable risk and affordable cost with consumer-grade software.

Proceedings Title

Proceedings of the International Symposium on Software Testing and Analysis

Conference Dates

July 1, 2006

Conference Location

Portland, ME

Conference Title

International Symposium on Software Testing and Analysis

Pub Type

Conferences

Keywords

conformance testing, error discovery, procedure library, return-on-investment, software results

Citation

Lyon, G. (1970), Estimating Error Discovery in Library Conformance Testing, Proceedings of the International Symposium on Software Testing and Analysis, Portland, ME (Accessed April 24, 2024)