Estimating Error Discovery in Library Conformance Testing
Gordon E. Lyon
Procedure libraries provide fresh insight into unit conformance testing. The library specification, as an ensemble, offers cost estimates for error discovery even prior to obtaining code to test. One extracts or decides counts of: (i) formal parameters for each procedure, (ii) test values for parameters, and (iii) procedures having the same number of parameters. A simplified, idealized formulation of this tally information employs randomization to nullify unwanted factors, normalization to control error variation, and weak ordering to schedule tests by return-on-investment. These techniques in turn support a novel strategy of truncated k-way testing. Truncated k-way requires no subtle combinatorial generation algorithms that often vex ordinary k-way testing. When test parameters take many values, cheaper truncated k-way testing becomes probabilistically equivalent to the more expensive k-way test. The approach addresses most conformance testing, but excludes high-assurance circumstances (e.g., flight-control software). Results underscore and quantify sometimes difficult tradeoffs faced in pursuing satisfactory test coverage, tolerable risk and affordable cost with consumer-grade software.
Proceedings of the International Symposium on Software Testing and Analysis
July 1, 2006
International Symposium on Software Testing and Analysis