Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Estimating Error Discovery in Library Conformance Testing


Gordon E. Lyon


Procedure libraries provide fresh insight into unit conformance testing. The library specification, as an ensemble, offers cost estimates for error discovery even prior to obtaining code to test. One extracts or decides counts of: (i) formal parameters for each procedure, (ii) test values for parameters, and (iii) procedures having the same number of parameters. A simplified, idealized formulation of this tally information employs randomization to nullify unwanted factors, normalization to control error variation, and weak ordering to schedule tests by return-on-investment. These techniques in turn support a novel strategy of truncated k-way testing. Truncated k-way requires no subtle combinatorial generation algorithms that often vex ordinary k-way testing. When test parameters take many values, cheaper truncated k-way testing becomes probabilistically equivalent to the more expensive k-way test. The approach addresses most conformance testing, but excludes high-assurance circumstances (e.g., flight-control software). Results underscore and quantify sometimes difficult tradeoffs faced in pursuing satisfactory test coverage, tolerable risk and affordable cost with consumer-grade software.
Proceedings Title
Proceedings of the International Symposium on Software Testing and Analysis
Conference Dates
July 1, 2006
Conference Location
Portland, ME
Conference Title
International Symposium on Software Testing and Analysis


conformance testing, error discovery, procedure library, return-on-investment, software results


Lyon, G. (1970), Estimating Error Discovery in Library Conformance Testing, Proceedings of the International Symposium on Software Testing and Analysis, Portland, ME (Accessed April 18, 2024)
Created August 26, 2016, Updated February 17, 2017