Oracle-Free Match Testing of a Program Using Covering Arrays and Equivalence Classes

This is a method and software for testing computer software. The method applies to a broad range of software but is particularly applicable to software with complex conditions and decisions, e.g., process control, avionics, and other areas.  This method also verifies equivalence classes for module unit testing. Both testing and equivalence class verification are achieved using a two-layer covering array, where some or all values of a primary covering array represent equivalence classes. A second layer covering array of the equivalence class values is computed, and its values are substituted for the equivalence class names in the primary array. It is shown that this method can also detect certain classes of errors without a conventional test oracle.

The program is tested against input values partitioned into equivalence classes. In a particular embodiment, representative values from each equivalence class are provided to the program, and the output of the program is stored, e.g., in memory as a result vector. It is contemplated that the program can be executable code (including units, modules, scripts, and the like), data, system configuration files, machine environment state settings, and the like. In a certain embodiment, the program is an executable program that receives the representative values as a substitute for the input values of the variable, which is the expected input by the program.

A process for testing a program includes: receiving a variable comprising a plurality of input values; producing a plurality of equivalence classes for the input values; producing a representative value per equivalence class; producing, by a processor, a primary covering array comprising a plurality of primary vectors; producing a secondary covering array comprising a plurality of secondary vectors; providing the secondary vectors to the program; and producing a result vector comprising a plurality of result entries to test the program. A computer system for testing the program includes: a memory; and a processor, in communication with the memory, wherein the computer system is configured to perform the process for testing the program. A computer program product for testing the program includes: a non-transitory computer-readable storage medium readable by a processor and storing program code for execution by the processor to perform the process.