Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

An Empirical Comparison of Combinatorial and Random Testing

Published

Author(s)

Raghu N. Kacker, David R. Kuhn

Abstract

Some conflicting results have been reported on the comparison between t-way combinatorial testing and random testing. In this paper, we report a new study that applies t-way and random testing to the Siemens suite. In particular, we investigate the stability of the two techniques. We measure both code coverage and fault detection effectiveness. Each program in the Siemens suite has a number of faulty versions. In addition, mutation faults are used to better evaluate fault detection effectiveness in terms of both number and diversity of faults. The experimental results show that in most cases, t-way testing performed as good as or better than random testing. There are few cases where random testing performed better, but with a very small margin. Overall, the differences between the two techniques are not as significant as one would have probably expected. We discuss the practical implications of the results. We believe that more studies are needed to better understand the comparison of the two techniques.
Proceedings Title
Seventh IEEE International Conference on Software Testing Verification and Validation (ICST 2014) Workshop on Combinatorial Testing (CT)
Conference Dates
March 31-April 4, 2014
Conference Location
Cleveland, OH

Keywords

Combinatorial Testing, Random Testing

Citation

Kacker, R. and Kuhn, D. (2014), An Empirical Comparison of Combinatorial and Random Testing, Seventh IEEE International Conference on Software Testing Verification and Validation (ICST 2014) Workshop on Combinatorial Testing (CT), Cleveland, OH, [online], https://doi.org/10.1109/ICSTW.2014.8 (Accessed December 8, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created April 4, 2014, Updated November 10, 2018