Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Combinatorial Approach to Detecting Buffer Overflow Vulnerabilities



Raghu N. Kacker, Yu Lei, David R. Kuhn, Wenhua Wang


Buffer overflow vulnerabilities are program defects that can cause a buffer overflow to occur at runtime. Many security attacks exploit buffer overflow vulnerabilities to compromise critical data structures. In this paper, we present a black-box testing approach to detecting buffer overflow vulnerabilities. In most cases the attacker can influence the behavior of a target system only by controlling its external parameters. Therefore, launching a successful attack often amounts to a clever way of tweaking the values of external parameters. Our approach identifies two conditions that must be met in order to trigger a buffer overflow, and is centered on how to tweak external parameter values in a systematic manner to satisfy these two conditions. A novel aspect of our approach is that it adapts a general software testing technique called combinatorial testing to the domain of security testing. In particular, our approach exploits the fact that combinatorial testing often achieves a high level of code coverage. We report a prototype tool, called Fugai, that implements our approach. The results of applying Fugai to five open-source programs show that our approach is very effective in detecting buffer overflow vulnerabilities in these programs.
ACM Transactions on Software Engineering and Methodology


software security, security testing, buffer overflow vulnerability


Kacker, R. , Lei, Y. , Kuhn, D. and , W. (2011), A Combinatorial Approach to Detecting Buffer Overflow Vulnerabilities, ACM Transactions on Software Engineering and Methodology, [online], (Accessed February 28, 2024)
Created June 14, 2011, Updated May 4, 2021