U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

PUBLICATIONS

SATE VI Report: Bug Injection and Collection

Published

Author(s)

Aurelien Delaitre, Paul E. Black, Damien Cupif, Guillaume Haben, Loembe Alex-Kevin, Vadim Okun, Yann Prono, Aurelien Delaitre

Abstract

The SATE VI report presents the results of a security-focused bug finding evaluation exercise carried out from 2018 to 2023 on various code bases using static analysis tools. Existing bugs were extracted from bug tracker reports and the CVE/NVD database, and additional bugs were injected using automated tools and manual analysis. The results of this exercise showed significant variability across tool effectiveness, depending on the test cases, bug classes, and bug complexity involved. The report discusses the shortcomings and difficulties encountered during the bug injection process, which marginally impeded the efficiency of the evaluation. The report emphasizes the correlation between high code complexity and tool difficulty in identifying bugs. Recall and discrimination rates were lower for the convoluted C Track than the considerably less complex Java Track. Across all languages and code bases, tools found bugs with lower complexity more readily than bugs with higher complexity. Finding rates varied for different bug classes, in line with the inherent complexity of each bug class (e.g., recall for simpler initialization errors was greater than on more intricate buffer errors). The report discusses the shortcomings of the bug injection process. Regardless of the test case, injected bugs were not found by tools at the same rate as existing bugs, implying that their quality needs to improve. The report also includes a summary of the Ockham Sound Analysis Criteria track, which focused on tools that do not report false positives or false negatives. The SATE VI report concludes that static analysis is a useful technique to find real security bugs in large code bases. The right set of tools, used properly, can help increase code quality and security. Potential users should test a tool or set of tools on their own code base before using them in production. The metrics presented in SATE VI are suitable for assessing tool fitness for such a use case.
Citation
Special Publication (NIST SP) - 500-341
Report Number
500-341
NIST Pub Series
Special Publication (NIST SP)
Pub Type
NIST Pubs

Download Paper

https://doi.org/10.6028/NIST.SP.500-341
Local Download

Keywords

Static Analysis, Cybersecurity, Bug Injection, Software Vulnerability, Software Assurance
Software testing, Cybersecurity measurement and Cybersecurity education and workforce development

Citation

Delaitre, A. , Black, P. , Cupif, D. , Haben, G. , Alex-Kevin, L. , Okun, V. , Prono, Y. and Delaitre, A. (2023), SATE VI Report: Bug Injection and Collection, Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.SP.500-341, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=936867 (Accessed April 16, 2024)

Created June 14, 2023