Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

SATE V Report: Ten Years of Static Analysis Tool Expositions

Published

Author(s)

Aurelien M. Delaitre, Bertrand C. Stivalet, Paul E. Black, Vadim Okun, Terry S. Cohen, Athos Ribeiro

Abstract

Software assurance has been the focus of the National Institute of Standards and Technology (NIST) Software Assurance Metrics and Tool Evaluation (SAMATE) team for many years.The Static Analysis Tool Exposition (SATE) is one of the team’s prominent projects to advance research in and adoption of static analysis, one of several software assurance methods.This report describes our approach and methodology. It then presents and discusses the results collected from the fifth edition of SATE. Overall, the goal of SATE was not to rank static analysis tools, but rather to propose a methodology to assess tool effectiveness. Others can use this methodology to determine which tools fit their requirements. The results in this report are presented as examples and used as a basis for further discussion. Our methodology relies on metrics, such as recall and precision, to determine tool effectiveness. To calculate these metrics, we designed test cases that exhibit certain characteristics. Most of the test cases were large pieces of software with cyber-security implications. Fourteen participants ran their tools on these test cases and sent us a report of their findings. We analyzed these reports and calculated the metrics to assess the tools’ effectiveness. Although a few results remained inconclusive, many key elements could be inferred based on our methodology, test cases, and analysis. We were able to estimate the propensity of tools to find critical vulnerabilities in real software, the degree of noise they produced, and the type of weaknesses they were able to find. Some shortcomings in the methodology and test cases were also identified and solutions proposed for the next edition of SATE.
Citation
Special Publication (NIST SP) - 500-326
Report Number
500-326

Keywords

Security Weaknesses, Software Assurance, Static Analysis Tools, Vulnerability

Citation

Delaitre, A. , Stivalet, B. , Black, P. , Okun, V. , Cohen, T. and Ribeiro, A. (2018), SATE V Report: Ten Years of Static Analysis Tool Expositions, Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.SP.500-326 (Accessed December 12, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created October 23, 2018, Updated May 4, 2021