SATE V Report: Ten Years of Static Analysis Tool Expositions

Published

October 23, 2018

Author(s)

Aurelien M. Delaitre, Bertrand C. Stivalet, Paul E. Black, Vadim Okun, Terry S. Cohen, Athos Ribeiro

Abstract

Software assurance has been the focus of the National Institute of Standards and Technology (NIST) Software Assurance Metrics and Tool Evaluation (SAMATE) team for many years.The Static Analysis Tool Exposition (SATE) is one of the team’s prominent projects to advance research in and adoption of static analysis, one of several software assurance methods.This report describes our approach and methodology. It then presents and discusses the results collected from the fifth edition of SATE. Overall, the goal of SATE was not to rank static analysis tools, but rather to propose a methodology to assess tool effectiveness. Others can use this methodology to determine which tools fit their requirements. The results in this report are presented as examples and used as a basis for further discussion. Our methodology relies on metrics, such as recall and precision, to determine tool effectiveness. To calculate these metrics, we designed test cases that exhibit certain characteristics. Most of the test cases were large pieces of software with cyber-security implications. Fourteen participants ran their tools on these test cases and sent us a report of their findings. We analyzed these reports and calculated the metrics to assess the tools’ effectiveness. Although a few results remained inconclusive, many key elements could be inferred based on our methodology, test cases, and analysis. We were able to estimate the propensity of tools to find critical vulnerabilities in real software, the degree of noise they produced, and the type of weaknesses they were able to find. Some shortcomings in the methodology and test cases were also identified and solutions proposed for the next edition of SATE.

Citation

Special Publication (NIST SP) - 500-326

Report Number

500-326

NIST Pub Series

Special Publication (NIST SP)

Pub Type

NIST Pubs

Download Paper

DOI Link

Keywords

Security Weaknesses, Software Assurance, Static Analysis Tools, Vulnerability

Citation

Delaitre, A. , Stivalet, B. , Black, P. , Okun, V. , Cohen, T. and Ribeiro, A. (2018), SATE V Report: Ten Years of Static Analysis Tool Expositions, Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.SP.500-326 (Accessed April 29, 2024)