SATE V Report: Ten Years of Static Analysis Tool Expositions
Aurelien M. Delaitre, Bertrand C. Stivalet, Paul E. Black, Vadim Okun, Terry S. Cohen, Athos Ribeiro
Software assurance has been the focus of the National Institute of Standards and Technology (NIST) Software Assurance Metrics and Tool Evaluation (SAMATE) team for many years.The Static Analysis Tool Exposition (SATE) is one of the teams prominent projects to advance research in and adoption of static analysis, one of several software assurance methods.This report describes our approach and methodology. It then presents and discusses the results collected from the fifth edition of SATE. Overall, the goal of SATE was not to rank static analysis tools, but rather to propose a methodology to assess tool effectiveness. Others can use this methodology to determine which tools fit their requirements. The results in this report are presented as examples and used as a basis for further discussion. Our methodology relies on metrics, such as recall and precision, to determine tool effectiveness. To calculate these metrics, we designed test cases that exhibit certain characteristics. Most of the test cases were large pieces of software with cyber-security implications. Fourteen participants ran their tools on these test cases and sent us a report of their findings. We analyzed these reports and calculated the metrics to assess the tools effectiveness. Although a few results remained inconclusive, many key elements could be inferred based on our methodology, test cases, and analysis. We were able to estimate the propensity of tools to find critical vulnerabilities in real software, the degree of noise they produced, and the type of weaknesses they were able to find. Some shortcomings in the methodology and test cases were also identified and solutions proposed for the next edition of SATE.