The NIST SAMATE project conducted the fourth Static Analysis Tool Exposition (SATE IV) to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement and speed adoption of tools. Briefly, participating tool makers ran their tool on a set of programs. NIST researchers performed a partial analysis of tool reports. The results and experiences were reported at the SATE IV Workshop in McLean, VA, in March, 2012. The tool reports and analysis were made publicly available in 2012. This paper describes the SATE procedure and provides our observations based on the data collected. We improved the procedure based on lessons learned from our experience with previous SATEs. One improvement was introducing tens of thousands of synthetic test cases, the Juliet 1.0 test suite, with precisely characterized weaknesses. Other improvements included a better procedure for characterizing vulnerabilities locations in the test cases selected based on entries in the Common Vulnerabilities and Exposures (CVE) dataset and providing teams with a virtual machine image containing the test cases properly configured and ready for analysis by tools. This paper identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software is available for empirical research. Second, our analysis of tool reports indicates kinds of weaknesses that exist in the software and that are reported by the tools. Third, the CVE-selected test cases contain exploitable vulnerabilities found in practice, with clearly identified locations in the code. Fourth, tool outputs for Juliet cases provide a rich set of data amenable to mechanical analysis. Finally, the analysis may be used as a basis for a further study of the weaknesses in the code and of static analysis.
Citation: Special Publication (NIST SP) - 500-297
NIST Pub Series: Special Publication (NIST SP)
Pub Type: NIST Pubs
Software security, static analysis tools, security weaknesses, vulnerability