Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Other Assurance Tool Test Collections
[SAMATE Home | IntrO TO SAMATE | SARD | SATE | Bugs Framework | Publications | Tool Survey | Resources]
DISCLAIMER: Certain trade names and company products are mentioned in the text or identified. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology (NIST), nor does it imply that the products are necessarily the best available for the purpose.
By selecting almost any of these links, you will be leaving NIST webspace. We provide these links because they may have information of interest to you. No inferences should be drawn because some sites are referenced, or not, from this page. There may be other web sites that are more appropriate for your purpose. NIST does not necessarily endorse the views expressed, or concur with the assertions presented on these sites. Further, NIST does not endorse any commercial products that may be mentioned on these sites.
In addition to the SARD, we provide our users with a list of other assurance tool collections and benchmarks that we are aware of. Test collections in this list must be designed to assess the capabilities of assurance tools. Tool test collections can include requirements analysis tool tests, design model analysis tool tests, source code analysis tool tests, static and dynamic binary analysis tool tests. Test collections on this list may include a harness or a test framework.
- Software-artifact Infrastructure Repository (SIR): This repository provides Java and C programs for use in experimentation with testing and analysis techniques, and materials facilitating that use. The primary purpose of the tests and testing framework is generation of experiments in software fault testing. That said, extension of the test suites and test objects beyond its current capability is encouraged by the developers.
- Test Suite Type: Source Code for Specific Versions of Open Source Applications
- Number of Programs: 85
- Number of Bugs: 572+
- Average Number of Lines of Code: 38,825
- Language: C, Java, C++, C#
- Supports Multiple Versions of Test Cases: Yes
- Test Harness: Yes
- FaultBench: is a set of real, subject Java programs for comparison and evaluation of actionable alert identification techniques (AAITs) that supplement automated static analysis.
- Test Suite Type: Source Code (via CVS) for Multiple Versions of Open Source Applications
- Number of Programs: 6
- Number of Bugs: 780
- Average Number of Lines of Code: 4973
- Language: Java
- Supports Multiple Versions of Test Cases: Yes
- Test Harness: No
- Documentation: Sarah Heckman and Laurie Williams, "On Establishing a Benchmark for Evaluating Static Analysis Alert Prioritization and Classification Techniques", Proc. Second Int'l Symposium on Empirical Software Engineering and Measurement, (ESEM 2008), October 9-10, 2008, Kaiserslautern, Germany, DOI: 10.1145/1414004.1414013
- OWASP Benchmark Project is a suite of synthetic test cases designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. It includes both test cases with weaknesses and without weaknesses (to test for false positives). The test suite is updated periodically.
- Test Suite Type: Source Code
- Number of Programs: Over 2500
- Number of Bugs: Over 1300
- Average Number of Lines of Code: 53 (As of version 1.2)
- Language: Java
- Supports Multiple Versions of Test Cases: Yes
- Test Harness: Yes
If you want to suggest other tool testing resources for inclusion on this list, please contact samate [at] nist.gov (SAMATE).