SATE 2008 Data

If you have questions, comments, or suggestions, email Vadim Okun -
vadim.okun@nist.gov

For details of SATE, see [1].

Please read the file CAUTIONS.txt (in directory sate_analysis)
which describes the important limitations of our analysis.

I. Directory Structure Overview

The three subdirectories are as follows:

1. sate_tool_reports - tool reports in the SATE format

2. sate_tool_analysis - our analysis of the tool reports

3. sate_additional - additional information submitted by the
participants

II. Directory Structure Details

1. sate_tool_reports

The directory sate_tool_reports contains two subdirectories (C
and Java) for the two SATE tracks. Each track directory contains
subdirectories for the tool makers that participated in that
track. Since Fortify and Veracode participated in both tracks,
there are subdirectories fortify and veracode under both C and
Java directories.

Each participant directory contains tool reports - files with
extension .xml and test case name in the file name. The XML schema
file for the tool output format, sate.xsd, is in the directory
sate_analysis. The schema file can be used for validation, for
example:

xmllint --schema sate_analysis/sate.xsd 
sate_tool_reports/C/flawfinder/lighttpd.flawfinder.xml

Most directories also contain supporting documentation, e.g.,
description of environment and options used.

Note that the directory for sofcheck contains two reports for
MvnForum (mvnforum_report.xml and myvietnam_report.xml), which
are outputs from the runs on the two subtrees of the test case.

Note: because of format conversion errors the warnings in the
Fortify reports generated with the -findbugs option (e.g.,
opennms_findbugs.xml) may contain incorrect weakness path
information. The corresponding files in the original Fortify
format (e.g., opennms_findbugs.fvdl) which contain the correct
weakness path information can be found in the directory
sate_additional/fortify.

2. sate_analysis

The directory sate_analysis contains the following:

* Our analysis of tool reports including

  ** Tool output with our analysis of tool warnings (in the
directories named after individual participants)

  ** Association lists in the directory sate_assoc

* sate.xsd - SATE tool output schema

* sate_eval.xsd - SATE analysis schema

* README - includes description of the output format, stages  of
analysis, changes that we made to the tool output

* CAUTIONS.txt - cautions on interpreting and using the SATE data

* sate_analysis_criteria.txt - a description of our criteria for
analysis.

* sate_report_weakness_categories.txt - the detailed list of
weakness categories used for presenting the SATE data in our
report, "Review of the first Static Analysis Tool Exposition
(SATE 2008)", which can be found in [1].

3. sate_additional

The directory sate_additional contains (in directories named after
individual participants) additional information, if any, submitted
by the participants. In particular:

* Several participants submitted reports in the tool's original format.
These reports can be found in subdirectories checkmarx, devinspect,
fortify, grammatech.

* Analysis of tool's reports by the tool maker (sofcheck, findbugs).

* The output from a later version of the tool: Findbugs v. 1.3.3
(submitted on March 19, 2008), Sofcheck v. 2.1.3 (submitted on March
9, 2008).

* The output from a run of Grammatech CodeSonar configured with
MALLOC_FAILURE_BEHAVIOR = DOESNT_FAIL option
(CodeSonar_tool_reports4.tar.gz).

III. References

[1] NIST Special Publication 500-279: Static Analysis Tool Exposition
(SATE) 2008, editors: V. Okun, R. Gaucher, P. E. Black.

