SATE 2009 Data

If you have questions, comments, or suggestions, email Vadim Okun -
vadim.okun@nist.gov

For details of SATE, see the NIST Special Publication (SP) 500-287:
The Second Static Analysis Tool Exposition (SATE) 2009. Vadim Okun,
Aurelien Delaitre, and Paul E. Black.

Please read the file CAUTIONS.txt (in directory sate_analysis)
which describes the important limitations of our analysis.

I. Directory Structure Overview

The three subdirectories are as follows:

1. sate_tool_reports - tool reports in the SATE format

2. sate_tool_analysis - analysis of the tool warnings

3. sate_additional - additional information submitted by the
participating teams

II. Directory Structure Details

1. sate_tool_reports

The directory sate_tool_reports contains subdirectories for each
participating team.

Each team directory contains tool reports - files with extension
.xml and test case name in the file name. The XML schema file
for the tool output format, sate_2009.xsd, is in the directory
sate_analysis. The schema file can be used for validation, for
example:

xmllint --schema sate_analysis/sate_2009.xsd 
sate_tool_reports/sofcheck/dmdirc_sate_messages.xml --noout

Most directories also contain supporting documentation, e.g.,
description of environment and options used.

2. sate_analysis

The directory sate_analysis contains the following:

* The directory "analysis_latest" - our latest analysis (April 2010):

- Tool reports with our analysis of selected tool warnings.  The file
names are, e.g., tool_testcase.xml.
- Lists of associations for each test case. The file names are, e.g.,
testcase_assoc.xml.

* The directory "analysis_2009_10_23" contains the previous version of
our analysis (as of October 23, 2009).

* The directory "reanalysis_changes" contains the changes that we made
during reanalysis, that is, changes between analysis_2009_10_23 and
analysis_latest.

- changes_final.txt - changes in the analysis of correctness
- changed_assoc.txt - associations added and removed during reanalysis

* analysis_by_teams - analysis of tool's reports by the tool maker
(several teams returned review of their tool's reports - optional
step of the SATE 2009 protocol).

* manual_findings - reports with manual findings by security consultants
for two test cases (Roller and IRSSI), and our analysis matching tool
warnings to the manual findings.

* sate_2009.xsd - SATE tool output schema.

* sate_2009.check_path.xsd - SATE tool output schema with a check for
path format.

* sate_2009_eval.xsd - SATE analysis schema. It is derived from
the SATE tool output schema.

* sate_2009_assoc.xsd - SATE association schema.

* CAUTIONS.txt - cautions on interpreting and using the SATE data

* sate2009_report_weakness_categories.txt - the detailed list of
weakness categories used for presenting the SATE data in our
report, NIST SP 500-287.

The guidelines for analysis of correctness and associating warnings
are here:

http://samate.nist.gov/SATE2009.html#Guidelines

3. sate_additional

The directory sate_additional contains (in directories named after
individual participants) additional information, if any, submitted
by the participants. In particular:

* Reports in the tool's original format - subdirectories for each
team that submitted the reports in the original format.

* An updated run of Grammatech CodeSonar - run2. The CodeSonar results
from the first run (run1) for IRSSI had several false warnings due to
a tool configuration error. We analyzed the warnings from run1.
Later, Grammatech submitted the updated run with the tool configured
correctly. The updated run is under sate_additional/grammatech/run2.

