SATE 2010 Data

If you have questions, comments, or suggestions, email Vadim Okun -
vadim.okun@nist.gov

For details of SATE, see the NIST Special Publication (SP) 500-283:
Report on the Third Static Analysis Tool Exposition (SATE) 2010,
Vadim Okun, Aurelien Delaitre, and Paul E. Black, editors.

Please read the file CAUTIONS.txt (in directory sate_analysis)
which describes the important limitations of our analysis.

Note. Per requests by Coverity and Grammatech, their tool output is not
released as part of SATE data.  Consequently, our detailed analysis of
their tool warnings is not released either.  However, the observations
and summary analysis in our paper (published as part of NIST SP 500-283)
are based on the complete data set.

I. Directory Structure Overview

The subdirectories are as follows:

1. sate_tool_reports - tool reports in the SATE format

2. sate_analysis - analysis of the tool warnings selected randomly,
based on CVEs, and based on manual findings.

3. sate_schemas - XML schema files for the SATE tool output format

4. sate_additional - additional information submitted by the teams

II. Directory Structure Details

1. sate_tool_reports

The directory sate_tool_reports contains subdirectories for each
participating team.

Each team directory contains tool reports - files with extension
.xml and test case name in the file name. The XML schema file
for the tool output format is in the directory sate_analysis.

Most directories also contain supporting documentation, e.g.,
description of environment and options used.

2. sate_analysis

The directory sate_analysis contains the following files and
subdirectories:

2.1. CAUTIONS.txt - cautions on interpreting and using the SATE data

2.2. sate2010_report_weakness_categories.cfg - the detailed list of
weakness categories used for presenting the SATE data in our
report (published as part of NIST SP 500-283).

2.3. warnings.txt - a list of tool warnings. For each warning, it provides
tool name, test case name, warning id, unique warning id (uid), and a flag
specifying whether the warning was selected for analysis randomly.

2.4. analysis_subset - our analysis of a subset of tool warnings
selected for analysis randomly:

* Tool reports with our analysis of selected tool warnings.  The file
names are, e.g., tool_testcase.xml.

* Lists of associations for each test case. The file names are, e.g.,
testcase_assoc.xml.

Note. We did not analyze any warnings from one of the tools, Marfcat.

* analysis_changes.txt - the list of warnings for which we changed our
analysis of correctness after September 22, 2010

The guidelines for analysis of correctness and associating warnings
can be found in Section 2.7 of our SATE 2010 report (published as part
of NIST SP 500-283).

2.5. analysis_by_teams contains subdirectories for those teams that
returned analysis of their tool's reports.  Specifically, as an optional
step of the SATE 2010 protocol, some teams returned their reviews of the
selected warnings from their tool reports and/or reviews of their tool's
reports for the CVE-selected test cases.

Note. Two teams, Marfcat and Red Lizards Goanna, describe their tool's
reports in the papers published as part of NIST SP 500-283.

2.6. CVEs - information about CVEs in the CVE-selected test cases

Directory sate_analysis/CVEs contains the following files and
subdirectories.

Note. Three CVEs (CVE-2008-5519, CVE-2007-6286, CVE-2006-7197) applied
exclusively to the C code portion of Tomcat.  Since Tomcat was in the
Java track, the numbers in our report do not include these CVEs.

* The XML files with lists of CVE locations in the vulnerable test cases

cve_wireshark-cve.xml
cve_chrome-cve.xml
cve_tomcat-cve.xml

* Matching tool warnings

We did not find any matching tool warnings for CVEs in Wireshark and Chrome.
We found matching tool warnings for 4 CVEs in Tomcat. The report for Tomcat
augmented with our analysis - matching tool warnings to the CVEs:

cve_tomcat-cve-eval.xml

The lists are in SATE format with the following additional attributes for
location:

- length - number of lines in the block of code for the CVE,
- type - one of:
	"fix" - location where the code was fixed
	"sink" - final step of the data/control flow
	"path" - location on the data/control flow leading to the sink

The matching tool warnings are listed using the tag <related>.

sate_2010_cve.pathcheck.xsd is an XML schema file for the CVE location
lists. It can be found in the top level directory sate_schemas.

* The subdirectory additional_cve_info contains the following files with
additional information about the CVEs:

- XML files with lists of CVE locations augmented with code fragments for
each location:

cve_wireshark-cve-snippets.xml
cve_chrome-cve-snippets.xml
cve_tomcat-cve-snippets.xml

- patches.txt - bug and patch information that we collected for the CVEs
in Wireshark and Chrome.

- cve2cwe.txt - a tab-separated list of CVE, CWE, test case

- wireshark_cve_summary.txt - a tab-separated list of CVEs with CWE id,
a short description of CVE and our opinion about how easily the tools
can detect the CVE.

* Summary of changes made during reanalysis of the CVE-selected test cases

- Updated line number for CVE-2010-2283
- Updated description for CVE-2010-2284
- CVE-2010-2304 was replaced by CVE-2010-1773 in the CVE and NVD databases
- CVE-2010-2303 was replaced by CVE-2010-1772 in the CVE and NVD databases
- CVE-2010-2302 - Updated the description, it was a copy-paste mistake
- Updated line numbers and description for CVE-2009-3243
- For Tomcat, we originally marked a warning (UID 13217) as related to
CVE-2008-0128. During reanalysis, we determined that the warning was not
related to the CVE.

2.7. manual_findings - reports with manual findings by security consultants.
The consultants analyzed the two general test cases: Pebble and Dovecot.

Pebble test case (ver. 2.5-M2) had an implementation bug (an
attempt to mitigate CSRF) that prevented many features from
working properly. The consultants analyzed the latest version
(as of the last week of August 2010).  The only change in the
latest version was removal of CSRF protection.

pebble_cigital.xml contains the report for Pebble, with matched
warnings from tools (using the tag <related>).  Additionally,
Pebble was found to be vulnerable to session fixation attacks
(not included in the report).

For Dovecot, the security consultants did not find any weaknesses
exploitable by an external attacker. Dovecot_Assessment.pdf
describes the assessment process used for Dovecot.

3. sate_schemas

* sate_2010.xsd - SATE tool output schema.

The schema file can be used for validation, for example:

xmllint --schema sate_schemas/sate_2010.xsd sate_tool_reports/cppcheck/cppcheck_dovecot.xml --noout

* sate_2010.pathcheck.xsd - SATE tool output schema with a check for
path format.

* sate_2010_eval.xsd - SATE analysis schema. It is derived from
the SATE tool output schema.

* sate_2010_cve.pathcheck.xsd - schema for CVE location lists. Also derived
from the SATE tool output schema.

* sate_2010_assoc.xsd - SATE association schema.

4. sate_additional

The directory sate_additional contains additional information, if any,
submitted by teams.  Specifically, several teams submitted the original
reports from their tool, in addition to the reports in the SATE output
format. The original reports from a team can be found in a subdirectory
named after the team.

