If you have questions, comments, or suggestions, email Vadim Okun -
vadim.okun@nist.gov

For details of SATE, see the NIST Special Publication 500-279:
Static Analysis Tool Exposition (SATE) 2008, editors: V. Okun,
R. Gaucher, P. E. Black.

Please read the file CAUTIONS.txt which describes the important
limitations of our analysis.

The directories [participant]-eval-report contain tool reports
augmented with our analysis. See the description of the evaluated
tool output format below.

The directory sate_assoc contains lists of associations for each
test case.

See sate_analysis_criteria.txt for a description of our criteria
for analysis.

I. Evaluated tool output format

The evaluated tool output format (XML schema - sate_eval.xsd),
which includes our analysis of tool warnings, has several fields
in addition to the tool output format (XML schema - sate.xsd).
Specifically, each warning has another id (UID), which is unique
across all tool reports. Also, the evaluation section has these
additional optional fields:

* Confirmed - "yes" means that the human determined that the
warning is correctly reporting the problem.

* Stage - a number that roughly corresponds to the step of the
SATE procedure:

  ** Stage 3 - (optional) participants' review of their own tool's
report.

  ** Stage 4 - review by the SATE analysts.

  ** Stage 5 - (optional) corrections by the participants. No
participant submitted corrections in the xml format at that stage;
however, Grammatech submitted a detailed document with corrections
to our analysis of their tool's warnings.

  ** Stage 6 - updates by the SATE analysts.

* Author - author of the evaluation. For each warning, the
evaluations by SATE analysts were combined together and a generic
name - "evaluators" - was used.

Additionally, the evaluated tool output format allows for more than
one evaluation section per warning.

II. Association List Format

The association list consists of sets of unique warning ids (UID),
where each set represents a group of associated warnings.  There is
one list per test case. Each set occupies a single line, which is a
tab separated list of UIDs. For example, if we determined that UID
441, 754, and 33201 refer to the same weakness, we associated them.
They are represented as:

441	754	33201

III. Notes about stages of analysis

* Most of the evaluations from stage 3 (by participants) include
the "falsepositive" field.

* Our evaluations (stages 4 and 6) include the "confirmed" field
instead.

* Most of the stage 6 evaluations are based on the corrections of
our stage 4 evaluations by the participants.

* Whenever both stage 4 and stage 6 evaluations exist for a
weakness, the stage 6 evaluation overrides the stage 4 evaluation.

IV. Changes to tool output

* We converted the file paths used in the location element to a
common form.

* Aspect Security combined different weaknesses of the same kind
into a single weakness. For our analysis, we separated them into
separate weaknesses. We converted the weakness ids, so that, for
example, for weakness id 17 which combined 3 different weaknesses,
we created 3 entries with ids 1701, 1702, 1703.

* For Grammatech CodeSonar, the reports with evaluations omit the
xmloutput section.

* For Grammatech CodeSonar, the tool did not assign severity to
the warnings, so we updated the severities for some weakness
classes.

* For Fortify SCA, we did not analyze the reports which were
generated with the -findbugs option.

