Source Code Analyzer Tool Assessment Guide and Test Suite

Notes to Reviewers
Source Code Analyzer Tool Assessment Guide and  Test Suite for the VVSG-NI, Version 1.0
April 1, 2009

The documents available from this page represent a source code analyzer tool guide and test suite for use by voting system testing labs as well as manufacturers of voting systems. Please note that this is NOT a guide or test suite for determining a voting system's conformance to the VVSG. It is for tool calibration and assessment only and is being made available to assist test labs and voting system software manufacturers in understanding, calibrating and using automated source code analysis tools against coding requirements prescribed in VVSG.

Test suite reviewers are advised to first read and understand the relevant material in the VVSG 2005 and the VVSG-NI relevant to the test suites under review, before reviewing the test suites. The following VVSG sections provided the technical requirements information needed to create this guide and tool test suite:

VVSG 2005:

• Volume I Section 5.2, Software Design and Coding Standards
• Volume II Section 5, Software Testing

VVSG-NI:

• Part 3 Section 4.5.1.A and 4.5.1.B, Source Code Workmanship Requirements

A complete version of the VVSG 2005 in PDF format can be found at:
http://www.eac.gov/testing_and_certification/2005_vvsg.aspx

A complete version of the VVSG-NI in HTML, MS-Word, or PDF formats can be found at http://www.eac.gov/vvsg. The source code workmanship requirements of Part 3 Section 4 can be found at:
http://www.eac.gov/vvsg/part_3_testing_requirements/chapter_4_documentation_and_design_reviews.aspx

Source code analysis is part of the "due diligence" performed by test labs in compliance with VVSG testing requirements. While still a human-intensive effort, static source code analysis today is augmented with automated tools that provide greater confidence that source code is examined in a thorough, reliable and repeatable way.

This tool guide provides a general overview of source code analysis tools, and is accompanied by tool tests (source code examples) that labs can use to calibrate those tools against VVSG coding requirements. The tool tests provided with this tool guide represent an initial collection written in C, C++ and Java languages. As this effort moves forward, additional tool tests in these languages and others will be added to strengthen the tool calibration procedures of test labs.

Commenting:

Please send comments on the test suites, by July 1, 2009, to: tools-test [at] nist.gov (subject: Source%20Code%20Analyzer%20Tool%20Assessment%20Guide%20and%20Test%20Suite%20Version%201.0%20for%20the%20VVSG%202005%20and%20VVSG-NI) (tools-test[at]nist[dot]gov).

You may provide comments directly in your email and/or send attachments in MS-Word or PDF. If you wish, you may embed your comments within the PDF documentation using the instructions provided here. In general, please tell us the features you like and provide us with comments, corrections, and suggestions on how to improve the test suites. Please provide the following items:

• Test suite version number (found in the test suite documentation, currently Version 1.0)
• Your name and affiliation (include contact information if desired)
• Identification of the particular tests for which your comment applies
• If including suggestions for changes to the tests, a description of the suggested change including an adequate justification for the change, or a draft replacement for the test including the justification and any other necessary documentation or commentary

All comments will be considered. After all comments have been received and incorporated into the test suites, a new version of the test suites will be posted on the NIST web site.

The source code analyzer tool guide and test suite is available is available here in a

• zip archive
• compressed tar archive