Details of Core Tests Reviewed to Determine Competency of Voting Systems Testing Laboratories
1. Review the technical data package for a vendor's voting system for conformance to federal standards. This requires evaluation of:
a. Overall system design, including subsystems, modules, and the interfaces among them.
b. Specific functional capabilities provided by the system.
c. Performance and design specifications.
d. Design constraints, applicable standards, and compatibility requirements.
e. Personnel, equipment, and facility requirements for system operation, maintenance, and logistical support.
f. Vendor practices for assuring system quality during the system's development and subsequent maintenance.
g. Vendor practices for managing the configuration of the system during development and for modifications to the system throughout its life cycle.
2. Conduct a physical configuration audit; that is, compare the voting system components submitted for qualification to the vendor's technical documentation. This comparison involves:
a. Establishing a configuration baseline of the software and hardware to be tested.
b. Confirming whether the vendor's documentation is sufficient for the user to install, validate, operate, and maintain the voting system.
c. Examining the vendor's source code to verify that the software conforms to the vendor's specifications.
d. Reviewing all drawings, specifications, technical data, and test data associated with the system hardware, if the software is to be run on any equipment other than a commercial off the shelf (COTS) mainframe data processing system, minicomputer, or microcomputer.
e. Assessing user acceptance test procedures and data, based on a review of vendor documents containing this information against the system's functional specifications.
f. Re-examining any changes to the baseline software configuration made during the course of testing. All changes to the system hardware that may produce a change in software operation are also subject to re-examination.
g. Assessing if the voting system meets accessibility requirements.
3. Conduct a source code review, which means:
a. Inspecting the source code to determine that the software works correctly and that there are no security glitches.
b. Verifying that the code is unmodified and that the default configuration options have not been changed.
c. Confirming that user selections and configuration changes are correct.
4. Conduct a functional configuration audit, which means:
a. Verifying every system function and combination of functions cited in the vendor's documentation.
b. Verifying the accuracy .and completeness of the system's Voter Manual, Operations Procedures, Maintenance Procedures, and Diagnostic Testing Procedures.
5. Conduct a system integration test, which means:
a. Designing and performing procedures that test the voting system capabilities for the system as a whole, to ensure that the system meets requirements defined in the VVSG 1.0 (2005).
b. Developing and implementing software test cases to be executed as part of functional testing of the system.
c. Establishing an operational profile of the common procedures, sequencing, and options among both general system requirements and those that are specifically recognized and supported by the vendor. Any portions of the source code not covered as part of this profile must be identified.
6. Conduct reliability and accuracy tests, which mean:
a. Verifying the ability of the voting system to record votes accurately at its maximum rated processing volume for a specified period of time.
b. Developing and implementing tests to assess voting system acceptance or rejection based on the number of relevant failures during equipment operation.
c. Verifying the vendor's mathematical model for predicting accuracy of the voting system.
7. Conduct security tests, which mean:
a. Verifying that access control mechanisms successfully mediate read and write access to system functions and data according to system documentation.
b. Verifying that system logs record key security functions according to system documentation.
c. Verifying that cryptographic functions can be used according to system documentation.