NVLAP Suspends Accreditation of SysTest Labs 10-28-08

October 28, 2008

Mark Phillips
Vice President of Compliance Services
SysTest Labs, Incorporated
216 16th Street, Suite 700
Denver, CO 80202-5115

NVLAP Lab Code 200733-0

Dear Mr. Phillips,

On behalf of the National Voluntary Laboratory Accreditation Program (NVLAP), I write to notify of you of NVLAP's decision to suspend its accreditation of SysTest's electronic voting testing program pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10. This letter provides an explanation of NVLAP's decision and describes the steps SysTest can take to reinstate its accreditation.

This action pertains to voting systems under review by SysTest to be recommended for certification by the Election Assistance Commission for future elections and is not pertinent to systems already deployed for the 2008 election which were certified under alternate systems.

Background Discussion

SysTest Labs, Incorporated is currently accredited by the National Voluntary Laboratory Accreditation Program (NVLAP), a program within the National Institute of Standards and Technology (NIST), to perform testing to federal standards in accordance with the Help America Vote Act of 2002 (HAVA). These standards are the 2002 Voting System Standards (VSS-2002) and the 2005 Voluntary Voting System Guidelines (VVSG-2005). On August 8, 2008, NVLAP sent SysTest Labs a letter outlining specific concerns with respect to SysTest's NVLAP-accredited testing of voting systems, including voting system test campaigns submitted to the Election Assistance Commission (EAC) under their voting system certification process. These specific concerns are documented in the March 2008 NVLAP on-site assessment checklist, produced as part of the normal reassessment process, and in communications between the EAC and NIST regarding issues that EAC staff identified with test reports submitted by SysTest Labs (enclosed). The August 8th letter (also enclosed) outlined three specific concerns. In short they were:

1. SysTest's lack of properly documented and validated test methods.   
2. Testing conducted by unqualified or untrained personnel.   
3. Improper assurances made to manufacturers regarding testing outcomes.

NVLAP directed SysTest to submit information to NVLAP, including a schedule of all accredited voting systems testing planned, within 14 days of receipt of the August 8th letter. NVLAP informed SysTest of its intention to conduct on-site monitoring of the testing of electronic voting machines. SysTest was notified by email on October 6, 2008 of NVLAP' s intention to visit their lab on October 14th through 16th to observe testing that had been scheduled during that period.

NVLAP assembled a team consisting of the NVLAP voting system technical assessor, the NIST/NVLAP program manager for voting system testing and four members of the NIST Information Technology Laboratory (ITL) involved in writing the federal voting system standards. In addition, two EAC staff members were invited to provide their observations. During the on-site visit this eight-member team witnessed several tests, interviewed testers, and examined documents related to the areas of concern.

Site Visit Observations

As a result of this on-site monitoring visit, NVLAP has serious concerns about SysTest's performance of voting system testing. These concerns were supported by observations of testing where the test methods being used were not fully developed, validated, mapped to the requirements of the applicable standards, and controlled under SysTest's document control policy.

From the team's observations it was unclear who at SysTest had the ultimate responsibility for test method development. During the observed tests, it appeared that the testers were running the tests for the first time. Changes were made to the test procedures to address items that should have been caught during an initial run-through of the test. Basic tests, such as the system readiness test, were not conducted successfully. Three test methods failed due to problems with the procedure, tester error, or unfamiliarity with the test set-up. Some anomalies or potential problems during testing were not reported by the testers but were pointed out by members of the on-site team.

During the team's visit SysTest personnel stated that their policy was to validate test methods during the actual testing of voting equipment. This approach is unacceptable. The lab must validate all test methods separate from actual testing so that equipment nonconformance can be isolated from test method problems. This validation must follow set documented procedures and show a clear chain of responsibility for the process.

SysTest has undergone numerous changes in personnel since its original accreditation and, in fact, since the March 2008 NVLAP on-site assessment. SysTest staff conducting testing during the monitoring visit demonstrated a lack of familiarity with the test equipment and procedures. Some personnel who participated in past on-site assessments were no longer associated with the NVLAP-accredited testing; they had been reassigned to work in support of state certification of voting systems. SysTest management's stated goal was to transfer the expertise and testing approach from their New York testingcampaign to the NVLAP/EAC accredited testing campaign. SysTest must improve the level of training of personnel involved in NVLAP/EAC accredited testing given that SysTest has reassigned experienced testers to other work. SysTest should consider bringing in outside instructors to train laboratory personnel.

SysTest was advised that an appearance of impropriety had occurred in a case where personnel had given a client an indication that their equipment would successfully pass testing. SysTest's response was that this was an isolated incident and the person involved had not intended to give this impression. SysTest further stated that their employees were given a quiz which they felt covered training in this situation. It is NVLAP's position that this quiz is insufficient and SysTest must provide specific training to their employees on professional ethics and document the employees' intent to adhere to SysTest's stated policy.

NVLAP's Decision

Pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10, NVLAP hereby suspends SysTest's accreditation effective as of the date of this letter. SysTest Labs, Incorporated is prohibited from using the NVLAP symbol on its test reports, correspondences, and advertising during the suspension period for all voting system testing. Accreditation may be reinstated only after such time that SysTest can demonstrate voting system testing in accordance with the requirements of the applicable voting system standards and NIST Handbook 150. This demonstration must be achieved through an on-site visit to SysTest to witness testing, review documentation, interview personnel, and any other means necessary to gather objective evidence in support of a decision regarding reinstatement.

This on-site visit will occur only after NVLAP is convinced, through the submission of documentation, that SysTest has taken the necessary steps to correct the areas of nonconformance herein addressed. This documentation will include, but is not limited to: procedures for test method development; procedures for test method validation; revised document control procedures that specifically address technical procedures; fully developed test methods showing validation, document control, and mapping to the federal voting system standards; and, procedures or policies that address methods by which SysTest will control statements or assurances to their clients regarding the outcome of voting system testing.

SysTest was accredited by NVLAP based on its ability to develop and perform competent testing within the framework of an effective management system. SysTest now needs to revise its management system to correct the nonconformances found during this visit and implement these system changes. NVLAP believes that the current SysTest management team is committed to accomplishing this goal and will work with them to that end.

Sincerely,

Jon Crickenberger
NIST/NVLAP Program Manager

Enclosures

Cc: Brian Hancock, Election Assistance Commission