Reinstatement of Accreditation of SysTest Labs, Incorporated

February 26, 2009

Mark Phillips
Vice President of Compliance Services
SysTest Labs, Incorporated
216 16th Street, Suite 700
Denver, CO 80202-5115

NVLAP Lab Code 200733-0

Dear Mr. Phillips,

On behalf of the National Voluntary Laboratory Accreditation Program (NVLAP), I write to notify you of NVLAP's decision to reinstate SysTest Labs Incorporated's accreditation in the NVLAP electronic voting systems testing program. The reinstatement is effective on the date of this letter.

In a letter to you dated October 28, 2008, NVLAP suspended the accreditation of SysTest Labs for voting systems testing. This suspension was made pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10. The suspension was based on nonconformities that had been sited from the last NVLAP on- site assessment visit in March 2008, findings from the Election Assistance Commission (EAC), and observations made during an October 2008 monitoring visit. These issues and findings were documented in the October 28th letter (enclosed).

After receiving the suspension letter, SysTest provided materials to NVLAP which focused on procedures and documentation in response to:

1. SysTest's lack of properly documented and validated test methods. 
2. Testing being conducted by unqualified or untrained personnel.   
3. Improper assurances made to manufacturers regarding testing outcomes.

SysTest Labs received a second monitoring visit by NVLAP on February 18th, 19th, and 20th, 2009, to assess the results of changes implemented and documented since the suspension letter. The visiting team consisted of two members of NVLAP and three from the EAC. The team interviewed SysTest employees involved in voting systems testing, assessed new and existing documentation, and witnessed testing.    As a result of this visit NVLAP found the following, which are numbered in relation to the above issues:

1. SysTest was able to show documented test methods and validations that comply with the voting standards. Specifically, SysTest was able to show the documentation of test modules that are used together to comprise a testing campaign and a process for validating said modules. The assessment team witnessed the performance of testing, recording of results, and reviewed the process flow and documentation.   
2. SysTest has made changes in its personnel that have resulted in a more competent testing team and was able to show evidence of new training and its effectiveness. Training courses and quizzes have been developed to document the competency of those involved in testing.   
3. SysTest was able to show that personnel had undergone training on professional ethics and signed a document attesting to the fact that they were aware of the factors involved in ethical behavior and the consequences that could ensue if a breach of ethics occurred. Although no direct breach of ethics had been alleged, NVLAP had been concerned about the appearance of unethical communications with lab customers.

SysTest was able to demonstrate through documentation and observations that they have corrected the nonconformities that lead to their suspension. NVLAP understands that SysTest has undergone some major changes in documentation and personnel and feels that the current team has done a great deal of work to reinvent some of the processes involved in the NVLAP accredited testing. SysTest has been successful in demonstrating their competency and NVLAP believes that the current technical and management team has ownership in the new processes and will work with NVLAP to ensure continued compliance.

Sincerely,

Jon Crickenberger
NIST/NVLAP Program Manager

Enclosure

Cc: Brian Hancock, Election Assistance Commission

+++++++++

October 28, 2008

Mark Phillips
Vice President of Compliance Services
SysTest Labs, Incorporated
216 16th Street, Suite 700
Denver, CO 80202-5115

NVLAP Lab Code 200733-0

Dear Mr. Phillips,

On behalf of the National Voluntary Laboratory Accreditation Program (NVLAP), I write to notify of you of NVLAP's decision to suspend its accreditation of SysTest's electronic voting testing program pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10. This letter provides an explanation of NVLAP's decision and describes the steps SysTest can take to reinstate its accreditation.

This action pertains to voting systems under review by SysTest to be recommended for certification by the Election Assistance Commission for future elections and is not pertinent to systems already deployed for the 2008 election which were certified under alternate systems.

Background Discussion

SysTest Labs, Incorporated is currently accredited by the National Voluntary Laboratory Accreditation Program (NVLAP), a program within the National Institute of Standards and Technology (NIST), to perform testing to federal standards in accordance with the Help America Vote Act of 2002 (HAVA). These standards are the 2002 Voting System Standards (VSS-2002) and the 2005 Voluntary Voting System Guidelines (VVSG-2005). On August 8, 2008, NVLAP sent SysTest Labs a letter outlining specific concerns with respect to SysTest's NVLAP-accredited testing of voting systems, including voting system test campaigns submitted to the Election Assistance Commission (EAC) under their voting system certification process. These specific concerns are documented in the March 2008 NVLAP on-site assessment checklist, produced as part of the normal reassessment process, and in communications between the EAC and NIST regarding issues that EAC staff identified with test reports submitted by SysTest Labs (enclosed). The August 8th letter (also enclosed) outlined three specific concerns. In short they were:

1. SysTest's lack of properly documented and validated test methods.   
2. Testing conducted by unqualified or untrained personnel.   
3. Improper assurances made to manufacturers regarding testing outcomes.

NVLAP directed SysTest to submit information to NVLAP, including a schedule of all accredited voting systems testing planned, within 14 days of receipt of the August 8th letter. NVLAP informed SysTest of its intention to conduct on-site monitoring of the testing of electronic voting machines. SysTest was notified by email on October 6, 2008 of NVLAP' s intention to visit their lab on October 14th through 16th to observe testing that had been scheduled during that period.

NVLAP assembled a team consisting of the NVLAP voting system technical assessor, the NIST/NVLAP program manager for voting system testing and four members of the NIST Information Technology Laboratory (ITL) involved in writing the federal voting system standards. In addition, two EAC staff members were invited to provide their observations. During the on-site visit this eight-member team witnessed several tests, interviewed testers, and examined documents related to the areas of concern.

Site Visit Observations

As a result of this on-site monitoring visit, NVLAP has serious concerns about SysTest's performance of voting system testing. These concerns were supported by observations of testing where the test methods being used were not fully developed, validated, mapped to the requirements of the applicable standards, and controlled under SysTest's document control policy.

From the team's observations it was unclear who at SysTest had the ultimate responsibility for test method development. During the observed tests, it appeared that the testers were running the tests for the first time. Changes were made to the test procedures to address items that should have been caught during an initial run-through of the test. Basic tests, such as the system readiness test, were not conducted successfully. Three test methods failed due to problems with the procedure, tester error, or unfamiliarity with the test set-up. Some anomalies or potential problems during testing were not reported by the testers but were pointed out by members of the on-site team.

During the team's visit SysTest personnel stated that their policy was to validate test methods during the actual testing of voting equipment. This approach is unacceptable. The lab must validate all test methods separate from actual testing so that equipment nonconformance can be isolated from test method problems. This validation must follow set documented procedures and show a clear chain of responsibility for the process.

SysTest has undergone numerous changes in personnel since its original accreditation and, in fact, since the March 2008 NVLAP on-site assessment. SysTest staff conducting testing during the monitoring visit demonstrated a lack of familiarity with the test equipment and procedures. Some personnel who participated in past on-site assessments were no longer associated with the NVLAP-accredited testing; they had been reassigned to work in support of state certification of voting systems. SysTest management's stated goal was to transfer the expertise and testing approach from their New York testingcampaign to the NVLAP/EAC accredited testing campaign. SysTest must improve the level of training of personnel involved in NVLAP/EAC accredited testing given that SysTest has reassigned experienced testers to other work. SysTest should consider bringing in outside instructors to train laboratory personnel.

SysTest was advised that an appearance of impropriety had occurred in a case where personnel had given a client an indication that their equipment would successfully pass testing. SysTest's response was that this was an isolated incident and the person involved had not intended to give this impression. SysTest further stated that their employees were given a quiz which they felt covered training in this situation. It is NVLAP's position that this quiz is insufficient and SysTest must provide specific training to their employees on professional ethics and document the employees' intent to adhere to SysTest's stated policy.

NVLAP's Decision

Pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10, NVLAP hereby suspends SysTest's accreditation effective as of the date of this letter. SysTest Labs, Incorporated is prohibited from using the NVLAP symbol on its test reports, correspondences, and advertising during the suspension period for all voting system testing. Accreditation may be reinstated only after such time that SysTest can demonstrate voting system testing in accordance with the requirements of the applicable voting system standards and NIST Handbook 150. This demonstration must be achieved through an on-site visit to SysTest to witness testing, review documentation, interview personnel, and any other means necessary to gather objective evidence in support of a decision regarding reinstatement.

This on-site visit will occur only after NVLAP is convinced, through the submission of documentation, that SysTest has taken the necessary steps to correct the areas of nonconformance herein addressed. This documentation will include, but is not limited to: procedures for test method development; procedures for test method validation; revised document control procedures that specifically address technical procedures; fully developed test methods showing validation, document control, and mapping to the federal voting system standards; and, procedures or policies that address methods by which SysTest will control statements or assurances to their clients regarding the outcome of voting system testing.

SysTest was accredited by NVLAP based on its ability to develop and perform competent testing within the framework of an effective management system. SysTest now needs to revise its management system to correct the nonconformances found during this visit and implement these system changes. NVLAP believes that the current SysTest management team is committed to accomplishing this goal and will work with them to that end.

Sincerely,

Jon Crickenberger
NIST/NVLAP Program Manager

Enclosures

Cc: Brian Hancock, Election Assistance Commission