Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 22, 2007
Draft Minutes


1. Administrative Updates (Allan Eustis)
2. Discussion plenary version of chapter 3 and change list, attached.
3. Discussion of explanatory white paper on usability benchmarks.
4. Other issues

Attendees: Alexia Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), Elle Colver (EAC), John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, NIST Statisticians, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Everyone is busy getting ready for the July 3rd plenary meeting. A draft agenda has been posted and the unofficial minutes from the May meeting will go up today. John Wack is working to compile the draft VVSG and it will go up on the web and CDs distributed early next week.

Significant changes to Usability, Accessibility, and Privacy since the May 2007 TGDC Meeting (John Cugini):

  • Throughout the entire section: rewording to conform to updating of glossary. Clarified terms include: "contest", "contest choice", "manufacturer" (not "vendor"), "vote".
  • 3.1.2: Added definition of "summative usability testing".
  • 3.1.3: Clarified interaction of requirements in sections 3.2 and 3.3, especially noting that all VEBD requirements apply as well to the Acc-VS. Note in section 3.3 refers back to this clarification.
  • 3.2.1: Section extensively re-written to reflect new metrics and benchmarks.
  • 3.2.2-D: Clarified requirement for notification of successful ballot casting and separated out what to do in case of failure.
  •, Added requirements for notification of ballot casting failure.
  • Added discussion on system support of privacy to clarify intent of requirement.
  • 3.2.4-D: Upgraded initial clause in "no bias" requirement.
  • 3.2.5: Clarification of "poor reading vision". Poor vision means not disabled.
  • 3.2.5-G, G.1, G.2: Legibility of paper upgraded from recommendation to requirement. Specified two "sufficient techniques" (font size and magnification).
  • 3.2.7-A.3: Added new requirement in Alternative Language section for "Auditability of Records for English Readers". This was verified with the EAC, they will bring it up also to the Standard Board.
  • Updated citation to Underwriters Laboratory safety standard.
  • 3.3.1-A.1: Requirement for "Documentation of Accessibility Procedures" (reporting requirements) moved to Volume IV.
  • 3.3.2: Use "low" rather than "partial" vision to conform to common practice.

Benchmarks White Paper (Sharon Laskowski):

Sharon reviewed in detail the benchmarks white paper - a final draft of the paper should be available by next Wednesday. This paper will be presented at the July 3rd plenary meeting. The goal of the white paper is to be able to present to the TGDC and EAC, in plain language, the overview and basic understanding of the usability benchmarks. The white paper summarizes the research conducted to develop the test, metrics, and benchmarks in response to a TGDC resolution and describes the resulting performance-based requirements included in the current draft of the VVSG. Supporting documentation of test data and test materials are included appendices. The report defines the test: the ballot and the instructions used and the test participants. It defines how the system will be measured: effectiveness, efficiency, and satisfaction (efficiency and satisfaction numbers will be reported but will not be used as benchmarks to pass/fail a system). The paper describes tests used to validate the results and processes used to analyze the results. The paper also explains what was done to show that we could get repeatable results from the test protocol. Sharon then explained how the benchmarks were set, explaining that the confidence interval was a repeatability measure. The research described in this white paper shows the validity and repeatability of our testing methodology and allowed HFP to determine benchmarks and specify usability performance requirements for voting systems. This methodology is technology-independent. Any system can be tested to these benchmarks.

The current explanatory white paper still needs to be shortened to about 5 pages before next week. Sharon has begun working on a more technical document that will provide a thorough analysis and will hopefully be ready when the VVSG is delivered to the EAC. Anyone wishing to analyze the data to review how the benchmarks were obtained will be able to do so using the raw data that will be made available.

After the VVSG is delivered to the EAC, time will be spent researching test methods. Tests have to be run to get reproducibility by the test labs. The flexibility of the test protocol has to be determined -- to ensure that this methodology can be reproduced in any VSTL, we will be performing research to determine how precisely the test conditions must be followed as part of our work on VVSG test method development. It needs to be determined how test methods are presented to the test labs.


It was proposed and accepted that at the next HFP subcommittee meeting members of the STS subcommittee would give an overview of the materials in the STS chapters (Chps 4-12). Whitney feels that a pre-brief will better help members get prepared for the July 3rd meeting. It was reminded that the CRT subcommittee has posted a change log <> for review.

NEXT HFP MEETING: Friday, June 29, 2007.

Meeting adjourned at 12:00 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]



Link to NIST HAVA Page

Last updated: July 25, 2007
Point of Contact

Privacy policy / security notice / accessibility statement
Disclaimer / FOIA
NIST is an agency of the U.S. Commerce Department