Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
March 9, 2007


1. Administrative Updates and planning for March TGDC meeting (Allan Eustis)
2. Summary of continuing discussions and clarifications about SI and accessibility (Whitney and Sharon)
3. Discussion of usability benchmarks (Sharon).
4. Other issues.

Our next telecon is scheduled for: Friday, March 16, 2007 at 11 AM ET

Attendees: Alexis Scott-Morrison, Alice Miller, David Baquis (U.S. Access Board), David Flater, John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Allan distributed an email regarding planning for the upcoming plenary meeting. There is an TGDC-internal meeting web page up (See: It will be posted externally (to the public) a week before meeting (March 15, 2007). The plenary is now scheduled to go to 4:30 on day 2.

Presentations/Materials to be presented at the plenary meeting were discussed.

Sharon discussed the material related to the usability benchmarks (she will be working on this over the weekend as well):

  • We have a draft of different ways to count errors and also time satisfaction.
  • We have data that shows that our methodology will work.
  • Do not have specific data per system; since we don't evaluate classes of voting systems, we're evaluating our methodology.
  • It would be nice to show, with some confidence, differences in performance of systems with our testing instruments.
  • Approximately 40 people were involved in the pilot testing.
  • We've been looking to see if we could set a conformance benchmark.
  • An expert statistician provided guidance on how to properly analyze data so that we get small enough confidence intervals. If you get large intervals, your benchmark gets set too low.
  • Question we are asking; Can we set a benchmark that allows us to pass systems that are "good enough" and to fail those that are not?
  • We need analysis to show what the data means and to show that the error is small enough so that we can actually say something's repeatable with similar people.
  • What are we measuring? We want to make sure that on a given system, we're not seeing clusters of errors, we're looking for usability flaws that could affect an election, and we want to make sure that we're not seeing large numbers of people having errors. We want to see that people that had errors had a low number and we want to see if there are particular tasks on the ballots that cause problems. We need to see how these errors would be multiplied for large numbers of users.
  • We want to present issues for assigning such a usability benchmark instrument.
  • We won't know exact scores because we're running hundreds of users through to make sure we get repeatability.
  • We'll be talking about the kinds of metrics we're looking at and how they'll be grouped into pass/fail.
  • We will mention that validity is being shown because our results are mirroring what others are finding.

Sharon and Whitney have been having an offline discussion regarding system testing and how that might fit into volume testing. If users are being brought in for volume testing, we could use them for benchmarking testing.

For accessibility, there is a need to do a whole system test of the process for different disabilities to make sure the system is definitely accessible. The other TGDC subcommittees want to do process testing, such as auditability, accessibility, setup validation, volume testing, etc. Sharon will include this as a "future activities" bullet.

Discussion was held regarding usability of the VVPAT papers rolls for audits, etc. It has generated a high level of problems for poll workers and election officials. This is not something HFP has spent a lot of time looking at because it has been concentrating on usability for voters. John Wack feels that changing the way items are printed onto flat sheets (preferable), will not offer a fundamental change to the voting system. The setup by poll workers is critical and needs to be looked at. John would like feedback from election officials. It might be good to look at the requirements written for the product and that would require a possible change to the paper used, e.g., stand up for 22 months with lots of handling. John will be engaging Dan Schutzer's input regarding this matter.

Software Independence (SI) and Accessibility:

Working with STS, we now have a clear definition about what SI is and what it means by voter verification. SI is making sure that any problems with the election can be detected afterwards. It is not necessary for everyone to verify their ballots for SI to hold. Voter verification has to be available for everyone. The discussion then clarified the difference between direct and indirect verification. A sighted voter can verify by reading the ballot printout, a blind voter would need assistive technology to indirectly verify their printout. The table regarding the four different approaches and their scores were discussed. Concern was expressed over the audio tape recordings and the possibility of tampering with them. There is also an issue with recounting two different types of media that is why there are minuses in the auditability and accessibility/usability columns of the paper.

Ron Rivest and Whitney will present a summary of this research paper at the plenary meeting for discussion by the full TGDC. The goal is to decide which, if any, of the approaches outlined, are suitable to meet the resolution that all systems are SI. Hopefully this will help to decide which architectures can be taken off the table, and which are suitable. There may be a possible resolution, but the discussion will allow subcommittee to move their work forward. At the next meeting (in May) we have to adopt final draft of requirements to forward to EAC.

ACTION ITEM: Whitney, Tricia, and David will make sure that the external (public) web page URL information is circulated when released and let people know that comments can be sent to

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]




Link to NIST HAVA Page

Last updated: July 25, 2007
Point of Contact

Privacy policy / security notice / accessibility statement
Disclaimer / FOIA
NIST is an agency of the U.S. Commerce Department