Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

TGDC Subcommittee Work - Historical Meetings - HFP Meetings - 2007

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
August 10, 2007
Draft Minutes

AGENDA

1. Administrative updates (Eustis)
2. Overview of status of benchmarks: renamed, reworded in Chapter 3. Revised white paper will be forthcoming.
3. Discussion of how to set benchmarks.
4. Summary of other HFP materials in preparation for TGDC meeting.
5. Other issues

Attendees: Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • IEEE has nominated Dr. Cem Kaner (Florida Institute of Technology) to be its representative on the TGDC. He has a PhD in experimental psychology, Juris Doctor, and an extensive background in computer science.
  • Last week Dr. Jeffrey announced his NIST departure for September 3, 2007.
  • The next scheduled STS teleconference (Tuesday, Aug 14) will be dedicated for reviewing the VVSG. All TGDC members invited to attend.
  • NIST has been working on a number of logistical items including an executive summary and companion documents for the VVSG. The web will have a set of matrices posted that map all requirements to their source
  • The TGDC plenary is still scheduled to begin at 11:30 ET. We will be using the TRACE hand raising software. We hope to have all power point presentations by next Wednesday.

Benchmarks (Sharon L):

  • Sharon has distributed a detailed technical report outlining the benchmarks work. She has also produced a 7 page overview summarizing the reporting and providing the benchmarks.
  • We have three benchmarks: voter inclusion index (a function of the average error rate, used to be called the accuracy index), total completion rate, and perfect ballot index (now a ratio).
  • The question we're currently dealing with is how to write them into the current standard. Currently the standard is written with the requirements stating the systems have to meet these 3 benchmarks, (which we are including current numbers for) with a statement saying that we are continuing to test and final numbers are forthcoming. Discussion was held about how high and how stringent those numbers should be. This question will be posed to the TGDC as a whole at the plenary meeting.
  • HFP would like to make recommendation to EAC that these benchmark numbers are reviewed periodically (this does not involve changing the standard or the requirements, just the benchmark numbers).

List of Notable Changes (John Cugini):

  • Introduction, section 2.2 and 2.3 (What's New) - nice summary of what's new from VVSG 2005
  • Introduction, section 3.5 (Relationship of VVSG and HAVA) - this applies to the whole standard but originated from HFP
  • Part 1, section 6.4.5 (Maintainability of voting system) - no longer in 3.2.8.2
  • Part 2, sections 3.6.2-A (vendor reporting of their own summative usability tests), 4.4.6-A (usability of poll worker documentation), 4.4.6-B (vendor must document the system's accessibility procedures) and 6.1-R (test lab must report the metrics for usability of the system) Moved back to Chapter 3 in Part 1 from the document requirements section.
  • Part 3, section 3.3 (Performance Testing (Benchmarking)) - talks about performance testing benchmarks in general
  • Part 3, section 4.5 (Accessibility Test Methods) - J. Cugini feels this could should be dropped

OTHER:

  • Sharon will be working on her presentation slides over the weekend to present benchmarking materials and recommendations, as well as changes to HFP's sections. Any comments should be emailed to her about any other material needing to be covered at meeting.
  • Allan discussed the approval process for the VVSG. Each subcommittee will present and then a resolution will be offered to approve that section. At the end of the meeting a resolution will be offered to approve the document in whole. Over the next 2 to 3 weeks edits will be made.
  • Everyone is invited to the STS teleconference on Tuesday, August 14, 10:30 am to go over the VVSG.
  • TGDC plenary on Friday, August 17, 2007, 11:30 a.m. ET.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
July 20, 2007
Draft Minutes

AGENDA

1. Administrative updates (Eustis)
2. Discussion of proposed changes of usability benchmark shalls to shoulds
3. Other issues

Attendees: Alexia Scott-Morrison, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, Nelson Hastings, Whitney Quesenbery, Philip Pearce

Administrative Updates (Allan Eustis):

Proposed Benchmark Changes:

Whitney reviewed the latest discussions that had taken place regarding the usability benchmarks being proposed. There are issues with putting in numbers because research is still being conducted. It was felt that there should be no difference in the effectiveness goals whether the voter had disabilities or not, but now additional tests are being conducted (initially in Chicago and Denver) to broaden demographics and to include individuals with disabilities. The requirements in the VVSG will have usability requirements for conformance testing and reporting with a statement that they are based on current data and that further research is still being performed. Based on the current research, the VVSG will recommend that the benchmarks be set at a specific number. The question to be asked of the TGDC at the plenary meeting will be how high to set the benchmark numbers. Currently they are set at a medium range, should they be higher? Should we be setting them so that vendors have an aspiration goal for improvement? These are questions for the TGDC and the EAC. [NOTE: Test performed to date has been on systems certified to VVSG 02 and used in the 2000 or later elections.]

Other:

  • Everyone should have a copy of the latest HFP section along with the changes list since the May meeting. The only item that may change would be on the performance benchmarks.
  • Nelson Hastings will distribute to both HFP and CRT the latest revision of the new IVVR chapter.
  • Whitney proposed canceling the next two HFP subcommittee meetings and convening the one on August 10. A note will be sent out about the final decision.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
July 13, 2007, 11:00 a.m.

Agenda:

1. Administrative updates (Eustis)

2. Discussion of usability benchmark white paper revisions and additional
reproducibility research (Sharon)

3. Review VVSG HFP to-do list for final revisions by John Cugini. (See Below)

4. Other issues

The next HFP teleconference is scheduled for Friday July 20 at 11AM ET

Attendees:  Alexia Scott Morrison, Allan Eustis, John Cugini, John Wack, Mark Skall, Nelson Hastings, Sharon Laskowski, Wendy Havens

Administrative Updates:

·The agenda for the July 3rd plenary meeting has been re-scheduled for August 17. 2007 (11:30 am EDT).  The reason for the change in date was to allow NIST to more fully examine some open issues in both security and human factors to insure the best outcome for the plenary. Want to discuss today to make sure we are on the same page and approach closure.

·There will be a new version of the draft VVSG available well in advance- probably by the end of next week (July 20, 2007)

Usability benchmark white paper revisions update and additional reproducibility research:

·In a review of the benchmarks and white paper, the NIST team consensus determined that it was important to articulate the methodology and expand with an executive summary that explains in layman’s terms the meaning of the benchmarks. The white paper is going out to usability experts in the Federal government for a technical review.

·NIST team discussed performing additional usability studies to confirm benchmarks. The plan is to conduct additional testing in Denver and Chicago first with 100 subjects per location for a total of 650 subjects for the usability benchmarks.  This will ensure that chosen demographics give good reproducibility.

·Updated paper will be re-circulated after peer review. Expect 2-3 week turn around.

·MS asked for time frame on additional testing. SL gave best case scenario for completion at the end of September. The paperwork reduction act paper work may need to be updated.

·MS noted the suggestion to change the “shall” requirements to “should” for the benchmarks since the testing will not be complete when the VVSG draft is scheduled for delivery.  The benchmarks could be updated during the public review process after the research is complete and the ‘shoulds’ changed to ‘shalls’. 

·SL noted questions from WQ on interpretation of definition of “should’ in the benchmark context. MS noted that his understanding is that you cannot fail a manufacturer on a testing benchmark until the “should” is changed back to a “shall”. However, the testing is mandated.

·JC asked for clarification from an editorial standpoint WRT twin requirements.  The VVSG requires reporting efficiency as a ‘shall’ in a completely different section of the VVSG. “You shall conduct tests and report efficiency”. There is no efficiency benchmark. So what we are really saying is that we will make the effectiveness benchmark a ‘should’ but keep the number there until research is complete.

·SL noted parallel to efficiency benchmarks in confidence in voting which are reported. 

·ASM felt comfortable with this approach if WQ and SL are in agreement. SL will discuss with WQ.

·MS noted that EAC indicated it would be appropriate to change the “shoulds” to ‘shalls” in the public review process. The TGDC adoption resolution could so specify the intent here.

·SL noted the need to do further testing to ensure that benchmarks are correct.

·JW noted need to send out an e-mail to entire TGDC on any outstanding issues from their perspective. NIST needs to know them quickly. First, we will address WQ’s concerns.
 

Review VVSG HFP to-do list for final revisions by John Cuguni. (See Below):

Current editing to-do list for HFP VVSG Draft

=============================================

Reconcile repetitive sections 3.2.8.2 and 15.4.5 duplication of Maintenance requirements.

Proposal is to delete 3.2.8.2 and put in a pointer to 15.4.5.

 

---------------------

 

Comment from David Wagner:

- 3.2.6.1E voter inactivity time    different for blind/ low vision? 

 

Analysis: 2 minutes is plenty we think.     we don't think they would  take any longer finding the button to push on the tactile controls at that  granularity.

 

It might be worth clarifying that the 2 minute period starts after the prompt has finished (not from the last voter action).  Would need to re-define "voter inactivity time" -- small fix but worth doing.

 

AE noted concern of B. Williams for voter time requirements. (will send out). U.S> access board noted that 2 minutes was a lot of time. SL noted that we will leave it for now, pending further comment, and will be the same as for audio ballot.  

 

---------------------

Comment from David Wagner:

 -  3.2.2D notification of ballot casting, which now has new wording from after the plenary.     Wagner notes a similar issue for poll workers: can the poll worker tell if voter successfully cast?     Any need to change wording further?

  

This also may be worth clarifying - it probably was the original intent. We have to think about scenarios, however -- VEBD vs. PCOS -- and the preservation of privacy.

 

---------------------

Check wording and measurements in accessibility figures e.g label for fig. 3 says 254 mm, but diagram says 255 and inches and mm are not numbered in the figures. JC will change to be consistent with reference to U.S. Access Board for hard conversion.

---------------------

Resolve all references - both to external sources and internal XREFs.

---------------------

Meeting adjourned at 11:47 am.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

*****************

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 29, 2007
Draft Minutes

Agenda:

1. Administrative updates and logistics for the July 3 TGDC plenary telecon (Eustis, Wack)
2. Summary of VVPR, Audits and Electronic Records sections (Hastings)
3. Go over final draft of usability benchmark white paper and draft presentations for the July 3 TGDC plenary telecon. (Sharon)
4. Other issues

Attendees: Alexia Scott Morrison, Allan Eustis, Alicia Clay, Barbara Guttman, Bill Burr, Elle Colver (EAC), John Cugini, Secretary John Gale, John Kelsey, John Wack, Mark Skall, Mat Masterson (EAC), Neal Erickson, Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates:

  • The agenda for the July 3rd plenary meeting was reorganized per Secretary Gale's suggestion to put STS first, then CRT, and HFP. (Note: the re-scheduling of the TGDC telcon plenary occurred after this subcommittee teleconference. The July 3 plenary teleconference will now take place on August 17.)
  • A summary of the May 2007 plenary minutes was prepared at the request of Secretary Gale. Longer, more detailed background minutes are still available.
  • TGDC members will call in for the plenary teleconference the same way as for normal subcommittee meetings. The goal is to move the approval process along and finish the meeting by 5:30.
  • Changes that are made to the VVSG (compared to the copy received by TGDC members) need to be pointed out. HFP has put a listing of changes on the plenary meeting website, CRT has an extensive change log, and STS will work to put something together.

Usability Benchmarks and Presentations:

Per instructions from Dr. W. Jeffrey, Sharon is writing an executive summary to clarify for the general public the benchmarks white paper. A change list of HFP's changes has been circulated - the new stuff having to do with the benchmarks material. The TGDC needs to discuss the policy/philosophy of the benchmarks. The benchmarks may need to be pushed higher, that should be discussed at the plenary. The benchmarks we've chosen to set are ballot casting (98%), 100% correct casting rate (70%), and accuracy rate/index (.35 w/possibility of moving to .4). [NOTE: Benchmark numbers can be changed during public comment period.]

The two items that will be covered regarding benchmarks at the July 3rd meeting will be a discussion on the demographics and the accuracy index. HFP feels that the accuracy index benchmark can be raised so that vendors strive for better systems. Full TGDC input will be requested.

Summary of VVPR, Audit, and Electronic Records Sections (Hastings/Clay/Kelsey):

Nelson Hastings summarized the changes to chapters 7-14. For all STS sections, "voting equipment" changed to "voting device" as well as other modifications to harmonize with other VVSG sections. All the security documentation requirements were moved to volume 4 with other documentation requirements.

  • Cryptography chapter: Changed audit record to election record. After STS debate it was decided that multiple keys were not needed for devices used to host multiple precincts during an election.
  • Setup Validation changed to Setup Inspection chapter. This change was a result of system integrity management requirements enabling us to relax focus on setup verification. Removed software verification by external hardware device requirement. Used clearer, better language.
  • Software distribution and installation chapter: Refocused to software installation capabilities of a device. All software distribution requirements and activities related to witness build moved to volume 5.
  • Access control chapter: Refocused to requirements on administration authentication. Voter authentication requirements were moved to e poll book requirements. Removed requirements on remote access.
  • System integrity management: Focus here on software integrity verification on boot up and loading into memory before execution and its affect on the setup verification chapter. Deleted requirements that are covered in other chapters. Re-scoped focus of malicious code real-time detection and backup recovery from election devices to make specific on election management systems.
  • Communications: Deleted requirements related to limiting remote activities and limiting number of active interfaces. Added new requirement related to air gap between networked devices.
  • System event logging: Re-scoped requirements. Added to list of items to be logged - acknowledged that some items needing logging must be done from procedural prospective..
  • Physical security: Added requirements about locks based on TGDC discussion.
  • OEVT: New section. Focuses on scope, team composition, level of effort, fail criteria, and rules of engagement.
  • Auditing: Took out any requirement that was not an equipment requirement or equipment documentation (removed anything that looked like election procedures). Removed descriptions of auditing steps - left what requirements were supposed to accomplish. Removed parallel and spot parallel testing. Changed what record was being used for final tally audit. Harmonized with CRT.
  • Electronic records: Deleted overlap with cryptography section. Added report for ballot counted and vote total.
  • VVPR: Added requirements that all records must be OCR readable. Added requirements per TGDC for the machine readability of the human readable paper records. Added requirements about the process of rejecting paper records and needing election official involvement. Changed requirements on cut-sheet VVPATs to allow split of cast vote record over more than one sheet.
  • SI: Added SI requirements (that it's required) to Conformance Clause, saying it "may" be achieved by independent voter verifiable records which VVPR is one way.

Meeting adjourned at 12:15 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 22, 2007
Draft Minutes

AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion plenary version of chapter 3 and change list, attached.
3. Discussion of explanatory white paper on usability benchmarks.
4. Other issues

Attendees: Alexia Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), Elle Colver (EAC), John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, NIST Statisticians, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Everyone is busy getting ready for the July 3rd plenary meeting. A draft agenda has been posted and the unofficial minutes from the May meeting will go up today. John Wack is working to compile the draft VVSG and it will go up on the web and CDs distributed early next week.

Significant changes to Usability, Accessibility, and Privacy since the May 2007 TGDC Meeting (John Cugini):

  • Throughout the entire section: rewording to conform to updating of glossary. Clarified terms include: "contest", "contest choice", "manufacturer" (not "vendor"), "vote".
  • 3.1.2: Added definition of "summative usability testing".
  • 3.1.3: Clarified interaction of requirements in sections 3.2 and 3.3, especially noting that all VEBD requirements apply as well to the Acc-VS. Note in section 3.3 refers back to this clarification.
  • 3.2.1: Section extensively re-written to reflect new metrics and benchmarks.
  • 3.2.2-D: Clarified requirement for notification of successful ballot casting and separated out what to do in case of failure.
  • 3.2.2.1-F, 3.2.2.2-F: Added requirements for notification of ballot casting failure.
  • 3.2.3.1-A: Added discussion on system support of privacy to clarify intent of requirement.
  • 3.2.4-D: Upgraded initial clause in "no bias" requirement.
  • 3.2.5: Clarification of "poor reading vision". Poor vision means not disabled.
  • 3.2.5-G, G.1, G.2: Legibility of paper upgraded from recommendation to requirement. Specified two "sufficient techniques" (font size and magnification).
  • 3.2.7-A.3: Added new requirement in Alternative Language section for "Auditability of Records for English Readers". This was verified with the EAC, they will bring it up also to the Standard Board.
  • 3.2.8.3-A: Updated citation to Underwriters Laboratory safety standard.
  • 3.3.1-A.1: Requirement for "Documentation of Accessibility Procedures" (reporting requirements) moved to Volume IV.
  • 3.3.2: Use "low" rather than "partial" vision to conform to common practice.

Benchmarks White Paper (Sharon Laskowski):

Sharon reviewed in detail the benchmarks white paper - a final draft of the paper should be available by next Wednesday. This paper will be presented at the July 3rd plenary meeting. The goal of the white paper is to be able to present to the TGDC and EAC, in plain language, the overview and basic understanding of the usability benchmarks. The white paper summarizes the research conducted to develop the test, metrics, and benchmarks in response to a TGDC resolution and describes the resulting performance-based requirements included in the current draft of the VVSG. Supporting documentation of test data and test materials are included appendices. The report defines the test: the ballot and the instructions used and the test participants. It defines how the system will be measured: effectiveness, efficiency, and satisfaction (efficiency and satisfaction numbers will be reported but will not be used as benchmarks to pass/fail a system). The paper describes tests used to validate the results and processes used to analyze the results. The paper also explains what was done to show that we could get repeatable results from the test protocol. Sharon then explained how the benchmarks were set, explaining that the confidence interval was a repeatability measure. The research described in this white paper shows the validity and repeatability of our testing methodology and allowed HFP to determine benchmarks and specify usability performance requirements for voting systems. This methodology is technology-independent. Any system can be tested to these benchmarks.

The current explanatory white paper still needs to be shortened to about 5 pages before next week. Sharon has begun working on a more technical document that will provide a thorough analysis and will hopefully be ready when the VVSG is delivered to the EAC. Anyone wishing to analyze the data to review how the benchmarks were obtained will be able to do so using the raw data that will be made available.

After the VVSG is delivered to the EAC, time will be spent researching test methods. Tests have to be run to get reproducibility by the test labs. The flexibility of the test protocol has to be determined -- to ensure that this methodology can be reproduced in any VSTL, we will be performing research to determine how precisely the test conditions must be followed as part of our work on VVSG test method development. It needs to be determined how test methods are presented to the test labs.

Other:

It was proposed and accepted that at the next HFP subcommittee meeting members of the STS subcommittee would give an overview of the materials in the STS chapters (Chps 4-12). Whitney feels that a pre-brief will better help members get prepared for the July 3rd meeting. It was reminded that the CRT subcommittee has posted a change log <http://vote.nist.gov/TGDC/crt/index.html&gt; for review.

NEXT HFP MEETING: Friday, June 29, 2007.

Meeting adjourned at 12:00 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 15, 2007
Draft Minutes

Agenda

1. Administrative Updates (Allan Eustis)
2. Discussion of Edits from Last Telecon
3. New Wording for Benchmark Usability Requirements
4. Other Issues

Attendees: Allan Eustis, David Baquis (U.S. Access Board), Elle Colver (EAC), John Cugini, John Wack, Nelson Hastings, Secretary John Gale, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates:

  • Allan E: We plan to have a draft agenda for the July 3rd plenary meeting to go to EAC next week. The meeting should start around 11:30 a.m. EDT, working lunch at 1:30 p.m. EDT, and ending no later than 5:30 p.m. EDT.
  • John W: We are putting together a web site containing the latest VVSG with latest chapters broken out for review http://vote.nist.gov/TGDC/vvsg07-report.htm

HFP Chapter Discussion:

  • The draft should be in good shape and deliverable to Allan Eustis by next Friday, June 22.
  • HFP will also have a 5 page explanatory paper on benchmarks by Friday, June 22, suitable for public and TGDC review.
  • John Cugini pointed to the requirements on personal assistive technology (3.3.1.C) and audio interfaces (3.3.3.C1 and 3.3.3.C2). These requirements allow for connectors to Acc-VS. There was an issue that these may have security risks, but it was clarified that we wrote requirements that did not preclude interfaces but that they MUST meet all security requirements to be allowed. The section will remain as written. Whitney pointed out that we want to make sure that TGDC did not write requirements that prohibit future innovative solutions.
  • The glossary has changed, and the chapter has been harmonized to match the new terms. E.g., ballot choice now means an option within a contest on the ballot, not the action by the voter.
  • The definition summative usability testing has been added and slightly modified per Alexia's comments.
  • The performance requirements were modified. The subcommittee was asked to review these carefully. There were five new metrics added along with their definitions. The last two, which are reporting requirements, will be moved to volume 5.
  • Section 3.2.5 (Poor Vision) was slightly tweaked to clarify the mean of poor vision per the request of Commissioner Davidson of the EAC.
  • Section 3.2.7 (Alternative Languages) which was discussed at the last couple HFP meetings was decided to be left as written, per instructions from the EAC.
  • There was a pointer added in section 3.3.1 (General) regarding usability requirements also pertain to Acc-VS.
  • Per recommendations from David Baquis, we are using "low vision" as opposed to "partial vision" to harmonize with the U.S. Access Board.
  • Sharon Laskowski is working on the 5 page explanatory paper on benchmarks. It will include a pointer to a website explaining measuring usability benchmarks.
  • John Cugini will be removing all pointers in the HFP chapter and cleaning it up for submission on June 22nd to Allan Eustis. He will provide a clean copy with a "notes and changes" sheet.

Process for July 3rd Teleconference

  • This meeting will be a document review.
  • HFP has 3 things to present: 1) short write-up explaining the benchmarks, 2) review of chapter, and 3) changes to chapter since May plenary.

Our next telecon is scheduled for: Friday, June 22, 2007 at 11 AM ET. The subcommittee will be discussing the benchmarks write-up.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 8, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of Edits for Cast Vote, and Other Small Edits
3. Benchmark Status
4. Other Issues

Attendees: Alexis Scott-Morrison, Allan Eustis, Commissioner Davidson (EAC), David Baquis (U.S. Access Board), John Cugini, Mat Masterson (EAC), Sharon Laskowski, Wendy Havens, Whitney Quesenbery (late)

Administrative Updates (Allan Eustis):

  • We are planning logistics for the July 3rd plenary teleconference. We will be using internet based "Hand Raising" software to cue up questions for the Chair at the meeting. Authors are busily working on chapters for the VVSG, we will be giving them as much time as possible. CD of the VVSG will be sent out a week before the plenary; Hard copies will be available on the 28th to those that request them.

Modifications to VVSG Chapter (John Cugini):

  • Under 3.1.2 (Special Terminology): Added the phrase "summative usability testing" to the "common industry format" definition. Added the definition for "summative usability testing" following ISO/IEC 25062:2006, with regards to voting. This answers a request by STS.
  • Under 3.2.2.D (Notification for Ballot Casting): Added requirements to include notifications for both successful ballot casting and for unsuccessful casting. It was broken into 3 requirements. Requirement pertains to DREs and PCOS systems. John Cugini feels it would not be good to discuss the testing strategy in the discussion field. Sharon will check with Whitney.
  • Under 3.2.3.1.A (Privacy): Added discussion to clarify requirement. No system shall provide mechanism for anyone to be able to tell how another person voted.
  • 3.2.5.G (Legibility): These are new requirements. [In the usability section, we use the term poor vision as opposed to partial or low vision.]
  • NOTE: The U.S. Access Board recommends changing <partial vision> to <low vision>. This is OK with subcommittee, John Cugini to make changes.
  • 3.2.7 (Alternative Languages): There were questions from the last HFP meeting about using English vs. alternative language vs. primary language. There's nothing changed here from last week, we're waiting on feedback from EAC.
  • 3.3.2 (Partial Vision): Per earlier conversation this will be changed to Low Vision. Added a reference to point to other visual requirements.

Usability Performance Benchmarks (Sharon Laskowski):

There were four different types of voting systems tested, with about 50 participants for each system. We'd like to have about 100 people each to tighten the confidence intervals. Using the statistics collected and analyzed, we will have benchmarks for completion rate, perfectly cast ballots, and accuracy. The voting systems used had been used before for federal elections and they were only certified with VVS 2002 standards. The fixes that need to occur on the voting systems appear to be easily accomplishable; we do not feel we are setting impossible benchmarks. Other data was also collected and will be reported for time and subjective satisfaction (there will not be benchmarks for these). The TGDC agreed that accuracy measures could be a pass/fail benchmark and that time/efficiency/satisfaction would be informational data points.

Sharon will pick benchmarks over the weekend and fill them into the standard. The data used to set these benchmarks will be available for anyone wishing to see how numbers were reached. The test protocol will be made public.

Other:

David Baquis brought up the issue of two font sizes and the requirements having to do with magnification. It was decided that the requirements would be left as written.

Our next telecon is scheduled for: Friday, June 15, 2007 at 11 AM ET

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
June 1, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of edits in response to TGDC plenary meeting. The newest version of our chapter is attached.
3. Issues of confusion in public comments:
- have strengthened partial vision requirements
- "requests for assistance" as a separate metric would have been desirable, but the cost (in terms of extra testing) was prohibitive. end-to-end accessibility and usability benchmark of errors for VS (not Acc-VS) do measure some of this characteristic
- pros/cons of bar codes to help with human readable text--> issue discussed and settled,
4. Performance metrics status, explanatory writeups needed
5. Other issues

Attendees:  Alexis Scott-Morrison, Allan Eustis, David Flater, John Cugini, John Gale, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis): 

  • The public plenary teleconference of the TGDC is scheduled for July 3, starting at 11:30 a.m.  It should last 3-4 hours.
  • We have added a link to the public comments received on the VVSG to the main voting web page http://vote.nist.gov/
  • We are in the process of creating a spreadsheet that will be posted on the TGDC site that is a collated chart showing all comments received and their disposition.
  • Reminder:  EAC will have a formal public comment process when the VVSG is delivered to them.

Discussion of VVSG Edits (John Cugini):

  • A lot of the VVSG material that was previously bracketed has gone away after the May plenary because the material was accepted by the TGDC.
  • Comments at the front of the chapter are the same as always.
  • HFP has been waiting for a customized version of the CIF which is still not available as it will be part of the test method worked after the VVSG is delivered.  The placeholder in the chapter will be removed.
  • Overall Effectiveness – still waiting for benchmarks
  • Notification of Ballot Casting:  David Flater of CRT suggested new wording (similar to what was in VVSG 05) stating that not only would a voter get an indication of a vote was cast successfully but also one if a vote was not cast successfully.  (These notifications are critical – if they do not work correctly, systems will not be certified.)  John Cugini will use David’s wording also adding a discussion point about falsifiability being a different test.
  • Editable Interfaces – clarified scope
  • Privacy at Polls:  The words “when deployed” were removed.  David Flater pointed out that CRT was adding a statement under general requirements that says the requirements of the product standard in terms of what is supposed to be delivered in functionality and performance is based on the assumption that the system is deployed, calibrated, and tested as per the instructions of the vendor. 
  • Privacy at Polls:  Ron Rivest’s concerns about the wording over “with cooperation of voter” will be discussed offline.
  • Low Vision and Legibility:  There are new legibility requirements.  There is confusion over the phrase “partial or low vision” used in the accessibility section.  It will be changed to read “partial vision” throughout this section.  The phrase “poor vision” will continue to be used in the usability section.
  • Auditability of Records for English Readers: There was discussion whether this should be changed to read “primary language of the jurisdiction”.  It was decided to leave this as is and the topic will be brought up to EAC and the Standards Board for discussion.
  • Safety Certification: Citation change only
  • Accessibility Throughout Voting Session:  Just a note that the vendor documentation requirement was moved to Vol. IV.
  • Partial vs. Low Vision: Not a new requirement.  The “partial or low vision” phrase will remain here, but changed to just “partial vision” throughout accessibility section.
  • Synchronized Audio and Video: There were public comments of concern over this requirement, but it was decided to leave as is.
  • Other Issue:  Alexis had a concern over whether in the magnifier required for low vision, if the vendor said certain magnifiers would work, would that be tested.  It was agreed that was the consensus.

Confusion in Public Comments:

  • There was confusion that since the requirement regarding two font sizes being available was moved to the usability section that it weakened the requirement under accessibility.  A pointer will be added to make sure this is not weakened.
  • HFP removed request for assistance metrics as they are too costly.
  • There was public confusion over the pros and cons of barcodes.  TGDC has reached a consensus on their use.

Benchmarks:

  • New data for the performance benchmarks is due in on Monday.
  • We need to put together explanations of our statistics, our approach, and our protocol.
  • We will make available the spreadsheets containing background data.

Next HFP meeting scheduled for Friday, June 8, 2007.

Meeting adjourned at 12:00 p.m.

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
May 18, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of small edits of Usability and Accessibility VVSG chapter, such as the rewrite for legibility and magnification
3. Discussion of what will be presented at the plenary meeting, including latest thoughts on the performance metrics.
4. Other issues

Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), David Flater, John Wack, Sharon Laskowski, Tricia Mason, Whitney Quesenbery

Administrative Updates:

  • Plenary meeting presentations are being worked on and will be available by COB today. They will be publicly available on Monday morning 5/23.
  • Reminder: There will be a reception on Sunday afternoon at 6:00 p.m. (Whitney encourages anyone available to attend as this could serve as a pre-briefing to the plenary meeting.) Copies of the current VVSG will be made available by Allan on Sunday night.
  • John Wack went over the agenda for the plenary meeting. Highlights include a brief introductory by Commissioner Davidson, with Brian Hancock discussing some Best Practices. John will be talking about the editing of the VVSG and the next steps in finishing the report. Subcommittee presentations will start with CRT, then HFP, finished by STS. Day 2 will consist of planning next steps in the process.
  • Future subcommittee telecons have been submitted to the federal register - plans are to continue on same days as has been the case.
  • Dr. Jeffrey testified at an EAC hearing on May 17, his testimony is available at vote.nist.gov. When asked about July deliverable, Dr. Jeffrey was confident of meeting the deadline.
  • Two slots on the TGDC are vacant.

HFP Plenary Presentation (Sharon Laskowski):

Sharon emailed everyone the draft presentation before the meeting. This subcommittee meeting was devoted to review of that presentation:

  • Still working on the benchmark slides, they are not included in the draft set.
  • The purpose of this presentation is to go over what has changed since the last plenary; to summarize significant changes since the VVSG 2005; and to give a progress report on the usability benchmarks.
  • One major issue to be noted for meeting: HFP has reworded the accessible voter verification requirement as agreed upon at the last plenary. Exact wording was submitted as a resolution, but there shouldn't be any problem with the modifications made. Changes have been run by STS.
  • Sharon then proceeded to go through each slide and request comments to the presentation. The following are notable points:
    • Slide 5 (Added Cast Ballot Notifications) - Sharon will change wording on slide to reflect CRT's changes in requirement.
    • Slide 8 (Moved VVPAT Requirements to VVPAT Section) - Our requirement was removed and added to the VVPAT section. It has been simplified to be less specific. Committee discussed whether to word it as screen comparison of record vs. verification. It was decided to be a comparison.
    • Slide 10 (Proposed New Wording for Low Vision on Paper Records) - This is new wording that has not yet been reflected in the current draft of the VVSG.
    • Slide 11 (Accidental Ballot Activation) - This is a shall requirement that calls to minimize accidental ballot activation errors, making this requirement hard to test. This does not apply to errors caused by voters intentionally casting ballots then changing their minds.
    • Slide 12 (Rewrite of Accessibility Subsection) - Change requested to slide to reflect that we are talking about systems designed for the general population.
    • Slide 14 (Rewording of Accessibility for Voter Verification) - This requirement was changed from previous TGDC approval to be more general. It has been circulated to STS. This is being called out special so that the committee realizes this is a change to the wording approved at last plenary. There is no need for an amended resolution as long as HFP chapter is approved in whole.
    • Benchmarking vu-graphs - not yet prepared. Sharon will include vu-graphs that cover results of the testing that was complete on May 17. Validity of errors and repeatability/reliability has been received. She will include testing suggestions on how to use benchmarks. After June 1, it will be possible to set benchmarks for a variety of systems. Timing results will be discussed - even if this is not a usability metric it may be taken into account when purchasing systems. Sharon will discuss the satisfaction response, there was no repeatability except in respect to confidence, this is a usability benchmark we would like. Sharon will also provide a timeline for completing the benchmarks section of the HFP chapter.

Meeting adjourned at 11:50 a.m.
 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
May 11, 2007
Draft Minutes

AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of further edits of Usability and Accessibility VVSG chapter; version for plenary meeting.
3. Other issues

Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis/John Wack) :

  • Agenda is up and sent out with CDs to TGDC, along with a copy of the current draft VVSG. Agenda is likely to change - including presentations by EAC and we are trying to finish meeting by lunchtime on day 2.
  • The draft VVSG will also probably change; the latest version will be available at meeting and on web site in advance. John is trying to make the document as usable as possible; half of the material has been incorporate into the document template.
  • There will be an informal reception before plenary on Sunday evening- 6 pm to 8 pm.
  • HFP will continue to have teleconferences after plenary - still Friday's at 11:00 a.m.

Review of Latest Draft of Usability and Accessibility Chapter (John Cugini):

Changed title of chapter to include "Privacy".

The first couple of notes in the draft refer to the benchmarks to be added and refer to the CIFs which are part of the testing.

Vendor Testing: Added new wording for usability testing by vendor, splitting the requirement into two. The HFP chapter requires testing be done; it is cross-referenced with Documentation Section for actual reporting requirement. All requirements for documentation and materials supplied by vendor are in the technical data package (vol. 4)

Privacy: This section was moved up in the chapter and a number of things were added. Big strategy in this section was to clarify that the requirements written here also applied to the voter verification process -- instead of pulling that out as a separate chapter. [NOTE: We're treating the whole process/voting session as one event instead of breaking it into stages.] There was a statement added for privacy when transporting ballots was necessary, e.g., privacy sleeve needs to be used.

Minimum Font Size: This was added to not only include requirements for voters but poll workers as well.

1.2.5.G. This was changed to include not only legibility of paper ballots but verification records as well. There was discussion on how to best write this: broaden it, should vs. shall, include options. John Cugini to draft requirement making this a shall and offering three options to meet requirement: system will allow for at least two font choices, system will provide magnifier, or system shall provide compatibility for a magnifier to be added and include list of compatible magnifiers. Sharon will present at TGDC meeting our current language and suggest this better alternative language.

1.2.5.H. Contrast ratio. Included poll workers.

1.2.7.A. General Support for Alternative Language. Add to include voter verification. Alexis expressed concern over being able to count verification records if they were in alternative language, are they bilingual? Should we add a requirement to make them be? Mat Masterson to check and see if this is required someplace else.

1.3.1. General Accessibility Requirement- Added two new requirements to support end-to-end testing.

1.3.1.A.1. Added comment to include paper records as well.

1.3.1.B. Need to add comment in the discussion field that the first page of the instructions that allow you to choose a larger font needs to be dealt with as well - currently that page is in small font.

1.3.1.C. Reworded to cover all disabilities so that all voters have similar opportunity for verification. Originally we thought all acc-VS would be electronic, but with paper verification for SI, this has caused issues. This requirement includes a sub-requirement for a readback mechanism for blind voters.

[NOTE: We are not writing requirements that limit technology and require specific solutions limiting what is available to officials.]

1.3.3.E; Ballot submission and vote verification- Added the vote verification clause. This requirement is a goal requirement but has no mechanism for achieving it.

Discussion of Benchmark Timeline (Sharon Laskowski):

  • 100 people have participated in testing. Another 50 are participating as sample voters over the weekend.
  • Beginning of June: Conduct field tests across 4 machines to see where to set final benchmarks.
  • June 1: HFP draft due (minus benchmarks)
  • May 15: Raw data in and analysis done from 3 tests (analysis has been getting repeatability in its testing)
  • May 25: Short description of statistics, looking at assumptions of stats and confidence levels
  • May 25: Sharon and John Cugini to look at way to count errors and compute tentative benchmarks
  • June 8: Additional data from last set of test. After that working on final benchmarks and methodology.
  • June 17: Show results to HFP and discuss
  • June 25: Show results to TGDC and discuss
  • July 1: Final draft edits due

OTHER:

  • July 1 is the HFP deadline to get material in. Final draft goes to EAC on July 31.
  • Next Steps will be discussed at the May plenary meeting.
  • We will need to work on a matrix of changes to provide both to the public and EAC for review. These include big changes/additions to help the review process. CRT has done something like this in their introductory section. HFP should look at to see if it's a template that can be followed.

Meeting adjourned at 12:10 p.m.

Our next telecon is scheduled for: Friday, May 18, 2007 at 11 AM ET

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
May 4, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of further edits of Usability and Accessibility VVSG chapter.
3. Report back on any items of interests from this weeks STS subcommittee telecon.
4. Other issues

Attendees: Alexis Scott-Morrison, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Jon Crickenberger (NVLAP), Nelson Hastings, Philip Pearce, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Only 2 weeks left until the TGDC plenary meeting. Allan and John Wack working on agenda to go to EAC today (5/4/07). There was a Committee member request that STS not be first to report at the plenary meeting. The proposed scheduled is now CRT - HFP - STS. Internal Web page is available with meeting information - more documents will be added as it becomes available (see http://vote.nist.gov/TGDC/TGDCmeeting052107.htm). Meeting Materials CD will be sent FedEx to TGDC members about a week before the meeting.
     
  • Next steps after plenary - There will continue to be subcommittee teleconferences as needed. Mark Skall hopes that final document editing would be turned over to NIST with final editing approval and delivery by TGDC chair - Bill Jeffrey. Members of TGDC present at today's meeting were uncomfortable with that. Whitney feels that there not be another plenary meeting, but a series of (public) teleconferences scheduled to approve chapter by chapter of the report. She feels there are too many controversial issues still to be discussed and approved by members of the TGDC. John Wack/Allan to report to Mark Skall so that this can be addressed in pre- briefing with Dr. Jeffrey.

Discussion of Chapter Edits (Sharon Laskowski):

Sharon forwarded a copy of the issues/concerns from last meeting to Whitney and Tricia. Whitney agreed with the changes as proposed.

Today's meeting will cover two major issues that can hopefully be resolved to be put into May 9th draft due to Allan Eustis.

Sharon's proposed rewording for the accessibility read back requirement: "If the Acc-VS generates a paper record (or some other durable, human-readable record) for the purpose of allowing voters to verify their ballot choices, then the system shall provide a means to ensure that the paper-based vote verification is accessible to all users with disabilities, as identified in section XXX."

The requirement as written is vague and untestable. Sub-requirements need to be written that are clear and testable. Sharon mentioned that STS had concerns about the goals of this requirement - John Wack and Whitney expressed that those concerns have already been addressed. Whitney posed the question about whether standards are being written for current system or future states of systems. HFP has been very clear about writing hi-level and sub-requirements to be very precise about what the requirements are. HFP's requirements deal with individuals mainly as opposed to the election as a whole, which is more to what STS and CRT are writing requirements towards.

John Cugini brought up the subject of what the sub-requirements for this requirement should be. Anything besides the audio read back? What about something for low vision voters or people with dexterity issues? Committee discussed the question about electronic ballot markers (EBMs) being used as read back mechanisms. STS subcommittee has a big concern about how much these additional requirements will cost election officials. It was decided that EBMs were acceptable as you could either put the ballot back in the same machine or take it to another machine for read back. Whitney pointed out that this read back is fine since it is not generated from the internal memory of the machine which is not acceptable.

The issue of privacy during transport from one machine to the verification machine was discussed. Preservation of privacy is a major issue. Using an envelope to transport was discussed. Not the best solution for individuals with dexterity issues. These things will be tested during the performance tests. We want to make sure that we write requirements that the vendor can look at and tell if there system should pass the tests.

It was decided that the requirement would be written with a couple sub-requirements to cover the audio read back, electronic magnification, dexterity (which is in the original standard), and privacy concerns. The audio readback can not be generated from the internal memory.

OTHER: Reminder, the benchmarks will not be here before the May 9th deadline. EAC has agreed that the report should be written with the information and the numbers can be changed or added later when they are received.

Cost of Testing Meeting:

John Cugini attended the cost of testing meeting. Sharon had come up with a rough estimate of $35K to $40K, to which there seemed to be no objection. HFP will have to show how they got there test results.

Our next telecon is scheduled for: Friday, May 11, 2007 at 11 AM ET

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
April 26, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Run through of current version of Usability and Accessibility VVSG chapter to discuss proposed edits and issues.
3. Other issues

Attendees: Alexis Scott Morrison, Allan Eustis, Commissioner Davidson (EAC), David Baquis (U.S. Access Board), John Cugini, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Sharon Laskowski, Steve Freeman (NVLAP), Wendy Havens

Administrative Updates (Allan Eustis):

  • Next week's CRT time slot on Thursday, May 3rd will be offered to STS. (Note: Since the meeting, this has changed - May 3rd there will be no subcommittee meetings).

Discussion of Usability and Accessibility Chapter (John Cugini):

The purpose of today's HFP meeting was to go over recent changes and issues of the HFP chapter of the VVSG. There will be a final iteration of changes after this via telcon before the May plenary.

Major Changes:

1.1.2 - Special Terminology - Common Industry Format (CIF)
We talked about getting a customized version but will not be able to by July deadline. We will put together a template to provide guidance so vendors don't have to second guess. Template will be ready after draft VVSG is released. Comment to be deleted in delivered version.

1.2.1.1. - A & C - Overall Effectiveness/Satisfaction
We don't have benchmarks for this yet. Completing tests of 100 users by June 1st. Calculations will be completed by June 15th. HFP will have a telecom with TGDC members to go over these numbers once received. HFP will write a white paper to explain how we got these numbers. Currently there is a hole in the standard.

1.2.1.1.D - Ability to Vote Without Human Assistance
We have one metric among many. We are not going to render human assistance because we want to see how people do by themselves. There is not way of counting. We are not proposing we do. It would be in as an untested requirement. We do not have the money to do separate tests and protocols. The question is whether we should leave this requirement in or remove. Consensus at meeting was to remove. Sharon will discuss with Whitney.

1.2.1.2 - Vendor Testing
This is a formatting issue that we think we've solved. Vendors must do usability testing and report results. The question is whether it goes in this chapter or the reporting section. The proposal is to keep requirement here and put the consolidate report in Volume 4. This issue applies to 6 or 7 requirements under this section.

1.2.2.D - Notification of Ballot Casting
This is a new requirement. It was in the CRT section but should be in the HFP chapter. This just states that there should be a signal when a ballot has been cast and a signal when it hasn't or a lengthy period of inactivity.

1.2.3.A - Completeness of Instructions
Used to say voting systems would provide instruction for operations. Changed to clarify that each voting station must provide instructions.

1.2.4.H - Visual Access to VVPAT
Current thinking is that this requirement belongs with other VVPAT requirements. (It will remain written as is unless there are objections.) Sharon to check on status of latest VVPAT section to make sure it gets moved there.

1.2.5.C - Accidental Activation
This is not controversial. Whitney has asked for a new discussion to be added for explanation of requirement.

1.2.8.A - Clarity of System Messages for Poll Workers
Added a section stating that all messages for poll workers be written in plain language as well as the instructions for voters.

1.3 - Accessibility Requirements
We added a statement that all requirements for usability apply to all systems, including accessible voting systems. Mat suggested a wording change to clarify.

1.3.1.A - Accessibility Throughout Voting System and 1.3.1.A.1 - Documentation of Accessibility Procedures
Added per request by Whitney. These requirements are to make sure there is ability to evaluate system at a higher lever - the end-to-end requirements. These are the new draft requirements - Sharon will run them by Whiteny. David Baquis supports these in as there consistent with ADA section 508.

1.3.1.E - Accessibility of Paper-Based Vote Verification
This was discussed at the last TGDC plenary meeting and was written based on the resolution passed at that meeting. STS does not feel that is adds much for security. David Baquis had comments about whether it was necessary or should be elevated to higher level to cover all disabilities. His comments will be posted on the public comments page of the TGDC website.

1.3.4.C - Ballot Submission
This is for voters with dexterity problems. It is a carryover from the 05 requirements. Is it practical, would it stop EBMs from being certified? No one wants to change it.

After the major changes were discussed, John Cugini ran through the document to discuss small wording changes that clarified requirements; changed wording to make them in plain language; and/or added discussions for clarification of requirements.

Meeting adjourned at 12:00 p.m.

Our next telecon is scheduled for: Friday, May 4, 2007 at 11 AM ET

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 ****************

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
April 20, 2007
Draft Minutes

DRAFT AGENDA

1. Administrative Updates (Allan Eustis)
2. Discussion of test methods and expertise for the VVSG with NVLAP staff.
3. If time allows, quick run through of current version of Usability and Accessibility VVSG chapter to discuss proposed edits and issues.
4. Other issues

Our next telecon is scheduled for: Thursday, April 26, 2007 at 1 PM ET

Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, John Crickenberger (NVLAP), John Cugini, John Wack, Matt Masterson (EAC), Nelson Hastings, Philip Pearce, Secretary John Gale, Sharon Laskowski, Steve Freeman (NVLAP), Wendy Havens, Whitney Quesenbery

Administrative Updates:

  • HFP TGDC members on the teleconference indicated they would be attending the upcoming May 21,22 2007 plenary at NIST.

Discussion of Test Methods:

This meeting was dedicated to a discussion with NVLAP regarding test methods. The goal of HFP was to take a look at every requirement and try and define an appropriate method for testing it. The requirements fit into three categories: high-level requirements that are specific (tested mostly through performance tests); high-level requirements that are vague (no specific features that map to testing, done via expert judgment testing); and low-level requirements that are specific (these are easy to test and write test methods).

Some examples were given. For color blindness, a low level, specific requirement would be that any information displayed in color must also be explained in text. For plain language, the requirement is a high-level, vague saying everything must be written in plain language and avoid passive language unless it is appropriate - this is subjective and can only be tested by experts in this field.

John Gale inquired if something over time could change from one testing category to another? John Cugini pointed out that if something passes the high level tests it may not be necessary to run through the low level requirements. As technology changes and requirements change, the test methods will change.

One issue for NVLAP is how to choose expert testers. They need to understand the type of expert knowledge that is required so that they can find the correct experts. Also there is the issue of who designs the tests. Up until now the test labs have not only done the testing, but designed the tests. The goal here is to centralize the tests - such as when the Access Board develops usability tests for standards and they should be used in each of the test labs therefore making consistency more attainable in test results. In the past VVSGs, we've only provided requirements, this time we will also be providing test methods.

What can the subcommittee do to help NVLAP? What issues does NVLAP have? Some of the standards in the current VVSG are ambiguous. (That will hopefully be fixed in the next version of the VVSG - we give some specific failure criteria and good descriptions about what is being done and what is being checked.)

[NOTE FROM NVLAP: NIST/TGDC members who are interested in our test methods are welcome to attend assessment visits to laboratories.]

NVLAP would like to require a cross reference matrix saying which test method is testing which requirement. This has never been made an official requirement, and legally it currently can't be forced. Whitney pointed out that we could write within the VVSG what test method went with which requirement.

Another problem with testing is that the labs want to test everything on a voting system in one test scenario. This is not practical. NVLAP was glad to see the language about not testing the 20,000 different criteria in one particular scenario. You can not catch every requirement by running one session. The scenarios are trying to come up with realistic variations on the use including people with different disabilities and different things that could happen.

Another issue of testing was the testing method for alternative languages. The testing does not specifically have to test every language; it should test the variation of technology (logic, storage, and presentation) that generates the languages. The testing needs to include the way languages are stored and the way they are presented.

Other issues that tNVLAP has come across is in the mobility requirements. This includes height requirements for tables. We assume that systems will be used as specified by the vendor, but not always the case. Machines can be tested in the lab but not "fit" appropriately in the polling place because vendor specifications are not followed completely.

Reporting requirements for testing has been an issue. The VVSG will include a general testing document that will cover general issues, including reporting issues making it clear who has to report what, when, and in what detail.

Testing by the VSTLs cannot cover everything. It was reminded that they can cover high level requirement testing, but the state testing and certification must also be done so that machines are certified in more detail with state and local laws. Whitney wants to make sure that it is clear who is to test what so that no gaps in testing occur.

There appears to be an issue that is caused by allowing "straight party" votes. Not sure what the fix of this is. We do have a requirement that states that electronic systems must prevent over voting. The problem is we can't tell how "straight party" voting interacts with individual votes also cast.

(Expert) Testers have to go through specific training, which will be documented. This is another place where test methods are invaluable because you can't qualify a person to undefined test methods. Also they will help NVLAP do a better job in accrediting labs, and we will have more consistency across the tests.

Expert testers will be covered by having a test method that is documented and validated, that the tester can perform it correctly because they have been qualified to perform that particular test method, and only people that are qualified are allowed to do the test method for final reporting and certification.

Meeting adjourned at 12:00 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 ****************

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
April 13, 2007
Minutes

DRAFT AGENDA

1. (5 mins.) Administrative Updates (Allan Eustis)
2. (30 mins.) Discussion to determine extent to which bar codes are allowed as an aid to accessibility and an usability aid for total recounts.
3. (5 mins.) Follow up on component solutions. See Matt Masterson's clarification at the end of this email.
4. (20 mins) NVLAP discussion. We can continue this at the next meeting as well and I will invite Steve Freeman to attend the next meeting.
5. Other issues.

Our next telecon is scheduled for: Friday, April 20, 2007 at 11 AM ET

Attendees: Alexa Scott-Morrison, Alice Miller, Allan Eustis, Barbara Guttman, Bill Burr, David Baquis (U.S. Access Board), David Wagner, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Ron Rivest, Sharon Laskowski, Sharon Turner-Buie, Steve Freeman (NVLAP), Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • The final draft of the current VVSG will be placed on the web the week of May 14th. The May plenary meeting may/may not have white papers at this point. The plan for the meeting is to go through the VVSG. Between now and then we plan to come up with a relevant list of what might be hot topics in the VVSG.

Bar Code Requirements:

The focus of the discussion today was to help the HFP subcommittee come to a conclusion on a recommendation about barcodes for the purpose of usability and accessibility. At the last HFP meeting, it was suggested that they be used with assistive technology with three constraints:

  • No information be included in the barcode that is not represented in clear text as well
  • The bar code is to be generated using a public standard and that standard be disclosed
  • The purpose of the bar code is for accessibility and for possible recounts, but not to replace hand audits

After the last meeting Secretary Gale . put out feelers to election officials about their input on this issue. The response indicated that election officials didn't see any reason that a system that utilizes bar codes shouldn't be certified.

David Wagner of the STS subcommittee wanted to separate the use of bar codes for recounts and audits from use of bar codes for accessibility. He also stated that bar codes must be able to be turned off or on depending on the user. David has two main concerns. First, not sure if the bar codes accomplish much, there could be the same problem that is caused by generating audio read back from electronic memory (there is a possibility for mismatched data), and second, if we put the information in for accessibility and it contains ballot choice information, then there will be temptation to use it for audits and recounts.

Ron Rivest of STS pointed out if you have the same information in two different formats on the same sheet, you have to audit a portion of this to make sure the information is being generated correctly. Ron supports the use of barcodes as long as the vendor supplies an open source for scanning. Barcodes should be readable but machines from different vendors than the voting machines.

At the end of the discussion it was decided that draft recommendations would be written. These draft requirements would include:

  • There must be an exact match between the encoded barcode and the text information (It must read back exactly what is printed, the goal is to hear no more or no less than what you're reading)
  • There must be test methods provided
  • There is required documentation of how it would be used and in what setting
  • There must be option for local officials to turn on or off

Vote-by-phone:

At the last teleconference vote-by-phone was discussed and Sharon took it to the EAC to find out about certification. The conclusion from the EAC certification manual is that a whole system has to be submitted for certification - you cannot certify components. Therefore, if you wanted to submit vote-by-phone as an accessible system, it must be submitted in entirety with documentation and the rest of the system to make it a complete accessible system. In order for a system to be certified as an ACC-VS, it must meet all the requirements in the accessibility chapter of the VVSG. There are no further action items on this issue.

NVLAP Discussion:

The purpose of this discussion was to set the agenda for next week's meeting. Accessibility and usability testing is something new to the test labs. They have been discovering issues when testing systems, such as in the mobility accessibility testing and the audio speed testing. Whitney was concerned that testers are not certified to test for accessibility and usability. The current standards seem to be missing details in test methods.

Agenda item for next week: What can NIST do to generate test methods in this stage to make sure enough repeatability is capable (how do we write clear requirements). Also, there is an upcoming summit on costs of testing systems. Possible discussion of HFP's input for summit.
Meeting adjourned at 12:10 p.m.

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 ****************

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
April 5, 2007
Minutes

AGENDA

1. Administrative Updates (Allan Eustis)

2. Review of 3/22-23 plenary (Sharon)

3. Issues to discuss for VVSG modifications (Sharon)

a. Should we move privacy section to its own chapter?
b. Vote-by-phone addresses audio-only accessibility but it could be a useful component of an accessible solution at the polling place. Should we modify requirements to allow for component solutions? Would and end-to-end requirement help with this?
c. Vendor usability tests could actually be part of the technical data package the vendor submits. Should these requirements be moved to that volume or kept in the usability and accessibility sections?
d. Usability of handling paper: next steps, work with other STS, CRT team members to address what we can.

4. Other issues.

Attendees: Alexa Scott-Morrison, Allan Eustis, John Cugini, John Gale, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Tricia Mason, Whitney Quesenbery

Administrative Items (Allan Eustis):

  • Next TGDC plenary meeting is scheduled for May 21-22, 2007, here at NIST. Informal reception Sunday night May 20th, 6pm to 8 pm.
  • All material from the March TGDC meeting is posted, including the adopted resolutions and the unofficial closed caption transcripts, as well as the power point presentations.
  • There will essentially be subcommittee teleconference calls once a week for each subcommittee until the May meeting. (see: http://vote.nist.gov/TGDC/telecons.html )

Performance Benchmark Testing Update: There are three sets of tests, two of which Sharon Laskowski can report to the TGDC in May. The 3rd big test is set for June 1-2. HFP will be writing the draft requirements and placeholder for the metrics from the June test, these numbers can be revised.

VVSG Modifications:

Whitney mentioned that we need to make sure that we write requirements that allow for the creation of voting systems that are affordable for very small districts as well as very large and/or complex districts. Costs of voting systems should include the cost of maintaining these systems as well. EAC is holding a Cost of Testing Summit (April 30-May 1) that will cover every possible cost we can imagine from testing systems to maintenance.

Should privacy be its own chapter? This is a very important concept, we don't want it to get buried, but is it worth restructuring this late in the game. It was decided that "privacy" would be added to the chapter title and the subsection would be moved up, right after performance requirements.

Vote-by-phone: There was a lengthy discussion on the use of vote-by-phone and how accessible these systems were. Vote-by-phone would be good for voters with low vision or blind voters. Accessible voting systems must cover all type of accessibility. We can write the requirement that all polling stations must have at least one system that meets all the requirements of the usability section, providing a unified solution for all disabilities. We also need to make sure that requirements are written such that when a system is certified through the accessibility section, it must also be a HAVA compliant system. One of the major concerns and issues with vote-by-phone is use by voters with dexterity disabilities. If the VVSG has an end-to-end accessibility requirement, the vendor that submits a vote-by-phone system would also have to submit what the end-to-end solution would be in order for it to be certified as an accessible voting system. The current draft of the HFP section has a draft requirement for the end-to-end accessibility requirement.

Vendor usability tests could actually be part of the technical data package the vendor submits. Should these requirements be moved to that volume or kept in the usability and accessibility sections? If we move it to the technical data package it may become lost and it will loose its readability factor. Suggestions were made to either put it in both sections, or put links from the technical data package back to the usability section. David Flater of the CRT subcommittee may have some issues with this. When compiling the final VVSG, they (CRT) are trying to keep similar types of requirements together. Sharon will talk to David.

Usability of handling paper: (clarified that this is in reference to poll workers): There was lengthy discussion on what was needed on this issue. The durability of the paper issue is out of HFP's scope. Sharon Laskowski will be getting together with John Kelsey of STS regarding the VVPAT section to make sure that usability is covered. There are lots of requirements for easily identifying the paper rolls and linking them to the machines they came from. It was suggested that bar codes be used also for this, as long as the barcodes only contained information already on the sheet and did not include any ballot information. John Wack thought that STS had an issue with using barcodes. Nelson Hastings will take this issue back to STS. (Whitney feels that there would have to be a serious security issue for removing this option as a possibility.) There was major concern over font sizes for auditing. Question at issue: Is the current font requirements for voters good enough for the poll worker requirements? Concern was expressed over the difference in what was necessary when doing a small audit versus doing a recount of the votes. Sharon summarized saying that 3.0 (font size) for paper rolls was fine if you have small audits; for larger audit, some other means to do the counting, such as barcodes, would be needed.

John Wack was not in agreement with HFP. John Gale pointed out that poll workers were hired to do a particular job, and that they were hired based on their ability to do so by state and local officials. In his experience, it is not necessary for requirements for accessibility to be as strong for poll workers as for voters. Sharon and Whitney will take this offline and try to reach a conclusion.

Other Business:

HFP would like to work with NVLAP representatives on a telcon.
HFP 2007 Next Draft is out for comments.

Our next telecon is scheduled for: Friday, April 13, 2007 at 11 AM ET

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
March 16, 2007
Minutes

AGENDA

1. Administrative Updates and planning for March TGDC meeting (Allan Eustis)
2. Summary of HPF discussion items for March TGDC meeting
3. Other issues.

Attendees: Allan Eustis, David Baquis (U.S. Access Board), John Cugini, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Materials have been sent to TGDC on CDs. Public web page has been posted with information for the meeting. The plenary meeting is currently structured so that non controversial requirements material will be discussed on Day 1, leaving Day 2 open for longer, complicated discussions. Mary Saunders of NIST will be presenting on Day 2 as well - her office supervises the NVLAP effort.

Sharon Laskowski started the meeting by going through her slides being presented at the plenary meeting. Basically the presentation is going to be a progress report, discussing changes or issues in the HFP sections, issues requiring further analysis, usability benchmark development, and next research steps.

Basically there are three significant changes since the December meeting:

  • First is regarding availability of font sizes and contrast for all visual systems, not just for accessible voting systems. This will be removed from HFP section, and added under general requirements.
  • Second is on general, continuous adjustability, voters can switch between 3 modes: visual, audio, or visual and audio. Voter will also be able to switch between languages.
  • Third, the safety requirement now refers to UL 60950 instead of OSHA requirement.

Issues that require further analysis:

  • Voter verification and accessibility issue. This will be discussed in detail on Friday at the SI & accessibility panel. We have written a draft requirement for a starting point for discussion.
  • Vendor usability testing. We're not changing the requirement, just want to support it better. We want to add specifics for voting systems for the ISO standard being used. Sharon to add a slide on the ISO CIF.
  • Performance metrics. Sharon will discuss overall performance metrics. Status for generating these metrics will be discussed later in presentation.
  • Dexterity. Do we discuss this at meeting? Definitely. This was put into to parallel a requirement for the blind, after comments on VVSG 05. This could disqualify some EBMs.
  • End-to-end accessibility testing/evaluation. This will cover the entire election process. Vendor must document how system should work and it must be testable. We will go over our motivation for this testing. The goal is to create a place for a test method to not only look at the piece parts, but how they fit together.

Benchmarks: Sharon is still working on these slides. The general goal is to have a quantitative benchmark requirement and to achieve that we had to develop a test protocol method, determine how to count errors, and define what errors looked like. Sharon will go over that methodology and report back the data that was found. She will include a slide of the characteristics a good test instrument must have. Results being reported are supported by others doing testing that have found the same results.

Next research steps will be the last slide:

  • We want to harmonize with the EAC accepted D4D ballot guidance
  • Voter specific plain language guidance
  • Safe color guidance
  • Icons for cognizant disabilities
  • Documentation guidance
  • Accessibility performance benchmarks

NOTE: Whitney commented that the VVSG as a whole was looking great, good job to everyone involved.
Meeting adjourned at 11:50 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
March 9, 2007
Minutes

AGENDA

1. Administrative Updates and planning for March TGDC meeting (Allan Eustis)
2. Summary of continuing discussions and clarifications about SI and accessibility (Whitney and Sharon)
3. Discussion of usability benchmarks (Sharon).
4. Other issues.

Our next telecon is scheduled for: Friday, March 16, 2007 at 11 AM ET

Attendees: Alexis Scott-Morrison, Alice Miller, David Baquis (U.S. Access Board), David Flater, John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • Allan distributed an email regarding planning for the upcoming plenary meeting. There is an TGDC-internal meeting web page up (See:http://vote.nist.gov/TGDC/TGDCmeeting032207.htm). It will be posted externally (to the public) a week before meeting (March 15, 2007). The plenary is now scheduled to go to 4:30 on day 2.

Presentations/Materials to be presented at the plenary meeting were discussed.

Sharon discussed the material related to the usability benchmarks (she will be working on this over the weekend as well):

  • We have a draft of different ways to count errors and also time satisfaction.
  • We have data that shows that our methodology will work.
  • Do not have specific data per system; since we don't evaluate classes of voting systems, we're evaluating our methodology.
  • It would be nice to show, with some confidence, differences in performance of systems with our testing instruments.
  • Approximately 40 people were involved in the pilot testing.
  • We've been looking to see if we could set a conformance benchmark.
  • An expert statistician provided guidance on how to properly analyze data so that we get small enough confidence intervals. If you get large intervals, your benchmark gets set too low.
  • Question we are asking; Can we set a benchmark that allows us to pass systems that are "good enough" and to fail those that are not?
  • We need analysis to show what the data means and to show that the error is small enough so that we can actually say something's repeatable with similar people.
  • What are we measuring? We want to make sure that on a given system, we're not seeing clusters of errors, we're looking for usability flaws that could affect an election, and we want to make sure that we're not seeing large numbers of people having errors. We want to see that people that had errors had a low number and we want to see if there are particular tasks on the ballots that cause problems. We need to see how these errors would be multiplied for large numbers of users.
  • We want to present issues for assigning such a usability benchmark instrument.
  • We won't know exact scores because we're running hundreds of users through to make sure we get repeatability.
  • We'll be talking about the kinds of metrics we're looking at and how they'll be grouped into pass/fail.
  • We will mention that validity is being shown because our results are mirroring what others are finding.

Sharon and Whitney have been having an offline discussion regarding system testing and how that might fit into volume testing. If users are being brought in for volume testing, we could use them for benchmarking testing.

For accessibility, there is a need to do a whole system test of the process for different disabilities to make sure the system is definitely accessible. The other TGDC subcommittees want to do process testing, such as auditability, accessibility, setup validation, volume testing, etc. Sharon will include this as a "future activities" bullet.

Discussion was held regarding usability of the VVPAT papers rolls for audits, etc. It has generated a high level of problems for poll workers and election officials. This is not something HFP has spent a lot of time looking at because it has been concentrating on usability for voters. John Wack feels that changing the way items are printed onto flat sheets (preferable), will not offer a fundamental change to the voting system. The setup by poll workers is critical and needs to be looked at. John would like feedback from election officials. It might be good to look at the requirements written for the product and that would require a possible change to the paper used, e.g., stand up for 22 months with lots of handling. John will be engaging Dan Schutzer's input regarding this matter.

Software Independence (SI) and Accessibility:

Working with STS, we now have a clear definition about what SI is and what it means by voter verification. SI is making sure that any problems with the election can be detected afterwards. It is not necessary for everyone to verify their ballots for SI to hold. Voter verification has to be available for everyone. The discussion then clarified the difference between direct and indirect verification. A sighted voter can verify by reading the ballot printout, a blind voter would need assistive technology to indirectly verify their printout. The table regarding the four different approaches and their scores were discussed. Concern was expressed over the audio tape recordings and the possibility of tampering with them. There is also an issue with recounting two different types of media that is why there are minuses in the auditability and accessibility/usability columns of the paper.

Ron Rivest and Whitney will present a summary of this research paper at the plenary meeting for discussion by the full TGDC. The goal is to decide which, if any, of the approaches outlined, are suitable to meet the resolution that all systems are SI. Hopefully this will help to decide which architectures can be taken off the table, and which are suitable. There may be a possible resolution, but the discussion will allow subcommittee to move their work forward. At the next meeting (in May) we have to adopt final draft of requirements to forward to EAC.

ACTION ITEM: Whitney, Tricia, and David will make sure that the external (public) web page URL information is circulated when released and let people know that comments can be sent to voting [at] nist.gov (voting[at]nist[dot]gov).

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP)*
March 2, 2007
Minutes

AGENDA

1. Administrative Updates and planning for March TGDC meeting (Allan Eustis)
2. Summary of VVSG HFP draft being submitted today.
3. Summary of joint HFP-STS SI and accessibility discussion from 2/1
4. Status of safety requirements, based on earlier 2/2 NIST internal discussion
5. Status of usability benchmarks and how to proceed with HFP subcommittee discussion
6. Other issues

Our next telecon is scheduled for: Friday, March 9, 2007 at 11 AM ET

Attendees: Alexis Scott-Morrison, Allan Eustis, John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Sharon Turner-Buie, Tricia Mason, Wendy Havens

Administrative Updates (Allan Eustis):

  • WRT recording of last joint telcon, we were able to filter out a lot of the noise from the joint HFP/STS meeting on the 3/1 so it's a little easier to understand Whitney's comments. (Posted at:http://vote.nist.gov/HFP/HFPSTS030107.mp3)
  • TGDC Plenary Meeting: Informal reception on the evening of the first day's meeting (22nd) at 6:00 p.m. The internal website <http://vote.nist.gov/TGDC/TGDCmeeting032207.htm> is beginning to be populated with information for the meeting. CDs will be mailed as usual to arrive one week in advance. Completed Portions of the draft VVSG will be included. A draft agenda will be distributed soon.

Quick Updates:

  • The HFP VVSG chapter draft for review at the plenary has been submitted for printing as a draft meeting document. Document is pretty much the same as discussed at meeting when we went over it point by point. Any problem areas are indicated in brackets.
  • Had an internal discussion regarding system safety requirements. At one point we had an OSHA citation that has now been resolved. We will be citing U.L. 60950 in the VVSG. There should be a requirement regarding the test labs performing the tests or subcontracting them out.
  • HFP had a joint meeting with STS on March 1 regarding SI and accessibility issues. Action item was to rewrite the white paper making technical changes and adding explanations. John Cugini has drafted a tentative requirement:
    • Long complicated discussion - bottom line is that relying on audio is probably OK, because audio would be verified/tested by sighted voters. It is desirable to urge vendors to supply a reading mechanisms that could take paper ballot and get it read directly. [Draft statement: If voting station generates a paper record for the purpose of allowing voters to verify by their ballot choices, then the system should provide a mechanism that can read that record and generate an audio representation of its contents.] The use of this mechanism is available to voters with dexterity disabilities.
    • This allows voters with sight disabilities to verify their ballots even if indirectly. The verification depends on the integrity of the mechanism. It does not depend on the internal electronic record of the system. SI is for the integrity of the system as a whole.
  • Status of usability benchmarks and how to proceed: We have received an initial technical report on the first round of tests that says our experiments to develop these benchmarks have validity. We have data that can be presented at the plenary meeting. Sharon will put together a summary for presenting. We will conduct further tests in the May/June timeframe.
  • John Wack expressed concerns about usability and paper records. Frustrated that TGDC is appearing to mandate inaccurate elections by mandating paper. (Manual counting is inaccurate and SI does not mandate this.) One area overlooked is usability of paper records that effect poll workers and election officials. John is concerned about reconciliation. It is an important step for auditing purposes.
  • John Wack requested that election officials on the TGDC read material and offer comments regarding usability. Sharon Turner-Buie and Alexis Scott-Morrison promised comments.

Allan Eustis mentioned that a possibility for the following plenary meeting would be end of May.

Allan announced that he had received notice that Senator Obama had introduced a bill to amend HAVA creating a new office for a voter advocate at the EAC. (See: http://electionlawblog.org/archives/di-bill.pdf).

 

Meeting adjourned at 11:30 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

Joint Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP) &
Security and Transparency (STS) Teleconference *
March 1, 2007, 11:00 a.m.
Draft Minutes

Agenda

1. Discussion of new version of Software Independence and Accessibility for the voter.
2. Discussion of where do we go from here for the March TGDC meeting.

Attendees: Alice Miller, Allan Eustis, Barbara Guttman, Commissioner Davidson (EAC), David Wagner, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Nelson Hastings, Ron Rivest, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbury

Discussion of New Version of Software Independence and Accessibility of the Voter:

This topic was discussed in great length. Sharon Laskowski would like to have as output/actions from this call: suggested changes to the white paper that will make it presentable at the March 22, 23 meeting, including John Wack's concerns about usability of audits. Given the pros and cons of the various approaches outlined in the paper, if we want strict SI, voter accessible verification, good voter accessibility and usability, and easy auditability, the problem has been over constrained.

Sharon has received Whitney's comments via email, and most of them will be integrated into the document.

Sharon wanted to start with Ron Rivest's comment, "At a high level it should be made clear that SI and auditability cannot be sacrificed, they are essential for voting system integrity. Voter verification and accessibility of the verification step may be adapted as best possible, but they are not so critical, technically, for the integrity of the election results." Ron feels this is a multi-dimensional problem and there will have to be some trade-offs.

Commissioner Davidson expressed concerns about the realism of getting this done. Also, VVSG 2005 has aduditable function in the guidelines.

HFP's main concern is accessibility. There's some compromise with usability that can be made.

Solution 1 that is proposed in the paper is accessible in that no one is excluded from voting individually. The verification step is not accessible, but the system integrity is preserved and the voting system as a whole is accessible. The question was asked if this was enough, or did the verification step on its own need to be simultaneously accessible and software independent? From the security point of view, it's enough for the integrity of the election for the system as a whole to be accessible.

Discussion continued about audio read back features for verification. Concerns were expressed over preserving this read back on tape for auditability, tapes are fragile and costs would be large. The proposal to have ballots scanned by a different system and read back for verification by voters with disabilities was discussed. This idea seems to be feasible, there have been prototypes. This should meet security needs if observational tests are performed on these systems. The difference between readback from internal memory of a voting system as compared to readback from a print out, re-scanned summary fed into a separate machine for readback was discussed.

Tricia Mason pointed out that security is very important, but its also very important that we do not disenfranchise anyone from the voting process. David Wagner expressed agreement in pointing out that accessibility in the sense that we done disenfranchise anyone is critical, that integrity of the overall system is critical, the ability for every voter to directly verify the record is not critical, and given that it is going to be difficult to find systems that meet all our goals, we may need to compromise some and consider observational testing adequate. We may want to encourage enabling as many people to directly verify the independent record as possible, but not make it a requirement.

Ron Rivest pointed out that we needed to stick with the original notion of SI - understood by the typical voter.

Requirements for verifiability for voters with disabilities can be covered with the readback feature, and we can use observational testing to make sure it is trustworthy.

Capability to do voter verification will be written as a "shall", direct verification will be written as a "should". Readback capabilities from scanning ballots will be written as a "should".

Auditability:

Having a system that is auditable in principle but not in practice is not good. Auditing off paper is going to be difficult without additional aides. How do we write requirements? Having test labs run through sample audits is a good start. Bar codes to track precincts, etc. could be used. Good labeling on ballots would be useful. Problems arise from paper spools when early voting occurs and voters from different precincts vote at one. Paper spool requirements need to be written.

John Cugini felt that a task analysis needed to be done which would give the committee a starting point. There is not much currently that can be said in the auditability column of this paper. HFP/STS may want to consult with Dan Schutzer who had some ideas on this point.

ACTIONS FROM TODAY'S MEETING:

  • Sharon has a number of small edits to the white paper.
  • Sharon will do an outline of the pros and cons of voter verifiability with voters with disabilities and the different options.
  • John Cugini will write draft recommended requirements.

The white paper needs more review before presenting as a white paper to TGDC March 22, 23. Whitney wants to point out procedures that go along with it. Feasibility concerns or where implementation doesn't exist, needs to be noted.

Discussion/review to continue via email.

Other Items:

Alice Miller mentioned a couple of resolutions that came out of a Standards Board meeting. It was decided that these would be discussed in detail at the TGDC Plenary Meeting.

Meeting adjourned at 12:15 pm

 

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 

 

Teleconferences from 200420052006 and upcoming in 2006.

*************

 

Joint Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP) &
Security and Transparency (STS) Teleconference *

February 23, 2007, 11:00 a.m.
Draft Minutes

Agenda:

1. Software Independence and Implications to Usability of Audits
2. Review of Software Independence Approaches Chart

Attendees: Alexis Scott-Morrison, Allan Eustis, Barbara Guttman, Bill Burr, David Baquis (U.S. Access Board), David Flater, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Mohammad Mareuf , Nelson Hastings, Philip Pearce, Ron Rivest, Sharon Laskowski, Sharon Turner-Buie, Wendy Havens, Whitney Quesenbery,

Administrative Items:

  • Allan introduced Mat Masterson, who will be working with Commissioner Davidson as a liaison between EAC and the TGDC.
  • Sharon Laskowski expressed a desire to have a consensus at the end of the discussion that this is the paper we want to release as an explanatory document at the next plenary and have some action item in the form of requirements in the accessibility section acknowledging how to address SI in accessibility.

Ron Rivest reported that a high level issue here with software independence- we have to be able to do audits - audits are how we detect errors. The auditing process itself is part of the whole package and must be usable by poll workers in a reasonable way and this must be tested. Ron and STS are looking to HFP for guidance on how best to do this from a usability stand point.

This topic about usability of SI and usability of audits was discussed in great detail.

  • Concerns were expressed over audio playback being slow, but it was pointed out that when auditing records, this was usually handled by one poll worker reading to another.
  • The record itself (the ballot being audited) can't be something that is degraded by being handled - this may be a core requirement issue.
  • A description of the envisioned auditing process should be provided by the vendor, as well as a time analysis.
  • For VVSG 07, what are the requirements, what kind of systems would fail?
  • Alexis expressed concerns that TGDC was stepping out bounds beyond writing requirements for systems and telling election administrators that they needed to conduct audits. It was clarified that TGDC is not requiring audits, what it is requiring is that if a state requires an audit, the voting system must be capable of performing the audit.
  • From a usability stand point, we're saying that "the accompanying audit procedure for the system must be usable." The vendor must clearly expose the process they envision for an election official/jurisdiction to use to conduct an audit, and an assessment of the time required to conduct the audit.
  • We can not do benchmarks or design guidelines at this point.
  • What would make a system fail? Without design guidelines, how do you decide what fails a system?
  • What would make an audit process unacceptable to an election official? Whatever the process is, it can not degrade the ballots/information.
  • One problem with a recent election was that the print out record did not match the electronic record in the type of information that was provided. This concurrence would be a necessary requirement.
  • John Wack mentioned the possibility of including bar codes that contained specific records and also keeping a bitmap file of the ballot.
  • Whitney pointed out that we need to keep two things in mind: we should not write requirements that cause harm, and we need to make sure that a certified system is a good (usable) system.
  • John Cugini pointed out that the paper being discussed covered high level requirements and potential ways to meet them.
  • John Wack inquired if we should put a statement in about "more study necessary", and this is "a safe way to do it for now"? John Cugini and Whitney agreed that this is always a good idea to have further research.
  • Reminder this is a discussion paper about what we should include. It would be nice to have feedback from EAC and other election officials.
  • David Flater suggested looking at 1990 VSS requirements regarding paper ballot stock and degradation requirements for wording.
  • A requirement must be keeping a durable (non-degraded) image of the vote for 22 months, and must survive numerous trips through auditing/voting system.

ACTION: Write high level requirement that states systems should be auditable and vendor must document procedures. We should look at specifics, such as re-reading ballots. Vendors must provide process so test labs can perform necessary tests. We must document what makes an audit technically possible and usable. Write definition of durability for surviving 22 months as required for voting records.


Reactions to SI Approaches Paper:

John Kelsey was happy that the procedural defense was added to the paper and feels that this provided the SI requirement - this is the one where users without disabilities are asked to test the accessible systems for voters with disabilities. Whitney feels that it will not pass through the Holt Bill.

Whitney does not believe we have an existing accessible system that will receive high marks in the 3 columns of the evaluation process noted in the paper.

John Wack mentioned that David Wagner felt that with IDV systems, there were no accessible systems that were SI.

Ron was asked to clarify the definition of SI - errors are detectable in principle, either during voting stage by the voter, or later from evidence results of the audit. It does not require the voter paying attention to what's going on, i.e. it doesn't require that the audit be used but the auditing equipment should be trustworthy.

John Wack felt that we need verification for audits and forensics for recounts. For example, we would preserve a recorded record of the audio verification provided to blind voters.

David Baquis indicated the Access Board was in favor of having all voters be able to verify their votes at the time they vote. David felt that the audio recording of the verification could be played back for the voter as a confirmation that what they just heard was what was recorded for later auditing if necessary.

ACTION: Simplify table in paper and add an extra column. Review section 3.2 to implement/consider David Baquis' /Access Board comments. Ron Rivest will provide comments via email. Barbara pointed out that comments should include the section, whether its yes or no, and how to make it yes if feasible.

Next joint HFP and STS meeting, Thursday, March 1, 2007.

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

 

Teleconferences from 200420052006 and upcoming in 2006.

*************

 

Joint Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP) &
Security and Transparency (STS) Teleconference *
February 9, 2007
Meeting Minutes

Participants: Secretary Gale, Allan Eustis, Wendy Havens, Benjamin Long, Barbara Guttman, Nelson Hastings, John Kelsey, John Cugini, David Flater, David Baquis (U.S. Access Board), Philip Pearce, Whitney Quesenbery, John Wack, Tricia Mason, Ron Rivest, David Wagner, Sharon Turner Buie

Agenda

1. Administrative Updates (Allan Eustis and John Wack)

2. Software Independence and Accessibility for the voter (Sharon and Nelson) -- Review of Initial white papers (We also need a better term than "end to end" accessibility, such as "complementary accessibility".)


3. Software Independence and implications to usability of audits (Sharon and Nelson) SI without innovation classes imply paper records. Is there anything we can do to improve the usability of audits of the paper?

4. Other issues (Safety -- need to hand off to safety expert?)

5. Next meeting topic: usability benchmark paper

[Next HFP teleconference is scheduled for: Friday, February 23, 2007 at 11 AM ET]

 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecom were preliminary and do not necessarily reflect the views of NIST or the TGDC.]


Administrative Updates: 2/9/07 Testimony to the EAC by David Alderman of NIST on NVLAP accreditation progress has been posted to: http://vote.nist.gov/NIST-CA-activities-under-HAVA-020807.pdf).

Software Independence and Accessibility for the voter: Whitney mentioned that starting point for this discussion has been that all aspects of the voting process need to be accessible or accommodated in an appropriate way. "Complimentary accessibility" seems a better term than "end to end" and avoids confusion with STS cryptographic use of the term "end to end". Ron agreed that verification for all voters sits at the intersection of security and usability. STS is looking for guidance and collaboration from HFP on the right path forward. Whitney pointed out we need to determine exactly what we are verifying.

(1) You need to be able to audit the election.
(2) Voter needs to be able to review ballot.
(3) Voter needs to be able verify vote is counted.

We are focusing on (2). John Cugini offered 4 possible technical approaches/solutions for discussion. He stressed that SI solutions are oriented towards blind voters. These are not mutually exclusive:

(a) Use audio as is (review of audio ballot sufficient)
(b) Introduce paper into process (such as EBM or OCR) and have device read back audio
(c) Use of audio tape to directly record what the voter is hearing
(d) Innovation class solution (E.g. independent "witness" system like Scytel reads back and records vote )

(a), (c), and (d) are paperless solutions. All have pluses and minuses. Some are not ready for prime time.

Discussion initiated by Ron Rivest on what of the four truly meet software independence. Tampering could occur with (a). Demanding SI for all voters may not feasible.(i.e. (a) may still be acceptable; (b) meets independent verification (IV) requirement. It does require an independent reader device; (c) also in (IV) class.

Dave Wagner reviewed his understanding of four technical approaches/solutions. First three appeared SI to him. (d) did not appear as SI. Whitney also reviewed her understanding of the four technical solutions from accessibility stand point and production of durable records.

Philip Pearce asked if any of the approaches violated private and independent verification requirement for all voters. John Kelsey said they all are private but the main issue is security of each approach. Whitney agreed and reviewed independence of each method; Issues of sampling size with some approaches. Philip Pearce posed question of how much responsibility we have to decide the "best" method. What are sufficient techniques?

David Baquis added four different challenges that need to be considered in this matrix:

1) those with low vision that do not routinely use audio
2) those with dexterity/mobility disabilities
3) those with cognitive disabilities

Whitney referenced a recent presentation at on cognitive disabilities presented at recent access board meeting. (See: http://www.access-board.gov/sec508/refresh/teitac3rd/clayton.ppt). Point here is that cognitive disabilities are on a spectrum with respect to barriers. We should not gloss over usability issues.

Whitney and participants reviewed the approaches (a)-(c) for each of the disabilities brought up by David Baquis. A magnifier could be used for those with low vision who do not use audio support routinely.

John Kelsey synthesized issue as a determination what appears on the screen and does not make it on to the papers that are relevant to the vote the voter he/she was casting. Also there are issues here related to alternative languages. Issue here is "one ballot versus many ballots "of alternative language ballots on a specific machine affecting privacy of vote. There are election procedures here to mitigate the loss of voter privacy.

David Wagner had concerns with co-mingling of ballots. Sharon Turner Buie pointed out that paper ballots are co mingled as they are read into the machine. Concern here by Wagner of one Chinese voter at that polling place. Turner Buie pointed out that ballot is tabulated irrespective of the language used to vote. (John Gale agreed.) Only in a recount would you possibly be able to detect the voter's identity. (Wagner expressed ongoing concerns related to privacy issues.)

Quesenbery brought up dexterity issue and transporting paper as a barrier to independence requirement for the voter. Scanner device was discussed. Does assistance with the paper ballot remove independence for the voter? We will get push back here.

Secretary Gale noted that state and local election officials operate a voting system, not just voting equipment. Trained election officials and poll workers operate the equipment within a system context. Election officials do their independent testing and assessment of the equipment to ensure that it is working accurately and effectively before an election. You do not want to discuss these issues in a vacuum separate from an election system. Our focus here seems to be to drill down to require the individual voter to do the assessment and evaluation. Meanwhile studies indicate that 30% or less of the voters verify their vote. If the average voter has trust in the certification and verification of the voting system, they are going to accept probability that the system is working correctly. That is they do not need to be responsible for reading the installation manuals etc. An analogy here: when we fly a commercial airplane we trust the system. We do not feel the need to read the service manuals or review the qualifications of the pilot because others do that. Are we making voting complicated for the average voter because we do not trust the election officials or the voting system to operate correctly? Should we not focus on assuring that the voter has ease of access and has a sense of confidence in the casting of their ballot? If we don't trust the software, then we should have independent audit verification of random precincts that use DRE equipment with VVPATs. Those selected precincts are going to have to count the VVPATs and compare them with the electronic result. That way, election officials are addressing the verification issues for all the voters in the precinct without passing the responsibility on to the individual voter.

Rivest agreed that the voter verification step is important from a security viewpoint if it is done by a sample of the voters. The idea of a random audit of the precincts is a good one especially with VVPATs.

Philip Pearce commented that, in our discussion, we should be asking the question for whom the voting equipment is working accurately and appropriately. Is it allowing voters, whenever they wish to do so, to ensure that the equipment allows for private and independent verification of their votes?

Secretary Gale agreed with this assessment, and commented that the voter had to have trust in the system's checks and balances. We seem to be in new arena of voter responsibility independent from the system. We should also realize that no equipment is going to be perfect just as no election is perfect. You have to rely on the system as a safety net back up.

Wagner noted that we are talking about voting systems producing non electronic records that will be used for permanent records and allowing voters to verify their votes. We want to eliminate barriers to the disabled voters' ability to verify his/her vote. However we are not giving the voter the right to follow the vote through every step in the process. We simply want to preserve the security.

Summarizing, Quesenbery commented that elections are indeed complex. We are not looking for full accessibility of the process; we are looking for equivalence. If the paper ballot is the ballot of record, we are looking for means to ensure that all voters can review that ballot. We do not require that all systems operate in the same way. We look at how voting system architectures satisfy the requirements for all voters. This makes it difficult to write guidelines to ensure that voters with disabilities can fully participate. The next step requires an answer to the question of what makes the ballot auditable. For example there are definite challenges in auditing an audio tape.

Discussion ensued on (a) thru (d) above relative to whether they satisfied as a sufficient SI technique. Each was reviewed. Rivest provided context for SI in terms of errors that are capable of causing changes in the election outcome without any detection. (Will an audit detect something wrong with some probability?) In (a) a sighted a voter could listen to the audio and determine if it is different than what is printed on the paper. This is an SI capability. Qeusenbery noted that we have to answer two questions. Is the approach sufficient to meet SI and is it sufficient to meet accessibility. (a) Meets accessibility. John Kelsey suggested a formal procedure to encourage (enabled) voters to vote on accessible audio ballot voting systems.

Quesenbery and Cugini determined that VVSG 2007 has requirements that audio and video interfaces work simultaneously. Rivest noted Selker's research indicating that sighted voters who used audio voted more accurately. Cugini cautioned that we are writing an equipment standard and can only recommend procedures.

Concluded that (a) might be SI, but is certainly accessible. (b) Produces a paper ballot. There is a way to read it for blind and low vision (magnify). There are transportation challenges for (a) and (b). The final (c) is accessible and auditable. But the ease of auditability is a factor.

There are issues with the first three. The fourth is still an unknown. We need a chart with all the issues laid out. We have maybe's for SI at this point.

Point is to scope out all cases before presentation and acceptance/rejection by full TGDC. HFP and STS Sub groups will need to agree first; need to get away from the theoretical to the practical. Rivest offered the opinion that (a), (b) and (c) have workable SI solutions. Gale noted complexity with (b) for transportation issues.

Cugini asks that we not downplay (d). There are prototypes of this architecture that help solve (b) issues.

Action item: NIST will document in a list/table format the four technical solutions with auditability and usability concerns clarified. The table will be sent out to STS and HFP for review before next joint meeting.

Software Independence and implications to usability of audits: This will be covered at the next HFP meeting. We will also plan a joint meeting to review auditability issues as well as resolving issues from this telcon.


[Next HFP teleconference is scheduled for: Friday, February 23, 2007 at 11 AM EST]

 

Teleconferences from 200420052006 and upcoming in 2006.

*************

 

HFP Subcommittee Teleconference*
Friday, January 26, 2007, 11:00 a.m.

Agenda:

1. Administrative Updates (Allan Eustis, John Wack)
2. Discussion of proposed edits to the VVSG based on our December meeting (Sharon and JohnC)
3. Other issues
4. Next meeting topic

Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, Ben Long, David Flater, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery, Ben Long

Administrative Updates:

  • Allan E: Began working on logistics for the March TGDC meeting. Deadlines for submission: draft of VVSG2007 by COB March 1; white papers for TGDC workbook by COB March 5; and Power Point presentations by noon March 15.
     
  • Allan E: Thanks to Barbara Guttman, and the willingness of Ben Long, we have a full-time lab manager for the voting equipment used in research for human factors. Ben will be an excellent addition to any research done at NIST on voting equipment.
     
  • Allan E: All ballots donated by Richard Nemi have been posted. These range from 1998-2006 and represent every state. (See: http://vote.nist.gov/ballots.htm)
     
  • John W: Attended the advisory board meeting on Tuesday and presented a short overview on our status. In regards to HFP issues, Jim Dickson doesn't feel he's getting an adequate answer from NIST on the use of icons or pictures on ballots. Concerns were expressed that we need more usability testing for poll workers (NOTE: section already covered in current draft). Requests were made that we include e-poll books in the VVSG 07. (Whitney stated that we need clarity on this from EAC, this is not in our scope. TGDC members have not seen these systems.)
     
  • Tricia Mason covered activities of the panel she was on at the advisory board meeting. Panel was specifically on accessibility and people with disabilities. Several presentations were given and two proposals were put forth. The proposal asking EAC to describe how voting systems are currently being evaluated for accessibility passed.

Discussion of Proposed Edits to the VVSG (John Cugini):

John Cugini had passed out a list of HFP issues raised at the December 2006 TGDC meeting. The subcommittee reviewed each issue:

3.1.3 - HAVA and VVSG. After drafting this general overview section, there is nothing specific to HFP. It should be moved to the general introduction by CRT, but HFP should put comment in about where it is. Change "voting equipment" to "voting system" (BE CONSISTENT), as voting system includes documentation and we are discussing what you get complete from the vendor.

3.2.2 - Clarify that when a voter receives a warning about under or overvoting that they are given the chance to fix the problem after the warning.

3.2.3-C - Should the specific guidelines point out that the plain language requirement is only for English and may not apply to other languages? Subcommittee feels this should be left alone.

3.2.4-C - Should all voter settings be adjustable throughout the voting session? Subcommittee feels that there should be "on demand" choices, but that the "switching mode" option should be a start over option. John Wack to ask EAC if we can pass this suggestion around and bring up at ITAA vendor meeting to get feedback.

3.2.4-E - Requirements should mandate at least two font size choices on all editable systems (not just accessible equipment).

3.2.4-H - Visual access to VVPAT. The goal would be if you make a selection on one piece of equipment, that it follows through to the next such as the printer. This will be discussed at joint meeting with STS regarding VVPAT. However, the requirement about physical posture of reaching both voting system and printer should remain.

3.2.4-J - Requirements should mandate at least two contrast settings on all editable systems (not just accessible equipment).

3.2.5.1 - Timing issues. Should we make a requirement about keyboard response time? No, not unless it proves to be a problem during testing. However, change the requirement to read "voter inactivity time" instead of "system response time".

3.2.5.1-D - Change "system activity indicator" to "system response indicator".

3.2.4.1-E - Voter inactivity time. Should we specify a time? If we did it would have to be on the high end. Leave as is for now.

3.2.7.1-A.4 - No receipts. We need to add a requirement stating that a receipt shall not be able to provide proof to another about how someone voted.

3.2.8 - Usability for poll workers. Should we mandate a state option whereby a poll worker could specify a state for which a system should be configured? NO.

3.2.8.1-A - Change "average" poll worker to "typical" poll worker.

3.2.8.2-B - Maintenance documentation requirement should be taken out and covered in documentation section of VVSG.

3.2.8.3-A - Should we change the regulations to reference and IEC standard? No, leave as is. We should state that systems must meet Federal Standards that are available.

3.2.8.3-B - Is the "elimination of hazards" section redundant? Leave in for now until further research is conducted.

3.3.5-B - Allowance for an assistant. Clarify spacing issues stating "adequate physical access".

Next meeting will be a joint HFP/STS subcommittee meeting on February 9, 2007.

[*Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. These teleconferences serve the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this teleconference are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

 

HFP Subcommittee Teleconference*
Friday, January 12, 2007

Agenda:

1. Administrative Updates (Allan)
2. Discussion of current and planned activities (Sharon)
3. Other issues

Participants: Alexis Scott-Morrison, Allan Eustis, David Baquis, David Flater, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery

Administrative Updates (Allan Eustis):

  • New disclaimer will be read at the beginning of every telecom. Meetings have been formally announced in the Federal Register. What is said at these meetings is public and NIST welcomes any and all new listeners.
  • New articles, documents, and public comments have been placed on the TGDC website.
  • Mark Skall and Allan attended Donetta Davidson's installation as Chief Commission of the EAC. Outgoing chief expects approval of new EAC members in January.
  • Transcripts from the TGDC December 4 and 5 meeting are on web page.
  • Any combined TGDC subcommittee meetings will be worked into current schedule of meetings. New meetings require a couple weeks notice to schedule.

Discussion of Current and Planned Activities for the Next Generation VVSG

Whitney stated that the subcommittee should set a goal to have all HFP related edits completed before the March TDGC plenary meeting for the VVSG2007, with an exception to security related usability material. After edits are complete, the next step will be to work on security and verification - this is a hot topic including potential upcoming Congressional activity. Third, HFP needs to deal with usability benchmarks, making sure we have the resources to say in July that we know where we're going with particular tasks. Fourth is ongoing research. We need to decide what are the immediate needs for the Next Generation VVSG and what are test methods and suites that can be worked out after July delivery (we have started discussing these in greater detail). Requirements will be in this version, including general testing requirements. Details of the test protocols can go in the test methods written after July.

HFP has been having continuous discussions regarding examining audit methods and usability in terms of some of the security methods that are going to apply to election officials and poll workers, in relation to software independence. These two can be bundled together

Still struggling with specific method and implementation design e.g. barcodes for easy recounts. Questions arise when you get to explicit implementation design - how do you write requirements that are high level but still useful.

The subcommittee reviewed the list of top four items on their list.

John Cugini has almost completed the (usability) updates to the VVSG from the comments received at the December meeting. He will complete them and have them ready for discussion at the next HFP meeting. Any issues will be discussed via email beforehand.

Usability Benchmarks - It has become exceedingly obvious that you can not just pick a number based on a data set. You have to know how to analyze and know if you're using the right statistical methods. HFP has received expert advice on this matter and is close to saying "this is how we're going to count" and "how we plan to analyze". We may need 100 subjects (ROM) to test. Tests are being conducted on error rates noticed when people are instructed how to vote, we are not currently looking at voter satisfaction or time periods.

Specific details on what test methods were discussed. There should be guidelines for expert review. Questions arose about what does broad guidance mean for technical documentation? Are accessibility benchmarks written as test methods? These will not be included in the July deliverable. Can addendums be added to this version of the VVSG? This will be brought up to the EAC for answer. John Wack will find out what the process for making additions and corrections will be. TGDC may also want to approach this topic with EAC. Allan pointed out that the document, once reviewed by EAC, would be sent out for public comment. Final will probably not be released until 2008. David Baquis wanted to know what EAC's process was for making changes. (Same public review process as for previous versions of VVSG.)

David B. felt that HFP should seek out involvement from disability groups and accessibility consultants in its discussion regarding possible test methods. (This would have to be done through the EAC). (There will be a cognitive expert at an upcoming TEITAK Telecommunications and Electronic and Information Technology Advisory Committee disability - meeting that might be a useful resource.)

The next two teleconference meetings will cover end-to-end testing and usability benchmarks, respectively.

David B. suggested re-titling end-to-end to read "complimentary accessibility". Whitney's two major points being emphasized are how accessibility and accommodation can work hand-in-hand and second; that it's not enough for each piece of a component in a system to be accessible, but you have to look at it as a whole system - from the voters point of view and all the things a voter has to do. John Cugini questioned if we get performance (test based) requirements for accessibility written, would this be the basis for this kind of test?

How do you make sure the systems overcome the challenges? Part of the documentation a vendor should submit to the test labs is an explanation of their system and an explanation of how it is intended to be used so that it can be tested in its intended method of operation - documentation should be lay out steps that are assumed people would go through. If there are different possibilities, those should be laid out as well. This would help people that are constructing usability tests. John Cugini inquired if this would require adding new requirements. Yes.

At the next meeting, HFP should decide what specific issues involve STS coordination. Also what does it mean to be an "accessible" secure system?

Next meeting, Friday, January 26, 2007, 11:00 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. These teleconferences serve the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this teleconference are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

****************

Created January 16, 2020