Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
TGDC Subcommittee Work - Historical Meetings - HFP Meetings - 2007
|
Technical Guidelines Development Committee (TGDC) AGENDA
Attendees: Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Whitney Quesenbery Administrative Updates (Allan Eustis):
Benchmarks (Sharon L):
List of Notable Changes (John Cugini):
OTHER:
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Attendees: Alexia Scott-Morrison, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, Nelson Hastings, Whitney Quesenbery, Philip Pearce Administrative Updates (Allan Eustis):
Proposed Benchmark Changes: Whitney reviewed the latest discussions that had taken place regarding the usability benchmarks being proposed. There are issues with putting in numbers because research is still being conducted. It was felt that there should be no difference in the effectiveness goals whether the voter had disabilities or not, but now additional tests are being conducted (initially in Chicago and Denver) to broaden demographics and to include individuals with disabilities. The requirements in the VVSG will have usability requirements for conformance testing and reporting with a statement that they are based on current data and that further research is still being performed. Based on the current research, the VVSG will recommend that the benchmarks be set at a specific number. The question to be asked of the TGDC at the plenary meeting will be how high to set the benchmark numbers. Currently they are set at a medium range, should they be higher? Should we be setting them so that vendors have an aspiration goal for improvement? These are questions for the TGDC and the EAC. [NOTE: Test performed to date has been on systems certified to VVSG 02 and used in the 2000 or later elections.] Other:
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) Agenda:
Attendees: Alexia Scott Morrison, Allan Eustis, Alicia Clay, Barbara Guttman, Bill Burr, Elle Colver (EAC), John Cugini, Secretary John Gale, John Kelsey, John Wack, Mark Skall, Mat Masterson (EAC), Neal Erickson, Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery Administrative Updates:
Usability Benchmarks and Presentations: Per instructions from Dr. W. Jeffrey, Sharon is writing an executive summary to clarify for the general public the benchmarks white paper. A change list of HFP's changes has been circulated - the new stuff having to do with the benchmarks material. The TGDC needs to discuss the policy/philosophy of the benchmarks. The benchmarks may need to be pushed higher, that should be discussed at the plenary. The benchmarks we've chosen to set are ballot casting (98%), 100% correct casting rate (70%), and accuracy rate/index (.35 w/possibility of moving to .4). [NOTE: Benchmark numbers can be changed during public comment period.] The two items that will be covered regarding benchmarks at the July 3rd meeting will be a discussion on the demographics and the accuracy index. HFP feels that the accuracy index benchmark can be raised so that vendors strive for better systems. Full TGDC input will be requested. Nelson Hastings summarized the changes to chapters 7-14. For all STS sections, "voting equipment" changed to "voting device" as well as other modifications to harmonize with other VVSG sections. All the security documentation requirements were moved to volume 4 with other documentation requirements.
Meeting adjourned at 12:15 p.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Attendees: Alexia Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), Elle Colver (EAC), John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, NIST Statisticians, Sharon Laskowski, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Significant changes to Usability, Accessibility, and Privacy since the May 2007 TGDC Meeting (John Cugini):
Benchmarks White Paper (Sharon Laskowski): Sharon reviewed in detail the benchmarks white paper - a final draft of the paper should be available by next Wednesday. This paper will be presented at the July 3rd plenary meeting. The goal of the white paper is to be able to present to the TGDC and EAC, in plain language, the overview and basic understanding of the usability benchmarks. The white paper summarizes the research conducted to develop the test, metrics, and benchmarks in response to a TGDC resolution and describes the resulting performance-based requirements included in the current draft of the VVSG. Supporting documentation of test data and test materials are included appendices. The report defines the test: the ballot and the instructions used and the test participants. It defines how the system will be measured: effectiveness, efficiency, and satisfaction (efficiency and satisfaction numbers will be reported but will not be used as benchmarks to pass/fail a system). The paper describes tests used to validate the results and processes used to analyze the results. The paper also explains what was done to show that we could get repeatable results from the test protocol. Sharon then explained how the benchmarks were set, explaining that the confidence interval was a repeatability measure. The research described in this white paper shows the validity and repeatability of our testing methodology and allowed HFP to determine benchmarks and specify usability performance requirements for voting systems. This methodology is technology-independent. Any system can be tested to these benchmarks. The current explanatory white paper still needs to be shortened to about 5 pages before next week. Sharon has begun working on a more technical document that will provide a thorough analysis and will hopefully be ready when the VVSG is delivered to the EAC. Anyone wishing to analyze the data to review how the benchmarks were obtained will be able to do so using the raw data that will be made available. After the VVSG is delivered to the EAC, time will be spent researching test methods. Tests have to be run to get reproducibility by the test labs. The flexibility of the test protocol has to be determined -- to ensure that this methodology can be reproduced in any VSTL, we will be performing research to determine how precisely the test conditions must be followed as part of our work on VVSG test method development. It needs to be determined how test methods are presented to the test labs. Other: It was proposed and accepted that at the next HFP subcommittee meeting members of the STS subcommittee would give an overview of the materials in the STS chapters (Chps 4-12). Whitney feels that a pre-brief will better help members get prepared for the July 3rd meeting. It was reminded that the CRT subcommittee has posted a change log <http://vote.nist.gov/TGDC/crt/index.html> for review. NEXT HFP MEETING: Friday, June 29, 2007. Meeting adjourned at 12:00 p.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) Agenda
Attendees: Allan Eustis, David Baquis (U.S. Access Board), Elle Colver (EAC), John Cugini, John Wack, Nelson Hastings, Secretary John Gale, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates:
HFP Chapter Discussion:
Process for July 3rd Teleconference
Our next telecon is scheduled for: Friday, June 22, 2007 at 11 AM ET. The subcommittee will be discussing the benchmarks write-up. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Attendees: Alexis Scott-Morrison, Allan Eustis, Commissioner Davidson (EAC), David Baquis (U.S. Access Board), John Cugini, Mat Masterson (EAC), Sharon Laskowski, Wendy Havens, Whitney Quesenbery (late) Administrative Updates (Allan Eustis):
Modifications to VVSG Chapter (John Cugini):
Usability Performance Benchmarks (Sharon Laskowski): There were four different types of voting systems tested, with about 50 participants for each system. We'd like to have about 100 people each to tighten the confidence intervals. Using the statistics collected and analyzed, we will have benchmarks for completion rate, perfectly cast ballots, and accuracy. The voting systems used had been used before for federal elections and they were only certified with VVS 2002 standards. The fixes that need to occur on the voting systems appear to be easily accomplishable; we do not feel we are setting impossible benchmarks. Other data was also collected and will be reported for time and subjective satisfaction (there will not be benchmarks for these). The TGDC agreed that accuracy measures could be a pass/fail benchmark and that time/efficiency/satisfaction would be informational data points. Sharon will pick benchmarks over the weekend and fill them into the standard. The data used to set these benchmarks will be available for anyone wishing to see how numbers were reached. The test protocol will be made public. Other: David Baquis brought up the issue of two font sizes and the requirements having to do with magnification. It was decided that the requirements would be left as written. Our next telecon is scheduled for: Friday, June 15, 2007 at 11 AM ET [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Attendees: Alexis Scott-Morrison, Allan Eustis, David Flater, John Cugini, John Gale, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Discussion of VVSG Edits (John Cugini):
Confusion in Public Comments:
Benchmarks:
Next HFP meeting scheduled for Friday, June 8, 2007. Meeting adjourned at 12:00 p.m.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), David Flater, John Wack, Sharon Laskowski, Tricia Mason, Whitney Quesenbery Administrative Updates:
HFP Plenary Presentation (Sharon Laskowski): Sharon emailed everyone the draft presentation before the meeting. This subcommittee meeting was devoted to review of that presentation:
Meeting adjourned at 11:50 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis/John Wack) :
Review of Latest Draft of Usability and Accessibility Chapter (John Cugini):
Discussion of Benchmark Timeline (Sharon Laskowski):
OTHER:
Meeting adjourned at 12:10 p.m. Our next telecon is scheduled for: Friday, May 18, 2007 at 11 AM ET [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Attendees: Alexis Scott-Morrison, Allan Eustis, David Baquis (U.S. Access Board), John Cugini, John Wack, Jon Crickenberger (NVLAP), Nelson Hastings, Philip Pearce, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Discussion of Chapter Edits (Sharon Laskowski): Sharon forwarded a copy of the issues/concerns from last meeting to Whitney and Tricia. Whitney agreed with the changes as proposed. Today's meeting will cover two major issues that can hopefully be resolved to be put into May 9th draft due to Allan Eustis. Sharon's proposed rewording for the accessibility read back requirement: "If the Acc-VS generates a paper record (or some other durable, human-readable record) for the purpose of allowing voters to verify their ballot choices, then the system shall provide a means to ensure that the paper-based vote verification is accessible to all users with disabilities, as identified in section XXX." The requirement as written is vague and untestable. Sub-requirements need to be written that are clear and testable. Sharon mentioned that STS had concerns about the goals of this requirement - John Wack and Whitney expressed that those concerns have already been addressed. Whitney posed the question about whether standards are being written for current system or future states of systems. HFP has been very clear about writing hi-level and sub-requirements to be very precise about what the requirements are. HFP's requirements deal with individuals mainly as opposed to the election as a whole, which is more to what STS and CRT are writing requirements towards. John Cugini brought up the subject of what the sub-requirements for this requirement should be. Anything besides the audio read back? What about something for low vision voters or people with dexterity issues? Committee discussed the question about electronic ballot markers (EBMs) being used as read back mechanisms. STS subcommittee has a big concern about how much these additional requirements will cost election officials. It was decided that EBMs were acceptable as you could either put the ballot back in the same machine or take it to another machine for read back. Whitney pointed out that this read back is fine since it is not generated from the internal memory of the machine which is not acceptable. The issue of privacy during transport from one machine to the verification machine was discussed. Preservation of privacy is a major issue. Using an envelope to transport was discussed. Not the best solution for individuals with dexterity issues. These things will be tested during the performance tests. We want to make sure that we write requirements that the vendor can look at and tell if there system should pass the tests. It was decided that the requirement would be written with a couple sub-requirements to cover the audio read back, electronic magnification, dexterity (which is in the original standard), and privacy concerns. The audio readback can not be generated from the internal memory. OTHER: Reminder, the benchmarks will not be here before the May 9th deadline. EAC has agreed that the report should be written with the information and the numbers can be changed or added later when they are received. Cost of Testing Meeting: John Cugini attended the cost of testing meeting. Sharon had come up with a rough estimate of $35K to $40K, to which there seemed to be no objection. HFP will have to show how they got there test results. Our next telecon is scheduled for: Friday, May 11, 2007 at 11 AM ET
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Attendees: Alexis Scott Morrison, Allan Eustis, Commissioner Davidson (EAC), David Baquis (U.S. Access Board), John Cugini, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Sharon Laskowski, Steve Freeman (NVLAP), Wendy Havens Administrative Updates (Allan Eustis):
Discussion of Usability and Accessibility Chapter (John Cugini): The purpose of today's HFP meeting was to go over recent changes and issues of the HFP chapter of the VVSG. There will be a final iteration of changes after this via telcon before the May plenary. Major Changes: 1.1.2 - Special Terminology - Common Industry Format (CIF) 1.2.1.1. - A & C - Overall Effectiveness/Satisfaction 1.2.1.1.D - Ability to Vote Without Human Assistance 1.2.1.2 - Vendor Testing 1.2.2.D - Notification of Ballot Casting 1.2.3.A - Completeness of Instructions 1.2.4.H - Visual Access to VVPAT 1.2.5.C - Accidental Activation 1.2.8.A - Clarity of System Messages for Poll Workers 1.3 - Accessibility Requirements 1.3.1.A - Accessibility Throughout Voting System and 1.3.1.A.1 - Documentation of Accessibility Procedures 1.3.1.E - Accessibility of Paper-Based Vote Verification 1.3.4.C - Ballot Submission After the major changes were discussed, John Cugini ran through the document to discuss small wording changes that clarified requirements; changed wording to make them in plain language; and/or added discussions for clarification of requirements. Meeting adjourned at 12:00 p.m. Our next telecon is scheduled for: Friday, May 4, 2007 at 11 AM ET [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Our next telecon is scheduled for: Thursday, April 26, 2007 at 1 PM ET Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, John Crickenberger (NVLAP), John Cugini, John Wack, Matt Masterson (EAC), Nelson Hastings, Philip Pearce, Secretary John Gale, Sharon Laskowski, Steve Freeman (NVLAP), Wendy Havens, Whitney Quesenbery Administrative Updates:
Discussion of Test Methods: This meeting was dedicated to a discussion with NVLAP regarding test methods. The goal of HFP was to take a look at every requirement and try and define an appropriate method for testing it. The requirements fit into three categories: high-level requirements that are specific (tested mostly through performance tests); high-level requirements that are vague (no specific features that map to testing, done via expert judgment testing); and low-level requirements that are specific (these are easy to test and write test methods). Some examples were given. For color blindness, a low level, specific requirement would be that any information displayed in color must also be explained in text. For plain language, the requirement is a high-level, vague saying everything must be written in plain language and avoid passive language unless it is appropriate - this is subjective and can only be tested by experts in this field. John Gale inquired if something over time could change from one testing category to another? John Cugini pointed out that if something passes the high level tests it may not be necessary to run through the low level requirements. As technology changes and requirements change, the test methods will change. One issue for NVLAP is how to choose expert testers. They need to understand the type of expert knowledge that is required so that they can find the correct experts. Also there is the issue of who designs the tests. Up until now the test labs have not only done the testing, but designed the tests. The goal here is to centralize the tests - such as when the Access Board develops usability tests for standards and they should be used in each of the test labs therefore making consistency more attainable in test results. In the past VVSGs, we've only provided requirements, this time we will also be providing test methods. What can the subcommittee do to help NVLAP? What issues does NVLAP have? Some of the standards in the current VVSG are ambiguous. (That will hopefully be fixed in the next version of the VVSG - we give some specific failure criteria and good descriptions about what is being done and what is being checked.) [NOTE FROM NVLAP: NIST/TGDC members who are interested in our test methods are welcome to attend assessment visits to laboratories.] NVLAP would like to require a cross reference matrix saying which test method is testing which requirement. This has never been made an official requirement, and legally it currently can't be forced. Whitney pointed out that we could write within the VVSG what test method went with which requirement. Another problem with testing is that the labs want to test everything on a voting system in one test scenario. This is not practical. NVLAP was glad to see the language about not testing the 20,000 different criteria in one particular scenario. You can not catch every requirement by running one session. The scenarios are trying to come up with realistic variations on the use including people with different disabilities and different things that could happen. Another issue of testing was the testing method for alternative languages. The testing does not specifically have to test every language; it should test the variation of technology (logic, storage, and presentation) that generates the languages. The testing needs to include the way languages are stored and the way they are presented. Other issues that tNVLAP has come across is in the mobility requirements. This includes height requirements for tables. We assume that systems will be used as specified by the vendor, but not always the case. Machines can be tested in the lab but not "fit" appropriately in the polling place because vendor specifications are not followed completely. Reporting requirements for testing has been an issue. The VVSG will include a general testing document that will cover general issues, including reporting issues making it clear who has to report what, when, and in what detail. Testing by the VSTLs cannot cover everything. It was reminded that they can cover high level requirement testing, but the state testing and certification must also be done so that machines are certified in more detail with state and local laws. Whitney wants to make sure that it is clear who is to test what so that no gaps in testing occur. There appears to be an issue that is caused by allowing "straight party" votes. Not sure what the fix of this is. We do have a requirement that states that electronic systems must prevent over voting. The problem is we can't tell how "straight party" voting interacts with individual votes also cast. (Expert) Testers have to go through specific training, which will be documented. This is another place where test methods are invaluable because you can't qualify a person to undefined test methods. Also they will help NVLAP do a better job in accrediting labs, and we will have more consistency across the tests. Expert testers will be covered by having a test method that is documented and validated, that the tester can perform it correctly because they have been qualified to perform that particular test method, and only people that are qualified are allowed to do the test method for final reporting and certification. Meeting adjourned at 12:00 p.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) DRAFT AGENDA
Our next telecon is scheduled for: Friday, April 20, 2007 at 11 AM ET Attendees: Alexa Scott-Morrison, Alice Miller, Allan Eustis, Barbara Guttman, Bill Burr, David Baquis (U.S. Access Board), David Wagner, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Ron Rivest, Sharon Laskowski, Sharon Turner-Buie, Steve Freeman (NVLAP), Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Bar Code Requirements: The focus of the discussion today was to help the HFP subcommittee come to a conclusion on a recommendation about barcodes for the purpose of usability and accessibility. At the last HFP meeting, it was suggested that they be used with assistive technology with three constraints:
After the last meeting Secretary Gale . put out feelers to election officials about their input on this issue. The response indicated that election officials didn't see any reason that a system that utilizes bar codes shouldn't be certified. David Wagner of the STS subcommittee wanted to separate the use of bar codes for recounts and audits from use of bar codes for accessibility. He also stated that bar codes must be able to be turned off or on depending on the user. David has two main concerns. First, not sure if the bar codes accomplish much, there could be the same problem that is caused by generating audio read back from electronic memory (there is a possibility for mismatched data), and second, if we put the information in for accessibility and it contains ballot choice information, then there will be temptation to use it for audits and recounts. Ron Rivest of STS pointed out if you have the same information in two different formats on the same sheet, you have to audit a portion of this to make sure the information is being generated correctly. Ron supports the use of barcodes as long as the vendor supplies an open source for scanning. Barcodes should be readable but machines from different vendors than the voting machines. At the end of the discussion it was decided that draft recommendations would be written. These draft requirements would include:
Vote-by-phone: At the last teleconference vote-by-phone was discussed and Sharon took it to the EAC to find out about certification. The conclusion from the EAC certification manual is that a whole system has to be submitted for certification - you cannot certify components. Therefore, if you wanted to submit vote-by-phone as an accessible system, it must be submitted in entirety with documentation and the rest of the system to make it a complete accessible system. In order for a system to be certified as an ACC-VS, it must meet all the requirements in the accessibility chapter of the VVSG. There are no further action items on this issue. NVLAP Discussion: The purpose of this discussion was to set the agenda for next week's meeting. Accessibility and usability testing is something new to the test labs. They have been discovering issues when testing systems, such as in the mobility accessibility testing and the audio speed testing. Whitney was concerned that testers are not certified to test for accessibility and usability. The current standards seem to be missing details in test methods. Agenda item for next week: What can NIST do to generate test methods in this stage to make sure enough repeatability is capable (how do we write clear requirements). Also, there is an upcoming summit on costs of testing systems. Possible discussion of HFP's input for summit.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Attendees: Alexa Scott-Morrison, Allan Eustis, John Cugini, John Gale, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Tricia Mason, Whitney Quesenbery Administrative Items (Allan Eustis):
Performance Benchmark Testing Update: There are three sets of tests, two of which Sharon Laskowski can report to the TGDC in May. The 3rd big test is set for June 1-2. HFP will be writing the draft requirements and placeholder for the metrics from the June test, these numbers can be revised. VVSG Modifications: Whitney mentioned that we need to make sure that we write requirements that allow for the creation of voting systems that are affordable for very small districts as well as very large and/or complex districts. Costs of voting systems should include the cost of maintaining these systems as well. EAC is holding a Cost of Testing Summit (April 30-May 1) that will cover every possible cost we can imagine from testing systems to maintenance. Should privacy be its own chapter? This is a very important concept, we don't want it to get buried, but is it worth restructuring this late in the game. It was decided that "privacy" would be added to the chapter title and the subsection would be moved up, right after performance requirements. Vote-by-phone: There was a lengthy discussion on the use of vote-by-phone and how accessible these systems were. Vote-by-phone would be good for voters with low vision or blind voters. Accessible voting systems must cover all type of accessibility. We can write the requirement that all polling stations must have at least one system that meets all the requirements of the usability section, providing a unified solution for all disabilities. We also need to make sure that requirements are written such that when a system is certified through the accessibility section, it must also be a HAVA compliant system. One of the major concerns and issues with vote-by-phone is use by voters with dexterity disabilities. If the VVSG has an end-to-end accessibility requirement, the vendor that submits a vote-by-phone system would also have to submit what the end-to-end solution would be in order for it to be certified as an accessible voting system. The current draft of the HFP section has a draft requirement for the end-to-end accessibility requirement. Vendor usability tests could actually be part of the technical data package the vendor submits. Should these requirements be moved to that volume or kept in the usability and accessibility sections? If we move it to the technical data package it may become lost and it will loose its readability factor. Suggestions were made to either put it in both sections, or put links from the technical data package back to the usability section. David Flater of the CRT subcommittee may have some issues with this. When compiling the final VVSG, they (CRT) are trying to keep similar types of requirements together. Sharon will talk to David. Usability of handling paper: (clarified that this is in reference to poll workers): There was lengthy discussion on what was needed on this issue. The durability of the paper issue is out of HFP's scope. Sharon Laskowski will be getting together with John Kelsey of STS regarding the VVPAT section to make sure that usability is covered. There are lots of requirements for easily identifying the paper rolls and linking them to the machines they came from. It was suggested that bar codes be used also for this, as long as the barcodes only contained information already on the sheet and did not include any ballot information. John Wack thought that STS had an issue with using barcodes. Nelson Hastings will take this issue back to STS. (Whitney feels that there would have to be a serious security issue for removing this option as a possibility.) There was major concern over font sizes for auditing. Question at issue: Is the current font requirements for voters good enough for the poll worker requirements? Concern was expressed over the difference in what was necessary when doing a small audit versus doing a recount of the votes. Sharon summarized saying that 3.0 (font size) for paper rolls was fine if you have small audits; for larger audit, some other means to do the counting, such as barcodes, would be needed. John Wack was not in agreement with HFP. John Gale pointed out that poll workers were hired to do a particular job, and that they were hired based on their ability to do so by state and local officials. In his experience, it is not necessary for requirements for accessibility to be as strong for poll workers as for voters. Sharon and Whitney will take this offline and try to reach a conclusion. Other Business: HFP would like to work with NVLAP representatives on a telcon. Our next telecon is scheduled for: Friday, April 13, 2007 at 11 AM ET
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA 1. Administrative Updates and planning for March TGDC meeting (Allan Eustis) Attendees: Allan Eustis, David Baquis (U.S. Access Board), John Cugini, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Sharon Laskowski started the meeting by going through her slides being presented at the plenary meeting. Basically the presentation is going to be a progress report, discussing changes or issues in the HFP sections, issues requiring further analysis, usability benchmark development, and next research steps. Basically there are three significant changes since the December meeting:
Issues that require further analysis:
Benchmarks: Sharon is still working on these slides. The general goal is to have a quantitative benchmark requirement and to achieve that we had to develop a test protocol method, determine how to count errors, and define what errors looked like. Sharon will go over that methodology and report back the data that was found. She will include a slide of the characteristics a good test instrument must have. Results being reported are supported by others doing testing that have found the same results. Next research steps will be the last slide:
NOTE: Whitney commented that the VVSG as a whole was looking great, good job to everyone involved. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Our next telecon is scheduled for: Friday, March 16, 2007 at 11 AM ET Attendees: Alexis Scott-Morrison, Alice Miller, David Baquis (U.S. Access Board), David Flater, John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Presentations/Materials to be presented at the plenary meeting were discussed. Sharon discussed the material related to the usability benchmarks (she will be working on this over the weekend as well):
Sharon and Whitney have been having an offline discussion regarding system testing and how that might fit into volume testing. If users are being brought in for volume testing, we could use them for benchmarking testing. For accessibility, there is a need to do a whole system test of the process for different disabilities to make sure the system is definitely accessible. The other TGDC subcommittees want to do process testing, such as auditability, accessibility, setup validation, volume testing, etc. Sharon will include this as a "future activities" bullet. Discussion was held regarding usability of the VVPAT papers rolls for audits, etc. It has generated a high level of problems for poll workers and election officials. This is not something HFP has spent a lot of time looking at because it has been concentrating on usability for voters. John Wack feels that changing the way items are printed onto flat sheets (preferable), will not offer a fundamental change to the voting system. The setup by poll workers is critical and needs to be looked at. John would like feedback from election officials. It might be good to look at the requirements written for the product and that would require a possible change to the paper used, e.g., stand up for 22 months with lots of handling. John will be engaging Dan Schutzer's input regarding this matter. Software Independence (SI) and Accessibility: Working with STS, we now have a clear definition about what SI is and what it means by voter verification. SI is making sure that any problems with the election can be detected afterwards. It is not necessary for everyone to verify their ballots for SI to hold. Voter verification has to be available for everyone. The discussion then clarified the difference between direct and indirect verification. A sighted voter can verify by reading the ballot printout, a blind voter would need assistive technology to indirectly verify their printout. The table regarding the four different approaches and their scores were discussed. Concern was expressed over the audio tape recordings and the possibility of tampering with them. There is also an issue with recounting two different types of media that is why there are minuses in the auditability and accessibility/usability columns of the paper. Ron Rivest and Whitney will present a summary of this research paper at the plenary meeting for discussion by the full TGDC. The goal is to decide which, if any, of the approaches outlined, are suitable to meet the resolution that all systems are SI. Hopefully this will help to decide which architectures can be taken off the table, and which are suitable. There may be a possible resolution, but the discussion will allow subcommittee to move their work forward. At the next meeting (in May) we have to adopt final draft of requirements to forward to EAC. ACTION ITEM: Whitney, Tricia, and David will make sure that the external (public) web page URL information is circulated when released and let people know that comments can be sent to voting [at] nist.gov (voting[at]nist[dot]gov). [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Technical Guidelines Development Committee (TGDC) AGENDA
Our next telecon is scheduled for: Friday, March 9, 2007 at 11 AM ET Attendees: Alexis Scott-Morrison, Allan Eustis, John Cugini, John Wack, Mat Masterson (EAC), Nelson Hastings, Sharon Laskowski, Sharon Turner-Buie, Tricia Mason, Wendy Havens Administrative Updates (Allan Eustis):
Quick Updates:
Allan Eustis mentioned that a possibility for the following plenary meeting would be end of May. Allan announced that he had received notice that Senator Obama had introduced a bill to amend HAVA creating a new office for a voter advocate at the EAC. (See: http://electionlawblog.org/archives/di-bill.pdf).
Meeting adjourned at 11:30 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
Joint Technical Guidelines Development Committee (TGDC) Agenda
Attendees: Alice Miller, Allan Eustis, Barbara Guttman, Commissioner Davidson (EAC), David Wagner, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Nelson Hastings, Ron Rivest, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbury Discussion of New Version of Software Independence and Accessibility of the Voter: This topic was discussed in great length. Sharon Laskowski would like to have as output/actions from this call: suggested changes to the white paper that will make it presentable at the March 22, 23 meeting, including John Wack's concerns about usability of audits. Given the pros and cons of the various approaches outlined in the paper, if we want strict SI, voter accessible verification, good voter accessibility and usability, and easy auditability, the problem has been over constrained. Sharon has received Whitney's comments via email, and most of them will be integrated into the document. Sharon wanted to start with Ron Rivest's comment, "At a high level it should be made clear that SI and auditability cannot be sacrificed, they are essential for voting system integrity. Voter verification and accessibility of the verification step may be adapted as best possible, but they are not so critical, technically, for the integrity of the election results." Ron feels this is a multi-dimensional problem and there will have to be some trade-offs. Commissioner Davidson expressed concerns about the realism of getting this done. Also, VVSG 2005 has aduditable function in the guidelines. HFP's main concern is accessibility. There's some compromise with usability that can be made. Solution 1 that is proposed in the paper is accessible in that no one is excluded from voting individually. The verification step is not accessible, but the system integrity is preserved and the voting system as a whole is accessible. The question was asked if this was enough, or did the verification step on its own need to be simultaneously accessible and software independent? From the security point of view, it's enough for the integrity of the election for the system as a whole to be accessible. Discussion continued about audio read back features for verification. Concerns were expressed over preserving this read back on tape for auditability, tapes are fragile and costs would be large. The proposal to have ballots scanned by a different system and read back for verification by voters with disabilities was discussed. This idea seems to be feasible, there have been prototypes. This should meet security needs if observational tests are performed on these systems. The difference between readback from internal memory of a voting system as compared to readback from a print out, re-scanned summary fed into a separate machine for readback was discussed. Tricia Mason pointed out that security is very important, but its also very important that we do not disenfranchise anyone from the voting process. David Wagner expressed agreement in pointing out that accessibility in the sense that we done disenfranchise anyone is critical, that integrity of the overall system is critical, the ability for every voter to directly verify the record is not critical, and given that it is going to be difficult to find systems that meet all our goals, we may need to compromise some and consider observational testing adequate. We may want to encourage enabling as many people to directly verify the independent record as possible, but not make it a requirement. Ron Rivest pointed out that we needed to stick with the original notion of SI - understood by the typical voter. Requirements for verifiability for voters with disabilities can be covered with the readback feature, and we can use observational testing to make sure it is trustworthy. Capability to do voter verification will be written as a "shall", direct verification will be written as a "should". Readback capabilities from scanning ballots will be written as a "should". Auditability: Having a system that is auditable in principle but not in practice is not good. Auditing off paper is going to be difficult without additional aides. How do we write requirements? Having test labs run through sample audits is a good start. Bar codes to track precincts, etc. could be used. Good labeling on ballots would be useful. Problems arise from paper spools when early voting occurs and voters from different precincts vote at one. Paper spool requirements need to be written. John Cugini felt that a task analysis needed to be done which would give the committee a starting point. There is not much currently that can be said in the auditability column of this paper. HFP/STS may want to consult with Dan Schutzer who had some ideas on this point. ACTIONS FROM TODAY'S MEETING:
The white paper needs more review before presenting as a white paper to TGDC March 22, 23. Whitney wants to point out procedures that go along with it. Feasibility concerns or where implementation doesn't exist, needs to be noted. Discussion/review to continue via email. Other Items: Alice Miller mentioned a couple of resolutions that came out of a Standards Board meeting. It was decided that these would be discussed in detail at the TGDC Plenary Meeting. Meeting adjourned at 12:15 pm
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
Teleconferences from 2004, 2005, 2006 and upcoming in 2006. ************* |
|
Joint Technical Guidelines Development Committee (TGDC) Agenda:
Attendees: Alexis Scott-Morrison, Allan Eustis, Barbara Guttman, Bill Burr, David Baquis (U.S. Access Board), David Flater, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Mohammad Mareuf , Nelson Hastings, Philip Pearce, Ron Rivest, Sharon Laskowski, Sharon Turner-Buie, Wendy Havens, Whitney Quesenbery, Administrative Items:
Ron Rivest reported that a high level issue here with software independence- we have to be able to do audits - audits are how we detect errors. The auditing process itself is part of the whole package and must be usable by poll workers in a reasonable way and this must be tested. Ron and STS are looking to HFP for guidance on how best to do this from a usability stand point. This topic about usability of SI and usability of audits was discussed in great detail.
ACTION: Write high level requirement that states systems should be auditable and vendor must document procedures. We should look at specifics, such as re-reading ballots. Vendors must provide process so test labs can perform necessary tests. We must document what makes an audit technically possible and usable. Write definition of durability for surviving 22 months as required for voting records.
John Kelsey was happy that the procedural defense was added to the paper and feels that this provided the SI requirement - this is the one where users without disabilities are asked to test the accessible systems for voters with disabilities. Whitney feels that it will not pass through the Holt Bill. Whitney does not believe we have an existing accessible system that will receive high marks in the 3 columns of the evaluation process noted in the paper. John Wack mentioned that David Wagner felt that with IDV systems, there were no accessible systems that were SI. Ron was asked to clarify the definition of SI - errors are detectable in principle, either during voting stage by the voter, or later from evidence results of the audit. It does not require the voter paying attention to what's going on, i.e. it doesn't require that the audit be used but the auditing equipment should be trustworthy. John Wack felt that we need verification for audits and forensics for recounts. For example, we would preserve a recorded record of the audio verification provided to blind voters. David Baquis indicated the Access Board was in favor of having all voters be able to verify their votes at the time they vote. David felt that the audio recording of the verification could be played back for the voter as a confirmation that what they just heard was what was recorded for later auditing if necessary. ACTION: Simplify table in paper and add an extra column. Review section 3.2 to implement/consider David Baquis' /Access Board comments. Ron Rivest will provide comments via email. Barbara pointed out that comments should include the section, whether its yes or no, and how to make it yes if feasible. Next joint HFP and STS meeting, Thursday, March 1, 2007.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.] Teleconferences from 2004, 2005, 2006 and upcoming in 2006. ************* |
|
Joint Technical Guidelines Development Committee (TGDC) Participants: Secretary Gale, Allan Eustis, Wendy Havens, Benjamin Long, Barbara Guttman, Nelson Hastings, John Kelsey, John Cugini, David Flater, David Baquis (U.S. Access Board), Philip Pearce, Whitney Quesenbery, John Wack, Tricia Mason, Ron Rivest, David Wagner, Sharon Turner Buie Agenda
[Next HFP teleconference is scheduled for: Friday, February 23, 2007 at 11 AM ET]
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecom were preliminary and do not necessarily reflect the views of NIST or the TGDC.]
Software Independence and Accessibility for the voter: Whitney mentioned that starting point for this discussion has been that all aspects of the voting process need to be accessible or accommodated in an appropriate way. "Complimentary accessibility" seems a better term than "end to end" and avoids confusion with STS cryptographic use of the term "end to end". Ron agreed that verification for all voters sits at the intersection of security and usability. STS is looking for guidance and collaboration from HFP on the right path forward. Whitney pointed out we need to determine exactly what we are verifying.
We are focusing on (2). John Cugini offered 4 possible technical approaches/solutions for discussion. He stressed that SI solutions are oriented towards blind voters. These are not mutually exclusive:
(a), (c), and (d) are paperless solutions. All have pluses and minuses. Some are not ready for prime time. Discussion initiated by Ron Rivest on what of the four truly meet software independence. Tampering could occur with (a). Demanding SI for all voters may not feasible.(i.e. (a) may still be acceptable; (b) meets independent verification (IV) requirement. It does require an independent reader device; (c) also in (IV) class. Dave Wagner reviewed his understanding of four technical approaches/solutions. First three appeared SI to him. (d) did not appear as SI. Whitney also reviewed her understanding of the four technical solutions from accessibility stand point and production of durable records. Philip Pearce asked if any of the approaches violated private and independent verification requirement for all voters. John Kelsey said they all are private but the main issue is security of each approach. Whitney agreed and reviewed independence of each method; Issues of sampling size with some approaches. Philip Pearce posed question of how much responsibility we have to decide the "best" method. What are sufficient techniques? David Baquis added four different challenges that need to be considered in this matrix:
Whitney referenced a recent presentation at on cognitive disabilities presented at recent access board meeting. (See: http://www.access-board.gov/sec508/refresh/teitac3rd/clayton.ppt). Point here is that cognitive disabilities are on a spectrum with respect to barriers. We should not gloss over usability issues. Whitney and participants reviewed the approaches (a)-(c) for each of the disabilities brought up by David Baquis. A magnifier could be used for those with low vision who do not use audio support routinely. John Kelsey synthesized issue as a determination what appears on the screen and does not make it on to the papers that are relevant to the vote the voter he/she was casting. Also there are issues here related to alternative languages. Issue here is "one ballot versus many ballots "of alternative language ballots on a specific machine affecting privacy of vote. There are election procedures here to mitigate the loss of voter privacy. David Wagner had concerns with co-mingling of ballots. Sharon Turner Buie pointed out that paper ballots are co mingled as they are read into the machine. Concern here by Wagner of one Chinese voter at that polling place. Turner Buie pointed out that ballot is tabulated irrespective of the language used to vote. (John Gale agreed.) Only in a recount would you possibly be able to detect the voter's identity. (Wagner expressed ongoing concerns related to privacy issues.) Quesenbery brought up dexterity issue and transporting paper as a barrier to independence requirement for the voter. Scanner device was discussed. Does assistance with the paper ballot remove independence for the voter? We will get push back here. Secretary Gale noted that state and local election officials operate a voting system, not just voting equipment. Trained election officials and poll workers operate the equipment within a system context. Election officials do their independent testing and assessment of the equipment to ensure that it is working accurately and effectively before an election. You do not want to discuss these issues in a vacuum separate from an election system. Our focus here seems to be to drill down to require the individual voter to do the assessment and evaluation. Meanwhile studies indicate that 30% or less of the voters verify their vote. If the average voter has trust in the certification and verification of the voting system, they are going to accept probability that the system is working correctly. That is they do not need to be responsible for reading the installation manuals etc. An analogy here: when we fly a commercial airplane we trust the system. We do not feel the need to read the service manuals or review the qualifications of the pilot because others do that. Are we making voting complicated for the average voter because we do not trust the election officials or the voting system to operate correctly? Should we not focus on assuring that the voter has ease of access and has a sense of confidence in the casting of their ballot? If we don't trust the software, then we should have independent audit verification of random precincts that use DRE equipment with VVPATs. Those selected precincts are going to have to count the VVPATs and compare them with the electronic result. That way, election officials are addressing the verification issues for all the voters in the precinct without passing the responsibility on to the individual voter. Rivest agreed that the voter verification step is important from a security viewpoint if it is done by a sample of the voters. The idea of a random audit of the precincts is a good one especially with VVPATs. Philip Pearce commented that, in our discussion, we should be asking the question for whom the voting equipment is working accurately and appropriately. Is it allowing voters, whenever they wish to do so, to ensure that the equipment allows for private and independent verification of their votes? Secretary Gale agreed with this assessment, and commented that the voter had to have trust in the system's checks and balances. We seem to be in new arena of voter responsibility independent from the system. We should also realize that no equipment is going to be perfect just as no election is perfect. You have to rely on the system as a safety net back up. Wagner noted that we are talking about voting systems producing non electronic records that will be used for permanent records and allowing voters to verify their votes. We want to eliminate barriers to the disabled voters' ability to verify his/her vote. However we are not giving the voter the right to follow the vote through every step in the process. We simply want to preserve the security. Summarizing, Quesenbery commented that elections are indeed complex. We are not looking for full accessibility of the process; we are looking for equivalence. If the paper ballot is the ballot of record, we are looking for means to ensure that all voters can review that ballot. We do not require that all systems operate in the same way. We look at how voting system architectures satisfy the requirements for all voters. This makes it difficult to write guidelines to ensure that voters with disabilities can fully participate. The next step requires an answer to the question of what makes the ballot auditable. For example there are definite challenges in auditing an audio tape. Discussion ensued on (a) thru (d) above relative to whether they satisfied as a sufficient SI technique. Each was reviewed. Rivest provided context for SI in terms of errors that are capable of causing changes in the election outcome without any detection. (Will an audit detect something wrong with some probability?) In (a) a sighted a voter could listen to the audio and determine if it is different than what is printed on the paper. This is an SI capability. Qeusenbery noted that we have to answer two questions. Is the approach sufficient to meet SI and is it sufficient to meet accessibility. (a) Meets accessibility. John Kelsey suggested a formal procedure to encourage (enabled) voters to vote on accessible audio ballot voting systems. Quesenbery and Cugini determined that VVSG 2007 has requirements that audio and video interfaces work simultaneously. Rivest noted Selker's research indicating that sighted voters who used audio voted more accurately. Cugini cautioned that we are writing an equipment standard and can only recommend procedures. Concluded that (a) might be SI, but is certainly accessible. (b) Produces a paper ballot. There is a way to read it for blind and low vision (magnify). There are transportation challenges for (a) and (b). The final (c) is accessible and auditable. But the ease of auditability is a factor. There are issues with the first three. The fourth is still an unknown. We need a chart with all the issues laid out. We have maybe's for SI at this point. Point is to scope out all cases before presentation and acceptance/rejection by full TGDC. HFP and STS Sub groups will need to agree first; need to get away from the theoretical to the practical. Rivest offered the opinion that (a), (b) and (c) have workable SI solutions. Gale noted complexity with (b) for transportation issues. Cugini asks that we not downplay (d). There are prototypes of this architecture that help solve (b) issues. Action item: NIST will document in a list/table format the four technical solutions with auditability and usability concerns clarified. The table will be sent out to STS and HFP for review before next joint meeting. Software Independence and implications to usability of audits: This will be covered at the next HFP meeting. We will also plan a joint meeting to review auditability issues as well as resolving issues from this telcon.
************* |
|
HFP Subcommittee Teleconference* Agenda:
Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, Ben Long, David Flater, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery, Ben Long Administrative Updates:
Discussion of Proposed Edits to the VVSG (John Cugini): John Cugini had passed out a list of HFP issues raised at the December 2006 TGDC meeting. The subcommittee reviewed each issue: 3.1.3 - HAVA and VVSG. After drafting this general overview section, there is nothing specific to HFP. It should be moved to the general introduction by CRT, but HFP should put comment in about where it is. Change "voting equipment" to "voting system" (BE CONSISTENT), as voting system includes documentation and we are discussing what you get complete from the vendor. 3.2.2 - Clarify that when a voter receives a warning about under or overvoting that they are given the chance to fix the problem after the warning. 3.2.3-C - Should the specific guidelines point out that the plain language requirement is only for English and may not apply to other languages? Subcommittee feels this should be left alone. 3.2.4-C - Should all voter settings be adjustable throughout the voting session? Subcommittee feels that there should be "on demand" choices, but that the "switching mode" option should be a start over option. John Wack to ask EAC if we can pass this suggestion around and bring up at ITAA vendor meeting to get feedback. 3.2.4-E - Requirements should mandate at least two font size choices on all editable systems (not just accessible equipment). 3.2.4-H - Visual access to VVPAT. The goal would be if you make a selection on one piece of equipment, that it follows through to the next such as the printer. This will be discussed at joint meeting with STS regarding VVPAT. However, the requirement about physical posture of reaching both voting system and printer should remain. 3.2.4-J - Requirements should mandate at least two contrast settings on all editable systems (not just accessible equipment). 3.2.5.1 - Timing issues. Should we make a requirement about keyboard response time? No, not unless it proves to be a problem during testing. However, change the requirement to read "voter inactivity time" instead of "system response time". 3.2.5.1-D - Change "system activity indicator" to "system response indicator". 3.2.4.1-E - Voter inactivity time. Should we specify a time? If we did it would have to be on the high end. Leave as is for now. 3.2.7.1-A.4 - No receipts. We need to add a requirement stating that a receipt shall not be able to provide proof to another about how someone voted. 3.2.8 - Usability for poll workers. Should we mandate a state option whereby a poll worker could specify a state for which a system should be configured? NO. 3.2.8.1-A - Change "average" poll worker to "typical" poll worker. 3.2.8.2-B - Maintenance documentation requirement should be taken out and covered in documentation section of VVSG. 3.2.8.3-A - Should we change the regulations to reference and IEC standard? No, leave as is. We should state that systems must meet Federal Standards that are available. 3.2.8.3-B - Is the "elimination of hazards" section redundant? Leave in for now until further research is conducted. 3.3.5-B - Allowance for an assistant. Clarify spacing issues stating "adequate physical access". Next meeting will be a joint HFP/STS subcommittee meeting on February 9, 2007. [*Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. These teleconferences serve the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this teleconference are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |
|
HFP Subcommittee Teleconference* Agenda:
Participants: Alexis Scott-Morrison, Allan Eustis, David Baquis, David Flater, John Cugini, John Wack, Nelson Hastings, Sharon Laskowski, Tricia Mason, Wendy Havens, Whitney Quesenbery Administrative Updates (Allan Eustis):
Discussion of Current and Planned Activities for the Next Generation VVSG Whitney stated that the subcommittee should set a goal to have all HFP related edits completed before the March TDGC plenary meeting for the VVSG2007, with an exception to security related usability material. After edits are complete, the next step will be to work on security and verification - this is a hot topic including potential upcoming Congressional activity. Third, HFP needs to deal with usability benchmarks, making sure we have the resources to say in July that we know where we're going with particular tasks. Fourth is ongoing research. We need to decide what are the immediate needs for the Next Generation VVSG and what are test methods and suites that can be worked out after July delivery (we have started discussing these in greater detail). Requirements will be in this version, including general testing requirements. Details of the test protocols can go in the test methods written after July. HFP has been having continuous discussions regarding examining audit methods and usability in terms of some of the security methods that are going to apply to election officials and poll workers, in relation to software independence. These two can be bundled together Still struggling with specific method and implementation design e.g. barcodes for easy recounts. Questions arise when you get to explicit implementation design - how do you write requirements that are high level but still useful. The subcommittee reviewed the list of top four items on their list. John Cugini has almost completed the (usability) updates to the VVSG from the comments received at the December meeting. He will complete them and have them ready for discussion at the next HFP meeting. Any issues will be discussed via email beforehand. Usability Benchmarks - It has become exceedingly obvious that you can not just pick a number based on a data set. You have to know how to analyze and know if you're using the right statistical methods. HFP has received expert advice on this matter and is close to saying "this is how we're going to count" and "how we plan to analyze". We may need 100 subjects (ROM) to test. Tests are being conducted on error rates noticed when people are instructed how to vote, we are not currently looking at voter satisfaction or time periods. Specific details on what test methods were discussed. There should be guidelines for expert review. Questions arose about what does broad guidance mean for technical documentation? Are accessibility benchmarks written as test methods? These will not be included in the July deliverable. Can addendums be added to this version of the VVSG? This will be brought up to the EAC for answer. John Wack will find out what the process for making additions and corrections will be. TGDC may also want to approach this topic with EAC. Allan pointed out that the document, once reviewed by EAC, would be sent out for public comment. Final will probably not be released until 2008. David Baquis wanted to know what EAC's process was for making changes. (Same public review process as for previous versions of VVSG.) David B. felt that HFP should seek out involvement from disability groups and accessibility consultants in its discussion regarding possible test methods. (This would have to be done through the EAC). (There will be a cognitive expert at an upcoming TEITAK Telecommunications and Electronic and Information Technology Advisory Committee disability - meeting that might be a useful resource.) The next two teleconference meetings will cover end-to-end testing and usability benchmarks, respectively. David B. suggested re-titling end-to-end to read "complimentary accessibility". Whitney's two major points being emphasized are how accessibility and accommodation can work hand-in-hand and second; that it's not enough for each piece of a component in a system to be accessible, but you have to look at it as a whole system - from the voters point of view and all the things a voter has to do. John Cugini questioned if we get performance (test based) requirements for accessibility written, would this be the basis for this kind of test? How do you make sure the systems overcome the challenges? Part of the documentation a vendor should submit to the test labs is an explanation of their system and an explanation of how it is intended to be used so that it can be tested in its intended method of operation - documentation should be lay out steps that are assumed people would go through. If there are different possibilities, those should be laid out as well. This would help people that are constructing usability tests. John Cugini inquired if this would require adding new requirements. Yes. At the next meeting, HFP should decide what specific issues involve STS coordination. Also what does it mean to be an "accessible" secure system? Next meeting, Friday, January 26, 2007, 11:00 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. These teleconferences serve the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this teleconference are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
**************** |