TGDC Subcommittee Work - Historical Meetings - CRT Meetings - 2007

Joint CRT and STS Teleconference
Thursday, June 28, 2007
11:00 a.m. EDT
Draft Meeting Minutes

Agenda:

1) Administrative updates and logistics for the July 3 TGDC plenary telecon. (Eustis, Wack)
2) Draft presentations for the July 3 TGDC plenary telecon. Please see attached Flater draft.
3) Any other items.

Attendees: Alan Goldfine, Allan Eustis, David Flater, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Wendy Havens

Administrative Updates (Allan Eustis):

• July 3rd Plenary Meeting: The meeting is scheduled to start 11:30 a.m. EDT, it will be webcast, with NIST staff in the NIST Employee Lounge and TGDC members (approx. 12-13) participating via teleconference (Phone # and Pass code the same as regularly scheduled subcommittee meetings). We will be using an on line hand raising tool, instructions on how to use have been emailed. Slides will be sent to members by Friday, June 29th. There will be hard wire network connections in the employees lounge for NIST use.
• The agenda has been revised per request from Secretary Gale, STS first, then CRT, and concluding with HFP.
• The meeting should conclude with 3 separate votes formally adopting the drafts with the editing token given to Bill Jeffrey for final changes.

CRT's Plenary Presentation (David Flater):

CRT only plans one presentation for the meeting, being given by David Flater.

• The presentation will be short (barring any issues raised), only five slides.
• The idea of the presentation is to review the CRT Issue Log which is a compilation of the 280 issues received, primarily from the TGDC members after the last plenary. The issue log documents to the responses to each issue.
• Slide 1 is just a cover slide.
• Slide 2 will discuss the issue log and the statistics.
• Slide 3 will highlight resolved issues in the log.
  • Missing shake & bake tests: no longer missing
  • Benchmarking test method: rewrote explanation
  • Frequently misconstrued terms: replaced with more explanatory wording
  • Software engineering practices: fixed nits
  • Volume test: more parameters for op-scan
  • Many requirements clarified through wording changes or defined terms
• Slide 4 will discuss issues that are pending, these are issues that have been resolved but did not make it into the VVSG draft that was distributed.
• Slide 5 will discuss open issues. The first being Ron Rivest's question on field immunity clarification. [per Alan Goldfine this issue may be closed, it will be discussed offline] Also a coordination issue with STS on harmonization with reporting and the definition of token.

This is the last scheduled meeting before the plenary. The need for future teleconferences will be decided after the July 3rd plenary.

Meeting adjourned at 11:20 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

Joint CRT and STS Teleconference
Thursday, June 14, 2007
11:00 a.m. EDT
Draft Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Issues received from TGDC members and the public (continued). See the files under "CRT VVSG finalization working materials:" at http://vote.nist.gov/TGDC/crt/index.html (Flater)
3) Any other items

Attendees: Alan Goldfine, Alicia Clay, Allan Eustis, Angela Orbaugh, Bill Burr, David Flater, David Wagner, John Wack, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Paul Miller, Philip Pearce, Ron Rivest, Thelma Allen, Wendy Havens, Rene Peralta

No Administrative Items.

CRT Issues (David Flater)

The issues log being maintained by CRT contains over 273 issues, with 27 remaining open as of today. Many of the open items are regarding terminology waiting on feedback from Paul Miller. In order to make the deadline for final draft, the open issues need to be closed by Friday, June 15.

• Issue # 6: Benchmarking Test Methods Look Wrong (Ron Rivest): There has been extensive email regarding this issue and will continue to try and resolve this via email. This has been re-written extensively in the language of classical hypothesis testing. This has hopefully been clarified per Ron's issues.
• Issue # 21: Threads and Memory Leaks (Ron Rivest): Should the following be mentioned somewhere: "memory leaks" (memory allocations that are never "freed") and "threads" (concurrent programming techniques). Response: Memory leaks are addressed in III.16.4.1.8, though not to the extent of proving their absence. Threads are not addressed. Suggestion is to ban multi-threaded voting application logic, but this could be controversial. It's common to have separate threads in gooey applications. There are ways to deal with multithreads in a structured way. The requirement should state that if a voting application's logic is multi threaded, the vendor be required to provide demonstration that it will not effect other programs or become deadlocked. David Flater to add language.
• Issue # 20: Ballots for opscan volume testing (David Wagner): Is it OK to substitute mechanically marked ballots for manually marked ballots during test. The proposed requirement states that it is not valid to substitute with mechanically marked ballots if the vendor specifies that the system is designed to count manually marked ballots. The 2nd issue was in order to reach the number of ballots needed to run the volume test, is it ok to recycle ballots during the test. The proposed requirement states that there must be at least 10,000 manually marked ballots to run through the test and after that you can recycle that 10,000 to get the number needed for volume testing. David W likes resolution proposed.
• Issue # 1: Terminology: Partisan contest to become party specific contest; non-partisan contest to become non-party specific contest; after discussion David F. will work to clarify the definition for primary election; the term general election will be deleted.
• Issue # 2: Terminology: Jurisdiction. This means different things in different states. David F. will find a new word to replace the word jurisdiction, keeping the definition currently used.
• Issue # 3: Terminology: Scratch vote vs. crossover vote. It was decided to replace with the term "straight party override".
• Issue # 15: Terminology: User serviceable: This term means for errors that are correctable by trouble shooter or election officials. The requirement will be changed to clarify that it is only consider user serviceable if the directions for fixing the error are provided in vendor documentation.
• Issue # 19: Terminology: Ballot activation: This term will be defined to mean what happens on the DRE itself. The epoll book issues ballot activation credentials and John Wack has been using credential issuance in his section.
• Issue # 23: Terminology: Ballot choice term will be changed to contest choice.
• Issue # 4: Terminology: Should there be a different definition for ballot? What should it be, there are so many options. Ron suggested that maybe there should be a listing of all the definitions of ballots with footnotes in the document to the one matching the text. David F to come up with a list of possible definitions and forward to Paul Miller for feedback from election community (with the understanding that comments may not be able to get into the July draft).
• Issue: Terminology: Is the term "during an election" ambiguous. This is used in the access control section. Nelson Hastings and David F will work on this issue offline to put in a requirement and will take the term issue off the table.
• Issue # 14: Indicator for Whom (David Wagner): There is a requirement that says that there will be a visual/audio indicator on the voting system about the status of the voting device. It was decided to change the requirement to narrow it to indicator for election judge.
• Issue # 27: Public Information Package (PIP): Proposed change to the requirement by David Wagner is to write the standard to say the test report must contain something labeled as the PIP and that the PIP must contain the test plan and test results to the extent possible. No objections from members present.

Next CRT meeting scheduled for Thursday, June 21. (Possibility it will be cancelled).

STS Issues:

Issue: Cyrptogrpahy: The issue of what records have to be signed on the voting system was discussed. There are multiple possibilities of what a voting system does on Election Day and it was questioned whether different keys were needed for different activities on the voting systems. It was decided that the requirement would be written that the machine only required one key, that everything must be signed, and there be able to be a count of what was signed. [Bill Burr will write requirements based on today's discussion.]

Issue: Security through obscurity: David Wagner thinks there should be a requirement that makes it clear that the system must not rely upon security through obscurity for security. His proposed text: The voting system must not rely upon the secrecy of its design, implementation, or documentation for the security or integrity of the election. The question is where does it go and how do you test. It should be considered a guiding principle the same way privacy is. It was decided that the requirement should go in the documentation standards. [Angela Orbaugh will update this section.]

Issue: Requires software independence: David Wagner was not able to find any place in the standard which requires the voting system to be software independent, as part of the normative text (rather than the discussion/introductory material). He feels a requirement needs to be added. Subcommittee was in agreement to add statement because the VVSG that is delivered to the EAC must say all voting systems shall be SI, period. John Wack suggestion was to add this requirement in the auditing chapter (being retitled Security Architecture) as a high level requirement with subrequirements on how to achieve SI, currently with VVPR. A glossary term for SI also needs to be included - using Ron's original definition.

Issue: VVPR should be mandatory: David Wagner feels there should be a requirement specifying that all voting systems must produce a voter-verified paper record. This is responsive to the TGDC resolution requiring software independence, combined with the decision that systems producing a voter-verified paper record are the only kind that will be currently supplied in the standard, and the innovation class will provide a way to path for certifying other kinds of voting systems. After discussion it was agreed that this requirement would NOT be included. VVPR is included as a "may" requirement for meeting software independence so that it leaves room for other means of meeting SI for innovation class designs. It must be made clear that any system in the innovation class must meet all requirements in the VVSG and that SI must be met, but it can be met in a new way. The proposal was to add a discussion to the requirement to say the intent of the requirement is that all non innovation class submissions must comply with VVPR but innovation class submissions can meet the requirement of SI by other means. David Flater suggested adding the caveat that any other way they system claims to be SI will be put under intent scrutiny to make sure it meets SI.

Next STS Meeting Tuesday, June 19th.

Meeting adjourned at 12:30 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

***********

CRT Teleconference
Thursday, June 7, 2007
11:00 a.m. EDT
Draft Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Issues received from TGDC members and the public (continued). See the files under "CRT VVSG finalization working materials:" at http://vote.nist.gov/TGDC/crt/index.html (Flater)
3) Any other items.

Attendees: Alan Goldfine, Allan Eustis, David Flater, John Wack, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Thelma Allen, Wendy Havens

Administrative Updates:

• TGDC Plenary Teleconference scheduled for July 3. We plan to distribute VVSG draft electronically, if anyone needs a hard copy email voting [at] nist.gov (voting[at]nist[dot]gov).

Issues Received (David Flater):

At the last meeting, it was decided that issues with terminology would be discussed with Paul Miller, CRT is awaiting feedback.

Issues discussed today:

• The "general election" term which had created a lot of controversy was deleted as the term was not used in VVSG requirements.
• The definitions partisan contest and non-partisan contest were changed to part specific contest and non-party specific contest, respectively.
• The definition for primary election was changed to match the definition in VVSG 05. The definition can be altered to make it more acceptable without damage to the requirements.
• The term jurisdiction had been used with multiple meanings by CRT and STS. The definition (awaited NASED approval) is: Administrative unit that is the entire scope within which the voting system is used; e.g., a county. Note: The jurisdiction corresponds to the top-level reporting context for which the system generates reports. STS to replace uses of the term jurisdiction with words to the effect of "system owner."
• Scratch vote vs. crossover vote: We are trying to avoid terms that already have pre-determined meanings. However, if the way we use crossover vote matches election officials, we will use that term instead.
• Ballot activation definition: CRT and STS have used the term in different ways. CRT is awaiting feedback on what meanings are in practice in the election community.
• The terms "vote" and "ballot" are sensitive. CRT is happy with the definition for vote. The definition for ballot is a lot trickier, it was left out of previous VVSG. The meaning of ballot is usually obvious from context, but we are not aware of any attempt to construct a single, universal definition of ballot that succeeded. CRT would like to discuss with TGDC.

The issue log is currently about 165 pages, with 1 page per issue. David Flater has incorporated many comments and given tentative resolutions for many. To summarize the issues, most are editorial, lots of glossary terms, David Wagner had a lot on coding standards which were not serious modifications but clarified a lot of ambiguities. A lot of comments were received from Ron Rivest and David Wagner, we are awaiting comments from other TGDC members.

Next meeting is scheduled for Thursday, June 14, 2007.

Meeting adjourned at 11:30 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

CRT Teleconference
Thursday, May 31, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Issues received from TGDC members and the public. (Note: These issues have not yet been processed; we expect to have some issue tracking and some issue discussion started before the plenary telecon.)
3) Any other items.

Attendees:  Alan Goldfine, Allan Eustis, Brit Williams, David Flater, John Wack, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Sharon Laskowski

Administrative Updates:

• July 3, 2007, will be the TGDC plenary teleconference meeting, as we will be able to have a quorum with 11 definite attendees.  The meeting should begin around 11:30 a.m.
• All the materials for the May plenary are available on the web, including the official transcripts <http://vote.nist.gov/meeting20070521.htm>.
• Everyone is reminded to look at the public comments received to date <http://vote.nist.gov/ECPosStat.htm>, there is a lot of good feedback.

Specific Issues on CRT Chapters Received from TGDC Members and Public (David Flater):

Open Issues:

• Define Jurisdiction:  There appeared to be a conflict between the term as used by CRT and STS.  Upon discussion it appears that both subcommittees were in agreement of the intended use (to mean top level reporting, whoever maintains the server, usually the county or city, sometimes the state).  Clearer words will be used and sent to Paul Miller for feedback/agreement.
• Integratability, Electronic Records, Open Export:  This was left hanging at the TGDC plenary meeting.  No proposed action today.
• Partisan Contest Defined Wrong:  Upon discussion it appears that the definition used for this term is correct; however, the term itself is not.  This is one of those terms that already has outside of VVSG meaning so it is best to come up with different words for the term.
• [NOTE:  Since there appears to be confusion/frustration over glossary terms, it was decided that maybe they should be discussed with election officials on the TGDC.  Reminder from David Flater, the purpose of the glossary is when there is a requirement in the VVSG that uses a specific term, that term can be looked up in the glossary to determine the meaning of the requirement.  This is not meant to be a dictionary of terms.  In cases when the term may be used in the voting field with a different meaning, a new term needs to be selected, not new definitions.]
• The terms audit and witness build were sent to STS for defining.  Nelson is working on and will discuss with David.
• Definition of Volume Test:  This is a term of art.  David will come up with definition.

Closed Issues:

• Terms such as test, test case, and test suite are used inconsistently.  Document will be reviewed and fixed.
• The term certification is out of VVSG scope.  The VVSG defines the requirements for the voting systems, it does not speak on the certification process which is the prerogative of the EAC.
• Acc-VS should be a sub-class of two other classes defined per HFP requests, voter editable ballot device (visual) and voter editable ballot device (audio).  Something that claims to be Acc-VS supports both.
• Assumptions That Instructions Were Followed:  There will be a statement added under general requirements that says the requirements of the product standard in terms of what is supposed to be delivered in functionality and performance is based on the assumption that the system is deployed, calibrated, and tested as per the instructions of the vendor.
• Mis-feed Rate Applied to EBMs: It was agreed that the mis-feed rate should be tweaked so that it is clear that feeding paper ballots into EBMs is also covered in the limitations of the numbers of mis-feeds that are tolerable in a conforming system.
• The Terms Scratch Vote and Cross-Over Vote:  Will be discussed with Paul Miller.
• Primary Election, General Election, Non-Partisan Contest:  All tied up with the term Partisan Contest
• Definitions for VVPAT and VVPR:  Global change from “verified” to “verifiable”
• Clarification of Draft:  There are several places where it says “click to add source or impact”, these will be deleted if not necessary.
• Confusion About Write-in Votes, Choices of Party, and Open Primary:  Explained in the discussion fields.
• There is a 0 tolerance for disenfranchisement failures and a non-0 tolerance for counting errors.  John Cugini from HFP had questioned this.  David says that it is correct because disenfranchisement failures are worse.
• “Shake and Bake” Test Dropped Out of Draft:  It had been decided that maybe this should be left out and to include a discussion making reference to the test procedures but to not include the contents of those procedures.  Alan Goldfine informed the group that this is still in a state of flux and he is currently working on new wording that will include some general requirements.
• Straight Party Voting, Absentee Voting, and Absentee Ballot:  Confusion over these terms because they are used in the voting world with different meanings than those used and intended in the VVSG.  David is working on a proposed resolution and will forward to the group for comments.

Next CRT meeting scheduled for Thursday, June 8, 2007.

Meeting adjourned at 11:50 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, May 17, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Draft Powerpoint presentations for May 21-22 TGDC plenary (to be distributed) (Goldfine, Flater, Wack)
3) Overview of latest draft of new VVSG class structure and changes, mechanics of requirement structure and workings, overview of material (Wack)
4) Upcoming TGDC plenary meeting items logistics, etc. (Eustis)
5) Schedule of future CRT phone meetings (Goldfine)
6) Any other items

Attendees: Alan Goldfine, Allan Eustis, Brit Williams, David Flater, Jon Crickenberger (NVLAP), Michael Koo, Paul Miller, Wendy Havens,

Administrative Updates (Allan Eustis):

• Dr. Jeffrey testifying today (5/17) at the EAC hearing. His testimony will be posted on the vote.nist.gov website later today. Mark Skall and John Wack attending hearing.
• There will be a meeting today to go over plenary logistics (final agenda and next steps) with Commissioner Davidson and the TGDC subcommittee chairs and co-chairs.
• TGDC members not attending the plenary in person will get their material via fedex.
• The powerpoint presentation for the plenary will be on the web and emailed out on Friday.

CRT's Plenary Presentation (Alan Goldfine and David Flater):

David and Alan gave a quick overview of the material to be presented at the plenary meeting (presentation emailed with agenda):

• Nothing new in the presentation that hasn't been covered at CRT telecons.
• Purpose of presentation is to bring rest of TGDC up to date on CRT activities.
• Most of slides in David's portion are about benchmarks.
• More time is spent on explaining meaning of terms.
• Discussing expectations of benchmarks, in respect to other kinds of testing.
• Presentation covers general changes to CRT's sections of VVSG since last meeting.
• Contains a list of changes made since VVSG 05.
• Slides cover our goals and what our final results are.
• Discuss requirements in regards to quality assurance/configuration management. This is in response to resolution 30-05 and direction of the TGDC at the December 2006 plenary meeting. It is a replacement/rewrite of material from VVSG 05. Changes were made since last plenary regarding the timing of the quality assurance manual per TGDC's instructions. Discussion text was clarified.
• Presentation will cover electromagnetic requirements in three categories. These are updated from VVSG 05 with latest standards. These changes are reflected in latest draft of VVSG.

Plenary Meeting (Allan Eustis):

Will have a better idea how day # 2 will run after today's meeting with subcommittee chairs. Bill Jeffrey would like to approve sections of the VVSG by subcommittee after their presentations. Significant portions of CRT material ready for approval.

Schedule of Upcoming CRT Telecons (Alan Goldfine):

We will continue to schedule CRT telecons for Thursday at 11:00, starting again on May 31 through the end of June. Email will be sent out to subcommittee regarding dates.
Meeting adjourned at 11:20 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, May 10, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) EMC Requirements (Goldfine)
3) Reliability and Accuracy Benchmarks (Flater) (tentative)
4) Overview of latest draft of new VVSG class structure and changes, mechanics of requirement structure and workings, overview of material (Wack)
5) Upcoming TGDC meeting items logistics, presentations overview (Eustis, Wack)
6) Any other items

Attendees: Alan Goldfine, Allan Eustis, Brit Williams, Dan Schutzer, David Flater, Michael Koo, Paul Miller

Administrative Updates (Allan Eustis):

• May Plenary Meeting: The draft agenda is out. We have a pre-briefing scheduled with Dr. Jeffrey on Wednesday, and EAC pre-briefing on Thursday of next week. We are trying to schedule a teleconference with the TGDC subcommittee chairs and co-chairs before the meeting - if you have any issues please let your chair and co-chair know.

EMC Requirements (Alan Goldfine):

The paper as discussed at the last CRT teleconference has been submitted to the editors for formatting and incorporation into VVSG draft. There was some tweaking to the informative language but nothing significant.

Reliability and Accuracy Benchmarks (David Flater):

Two small points: 1) We clarified the issue of how the device class control tabulator only listed benchmarks for non-user serviceable failures. The amendment made was under failure type, instead of non-user serviceable, it was changed to "all" with a footnote saying that apart from misfeeds (and jams) which are handled by a separate benchmark, TGDC experience is that central tabulator failures are never user serviceable. 2) The second issue was in regards to the misfeed rate. The benchmark was set on a 99% curve, meaning the benchmark was set so that a conforming device should leave no more than 1% chance of blowing the benchmark in the example election for the volume positive. In case of misfeeds, the benchmark was set to the rate quoted, 1 in 500.

Draft VVSG (Allan Eustis for John Wack):

The VVSG draft will go up on the web Friday and will be substantially the version that is printed for discussion at the May 21-22 plenary meeting. It is a near final draft, however some security sections are still preliminary. The plenary will focus mainly on issues of major concerns. On day 2, there will be discussion of the next steps and future TGDC meetings. We will also discuss the VVSG final review process. We are completing this document under the laws established by HAVA.

TGDC Logistics (Allan Eustis):

The web page is up as well with logistic information and draft agenda. TGDC members will receive a CD will all the information on it. Final VVSG will go on the web on Friday and also burned to CD sent to TGDC.

Other:

• Alan Goldfine: Next CRT meeting will be next Thursday, May 17. We will discuss plenary meeting presentations which will be sent out next week. We will also schedule future subcommittee meetings - still useful to have weekly meetings.
• Allan Eustis: Steve Berger has resigned from the TGDC. IEEE is in the process of providing a new appointment. Seat currently vacant.
• Allan Eustis: All subcommittees will probably continue with weekly meetings after plenary.
• David Flater: There was significant discussion about the interoperability section of the VVSG at the STS meeting this week. Changes resulted to the section to make the language more precise when discussing interoperability and integratability. Please review if you are interested in changes.
• Brit Williams: IEEE is reconstituting its standards coordinating committee and the way that they provide input to the TGDC. NIST may be interested in joining committee.

Meeting adjourned at 11:30 a.m.
 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, April 26, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) EMC Requirements (Goldfine)
3) QA/CM Requirements (Goldfine)
4) E-pollbook Issues (Wack)
5) Any other items.

Attendees: Alan Goldfine, Allan Eustis, Dan Schutzer, John Wack, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Michael Kass, Michael Koo, Nelson Hastings, Philip Pearce, Sharon Laskowski, Steve Freeman (NVLAP)

Administrative Updates (Allan Eustis):

• We have received and posted the official transcripts from the March plenary meeting. There was a gap in between tapes that left out a discussion by Alan Goldfine and David Flater regarding the ISO 9000/9001 Quality Assurance/Configuration Management. This section was important because a decision was reached for alternative wording on a requirement. This gap has been filled with an addendum that has been posted.

EMC Requirements:

The latest draft requirements were posted. This is first version since the one discussed at the TGDC plenary meeting. It includes draft requirements for radiated emissions and telecommunication interference. Still in rough format - we are working on informative text, organization, some references, and testing procedures. It is complete in that it covers everything we intended to include.

QA/CM Requirements:

The complete version was discussed two weeks ago. As a result minor word-smithing changes were incorporated. This version has been sent to be included in current VVSG draft.

E-poll books (John Wack):

John summarized the joint STS/CRT/HFP subcommittees meeting. John Kelsey had discussed possible threats to privacy, being concerned over personally identifiable information being contained on one system that is used to activate ballots. The risk is that someone could piece together the information to figure out who voted for whom. There was also discussion about externally networking epoll books and ballot activation, concerns being for security and reliability. The decision at the meeting was to allow epoll books to activate ballots. The activation mechanism needs some additional requirements. David Flater has written some additional general applicable requirements. These need to be specific and strengthened. The decision was also made if epoll books were externally networked; they were not to be used for ballot activation. Some alternative options were discussed and these will be presented to the entire TGDC at the May plenary meeting.

Next CRT meeting scheduled for May 3rd may be canceled or given to another subcommittee.
 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, April 19, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Reliability and Accuracy Benchmarks (Flater) (see http://vote.nist.gov/TGDC/crt/misc/NewBenchmarks1.doc
3) Any other items.

Attendees: Alan Goldfine, Allan Eustis, David Flater, John Crickenberger (NVLAP), John Wack, Mat Masterson (EAC), Michael Kass, Michael Koo, Nelson Hastings, Paul Miller, Sharon Laskowski, Wendy Havens

Administrative Updates (Allan Eustis):

• There will be a workshop on remote access voting/internet voting at the Swiss Embassy next Monday co-sponsored by MIT/CalTech. There will be a number of election officials from Switzerland making presentations. A number of NIST staff are attending; we will report back on the meeting.

Reliability and Accuracy Benchmarks (David Flater):

The document posted on the web (see above) reflects the changes that were discussed at last weeks meeting. David went over several items in the paper. It was decided to keep the 120,000 voting sessions for EBM ballots. There will be a 1 in 500 allowance for paper feed errors/jams. It was decided that there would be no more than 6 errors (failures) allowed of any sort pending Paul Miller's final review. David was planning to finalize document at end of day so any comments were due immediately.

E-PollBooks (John Wack):

John Wack brought up the topic of e-pollbooks. The EAC requested the TGDC to take a look at writing requirements for e-pollbooks - at a minimum in regards to the use of them for ballot activation. Due to the short time left until the next iteration of the VVSG is due out, it was decided that we could not provide more than high-level requirements. That is what David Flater has done - he has added a device class in the VVSG for ballot activators.

The specific questions today were in regards to allowing networked devices, such as epollbooks, to also activate ballots. (The requirement in the VVSG currently states that any part of the voting system can not be externally networked and if epollbooks are used as ballot activators, they are part of the voting system.) Some states are currently using the networked epollbooks to activate ballots. The main concerns are security, availability and privacy. Paul Miller understands the availability issue being a major concern, but also thinks there are possibilities of handling this by having backup ballot activation techniques. This topic will be discussed by all three TGDC subcommittees on April 24th.
 

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, April 12, 2007
11:00 a.m. EDT
Meeting Minutes

Agenda:

1) Administrative updates (Eustis)

2) Reliability and Accuracy Benchmarks (Flater) .

3) QA/CM requirements (Goldfine) .

4) Ongoing Maintenance of Draft (Flater)

5) Any other items.

Note: The QA/CM paper (revision 3) is complete, pending CRT review. Track Changes (in the .doc file) documents all changes made since the plenary version.

Attendees: Alan Eustis, Alan Goldfine, Alicia Clay, David Flater, John Crickenberger, John Wack, Mat Masterson (EAC), Michael Kass, Paul Miller, Sharon Laskowski, Wendy Havens

Administrative Updates:

• Allan Eustis: Voting team met on Wednesday, April 11, and went over some deadlines. By May 14 there will be a near final draft of the VVSG available on the web. On Sunday, May 20, there will be an informal reception at the hotel.
• John Wack: At the team meeting, it was discussed what we needed to accomplish at the plenary meeting - we want to approve as much material of the VVSG as possible. John will send an email to the TGDC about our intentions. The only way we'll accomplish this is for the TGDC to come prepared, having read the material, and ready to discuss any complex issues.
• Alan Goldfine: Due to the short deadlines these days, it may not always be possible to get documents for the weekly meetings send out a week before the meeting.

Latest News on Reliability and Accuracy Benchmarks (David Flater):

There are no modifications to this week's document from last week. David is currently waiting on feedback from the vetting Paul Miller is doing with NASED. Paul said the feedback he is receiving is pretty consistent. There might be small modifications, but nothing substantial. Based on this input, David will work to finalize document and harmonize to the official draft. He will make final changes to the draft in the time frame May 7 - 9. There will still be some opportunity for change before the July deadline. Paul Miller wanted to know if David planned on making any changes to the background assumption material that he had provided; only editorial changes, nothing significant. Some of the language Paul used will be used to help clear up some definitions.

Quality Assurance/Configuration Management (Alan Goldfine):

Please see the document referenced to in the email - the copy on the web does not include the track changes that were specifically left in.

There were quite a few editorial and formatting changes, but only a few major changes that are pointed out. First, on page 6, requirement 2.1-a regarding the quality assurance manual's delivery deadline was changed to remove the statement "delivered before the start of the design and development process". There is no time specification. Second, on page 8, the list of critical parts and components was changed based on conversation with Philip Pearce. We added to conditions for critical parts and components, 1) components whose failure would violate a voter's ability to vote independently and 2) would impede the usability of the voting system by all voters. These two components support usability and accessibility.

John Wack felt there were a few requirements that STS should review, such as physical identification of the system and the requirements about voting system configuration log.

Ongoing Maintenance of Draft (David Flater): Nothing new - made changes discussed last week.

Other Issues:

John Wack suggested a meeting to over issues of concern in the VVSG for each subcommittee before the plenary meeting on May 21-22.

There will be an STS meeting in two weeks on e-poll books. CRT and HFP are invited to participate in the meeting.

Meeting adjourned at 11:30 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, April 5, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates (Eustis)
2) Reliability and Accuracy Benchmarks (Flater)
3) Ongoing Maintenance of Draft (Flater)
4) Any other items.

Attendees: Allan Eustis, Brit Williams, David Flater, John Crickenberger, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Sharon Laskowski

Administrative Updates:

• Next TGDC Plenary meeting is scheduled for May 21-22, 2007, here at NIST. Agenda will focus on the review of the near final VVSG draft.
• All material from the March meeting has been posted, including the two resolutions that were passed.
• The April-May telecon dates have been posted on web - any member of TGDC invited to attend any of the telecons.

Reliability and Accuracy Benchmarks (David Flater):

Current state of the Benchmarks research paper by Paul Miller and David Flater was forwarded to the subcommittee. This was prepared in response to the direction of the TGDC at the last plenary meeting - namely to attempt to gather what data/guestimates we could about different volumes of usage of different devices so as to come up with an estimate for what the benchmark for reliability and accuracy should be in at least the correct order of magnitude. From NASED feedback, no given number is going to be correct for the usage in every possible jurisdiction. Part of the direction that was taken was that different severities would be assigned to different types of failures in a manner that was not entirely different than what was done in the 1990 voting system standards.

[NOTE: David Flater asked why the scoring standard in the 1990 standard was taken out of the 2002 revision that described what a relative failure is and assigned different weights depending on what kind of failure it was. Brit Williams said there was not a major show stopping reason it was deleted. He noted that Appendices were not par of the 1990 VSS.]

Paul Miller is going to talk to other members of NASED to confirm numbers and approach. Assuming the process comes out favorably, the draft will be changed to expand the single benchmark into a collection of benchmarks that are tailored to individual types of equipment.
The paper contains definitions of several different types of voting equipment. These definitions are compatible to what is in the draft. These are written in plain language and the ones in the draft are more formal.

After that, there is an attempt to answer what constitutes a typical volume for these different types of equipment, understanding that each jurisdiction will understand different what "typical" means. There's a breakdown of how many errors/failures a precinct can tolerate on Election Day. Failures are classified differently depending on whether personnel present can easily remedy situation versus calling in specialized assistance.
There is a system level benchmark for accuracy. Apart from the general consensus that we don't want any vote to become unrecoverable, there was a feeling that the benchmark in the current standard was fine, we just need to understand how it will be applied. The two benchmarks in 05 were done using a sequential probability ratio test, which involves selecting a lower benchmark which conformity could be demonstrated at 95% confidence. The new test is different, using a fixed length test plan. [The difference between the tests was explained later by David Flater] We need a single benchmark to say what the requirement is - and if you don't meet this requirement, we can not recommend the equipment for certification. [Dave's December plenary presentation has a number of slides showing difference in testing methods.]

Next Step: Paul Miller will confirm or modify numbers based on discussion with NASED members. David will get material prepped for inclusion into draft VVSG.

Ongoing Maintenance of Draft (David Flater):

David is still catching up with changes discussed at plenary meeting. Including such changes as:

• Clarify testing terminology - test protocol out
• All procedural requirements are being changed to informative text explaining assumptions being made by product requirements
• Purging use of the phrase "may not" based on different interpretations
• A whole series of changes in response to comments from David Wagner
• Definition change for EBMs
• Defined what logic defect is
• Clarified logic defects found during conformity assessment are not field serviceable
• Clarify use of assertion and constraint
• Clarify vendor versus lab responsibility in logic verification
• Discuss phenomena such as numeric overflows and invocation of undefined behavior in both coding convention and logic verification section
• Positional changes still need to be done
• Clarify reporting of irrelevant contests
• Word-smithing in conformance clause
• Paper durability - start with JPC (joint Committee on Printing standards
• Add new device classes - audit devices, activation devices
• Add system level classes - 1 for independent dual verification for handling innovation class and for handling election verification which maps to end-to-end cryptographic systems

Meeting adjourned at 11:45 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference
Thursday, March 15, 2007
11:00 a.m. EDT
Meeting Minutes

Draft Agenda:

1) Administrative updates and TGDC plenary information (Eustis)
2) Discussion of revised QA/CM research paper (Goldfine)
http://vote.nist.gov/TGDC/crt/QA-CM-white-paper-for-CRT-r2.doc
3) Discussion of revised EMC Draft (Goldfine)
http://vote.nist.gov/TGDC/crt/EMC-Draft-4-R2-for-CRT.doc
4) Run-through of draft March 22 CRT plenary presentations (Goldfine, Flater)
5) Any other items.

Attendees: Alan Goldfine, Allan Eustis, Dan Schutzer, David Flater, John Wack, Lynne Rosenthal, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Sharon Laskowski, Steve Berger

Administrative Updates:

• Allan: TGDC members should receive by FedEx today their travel packages as well as their informational CDs. There will be an informal reception on the night of the 22nd. Next plenary meeting will possibly be at the end of May.
• John Wack: The format for the meeting we're going for is to discuss and decide on all non-controversial material on Day 1, saving Day 2 for open, longer discussions.

Discussion of QA/CM Research Paper: This paper has been posted on the web for a week. The material in this paper has not been included in the VVSG yet, it should go to coordinator of paper in the next week.

Alan Goldfine discussed the plenary presentations. The CRT subcommittee plans to cover 5 topics: 1) electromagnetic compatibility, 2) quality assurance/configuration management requirement, 3) review of CRT changes, 4) discussion of benchmarks, and 5) general discussion.

For electromagnetic compatibility, the requirements are still in the process of being developed. These requirements have been divided into three parts: conducted disturbance (should be completed next week), radiated disturbances (draft set of requirements by March 30), and telecommunications (incomplete, 1st draft by March 30). (Steve Berger forwarded these to IEEE for comments which are forthcoming.)

QA/CM requirements raise an issue. The requirements are being done as a result of a TGDC resolution directing the review of QA/CM. At the December 2006 meeting, it was decided that the ISO 9000/9001 standard would be used for quality assurance. This is just the beginning of the process . Details are needed to state what conformance means. Vendor requirements are more specific - required documentation, data delivered to test labs and EAC. The draft requirements state two different alternatives which were discussed in great detail. The first alternative requires that vendors provide their quality assurance procedures ( quality manual) early in the process, before design/development of system begins. The other alternative requires that the quality assurance procedures be done, but there is no specification. The problem identified : if a system's quality assurance procedures are not approved, and the design/development is well on its way, a system may not be approved and there would be wasted time and effort. Opinions were expressed that this did not seem to be a problem, that the real question was if the system passed certification, the manufacturing process must produce systems of the same quality. This topic did not appear to be resolved. It will be brought up at TGDC plenary meeting for discussion and possible consensus. It was pointed out that there was no time specified in the EAC manual regarding the registration process. Mat Masterson will bring this up to the EAC.

David Flater then discussed what he would be presenting at the plenary meeting. The first topic would be a review of the changes as discussed at the last CRT subcommittee meeting. David will also be presenting information regarding benchmarks. CRT has received feedback from NASED regarding the question about resetting benchmarks on reliability and accuracy in the next VVSG. Defensible numbers need to used for the benchmarks. The response from NASED in response to reliability said that no failures that lead to unrecoverable votes are acceptable, in other cases, our tolerance for failure depends on how hard it is to recover from those failures, there is no typical volume on which to base a benchmark. We currently do not have a particular volume to base a benchmark; however, if test labs are going to advise rejection of systems that perform unreliably during testing, there needs to be a benchmark for what constitutes an unacceptable rate of failure.

With respect to accuracy, we have a real requirement that does not map to a particular benchmark. The acceptable number of errors is 1 less than the vote margin between first and second place. The current benchmark is 1 in 10M ballot positions - this was set as a compromise based on costs of testing. NASED acknowledged the need to review test methods and also expressed concern that 1 in 10M is probably not achievable for real ballots. Since we want volume testing to produce realistic ballots, the benchmark should meet everyone's requirements but also be attainable. Should the benchmark be relaxed so that it is attainable? The route we went through did not produce the data we need to produce defensible benchmark numbers. There is not a lot of time left. At the TGDC meeting, we plan to ask our customers for input about what this number should be.

Paul Miller informed the group that NASED had concerns about putting into writing indicating an acceptable rate of failure. There was also a concern about the definition of terms. We have to look at what are "failures". If poll workers can't set up systems, that's a failure. If a printer jams, that's a failure. However, there are manageable failures and those should be identified. David stated that failures were defined as equipment breakdown, including software, so that continuous service is worrisome or impossible. Complete testing of systems in voting environment cannot be tested as we do not know how many types of machines a precinct will have and how many as backup, if any. Failures fall anywhere in a large spectrum. If we ask test labs to try and evaluate the severity of failures, we will get ambiguous results.

Any comments/questions should be sent via email.

The meeting adjourned at 12:10 p.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Subcommittee Teleconference*
Thursday, February 15, 2007, 11:00 a.m. EDT

Agenda:

1) Administrative updates (Eustis)
2) Letters to NASS and NASED re: benchmarks (Flater/Wack)
3) Discussion of QA/CM research paper (Goldfine)
4) Discussion of EMC Draft with outline (Goldfine)
5) Revised discussion draft (Flater)
6) Any other items.

Attendees: Alan Goldfine, Allan Eustis, Brit Williams, Commissioner Davidson (EAC), David Flater, John Cugini, John Wack, Lynne Rosenthal, Matt Masterson (EAC), Nelson Hastings, Paul Miller, Philip Pearce, Steve Berger

Administrative Updates (Allan Eustis):

• Welcome to Matt Masterson from EAC. Mat will be working with Commissioner Davidson and Brian Hancock and TGDC to serve as a liaison in the development of the VVSG.
• David Alderman's (NIST) testimony to the EAC on conformity assessment activities and review of the laboratories has been posted on the web. See: http://vote.nist.gov/NIST-CA-activities-under-HAVA-020807.pdf).

Letters to NASS and NASED Regarding Benchmarks (David Flater):

It was determined at the end of the last TGDC meeting that we needed election community buy-in of the benchmarks in terms of accuracy and reliability. Letters were sent to NASS and NASED asking for comments/consensus on these issues. NASS has declined to take a position, we have not heard back from NASED. (It was noted that due diligence was taken to get feedback.) Comm. Davidson suggested contacting State Election Directors to find out what they are doing. CRT had been advised by NIST General Counsel not to do this, but it was suggested that EAC might be able to. Comm. Davidson will check with EAC General Counsel.

ACTION: TGDC members present were asked to provide any information that had on this subject to CRT.

Discussion of Quality Assurance (QA)/Configuration Management (CM) Research Paper:

This paper was written after consensus at the December meeting that the ISO 9000 and 9001 standards should provide the framework for the VVSG 07 requirements dealing with QA/CM. This is a first draft of a set of requirements that attempted to do that. ISO 9000 is a general set of requirements - the real key is to design specific requirements that work for voting systems - that is what this paper tries to do. Alan asked for comments/feedback - did we address key issues, is it outside our scope, did we address everything within our scope? Steve Berger commented that this was going in the direction he had in mind. Steve also wanted to know if a vendor changed hardware/software, did it show up as a version change? Alan noted this specific issue was probably covered in configuration management. It is a valid concern and he will look into it.

ACTION: Specific technical comments should be discussed with Alan offline via email or phone.

Re 1.2.8.6, Philip Pearce asked about the critical parts list of what constitutes a failure of a part or a component - should there be something added regarding diminished accessibility or usability. Alan agreed and asked if Philip would draft up some appropriate bullets.

ACTION: Does group agree with list of critical items, please let Alan know if not.

[NOTE: In 2005, TGDC requested process model - CRT's working draft contains informative process model.]

EMC Draft with Outline:

Discussion was held at a previous meeting about revisions of electrical requirements within environmental requirements section of the VVSG. It was suggested CRT reach beyond the voting team to experts in the field of electrical equipment testing. We've talked to experts (at NIST) regarding radiated interference and conducted electromagnetic interference. Based on these discussions, we have outlined work and some draft requirements for this area. Alan inquired if there were any general opinions or concerns regarding this outline.

Technical questions should be sent via email to Alan Goldfine. Steve Berger questioned if there was any field data about problems currently with these issues? The answer was no, but that should not stop the group from looking into it. Steve also mentioned that there were a lot of "to be determined" specs; including requirements question marks. Alan pointed out that this represented discussion about whether certain topics need to be developed within volume 5 of the VVSG. Are these testing standards or are they specifically testing scenarios. This is a work in progress - all the TBDs will be determined. Comm. Davidson suggested getting comments from the vendors about costs after Alan receives Steve's questions via email. John Wack, Allan Eustis, and Alan Goldfine will discuss offline about bringing this up at the next vendor's meeting.

Discussion Draft (Summary of Changes):

Revised set of draft requirements are available on the web. (Specifically this is David Flater's material which has been submitted to the editorial team for review and formatting.) The change log file contains a count of things that were changed between drafts. David went over a few of the changes. Biggest change was in the introductory informative text to coding conventions (C language with use of COTs add ons becomes structured language) , and in the follow on changes to the conventions themselves. Other changes (subject to revision) included: data to be provided, functional testing, conformity assessment and use of components.

The question was asked if for the definition of EBMs (electronically assisted ballot markers), was consideration given to blindness or visually impaired. David noted that he clarified the definition of EBMs but that the issue with blindness or visually impaired was being looked at specifically by HFP.

Meeting adjourned at 12:00.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Subcommittee Teleconference*
Thursday, February 1, 2007, 11:00 a.m. EDT

Agenda:

1) Administrative updates (Allan E.)
2) Document delivery plans in advance of the next TGDC plenary (Allan E.)
3) Report from the EAC Advisory Board meeting (John W.)
4) Discussion of potential requirements for e-pollbooks (John W.)
5) Any other items.

Participants: Alan Goldfine, Allan Eustis, Brit Williams, David Flater, Commissioner Donetta Davidson, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Wendy Havens, Dan Schutzer

Administrative Updates (Allan Eustis):

• Sent out an email regarding a calendar of voting events which will be posted on the web - please review for items of interest and pleas forward items of upcoming events.
• Lucy Salah will be sending a logistics letter soon for next TGDC meeting, including travel, hotel, etc. The meeting will begin the morning of March 22, ending around 2:30 on the 23rd. Teleconference capabilities will be available for anyone not able to attend meeting.
• Deadlines for document development process - they will be posted on the web. We plan to send to TGDC members a draft VVSG and related white papers before the meeting as per other TGDC meetings..

Report from the EAC Advisory Board (John Wack):

• EAC Advisory board seemed like a good forum, 20 to 30 members in attendance - a good atmosphere to get work done.
• John attended to give an overview of the TGDC meeting and discuss the resolutions.
• People appeared to be interested in getting exported electronic records from different kinds of equipment (same vendor) to be interoperable with the consolidation software.
• Need to make sure that software getting results from DREs or absentee ballots can talk to each other. Some errors that were discussed at the board meeting were due to programming errors.
• Testing every piece of equipment to verify software would cause a lot of testing and take a while. Individual components of voting systems are not checked every time.
• Best we can do is conformance to the standards.
• Interoperability may take several years. Vendors may want to get together and do own interoperability testing.
• To give general interoperability requirements, we would have to define interfaces.
• There was a resolution passed at a TGDC meeting regarding common ballot format specifications - basically directing NIST to look at what sort of election markup language is out there and see if it sufficient for cast vote records, and then make a recommendation to the TGDC.
• Second item of the board meeting: Brian Hancock and some election officials have requested VVSG 07 have some requirements for e-poll books. Currently all general requirements of VVSG 07 cover e-poll books. Concern was expressed by subcommittee members that it is too early in the development process to write functional requirements for e-poll books.
• Philip Pearce discussed a little of his panel discussion at the board meeting, making sure that usability and accessibility were being considered when voting equipment was tested. He mentioned Jim Dickson's concerns regarding icons.

Other Issues:

Commissioner Davidson mentioned a question brought up to her at a meeting about problems with sending results over analog lines. Allan had heard the same problem in WY.

Future CRT phone meetings are scheduled for:

Thursday, February 15 at 11:00 a.m. (Topics of discussion to include technical aspects proposed for quality assurance and configuration management as well as electromagnetic requirements.)

Thursday, March 1 at 11:00 a.m.

Thursday, March 15 at 11:00 a.m.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************

CRT Teleconference*
January 11, 2007

Agenda:

1) Administrative updates (Allan E.)
2) Volume testing, reliability and accuracy testing discussion (David F.)
3) Election Management Systems discussion (David F.)
4) Data collection for reliability and accuracy benchmarks (David F.)
5) Any other items.

Participants: Alan Goldfine, Allan Eustis, Brit Williams, Dan Schuzter, David Flater, Max Etschmaier, Nelson Hastings, Paul Miller, Sharon Laskowski, Stephen Berger, Philip Pearce, Wendy Havens

Administrative Updates (Allan Eustis):

• New disclaimer will be read at the beginning of every telecom. Meetings are formally announced in Federal register. What is said at these meetings is public and NIST welcomes any and all new listeners.
• New articles, documents, and public comments have been placed on the TGDC website.
• Mark Skall and Allan attended Donetta Davidson's installation as Chief Commission of the EAC. Outgoing chief expects approval of new EAC members in January.
• Transcripts from the TGDC December 4 and 5, 2006 meeting are on web page.
• Any combined TGDC subcommittee meetings will be worked into current schedule of meetings. New meetings require a couple weeks notice to schedule.

Volume testing, reliability, and accuracy testing discussion paper (David Flater):

This write up is is a realization of things that were discussed at TGDC meeting. First major addition is requirements to conduct volume testing similar to CA volume testing reliability protocol. Specific parameters were modeled after this protocol -- one exception is with respect to central tabulators. The other part has to do with test methods for accessing conformance to the benchmarks for reliability accuracy and probability of misfeeds for paper based tabulators. Following that are specific requirements for reliability, accuracy, and probability for misfeeds.

Discussion regarding this paper among subcommittee members followed. After discussion, Allan asked Steve Berger to summarize his concerns so that David Flater would know which direction to proceed.

Steve: Looking at the testing resource required by the draft (Table 7 below), we should look at it two ways. First, determine what resource is being assumed and explicitly state that this current draft would require "x" number of man hours of testing. Second, if the minimum level of testing is performed, what's the confidence that can be stated; if there's not enough testing to make a credible statement, then we need to look at why we're doing the test in the first place. Second point, do the tests, as currently prescribed, isolate problems being observed in the field. For example, the calibration issue. Do the tests we currently have screen out future systems that may require too frequent calibrations. Do we understand the causal factor? If we don't, then we need to do research.

David Flater provided a high level preliminary answer. We have not specified a particular test suite. We have specified a test method. Specific resources and results at end of conformity assessment depend in large part on the test suite used, which is currently developed by the test labs and approved by the EAC. With respect to isolation of specific issues, those issues are targeted by a series of environmental tests, from operational and non-operating points of view. We can always do research on causal issues.

Allan Eustis pointed out that some of these issues would be examined at an upcoming workshop by EAC on cost of testing. Also, relevant policy-related issues need to be referred to the EAC.

Steve Berger stated that the fundamental question had to be asked "would problems in the field today be prevented by the VVSG 2005 (as published), if not have we rectified the concerns in order that issues would be effectively identified and handled under VVSG 2007?"

We can use this methodology on a number of field issues. We should take a systems view. (Max's next paper on Quality Assurance will address some of these issues and will be released in the next week or so. It will be a discussion item at an upcoming CRT meeting.)

Election Management Systems (EMS) Discussion Paper (David Flater):

There is confusion over the definition of the Election Management System (EMS) as defined in the 2002 VSS. The current definition covers both pre and post voting functions. The question posed is whether the definition should be revised, making a distinction between pre and post requirements. (Brit Williams pointed out that originally (VSS 1990) the definition referred to firmware and software, and that pre and post election were added later. Some vendors group both functions into one software package, others separate it by function. EMS is run on standard desktop/laptop computers.) Consensus at the meeting was that the current definition was on track, and that follow up concurrence would be asked for from ITAA.

Data Collection for Reliability and Accuracy Benchmarks (David Flater):

In order to set benchmarks for reliability and accuracy, we need to know what the customer wants to see in terms of reliability and accuracy. Since se have controversy over testing, it puts benchmarks on hold. The way we do this data collection is currently being reviewed by the legal office.

The question arose about availability. How is availability defined? Turn on a machine and if it works it means it's available. Concerns were expressed over the fact that logic and accuracy testing was not performed at the precincts before use. It was also discussed that some issues/problems thought to deal with calibration problems turned out to be usability problems. Some of these issues are being looked at by the HFP subcommittee.

Concerns expressed by CRT participants: Does a system go out of calibration during transport? Are the environmental tests being performed sufficient to determine this? What about older technology problems such as memory packs that are either old, or the technology on them is old? Separate component issues were raised. We need to evaluate the components of the system, as well as the system as a whole. Manufacturers of components need to be contacted about expectations of individual components. We need to know if touch screen systems are being deployed in a manner that goes against manufacturers' advice or if they are being subjected to expectations beyond what manufacturers say are reasonable. We then need to look at the process to detect when this is happening. We also need to make sure that vendors are in agreement with component manufacturers about what to expect. Vendors do not want products used in unacceptable operating conditions.

Next teleconference is scheduled for February 1, 2007, at 11:00 a.m. EST.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC in directing NIST staff and coordinating its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

• Teleconferences from 2004, 2005, 2006 and upcoming in 2006.

************