Thursday, October 12, 2006
Alan Goldfine, Allan Eustis, David Flater, Max E., Nelson Hastings, Paul
Miller, Philip Pearce, Sharon Laskowski, Wendy Havens
updates (Allan E.)
2) Rescheduling the November 2 CRT phone meeting to October 26.
3) Discussion of revised "On Accuracy Benchmarks, Metrics, and
Test Methods" (David F.). Please read:
4) Discussion of "Voting Machines: Reliability Requirements, Metrics,
and Certification" (Max E.). Please read:
NOTE: Max's PowerPoint presentation is attached to this email.
5) Discussion of the remaining issues from "Issues List" (David
F.). Please read:
6) Any other items.
- Allan welcomed
Philip Pearce as an official member of the TGDC. Also Paul Miller, as
of today October 12, 2006 is now an official member.
- New members will
be getting an orientation package very soon. As soon as the fourth new
member joins (hopefully within the next couple of weeks) we will have
an orientation teleconference with EAC Commissioner Davidson.
- Alan Goldfine
discussed rescheduling CRT meetings, moving the meeting on Nov. 2 to
Oct. 26 at 11:00 a.m. Nov. 16 & 30 meetings also moved to 11:00
On Accuracy Benchmarks,
Metrics, and Test Methods - David Flater
This came about because
there was an issue highlighted in the draft about whether we want to use
a single high-level end to end error rate for the system or do we want
to retain the individual error rates that were specified for each low
level operation that were in the previous editions of the standard.
In accuracy assessment,
no value in having low level error rate - found other issues:
- Low-level versus
single end-to-end error rate - when done as full analysis, we get predictions
of end-to-end error rate. Context of doing this in a test lab is too
narrow. No value. Error rates are not observable in a system level test.
- Other versions
used a probability ratio sequential test for the design of accuracy
assessment. Assumes you're doing a single test. More valuable to collect
data through entire campaign.
- Fixed length
versus sequential test plan - when enough evidence is collected to verify
system doesn't meet accuracy benchmark, you can terminate testing. We
may want to run the entire test campaign for other reasons.
- Validity as a
system test - The accuracy testing specified in VVSG 05 allowed the
test lab to bypass portions of the system that would be exercised during
an actual election. Issues such as those reported in CA volume testing,
and the cost issue. David position is we have to do end-to-end testing.
- The accuracy metric
in the 2002 VSS and VVSG'05 is ambiguous. Need to clarify.
- Definition of
"volume" i.e. filled in oval. 1990 VSS Votes versus ballots.
Define volume as votes and not detection of marks on a ballot.
- The Bernoulli
process assumed by the 2002 VSS and VVSG'05 is an invalid model of tabulation.
The system can do worse than miscount, it can manufacture votes. The
Poisson method is a more valid model, allowing for the possibility of
more than one error per unit of volume.
- In the determination
of error, it is unclear how inaccuracies in ballot counts and totals
of under votes and over votes factor in.
- All changes have
ramifications for what kind of benchmark make sense. Old standard specified
1 error in the volume of 10M. In testing, only 1 error in 500K.
- We may want to
modify the benchmark to what can be practically demonstrated in testing.
- Is the new approach
- Is the 1 in 10M
benchmark still appropriate? If not, what should it be?
- Is the 90 % confidence
level appropriate? If not, what should it be?
- Should the test
plan be fixed length, or should we stop as soon as there is sufficient
evidence that the accuracy benchmark is not satisfied?
- Allan - Is this
dependent on the acceptance by TGDC of the volume test program?
- David F - Some
changes move us closer to a "California" test method - an
end-to-end system test. Only requirement is to mitigate sufficient volume
to change risks.
- Alan G - One
reason why California is a disadvantage, it decreases reproducibility
of our (volume) tests.
- Paul Miller -
Saw the volume test in San Diego - concerns about interface user interaction
to the screen freezing and jams with the VVPAT systems. Are these considered
errors and how they can be fixed. (David: These are operational (reliability
Reliability Requirements, Metrics and Certification - Max Etschmaier
- Voting system
functions: Look at the whole thing - but then look at individual details.
- In my last presentation
I laid out my general philosophy. Hopefully you've looked at report.
The section is limited in its scope. Documented a lot of analysis. This
is the first phase of reliability requirements. Transcends into other
- Example of a voting
system. Options generation: sets up voting machine for election, separate
from machine, but inputs data so security issues. Control program: core
of voting machine, model is invariant, does not change. I/O Device:
self contained device, if fails, no critical effect on other systems.
Verification unit: verifies machine working properly before, during
and after election. Machine core: physical structure, holds all other
components together. Alternate record: depends on legal requirements
as to whether it's needed.
- Discussed critical
failures and usage pattern at last meeting.
- With all we have,
we can do a functional failure analysis.
- Procedure: For
every critical function - identify design requirements to avoid it,
if none found, set limit on failure probability. For every non-critical
function - determine failure probability, set limit if possible.
- Design requirements
from analysis. Do we look at the machine as a whole or do we want to
break it down and understand the pieces to make an analysis - more meaningful
statements. Architecture needs to be transparent - requires separation
of code and data. Components need to be separate between input and output
devices. Fail safe architecture where possible. No modification after
certification. [Discussion: Alan G - you're explicitly forbidding scenario
where any changes can be made by vendor before machine is used. David
F - EAC posted a firm statement that says if changes to machines have
not been reviewed, the machine will not be certified. Legally it is
the responsibility of the state to make sure machines are certified.
Paul M - Any modifications that do not affect machine workings, will
be considered certified - vendor and county (now state) decision.]
- Reliability requirements.
Critical and non-critical components.
- Results are a
voting machine that provides security repository of the original vote,
feeds into precinct and regional systems, recount (through verification
unit) available anytime, and very few failures - any precinct would
be able to make due with one spare voting machine. With this rate of
failure, strategy # 2 discussed before not necessary.
- A prototype should
- Two parallel paths
increase failure probability and decrease reliability.
requires very careful analysis. Vendor is responsible for compliance.
Volume testing as validation, and start of in-status ongoing performance
monitoring. No modification after certification.
- Submission for
certification. Allan - What's different from the current EAC requirements?
David F - the requirements stating certification of component reliability
and enforceable assurance of technical and financial fitness.
- Next steps. Find
a path to implementation: 1) Define requirements for quality and configuration
management, 2) transfer results to other parts of the VVSG, 3) examine
transition for jurisdictions, and 4) examine transition for vendors.
Formulate reliability requirements for VVSG: 1) specify format and 2)
set limits on probabilities. [Decisions for TGDC as a hold]
- Currently working
on the quality assurance and configuration management which was handled
completely different in VVSG 05 and previous versions. Max will be delivering
a report on this by the December meeting.
- This is a framework.
Decision needs to be made on how we want to go.
- Discussion will
continue at the next CRT meeting.
Next CRT meeting
will be October 26 at 11:00 a.m.
to NIST HAVA Page
Last updated: July 25, 2007
Point of Contact
policy / security notice / accessibility statement
NIST is an agency of the U.S. Commerce Department