Technical Guidelines Development Committee (TGDC)
Security and Transparency Subcommittee* Teleconference
July 17, 2007, 10:30 a.m.

Draft Minutes

Agenda

1) Administrative Updates

2) Discussion of open ended vulnerability testing (OEVT) (Background material: see below)

3) Other items

4) Next call July 24, 2007 at 10:30AM

Attendees: Allan Eustis, Wendy Havens, Ron Rivest, Alicia Clay, Matt Masterson (EAC), Barbara Guttman, Angela Orbaugh, Nelson Hastings, John Kelsey, Quinh Dang, John Wack, Alexia Scott Morrison, Santosh Chokani, Andrew Regenshied,

Administrative Updates:

· Next TGDC plenary telcon on track for August 17, 2007 beginning at 11:30 EDT.

· Final draft of the VVSG document should available later this week for review.

Discussion of open ended vulnerability testing (OEVT) (Ron Rivest):

· RR notes that OEVT document looks good.

· AC summarized David Flater’s issues. OEVT has been used as a test reference along with source code review.

· The issue was raised of whether OEVT team had to examine various requirements as opposed to setting their own agenda. Test was proposed/suggested to clarify this at the end of the introductory material: “OEVT will generally be conducted in conjunction with or following inspection, functional performance or vulnerability testing driven by test suites. All aspects of the voting system, devices or use procedures may come under the purview of OEVT. However, in some instances, when the primary method of assessing conformance to a requirement is source code review, OEVT is specifically listed as a test reference because it may provide a faster, more efficient, and less expensive alternative.”

· RR : test references left in. AC yes. NH noted that this occurred about five times in the requirements.

· RR puzzled by operational impact of wording. SC noted they are complimentary techniques. AC noted possibility of deleting the word “alternative” and substituting “technique” or “methodology”.

· OEVT team and source code review team can be different groups or, more often, the same evaluation team.

· RR suggested demoting them from test references where appropriate to a discussion field item.

· NH: Original motivation for OEVT method was incongruity of having a requirement with no test reference. JW noted the special case of OEVT that could be explained in discussion field. (This was agreed to by STS teleconference participants)

· LR made recommendation to avoid language that specified the order of testing.

· RR: OEVT is most often done near the end of the process.

· Discussion ensued on configuration of voting system for testing versus elections. MM had question on “re-configuration” and whether this changed the voting system submitted for certification? RR noted that it meant setting up for different elections. That would probably be in scope whereas disabling controls would require clearance.

· AC noted that rules of engagement for OEVT team are included in the requirements.

· Discussion of mock elections as part of testing ensued. JW noted that volume testing is in effect a mock election. All agreed this needs to be spelled out more clearly- including size of volume test. RR brought up possibility of a small realistic mock election. BG noted that inspection and set up requirements may accomplish this.

· MM brought up need for clear definition of “plausible description” and “plausible analysis” in 1.1.2.g. RR noted that bright defining line is not possible here. AC noted that someone will have to sit in judgment. Discussion ensued on defense in depth as an example. MM noted that EAC will need guidance here. RR noted that a definition for plausible threat could include number of individuals required to exploit vulnerability.

· AC will come up with draft text for “plausible definition and will circulate to STS via e-mail for review. RR noted criteria for failure is at the VSTL level. MM pointed out that VSTL’s do not like to make judgment calls.

· RR indicated that perhaps NISt could produce guidance documents for VSTLs in this and associated requirements.

Topics for upcoming STS calls (Alicia Clay):

· Will review IVVR next Tuesday at STS Teleconference.

Next STS meeting is scheduled for Tuesday, July 17th. The discussion will be on OEVT.

Meeting adjourned at 11:22.

[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion is for the purposes of the STS subcommittee of the TGDC to direct NIST and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

1.1 [[Open-Ended Vulnerability Testing

Vulnerability testing is an attempt to bypass or break the security of a system or a device. Like functional testing, vulnerability testing can falsify a general assertion (namely, that the system or device is secure) but it cannot verify the security (show that the system or device is secure in all cases). Open ended vulnerability testing (OEVT) is conducted without the confines of a test suite. It instead relies heavily on the experience and expertise of the OEVT Team Members, their knowledge of the system, its component devices and associated vulnerabilities, and their ability to exploit those vulnerabilities. The goal of OEVT is to discover architecture, design and implementation flaws in the system which may not be detected using systematic functional, reliability, and security testing and which may be exploited to change the outcome of an election, interfere with voters’ ability to cast ballots or have their votes counted during an election or compromise the secrecy of vote. The goal of OEVT also includes attempts to discover logic bombs, time

bombs or other Trojan Horses that may have been introduced in the system hardware, firmware or software for said purposes.

1.1.1 OEVT Scope and Priorities

ž 1.1.1-A Scope of Open Ended Vulnerability Testing

The scope of open ended vulnerability testing shall include [[but not

be limited to the]] voting system security during all phases of the

voting process and [[shall]] include all vendor supplied voting

system use procedures.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

The TGDC open-ended vulnerability search and research should not

exclude those involving collusion between multiple parties (including

vendor insiders). The scope of OEVT includes but is not limited to the

following:

• Voting System Security

• Voting System Physical security while voting machines are:

o In storage

o Being configured

o Being transported, and

o Being used

• Voting System Use Procedures

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.1-B Focus of Open Ended Vulnerability Testing

OEVT Team members shall seek out vulnerabilities in the voting

system that might be used to change the outcome of an election, to

interfere with voters’ ability to cast ballots or have their votes

counted during an election or to compromise the secrecy of vote.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.1-C OEVT General Priorities

The OEVT team shall prioritize testing efforts based on:

• threat scenarios for the voting system under investigation,

• the availability of time and resources,

• the OEVT team’s determination of easily exploitable

vulnerabilities and

• the OEVT team’s determination of which exploitation

scenarios are more likely to impact the outcome of an

election, interfere with voters’ ability to cast ballots or have

their votes counted during an election or compromise the

secrecy of the vote.

NIST User! 6/26/07 2:31 PM

Comment: [[Add a formal citation and use the

standard reference here.]]

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Following are suggestions for OEVT prioritization in the areas of threat

scenarios, COTS products and Internet based threats. The intent here is

to provide guidance on how to prioritize testing efforts given specific voting

device implementations.

Threats that can be exploited to change the outcome of an election and

flaws that can provide erroneous results for an election should have the

highest priority.

Threats that can cause a denial of service during the election should be

considered of very high priority.

Threats that can compromise the secrecy of the vote should be considered

of high priority.

A threat to disclosure or modification of metadata (e.g., security audit log)

that does not change the outcome of the election, does not cause denial of

service during the election, or does not compromise the secrecy of ballot

should be considered of medium priority.

If the voting machine uses COTS products, then the OEVT team should

also investigate publicly known vulnerabilities.

The OEVT team should not consider the voting machine vulnerabilities that

require Internet connectivity for exploitation if the voting machine is not

connected to the Internet during the election and otherwise. However, if the

voting machine is connected to another machine which in turn may have

been connected to the Internet (as in the case of epollbooks), Internet

based attacks may be plausible and should be investigated.

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž [[1.1.1- D Rules of Engagement – Context of testing

Open ended vulnerability testing shall be conducted within the

context of a process model describing a specific implementation of

the voting system and a corresponding model of plausible threats.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Vendor to supply both models as a part of the TDP???

Source:

Impact: Click here to add the Impact

ž 1.1.1-E- Rules of Engagement – Adequate system model

The OEVT team shall verify that vendor provided system model

sufficiently describes the intended implementation of the voting

system.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Vendor’s system model and associated documentation should reliably

describe the voting system and all associated use procedures given the

environment in which the system will be used.

Source:

Impact: Click here to add the Impact

ž 1.1.1-F- Rules of Engagement – Adequate threat model

The OEVT team shall verify that the threat model sufficiently

addresses significant threats to the voting system.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Significant threats are those which could:

• change the outcome of an election,

• interfere with voters’ ability to cast ballots or have their votes

counted during an election or

• compromise the secrecy of vote

Team may modify threat model to include plausible threats missing from

the vendor’s threat model.

Source:

Impact: Click here to add the Impact

ž 1.1.1-G- Threat model - Failure

Voting systems shall fail open ended vulnerability testing if the

vendor’s model of the system along with associated use procedures

and security controls does not adequately mitigate all significant

threats as described in the threat model.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Team may use a threat model that has been amended based on their

findings in accordance with 1.1.2-A

Source:

Impact: Click here to add the Impact ]]

1.1.2 OEVT Requirements and Guiding Principles

ž 1.1.2-A OEVT Team Resources

The OEVT team shall use the vendor supplied Technical Data

Package (TDP) and User documentation, have access to voting

devices configured similar to how they are to be used in an election,

and have access to all other material and tools necessary to conduct

a thorough investigation.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Materials supplied o the OEVT team should include but not be limited to

the following:

• Threat analysis describing threats mitigated by the voting system

• Security architecture describe the philosophy or protection (i.e., security

model) and how threats to the voting system are mitigated

• High level design of the system

• Any other documentation provided to the testing laboratory

• Source code

• Operational voting system configured for election, but with the ability for

the OEVT team to reconfigure it

• Testing reports from the developer and from the testing laboratory

including previous OEVT results

• Tools sufficient to conduct a test lab build

• Physical and procedural security procedures

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2- B Open Ended Vulnerability Team Establishment

The test lab shall establish an OEVT team of at least 3 security

experts and at least one election management expert to conduct the

open ended vulnerability testing

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

See Chapter 3 of Volume 4 for a description on the Technical Data

Package and associated requirements.

Source: Click here to add the Source

Impact: Click here to add the Impact

ž 1.1.2-C OEVT Team Composition – Security Expert

The OEVT team shall have at least one member with [[6]] or more

years of experience in the area of software engineering, at least one

member with [[6]] or more years of experience in the area of

information security, at least one member with [[6]] or more years of

experience in the area of penetration testing and at least one

member with [[6]] or more years of experience in the area of voting

system security.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

OEVT team knowledge should include but not be limited to the following:

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-D OEVT Team Composition- Election Management Expert

The OEVT team shall have at least one member with at least [[4]]

years of experience in the area of election management.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

The OEVT team will require consultation from an elections expert who is

familiar with election procedures; how the voting systems and installed and

used; and how votes are counted.

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-E OEVT Team Knowledge

The OEVT team knowledge shall include but not be limited to the

following:

• Good knowledge of work done to date on voting system design, research

and analysis conducted on voting system security, and known and

suspected flaws in voting systems;

• Good knowledge of threats to voting systems;

• Knowledge equivalent to a Bachelor’s degree in computer science or

related field;

• Experience in design, implementation, security analysis, or testing of

technologies or products involved in voting systems

• Experience in the conduct and management of elections

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

Alicia Clay Jones! 6/28/07 11:27 AM

Comment: Redundant?

D I S C U S S I O N

• Examples of threats to voting systems can be found in the information

presented at the October 2005 NIST Workshop on voting system threats;

• Examples of fields related to computer science are Mathematics,

Software Engineering, Information Security and Electrical Engineering.

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-F OEVT Fail Criteria[[ – Violation of Requirements]]

[[The voting device shall fail open ended vulnerability testing if the

OEVT team shall define fail criteria using finds vulnerabilities and or

errors detected in the voting device that violate requirements in the

VVSG and the vendor supplied Technical Data Package.]]

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

[[While the OEVT is directed at issues of device and system security, a

violation of any requirement in the VVSG can lead to failure.]] Following

are examples of issues for which the test lab must give a recommendation

of “fail”:

Evidence that any single person can cause a violation of a voting system

security goal (e.g. integrity of election results, privacy of the voter,

availability of the voting system), assuming that all other parties follow

procedures appropriate for their roles as specified in the vendor

documentation,

Vendor's documentation fails to adequately describe:

(a) the potential vulnerabilities of the voting system (e.g. as

given for example in the Brennan Center report [cite]), and

(b) the security mechanisms or procedures that serve as

countermeasures to these vulnerabilities in the submitted voting system.

Use of a cryptographic module that has not been validated against 140-2;

(7.1.1-A)

Ability to modify electronic event logs without detection.(13.2.1-B)

A VVPR that has an inaccurate or incomplete summary of the cast

electronic vote (1.2-A)

Unidentified software on the voting system. (8.3.2.1-C)

Identified software which lacks documentation of the functionality it

provides to the voting device.(1.3.1.1-B.3)

Access to configuration files without authentication (1.4-O.3)

Ability to cast more than one ballot within a voting session (1.2.1-G)

Ability to perform restore operations in Activated Mode (1.2.4-E)

Enabled remote access in Activated Mode (1.2.3-H)

Ballot boxes without appropriate tamper evidence countermeasures (1.2.5)

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-G [[OEVT Fail Criteria – Critical Flaws]]

[[The voting device shall fail open ended vulnerability testing if the

OEVT team provides a plausible description of how test lab shall

recommend a “fail” after having discovered vulnerabilities or errors

found in a voting device or the implementation of its security features

could be used to:

• change the outcome of an election,

• interfere with voters’ ability to cast ballots or have their votes

counted during an election or

• compromise the secrecy of vote

without having to demonstrate a successful exploitation of said

vulnerabilities or errors.]]

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

[[The OEVT team does not have to develop an attack and demonstrate the

exploitation of the vulnerabilities or errors they find. They do however have

to offer a plausible analysis to support their claims.]]

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-H OEVT Level of Effort – Test Plan

In determining the level of effort to apply to open ended vulnerability

testing, the test lab shall take into consideration the size and

complexity of the voting system, any available results from the “close

ended” functional, security and usability testing laboratory analysis

and testing activities; the number of vulnerabilities found in previous

security analysis and testing of the voting system and its prior

versions.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-I OEVT Level of Effort – Commitment of Resources

The OEVT team shall examine the system for a minimum of 12 staff

weeks.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-J OEVT Reporting Requirements

The OEVT team shall record all information discovered during the

open ended vulnerability test, including but not limited to:

• Names, organizational affiliations, summary qualifications, and resumes

of the members of the OEVT;

• Time spent by each individual on the OEVT activities;

• List of hypotheses considered;

• List o hypotheses rejected and rationale;

• List of hypotheses tested, testing approach, and testing outcomes;

• List and description of remaining vulnerabilities in the voting system.

o A description of each vulnerability including how the vulnerability

[[violates one or more VVSG requirements or how it can be exploited and

the nature of the impact to change the outcome of an election, to interfere

with voters’ ability to cast ballots or have their votes counted during an

election or to compromise the secrecy of vote. ]]

o For each vulnerability, the OEVT team should identify the VVSG 2007

requirement(s) violated

o The OEVT team should flag those vulnerabilities as serious if the

vulnerability can result in the violation of one or more VVSG 2007

requirements; a change of the outcome of an election; or a denial of

service (lack of availability) during the election.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

Examples of the impact of an exploited vulnerability are over-count of

ballots for a candidate; undercount for a candidate; very slow response

time during election; erasure of votes; and lack of availability of the voting

machine during election.

Source: [OEVT white paper]

Impact: Click here to add the Impact

ž 1.1.2-K VSTL Response to OEVT

The VSTL shall examine the OEVT results in the context of all other

security, usability and core function test results and update their

compliance assessment of the voting system based on the OEVT.

Applies to: Click here to add the Applies to text

Test Reference: Click here to add the Test Reference

D I S C U S S I O N

The testing laboratory should examine each vulnerability that could result

in the violation of one or more VVSG 2007 requirements; a change of the

outcome of an election; or a denial of service (lack of availability) during the

election and use the information to form the basis for non-compliance. If

significant vulnerabilities are discovered as a result of open ended

vulnerability testing, this may be an indication of problems with test lab

procedures in other areas as well as voting system design or

implementation.

Source: [OEVT white paper]

Impact: Click here to add the Impact