Improving the Voting Process: A Multi-Disciplinary and Politicized Problem
by
Roy G. Saltman
Independent Consultant and Author
The Monthly Colloquium Lecture
National Institute of Standards and Technology
Gaithersburg, Maryland
September 21, 2007

ABSTRACT

The current problems of the voting process are presented in the context of the Constitutional requirement of division of responsibilities between the federal and state governments. Developments beginning in the late 19th century are described, involving voting technology, improved ballot secrecy, and elimination of paper ballots because of extensive fraud. The difficulties resulting from the use of computing technology, beginning in the 1960s, such as fear of software fraud and ambiguity of results due to “hanging chads,” are elaborated. The ground-breaking NBS reports of 1975 and 1988 are reviewed for their recommendations and anticipation of present problems. Current issues discussed include the questions of software correctness, testing of commercial off-the-shelf software, independent verification with and without paper trails, voter registration integrity, and partisanship at the highest levels of supposedly evenhanded administration.

1. The US System of Elections

Dr. Turner, Dr. Migdall, members of the Colloquium Committee, and members of the audience: I am deeply honored by the invitation to speak today, given the distinguished reputations of those who have preceded me in this lecture series.

The political crisis that resulted from the 2000 Presidential election in Florida raised public awareness of issues of election administration, from previously close to zero to a near fever-pitch in some quarters. Debates began and continue today on measures necessary to improve the process. Congress enacted the Help America Vote Act (HAVA) in October 2002 and, possibly, more legislation will be adopted in the near future. In order to discuss current concerns, an understanding of the history of this subject is very useful. Situations in this field seem to repeat themselves in interesting ways; parallels will be pointed out.

Elections, even elections for federal office, are carried out by the states, not the federal government. Under the Constitution, the federal government has the right to determine the “times, places, and manner” of conducting elections for federal office but has no responsibility for their administration. The federal government has no authority whatsoever over the administration of elections for state and local government offices. However, the federal government has had the power to intervene in elections run by the states in the case of civil rights questions since the ratification of the Fourteenth Amendment in 1868. Constitutional requirements were finally implemented with adoption of the Voting Rights Act of 1965 as amended. As a result, the problem of lack of access to the ballot by persons of any race or ethnicity was considerably ameliorated, if not totally solved, by the last decade of the 20th century.

The carrying out of elections by the states, as well as the devolution of authority in many states to local jurisdictions such as counties, cities, and even towns and townships, has created disparities: -- in election equipment, in forms of ballot presentation, and in many types of regulations, such as those on polling hours, use of absentee ballots, early voting, and voting by ex-felons. This high level of disaggregation is unique among democratic nations, even those with a federal system of government. Its effect is to require that mandated national reforms can only be instituted by an Act of Congress and, in many situations, these reforms can apply only to federal elections.

State agreement to adopt the federal reforms for state elections often occurs because of the extra costs and duplicative systems that states would incur by providing two election administrations, one for federal offices and one for state and local government offices. States often consolidate their elections with federal elections to reduce costs, increase convenience for voters, and enable candidates for different offices of the same party to campaign together. A federal law applying to federal elections that has been generally implemented by the states for their own elections to avoid extra costs is the National Voter Registration Act (the so-called “motor-voter” act) of 1993. This law has been often cited as an “unfunded mandate” because no federal funding was provided to the states with passage of the act.
The intricacies of our election system and its associated difficulties are not understood by many citizens. During the debate held on July 23, this year, among competing Democratic candidates for the Presidential nomination, the following question was asked by a member of the audience, speaking over the Internet:

“If I can go out into any state and get the same triple grande, nonfat, no-foam vanilla latte from Starbucks, why can’t I go to any state and vote the same way?”
[The Washington Post, July 24, 2007, p. A6]

Ordinary citizens cannot be expected to be Constitutional scholars or experts on government, but one would hope that our educational system would prevent the type of appalling naiveté demonstrated in this question. Of course, it would be possible to vote from anywhere in the country if each and every citizen could be positively identified remotely, if all possible ballot formats were available on-line, if the security and privacy of Internet voting were assured, and if sufficient money were appropriated to make all of this possible. At this time, we have not sufficient funds to carry out much simpler tasks which are far more fundamental.

2. A History of Procedural and Technological Change

2a. The Late 19th Century

The national crisis in election administration that resulted from the 2000 Presidential election was not the first of its kind. A desire for significant change in the manner of conducting elections began in the Gilded Age, the period between the end of the Civil War in 1865 and the beginning of the 20th century. That era is known as a time of extensive election fraud. Multiple voting by paid individuals occurred in many places; hence the command to “vote early and often.” Several types of ballot manipulations were widely used, such as ballot stuffing, deliberate invalidation of opponents’ ballots, and destruction of real ballots and their replacement by pre-marked ballots. Bribery and intimidation were widespread because of defects in the voting system. A dethroned Boss Tweed admitted to false counting in 1878, stating:

“The ballots made no result; the counters made the result.”

Two innovations were implemented in response. Neither development was instituted by federal action, and each required many years for acceptance by states acting individually. The first was the adoption of the so-called “Australian” or secret ballot. This type of ballot, often called the “blanket ballot,” included the candidates from all parties and all contests; it was issued to the voter at a polling station and filled out by the voter and cast at that location. Before use of the neutral and secret ballot, political parties distributed their own ballots. There was violence at polling stations as party stalwarts vied to have a prospective voter accept a particular ballot. In some locales, hired thugs called “shoulder-hitters” attempted to physically prevent known opposition voters from approaching the polling station. (Women were generally not subject to these problems because, in those times, they could not vote in most states.) Since the party-produced ballots were visually distinctive, it was often clear which ballot a voter deposited in the ballot box. The secret ballot was adopted statewide, first in Massachusetts in 1888 and, by 1896, about 40 states had individually adopted it. The remaining states did not accept the new system until various times in the 20th century.

The second innovation, the invention of mechanical voting devices, began as a response to the ballot frauds of the Gilded Age and the desire for quick and correct determination of results. The use of the first voting machine occurred in a local election in Lockport, New York, in 1892. Most mechanical machines that were in use in 2006 are based on the inventions of Alfred Gillespie, revealed in his patents of 1897 and 1899. Gillespie, originally from Atlantic, Iowa, invented the reversible small levers that voters use to select candidates as well as the large lever, connected to the privacy curtain, which causes the votes to be cast and the small levers to be reset when the curtain is opened. In the design of almost all mechanical voting machines, the votes of each voter add to summing counters, one counter for each candidate, located within the machines. The total counts are viewed on these counters after the polls are closed and the machines opened. In 1964, about two-thirds of all voters in this nation cast their votes on lever machines. It is fascinating that the success of mechanical voting machines was due to their elimination of paper ballots and the time for counting them. In contrast, since 2003, a strong effort has been waged by activists to demand, through a new federal law, the requirement for use of paper ballots in all contests. More about this later.

An interesting sidelight on the use of mechanical voting machines is that in 1871, a federal law was adopted which required “that all votes for representatives in Congress shall hereafter be by written or printed ballot ...” This law may have been enacted to assure the elimination of oral voting, a process prevalent earlier in the 19th century but, by 1871, used only in Kentucky and Oregon. The mechanical no-ballot machines proved their validity and acceptance in a federal election in Rochester, New York, in 1896, but a protest to the House of Representatives resulted, due to the 1871 law. The protest was turned down by the House, but the Congress modified its requirement. Legislation enacted in 1899 also permitted (besides paper ballots) “voting machines the use of which has been duly authorized by the State law ..” This act began the state requirement for approval of voting devices used within the state. There were no additional federal requirements or conditions imposed on methods for casting and counting of votes between 1899 and 2002, despite the vast changes in technology during that period.

Mechanized data processing with punched cards began with Herman Hollerith, born in Buffalo, New York, in 1860. His first three patents on “improvements in the art of compiling statistics” were issued in 1889. Many of you may know that the use of his punched cards and the machines that he invented for creating and processing them made possible significantly increased data reductions and analyses from the 1890 national census. Eventually, the company he founded became part of International Business Machines Corporation, now known as IBM.

2b. The Late 1950s Through Early 1970s

With the invention of the electronic digital computer, the use of punched cards for data input and output became ubiquitous in the 1950s and 1960s. Persons displaced or even concerned by the increasing use of automation and information technology hoped that the legend on the cards, “Do not fold, spindle or mutilate,” would also apply to them.

At some time before 1960, IBM had invented the “pre-scored” punched card. This is a card in which the locations on the card to be punched are perforated. Then, a person can manually punch out any necessary location with a metal-tipped stylus if the card is placed on a particular type of support. The support was called a “Port-a-punch” unit. It was intended for use by utility company employees who would go to field locations to read meters and manually punch the meter values into pre-scored cards. The purpose of these cards was to eliminate the duplicative task of re-writing the values and thereby prevent transcription errors. There was no initial intention by IBM for pre-scored punched cards to be used en masse by millions of persons.

Joseph P. Harris, a political scientist, who began the movement for effective election administration with research leading to his books of 1929 and 1934, adapted the concept of the Port-a-punch unit using pre-scored punch cards to invent the “Votomatic” vote recorder system in 1962. He achieved modest success selling it to local governments, but he sold the system to IBM in 1965. Thus, hard-copy ballots returned, because they were computer-readable. Criticism of the system by some voters who used it caused IBM to stop selling it in 1969, and to sell the rights to manufacture the system to other companies. IBM’s computing systems were almost always sold to corporations, and any criticisms were handled privately on a company-to-company basis. The public nature of criticism of voting devices was unacceptable to IBM.

A concern about software fraud soon made its appearance. The Los Angeles Times reported on July 8, 1969, that several computer scientists had undertaken an experiment demonstrating that computer programs used to count votes could be manipulated and that the malicious code could be subsequently erased. With this process, it was implied, election results could be altered and no one in authority would know. The stir raised in southern California because of this report did not deter increasing use of computerized voting.

Computer Election Systems, also called CES, formed by former IBM salesmen, was one of the buyers of the rights to the Votomatic system, and was successful in marketing it. By 1972, 30 states had adopted legislation permitting its use and 16 of the nation’s 100 largest cities were employing it. Advantages were that the Votomatic units were inexpensive, small, and light in weight, in comparison to mechanical lever voting machines, which were expensive, large, and heavy.

Furthermore, with the punch-card units, several voters could punch out their ballots in parallel, eliminating waiting lines, a serious problem of lever machines.

Along with the increasing use of computerized voting in the late 1960s, there were well-publicized instances of difficulties in processing of the votes and significant delays in reporting results. In some instances, delays lasted several days before initial results could be obtained. The problems brought by “chads,” the removed insides of punched locations on pre-scored punch-card ballots, were evident in elections in Los Angeles County in 1968 and 1970, and also in Detroit in 1970. Despite the wide publication of the problems at that time, and the continued extensive use of pre-scored punch-card ballots throughout the 1970s, 80s, and 90s, the general public, as well as many politicians, appeared to be astounded by the problem of chads that they suddenly discovered in November 2000.

3. NIST’s Research and Publications in the Period 1974 to 1988

3a. The 1975 Report

In 1971, US Representative William Keating of Cincinnati, Ohio, was disturbed by many reports of difficulties in computerized elections. One of those situations had occurred in his own city, just a few weeks before he rose to speak on the floor of the House of Representatives in late November of that year. At that time, Keating offered an amendment to the bill that became the Federal Election Campaign Act of 1972. The amendment, which was adopted, created the federal government’s Clearinghouse on Election Administration, a five-person office with minimal funding. Keating wanted to establish an organization whose function would be to find the best practices then in place and communicate them to election administrators. As a result of a fortunate series of events that followed, NIST, under its former name, National Bureau of Standards (NBS), got the opportunity to publish, in 1975, the first comprehensive report on the integrity of computerized voting. The report included several recommendations to improve the accuracy and security of the vote-tallying process. One of its conclusions was that:

“The assurance that steps are being taken by election officials to prevent unauthorized computer program alteration or other computer-related manipulations remains, nationwide, a continuing problem for the maintenance of public confidence in the election process.” [p. 4]

This statement would be still be meaningful if issued now in 2007. That is unfortunate, because it indicates that there has been insufficient implementation of measures for security, as well as measures for audit and control, in the 32 years since the report was released.

Many persons have asked me I how became involved in the issue of integrity of computerized elections. I joined NBS in 1969, to work in that part of NBS that is now called the Information Technology Laboratory. The laboratory’s first incarnation was initiated in response to a federal law of 1965 that mandated that NBS establish standards for federal computer systems and consult for other agencies of the federal government on use of computers. In 1973, I was asked to find other-agency consulting opportunities for NBS in my field of expertise. I was surprised to find out, when I joined NBS, that the agency’s work was not fully funded through appropriations and that other-agency projects were a common phenomenon.

Due to my increasing interest in the application of computers in the public sector, I attended meetings of organizations of computer users working for state and local governments and, at one of these meetings, I met Dr. Gary Greenhalgh, at that time the director of the Clearinghouse on Election Administration. Gary told me that the General Accounting Office (GAO, now called the Government Accountability Office), in which the Clearinghouse was then located, had received a letter from a member of Congress from California, asking that a study of computerized voting be undertaken. The letter evidenced concern for the correctness of reported election outcomes due to the widely reported difficulties. Gary and I developed a plan of a one-year project to be carried out by NBS, and the plan was approved by both agencies. With the agreement of Elmer Staats, then Comptroller-General, $100,000 was transferred to NBS in early 1974 from GAO for that purpose.

The output of the project was the report that I authored, called Effective Use of Computing Technology in Vote-Tallying, identified originally as NBSIR (IR for interagency report) 75-687. It was reprinted as Special Publication (SP) 500-30 in 1978 in order to make it available in federal depository libraries.
With the understanding that, in 1975, personal computers did not exist, and that essentially all computerized voting employed hard-copy ballots that were manually punched or marked, five of the findings and conclusions of the report were the following:

(1) To eliminate as many security threats as possible, the least complex operating system that provides the necessary capabilities should be used.

(2) Ballot reconciliation should be undertaken: that is, the sum of the number of ballots used, unused and spoiled in each precinct at the close of polls must equal the number of blank ballots received when the polls were opened.

(3) Recounting of ballots should be undertaken, for example, by “machine recounting on alternate, independently-managed systems.” Furthermore, the closer the totals of the two leading candidates in each contest, the greater the percentage of ballots that should be recounted.

(4) Research needs to be undertaken on the human engineering of voting equipment.

(5) An election systems standards laboratory should be established to set national standards for federal election procedures and for election equipment and systems performance.

The report was widely distributed to election administrators, but its implementation was limited. One person who used the recommendations was Marie Garber, former election administrator of Montgomery County, Maryland, and later Maryland state election administrator. In the latter position, Ms. Garber adopted the recommendation that recounting on an independently managed system should be carried out. As a result, when punch-card ballots from Carroll County were rerun on the computing system of Frederick County following a 1984 election, a discrepancy was revealed. The error was determined to have been caused by the unintentional misuse of a data-entry program in Carroll County. The mistake had caused a wrong outcome in an election for the county school board to be reported. With the replacement of the incorrect program, results on the computers of both counties matched; manual recounting also verified that result.

The current situation with regard to the five identified recommendations of the 1975 report is as follows:

(1) Multi-function operating systems are being used unnecessarily in precinct-located voting equipment. This situation will be further discussed.

(2) Reports from the field show that ballot reconciliation is being done poorly or not at all in some places, resulting in election results that can be reasonably questioned. Requirements for ballot reconciliation need to be instituted where they do not exist. If requirements exist, better implementation needs to be undertaken.

(3) Partial manual recounts are required in few states, but not in all; no variable percentage recount has been implemented in any state, but there is a bill in Congress now that would mandate it nationally.

(4) The enactment of HAVA has made possible research on human factors in voting, but much more needs to be done. Statistical analyses after the 2000 election showed that persons of low educational level could not easily use the pre-scored punch-card voting system.

(5) The development of voluntary standards proceeded at a snail’s pace. The Federal Election Commission (FEC), to which the Clearinghouse was moved in 1975, decided that it had no authority to pursue standards development It requested authority from Congress, and in 1980, Congress legislated that the FEC undertake a feasibility study “with the cooperation and assistance of the National Bureau of Standards.” The study, undertaken by an independent contractor, the late Robert Naegele, reported that “performance standards for voting systems are both needed and feasible.” It was submitted to Congress in 1983. Development of the first set of voluntary standards was begun in 1984, but the standards were not completed and issued until 1990.

I have noted already that the problems raised by the use of pre-scored punch-card ballots were known in 1975. However, GAO representatives had made very clear to NBS that voting technology was not to be disparaged in order not to reduce public confidence in the voting process. Thus, the “findings and conclusions” of the 1975 report said nothing about the failings of pre-scored punch-cards. Nevertheless, on pages 73 and 74 of that report, under the title of “Acceptance Testing of Vote-Tallying Equipment,” the difficulty was made clear by implication. Some of the wording in that section is the following:

“In an election, the [voting] system is to be used by voters of varying abilities. The concept of the voting system must be that it is there to serve the voters, and the system must be geared so that the overwhelming majority of voters, approaching 100%, can use it to record their votes as they intend. ... Thus, the ballot, vote-encoding equipment, and sensor combination should be given an acceptance test ... If too many voters, told beforehand how to operate the system, cannot have their choices recorded correctly, the system must be rejected.” [pp. 73, 74]

The viewpoint of this paragraph was contradicted by the concurring opinion of Chief Justice Rehnquist in the Bush v. Gore decision of December 2000. His opinion stated that, since voters had been told how to operate the system, it was up to the voters to get it right. The court made no allowance for voters who were not literate, or educated, or dexterous, or who had impaired vision.

This section of the 1975 report continues:

“What is being proposed here is a controlled experiment involving the human element in the voting process. Such experiments, if they had been carried out when punch-card systems were first introduced, might have made clear the difficulties caused by hanging chad on ballots, loose design specifications for ballot holders, and card reader jams that plagued such systems.” [p. 74]

There is no evidence that such controlled experiments were ever carried out anywhere. Nevertheless, the FEC’s voluntary standards of 1990 stated that:

“Punching devices ... shall ..ensure that the chad … is completely removed.” [Section 3.2.4.1.2, p. 27]

Florida, and other states, when they adopted the federal voluntary standards, undertook no actions to meet this requirement. This failure of concern for the usability of these voting devices significantly contributed to the embarrassing disaster of the 2000 election.

3b. The 1988 Report

In the decade following the publication of the 1975 NBS report, significant advances in computer technology occurred. Large-scale integration of solid-state logic units was developed, allowing for the use of precinct-located computerized voting equipment. As a result, the use of optically scanned ballots, many tallied in equipment located at precincts, became common. The percentage use of pre-scored punch-card ballots leveled off and began a slow decline.

Direct-recording electronic (DRE) voting equipment started to be used as precinct-located devices became possible. Some of this equipment simply replaced the levers on mechanical voting machines with pushbuttons or micro-switches. Other DREs used electronic terminals to display choices on a succession of screens; again, selections were made through the use of pushbuttons and, in some of the latest implementations, touch-screens. The concept used in lever machines was retained: that is, individual selections for each candidate were summed and stored in an electronic counter. As with lever machines, there was one counter for each candidate. DREs had become important, but their fraction of nationwide use was still less than three percent in 1984. There was no significant public opposition to their employment at that time; after all, they replaced lever machines, and lever machines had never required hard-copy ballots.

In 1985, beginning on July 29, The New York Times published a series of articles on computerized voting. The headline on the first article, which began on Page One, was “Computerized Systems for Voting Seen as Vulnerable to Tampering.” The first article stated:

“The computer program that was used to count more than one-third of the votes cast in the Presidential election last year is very vulnerable to manipulation and fraud, according to expert witnesses in court actions challenging local and Congressional elections in three states ...”

The computer program in question was actually counting the votes on pre-scored punch-card ballots. Consequently, there were ballots available to be recounted manually or on an independently programmed system. This recount possibility to erase doubt was not made clear in the series of articles. The same concern about software fraud directed against DRE voting systems not using ballots would not arise for another 18 years.

The New York Times series was seen by officials of the John and Mary R. Markle Foundation, a private, non-profit foundation headquartered in New York City. The foundation decided to fund a study of the problem. It selected NBS as its organization of choice to carry out the study, based on the quality of the 1975 report. In 1986, NBS accepted the funds, about $200,000, to carry out the research and produce a report. The receipt of a grant by a federal agency from a private foundation is perhaps unusual,but both NBS and I were gratified by the confidence placed in us.

The result of the work was my 1988 report entitled Accuracy, Integrity, and Security in Computerized Vote-Tallying, identified as NBS SP 500-158. A significant recommendation of the report was that:

“The use of pre-scored punch-cards should be ended .... It is generally not possible to exactly duplicate a count obtained on pre-scored punch-cards, given the inherent physical characteristics of these ballots and the variability in the ballot-punching performance of real voters.” [p. 5]

This recommendation could be made because NBS was not constrained by Markle to refrain from criticizing technology (as we were by the GAO). Despite the wide distribution of the 1988 report, this recommendation was generally not acted on by election administrators prior to 2001. In two cases, however, states acted when they had their own failures. In 1993, Wisconsin told its towns not to purchase any more systems using pre-scored punch cards (although those then in place could continue to be used) and, in 1997, Massachusetts totally revoked its approval for their use. The recommendation of the 1988 report was remembered after the Florida fiasco of 2000, and then it was reported in the media, for example, in The Miami Herald.

An important concern of the 1988 report is the design of DRE voting devices. With the use of these units, there is no independent audit trail. There is a lack of independent verification that the choices selected by the voter have, in fact, been recorded correctly. The report states:

“The fact that the voter can see his or her choices on a display, or even receives a printout of the choices made, does not prove that those were the choices actually recorded in the machine to be summarized for generating the results of the election.” [p. 41]

The report goes on to state that the correct recording of voter’s choices in DRE machines must be bolstered with extensive pre-election and post-election review and testing of the logic of the machine. Two other recommendations were that (1) undervotes, i.e., failures to vote in any contest, be positively recorded, rather than be determined by inference and (2) that “voter-choice sets,” the machine’s record of all choices of a voter, be retained in a permanent storage unit. Neither of these actions could be carried out with lever machines. The recommendation on “voter-choice sets” was accepted and included in the FEC’s 1990 voluntary standards. In that document, they were re-named “electronic ballot images” or EBIs.

Additionally, a major recommendation of the report is certification, which implies state approval. The report makes clear that:

“Products to be certified should include all vote-tallying software and all software to be mounted together with vote-tallying software.” [pp. 3, 4]

The issue of testing of other software, besides the vote-tallying software, remains contentious, and it is discussed further, as a current problem.

4. The 2000 Florida Election and HAVA

4a. Some General Provisions of HAVA

HAVA was adopted in October 2002 by Congress as a result of the November 2000 Florida election disaster. For more complete information about that election, you may read Chapter One of my book, The History and Politics of Voting Technology: In Quest of Integrity and Public Confidence. (Don’t wait for the movie.) Many references, also available for your review, are given in the book. However, most of the numerical data, which showed that Gore would have won if a complete recount of the entire state had been undertaken, are not available in any other commercially published book, to the best of my knowledge. These data, developed by a large consortium of media companies, were provided to me by Dan Keating of The Washington Post Keating was one of the technical leaders of the study. Keating had made much of the data available in a presentation to the American Political Science Association at their annual meeting in Boston in 2002.
HAVA established the bi-partisan, four-member Election Assistance Commission (EAC). A mandatory requirement of HAVA is that persons with disabilities, for example, vision-impaired individuals, must be able to use voting equipment at polling stations without assistance. This mandate expands democracy, but has made the selection and implementation of voting equipment a more complicated process. Another requirement of HAVA is that voting equipment must provide an audit trail, but mechanical voting machines cannot do that. Thus, these machines should no longer be used, but they were still being used throughout New York State in 2006.

HAVA included an important provision on voter registration technology. The law stated that each state is to implement a centralized interactive computerized statewide voter registration list defined, maintained, and administered at the state level. There is evidence that some states are carrying out this requirement in a manner not fully consistent with the mandate for administration in a “top-down” manner, with full control at the state level. More will be said on this subject.
Several issues covered in HAVA do not result in requirements imposed on any state, if a state decides not to accept its share of the $3 billion appropriated under the act. Here, we see the strong resistance of states to federal control, some more than others. For example, Idaho did not accept any funds to update its voting equipment. Over half of the voters of that state continue to employ, by their choice, the pre-scored punch-card voting system that created havoc in Florida in 2000. They have, apparently, total disregard for this system’s defects. Adults can continue to act like obstinate children when there is no penalty for immaturity.

4b. HAVA Provisions Relevant to NIST

The EAC is not a regulatory agency, but it is empowered to undertake research, to approve new or revised sets of Voluntary Voting System Guidelines (VVSG), and to approve the accreditation of Voting System Testing Laboratories (VSTLs). The guidelines, which may be made mandatory within any state by the state’s own volition, are developed by the Technical Guidelines Development Committee (TGDC), established under HAVA. NIST’s Information Technology Laboratory provides administrative and technical support to the committee. The accreditation of VSTLs is recommended by NIST’s National Voluntary Laboratory Accreditation Program (NVLAP) following their evaluations.

5. Some Current Problems of Technology and Policy, and Recommendations

5a. Software Correctness

As I have pointed out, the possibility of fraud by manipulated software has been raised for as nearly as long as computerized voting equipment has been used. As a result of the 1969 article in The Los Angeles Times, the state of California decreed, soon after, that a one-percent manual recount of each election using computer-readable ballots must be carried out at no cost to any candidate. In the 1975 NBS report, I developed a mathematical formulation that demonstrated that, for a particular confidence level, the percent recount should be greater as the totals of the two leading candidates become closer. I concluded that the one-percent recount was insufficient, and that possibly, a 5 to 10 percent recount might be needed in a very close contest if a very high confidence level were demanded and no candidate had paid for a complete recount.

With the use of DRE voting equipment, there are no ballots to recount. The electronic ballot images, which may be printed after the polls are closed, are considered under HAVA to meet the requirements for an audit trail. The percent use of DREs continued to increase from their first implementations up to 2000, when they were used by 13 percent of the voting population nationwide. None were used in Florida in that year. A backlash against DREs began only in 2003, after Florida acquired some of them and a lawsuit was filed against their use. Another lawsuit with a similar purpose was filed in Maryland also in 2003, after Maryland’s acquisition of DREs. The lawsuits have been unsuccessful. In addition, David Dill, a professor of computer science at Stanford University, publicly asked his local county in California not to procure DREs. He began an organization called the Verified Voting Foundation and established a website. Several thousand individuals, including many computer scientists, have emailed their support of his efforts.

Dill and other computer scientists have pointed out that, theoretically, proof of correctness of all but the simplest of computer programs is impossible. This is certainly true, yet computer programs are widely used in situations where their incorrectness would be life-threatening, such as in control of aircraft stability in flight. In December 2003, Professor Dill traveled on an aircraft from his home on the West Coast to a symposium at NIST here on the East Coast to personally speak of his concern about software incorrectness. The key to integrity is thorough testing as well as protection of the final code against unapproved change, as certainly Professor Dill is aware. One of Dill’s antagonists is Professor Brit Williams, the state of Georgia’s election advisor for technical matters, who spoke at the same NIST symposium. Williams and another Georgia official testified to Congress in July 2004 that:

“The conjecture that using current technology, we are unable to make such a simple system [as a voting machine] secure and accurate is contradicted by the facts of our daily existence.”

Furthermore, Conny B. McCormack, the election administrator of Los Angeles County, the nation’s most populous, told Congress in June 2005 that:

“The fact is that existing DRE systems ... have the proven track record of doing the best job of all available voting systems.”

Many election administrators want to retain the use of DREs without paper ballots because of their advantages. These are, (1) with DREs, paper ballots need not be distributed to precincts and then collected and their use accounted for, (2) without hard-copy ballots, the question of “intent of the voter” in the analysis of a non-standard mark (or of a hanging chad) never arises, so that there cannot be disputes over that subject and (3) without ballots, the different languages required to be presented on ballots due to the requirements of the 1975 amendments to the Voting Rights Act may be provided by software and not by printing.

There are, of course, disadvantages to the use of DREs, and these include (1) the inability to provide an independent audit trail, (2) the likelihood of the formation of waiting lines of voters if an insufficient number of machines have been provided, and (3) the sense of incompleteness felt by the voter because the end of the voting activity does not generate a piece of paper, nor is it accompanied by the significant mechanical action of opening the curtain of a mechanical lever machine and seeing the levers return to their neutral locations. This sense of incompleteness may be a significant, if unarticulated, source of the public’s dissatisfaction.

A number of computer scientists have been very active in identifying security flaws in the operation of DRE voting equipment. Professor Ed Felten of Princeton noticed that access to the circuit board storing the program could be achieved in a particular vendor’s machine because a only simple lock was being used to enclose its compartment. With this access, the circuit board could be removed and replaced with another storing false code. Professor Avi Rubin of Johns Hopkins claimed that the “smart card” given a voter that allows the voter to vote on a particular vendor’s machine could be duplicated, allowing the voter to vote more than once.

The news media have been very receptive to computer scientists making these claims, giving them wide publicity. However, the fact that these flaws would be very hard to exploit without massive collusion and that the vendors have instigated corrective measures are rarely, if ever, reported. A result of this situation is that public confidence in DRE machines and in reported results of elections in general has been lowered, although no evidence whatsoever has been presented that any person has actually tried, much less succeeded, in manipulating any DRE machine during an election.

The situation that occurred in Sarasota County, Florida, in November 2006, has added to the lack of confidence in DRE machines. The two opposing candidates for a seat in the House of Representatives together received 18,000 fewer votes than candidates for the contest just above it and just below it on the ballot. A protest that this unusual situation was caused by a flaw in the software of the voting machines was instigated by the loser of the election. The same type of drop-off did not occur in Sarasota County for voters casting absentee ballots, nor did it occur in neighboring counties voting in the same contest but using different machines.

The loser went to court and asked a judge to permit her experts to examine the machines, with the understanding that the experts would be willing to sign a non-disclosure agreement to protect the trade secrecy of the vendor’s software. The judge decided not to permit the examination, in agreement with the views of the vendor of the equipment and of the State of Florida. After that decision, the state convened a committee, including its own experts but not the loser’s experts, to examine the machines’ hardware and software. The state’s experts, who were a highly professional group, said that there was nothing wrong with the machines. A reasonable supposition is that the problem was one of human factors, specifically the manner of presentation of the contest on the screen of each machine.

However, the exclusion of the loser’s experts from the examination was disgraceful, in my opinion, and has allowed continuing doubts to persist.
What is unfortunate, also, is that computer scientists who have disparaged the ability of testing to assure software correctness have made no effort to assist NVLAP in its evaluation of the testing laboratories. I challenge these individuals to answer the following questions:
Under the claim that no significant computer program can be proven correct, should the certification of software using accredited VSTLs be abolished? If yes, detail the methods that should replace it to assure public confidence in the announced results. If no, answer the following questions:

(1) Are the procedures used by the VSTLs for software testing effective and, if not, are there better procedures that should be used?

(2) Are administrative controls in place that prevent conflict-of-interest situations within VSTLs? If not, propose improvements.

(3) Is there effective oversight of VSTL activities? If not, propose improvements.

(4) How should the work of NVLAP in the area of accreditation of VSTLs be otherwise improved?

I propose that an advisory committee of computer scientists be established to assist NVLAP and the TGDC in the evaluation of the procedures used by the VSTLs.

5b. Restricting the Commercial Off-The-Shelf (COTS) Testing Exemption

The software that is to be tested that I have just discussed may be included in the category of “application” software, that is, the software that, for a touch-screen DRE device, actually causes the presentation of the candidates on the screen to occur, records each voter’s selections made via the touch-screen, and sums the votes for each candidate in each contest. For a different type of voting machine, one that tallies computer-readable ballots, the “application” software need not be concerned with a complex screen presentation, but still must record each voter’s selection sensed by an optical scanner and sum the votes for each candidate.
Some electronic voting machines that are designed around a computer provide for this “application” software to execute under control of an “operating system” or management program. A commercially provided operating system is included in the category of COTS software. Professor Avi Rubin has written, in his 2006 book Brave New Ballot, that:

“DRE machines [are] essentially personal computers running a special application.” [p. 13]

Three pages later, he states that:

“At its heart, an e-voting machine is a computer running a version of the Windows operating system.” [p. 16]

As written without qualification, and Rubin inserts none, these two statements are not wholly correct. In some cases, such as the voting machines made by Diebold, whose software was examined by Rubin, the assertions are true. With another vendor’s voting machines, the statements may not be true. There is no inherent requirement that the application program of a precinct-located, single-purpose voting machine must execute under control of a multi-function operating system.

Here we are faced with a situation which I believe has arisen due to the narrow education and limited experiences of many computer scientists. These professionals are used to analyzing computers with multi-function operating systems that support many application programs at the same time. These computers may be, simultaneously, connected to the Internet or other communications facilities through which they receive and send messages. The likelihood that there are “bugs” in the software of such computers is very high, because of the large size and complex nature of the programming. Software testing, of the type undertaken by the VSTLs, may not find all the bugs.

Some computers are used for a totally different purpose that is apparently unfamiliar to many computer scientists. These computers are used for a single-function, real-time application that has no external connections except those from the on-going process that it is controlling. There is no need for a multi-function operating system, because there are no separate, independent programs running and no unanticipated interrupts. Examples of situations in which single-function, real-time computers may be found, besides control of in-flight stability of an aircraft, are control of the process of combustion in an advanced automobile, control of parameters of an ongoing chemical process such as oil refining, control of the firing of a missile for military purposes, and control of parameters of medical equipment during a life-threatening operation on a human. In some of these applications, the computer is actually embedded in the process hardware and is not seen by the human operators. The programs of these computers are likely to be designed by engineers with the appropriate specialty who are very knowledgeable of the process being controlled and who also understand both the design of computers on the level of single machine instructions and the design of data input and output devices.

It is my view that precinct-located voting devices are engaged in an activity similar to a single-function, real-time process-control application. The application has no requirement for simultaneous execution of other programs or of communications connections. Therefore, the use of a multi-function operating system is totally unnecessary.

I have raised this issue because COTS software has been given an exemption from testing by the VSTLs in the most recent VVSG and in the next planned release. The history of this exemption is that, when the first voluntary standards were being developed in the 1980s, a large percentage of vote-processing of computer-readable ballots was carried out, not in the precincts, but centrally, by mainframe computers. It would have been unreasonable, if not impossible, to test the non-applications programs of these computers, including the operating systems, utility programs for peripheral support, and compilers converting source code into object code. With the development of precinct-located, single-purpose computers as voting devices, the benefit to the voting public of the continuation of this COTS exemption has not been shown.

Nevertheless, the likelihood that malicious code could be introduced into vote-counting software from any of the exempted computer programs is just as great as the likelihood of fraud through means of the physical security flaws that have been widely publicized recently.

Therefore, I propose again, as I did in a presentation to the TGDC in September 2004, that all software in precinct-located vote-processing computers be subject to testing by the VSTLs. The cost of testing imposed by the VSTLs on the vendors should increase with the size of the program being tested, as a penalty for the inclusion of operating system functionality that adds complexity but is unnecessary and unused.

5c. Independent Verification—Paper-Ballot Systems

As has been noted, the 1975 NBS report stated that an aid to the audit of vote-tallying calculations would be machine-recounting of ballots on an alternate, independently managed system. When ballots are used that are both human-readable and computer-readable, recounting by hand or on an independently managed system is possible. The percent of ballots recounted without cost to any candidate should increase as the vote totals become closer.

5d. Independent Verification—DRE Systems

With the increasing use of DRE systems, in which there is no possibility of independent verification, concerned citizens began to demand some form of greater assurance. Professor Dill’s website, established in 2003, contained the following:

“It is ... crucial that voting equipment provide a voter-verifiable audit trail, by which we mean a permanent record of each vote that can be checked for accuracy by the voter before the vote is submitted ...”

Nevada was the first state to mandate a “paper trail” for DRE systems and California soon followed in 2004. The implementation in these states is often in a very user-unfriendly form, but it is inexpensive to accomplish. A printer is added to each DRE. It is put to use only after the voter completes voting all contests and indicates that fact on the screen. Then, a paper record of votes cast is printed so that the voter can view it under a transparent cover. The voter cannot touch the printout.

With this paper audit trail, the voter has the opportunity to scan the printout and compare the printed record against the content of the summary screen. The summary screen is the electronic record of what the voter has selected. The printout and summary screen must match. If they match, the voter so indicates by clicking on the screen, and the printed record is retained as the ballot of record. If the printout and screen do not match, a poll-worker must be called over and shown the discrepancy. It is evidence of computer program error. Note that if a difference exists, the voter loses the right to a secret ballot by demonstrating the problem. This is where deep thinking in one narrow dimension has taken us. Effective election administration is a multi-dimensional issue.

Another problem with printing of the hard-copy after all votes are cast, is that very few voters actually make the effort to completely compare the results of all contests shown in both places. This was demonstrated by a detailed review by highly qualified persons of a videotape of an actual election in Nevada. (The taping was careful not to show the actual votes cast by the voters.) Most voters do not make the full comparison because (1) it takes extra time, (2) once they have finished voting, they want to leave, (3) the comparison is difficult because of the different formats of the printout and the screen, or (4) they have confidence that the screen result will be properly recorded.

The human-factor difficulties in printout review have been noted by authoritative researchers such as Ted Selker of MIT and Don Norris of University of Maryland at Baltimore County. A printout that is not reviewed by a voter does not serve the purpose for which it was intended: to assure that the result actually recorded is the same result as that shown on the DRE summary screen. It is a waste of time and resources to create such printouts and have them serve as the ballots of record.

Furthermore, some printers that were supposed to produce ballots of record have failed during recent elections, leaving ambiguity of results in their wake.
It appears, then, that those well-meaning persons who have demanded a “paper trail” have not made the effort to distinguish between a paper trail that carries out its intended function and one that does not. A bill coursing its way through the Congress now, called H.R. 811 and similarly, a companion bill in the Senate, S. 1487, fails to make this distinction. This situation appears to me to be reminiscent of the adoption of pre-scored punch-cards, which were very cheap but had serious human-factor problems that affected the outcomes of elections in which they were used. With the use of pre-scored punch-cards, voters were told to be sure to remove the hanging chads, but most voters did not do that over the many years that these ballots were in use. H.R. 811, similarly, calls for administrators to inform voters to be sure to compare the printout with the screen, which only proves that the authors of the bill know very well already that voters aren’t doing it. Passing a law that only urges compliance will not change the human-factor parameters of the situation.

There are voting devices producing printouts which do not have the defects of the “post-vote” or “receipt” printout that I have just described. In this different type of printout, the printing of the voter’s selection occurs almost immediately (not more than a few seconds) following each candidate choice by screen or pushbutton. Since the voter can see the corresponding selection on the printout essentially contemporaneously with each electronic selection, there is no extra time required to compare the screen with the printout after voting has been completed. This system overcomes the human-factor difficulties of the “post-vote” printout. One such device is being marketed by Liberty Election Systems for use in New York State.

It is my personal view that H.R. 811 should not be enacted unless it disallows, as not responsive to the requirement for voter-verification, printouts from DRE voting equipment which are not produced for voter review until all votes are cast.

Another unfortunate aspect of H.R. 811 is that it requires an audit trail in paper. The needed concept is “independent verification,” not paper technology. As many business and government operations have eliminated paper, such as salary checks and stock certificates, it seems strange and retrograde to me that demands are being made to require the use of paper in elections. It is possible not to use paper and still carry out independent verification; bills submitted in Congress ought to allow for forward-thinking solutions, and not just respond with knee-jerk answers. The passage of HR. 811, as currently written, would restrict voting technology to the concepts of 50 years ago.

One possible solution without paper, and there are others, is the employment of two parallel and assured identical calculations of the summaries of all voters’ selections on the screens of a DRE machine. Each tested program performing the calculations would be written by an independent vendor. The first set of calculations would be contained in a DRE with the screen on which the voter makes his or her selections. The summary screen, when completed by each voter, would be transmitted as a streaming video to the second device in which the parallel calculations would be carried out. The results produced by both sets of calculations should match. The streaming videos could also be retained and stored independently in a third device as an independent check on both sets of calculations. The retained summary screens would constitute the official ballot-of-record for each voter and could be printed out following the close of polls.
The idea of a second set of parallel calculations receiving the voter’s selections as a streaming video has been proposed by a company named Democracy Systems of Ormond Beach, Florida. I have no financial interest in this company, or in Liberty Election Systems, or for that matter, in any company manufacturing or selling election equipment.

5e. Voter Registration Integrity

A major aspect of election administration is the maintenance of a list of qualified registrants. As early as 1837, when registration was first considered for Philadelphia, supporters said that the law would significantly reduce fraud, while opponents said that it would discriminate against the poor. These are the essentially the same arguments that are used today, 180 years later, in debates concerning the adoption of voter registration regulations. The lack of a national and personal identity card in the US significantly increases the difficulty of achieving a correct list of registrants. Mexico has such a system and does a better job with voter registration than the US.

The capability for a person to make an application for voter registration at a state motor vehicle agency began in Michigan in 1975. Several other states had adopted the idea by 1993, the year of passage of the federal National Voter Registration Act (NVRA). That act, which mandated that registration applications must be made available at certain public facilities including motor vehicle agencies, was required to be implemented for federal elections by all states except North Dakota, which had no voter registration requirement, and several other states that allowed election-day registration. A most important provision of NVRA was that:

“Any state program ... shall not result in the removal of the name of any person from the official list of voters ... by reason of the person’s failure to vote.”

Many states had used purging from their lists for failure to vote as the primary way that they kept their lists current. A purge after failure to vote in two successive federal general elections (including any election in-between) is not unreasonable, in my opinion. Under NVRA, without a positive indication that a registrant has moved, died, has been convicted of a felony or has been declared mentally incompetent, a removal from the official list is permitted under only limited conditions. These circumstances involve receipt of an “undeliverable” notice by the Postal Service concerning correspondence, a confirming failure to return a notice from the election administration, as well as failure to vote in two successive federal general elections.

It is not surprising, therefore, that many election administrations have registration lists that contain large numbers of entries which, if the facts were known, would have been eliminated. Accurate maintenance of voter registration lists is a very large data communications problem in which many of the necessary links have not been adequately implemented. The existence of large numbers of incorrect entries is an invitation to fraud. It is the modern equivalent of the effort to “vote the cemetery,” a ploy in years gone by. The possibility lowers public confidence in announced election outcomes, whether or not the fraud has or has not actually been perpetrated.

The likelihood of registration fraud, however small, has caused a number of states to require a photo identification document from each voter. Again, advocates for the poor and elderly have opposed these efforts on the basis that it would disadvantage that particular part of the population. If the state demands that a voter pay for such a document, the effort may be considered a poll tax, outlawed under a Constitutional amendment and by the US Supreme Court. In order to overcome this objection, states requiring photo IDs have exempted the poor from the payment requirement, but a person who wants the exemption must make an application and either reveal the personal financial situation or sign, under penalty of perjury, that he or she has less than a certain income.

The problem of inadequate administration was seen in Florida in the late 1990s and in 2000. An election in 1997 for mayor of Miami had been thrown out because of registration fraud. As a result, the Florida legislature contracted with a private company in 1998 to purge the voter rolls of ineligible voters. The purge was badly done and generated much publicity claiming that the purpose of the effort was to reduce the number of minority voters on the registration lists. In the 2000 general election, said a book written by staff of The Miami Herald,

“Thousand of Floridians cast illegal votes on November 7; they swore they were eligible to vote, but they were not. The ballots, all of which were counted, came from unregistered voters, ineligible felons, and a handful of senior citizens who voted absentee first, then voted again at their local precincts after swearing that they hadn’t voted before. ..”

Disconnects between the motor vehicle agencies and the election administrations were experienced by voters in that election. In a hearing held by the US Commission on Civil Rights after the election, a Florida poll-worker reported:

“There were people who had registered to vote through [the 1993 National Voter Registration Act] and somehow their registration was not transmitted to the supervisor of elections office. I saw that with married couples in my own precinct. One person would be registered to vote; the other person would not ...”

A serious omission in HAVA is the lack of consideration of the problem of voters moving between states. Interstate moves by residents are significant. Nevertheless, there is no requirement in the act for each state to report the old address of a newly registered voter to the state from which the voter came. This problem is understood by the Election Assistance Commission. The commission has contracted with the Computer Science and Telecom-munications Board (CSTB) of the National Academy of Sciences to consider the issue, and the CSTB held a workshop this past August 6.

While some states have arranged to transmit changes of address among themselves, the process is by no means universal among the 50 states, the District of Columbia, and US territories. Furthermore, under HAVA, each state is to assign a unique identifier to each legally registered voter. There is apparently no effort underway among the states to coordinate the algorithms for determining the identifiers.

My recommendation is that the Election Assistance Commission be given greater authority to set mandatory national standards for collection of data by each state for the purposes of voter registration, for standards to identify voters, and for the interchange of voter registration data among the states, the district, and the territories There already exists a standard electronic messaging system widely used in commerce that is independent of computer makes and models. NIST had a hand in developing the American national and international standards for this system, called electronic data interchange or EDI. This scheme could be adapted easily for data interchange among states.

In its recent report to Congress about the National Voter Registration Act, the EAC described the difficulty of obtaining basic voter registration statistics that are consistent within states and comparable among states. The EAC said that “The missing data in this report demonstrates the inability of many states and jurisdictions to provide basic voter registration information and data.”

It is a fact that the making of public policy requires the collection and evaluation of much data about the current situation. Implemented changes can only be evaluated by continuing data collection. More data is needed for policy-making in election administration, and its collection by states and localities should be mandated.

5f. Election Administration Must Be Non-Partisan

The final issue that I will present today is partisanship in election administration. Many of you will recall that, in Florida in 2000, that state’s Secretary of State, the person responsible for policy-making for elections, had been elected to office as member of a political party. She served, at the same time, as the co-chair of the state’s political campaign for a leading presidential candidate. Decisions that she made during the post-election legal battle were highly partisan. In Ohio, in 2004, that state’s Secretary of State, similarly, served as co-chair of the state’s campaign for the same presidential candidate. In addition, this very person ran on a partisan ticket for the office of governor of Ohio in 2006 without resigning his position. In effect, he oversaw his own election. He was defeated.

Here is another example. On May 9, 2007, The Denver Post newspaper editorialized as follows:

“In his campaign to become Colorado’s secretary of state, Mike Coffman pledged to uphold the integrity of an office that had been diminished by partisan rule-making under previous occupants. ... It was the right thing to say and it’s the right thing to do. Unfortunately, he hasn’t practiced what he preached. Coffman misfired by hiring a political activist ... as the state’s elections technology manager. ...”

We permit, in this country, highly partisan political speech throughout our extensive election campaigns; we wouldn’t want it any other way. Election administration, however, is a different matter. In my opinion, it ought to be scrupulously non-partisan or, at least have balanced partisanship. The federal government, with its capability under the Constitution to change the “manner” of conducting federal elections, ought to correct this problem.

One possible solution is, by federal law, to require each state to manage federal elections with a bi-partisan commission that is appointed in a manner similar to the appointment process of the federal Election Assistance Commission. Each state commission would then name a non-partisan executive director to administer the state’s elections.

6. Summary

6a. Some general conclusions

The shared responsibility of the federal government and the states in the administration of federal elections creates a diversity that is difficult to manage. The nature of our federalism allows differences in “equal protection” in election administration if accomplished by different states. There was a crisis in the late 19th century due to extensive fraud that was solved essentially by state action, but it took many years for all states, acting individually, to implement a solution. The difficulties in adopting computers to elections in the late 20th century was again handled by the states themselves, with only minor assistance from the federal government. NBS was a part of this small effort, producing two reports with no force of law that provided useful recommendations, but little implementation by administrators. The federal government exercised no real leadership until the crisis of the Florida fiasco of 2000 forced action. Even now, the ability of the federal government to act through the EAC is limited. Furthermore, policy-making in election administration requires multi-disciplinary thinking, a commodity not easily achieved. In addition, much of the data needed to justify policy actions are not available, because the mandates to collect such data do not exist.

6b. Recommendations to the TGDC and NIST

(1) On software correctness, public lack of confidence would be considerably eased if the computer science community would respond to my challenge: either to demand the end to software testing and provide generally acceptable alternative techniques, or to serve to evaluate the testing methods being used by the VSTLs and to propose improvements if needed. An advisory committee should be established by TGDC to enable this effort to be undertaken.

(2) The testing exemption granted to COTS software should be eliminated for precinct-located voting equipment. There is no justification for the use of multi-function operating systems in this application. The price of software testing should increase with the size of the software.

6c. Recommendations to Congress on legislation

(1) On a DRE machine, a useless paper trail is one which is not made available until all votes are cast. It does not serve the function of voter-verification since few voters review it. Federal legislation should not permit its use for independent verification. A printout in which each choice is printed and seen as soon as selected will satisfy the requirement.

(2) Independent verification of DRE results without paper is possible, for example, with the parallel computation of results by independently prepared computer programs and the electronic retention of the summary screen of each voter. Federal legislation should not restrict technology to mid-20th century solutions.

(3) Integrity of voter registration lists will be improved by EAC efforts to impose standards for voter identification and for interstate data interchange of voter changes in residence. If the EAC does not have the authority to undertake this activity, Congress should give it the power.

(4) Partisanship in election administration must be ended, most urgently at the highest levels. Congress should mandate a bi-partisan commission in each state to set policy for federal elections. An executive director, hired on merit only, should report to each commission to administer federal elections throughout the state.

Thank you for your attention.

Return to the NIST HAVA Page