Not Quite a Blog

On October 13, Aaron Burstein and I sent a letter to the EAC's Director of Voting System Testing and Certification, Brian Hancock, signed by many of the California Top-To-Bottom Review (TTBR) investigators ("CA TTBR Investigators Send Letter to the EAC"). The letter showed that the EAC had approved a voting system test plan after a testing laboratory misunderstood the results of the California TTBR.

On October 21, The EAC replied. Today, Aaron and I responded (on behalf of ourselves).

The EAC has done a great service by making the test plan and test report available, but the substance of the documents falls short of assuring us that iBeta’s tests covered the TTBR’s findings. Taken together, these documents do not demonstrate how iBeta’s tests covered the issues raised by the TTBR.

(Cross-posted at Freedom to Tinker.)

Short version: we now have gobs of voting system ITA reports, publicly available and hosted by the NSF ACCURATE e-voting center.

Long version: Before the Election Assistance Commission (EAC) took over the testing and certification of voting systems under the Help America Vote Act (HAVA), this critical function was performed by volunteers. The National Association of State Election Directors (NASED) recognized a need for voting system testing and partnered with the Federal Election Commission (FEC) to establish a qualification program that would test systems as having met or exceeded the requirements of the 1990 and 2002 Voting System Standards.^*

However, as I've lamented many, many times over the years, the input, output and intermediate work product of the NASED testing regime were completely secret, due to proprietary concerns on behalf of the manufacturers. Once a system completed testing, members of the public could see that an entry was made in a publicly-available spreadsheet listing the tested components and a NASED qualification number for the system. But the public was permitted no other insight into the NASED qualification regime.

Researchers were convinced from what evidence was available that the quality of the testing was highly inadequate and that the expertise didn't exist within either the testing laboratories to perform adequate testing or the NASED technical committee to competently review the ultimate test reports submitted by the laboratories (called Independent Testing Authorities (ITA)). Naturally, when reports of problems started to crop-up, like the various Hursti vulnerabilities with Diebold memory cards, the NASED system scrambled to figure out what went wrong.

I know have more moderate views with respect to the NASED regime: sure, it was pretty bad and a lot of serious vulnerabilities slipped through the cracks, but I'm not yet convinced that just having the right people or a different process in place would have resulted in fewer problems in the field. To have fixed the NASED system would have required improvements on all fronts: the technology, the testing paradigms, the people involved and the testing and certification process.

The EAC has since taken over testing and certification. Their process is notable in its much higher level of openness and accountability; the test plans are published (previously claimed as proprietary by the testing labs), the test reports are published (previously claimed as proprietary by the vendors) and the process is specified in detail with a program manual, a laboratory manual, notices of clarification, etc.

This is all great and it helps to increase the transparency of the EAC certification program. But, what about the past? What about the testing that NASED did? Well, we don't know much about it for a number of reasons, chief among them that we never saw any of the materials mentioned above that are now available in the new EAC system.

Through a fortunate FOIA request made of the EAC on behalf of election sleuth Susan Greenhalgh, we now have available a slew of ITA reports from one of the ITAs, Ciber.

The reports are available at the following location (hosted by our NSF ACCURATE e-voting center):

http://accurate-voting.org/docs/ita-reports/

These reports cover the Software ITA testing performed by the ITA Ciber for the following voting systems:

• Automark AIMS 1.0.9
• Diebold GEMS 1.18.19
• Diebold GEMS 1.18.22
• Diebold GEMS 1.18.24
• Diebold AccuVote-TSx Model D
• Diebold AccuVote-TSx Model D w/ AccuView Printer
• Diebold Assure 1.0
• Diebold Assure 1.1
• Diebold Election Media Processor 4.6.2
• Diebold Optical Scan Accumulator Adapter
• Hart System 4.0
• Hart System 4.1
• Hart System 6.0
• Hart System 6.2
• Hart System 6.2.1

I'll be looking at these in my leisure over coming weeks and pointing out interesting features of these reports and the associated correspondence included in the FOIA production.

^*The distinction between certification and qualification, although vague, appears to be that under the NASED system, states did the ultimate certification of a voting system for fitness in future elections.

California Secretary of State Debra Bowen has sent a letter to Chair Gineen Beach of the US Election Assistance Commission (EAC) outlining three proposals that she thinks will markedly improve the integrity of voting systems in the country.

I've put a copy of Bowen's letter here (87kB PDF).

Bowen's three proposals are:

• Vulnerability Reporting -- The EAC should require that vendors disclose vulnerabilities, flaws, problems, etc. to the EAC as the system certification authority and to all the state election directors that use the affected equipment.
• Uniform Incident Reporting -- The EAC should create and adopt procedures that jurisdictions can follow to collect and report data about incidents they experience with their voting systems.
• Voting System Performance Measurement -- As part of the Election Day Survey, the EAC should systematically collect data from election officials about how voting systems perform during general elections.

In my opinion, each of these would be a welcome move for the EAC.

These proposals would put into place a number of essential missing elements of administering computerized elections equipment. First, for the users of these systems, election officials, it can be extremely frustrating and debilitating if they suspect that some voting system flaw is responsible for problems they're experiencing. Often, when errors arise, contingency planning requires detailed knowledge about specific details of a voting system flaw. Without knowing as much as possible about the problem they're facing, election officials can exacerbate the problem. At best, not knowing about a potential flaw can do what Bowen describes: doom the election official, and others with the same equipment, to repeatedly encounter the flaw in subsequent elections. Of course, vendors are the most likely to have useful information on a given flaw, and they should be required to report this information to both the EAC and election officials.

Often the most information we have about voting system incidents come from reports from local journalists. These reporters don't tend to cover high technology too often; their reports are often incomplete and in many cases simply and obviously incorrect. Having a standardized set of elements that an election official can collect and report about voting system incidents will help to ensure that the data comes directly from those experiencing a given problem. The EAC should design such procedures and then a system for collecting and reporting these issues to other election officials and the public.

Finally, many of us were disappointed to learn that the 2008 Election Day survey would not include questions about voting system performance. Election Day is a unique and hard-to-replicate event where very little systematic data is collected about voting machine performance. The OurVoteLive and MyVote1 efforts go a long way towards actionable, qualitative data that can help to increase enfranchisement. However, self-reported data from the operators of the machinery of our democracy would be a gold mine in terms of identifying and examining trends in how this machinery performs, both good and bad.

I know a number of people, including Susannah Goodman at Common Cause as well as John Gideon and Ellen Theisen of VotersUnite!, who have been championing one or another of these proposals in their advocacy. The fact that Debra Bowen has penned this letter is a testament to the reason behind their efforts.

Aaron Burstein (UC Berkeley Law) and I just finished a large regulatory filing that will be of interest to any of you election technology geeks out there. Enjoy and be sure to let us know if you have feedback (there's another round of comments sometime in the future on a revised draft).

ACCURATE Submits Comments on the Voluntary Voting System Guidelines

5 May 2008

A Center for Correct, Usable, Reliable, Auditable and Transparent Elections (ACCURATE) submitted public comment today to the U.S. Election Assistance Commission on their draft Voluntary Voting System Guidelines (VVSG), urging the Commission to adopt certain key features fo the draft. The VVSG provides a national certification framework for U.S. voting systems against which 40 states require their voting systems to be certified.

ACCURATE's comments laud the new draft as a groundbreaking and badly needed overhaul of our national voting system standards while making constructive suggestions for further development.

The most significant element of the draft VVSG is the requirement for software independence, which would require voting systems to be designed so that undetected flaws in the voting system software cannot cause changes in the vote count. ACCURATE fully supports requiring software independence as the backbone of a robust and comprehensive next-generation voting system certification regime.

The commentary goes on to emphasize the importance of welcome features of the draft: adversarial vulnerability testing, volume testing, the new framework for usability and accessibility testing and comprehensive voting system documentation requirements. The comment closes by pointing out areas of the VVSG that will require increased institutional support outside of the VVSG process, including the crucial innovation class and a closed loop for incident reporting and feedback.

ACCURATE plans to participate further as the draft VVSG is modified and extended.