Not Quite a Blog

Today, UC Berkeley Professors Philip Stark, Jasjeet Sekhon and Henry Brady, along with me, sent the California Secretary of State a brief comment about her effort to provide precinct-level elections data for all 58 California counties.

We applaud the project and are very excited about the prospects of precinct-level data from each county in near real time. In the comment, we point out how the data isn't nearly as usable as it could be and how the focus right now should be on making sure that counties can get the data out of their back-end systems and then the SoS (with help, if she wants it!) can worry about transforming the data into something more usable.

Find the PDF of the comment here:
http://josephhall.org/papers/county-data-comments-final.pdf

UPDATE [2009-11-05T17:17:41]: The EAC replied and we responded.

Today we sent a letter to the Election Assistance Commission signed by a number of the investigators involved in the California Secretary of State's Top-to-Bottom Review of voting systems.

The letter points out that the EAC, and a lab that performs certification testing of voting systems, disregarded technical information about voting system security from state-level evaluations of a particular voting system.  This specific instance raises broader questions about the EAC's treatment of such information under its testing and certification program.

The signatories include:

• Matt Bishop, CA TTBR Principal Investigator, Department of Computer Science; University of California, Davis

• David Wagner, CA TTBR Principal Investigator, Computer Science Division; University of California, Berkeley

• Matt Blaze, Computer & Information Science; University of Pennsylvania

• J. Alex Halderman, Department of Electrical Engineering and Computer Science; University of Michigan

• Candice Hoke, Cleveland-Marshall College of Law; Cleveland State University

• Richard Kemmerer, Department of Computer Science; University of California, Santa Barbara

• Deirdre Mulligan, School of Information; University of California, Berkeley

• Elliot Proebstel, Department of Computer Science; University of California, Davis

• Eric Rescorla, Principal, RTFM, Inc.

• Hovav Shacham, Department of Computer Science and Engineering; University of California, San Diego

• Giovanni Vigna, Department of Computer Science; University of California, Santa Barbara

• Dan Wallach, Department of Computer Science; Rice University

(Cross-posted on the ACCURATE blog: ACCURATE Comment on VVSG v1.1)

A Center for Correct, Usable, Reliable, Auditable and Transparent Elections (ACCURATE) submitted public comment today to the U.S. Election Assistance Commission on their draft Voluntary Voting System Guidelines, version 1.1 (VVSG v1.1). The VVSG provides national certification requirements and testing protocols for voting systems against which many states require their voting systems to be certified.

Read more »

(Cross-posted at Freedom to Tinker.)

Short version: we now have gobs of voting system ITA reports, publicly available and hosted by the NSF ACCURATE e-voting center.

Long version: Before the Election Assistance Commission (EAC) took over the testing and certification of voting systems under the Help America Vote Act (HAVA), this critical function was performed by volunteers. The National Association of State Election Directors (NASED) recognized a need for voting system testing and partnered with the Federal Election Commission (FEC) to establish a qualification program that would test systems as having met or exceeded the requirements of the 1990 and 2002 Voting System Standards.^*

However, as I've lamented many, many times over the years, the input, output and intermediate work product of the NASED testing regime were completely secret, due to proprietary concerns on behalf of the manufacturers. Once a system completed testing, members of the public could see that an entry was made in a publicly-available spreadsheet listing the tested components and a NASED qualification number for the system. But the public was permitted no other insight into the NASED qualification regime.

Researchers were convinced from what evidence was available that the quality of the testing was highly inadequate and that the expertise didn't exist within either the testing laboratories to perform adequate testing or the NASED technical committee to competently review the ultimate test reports submitted by the laboratories (called Independent Testing Authorities (ITA)). Naturally, when reports of problems started to crop-up, like the various Hursti vulnerabilities with Diebold memory cards, the NASED system scrambled to figure out what went wrong.

I know have more moderate views with respect to the NASED regime: sure, it was pretty bad and a lot of serious vulnerabilities slipped through the cracks, but I'm not yet convinced that just having the right people or a different process in place would have resulted in fewer problems in the field. To have fixed the NASED system would have required improvements on all fronts: the technology, the testing paradigms, the people involved and the testing and certification process.

The EAC has since taken over testing and certification. Their process is notable in its much higher level of openness and accountability; the test plans are published (previously claimed as proprietary by the testing labs), the test reports are published (previously claimed as proprietary by the vendors) and the process is specified in detail with a program manual, a laboratory manual, notices of clarification, etc.

This is all great and it helps to increase the transparency of the EAC certification program. But, what about the past? What about the testing that NASED did? Well, we don't know much about it for a number of reasons, chief among them that we never saw any of the materials mentioned above that are now available in the new EAC system.

Through a fortunate FOIA request made of the EAC on behalf of election sleuth Susan Greenhalgh, we now have available a slew of ITA reports from one of the ITAs, Ciber.

The reports are available at the following location (hosted by our NSF ACCURATE e-voting center):

http://accurate-voting.org/docs/ita-reports/

These reports cover the Software ITA testing performed by the ITA Ciber for the following voting systems:

• Automark AIMS 1.0.9
• Diebold GEMS 1.18.19
• Diebold GEMS 1.18.22
• Diebold GEMS 1.18.24
• Diebold AccuVote-TSx Model D
• Diebold AccuVote-TSx Model D w/ AccuView Printer
• Diebold Assure 1.0
• Diebold Assure 1.1
• Diebold Election Media Processor 4.6.2
• Diebold Optical Scan Accumulator Adapter
• Hart System 4.0
• Hart System 4.1
• Hart System 6.0
• Hart System 6.2
• Hart System 6.2.1

I'll be looking at these in my leisure over coming weeks and pointing out interesting features of these reports and the associated correspondence included in the FOIA production.

^*The distinction between certification and qualification, although vague, appears to be that under the NASED system, states did the ultimate certification of a voting system for fitness in future elections.

California Secretary of State Debra Bowen has sent a letter to Chair Gineen Beach of the US Election Assistance Commission (EAC) outlining three proposals that she thinks will markedly improve the integrity of voting systems in the country.

I've put a copy of Bowen's letter here (87kB PDF).

Bowen's three proposals are:

• Vulnerability Reporting -- The EAC should require that vendors disclose vulnerabilities, flaws, problems, etc. to the EAC as the system certification authority and to all the state election directors that use the affected equipment.
• Uniform Incident Reporting -- The EAC should create and adopt procedures that jurisdictions can follow to collect and report data about incidents they experience with their voting systems.
• Voting System Performance Measurement -- As part of the Election Day Survey, the EAC should systematically collect data from election officials about how voting systems perform during general elections.

In my opinion, each of these would be a welcome move for the EAC.

These proposals would put into place a number of essential missing elements of administering computerized elections equipment. First, for the users of these systems, election officials, it can be extremely frustrating and debilitating if they suspect that some voting system flaw is responsible for problems they're experiencing. Often, when errors arise, contingency planning requires detailed knowledge about specific details of a voting system flaw. Without knowing as much as possible about the problem they're facing, election officials can exacerbate the problem. At best, not knowing about a potential flaw can do what Bowen describes: doom the election official, and others with the same equipment, to repeatedly encounter the flaw in subsequent elections. Of course, vendors are the most likely to have useful information on a given flaw, and they should be required to report this information to both the EAC and election officials.

Often the most information we have about voting system incidents come from reports from local journalists. These reporters don't tend to cover high technology too often; their reports are often incomplete and in many cases simply and obviously incorrect. Having a standardized set of elements that an election official can collect and report about voting system incidents will help to ensure that the data comes directly from those experiencing a given problem. The EAC should design such procedures and then a system for collecting and reporting these issues to other election officials and the public.

Finally, many of us were disappointed to learn that the 2008 Election Day survey would not include questions about voting system performance. Election Day is a unique and hard-to-replicate event where very little systematic data is collected about voting machine performance. The OurVoteLive and MyVote1 efforts go a long way towards actionable, qualitative data that can help to increase enfranchisement. However, self-reported data from the operators of the machinery of our democracy would be a gold mine in terms of identifying and examining trends in how this machinery performs, both good and bad.

I know a number of people, including Susannah Goodman at Common Cause as well as John Gideon and Ellen Theisen of VotersUnite!, who have been championing one or another of these proposals in their advocacy. The fact that Debra Bowen has penned this letter is a testament to the reason behind their efforts.

(cross-posted to Freedom to Tinker)

The Office of the California Secretary of State has issued a set of proposed emergency regulations for post-election manual tallying of paper election records. In this post, my first at FTT, I'll try to explain and contextualize this development.

Since her election to office, California Secretary of State (CA SoS) Debra Bowen has methodically studied the shortcomings in California's election equipment. She first initiated a Top-To-Bottom review (TTBR) of California's voting systems that found them to be of poor technical quality and vulnerable to a myriad of security vulnerabilities, accessibility flaws, reliability issues and inadequate documentation and testing (a number of FTT regulars participated in the TTBR). For this year's presidential primary in California, Bowen worked to mitigate these problems by decertifying this equipment and then recertifying it subject to a list of about 40 different conditions. One such condition is that the usual 1% manual tally under California law -- counties must randomly choose and hand tally ballots cast in 1% of precincts -- would be modified to include escalation that would mandate increased tallying for close races (where even small amounts of possible fraud and/or error could make a difference in the outcome of a contest).

Bowen issued these additional requirements (the "PEMT Requirements") under her authority as CA SoS to regulate election technologies (here are the original PEMT Requirements). Unfortunately, the Registrar in San Diego County sued Bowen arguing that she 1) didn't have such broad authority and 2) that, even if she did, she could only issue the PEMT Requirements through the California regulatory procedure (specified by the CA Administrative Procedure Act). A state Superior Court found in favor of the CA SoS but a Court of Appeal found that the PEMT Requirements did indeed betray characteristics of regulations and should therefore have gone through the regulatory procedure (for the legal eagles out there, see: County of San Diego v. Debra Bowen (2008) 166 Cal.App.4^th 501).

By the time the Court of Appeal had made its decision on August 29, there was no time to follow the normal regulatory process, which takes about four months. Instead, the CA SoS had to follow the process for adopting an emergency regulation which applies when a regulation "is necessary for the immediate preservation of the public peace, health and safety, or general welfare."

What is so special about these emergency manual tally provisions? First, it represents the increasing relevance and importance of adversarial considerations in the design of an election audit process. As we describe in the NYU Brennan Center / UC Berkeley Samuelson Clinic report on post-election audits ("Post-Election Audits: Restoring Trust In Elections"), fixed-percentage audits of election records are only particularly useful in detecting wide-ranging anomalies in vote counts. Methods that "tune" the amount of records audited depending on the margin in contests on the ballot do a much better job of ensuring that they'll find evidence of possible error or fraud. Per the emergency PEMT Regulations, any contest with a margin (difference between the winning and losing choice in a contest) of 0.5% or lower is subject to a 10% manual tally, an order of magnitude more scrutiny than the statutory default.

Second, the CA SoS' emergency PEMT Regulations reflect many best practices from audit theory and research: precincts to audit must be chosen randomly; the precincts to audit are only chosen after the semi-official vote tallies are arrived at; tally activities must be announced publicly and available for public observation; tallies must be conducted under "blind count" rules where the talliers do not know the totals in the precincts they're tallying; differences between machine and hand counts must be explained or investigated.

The elephant in the room is always Los Angeles County; LA is so amazingly enormous for an election jurisdiction that some things simply aren't possible. (For example, they frequently pick up ballot materials from precincts in helicopters; that is, traffic in LA is so bad and there are so many polling places (~5,000 or so) that the most reliable form of ballot transmission is via helicopter.) These rules are going to be exceedingly difficult for LA to comply with. I expect they will hire an army of tally managers and talliers to perform their tally and that it will be a race against the clock, counting 24 hours a day, seven days per week, to try and get it all done in the 28-calendar day canvass period.

Yesterday, I filed my dissertation and got my sucker (Berkeley gives you a tasty See's Cafe Latte hard candy sucker when you graduate).

My dissertation is about a very poignant topic: policy mechanisms for making electronic voting systems more transparent. After refining what we mean when we talk about transparency, I examine open source, procurement contracts and auditing voting systems and show that each of these hold real promise for peeling back the layers of opacity in which we now find our voting systems enshrouded.

If you'd like to read it, here is a link and the abstract:

Policy Mechanisms for Increasing Transparency in Electronic Voting

In the early years of the American republic, only white male landowners could vote, and then typically by expressing their preferences in a public setting, for all to witness. Our electoral system has changed drastically since that time; now almost all Americans cast votes with the assistance of computerized equipment. While much good stems from the use of computerized equipment in elections—notably increased efficiency, enfranchisement and flexibility—unintended consequences of this mechanization have left us with complicated, insecure and opaque voting systems.

My PhD thesis focuses on the issue of transparency in evoting; that is, what public policy mechanisms can serve to make our voting systems less opaque? After exploring what we mean by “electoral transparency”, I examine the question of evoting transparency on three fronts. I analyze the role of disclosed and open source software in election systems and conclude that, while fully disclosed source code is a valid goal, limited disclosure to experts serves many of the same goals in the shortterm while preserving vendor trade secrecy. I investigate how contractual provisions between local election jurisdictions and voting system vendors serve to frustrate transparency and find that election officials need to be more careful in these negotiations. Finally, I turn to the question of auditing black box elections systems; that is, since we cannot know how these systems work in the fulldisclosure (“white box”) case, possibly because of contractual provisions that limit investigation, what methods and procedures can we use for “checking the math” behind our elections?

If you're in NYC on Friday, come see me (and others) speak at the American Bar Association Annual Meeting.

I'll be talking about types of voting technology, the recent CA TTBR and EVEREST reviews and implications for the 2008 elections (slides are here). This will be part of a panel entitled, "The Devolution of Voting Technology: From Pen and Paper to Touch Screens and Back Again?" with Jack Feeney, Jacquelynne Harris and Candace Hoke.

Here's the deets:

The Devolution of Voting Technology: From Pen and Paper to Touch Screens and Back Again?

The American Bar Association Standing Committee on Election Law presents a free CLE program...

Friday, August 8, 2:00 p.m. to 4:00 p.m.
Conference Room I, Executive Conference Level
Sheraton New York Hotel and Towers

Panelists

• John C. Keeney, Jr., Washington, DC (Moderator)
• Joseph Lorenzo Hall, Berkeley, California
• Jacquelynne Harris, Charlottesville, Virginia
• Professor S. Candice Hoke, Cleveland, Ohio

Additional Sponsors: Section of Administrative Law and Regulatory Practice, Section of Science and Technology Law, Section of State and Local Government Law, Council on Racial and Ethnic Justice

Doug Chapin and Chris Backert pointed me to the new Voting Information Project (Chris entitled his post, “Just About The Coolest Thing”). My initial reaction was much like the title of Chris’ post: it did seem very cool. Not only that, but it’s funded/coordinated by Google, Inc., Pew’s Center on the States and the JEHT Foundation. This will be a wonderful resource when finished and will start to regularize structured data publication by local election jurisdictions. Many thanks and my gratitude to those behind VIP for their efforts and investment.

However, after looking a bit more deeply into the current content of the VIP site, I’m left with some serious questions. I hope we can work through some of these comments to best position VIP to fulfill its goals.

What’s the vision?

It’s hard to find a coherent vision for the project. They have a FAQ and a longer description of the project, but no mission, vision or roadmap and no indication of who (as in people, individuals) are participating (other than the developers listed on their Google Code page).

Right now the VIP appears to be a data standard (a markup language specification), a funding program for states to collect data in this format and a method of uploading such data to the VIP site. (In the longer description of the project, they say they’ll soon provide data feeds.)

It would be nice to know what else they have planned. One concern during an election year is a data collection overload; the VIP shouldn’t develop in a vacuum, which it appears to have over the past eight months. That is, much of this information is regularly made available by NGOs. Ballot information can be had at the state and local level via LWV’s smartvoter, for example. The Verified Voting Foundation has invested quite a bit of time and effort in producing a mapping of the types of voting technologies used at the local level (check out their Verifier tool). I can understand how this is a “push” effort rather than a “pull” effort---that is, the idea is for government entities to start providing this data instead of NGOs to have to collect it---but a more likely outcome for this project is that a few states provide feeds in 2008 and then stop maintaining this information when the attention dissipates.

(NB: The specification doesn’t have a name other than the “Voting Information Project’s open format” (or, in various places, “EVP XML” or “EIP XML”). I’ll call it the “VIP XML format”.)

Where’s the EML?

The VIP XML format specification document (via this very ugly URL), doesn’t mention the OASIS’ Election Markup Language (EML). EML is an international standard markup language for election data that is currently in the process of becoming an ISO standard. It would be good if the VIP folks could say why they decided to make their own standard instead of reusing pieces of the EML standard. I suspect this is because the EML standard is general to more than just US elections and that it might not include elements that they need for their goals. However, it would make sense to see how EML could be changed to accommodate these needs rather than eschewing it altogether.

This is an important point: There is no mention on the VIP site that they’re working with voting system vendors. Vendors are planning on supporting data interchange in EML format and a number of us have made the case that the new US VVSG standard should require vendors to produce data in EML (see the text of ACCURATE’s VVSG comments). The VIP project would be much more useful if they also invested time in XSL transforms that would allow EML interoperability with the VIP XML format.

Where are the data entry tools, etc.?

Related to this last point, if the idea is to provide regularly-updated “feeds” of elections-related information, there needs to be mechanisms for getting this information out of the tools that jurisdictions currently use, or providing data-entry “wizards” that will walk local election officials---where most of this information will reside---through the process of creating a data feed.

Obviously, data “feeds” are best and most useful when automatically created rather than manually updated. There isn’t any attention on the VIP site to the integration needed with existing elections information management tools and election management systems to best support data publication and dissemination such that the end-to-end connection is made from tools that contain this information to publication of the information via the VIP feeds mechanism.

What will encourage jurisdictions to enter/provide data?

In all seriousness, why would a jurisdiction want to provide this information? Unless election officials see a clear benefit from providing the data and that the process for doing so is easy (and hard to provide incorrect, incomplete, etc. data), I can’t imagine many jurisdictions will take the time to do this.

The VIP project seems to think that this is a problem of initial costs. To remedy this, they’re offering $20,000 per state to help with these fixed costs. I’m not convinced that’s adequate. It might make more sense if there was some sort of incentive offered for providing data, similar to a non-profit equivalent to Dan Tokaji and Thad Hall’s proposal for providing federal funding contingent upon completing a detailed election data survey instrument.

I don’t think financial incentives are absolutely necessary to get comprehensive, high-quality data. Instead, election officials need to see there is some sort of network effect that compels them to provide data. For example, if there was a social network-component or if, through using it, they got access to some tools that made their jobs easier (such as voter-oriented elections information portals).

What’s with the draconian terms?

Being not-quite-a-lawyer, I was really interested to see the VIP project’s Terms of Service. There are a few interesting, and possibly troubling, features of this ToS:

• CC licensed specification: The specification document is licensed under a Creative Commons Attribution-ShareAlike 2.5 license. It appears that the text of recent Google and Microsoft specifications are so licensed but, I’d feel more comfortable with a license that recognizes that a specification is more than just text (see Creative Common’s Mike Linksvayer’s post “What good is a CC licensed specification?”).
• Changes could be arbitrary: For a site that aims to provide data feeds, it’s a bit troubling that the ToS says anything can happen at any time without any notice. Some notion of a commitment to provide data that people are relying on would be a good thing.
• It’s a contract: Unlike most ToSs, it explicitly says that the ToS is a “contract” that we, as visitors, enter into by using the website. In the world of terms of use, terms of service, etc., it’s rare to see the word “contract” used; usually ToS writers prefer less-specific language. I’m not sure if this is necessarily a problem and would be interested if any legal minds out there have an opinion on such adhesion contracts on the internet. (I just might not be aware of recent developments that advise using the more specific “contract” language.)

My point here is that these terms seem somewhat draconian and reflect the desire of the backers to avoid legal messes and liability. But will these terms serve their goals of creating a interactive data community? I’m doubtful. The question of what terms should apply to “open services” will have to wait for a future article.

What kind of standard is their spec?

Finally, the big kicker for me: This doesn’t appear to be an open standard. I would call it a disclosed standard or a limited open standard (after my paper on source code disclosure in voting systems... that I’m currently updating for my thesis publication): Only a few possible constituencies are able to provide input into the VIP XML standard and it can change arbitrarily at any point in time. If one of the goals is to support data publication in an end-to-end manner, it would be best to include users of the data as well as election management system developers (vendors) in development of the standard. As it stands now, it appears to be an effort entirely contained within Google and a set of unidentified state-level election officials.

I don’t want to sound too harsh; I think this is exactly the kind of endeavor we need more of. However, the overall feeling I get from the VIP is not as promising as it could be.

UPDATE [2008-05-28T12:26:47PDT]: Added some sugar to my criticism.

I gave my dissertation talk today. It went surprisingly well. I felt my delivery was smooth---with nary an “um”---and people seemed interested! I’ve learned that I have to practice talks 3-4 times before I’m ready to give them. (and the “presenter tools” feature in powerpoint is killer, I must say.)

The dissertation talk is a curious beast on the academic landscape. It involves summing up 5 years of work (at least!) in about 50 minutes. I found it very challenging to both adequately explain the work I’ve done and then make that explanation accessible to a multidisciplinary audience. Apparently, I didn't do such a bad job; I can’t explain how gratifying that is for someone who enjoys teaching.

I’ll post my slides one slideshare in a bit and update this post, but they do need a bit of massaging---I used animation to make images appear and disappear over the text, which doesn’t seem to work with slideshare.

Anyway, I’s tie tie now.

(image by the Schultz-enator)