Diebold Admits Systemic Audit Log Failure; State Vows Inquiry
SACRAMENTO, California — Premier Election Solutions (formerly Diebold Election Systems) admitted in a state hearing Tuesday that the audit logs produced by its tabulation software miss significant events, including the act of someone deleting votes on election day.
The company acknowledged that the problem exists with every version of its tabulation software.
The revelation confirmed that a problem uncovered by Threat Level in January, and reiterated in a report released two weeks ago by the California secretary of state’s office, has widespread implications for election jurisdictions around the country that use any version of the company’s Global Election Management System (GEMS) software to tabulate votes. The GEMS software is used to tabulate votes cast on Premier/Diebold touch-screen and optical-scan machine, and is used in more than 1,400 election districts in 31 states. Maryland and Georgia use Premier/Diebold systems exclusively, therefore the GEMS software counts every vote statewide.
“Today’s hearing confirmed one of my worst fears,” said Kim Alexander, founder and president of the non-profit California Voter Foundation. “The audit logs have been the top selling point for vendors hawking paperless voting systems. They and the jurisdictions that have used paperless voting machines have repeatedly pointed to the audit logs as the primary security mechanism and ‘fail-safe’ for any glitch that might occur on machines.
“To discover that the fail-safe itself is unreliable eliminates one of the key selling points for electronic voting security,” Alexander said.
Following a public records request of GEMS logs, Threat Level previously reported that the Premier/Diebold logs did not indicate when election officials in Humboldt County, California, intentionally deleted more than two dozen batches of ballots from their system during the November general election.
The finding raised questions about the integrity of elections conducted with the system, but it was unknown at the time whether the problem with the audit log existed with other versions of the GEMS software used in other counties in California and across the country. Premier/Diebold didn’t respond to phone calls seeking information at the time.
The secretary of state’s report (.pdf) discussed the same problem with the logs but also did not indicate whether the problem existed with every version of the GEMS software.
A Premier/Diebold representative confirmed at the hearing that none of its logs records such events.
When asked by a member of the California secretary of state’s staff if the company had done anything to address the problem, Justin Bales, general service manager for Premier/Diebold’s western region said, “No, not yet.”
Bales went on to say that the GEMS logs have been the same since the software was first created more than a decade ago.
“We never, again, intended for any malicious intent and not to log certain activities,” Bales said. “It was just not in the initial program, but now we’re taking a serious look at that.”
California Secretary of State Debra Bowen called the audit logs
“useless” and vowed to investigate the issue further. She told Threat
Level after the hearing that an examination of audit logs in other voting systems was also merited in light of these revelations.
“Clearly, we’re going to have to look at this,” Bowen said. “That’s one of the obvious next steps.”
The secretary of state’s office was holding a hearing to discuss a report it released two weeks ago examining what occurred on a
Premier/Diebold system in Humboldt County that “lost almost 200 ballots during the November presidential election.
Premier/Diebold has stated that a programming flaw in version 1.18.19
of its GEMS software caused the ballots to be deleted but has said the problem was fixed in a later version of the software.
To investigate the issue, California officials turned to the GEMS audit logs to see what occurred in the system when the votes were deleted.
But they quickly discovered that the logs could provide them with no clues about what went wrong.
“In terms of being able to track down the precise mechanism by which the problem had occurred in this election, critical information was simply never recorded,” said Lowell Finley, deputy secretary of state for voting systems technology and policy, who testified at the hearing.
Finley said his staff was also shocked to find that two of the logs contained a “clear” button that allowed officials to delete them.
Finley said this violated federal voting system standards, which require voting systems to maintain an indestructible archival record of all system activity related to the vote tally and, in particular, any activity involving unusual intervention by an election official.
The “clear” button was removed from a later version of the GEMS
software, but Finley said three counties in California still used the
1.18.19 version containing the button, as do jurisdictions in Texas and
Bales explained that the “clear” button was installed in the software to aid a few counties that used the GEMS database as a template “for creating subsequent elections.” The clear button was included to allow them to erase a log after copying the template.
“With the benefit of hindsight, we saw that as definitely not the best avenue to do,” he said. “It was in there with no malicious intent.”
Bowen’s office is evaluating whether it will de-certify GEMS version
1.18.19. Bales said Premier/Diebold fully supported de-certifying this version in California. He did not address whether his company would make the same recommendation to other states using the software.
But even if that version is de-certified, voting activists who testified at the hearing said the systematic issue with the GEMS audit logs points to a fundamental problem that won’t go away.
“I believe the quality of this product has proven to be highly questionable, and the voters are sick and tired of this kind of abuse of the vote count,” said Gail Work, chair of the election integrity committee for the San Mateo County Democratic Central Committee.
Software for the Premier/Diebold voting machines was first written more than a decade ago by a company named I-Mark, which was bought by a company named Global Election Systems. Diebold bought Global Election
Systems in 2002 to launch its election division, Diebold Election
Systems. In 2007, in the wake of bad publicity, Diebold changed the name of its election division to Premier Election Solutions.
– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –
Report: Diebold Voting System Has ‘Delete’ Button for Erasing Audit Logs
After three months of investigation, California’s secretary of state has released a report examining why a voting system made by Premier Election Solutions (formerly known as Diebold) lost about 200 ballots in Humboldt County during November’s presidential election.
But the most startling information in the state’s 13-page report (.pdf) is not why the system lost votes, which Wired.com previously covered in detail, but that some versions of Diebold’s vote tabulation system, known as the Global Election Management System (Gems), include a button that allows someone to delete audit logs from the system.
Auditing logs are required under the federal voting-system guidelines, which are used to test and qualify voting systems for use in elections. The logs record changes and other events that occur on voting systems to ensure the integrity of elections and help determine what occurred in a system when something goes wrong.
“Deleting a log is something that you would only do in de-commissioning a system you’re no longer using or perhaps in a testing scenario,” said Princeton University computer scientist Ed Felten, who has studied voting systems extensively. “But in normal operation, the log should always be kept.”
How to Make Your Browsing Data More Private than a Thousand Incognito Windows
Yet the Diebold system in Humboldt County, which uses version 1.18.19 of Gems, has a button labeled Clear, that “permits deletion of certain audit logs that contain — or should contain — records that would be essential to reconstruct operator actions during the vote-tallying process,” according to the California report.
The button is positioned next to the Print and Save As buttons (see image above), making it easy for an election official to click on it by mistake and erase crucial logs.
In fact, the report says, this occurred recently in a California county when an official, while attempting to print out a copy of a so-called “poster log,” inadvertently deleted it instead.
The system provides no warning to the operator that clicking on the button will result in permanent deletion of records in the log, nor does it require the operator to confirm the action before executing it.
Apparently Premier/Diebold was aware that having a Clear button on its system was a bad idea. According to California’s report, one of the system’s developers wrote in an e-mail in 2001: “Adding a Clear button is easy, but there are too many reasons why doing that is a bad idea.” Yet the company included the button in its system anyway.
The button was removed from software versions 1.18.20 following, but Premier/Diebold never went back to jurisdictions using previous versions to upgrade them, and version 1.18.19 is still used in three California counties as well as in other states. It’s unclear how many previous versions of the software had the button, or why it was included in the first place.
According to the report:
The Clear buttons … allow inadvertent or malicious destruction of critical audit trail records in all Gems version 1.18.19 jurisdictions, risking the accuracy and integrity of elections conducted using this voting system. Five years after the company recognized the need to remove the Clear buttons from the GEMS audit log screens, not only Humboldt, San Luis Obispo and Santa Barbara Counties in California but jurisdictions in other parts of the country, including several counties in Texas and Florida, continue to use Gems version 1.18.19….
The report states that the inclusion of the button violated the federal voting-system standards under which the Premier/Diebold system qualified to be used in elections. The standards require that voting-system software automatically creates and permanently retains electronic audit logs of important system events that occur on the machine.
Premier/Diebold did not respond to a request for comment.
The Clear button isn’t the only problem with the audit log in the Premier/Diebold system. Wired.com previously reported other issues with the logs — for example, they don’t record significant events that occur in the tabulation system, such as when someone deletes votes from the software.
The California report states that the Clear button and other issues should have been a red flag to the testing laboratories that certified the system. The system should have flunked certification-testing and been banned from the election.
Under the official voting-system standards, “each of the errors and deficiencies in the Gems version 1.18.19 software described in this report, standing alone, would warrant a finding … of ‘Total Failure’,” the report concludes.
“Presumably some organization, some lab, looked at this system and decided they thought it complies with the standard,” said Felten. “And, obviously, they were wrong. Any state that uses Gems should be looking at this seriously.”
It’s unclear what the states currently using the Gems system will do now that they know their voting software does not create an adequate audit trail.
California’s secretary of state has scheduled a public hearing on March 17 (.pdf) to discuss the report and whether version 1.18.19 of Gems should be decertified in the state. That would force counties in the Golden State to upgrade to a different version.
As for addressing the fundamental problems with the audit logs in all versions of the GEMS software, a spokeswoman for the secretary of state’s office said only that the state sent the report to the federal Election Assistance Commission to communicate the issue to election officials in other states.
A spokeswoman for the EAC told Wired.com that the commission has no authority to address problems with voting systems that were tested and qualified prior to 2002, when Congress gave the organization oversight responsibility.
“There’s no regulatory action that we could take,” said EAC spokeswoman Jeannie Layson. “But certainly … [we] make sure that the test labs and independent reviewers who look at the test reports are aware of all that information.”
The lab that was responsible for testing and qualifying Gems version 1.18.19 with the Clear button is Colorado-based Ciber. In 2007, the lab was suspended from testing voting systems for not following quality-control procedures and for failing to document that it was conducting all the required tests. But the EAC restored the lab’s accreditation to test voting systems last year.
Ciber did not respond to a call for comment about its examination of the Premier/Diebold system and its approval of the Clear button.
The California report is the result of an investigation into what occurred in Humboldt Countyduring the November 2008 presidential election.
After the election, county officials discovered that their tabulation software had dropped 197 ballots without giving any notice to election officials that it was doing so. Humboldt uses a Premier/Diebold central-count optical-scan system. The company acknowledged that a programming flaw in version 1.18.19 of Gems could drop votes when used with a central-count scan system, and that it had known about the problem since October 2004.
Premier/Diebold sent some election officials a workaround at the time, though Humboldt County election director Carolyn Crnich never received it. The company also never notified California state officials or the federal EAC so that election officials around the country could be notified.
The flaw was fixed in May 2005. But until then, the vendor let jurisdictions use five flawed versions of the software and never explained the problem or the workaround in user documentation. Diebold has said that no jurisdiction outside California used these versions of Gems with a central-count scan system and therefore were not at risk from the flaw. California officials backed this claim in their report.
Secretary of State Debra Bowen has sponsored legislation that would require a voting-machine vendor to notify the state in writing (.pdf) any time it discovers a problem with its voting system. The vendor would have to notify the state — and any California jurisdiction using the voting system — within five working days of discovering a flaw in software or hardware.
The bill also requires a vendor to disclose any flaws it already knows about systems that are currently in use in the state. These reports will then be submitted to the EAC so that officials in other states will know about them as well. The bill provides for civil penalties of $10,000 per violation against vendors for undisclosed flaws or for making unauthorized changes to a voting system.
Kate Folmar, spokeswoman for the secretary of state’s office, said Bowen hopes that the bill, if passed, “could become a model for other states for dealing with similar anomalies and problems that pop up with their voting systems.”