The Washington PostDemocracy Dies in Darkness

Iowa Democrats kept their app secret to prevent hacks. Instead, they got confusion and chaos.

February 4, 2020 at 8:30 p.m. EST
The delay in results from Iowa’s first-in-the-nation caucuses Feb. 3 was due in part to problems with a mobile app used to report figures to the state party. (Video: The Washington Post)

Linda Nelson, a 68-year-old retired elementary school teacher in Council Bluffs, received a notice over the weekend that a new version of the mobile app she would use three days later for Iowa's first-in-the-nation caucuses was ready for her to test.

But when Nelson downloaded the software on her smartphone, she couldn’t figure out how to log in. She tried one PIN number. She tried the ID for the precinct she would be chairing in the western Pottawattamie County. She tried another PIN number. “Nothing works!” Nelson wrote in an email to the state party.

Her problems were replicated across the state on Monday, as precinct leaders failed to access the app or reported being locked out as they sought to punch in numbers to report them to state party officials. In one especially stark example, just one precinct leader in all of Scott County, which is part of the Quad Cities region, was able to successfully use the software, according to the county’s Democratic Party chair.

The technical setbacks generated mass confusion that played out on live television Monday night, turning the Democrats’ much-anticipated, first-in-the-country presidential nominating contest into a global embarrassment — leaving an international audience in the dark and putting another blemish on the United States’ democratic system.

Buttigieg, Sanders leading in early results

By Tuesday, as state party officials began to release some of the results while apologizing for the failure, the debacle fueled calls for Iowa to relinquish its coveted status at the top of the presidential nomination calendar.

Attention focused on the state’s arcane voting system, illustrated by viral images of intricate caucus math worksheets used to allocate delegates at about 1,700 caucus sites. And it provided fodder for prominent allies of President Trump, including his campaign manager and adult sons, to question the legitimacy of the Democratic contest. Sen. Lindsey O. Graham (R-S.C.) took to Twitter to suggest without evidence that last-minute uncertainty was engineered to damage Sen. Bernie Sanders (I-Vt.).

Some explanations for the meltdown began to emerge — including the fact that many precinct organizers had not been able to adequately test the app designed for reporting results to party headquarters. But questions also remained about why the state party chose Shadow, a Democratic-aligned start-up, for such a high-stakes moment.

The app had been rolled out for testing in mid-January, just weeks before the caucuses. The delay was intentional, according to a state party official who spoke on the condition of anonymity to discuss sensitive matters.

Iowa Democrats waited to introduce the software to avoid giving potential hackers time to penetrate it, the official said, adding that the party chose to keep the name of the vendor secret on the advice of national cybersecurity consultants.

In the end, the problem on caucus night was not a hack. Nor could it be explained alone by difficulty in downloading the software.

It was a more basic coding error that caused the problems. When caucus leaders started successfully reporting data through the app, a separate system also developed by Shadow collated the information but spit out only partial results, according to a person familiar with the events.

State party officials soon discovered the coding error, which would delay the final count by more than a day and raise new questions about the judgment of party heads.

State party leaders insisted that the vote count itself was not affected.

“This was a coding error in one of the pieces on the back end,” said Troy Price, the chairman of the Iowa Democratic Party. “But the raw data, the data that has come in, is secure.”

Still, the technical failures became another damaging flash point for Democrats, who were subject to hacks and disinformation in 2016, when Russian nationals penetrated party servers as part of a scheme to interfere in the presidential race.

Monday was a confusing night for Democrats. President Trump is smiling.

The Nevada Democratic Party, which had planned to use the same software for its Feb. 22 caucuses, said Tuesday that it would forgo the app.

“We will not be employing the same app or vendor used in the Iowa caucus,” the state party’s chairman, William McCurdy II, said in a statement. “We had already developed a series of backups and redundant reporting systems, and are currently evaluating the best path forward.”

Iowa was caught in suspended confusion, with former South Bend, Ind., mayor Pete Buttigieg declaring victory Monday night before formal results were released and Sanders predicting a strong finish for himself. Meanwhile, an attorney for the campaign of former vice president Joe Biden, who was in fourth place based on initial results released Tuesday, wrote a letter to state party officials citing “considerable flaws” in the caucus reporting system.

Unfounded conspiracy theories coursed through social media, pushed by Trump allies as well as backers of Democratic hopefuls, as Iowa party leaders struggled to salvage confidence in the vote.

“Even though the reporting was a hitch, there is no hint of and there hasn’t been any skulduggery, intrigue or scalawagging,” said David Nagle, a former state party chair who helped lead caucus planning after the 2016 election. “The party was faced with a quandary: Do we throw numbers out there, or do we get it right?”

Tuesday brought new scrutiny to Shadow, the technology company registered in Colorado that developed the software and is led by veterans of Hillary Clinton’s presidential campaign. The company was founded as Groundbase in 2017 and rebranded in 2019 with a major investment from Acronym, a nonprofit digital outfit with an affiliated political action committee.

Shadow received about $154,000 from federal campaign committees, state parties and politically active groups in 2019, Federal Election Commission records show. Among its biggest clients was For Our Future, an independent operation supporting Democrats, which paid the company more than $10,600 for “digital communications.”

Democratic campaigns hired the firm, as well, to help with fundraising and digital communications, including the campaigns of Buttigieg, Biden and Sen. Kirsten Gillibrand of New York, who dropped out of the race last year.

Biden’s campaign dropped Shadow after the former vice president’s IT team raised concerns about the software it was providing for sending text messages, according to a spokesman for the campaign.

Shadow released a statement Tuesday on Twitter expressing regret for the delay in Iowa and saying the underlying technical issue had been corrected. In 2016, the state party used an app developed by Microsoft. “It worked just fine,” said Andy McGuire, who was the state party’s chairwoman at the time.

How the Shadow app stumbled in Iowa

The state party turned to the tech company late last year, finalizing a contract in the fall, and kept the plans under wraps, according to Democratic officials and outside consultants who spoke on the condition of anonymity to discuss private deliberations. When Iowa Democrats participated in a November simulation run by the Defending Digital Democracy Project at Harvard’s Belfer Center — an effort to test the security of the upcoming caucuses — they were still finalizing plans with the firm, according to an individual who was present at the exercise, held in Des Moines.

Iowa Democrats paid Shadow about $63,000 in two installments in November and December, records show. Nevada also made payments through the end of last year.

“Back in late 2019, they said there would be an app coming out,” said Steven Drahozal, the Democratic chairman in Dubuque County. Beyond the instructions that accompanied the app’s rollout, he added, there was little training from the state party on how to use it.

A spokeswoman for the state party, Mandy McClure, said there were office hours during which precinct leaders could call in for help.

“This was way too hurried,” said David Jefferson, a computer scientist at the Lawrence Livermore National Laboratory who has counseled the national party on technology issues and said experts had warned about risks posed by a mobile app for recording results. “They couldn’t possibly have enough testing time, and they couldn’t possibly test it at scale.”

A spokeswoman for the Department of Homeland Security, Heather Swift, said that the department’s Cybersecurity and Infrastructure Security Agency had offered to test the app for the state party but that leaders in Iowa declined. Price, in response, said the party’s “systems were tested by independent cybersecurity consultants.” He added Tuesday that he was not aware of the DHS offer.

Dan Wallach, who runs Rice University’s computer security lab, said the lack of transparency erodes trust.

“There was demonstrably not adequate training and testing,” he said, pointing to apparent connectivity problems in rural parts of the state.

In some areas, difficulty installing the app or completing verification with a PIN led county officials to tell their volunteers to rely on calling in their results. As Monday night wore on, the phone lines at the state party were overwhelmed.

“When our chairs are calling, it’s a wait time of an hour and a half or two hours,” said Sean Bagniewski, the chairman of the Polk County Democratic Party. “In some cases, they have dropped the call.”

In Johnson County, one caucus site leader who finally got through passed around a phone with an open line so other chairs could report their results, according to Brian Deese, a party volunteer who witnessed the cooperation. “Pretty much everybody defaulted to phone lines, and it got jammed,” he said.

At one point, Judy Downs, the executive director of the Polk County party, took photos on her phone of the final results at multiple caucus sites and drove over to the state party headquarters, where she was turned away, Bagniewski said.

Before the caucuses, the Democratic National Committee had boasted about its efforts to help state parties such as Iowa secure their election processes against hacking and other manipulation. Iowa Democrats scrapped plans for “virtual caucuses” — which would have allowed Iowans to participate by phone — after national leaders moved to block them in August, warning of ­cybersecurity threats.

Michelle Ye Hee Lee contributed to this report.