Inside the 2024 Precinct Project

Getting the raw data from states is, at most, a fifth of the battle. The bulk of our effort goes to parsing and standardizing those raw data with the overarching goal of accurately preserving the information it contains whenever possible. We encounter data in all variety of fantastic and horrifying formats, but raw data files generally fall into one of three buckets:

Tabular

When data come in tabular files (.csv, .tsv, .xlsx, etc.) cleaning is generally a straightforward matter of standardizing entries to be consistent with our codebook through pivoting, recoding, and merging in geographic identifiers like FIPS codes.

Born-digital documents

Just as, if not more, common than tabular files are documents (.pdf) created on a computer (i.e. they are not scans of a physical document). Some states, like Kentucky, report election results in documents with the exact same format across counties (often PDF exports of spreadsheets). We parse these documents with tools like pdfplumber that use optical character recognition (OCR) to identify and extract their contents. This is usually straightforward for born-digital documents (angled text and complicated tables still prove to be tricky) but still requires tailoring an extraction script for a given file or file format. LLM-based coding assistants like GitHub Copilot, Claude Code, and GPT Codex can generate scripts to extract and wrangle data from these documents with limited prompting. 

Digitized documents

Scans of physical documents are the bane of our existence in this process. In most cases, these do not already have an OCR layer (i.e. you cannot search for text within them) and cannot have one easily applied due to differing page orientations, poor resolution, artifacting, or being cut off—see, for example, the case from Wallowa County, OR shown below. For most of the project’s history, such cases required manual extraction, which is both extremely time consuming and prone to errors—just about anyone’s eyes will glaze over as they try to transcribe hundreds of pages that look like the example. While we still resort to manual extraction when absolutely necessary, AI tools considerably reduce the need.

Once we’ve collected and extracted the data for a state, it enters our quality assurance (QA) process comprising three phases. In the first phase, the wrangler—the person performing extraction for the state—assigns a member of the research team as the first reviewer. This reviewer performs a series of validation checks on the data and provides detailed feedback about issues with the data. The wrangler then applies fixes to address reviewer feedback before kicking the data back to the reviewer for another round of QA; rinse and repeat until the reviewer no longer sees any issues. A second reviewer then performs validation checks independently. If they find further issues, the state goes back to the first phase, otherwise they give their stamp of approval. At this point, we make the data available in our 2024 election data repository so the public can use it and report issues they encounter.

After all states pass two rounds of QA, we enter the final phase in which the research director addresses issues reported by the public and performs a final round of validation checks. Sometimes, as was the case for Indiana this time around, additional data has been released since our initial collection, necessitating further wrangling and rounds of QA.

You may be wondering what our validation checks actually check. Our workhorse for automated checks is an internal tool called QAPP (Quality Assurance for the Precinct Project) that expands on a QA engine built by a previous research associate Marcos Zárate and the previous research director, Samuel Baltz. Rather than having reviewers manually hunt common errors, it runs around 200 checks across 15 categories and produces a single report indicating what may need closer inspection. Some of these checks are basic: are all required columns present, are values in the expected format, and are state and county codes consistent with official reference files? Others look for problems that can be difficult to spot, such as duplicate or conflicting rows, blank candidate names, non-numeric or negative vote totals, suspicious zero-vote precincts, and inconsistent district labels. 

QAPP also helps us check whether the dataset makes sense as a whole, not just row by row. It summarizes missing or unusual values, flags unusual vote distributions, and, perhaps most importantly, it aggregates precinct results up to statewide totals for major offices so we can compare them against state-level results where available. The engine does not modify the data on its own–it only directs attention to the places where errors are most likely and allows the reviewer to document issues for the wrangler to fix. 

Our effort to produce a nationwide dataset of precinct-level returns continues to evolve as we refine our data pipeline, learn from previous releases, and incorporate new technologies. The 2024 dataset took over 500 days from when polls closed on November 5, 2024 to be ready for release. However, as the work required to produce the data can increasingly be automated, be it collecting, extracting, or quality assuring the data, we hope to considerably reduce the time between when states report official results and when we release our dataset. 

But this is not without challenges. For one, advances in AI mean a rapidly changing understanding of how one can effectively use these tools—whether it’s optimizing prompts, orchestrating subagents, or knowing when to use skills. Consensus around any of these is elusive, changing day by day and model to model. A downstream consequence, exacerbated by the stochastic nature of LLMs, is the potential to create data (and ultimately research) that cannot be reproduced.

To date, most social science work probing this focuses on the problem posed by non-determinism in text annotation tasks. But even using LLMs to collect and wrangle data, as we do, has the potential to cause reproducibility problems if care isn’t taken. Suppose a human wrangler comes across a particularly awful digitized document containing a county’s data and asks Gemini to extract it. Gemini spits out a CSV file with cleaned data that looks reasonable enough and seems to accurately reflect the raw data, so much so that it passes through QA and is published with these missing races. When a user eventually encounters and (hopefully) reports the error, it will not be possible to determine what went wrong with the extraction—–the document will have to be entirely reparsed and QAed, consuming time and model credits that could have been saved if the wrangler had simply prompted Gemini to write a script that produces the cleaned data from the raw. 

A broader challenge presented by AI is the redistribution of labor instigated by automation of entry-level tasks. The oft-repeated example is coding assistants replacing the junior developer who eventually becomes the senior developer, but researchers face a similar dilemma. Much academic research has traditionally rested on the work of undergraduate and graduate research assistants performing repetitive data entry, cleaning, and classification tasks—–the very tasks AI models excel at. While not the most thought-provoking or interesting work, there is an argument to be made that getting one’s hands dirty with data in this way is essential to developing the skills and expertise needed to succeed at high level tasks. But not too long ago one may have said the same thing to a junior developer about the necessity of learning assembly language to effectively program in C++. 

In the context of wrangling election data, there is validity to both perspectives. It may very well be the case that students can now gain more practical experience learning how to effectively automate data tasks rather than doing them manually or with human-written code. Yet a deep understanding of the structure and limitations of the underlying data is still indispensable when conducting rigorous research.