The Tyranny of the Tidy Dataset: When Cleanliness Obscures Truth

In the world of open data, there is a sacred cow we rarely question: the tidy dataset. We are taught that good data is clean data. It must be scrubbed, normalized, and structured into neat rows and columns, free from the messiness of human inconsistency. This is the received wisdom, the unassailable goal. But in our zeal for cleanliness, in our pursuit of a perfectly queryable CSV, we risk performing a quiet act of archival violence. We are laundering reality to make it machine-readable, often at the cost of making it less human-meaningful.

The process of "cleaning" data is inherently interpretive. Anomalies are smoothed out, outliers are removed, and inconsistent entries are forced into a predetermined schema. A handwritten comment in the margin of a public ledger—a snarky aside, a heartfelt plea, a correction—is the first thing to be deleted. It doesn’t fit the column. It is relegated to a separate ‘notes’ field, if it is saved at all, severed from the record it annotates. In our quest for order, we strip away the context, the friction, and the narrative that gives the data its soul and its true evidential value.

The Sterile Garden of Perfect Information

What we are left with is a sterile garden where every plant is identical and rows are perfectly straight. It is a beautiful, useless fiction. The real world is messy. The truth is often found in the inconsistencies: the two spellings of a street name that reveal a neighborhood’s linguistic history, the duplicate entries that point to a bureaucratic error with real-world consequences, the blank fields that signify something more profound than mere absence. A tidy dataset hides its own history. It conceals the corrections, the debates, and the mistakes that were part of its creation. It presents itself as a pristine, objective truth, when it is, in fact, a heavily edited and curated version of events.

This obsession with tidiness is a form of gatekeeping. It assumes that the value of public records lies solely in their ability to be ingested by Python scripts and Tableau dashboards. It disregards the researcher, the journalist, or the curious citizen who might learn more from the chaotic, unvarnished original—the scan of the crumpled paper form, the un-normalized database dump, the version history of a document. This raw material tells a richer, more complex story. It allows us to see the seams, and seeing the seams is how we learn how the garment was made.

This is not a call for chaos. Structure is necessary. But it is a plea for humility and transparency in our preservation. When we publish open data, we must provide not only the "clean" version but also the "raw" one, with a clear ledger of the changes made. We must document our choices, our schemas, and our rationales for what was altered or removed. We must value the scan as highly as the spreadsheet. Because an archive that only keeps the tidy summary is an archive that has already chosen the story it wants to tell, and in doing so, has silenced the more interesting, complicated truth.

Notes & further reading

A few pages I came back to while writing this: