The Seductive Flaw in Our Obsession with Scraping

In the world of open data and web archiving, there is a sacred commandment: Thou shalt scrape. The logic is seductive and appears unassailable. Data is trapped on a website, hidden behind a labyrinthine interface, locked in a proprietary database. Our duty, we are told, is to liberate it. To build a bot, write a script, and pull that data into a clean, machine-readable format—a CSV, a JSON dump, a proper database. We celebrate these acts of digital emancipation as victories for transparency and preservation. But I want to propose a heretical idea: our fetish for scraping might be destroying the very evidence it seeks to preserve, leaving us with hollow data that has lost its crucial, inherent context.

When we scrape, we make an implicit and enormous claim: that we know what is important. We identify the ‘data’—the numbers, the names, the dates—and we discard the ‘presentation’—the layout, the surrounding text, the navigation, the visual hierarchy. We strip the wheat from the chaff. But what if the chaff is not just decorative chaff? What if the presentation is the data? The journey a citizen takes to find a public record on a government website—the clicks, the dropdown menus, the warnings, the footnotes—is a piece of civic testimony in itself. It tells a story about priorities, about bureaucratic logic, about who the site assumes its user to be. Scraping the final dataset erases that story, leaving us with the what but not the how or the why.

The Ghost in the Machine-Readable Format

This is the ghost we accidentally exorcise. A perfect, normalized database of municipal spending is a wonderful tool for analysis, but it cannot tell you that the original website buried the report seven clicks deep, labelled it with obscure jargon, and published it in a format that broke screen readers. That specific, material reality, which speaks volumes about the transparency (or lack thereof) of the institution, is vaporized by the very act of making the data ‘accessible.’ We have preserved the bone, but we’ve lost the flesh and blood that gave it meaning.

The web archiving community, with its focus on WARC files and mementos, understands this at a macro level. They try to capture the whole page. But even there, the pressure is often to then extract the ‘useful bits.’ The argument I’m making is more radical: sometimes, the ‘useful bit’ is the friction. It’s the difficulty. It’s the terrible user interface. To truly preserve a digital artifact as a public record, we need to resist the urge to clean it up for posterity. We must preserve the experience of the thing, not just its distilled content.

This isn’t an argument against scraping as a tool. It is an argument against scraping as a default philosophy. Before we write the script, we should ask: what evidence are we losing in the extraction? Are we, in our quest for pristine data, unknowingly committing an act of archival violence, smoothing over the rough, truthful edges of how information actually existed in its native habitat? The most complete preservation might not be a perfect copy of the data, but an imperfect, frustrating, and beautifully contextual copy of the original system. Sometimes, the lock and the labyrinth are the most important parts of the record.

Notes & further reading

A few pages I came back to while writing this: