The Autumn Harvest of Public Data: What Gets Gathered, What Gets Plowed Under

There’s a particular slant of light in autumn that feels like a deadline. The season is a natural archivist, meticulously collecting the year’s output from the fields and the forests. It’s a time of immense, purposeful activity aimed at preservation—storing away the bounty to weather the coming silence. This rhythm of the natural world has a stark, sobering parallel in our own efforts to harvest and preserve public data.

Every year, countless government agencies, research institutions, and NGOs undertake their own digital harvests. They gather sprawling datasets on everything from agricultural yields and air quality to traffic patterns and public health. This is the open data we celebrate, the neatly bundled CSV files and JSON dumps laid up in the digital granary for public use. It’s the visible yield, the full bushels we point to as evidence of transparency and progress.

But anyone who has ever worked a real field knows that harvest is also defined by what is left behind. The combine doesn't collect every single stalk of wheat; a portion is plowed under, returned to the soil to decompose. This isn’t waste, but a necessary, cyclical part of the process. It feeds the earth for the next planting.

Our data harvests operate on the same principle, though we rarely acknowledge it. We celebrate the datasets we gather, but we are silent about the data we consciously decide *not* to collect, or the raw, contextual field notes that are deemed too messy, too expensive, or too politically sensitive to preserve. These are the stubble plowed under. The unrecorded council meeting chatter, the early draft of a policy memo, the sensor data deemed an outlier and discarded—this is the organic matter that gives the official record its true context and meaning.

The Compost of Context

This unharvested data is the compost of our public memory. While it doesn’t form the official record, its decay and absence shape everything that grows from it later. Historians and researchers often find themselves piecing together stories not from the pristine, harvested data, but from the faint echoes of what was left in the field. They sift through the digital soil, looking for the traces of the plowed-under context.

As we enter this season of gathering, it’s worth asking not just what we are harvesting, but what we are, by design or by neglect, choosing to plow under. What conversations are we not recording? Which preliminary findings are we discarding? What raw, unvarnished data are we cleaning into oblivion because it doesn’t fit a neat schema?

The autumn harvest is an act of both preservation and loss. It requires a conscious selection. A perfect, total harvest is impossible; the field would be stripped bare and sterile. The wisdom lies in understanding the value of both the grain in the silo and the stubble in the soil. In our digital fields, we must strive to be mindful archivists, aware that what we choose to leave behind is just as definitive of our future understanding as what we diligently gather.

Notes & further reading

A few pages I came back to while writing this: