The Astronomer's Ledger: On Borrowing Provenance from the Stars
For centuries, astronomers have faced a problem of scale not unlike our own in the realm of digital preservation. They observe a sky teeming with countless points of light, a seemingly chaotic and overwhelming dataset. Their solution, refined over generations, wasn't just better telescopes; it was a meticulous system of recording not just the observation, but the context of the observation. This practice, the unbroken chain of custody and calibration for a single data point, is called provenance. And it's a discipline we desperately need to borrow for our own terrestrial constellations of public data.
Consider a raw astronomical datum: the magnitude of a star measured on a specific night. To a future researcher, that number is meaningless noise without its accompanying metadata. It requires the telescope's calibration logs, the atmospheric conditions, the specific filter used, the software version of the reduction pipeline, and the identity of the observer. Each layer of context transforms a floating-point number into a credible, reusable piece of knowledge. This is the astronomer's ledger—a deep, intrinsic commitment to the story behind the data.
Now, look at a typical public dataset, a CSV file of municipal spending or climate readings. How often is it posted online as a naked, context-starved artifact? We get the numbers, but rarely the full story of their origin. Who compiled it? On what authority? What were the cleaning rules? Which software version generated it? Was a specific row excluded and why? This data is presented as a finished product, not as a snapshot in a continuous process of observation. We have the star's magnitude, but we've lost the logs for the telescope.
Applying the astronomer's method would mean packaging every public record with its own inherent ledger. A weather sensor's data dump would be inseparable from its maintenance history and calibration reports. A published government statistic would carry a lightweight, embedded manifest detailing its source systems, its transformation scripts, and its curators. This shifts the paradigm from data as a static publication to data as a recorded observation, complete with all the necessary caveats and credentials that make it scientifically valid.
This isn't merely an academic exercise. It is the difference between data that is simply available and data that is truly trustworthy and resilient over time. By learning from the astronomers, we can stop archiving mere numbers and start preserving credible evidence. We can build a practice where every data point, from a local budget line item to a national census record, comes with its own story intact, allowing future generations to understand not just what we recorded, but how and why we knew it to be true.
Notes & further reading
A few pages I came back to while writing this: