The Garden Hose Protocol: Saving a Public Dataset Before the Spigot Closes

We talk a lot about the grand, automated crawls that preserve the web. The Internet Archive’s bots, national library projects—these are the fire trucks of digital preservation. But what about the garden hose? I’m talking about the small, vital, and often fragile datasets hosted on a single government or university server. The kind where you get a sinking feeling, a premonition, that the ‘temporary’ page holding the CSV files might not be there next budget cycle. You don’t need a fire truck for this. You need a protocol.

I call it the Garden Hose Protocol: a manual, mindful method for creating a usable, citable snapshot of a public dataset in an afternoon. It’s for when the API is a fantasy and the ‘download all’ button gives you one monolithic, poorly-documented file. The goal isn’t just to grab the data, but to capture enough context that someone (including your future self) can actually understand it six months later.

The Three-Jar Snapshot

First, you need three ‘jars’—three folders on your computer. Name them deliberately: 1_raw_source, 2_documentation, and 3_processed_snapshot. The magic is in the discipline of what goes where.

Start with 2_documentation. Before you touch a single data file, take screenshots. Capture the webpage that hosts the dataset. Scroll and stitch if you must. Get the ‘About’ page, the data dictionary, the contact email, the last updated date. Save the page as HTML (most browsers have a ‘Save Page As…’ option). This folder is your provenance. It answers the critical question: “What did the source say it was providing on the day I took it?”

Now, 1_raw_source. Download every file as offered. Do not rename them. Do not open and re-save them in Excel, which can silently mangle formats. Let the original filenames, with their cryptic codes and dates, remain intact. Their very obscurity can be a clue later. If there are multiple formats, take them all.

Finally, 3_processed_snapshot. This is where you make the data useful. Create a single, clean CSV or SQLite file with sensible column names. But crucially, include a README.txt in this folder that explicitly links your clean data back to the raw files. Write: “Column ‘precip_in’ corresponds to ‘PRCP_MM’ in raw file ‘weather_station_44a_2023.Q3.zip’, as described in the screenshot ‘data_dictionary_20231011.png’.” This creates the chain of custody.

The final, vital step is to create a checksum. On the command line, run shasum -a 256 * on your 1_raw_source folder and pipe it to a file called MANIFEST.txt. This string of characters is the dataset’s fingerprint. If anyone ever questions the integrity of your snapshot, this checksum proves nothing has altered.

This isn’t glamorous work. It won’t archive the entire web. But it addresses a specific, quiet vulnerability: the data that is public, but not persistent. It turns a nervous download into a curated, explainable package. You are not just hoarding bits; you are building a small, sturdy bridge between a moment of public access and an uncertain future. Sometimes, the most honest preservation tool isn’t a robot, but a gardener, carefully filling jars before the rain stops.

Notes & further reading

A few pages I came back to while writing this: