The Humble Robots.txt: How to Read the Archive's 'Do Not Enter' Signs

We often think of web archives as vast, all-seeing libraries, capturing every public whisper of the internet. The reality is far more nuanced, shaped by a quiet, decades-old protocol that acts as a polite request: the robots.txt file. For archivists and researchers, understanding how to read these files isn't just technical trivia; it's the key to understanding the intentional gaps in our collective memory.

A robots.txt file is a small text document placed in a website's root directory. Its purpose is simple: to instruct automated web crawlers—be they from search engines or archives—which parts of the site they are allowed to access. A line like 'Disallow: /private/' is a clear 'keep out' sign. For live search engines, this is a crucial courtesy. For web archives, however, it becomes a complex ethical and practical directive. Major archives, like the Internet Archive's Wayback Machine, generally honor these exclusions out of respect for a site owner's wishes, even long after the original site has vanished.

So, how do you practically investigate this for yourself? The technique is straightforward. When you find an archived page, don't just look at the snapshot. Look at the structure of the URL. Strip it back to its domain root. For example, if you're looking at 'https://web.archive.org/web/20231001120000/https://www.example.com/old-page.html', the target domain is 'www.example.com'. Now, construct a new Wayback Machine URL to check for a saved robots.txt file: 'https://web.archive.org/web/20231001120000/https://www.example.com/robots.txt'. Simply replace the original page's path with '/robots.txt'.

What you find can be revealing. A disallowed '/archive/' path on a news site might hint at a trove of unpublished drafts or internal content that was never meant to be public. A disallowed '/cgi-bin/' directory on an old academic site might signal where dynamic, database-driven content lived—content that was often impossible for early crawlers to capture meaningfully. By reading the robots.txt file contemporary to your archived page, you move from being a passive viewer of what was saved to an active investigator of what was intentionally left out. You're not just seeing the archive; you're seeing the shape of its absences.

This practice transforms the robots.txt from a simple technical filter into a historical document itself. It reveals the anxieties and intentions of a website's stewards at a specific point in time—what they considered private, dynamic, or unimportant. The next time you dive into a web archive, take a moment to seek out this humble file. It is the quiet custodian of the gaps, and listening to it tells a richer, more complete story of what we've chosen to remember, and what we've asked to be forgotten.

Notes & further reading

A few pages I came back to while writing this: