The Unwritten Rules of the Wayback Machine: What Gets Passed Over?
We often talk about the web archives we have, celebrating the terabytes of data saved from the digital void. The Internet Archive’s Wayback Machine is a modern marvel, a library of Alexandria for the digital age. But every library has its gaps. Every collection is shaped by conscious and unconscious decisions. For every page saved, another is passed over. The question that nags at me isn’t about what’s been preserved, but what are the unwritten rules that determine what gets left behind?
It’s easy to assume the archive is a neutral, all-seeing eye. The reality is far more human. The process begins with crawling, and crawlers follow rules. They are programmed to respect robots.txt files, the small text documents websites use to politely ask automated agents to stay out. A well-meaning webmaster, perhaps trying to conserve server bandwidth or protect user privacy, might add a Disallow: / directive. It’s a simple act, but its effect on history is profound. That entire swath of the web, with all its conversations, its culture, its context, becomes invisible to the primary tool we have for remembering it. The archive, in its politeness, agrees to forget.
Then there’s the matter of links. Crawlers largely traverse the web by following hyperlinks from one page to another. This creates a inherent bias toward the well-connected, the popular, the central hubs. But what of the digital cul-de-sacs? The small, personal Geocities page that no one linked to, the forum thread buried deep within a now-defunct site, the online art project that existed solely at one unique URL—these are the first to vanish. Their obscurity is their vulnerability. Without a path leading to them, the crawler’s net simply drifts by.
The Silence of the Un-Crawled
Beyond the technical, there are the logistical and economic constraints. The web is impossibly vast, and crawling it is incredibly expensive in terms of time, storage, and processing power. Archivists must make choices. They prioritize certain top-level domains (.gov, .edu) or seed their crawls with specific lists of important websites. This is a necessary triage, but it means the ‘everyday web’—the small business sites, the local club pages, the forgotten blogs of teenagers—often slips through the cracks. It’s not deemed unimportant, but it might be deemed impractical to save.
This creates a historical record that is skewed toward the institutional and the popular, potentially at the expense of the personal and the marginal. We are, in a way, building a history of the web written by its winners and its most central players. The quiet, the private, the unlinked, and the politely refused—these spaces form a silent archive of their own, a negative image of what we’ve collectively decided to keep. Understanding these gaps isn’t a criticism of the monumental work being done; it’s an essential step in reading the archive critically, and perhaps, in finding ways to whisper a few of those forgotten names back into the record.
Notes & further reading
A few pages I came back to while writing this: