The Unseen Cartographers: On the Lost Art of the Sitemap
We speak of web archiving in grand, sweeping terms: crawling, indexing, preserving petabytes of data for the ages. We marvel at the scale of institutions like the Internet Archive. But we rarely discuss the humble, human-made documents that were meant to guide these digital explorers—the sitemaps that websites once proudly offered, and which have now largely faded into obsolescence.
Before the era of sophisticated, automated crawlers that relentlessly probe a site's structure, there was a gentler, more cooperative model. A website maintainer, thinking like an archivist themselves, would craft an XML sitemap or, in an even earlier time, a simple HTML page listing every public-facing document. This was an act of cartography. It was a website saying, "Here I am. This is my complete form. Please, take a copy." It was a handshake between creator and preserver, a declaration of intent that every page was meant to be found.
Today, that practice has atrophied. The drive for dynamic, personalized content means many sites no longer have a "complete" form to present. Pages are generated on the fly, tailored to the user, and exist in a state of perpetual flux. The sitemap, if it exists at all, is a technical SEO requirement, a bare-minimum list of priority pages fed to search engines to ensure commercial visibility. It is no longer a true map of the territory; it is a tourist brochure highlighting only the main attractions.
This shift has profound implications for preservation. When an archiver's bot arrives now, it is often an uninvited guest, forced to deduce a site's structure from links and menus, guessing at the depth and breadth of what's available. It can easily miss entire sections hidden behind complex JavaScript or orphaned pages with no inbound links. The silent retirement of the comprehensive sitemap means we are losing a key tool of intentional, accurate preservation. We are relying on machines to reverse-engineer a human's design, rather than benefiting from the designer's own blueprint.
This loss speaks to a broader change in our digital philosophy. The web has moved from a library of documents to be cataloged to an endless stream of experiences to be consumed. The act of providing a full map was an act of acknowledging that a website was a finite, tangible collection—a thing that could, and should, be captured in its entirety. Its decline is the victory of the transient over the permanent, the interactive over the static. The unseen cartographers have put down their pens, and in their absence, we are left with explorers charting a coastline that shifts with every tide.
Notes & further reading
A few pages I came back to while writing this: