The Invisible URL: How to Map a Site Before It Vanishes

We talk a lot about saving pages, but sometimes the most critical act of preservation happens before you download a single byte. It’s the act of seeing the site as a structure, understanding its shape and connections. When a website is about to go dark—due to a domain lapse, a server migration, or simply a creator moving on—its greatest threat isn't the loss of the homepage. It's the loss of everything you never knew was there, the pages with no inbound links, the resources buried three clicks deep. To save it, you first have to map it.

This is where the simple, powerful technique of creating a site map comes in. Not an XML sitemap submitted to search engines, but a human-readable, navigational chart of the territory. For small to medium-sized static sites, this can be done with a free, command-line tool called wget, used not to download, but to probe. The command is deceptively simple: wget --spider --force-html -r -l 2 [URL] 2>&1 | grep '^--' | awk '{ print $3 }' | sort -u. Let's break down what this incantation does.

The Spider's Web

The --spider flag tells wget to only check for the existence of pages, not download them. --force-html treats files as HTML even if they have a non-standard extension, -r enables recursion, and -l 2 limits the crawl depth (you can increase this). The output is piped through a series of filters to extract just the clean list of discovered URLs. Run this against a personal blog, a defunct project page, or a small organizational site, and you'll get a raw list of every link the spider could find. It’s the blueprint of the place.

But the list is just data. The preservation act is in the analysis. Open that list in a text editor. Look for patterns. Are there directories like /notes/ or /old/ that you might have missed? Are there file extensions like .pdf or .zip hiding in corners? This map reveals the scope of the task. It shows you the lonely pages, the ones only linked from a single obscure archive page. These are your high-priority targets, the digital ephemera most likely to vanish without a trace.

This technique is humble, almost archaic. It doesn't require an API key or a fancy service. It acknowledges a fundamental truth: you cannot preserve what you cannot see. In the rush to grab a homepage with a bulk archiving tool, we often miss the cul-de-sacs and back alleys of a web space. Creating a map first forces a moment of contemplation. It turns the act of archiving from a blanket capture into a deliberate, informed rescue mission. You are no longer just saving a page; you are documenting a structure, understanding the creator's intent in how they linked their work together.

Next time you hear a site is on the chopping block, or you feel that familiar twinge of digital precarity about a corner of the web you cherish, don't start with the save button. Start with the spider. Draw your map. The list of URLs it produces is, itself, a minimal, durable record—a text-file testament to what was there, ready to guide your hand or, at the very least, to mark what has been lost.

Notes & further reading

A few pages I came back to while writing this: