The Unseen Index: How to Build a Personal Web Archive with Wget and a Text Editor

We talk a lot about the grand archives, the Internet Archive's Wayback Machine, the vast institutional repositories. But there's another layer of preservation, one that happens closer to the bone: the personal collection. It’s the set of pages, articles, or small sites you encounter that feel vital, that you know you’ll want to reference again in a year, or five, or ten, long after the live web has reshaped itself around their absence. The question isn't whether to save them, but how to save them in a way that remains useful and accessible.

Forget complex software or proprietary formats. The most durable tool for this job is often the simplest: the command-line utility wget. It’s a quiet workhorse, available on every major operating system, designed for one thing: retrieving content from the web. Its power for personal archiving lies in its ability to grab not just a single file, but an entire context—the page, its images, the local CSS—and structure it all logically on your hard drive.

Here’s the concrete technique. Open your terminal and navigate to where you want this little archive to live—a folder called ‘holdings’ or ‘library’. Then, use this command:

wget --recursive --page-requisites --convert-links --no-parent --wait=2 https://example.com/your-precious-page

Let’s break down this incantation. --recursive gets linked pages, to a default depth of 5. --page-requisites is the magic flag that grabs all the assets (images, stylesheets) needed to display the page properly offline. --convert-links rewrites the links in the downloaded files to point to each other locally, not back to the live web. --no-parent keeps Wget from crawling up and away from the specific content you care about. And --wait=2 is a gesture of goodwill, a two-second pause between requests to be polite to the server.

Building the Finding Aid

You now have a functional, navigable copy on your machine. But a box of files is not yet an archive. An archive needs a finding aid. This is where the text editor comes in. Open a new file called index.txt right there in the folder.

In it, write a brief entry. Note the original URL. Add the date you captured it. Write a sentence or two on why it mattered. What was the context? What idea sparked the need to preserve this? This tiny act of metadata creation is what transforms a download into a curated item. It’s the note you slip into a physical book, the note your future self will thank you for. Over time, this index.txt file becomes the table of contents for your own unique collection of the web’s ephemera.

This method won’t capture complex web apps or deeply interactive content. It’s for the textual web, the essays and the documentation and the oddities. But its simplicity is its strength. The output is just HTML, CSS, and images in a folder—formats that are radically open and likely to be readable for decades. You’ve bypassed the need for any special software to view your archive. You own it, completely. You have not just the data, but the beginnings of its story.

Notes & further reading

A few pages I came back to while writing this: