The Quiet Harvest: How to Build a Simple Web Archiver with Python and Wget
We talk a lot about the importance of web archiving, of preserving digital moments before they vanish into the ether. It can feel like a monumental task, one best left to large institutions with crawling budgets and server farms. But the most meaningful archives often start small, personal, and hyper-specific. You don’t need to save the entire internet; sometimes, you just need to save one thing, well.
I want to show you how to build a simple, automated web archiver. This isn't about complex software or expensive services. We're going to use two humble, powerful, and free tools: a few lines of Python code and the venerable command-line program Wget. This combination creates a 'quiet harvester'—a script that can diligently and respectfully preserve a single website or a section of the web on a schedule you set.
The Tools of the Trade
First, ensure you have Python installed on your computer (it likely already is). Next, you’ll need Wget. On Linux and Mac, it’s usually pre-installed. Windows users can easily download it. The magic of Wget is its ability to recursively download websites, following links to grab not just the page you point it at, but the pages it leads to, along with all the images, stylesheets, and other assets that make it whole.
Our Python script won't do the downloading itself. Instead, it will act as a scheduler and a commander, telling Wget what to do and when to do it. This separation is key. It lets us use the robust, battle-tested downloading power of Wget while adding a layer of automation and logging with Python.
The core of the script is simple. We use Python’s `subprocess` module to call Wget with a specific set of instructions. A basic command might look like this:
wget --recursive --page-requisites --adjust-extension --convert-links --restrict-file-names=windows --wait=5 --random-wait --limit-rate=500k --no-parent --timestamping --directory-prefix=./archive https://example.com
This string of commands tells Wget to: download recursively, grab all necessary files, make the files work offline, wait politely between requests to avoid hammering the server, and save everything into an 'archive' folder. The `--timestamping` flag is crucial—it means Wget will only download files that have changed since your last visit, making subsequent runs efficient.
Building the Loop
The Python script wraps this command in a `while True` loop, runs it, and then uses `time.sleep()` to pause for a set interval—say, 24 hours—before running it again. You can add a simple timestamped log message each time the harvest begins and ends, creating a small ledger of your preservation attempts.
You run this script once, and it sits in the background of your machine, a silent custodian. Every day, or week, or month, it wakes up and checks on your chosen corner of the web. It gathers what is new and changed, and tucks it away into your growing archive.
This isn't the answer for every archiving need. It’s a gentle, personal approach. It won't handle complex JavaScript-heavy sites as well as a browser-based tool might, and you must always be mindful of a website’s `robots.txt` rules. But for preserving a blog, a news section, a digital project, or any relatively static site, it is remarkably effective. It turns the abstract ideal of preservation into a concrete, quiet practice. You are not just hoping something stays online; you are ensuring a copy exists, right there on your own drive, one day at a time.
Notes & further reading
A few pages I came back to while writing this: