How to Unshrink Your Web: A Guide to Playing with the Internet Archive's Wayback CDX API

We often think of the Wayback Machine as a kind of digital séance. You enter a URL, click a date on a calendar, and a ghost appears. It’s a powerful, almost mystical interaction. But behind that simple search box lies a database of staggering depth, accessible through a tool that transforms the archive from a museum into a workshop: the CDX Server API.

The CDX API is the engine room of the Wayback Machine. Instead of showing you a single, fully-rendered page, it hands you the raw index—the catalogue of all the snapshots it holds for a given website or even the entire web. This index is a simple text file, but within its rows of data lies the power to ask questions you can't ask through the web interface. You can, for instance, find every capture of a specific image file across the entire archive, track the evolution of a site's footer over a decade, or map out the precise moments when a government agency updated its policy documents. The web interface gives you a framed painting; the CDX API gives you the curator’s card catalogue, the provenance notes, and the keys to the storage room.

Your First Foray into the Index

Let's do something concrete. You don't need to be a programmer to start; a web browser will suffice. The basic syntax for a query is a URL. Let's say you want to investigate the life of a single webpage. Point your browser to this address:

https://web.archive.org/cdx/search/cdx?url=example.com&output=json

Replace `example.com` with the URL you're curious about. The `output=json` parameter tells the API to return the data in a structured format that's relatively easy to read. What you'll get back is a list. Each entry represents a snapshot. You'll see a timestamp (like `20220102130405` for January 2nd, 2022, at 1:04:05 PM), the original URL, the MIME type, the HTTP response code, and a hash. This list is the chronology of that URL's existence in the archive.

Now, let's get more specific. What if you wanted to see only captures from a specific year? You can filter by a timestamp prefix:

https://web.archive.org/cdx/search/cdx?url=example.com&from=2022&to=2023&output=json

The `from` and `to` parameters let you define a temporal window. Suddenly, you're not just looking at a timeline; you're focusing a microscope on a specific period.

This is useful, but the real power emerges when you start combining filters. Let's say you're researching a site that changed its design and you want to find all the old CSS files that defined its look. You could search for captures where the URL ends with `.css`:

https://web.archive.org/cdx/search/cdx?url=*.example.com/*.css&output=json

The asterisks act as wildcards. This query will return the index of every captured CSS file from every subdomain of example.com. You've just moved from studying a single page to analyzing the architectural components of an entire domain. You can then use the timestamps from these results to visit the actual archived CSS files in the Wayback Machine, watching the visual identity of a site evolve line by line.

Playing with the CDX API is an act of unshrinking the web. It shifts your relationship with the archive from passive observer to active investigator. You're no longer limited to the paths the Wayback Machine's interface offers. You can follow the digital breadcrumbs, uncover hidden patterns, and build your own narratives from the raw material of preserved time. It’s not just about what was saved; it’s about the questions you can now ask of what was saved.

Notes & further reading

A few pages I came back to while writing this: