The Digital Iceberg: Two Approaches to Preserving the Modern Web

We often imagine web archives as vast, silent libraries, methodically capturing every page and pixel. But when it comes to preserving the modern web, a fundamental schism has emerged, creating two distinct philosophies that approach the task like a cartographer versus a deep-sea diver.

On one side is the surface crawler, exemplified by tools like the Internet Archive’s Wayback Machine. This is the cartographer’s approach: it meticulously charts the visible coastline of the web, capturing the HTML, CSS, and images that a browser renders. It aims for breadth and scale, creating a two-dimensional map of the digital landscape at a given moment. It’s an invaluable record of what was publicly presented, the official face of a website. But like any map, it simplifies the territory. It often misses the complex, dynamic data flowing beneath the surface.

Contrast this with the approach of projects like Webrecorder or Conifer. This is the deep-sea diver’s method. Instead of mapping the surface, it dives into a single, specific user session. It doesn’t just take a picture; it records the interaction. It captures the API calls, the WebSocket messages, the logged-in experiences, and the personalized JavaScript-rendered content that a surface crawler can never see. It preserves the web not as a static document, but as a functional, replayable event.

The difference is stark. A surface crawl of a social media feed might capture a list of usernames and post times. A session-based capture can preserve the infinite scroll, the auto-playing videos, the real-time notifications popping in, the very feel of being there. One is a photograph of a machine; the other is the machine itself, frozen in time.

Neither approach is inherently superior; they answer different questions for different futures. The cartographer’s archive asks, “What was publicly available on this day?” It provides context and breadth. The diver’s archive asks, “What was it actually like to use this?” It provides depth and fidelity. The modern web is an iceberg, with its most profound interactions hidden beneath the waterline of a simple HTTP request. To truly preserve it, we need both the maps that show its scale and the dives that reveal its hidden mass.

Notes & further reading

A few pages I came back to while writing this: