The Lost Libraries and the Index That Never Was
In the mid-19th century, before the internet, before microfilm, there was a man with a distinctly modern ambition: Antonio Panizzi, the fiery Principal Librarian of the British Museum. His dream was to create a universal catalog for the world’s largest library, a tool so comprehensive it would reveal the totality of human knowledge contained within its walls. He envisioned a grand, singular index, accessible and clear. His fight for this vision, and its ultimate partial failure, feels startlingly familiar today. It is a pre-digital parable about the immense challenge of making information not just collected, but truly open and navigable.
When Panizzi took the helm, the British Museum’s library was a magnificent but chaotic jumble. Books were cataloged inconsistently, if at all. Finding a specific volume was often a matter of luck or insider knowledge. It was a library with a closed data model, its riches locked away by poor indexing. Panizzi’s response was his famous 91 Rules for cataloging, a radical standardisation that demanded strict, uniform entries for author, title, and subject. He was, in essence, creating a metadata schema for a physical collection, arguing that without such rigorous structure, the library was useless. He famously stated that he wanted a poor student to be able to find a book as efficiently as the wealthiest patron. This ethos—accessibility through standardisation—is the very bedrock of modern open data principles.
His magnum opus was to be a mammoth printed catalog, a single, searchable (by human eye, of course) volume of everything. The project began, but the pace of acquisition quickly outstripped the pace of cataloging. The index began to fall behind, volumes piled up, and the dream of a complete, up-to-date central index started to crumble. The ‘data’ was growing exponentially, and the ‘system’ couldn’t keep up. It’s a problem any web archivist wrestling with the scale of the modern internet will recognise instantly. The sheer volume of information defeated the dream of a perfect, monolithic index.
So, what happened? The library didn’t abandon the catalog. Instead, it evolved. It moved to a distributed model. They began creating individual slips for each book, which could be filed and refiled. The single, authoritative volume gave way to a card catalog—a system that was inherently more flexible, more expandable, and more resilient. It was a move from a brittle, centralised database to a modular, adaptable one. The ‘index’ became a living, growing entity, not a frozen monument.
Panizzi’s story is a powerful reminder for those of us working with digital preservation and open records today. The ambition to create the ultimate, perfect archive—the Wayback Machine for everything, the one true dataset—is noble. But history suggests that our efforts might be more enduring if we focus on creating robust, standardised, and interoperable pieces: the digital equivalent of those catalog slips. The goal isn’t necessarily a single, perfect index, but a network of well-described, accessible fragments that, together, can be reassembled into countless different pictures of the past. The legacy of the British Museum’s catalog isn’t the book that was never finished, but the system that learned to adapt, a lesson in building for flux rather than fighting it.
Notes & further reading
A few pages I came back to while writing this: