Lessons learned from twelve years’ operation of the Web Archiving Project (WARP)

The National Diet Library (NDL) has been operating the Web ARchiving Project (WARP) since 2002, to collect and keep available for future access websites published in Japan. This paper describes the purpose of, history behind, and system used for this project, and introduces actual case studies to demonstrate the challenges faced in fulfilling the potential of this project. WARP has been attempting to create a comprehensive archive of websites published by public agencies in Japan, as prescribed in the 2010 revision of the NDL Law. It also archives, with permission of the publishers, the websites of private universities, websites promoting cultural or international events held in Japan, and websites related to the Great East Japan Earthquake. As of March 2015, the archived content reached 85,764 items, comprising 533 TB of data and 3.1 billion files. WARP was created using Open Source Software (OSS), such as Heritrix, Wayback and Solr, with some original software and user interfaces. Publications significant for public use, which are included in the collected websites, are cataloged individually, and made accessible together with other digitized materials. WARP metadata can also be searchable via other integrated search services. Some public agencies even guide their users to WARP in order to ensure access to older information that is no longer available on their own websites. Since it does not seem practicable for individual public libraries in Japan to conduct web archiving on their own, the NDL will take a step further in promoting WARP within the framework of digital resource sharing programs. We consider this an important part of the NDL’s mission as a national library responsible for disseminating cultural heritage through configuration of platforms and networks for digital resource sharing.