icdar2021-historical-document-dating

# ICDAR 2021 Historical Document Classification — Task 2 (Dating) **13,810 manuscript page images** labelled with the period in which they were produced. Images come from [e-codices](https://www.e-codices.unifr.ch/), the virtual manuscript library of Switzerland. | Split | Images | Date range | Median span | Dated to a single year | |---|---|---|---|---| | train | 11,294 | 800–1899 | 45 years | 1,409 | | test | 2,516 | 800–1921 | 49 years | 264 | ## The label is an interval, not a year Palaeographers date a manuscript to a **range**, and the width of that range encodes their confidence. This dataset preserves that: - `date_start`, `date_end` — the range as published - `date_span` — `end - start`. **0 means dated to a single year.** - `date_mid` — midpoint, a convenience point-target Precision varies enormously: some pages are pinned to one year (`1505–1505`), others sit inside a **two-century** window (`1250–1449`). Any sensible model or metric has to handle that heterogeneity — a plain regression on `date_mid` throws away the confidence signal, and a plain accuracy metric treats a two-century guess as equal to a one-year one. Consider weighting by `date_span`, or scoring interval overlap rather than point error. ## Gotchas in the source deposit Two things will silently corrupt a naive build. Both are handled here. **The two ground-truth files use different delimiters.** ``` dating/training/gt.csv img/0_1400_1430.jpg,1400,1430 <- COMMA dating/test/gt.csv img/667cd1e1....jpg;1200;1299 <- SEMICOLON ``` Parse both the same way and the test split collapses into a single junk column — with a plausible-looking row count. Verified: with the right delimiters, all 2,516 test images join to a label, and all 11,294 train images do too. **The test labels are NOT missing — despite appearances.** Zenodo publishes a separate `task2-dating-test-meta.csv` alongside the archive that maps test images to their source manuscript and contains **no dates at all**. It is a provenance file, not the ground truth. The real test labels live inside `dating.tar.gz` at `dating/test/gt.csv`. Anyone glancing at the Zenodo file list would reasonably conclude this is a competition set with a withheld test split. It isn't. Also: training filenames encode the range (`5639_1467_1467.jpg`), but **test filenames are opaque hashes** — you must use the CSV. ## Load ```python from datasets import load_dataset # 28 GB of page images - stream unless you want the whole deposit on disk ds = load_dataset("biglam/icdar2021-historical-document-dating", split="train", streaming=True) # keep only tightly-dated pages; date_span == 0 means dated to a single year tight = ds.filter(lambda r: r["date_span"] <= 25) ``` ## Source & credit Mathias Seuret, Anguelos Nicolau, Dalia Rodríguez-Salas et al. *ICDAR 2021 Historical Document Classification Dataset for Task 2 — Dating.* Zenodo, 2021-05-28. <https://zenodo.org/records/4836687> — CC-BY-4.0. Images courtesy of e-codices, Virtual Manuscript Library of Switzerland. This repository converts the 28 GB deposit to Parquet. Please cite the original authors and the ICDAR 2021 competition. ```bibtex @dataset{seuret_2021_icdar_dating, author = {Seuret, Mathias and Nicolau, Anguelos and Rodr{\'i}guez-Salas, Dalia and Weichselbaumer, Nikolaus and Stutzmann, Dominique and Mayr, Martin and Maier, Andreas and Christlein, Vincent}, title = {{ICDAR 2021 Historical Document Classification Dataset for Task 2 --- Dating}}, year = {2021}, publisher = {Zenodo}, doi = {10.5281/zenodo.4836687} } ```