Reading or Reciting? Source Library OCR-Eval v0.3: reference-scored OCR of historical printed and manuscript pages, with a memorization control

Reference-scored OCR observations on 44 pages of early-modern printed books and manuscripts that Source Library (sourcelibrary.org) holds, in Greek, Armenian, Latin, Chinese, German and Hebrew. Every reference passage is labelled canonical (texts that vision-language models have plausibly memorised) or non-canonical (prefaces, front matter, passages of rarely digitised works). The gap between the two on matched pages estimates how much better a model scores on text it can recite than on text it can only read. Four within-work pairs hold a canonical and a low-canonicity passage of the same work in the same scan. Files. v0.3/pages.jsonl: 44 pages with book metadata, visually audited layout and canonicity covariates, and measured image resolution. v0.3/references.jsonl: 44 reference passages with source, licence and sha256; the text is included for 27 and withheld for 17 whose source does not permit redistribution or whose licence could not be confirmed. v0.3/runs.jsonl: 1,737 raw model outputs from about 15 model and serving arms (Gemini 3.x, Claude Sonnet, Mistral-OCR, Qwen3-VL, DeepSeek-OCR, Gemma 3 and 4, and the Source Library pipeline), with alignment and character accuracy. v0.4-difficulty/pages.jsonl: a companion set of 334 hard pages (20 difficulty classes) for reference-free measures; it has no references and no runs. Each folder has a README with field definitions and caveats; checksums.txt covers every file. Files in folders are named folder__file on Zenodo. Limits. 44 pages and unbalanced cells; canonical scores are memory-assisted upper bounds; the serving provider is part of the system under test; scoring_version was not stamped, so re-score from the raw text for a pinned metric. Licences. v0.3/references.jsonl is CC BY-SA 4.0 (it contains CC BY-SA text from Wikisource, First1KGreek, the Deutsches Textarchiv and Sefaria; each row names its source). All other files are CC BY 4.0. Page images are not included. Contains a canary GUID (CANARY.txt): do not train on this data. Exported by scripts/eval/export-eval-dataset.mjs and assembled by scripts/eval/build-ocr-eval-deposit.mjs in the sourcelibrary-v2 repository (AGPL-3.0); issues #3235 and #7462. No reader or personal data.