londons-pulse-moh

# London's Pulse: Medical Officer of Health reports (page images + OCR text) Page-level scans of the Wellcome Collection [**London's Pulse**](https://wellcomelibrary.org/moh/) Medical Officer of Health (MOH) reports (1848–1972), paired with OCR text, per-report licence, and full provenance. Built for **OCR / VLM / document-understanding** work on real historical public-health records — dense statistical tables, mixed layouts, century-old print. ## Configs | config | rows | what | |---|---|---| | `default` | 391,964 pages / 4,886 reports | every **page image** + its report's OCR text + licence | | `tables` | 3,000 table pages / 288 reports | pages that **contain a table**, each paired with that page's **extracted-table ground truth** — dual-purpose: page-type/classifier training **and** an OCR/VLM **table-extraction eval set** | ```python from datasets import load_dataset # full corpus (stream — it's ~110 GB) ds = load_dataset("biglam/londons-pulse-moh", split="train", streaming=True) # tables subset (small; image + ground-truth tables) tab = load_dataset("biglam/londons-pulse-moh", "tables", split="test") ``` ## `default` config — columns | column | type | description | |---|---|---| | `image` | `Image` | the page scan (full native resolution) | | `b_number` | `string` | Wellcome report id (join key) | | `page_index` | `int32` | 1-based page (scan) number within the report | | `n_pages` | `int32` | total pages in the report | | `report_text` | `string` | OCR text of the **whole report** (see caveat) | | `license` | `string` | normalised image licence (all `cc-by-nc` here) | | `table_ground_truth` | `string` | reserved (empty in `default`; populated in the `tables` config) | | `signals` | `string` | JSON provenance (`manifest_url`, `image_service_id`, size/format, `license_raw`, multi-volume linkage) | > [!IMPORTANT] > **`report_text` is report-level, not page-aligned** — the source OCR is a flat per-report > dump with no reliable page boundaries, so the same full-report text repeats across every > page of that report. Group/dedupe by `b_number`. ## `tables` config — columns One row per **table page**: the page image plus the machine-extracted table(s) Wellcome published for that page. Use it two ways — (1) as `table`-labelled training data for page-type/layout classifiers (the `table` class is scarce in book-domain sets), and (2) as an **OCR/VLM table-extraction benchmark**: feed `image`, score the model's output against `table_ground_truth`. | column | type | description | |---|---|---| | `image` | `Image` | the page scan | | `page_type` | `string` | `"table"` | | `table_ground_truth` | `string` | JSON list of `{table_id, csv}` — the extracted table(s) on that page | | `n_tables_on_page` | `int32` | number of tables on the page | | `b_number`, `page_index`, `printed_page` | | report id, scan index, printed page | | `source_collection`, `label_source`, `split`, `signals` | | provenance (`label_source="table-export"`) | Stratified across all 12 decades; grouped train/val/test split (a report is wholly in one split — no page leakage). `table` = a page from which Wellcome extracted ≥1 table (so a page may also contain prose); printed→scan alignment via IIIF canvas labels, verified, with composite/multi-section reports excluded. > [!NOTE] > **Ground-truth quality — silver, not gold.** The tables are **machine-extracted** (OCR-based table recognition), not hand-transcribed. Spot-checks against the page images show **numeric cell values are highly accurate** (often exact), while **captions/text labels carry occasional OCR errors** (e.g. `classes`→`olasses`, `(S.1)`→`(S.l)`), multi-level headers are flattened, and sparse tables have imperfect empty-cell alignment / inconsistent nil markers (`-`, `_`, blank). Each CSV is prefixed with an id line, a `Page,NNN` line, and the table caption. **Use as a silver benchmark target**: score numeric cells exactly, fuzzy-match text, and don't penalise a model for the GT's own OCR noise. A small hand-corrected gold subset is recommended for headline results. ## Provenance & reproducibility - **Images**: Wellcome [IIIF Presentation v2 API](https://iiif.wellcomecollection.org/), keyed on each report's b-number, full native resolution. - **Text** (`default`): the bulk `Fulltext.zip` corpus from [wellcomelibrary.org/moh](https://wellcomelibrary.org/moh/), joined on b-number. - **Tables** (`tables`): Wellcome's `All_Report_Tables` export (~275k machine-extracted tables); each table's `Page,NNN` line resolves to a scan image via the manifest's numeric canvas labels. - `signals` records the exact manifest + image service for every page. ## Licence Page **images** are **CC-BY-NC 4.0** (per Wellcome's IIIF manifests; per-row `license` + `signals.license_raw`). The OCR **text** corpus is CC-BY 4.0; the extracted-tables export is CC-BY 4.0. Reports whose image licence was not open were excluded. **Reuse is non-commercial**, with attribution to **Wellcome Collection**. | licence | pages (`default`) | |---|---| | cc-by-nc | 391,964 | ## Source & attribution Wellcome Collection, *London's Pulse: Medical Officer of Health reports 1848–1972*. Images: `iiif.wellcomecollection.org` · Text/tables: `wellcomelibrary.org/moh`. ## Intended uses Historical OCR/VLM evaluation, document layout analysis, page-type classification, and structured extraction from century-old public-health tables (disease incidence, mortality). The `tables` config is the substrate for an image→table extraction benchmark.