Cross-Paper Result Comparability: A Dataset, Protocol-Partitioned Leaderboards, and a Census

Cross-Paper Result Comparability is a provenance-linked resource for studying whether quantitative results reported in different machine-learning papers can validly be compared. It is built from a frozen Papers with Code snapshot dated 2025-07-28 and a curated sample of 1,625 arXiv papers. Contents: 3,058 candidate cross-paper disagreement pairs: two papers reporting the same canonical (method, dataset, metric) cell with a differing value. Each pair carries pointers to both source papers (arXiv identifier, URL, section or table location), the values, a beyond-noise decision, a model-suggested comparability label (explicitly not gold), and an identity grade. A 200-pair human-labeled reference subset with cause labels and reliability flags. Protocol-partitioned leaderboards: 34,247 entries across 4,438 leaderboards, grouped by observed protocol signature. Of these, 16,215 are ranked within observed-protocol clusters and 18,032 are flagged comparability-unknown rather than silently ranked. An author-annotated census of disagreement causes, comparability certificates, a datasheet (following Datasheets for Datasets), a JSON schema, a Python loader, and SHA-256 manifests. Honest scope (documented in the datasheet): Model-suggested labels are navigation aids, not gold. Only the 200-pair census subset is human-labeled, by a single annotator, with intra-annotator reliability reported. No third-party paper text is redistributed. Each record points to its source, and the reader consults the paper directly. The partition reliably flags incomparability, but a shared observed-protocol cluster does not by itself guarantee comparability. License: the data, annotations, and partitioned leaderboards are released under CC-BY-SA 4.0 (inheriting Papers with Code's CC-BY-SA terms); the loader and supporting scripts are under the MIT License. A companion paper describing the resource is under review (JCDL 2026 Resources Track). See the repository README, DATASHEET.md, and CITATION.cff for full documentation and citation details.