Reader-level annotations and analysis code for: Binary Referable Evaluation Conceals Half the Annotation Error in the APTOS 2019 Diabetic Retinopathy Benchmark

Reader-level annotations and complete analysis code supporting a grade-stratified audit of annotation error in the APTOS 2019 Blindness Detection dataset. Contents: release/reader_level_data.csv — A de-identified reader-level dataset covering the 190 sampled images, carrying for each image the dataset label, the sampling stratum, the three readers' independent ICDR grades (0-4, or U for ungradable, with the third reader's grade present on the 22 images the first two graded differently) and a duplicate-pair flag. The reference standard is derived from these grades by the deposited rule; it is not shipped as a column. The records carry no image identifier, and the key mapping them to APTOS filenames is withheld permanently, so the deposit does not identify which specific public images a reader graded as mislabeled. Fundus images are not redistributed; they are obtained from Kaggle under the original terms. release/duplicate_manifest.csv — A manifest of the byte-identical duplicate groups found in the dataset, carrying one row for each of the 251 member files of the 123 groups and giving the group's MD5 digest and size, the member's public APTOS image identifier, the grade the official label file assigns it, and whether the group's labels conflict. REFERENCE_STANDARD_RULE.md — The adjudication rule, fixed in writing on 31 July 2026 before the deciding reads were collected, with its original timestamp intact. PROTOCOL_AMENDMENTS.md — Dated log of every deviation from the pre-specified analysis plan. READER_INSTRUCTIONS_READER2.md and READER_INSTRUCTIONS_READER3.md — The written instruction sheets issued to Readers 2 and 3, reproduced verbatim. scripts/ — Eleven Python scripts, an itemized list rather than a working directory, each one described in the deposit's README. They reproduce every reported reader-based value, including the design-weighted estimator, the Dirichlet-Jeffreys posterior used for interval estimation, the coverage simulation together with its recorded output (scripts/coverage_simulation_output.txt), the duplicate audit, and the checks of the hand-written statistics against reference implementations. Reproducing the reported values, from the root of the extracted deposit, with no images and no withheld files: python3 scripts/final_intervals.py --source release — the headline estimates with their credible intervals, under the primary reference standard, the two single-reader standards and the Grade 3 exclusion. python3 scripts/build_results_json.py --source release --out results.json — a machine-readable archive of every estimate the paper reports. python3 scripts/reported_quantities.py — the quantities the Results and Discussion give in prose rather than in a table. python3 scripts/build_tables.py --source release — Tables 1, 2 and 2B. python3 scripts/validate_against_reference_impls.py — the agreement statistics, and the checks of the hand-written kappa and Dirichlet code against scikit-learn and NumPy. Two exclusions are stated rather than left to be discovered. The duplicate counts are carried in the deposited manifest, but regenerating them the way they were obtained, by hashing the 3,662 distributed files, additionally requires the Kaggle download. The model's per-image output is not released, so the model-versus-label comparison, the model test-retest check, and the derivation of the ten sampling-cell sizes from that output cannot be recomputed inside the deposit; those ten sizes are deposited as a recorded constant of the design, so every reported estimate still recomputes. Every estimate, interval and table reported in the paper is reproduced using the Python standard library alone. Three scripts need more, and none of them is needed for a reported number: scripts/build_figures.py requires matplotlib to render the figures and Pillow to flatten them for TIFF output, and scripts/validate_against_reference_impls.py and scripts/zero_event_comparison.py require scikit-learn, NumPy and SciPy to check the hand-written statistics against reference implementations. Each of the three states its own requirement and exits with an install message rather than a traceback if it is unmet.