MarCompBench v1 — reference and model verdicts for DP operations manual gap analysis

Superseded by v1.2.0 (DOI 10.5281/zenodo.21365194), the citable version. The files of this superseded version are restricted. A benchmark for evaluating compliance-gap assessment of Dynamic Positioning (DP) Operations Manuals against a 99-item requirement set derived from published MTS/IMCA DP operations guidance, together with the first inter-rater reliability data for that task. This record accompanies the paper: Torgersen (2026), Machine reproducibility and human disagreement in maritime compliance assessment: evidence from a triangulated benchmark on a DP operations manual. Headline finding: three qualified reference assessments of the same DP Operations Manual agreed on 37–51% of items pairwise (Cohen's κ 0.13–0.31; Gwet's AC1 0.19–0.38; Krippendorff's α 0.21 nominal, 0.29 ordinal on the three-point coverage scale), below standard reliability thresholds. Of 98 jointly rated items, 26 were unanimous, 56 split two against one, and 16 produced three different verdicts. The eight-model panel, run under one shared system prompt, agreed on 68.2% of items pairwise (mean; median 70.7%), and the three open-weight models returned identical verdicts on every item across repeated runs. The record contains verdict codes and derived statistics only: the requirement-item structure (ids, categories, source-guidance citations), the two grading references' per-item verdicts, all eight models' verdicts across three runs, the four-cell Mistral Large prompt ablation, the per-item override and endorsement record, the full statistics file, and a dependency-free deterministic script (compute_agreement.py) that regenerates the statistics from the CSVs. The third reference's verdict codes derive from a third-party controlled professional report and are held in a restricted companion record (DOI 10.5281/zenodo.21357285), available on request for verification under confidentiality. The requirement-item texts are the intellectual property of the Marine Technology Society and are not redistributed here; download TECHOP O-01 and its companion spreadsheet free of charge from the MTS DP Committee at dynamic-positioning.com and join on item order. See README.md for full documentation, scale conventions, and licensing (CC-BY-4.0 data, MIT code).