De-identified item-level scoring data: LLM as a second rater for standardized patient history-taking examinations

Item-level scoring dataset for the manuscript 'Transcript-based large language model scoring as a second rater for standardized patient history-taking examinations: a retrospective adjudication study' (JMIR Medical Education, under review; manuscript ID 110683). 8,405 checklist items from 240 real high-stakes standardized-patient history-taking encounters (92 students, six cases), each scored by the on-site examiner, by three LLMs (DeepSeek-V4-Pro, DeepSeek-V4-Flash-0731, Qwen3-Next-80B-A3B-Instruct), and, for the 882 discordant items, by three blinded faculty experts. Encounter and student identifiers are anonymized; no names, audio, or transcripts are included. run_analysis.py reproduces the manuscript's headline analyses; v2 added an anonymized examiner identifier (E01-E12) and cross-classified sensitivity analyses; v3 (second-round revision) adds the October 2026 two-pass re-scoring data (october_rescore_analysis_set.csv), complete development-set expert scores (development_set_scores.csv), a script reproducing every reported stability number from the two public CSVs alone (reproduce_p4_from_public_data.py), the whole-examination accuracy computation (whole_exam_accuracy.py), and the re-scoring pipeline (rescore_20261023.py, analyze_rerun_20261023.py). Authors: Jun Feng, Shaoting Wang, and Xiaoxing Gao contributed equally. Corresponding author: Xuefeng Sun ([e-mail]). Ethics approval: Peking Union Medical College Hospital Ethics Committee (No. I-26PJ0511).v4 (24 October 2026): fixes a column-labelling error in development_set_scores.csv (sp_case now reads SP01–SP06) and rewrites whole_exam_accuracy.py to compute all counts live from the CSV. No numbers change. v5 (31 October 2026): completes october_rescore_analysis_set.csv to all 240 analysis-set encounters (the four previously untraceable encounters were resolved via examination-room rosters, secondary-model score sequences, and expert adjudication records); adds reproduce_round3_numbers.py. Reported stability figures are unchanged to the reported precision. v6 (31 October 2026, later the same day): fixes a NaN-comparison slip in the majority-vote section of reproduce_p4_from_public_data.py (the 11 items with no three-run majority had been counted as discordant; corrected: 807/8,394 defined-vote discordances, 818 items routed). Accuracy figures unchanged. v7 (4 November 2026): examiner identifiers replaced with a random permutation (seed 20251110) unrelated to room numbers, following an editorial comment; scores and all other columns unchanged, no analysis result affected.