Agreement Between Large Language Models and Humans in Research Proposal Review - Data and Scripts

Agreement Between Large Language Models and Humans in Research Proposal Review — Data and Code This repository contains the data and code required to reproduce the analyses, statistical results, and figures presented in the associated manuscript. Files are organized by function and described below. All research proposals are anonymized and labeled with non-identifying identifiers (A, B, C, …). Reviewer identities were never provided to the authors. To prevent inadvertent disclosure, all free-text review content from both human reviewers and large language models (LLMs) has been removed; only the numerical evaluation data required to reproduce the reported analyses are included. Data files Human_raw_scores.csv Individual numerical scores assigned by human reviewers, one row per reviewer × proposal × criterion. Used to compute panel-level summary statistics and the inter-reviewer reliability metrics reported in the manuscript. Human.csv Proposal-level human reference scores (one row per proposal) used as the human benchmark against which LLM scores and rankings are compared. LLM_data_combined_clean_filtered.csv All numerical scores generated by the evaluated LLMs. Preprocessed to remove evaluations in which a model failed to return one or more required numerical scores (see Data provenance and known limitations for counts). review_criteria.txt The evaluation criteria and rating scales. See the note under Data provenance regarding the 2023 vs. 2024 criterion naming. Data dictionary Human_raw_scores.csv Column Description Applicant Anonymized proposal identifier (A, B, C, …). Cycle Review cycle the proposal belongs to (2023 or 2024). Label Scoring criterion: Intellectual merit, Potential for impact, Collaborative Potential, or Overall ranking. Reviewer_Seq Reviewer index within a proposal (1, 2, 3, …). Identities are unknown; this index only links a single reviewer's ratings across criteria for the same proposal, in source-file order. It is not consistent across proposals (reviewer 1 for proposal A is not reviewer 1 for proposal B). Rating Numerical score. Criterion ratings use a 1–5 scale; Overall ranking uses a 1–3 scale (3 = fund, 2 = fund with modifications, 1 = do not fund). Human.csv Column Description Name Anonymized proposal identifier (matches Applicant above). Cycle Review cycle (2023 or 2024). IM, Impact, Collab, Overall Panel-mean scores for the four criteria. Score Weighted composite panel score, computed as 0.4·IM + 0.3·Impact + 0.3·Collab, matching the weighting applied to the LLM composite scores. LLM_data_combined_clean_filtered.csv Column Description Name Anonymized proposal identifier (matches Human.csv). Type Input given to the model: Abstract or Full_Proposal. Model Model identifier. For models with controllable reasoning depth, the tier is appended as a suffix (_low, _medium, _high). The portion before the first underscore is the root model. Prompt Prompting strategy (OneShot or CoT). Seed Requested random seed. For models that did not support seed specification at the time of execution (the reasoning models listed in the Methods), the API ignored this value and it functions only as a replicate index; output is not reproducible from it for those models. Temp Sampling temperature (0.1, 0.5, 0.9). IM, Impact, Collab Criterion ratings (1–5 scale). Overall Overall recommendation (1–3 scale). See limitation note on out-of-scale values. Score Weighted composite, 0.4·IM + 0.3·Impact + 0.3·Collab. Category Reasoning/architecture category of the model. Year Evaluation wave in which the run was performed (proposals were re-evaluated as new model generations were released); this is not the proposal's submission cycle. Use Cycle in the human files for submission cycle. Data provenance and known limitations We document the following so that users can interpret the data accurately. Two review cycles, combined. The 28 proposals come from two internal seed grant cycles: 15 from 2023 and 13 from 2024 (Cycle column). For 2024 proposals, proposal-level means in Human.csv are the official institute panel means; for 2023 proposals they are computed from the individual ratings in Human_raw_scores.csv. Reviewers per proposal. Panels ranged from 3 to 6 reviewers. Reviewer identities were never provided; the human inter-reviewer reliability is therefore estimated with a one-way random-effects model (ICC(1,1)), which is the appropriate model when each proposal is rated by a different, unidentified set of reviewers. Six 2024 reviews not present at the individual level. For six 2024 proposals (G, R, T, W, X, Z), one reviewer's scores were submitted without written comments and are not included in Human_raw_scores.csv. For these proposals, Human.csv carries the official institute panel means, so the panel mean in Human.csv and the mean recomputed from Human_raw_scores.csv differ slightly. The reproducibility check in 06_Human_data.ipynb confirms exact agreement for all proposals with complete individual records and reports the expected small differences for these six. Out-of-scale LLM Overall ratings. A small number of responses (≈1.2% of reviews, almost entirely from gpt-3.5-turbo) rated the overall recommendation on a 1–5 scale rather than the requested 1–3 scale, in a format the parser could not distinguish. The analysis code masks values outside the valid range before computing any Overall-based result; the composite Score does not use Overall and is unaffected. Excluded LLM evaluations. Evaluations in which a model failed to return one or more required numerical scores were removed prior to analysis. The file provided here is the post-exclusion (analyzed) dataset. Code notebooks Two groups of notebooks are provided: (i) the LLM evaluation pipeline and (ii) statistical analysis and figure generation. LLM evaluation pipeline (Notebooks 1–4) Documentation of the methodology used to generate the LLM evaluations. These use synthetic examples and contain no confidential data, API credentials, or real proposal text. 01_pipeline_overview.ipynb — architecture, configuration, criteria, output format, evaluation matrix. 02_prompting_strategies.ipynb — one-shot and chain-of-thought prompting; example selection; text vs. vision input. 03_response_parsing.ipynb — regex extraction of ratings and comments; error handling; decimal ratings. 04_example_evaluation.ipynb — end-to-end workflow on synthetic data. Statistical analysis and figures (Notebooks 5–6) 05_Data_Processing.ipynb — ANOVA and effect sizes; empirical absolute score differences; ICC(2,1) and Spearman correlations vs. the human panel; Monte Carlo comparisons; scatter, slope, and bias figures. 06_Human_data.ipynb — ICC(1,1)/ICC(1,k) for human reviewers with bootstrap confidence intervals; Monte Carlo single-reviewer vs. leave-one-out panel rank agreement; consistency check of Human.csv against Human_raw_scores.csv. Reproducibility Running 05_Data_Processing.ipynb and 06_Human_data.ipynb against the included data reproduces the statistical results and figures in the manuscript. Notebooks 1–4 document the evaluation pipeline using synthetic examples. Requirements pandas numpy scipy statsmodels scikit-learn matplotlib The LLM evaluation pipeline notebooks (1–4) additionally use the packages listed in requirements.txt. Citation If you use this data or code, please cite the associated manuscript and this archive: [Author list]. Agreement Between Large Language Models and Humans in Research Proposal Review. [Year]. Zenodo. https://doi.org/[concept DOI] License Data are released under CC BY 4.0; code is released under the MIT License.