Supplementary Materials: Anonymized Student Assignments, AI Scoring Program, Inter-Rater Reliability Analysis Program, and Scoring Results Summary
收藏资源简介:
To support transparency and reproducibility, the following supplementary materials are provided alongside this manuscript: (1) Anonymized Student Assignments — A folder containing the complete set of student midterm assignments from the Social Research Methods course, used as the source texts for scoring by the human teacher and the two AI raters (Gemini and Kimi). All personally identifiable information, including student names, identification numbers, and document metadata, has been removed to protect participant privacy. (2) AI Automated Scoring Program — A Python script (an automated scoring program for the Social Research Methods midterm assignment.py) used to invoke the Gemini API and the Kimi API for automated scoring of the anonymized assignments. The complete prompts provided to both AI raters are embedded within the script as string variables, ensuring that the exact instructions used for AI-based scoring are fully documented and reproducible. API keys have been replaced with placeholders; users wishing to re-run the script must supply their own credentials. (3) Inter-Rater Reliability Analysis Program — A Python script (Inter-rater reliability analysis among three raters.py) implementing the statistical procedures reported in the manuscript, including intraclass correlation coefficients (ICC), Pearson correlations, and Bland–Altman agreement analyses among the three raters (Teacher, Gemini, Kimi). Running this script on the provided scoring data will reproduce all inter-rater reliability results reported in the Results section. (4) Scoring Results Summary — A CSV file (Scoring Results Summary (Anonymized).csv) containing the raw scoring data from all three raters across every anonymized assignment. This file serves as the direct input for the reliability analysis program and provides the complete numerical basis for the findings reported in the manuscript. Together, these materials allow readers and reviewers to fully verify the study's data, procedures, and analytical outcomes.



