"Leveraging Large Language Models to Assist Teachers in Code Grading" (Supplementary Materials: Dataset and Reproducibility Code)

Paper Information Dataset and supplementary material of the paper: "Leveraging Large Language Models to Assist Teachers in Code Grading" If you want to refer to this work, please cite this Zenodo entry as well as the published paper below: @InProceedings{10.1007/978-3-031-98459-4_15,author="Cipriano, Edoardoand Ferrato, Alessioand Limongelli, Carlaand Schicchi, Danieleand Taibi, Davide",editor="Cristea, Alexandra I.and Walker, Erinand Lu, Yuand Santos, Olga C.and Isotani, Seiji",title="Leveraging Large Language Models to Assist Teachers in Code Grading",booktitle="Artificial Intelligence in Education",year="2025",publisher="Springer Nature Switzerland",address="Cham",pages="204--217",isbn="978-3-031-98459-4"} Research Purpose This repository contains data and resources used in research examining the effectiveness of Large Language Models (LLMs) for automated code grading. The project aims to: 1. Establish a benchmark dataset for evaluating LLM performance in code grading 2. Compare different LLMs to automated code grading Directory Structure ### `/code` Contains the code of the students. File naming convention: `YY_ID_row_topic.txt` - YY: Year (e.g., 21 for 2021) - ID: Exercise identifier - row: Exam row/version (e.g., A, B, C) - topic: Exercise topic (e.g., array, recursion) ### `/responses` Contains LLMs responses and evaluations, serving as the core benchmark dataset - `claude-3.5-sonnet/` - Directory containing Claude 3.5 Sonnet's evaluations - `21_array_002_C.txt` - Evaluation for array problem #002 part C File naming convention: `YY_topic_ID_row.txt` - YY: Year (e.g., 21 for 2021) - topic: Exercise topic (e.g., array, recursion) - ID: Exercise identifier - row: Exam row/version (e.g., A, B, C) ### `/exam_materials` Contains reference materials for establishing ground truth - `professors_grades.csv` - Expert-assigned grades (ground truth) - `/exam_rubrics/` - Detailed instructor rubrics - `rubric_21_recursion_A.txt` - Rubric for recursion Test A - `rubric_21_recursion_B.txt` - Rubric for recursion Test B - Original exam texts and evaluation criteria ### `/prompts` Contains the notebook with the prompts used for the LLMs and exploits openrouter API Users can just change the api key and run the notebook to get the results or implement their own prompts. ### `/grade_extraction_plus_results` Contains grade extraction from the LLMs responses and the analysis of the results. - Scripts for extracting and analyzing LLM evaluations - Tools for measuring grading accuracy