IsmaelMousa/engsaf

--- license: apache-2.0 task_categories: - question-answering - table-question-answering - text-generation language: - en tags: - '2024' - engineering-exams - official - original - real pretty_name: Engineering Short Answer Feedback size_categories: - 1K<n<10K --- # Engineering Short Answer Feedback A collection of real short-answer responses from engineering exams across multiple engineering domains. ## Background In recent years, there has been a growing interest in using Artificial Intelligence (AI) to automate student assessment in education. Among different types of assessments, summative assessments play a crucial role in evaluating a student's understanding level of a course. Such examinations often involve short-answer questions. However, grading these responses and providing meaningful feedback manually at scale is both time-consuming and labor-intensive. Feedback is particularly important, as it helps students recognize their strengths and areas for improvement. Despite the importance of this task, there is a significant lack of publicly available datasets that support automatic short-answer grading with feedback generation. To address this gap, we introduce Engineering Short Answer Feedback (EngSAF), a dataset designed for automatic short-answer grading with feedback. The dataset covers a diverse range of subjects, questions, and answer patterns from multiple engineering domains. For more details: [https://arxiv.org/abs/2407.12818](https://arxiv.org/abs/2407.12818). ## Abstract The EngSAF dataset, in its raw and unprocessed form, consists of approximately 5,800 short-answer responses collected from real-life engineering examinations administered at a reputed academic institute. These responses are spread across 119 unique questions drawn from a wide range of engineering disciplines, making the dataset both diverse and domain-specific. Each data point includes a student’s answer and an associated human-annotated score, serving as a benchmark for evaluating automated grading models. The dataset is divided into three primary subsets: 70% is allocated for training, 16% is reserved for evaluation on unseen answers (UA), and 14% is dedicated to evaluating performance on entirely new questions (UQ). At this stage, it is important to note that the dataset is considered in its original state; no preprocessing, transformation, or filtering has yet been applied. All subsequent improvements and refinements to the data will be described in later sections. This dataset is known as EngSAF version 1.0 and was introduced in the paper titled *"I understand why I got this grade": Automatic Short Answer Grading (ASAG) with Feedback*, authored by Aggarwal et al., and set to appear in the proceedings of AIED 2025. The dataset is released strictly for academic and research purposes; any commercial use or redistribution without explicit permission is prohibited. ## Introduction The AEG with feedback problem is as follows: given a question, a reference answer, and a student’s answer, the aim is to assign a label indicating the degree of correctness in the student’s answer compared to the reference answer and provide content-focused elaborated feedback/explanation for the same. Note that the degree of correctness is limited to only three labels here, namely correct, partially correct, and incorrect. In this problem, we only focus on questions where the answers are short, varying between a sentence and a short paragraph. This task involves evaluating the alignment between the student’s answer and the reference answers which is expressed via the degree of correctness along with proper reasoning as to why that label is assigned. <div align="center"> <a href="https://arxiv.org/abs/2407.12818" target="_blank"> <img src="ASAG.png" height="120" alt="ASAG" title="ASAG"/> </a> </div> ## Methodology We followed a systematic process beginning with thorough data analysis and preprocessing. #### Data Analysis The initial analysis revealed that the training split contained a small number of missing values, specifically in the Question_id and Student Answer columns. Given that these were minimal; just 12 rows; we dropped them outright. Furthermore, we decided to eliminate the Question_id column altogether, as it held no value for our modeling goals. To bring consistency and alignment with our pipeline, we standardized the column names: "Question" became question, " Student Answer" was renamed student_answer, "Correct Answer" became reference_answer, "output_label" was renamed score, and "feedback" was renamed rationale. Although the mark_scheme column was not included in the original CSV, it was documented in the dataset’s official resources, and we added it later during feature engineering. In terms of duplication, we discovered a few records that were fully duplicated; these were removed. After cleaning, the training data comprised 3,662 entries with 106 unique questions, 3,516 unique student answers, and 3,614 unique feedback texts. The distribution of the score labels was nearly balanced, with a slight bias toward label 2 and the lowest representation in label 0. Importantly, student answers and feedback lengths varied substantially, introducing valuable diversity into the dataset. A similar set of operations was performed on the unseen_answers split, which contained 980 entries across 103 unique questions. We found 954 unique student answers and 963 unique feedback entries, and we followed the same cleaning and renaming steps. The same applied to the unseen_question set, which was already cleaner; it had no missing or fully duplicated rows, and included 765 samples with 12 unique questions, 751 unique student answers, and 765 unique feedbacks. We then turned our attention to semantic similarities and potential data leakage between these subsets. We wanted to quantify how many evaluation entries were semantically similar to training entries, which could artificially inflate performance metrics. To achieve this, we concatenated the values of each row into a single string and embedded them using the all-MiniLM-L6-v2 sentence-transformer, a fast and effective model for short text sequences. We indexed these embeddings using FAISS and searched for semantic overlaps above a 90% similarity threshold. This threshold was selected after iterative testing, starting from 80%, where we observed too many false positives due to repeated questions and correct answers. The 90% cutoff better reflected genuine semantic overlap without penalizing natural repetitions. The results were revealing. In the unseen_answers split, 909 of the 976 entries showed high semantic similarity to training data, leaving only 67 non-leaking samples. In contrast, the unseen_question split had only 17 overlapping samples, leaving 748 that were clean. The val set contained 376 overlapping entries and just 29 non-leakages, making it unsuitable for evaluation. Rather than discarding these leakages, we leveraged them during the resplitting phase. We isolated all non-leakage entries from unseen_answers, unseen_question, and val, merged them into a unified unseen set, and then split this into validation and test subsets with a 40-60 ratio, respectively. After filtering out duplicate records, we ended up with 844 unique, clean entries; the validation set was assigned 338 entries, and the test set received 506 entries. While a few student answers were repeated, each was associated with a unique feedback response, aligning with our expectation of varied responses to the same question. The remaining overlapping entries; previously filtered due to leakage; were pooled and merged back with the training set. After removing a few full duplicates and dropping repeated student_answer values to enhance training diversity, we finalized a training set size of 4,735 clean entries across 107 unique questions. This preprocessing ensured that our model would train on non-redundant, high-quality data without any semantic contamination of evaluation splits. Finally, we checked the text lengths across splits. The longest sentence in the training set contained 481 words, while the longest in the evaluation sets had 361. #### Processing The processing stage implemented all the data cleaning strategies identified during data analysis while preparing the datasets for model training and evaluation. Building on the data analysis findings, we developed a comprehensive processing pipeline using a custom Preprocessor class. We first addressed the quality issues identified during data analysis. This included removing the unnecessary Question_id column, dropping rows with missing values (only 12 instances), and standardizing column names to more descriptive formats (question, student_answer, etc.). We eliminated full duplicate records while preserving cases where only the student answers or feedback were duplicated, as these represented legitimate variations in responses to the same question. The dataset was then split according to our leakage-aware strategy, combining non-leaky records into validation and test sets while merging appropriate leaky records with the training data. This resulted in a final training set of 4,735, validation set of 338, and testing set of 506 clean records. For more details: [https://github.com/IsmaelMousa/automatic-essay-grading](https://github.com/IsmaelMousa/automatic-essay-grading) ## Experiments Comprehensive instruction tuning, evaluation, and optimization of large language models (LLMs) for automated essay grading, using refined data analysis, preprocessing, and feature engineering. Conducted 39 experiments on models like Mistral, Qwen2.5, and SmolLM2 to assess their performance across real datasets using both score and rationale evaluation. [](https://api.wandb.ai/links/ismael-amjad/783p4r3l) [](https://api.wandb.ai/links/ismael-amjad/rav48wc1) [](https://api.wandb.ai/links/ismael-amjad/osmeqlj3) Our experiments show Mistral achieves the highest accuracy in score prediction, while Qwen2.5 outperforms others in generating quality rationales. Tables 1 and 2 summarize the top-performing models for score and rationale evaluations, respectively, across key metrics. #### Table 1: Top-performing model per type based on `score` evaluation: | **Model** | **F1** | **Precision** | **Recall** | **Accuracy** | **CKS** | **RMSE** | |:-----------------------------------------------------|:---------:|:-------------:|:----------:|:------------:|:---------:|:----------:| | Mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-231K-tokens | **0.642** | **0.683** | **0.633** | **0.65** | **0.457** | 0.707 | | Qwen2.5-3B-Instruct-EngSaf-628K | 0.6141 | 0.6415 | 0.6046 | 0.62 | 0.4123 | **0.6633** | | SmolLM2-1.7B-Instruct-EngSaf-429K | 0.3614 | 0.4496 | 0.3939 | 0.4 | 0.0789 | 1.0392 | #### Table 2: Top-performing model per type based on `rationale` evaluation: | **Model** | **F1** | **Precision** | **Recall** | |:-----------------------------------------------------|:----------:|:-------------:|:----------:| | Mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-231K-tokens | 0.633 | 0.638 | 0.633 | | Qwen2.5-3B-Instruct-EngSaf-628K | **0.6438** | **0.653** | **0.6382** | | SmolLM2-1.7B-Instruct-EngSaf-429K | 0.6335 | 0.6381 | 0.6333 | Try it live: [https://huggingface.co/spaces/IsmaelMousa/AEG-Qwen2.5](https://huggingface.co/spaces/IsmaelMousa/AEG-Qwen2.5) ## Usage ```python from datasets import load_dataset engsaf = load_dataset("IsmaelMousa/engsaf") print(engsaf["train"][0]) ``` ``` {'question': 'Q1. State TRUE or FALSE and justify. No correct justification, no marks.\ne. If a parent process forks a new child process, the exec system call cannot be used in the parent process until the child process terminates.', 'student_answer': 'FALSE', 'reference_answer': 'False, parent and child processes are two independent processes. Parent can perform any operation that it likes including an exec system call.', 'mark_scheme': "{'0': 'Incorrect response', '1': 'Partially correct response', '2': 'Correct response'}", 'score': 0, 'rationale': 'Your answer is incorrect. Parent and child processes are independent and can execute concurrently. The parent process can make system calls, including exec, without waiting for the child process to terminate.'} ``` ## Citation If you use this dataset or reference the companion study, please cite the following. ```bibtex @misc{mousa2024engsaf, author = {Ismael Mousa}, title = {Automatic Essay Grading: Insights from EngSAF}, year = {2024}, note = {https://github.com/IsmaelMousa/automatic-essay-grading} } ``` ## License This dataset is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).license: apache-2.0 任务类别： - 问答 - 表格问答 - 文本生成 语言： - 英语 标签： - 2024 - 工程考试 - 官方 - 原始 - 真实 展示名称：工程简答题反馈数据集（Engineering Short Answer Feedback） 样本规模类别： - 1K<n<10K # 工程简答题反馈数据集（Engineering Short Answer Feedback） 本数据集收录了来自多个工程领域真实工程考试的简答题作答内容。 ## 背景 近年来，教育领域中利用人工智能（Artificial Intelligence, AI）实现学生学业评估自动化的需求日益增长。在各类评估形式中，终结性评估对于衡量学生对课程内容的掌握程度至关重要。此类考试通常包含简答题题型，但大规模手动批阅学生作答并出具针对性反馈既耗时又耗力。反馈环节尤为关键，能够帮助学生明确自身优势与待改进之处。尽管该任务意义重大，但当前公开可用的、支持自动简答题批阅与反馈生成的数据集仍存在显著缺口。为填补这一空白，我们推出了**工程简答题反馈数据集（Engineering Short Answer Feedback, EngSAF）**，专为支持自动简答题批阅与反馈生成任务打造。本数据集涵盖多个工程领域的多样化科目、试题与作答范式。 更多细节请参阅：[https://arxiv.org/abs/2407.12818](https://arxiv.org/abs/2407.12818)。 ## 摘要 原始未处理的EngSAF数据集包含约5800份来自某知名学术机构真实工程考试的简答题作答内容。这些作答覆盖了119道来自多个工程学科的独特试题，确保数据集兼具多样性与领域针对性。每条数据均包含学生作答与人工标注的得分，可作为自动化批阅模型的评估基准。数据集最初分为三个主要子集：70%用于模型训练，16%用于未知作答（Unseen Answers, UA）场景的评估，剩余14%用于全新试题（Unseen Questions, UQ）场景的性能测试。需注意，本阶段数据集保持原始状态，未经过任何预处理、转换或筛选操作，后续所有数据优化与精炼步骤将在后续章节详述。本数据集为EngSAF 1.0版本，出自Aggarwal等人撰写的论文《"I understand why I got this grade": Automatic Short Answer Grading (ASAG) with Feedback》，将收录于AIED 2025会议论文集。本数据集仅面向学术与研究用途发布，未经明确许可严禁任何商业使用或二次分发。 ## 介绍 带反馈的自动简答题批阅（Automatic Essay Grading, AEG）任务定义如下：给定试题、参考答案与学生作答，目标是为学生作答与参考答案的匹配程度分配正确性标签，并生成针对性的详细反馈与解释。需注意，本任务中正确性标签仅包含三类：正确、部分正确与错误。本任务仅针对作答长度在单句至短段落之间的简答题。该任务需要评估学生作答与参考答案的对齐程度，通过正确性标签体现，并需附带该标签的判定依据。 <div align="center"> <a href="https://arxiv.org/abs/2407.12818" target="_blank"> <img src="ASAG.png" height="120" alt="ASAG" title="ASAG"/> </a> </div> ## 方法 我们遵循系统化流程开展工作，首先进行全面的数据分析与预处理。 #### 数据分析 初始分析发现，训练集中存在少量缺失值，主要集中在Question_id与Student Answer列。由于缺失量极少（仅12条数据），我们直接删除了这些记录。此外，由于Question_id列对建模目标无实际价值，我们将其完全移除。为统一流程并适配后续管线，我们对列名进行了标准化处理：将"Question"重命名为`question`，"Student Answer"重命名为`student_answer`，"Correct Answer"重命名为`reference_answer`，"output_label"重命名为`score`，"feedback"重命名为`rationale`。尽管原始CSV文件未包含mark_scheme列，但该字段已在数据集官方资源中予以说明，我们在后续特征工程阶段补充了该列。 针对重复数据，我们删除了所有完全重复的记录。清洗后的训练集包含3662条数据，对应106道独特试题、3516条独特学生作答与3614条独特反馈文本。得分标签的分布基本均衡，仅存在轻微偏向标签2的倾向，标签0的占比最低。值得注意的是，学生作答与反馈文本的长度存在显著差异，为数据集带来了宝贵的多样性。 我们对未知作答（unseen_answers）子集执行了相同的清洗与重命名操作，该子集包含980条数据，对应103道独特试题，包含954条独特学生作答与963条独特反馈记录。未知试题（unseen_question）子集的原始数据更为干净，无缺失值或完全重复的记录，包含765条数据，对应12道独特试题、751条独特学生作答与765条独特反馈记录。 随后，我们聚焦于各子集间的语义相似度与潜在的数据泄露问题，旨在量化评估集中与训练集存在语义相似性的记录数量——此类相似性可能会人为抬高模型的性能指标。为此，我们将每条记录的所有字段拼接为单个字符串，使用all-MiniLM-L6-v2句嵌入模型（专为短文本序列设计的高效快速模型）生成嵌入向量，再通过FAISS构建索引并检索相似度阈值高于90%的语义重叠记录。该阈值经过多次迭代测试确定：最初尝试80%阈值时，由于试题与参考答案的重复内容导致过多假阳性结果；90%阈值则更精准地反映真实语义重叠，同时不会对自然重复内容造成误判。 分析结果显示：未知作答子集中的976条记录里，有909条与训练集存在高度语义相似性，仅67条无泄露风险；未知试题子集仅存在17条重叠记录，剩余748条均为干净数据；验证集（val）则包含376条重叠记录，仅29条无泄露，因此不适合用于评估。 我们并未直接丢弃这些存在泄露风险的记录，而是在重划分阶段对其加以利用：从未知作答、未知试题与验证集中分离出所有无泄露的记录，合并为统一的未知集，再按4:6的比例划分为新的验证集与测试集。去重后，最终得到844条独特干净数据，其中验证集338条，测试集506条。尽管存在少量重复的学生作答，但每条作答均对应唯一的反馈文本，符合同一试题存在多样化作答的预期。 此前因泄露问题被分离的重叠记录被合并回训练集，在删除少量完全重复记录并移除重复的学生作答以提升训练集多样性后，最终训练集包含4735条干净数据，对应107道独特试题。该预处理流程确保模型在无冗余的高质量数据上训练，同时评估集不会受到语义污染。 最后，我们统计了各子集的文本长度分布：训练集中最长文本包含481个单词，评估集中最长文本则为361个单词。 #### 数据处理 数据处理阶段落实了数据分析阶段确定的所有清洗策略，并将数据集适配至模型训练与评估流程。基于数据分析结果，我们通过自定义Preprocessor类构建了完整的处理管线。我们首先修复了数据分析阶段发现的质量问题：移除无用的Question_id列，删除仅12条存在缺失值的记录，并将列名标准化为更具可读性的格式（如`question`、`student_answer`等）。我们删除了所有完全重复的记录，但保留了仅学生作答或反馈文本重复的情况——此类重复代表同一试题的合法作答变体。随后，我们基于泄露感知策略对数据集进行划分：将无泄露的记录合并为验证集与测试集，将合适的泄露记录并入训练集，最终得到训练集4735条、验证集338条、测试集506条的干净数据。 更多细节请参阅：[https://github.com/IsmaelMousa/automatic-essay-grading](https://github.com/IsmaelMousa/automatic-essay-grading) ## 实验 我们基于优化后的数据分析、预处理与特征工程流程，对用于自动简答题批阅的大语言模型（Large Language Model, LLM）开展了全面的指令微调、评估与优化工作。我们在Mistral、Qwen2.5与SmolLM2等模型上完成了39组实验，从得分预测与反馈生成两个维度评估模型在真实数据集上的性能。 [](https://api.wandb.ai/links/ismael-amjad/783p4r3l) [](https://api.wandb.ai/links/ismael-amjad/rav48wc1) [](https://api.wandb.ai/links/ismael-amjad/osmeqlj3) 实验结果显示，Mistral在得分预测任务中实现了最高的准确率，而Qwen2.5则在高质量反馈生成任务中表现更优。表1与表2分别总结了得分评估与反馈生成任务中表现最优的模型及其核心指标。 #### 表1：基于`score`评估的各类型最优模型： | **模型** | **F1值** | **精确率** | **召回率** | **准确率** | **CKS** | **均方根误差（RMSE）** | |:-----------------------------------------------------|:---------:|:-------------:|:----------:|:------------:|:---------:|:----------:| | Mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-231K-tokens | **0.642** | **0.683** | **0.633** | **0.65** | **0.457** | 0.707 | | Qwen2.5-3B-Instruct-EngSaf-628K | 0.6141 | 0.6415 | 0.6046 | 0.62 | 0.4123 | **0.6633** | | SmolLM2-1.7B-Instruct-EngSaf-429K | 0.3614 | 0.4496 | 0.3939 | 0.4 | 0.0789 | 1.0392 | #### 表2：基于`rationale`评估的各类型最优模型： | **模型** | **F1值** | **精确率** | **召回率** | |:-----------------------------------------------------|:----------:|:-------------:|:----------:| | Mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-231K-tokens | 0.633 | 0.638 | 0.633 | | Qwen2.5-3B-Instruct-EngSaf-628K | **0.6438** | **0.653** | **0.6382** | | SmolLM2-1.7B-Instruct-EngSaf-429K | 0.6335 | 0.6381 | 0.6333 | 在线体验：[https://huggingface.co/spaces/IsmaelMousa/AEG-Qwen2.5](https://huggingface.co/spaces/IsmaelMousa/AEG-Qwen2.5) ## 使用方法 python from datasets import load_dataset engsaf = load_dataset("IsmaelMousa/engsaf") print(engsaf["train"][0]) {'question': 'Q1. State TRUE or FALSE and justify. No correct justification, no marks. e. If a parent process forks a new child process, the exec system call cannot be used in the parent process until the child process terminates.', 'student_answer': 'FALSE', 'reference_answer': 'False, parent and child processes are two independent processes. Parent can perform any operation that it likes including an exec system call.', 'mark_scheme': "{'0': 'Incorrect response', '1': 'Partially correct response', '2': 'Correct response'}", 'score': 0, 'rationale': 'Your answer is incorrect. Parent and child processes are independent and can execute concurrently. The parent process can make system calls, including exec, without waiting for the child process to terminate.'} ## 引用 若您使用本数据集或引用相关研究，请按以下格式引用： bibtex @misc{mousa2024engsaf, author = {Ismael Mousa}, title = {Automatic Essay Grading: Insights from EngSAF}, year = {2024}, note = {https://github.com/IsmaelMousa/automatic-essay-grading} } ## 许可证 本数据集采用[Apache 2.0许可证](https://www.apache.org/licenses/LICENSE-2.0)发布。