Replication Data for: Bridging the Grade Gap: Reducing Assessment Bias in a Multi-Grader Class

Many large survey courses rely on multiple professors or teaching assistants to judge student responses to open-ended questions. Even following best practices, students with similar levels of conceptual understanding can receive widely varying assessments from different graders. We detail how this can occur and argue that it is an example of differential item functioning (or interpersonal incomparability), where graders interpret the same possible grading range differently. Using both actual assessment data from a large survey course in Comparative Politics and simulation methods, we show that the bias can be corrected by a small number of “bridging” observations across graders. We conclude by offering best practices for fair assessment in large survey courses. These files should fully replicate the findings in \"Bridging the Grade Gap: Reducing Assessment Bias in a Multi-Grader Class,\" accepted at Political Analysis in April, 2021.

诸多大型通识调查类课程通常会依托多位教授与教学助教，对学生开放式问题的作答进行评分。即便遵循评分最佳实践，概念理解水平相近的学生仍可能收到不同评分者给出的差异极大的评定结果。我们详细阐释了该现象的产生机制，并指出这属于项目功能差异（Differential Item Functioning，DIF），亦可称为人际不可比性——即不同评分者对同一既定评分区间的解读存在偏差。通过采用某大型比较政治学通识调查类课程的实际评分数据与模拟仿真方法，我们证实：仅需少量跨评分者的桥接观测样本，即可修正这类评分偏差。最后，我们针对大型通识调查类课程的公平评分，提出了相应的最佳实践方案。本数据集附带的全部文件可完整复现《缩小评分差距：降低多评分者班级中的评估偏差（Bridging the Grade Gap: Reducing Assessment Bias in a Multi-Grader Class）》一文的研究结果，该论文于2021年4月被《政治分析（Political Analysis）》期刊录用。