Eleven years of Progress Testing at Unicamp: a test validity study

Abstract: Introduction: The Medical School of the State University of Campinas (UNICAMP) has been testing the cumulative cognitive acquisition among its students through the Progress Test (PT) for over a decade, making it possible to analyze the utility of the test as a strategy to support pedagogical decisions and to point out the main threats to its validity. Objective: To provide an analysis of the validity of the PT, explaining opportunities for its use, especially in determining standards of cognitive sufficiency for progression, standards of cognitive sufficiency at the end of the course, and identification of students at risk. Method: Retrospective observational study of historical series of successive written tests performed to analyze cognitive accumulation, covering a period from 2006 to 2016, totaling 11 years and 6 consecutive classes. At each instance of measurement (test application), the study uses a mixed model where exposure (test performance) and outcome (test score) are evaluated at the same time point, characterizing a cross-sectional study, the successive results of which will generate the cognitive growth curves. Result: A cognitive accumulation of around 6 percentage points was observed with each new test. Students, upon completing the 6th year, scored around 65.7% (± 9.1). A “floor effect” was determined for each test to identify students with below-average performance, which in general was about 1.5 SD below the average of the respective class. Conclusion: TP-Unicamp offers reliable data to support important pedagogical decisions, such as the identification of students at academic risk for low performance, criteria for progression and cognitive performance at the end of the course. As reliability is influenced by sampling, increasing the number of items in each test and increasing the frequency of tests could represent strategies to overcome potential limitations.

摘要：引言：坎皮纳斯州立大学（State University of Campinas, UNICAMP）医学院十余年来一直通过进度测试（Progress Test, PT）对学生的累积认知习得情况开展测评，借此可分析该测试作为支撑教学决策策略的效用，并明确影响其效度的主要威胁因素。目的：对进度测试的效度展开系统性分析，阐明其应用场景，尤其可用于确定学业进阶所需的认知达标标准、毕业时的认知表现标准，以及识别存在学业风险的学生。方法：本研究为回顾性观察性研究，分析2006年至2016年间共11年、6届连续学生的系列书面测试历史数据，以探究认知累积特征。在每一次测评（测试实施）环节，本研究采用混合模型，于同一时间点评估暴露因素（测试表现）与结局指标（测试得分），属于横断面研究，其连续测试结果将用于构建认知成长曲线。结果：每一次新增测试均可观察到约6个百分点的认知累积提升。学生完成第6学年时，测试得分约为65.7%（±9.1）。本研究为每项测试确定了用以识别表现低于平均水平学生的“地板效应（floor effect）”阈值，总体而言该阈值约低于对应班级平均分1.5个标准差。结论：坎皮纳斯大学进度测试可提供可靠数据以支撑重要教学决策，例如识别学业表现不佳的风险学生、学业进阶评判标准以及毕业时的认知表现标准。由于测试信度受抽样影响，增加每项测试的题目数量与测试实施频率，或可成为克服其潜在局限性的可行策略。