Eleven years of Progress Testing at Unicamp: a test validity study

Abstract: Introduction: The Medical School of the State University of Campinas (UNICAMP) has been testing the cumulative cognitive acquisition among its students through the Progress Test (PT) for over a decade, making it possible to analyze the utility of the test as a strategy to support pedagogical decisions and to point out the main threats to its validity. Objective: To provide an analysis of the validity of the PT, explaining opportunities for its use, especially in determining standards of cognitive sufficiency for progression, standards of cognitive sufficiency at the end of the course, and identification of students at risk. Method: Retrospective observational study of historical series of successive written tests performed to analyze cognitive accumulation, covering a period from 2006 to 2016, totaling 11 years and 6 consecutive classes. At each instance of measurement (test application), the study uses a mixed model where exposure (test performance) and outcome (test score) are evaluated at the same time point, characterizing a cross-sectional study, the successive results of which will generate the cognitive growth curves. Result: A cognitive accumulation of around 6 percentage points was observed with each new test. Students, upon completing the 6th year, scored around 65.7% (± 9.1). A “floor effect” was determined for each test to identify students with below-average performance, which in general was about 1.5 SD below the average of the respective class. Conclusion: TP-Unicamp offers reliable data to support important pedagogical decisions, such as the identification of students at academic risk for low performance, criteria for progression and cognitive performance at the end of the course. As reliability is influenced by sampling, increasing the number of items in each test and increasing the frequency of tests could represent strategies to overcome potential limitations.