Data_Sheet_1_Efficiency of computerized adaptive testing with a cognitively designed item bank.docx

An item bank is key to applying computerized adaptive testing (CAT). The traditional approach to developing an item bank requires content experts to design each item individually, which is a time-consuming and costly process. The cognitive design system (CDS) approach offers a solution by automating item generation. However, the CDS approach has a specific way of calibrating or predicting item difficulty that affects the measurement efficiency of CAT. A simulation study was conducted to compare the efficiency of CAT using both calibration and prediction models. The results show that, although the predictive model (linear logistic trait model; LLTM) shows a higher root mean square error (RMSE) than the baseline model (Rasch), it requires only a few additional items to achieve comparable RMSE. Importantly, the number of additional items needed decreases as the explanatory rate of the model increases. These results indicate that the slight reduction in measurement efficiency due to prediction item difficulty is acceptable. Moreover, the use of prediction item difficulty can significantly reduce or even eliminate the need for item pretesting, thereby reducing the costs associated with item calibration.

题库是计算机化自适应测验（Computerized Adaptive Testing, CAT）应用的核心要件。传统题库开发模式要求内容专家逐一设计每一道测验题目，该过程耗时耗力且成本高昂。认知设计系统（Cognitive Design System, CDS）方法通过实现题目生成自动化，为题库开发提供了可行解决方案。但该方法存在一套特定的题目难度校准或预测流程，会对CAT的测量效率产生影响。本研究通过模拟实验，对比了采用难度校准模型与难度预测模型的CAT的测量效率。结果表明，尽管预测模型——线性逻辑特质模型（linear logistic trait model, LLTM）的均方根误差（root mean square error, RMSE）高于基准模型拉什模型，但仅需少量额外题目即可达到与基准模型相当的RMSE水平。值得注意的是，所需额外题目的数量会随着模型解释率的提升而减少。上述结果表明，因采用题目难度预测方法导致的测量效率小幅下降是可接受的。此外，采用题目难度预测方法可大幅减少甚至完全免去题目试测环节，进而降低与题目校准相关的成本。