Replication Data for: Modeling Guessing Components in the Measurement of Political Knowledge

Due to the crucial role of political knowledge in democratic participation, the measurement of political knowledge has been a major concern in the discipline of political science. Common formats used for political knowledge questions include multiple-choice items and open-ended identification questions. The conventional wisdom holds that multiple-choice items induce guessing behavior, which leads to underestimated item-difficulty parameters and biased estimates of political knowledge. This article examines guessing behavior in multiple-choice items and argues that a successful guess requires certain levels of knowledge conditional on the difficulties of items. To deal with this issue, we propose a Bayesian IRT guessing model that accommodates the guessing components of item responses. The proposed model is applied to analyzing survey data in Taiwan, and the results show that the proposed model appropriately describes the guessing components based on respondents' levels of political knowledge and item characteristics. That is, in general, partially informed respondents are more likely to have a successful guess because well-informed respondents do not need to guess and barely informed ones are highly seducible by the attractive distractors. We also examine the gender gap in political knowledge and find that, even when the guessing effect is accounted for, men are more knowledgeable than women about political affairs, which is consistent with the literature.

鉴于政治知识在民主参与中的关键作用，政治知识测量一直是政治学（political science）学科的重点研究议题。政治知识类试题的常见题型包括多项选择题（multiple-choice items）与开放式识别题。学界主流共识认为，多项选择题会诱发猜测行为，进而导致试题难度参数被低估，且政治知识的估计结果存在偏倚。本文针对多项选择题中的猜测行为展开研究，并提出：成功的猜测需依赖于与试题难度相匹配的特定知识水平。为解决这一问题，本文提出一种适配试题作答猜测成分的贝叶斯项目反应理论（Item Response Theory, IRT）猜测模型。本文将所提模型应用于中国台湾地区的调查数据分析，结果表明，该模型能够基于受访者的政治知识水平与试题特征，合理刻画作答猜测成分。具体而言，整体来看，知识储备中等的受访者更易成功猜对答案：知识渊博的受访者无需猜测，而几乎一无所知的受访者则极易被极具吸引力的干扰项（distractors）误导。此外，本文还考察了政治知识层面的性别差距，结果发现，即便控制猜测效应的影响，男性在政治事务方面的知识储备仍优于女性，这与现有研究结论一致。