REPLICATION MATERIAL to Question-order Effect in the Study of Satisfaction with Democracy. Lessons from Three Split-ballot Experiments (Papp et al. 2024, Sociological Methodology)

Papp, Zsófia, Pál Susánszky &amp; Andrea Szabó (2024) Question-Order Effect in the Study of Satisfaction with Democracy: Lessons from Three Split-Ballot Experiments. Sociological Methodology, 54(2): 404-417, First published 28 May 2024; https://doi.org/10.1177/00811750241254363<br>The study of attitudes toward democracy and democratic performance gains momentum with the gradual degradation of democracies worldwide (Bermeo 2016). During what some call the third wave of autocratization (Lührmann and Lindberg 2019; Waldner and Lust 2018), we see the halt of the liberal democracy projects in Eastern European countries such as Hungary and - formerly Poland, and the Czech Republic (Anna Vachudova 2020; Bustikova and Guasti 2017; Sitter and Bakke 2019). Arguably, citizen satisfaction with democracy (SWD) plays a role in the future of liberal democracy in that it has the potential to stabilize democracy and prevent radical changes (Bernauer and Vatter 2012). Behavioral consequences of democratic dissatisfaction include support for anti-democratic parties, participation in illegal political action, and decreasing willingness to vote (Anderson et al. 2005; André and Depauw 2017; Bengtsson and Christensen 2016).It is no surprise that SWD has been in the focus of a vast number of studies. At the same time, the SWD survey item has been criticized because of inter- and intra-country variation that prohibit the meaningful comparison of individuals and countries (Canache, Mondak, and Seligson 2001; Clarke, Dutt, and Kornberg 1993) and that it is highly sensitive to the institutional context (Linde and Ekman 2003). Thus, SWD is demonstrably not a suitable indicator of political support for democracy. However, it is generally considered a good measure of the citizens’ subjective assessment of the quality of democracy as they perceive it (Cutler, Nuesser, and Nyblade 2013). Recent evidence puts researchers on guard by demonstrating that different cross-national survey projects report varying levels, trends, and determinants of SWD raising further questions about measurement validity (Valgarðsson and Devine 2022). At the same time, these survey projects differ in terms of the wording and the scale of the SWD item, as well as the general context of the questionnaire further hampering comparative efforts. <i>This research note focuses on how the context of the survey affects responses to the SWD question.</i>Our research question is informed by the European Social Survey’s core questionnaire (ESS). ESS is a popular source for the study of SWD (among others, Christmann 2018; Harteveld et al. 2021; Magalhães 2016; Nemčok and Wass 2021; Stecker and Tausendpfund 2016; Vlachová 2019; Zilinsky 2019). It offers high-quality, wide-range comparative and longitudinal data on people’s social and political attitudes. In the ESS, the SWD measure (STFDEM) ‘… on the whole, how satisfied are you with the way democracy works in your country?’ (ESS Round 10: European Social Survey 2022) follows several satisfaction measures, namely, ‘All things considered, how satisfied are you with your life as a whole nowadays?’ (STFLIFE), ‘On the whole, how satisfied are you with the present state of the economy in [country]’ (STFECO), and ‘Now thinking about the [country] government, how satisfied are you with the way it is doing its job?’ (STFGOV). All items are measured on a 0 to 10 scale with ‘Extremely dissatisfied’ (0) and ‘Extremely satisfied’ (10) at the two extremes. While these survey items are not visually presented as a battery, due to their joint focus (i.e. satisfaction) the respondents may very well think they are clustered together for a reason. Therefore, there is a possibility that this series of connected questions may be subject to question-order bias, and perhaps the order of questions affects the level of SWD as well as its determinants. Out of the three preceding satisfaction items, <i>we zoom in on satisfaction with the state of the economy (SWE) and how its relative position in the questionnaire affects SWD responses.</i> We carried out three independent split-ballot experiments in Hungary between May 2021 and May 2022. We conclude that if the SWE question is asked first (i.e. the SWD question is primed by SWE), the mean value of SWD is significantly and meaningfully lower than in the case when the SWD question is not primed. Importantly, question order only affects SWD but not SWE. Our results are relevant to the (1) creators of ESS for the further development of the core questionnaire, (2) to creators of other large, comparative studies that have yet to include a full satisfaction battery, (3) to researchers designing smaller studies who may consider including several satisfaction items, and (4) to the users of such studies to receive additional information on data quality.<b>Data providers:</b>- NRC Marketingkutató és Tanácsadó Kft. (data1, data2)- Pharmaproject - Statisztika Tanácsadó és Szolgáltató Bt. (data3)- World Values Survey Wave 5; Inglehart, R., C. Haerpfer, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris, E. Ponarin &amp; B. Puranen et al. (eds.). 2018. World Values Survey: Round Five - Country-Pooled Datafile. Madrid, Spain &amp; Vienna, Austria: JD Systems Institute &amp; WVSA Secretariat. doi.org/10.14281/18241.7 ; https://www.worldvaluessurvey.org/WVSDocumentationWV5.jsp<b>Replication code</b>: Zsófia Papp<b>Software</b>: R 4.3.2<br>

Papp, Zsófia, Pál Susánszky 与 Andrea Szabó（2024）《民主满意度研究中的问题顺序效应（Question-Order Effect）：来自三项分割选票实验（split-ballot experiments）的启示》，《社会学方法》（Sociological Methodology），第54卷第2期：404-417，2024年5月28日首次出版；https://doi.org/10.1177/00811750241254363

全球民主逐渐退化的背景下，民众对民主及民主运行表现的态度研究热度持续攀升（Bermeo 2016）。在部分学者所称的第三波专制化浪潮（third wave of autocratization）中（Lührmann与Lindberg 2019；Waldner与Lust 2018），匈牙利、前波兰与捷克等东欧国家的自由民主进程陷入停滞（Anna Vachudova 2020；Bustikova与Guasti 2017；Sitter与Bakke 2019）。可以说，民众的民主满意度（Satisfaction with Democracy, SWD）对自由民主的未来至关重要，因其具备稳定民主制度、防范激进变革的潜力（Bernauer与Vatter 2012）。民主不满带来的行为后果包括支持反民主政党、参与非法政治行动以及投票意愿下降（Anderson et al. 2005；André与Depauw 2017；Bengtsson与Christensen 2016）。

无怪乎民主满意度已成为海量学术成果的聚焦对象。与此同时，民主满意度调研题项也饱受诟病：其国与国、国家内部的差异使得个体与国家间的有效比较难以实现（Canache、Mondak与Seligson 2001；Clarke、Dutt与Kornberg 1993），且其对制度语境高度敏感（Linde与Ekman 2003）。因此，有证据表明，民主满意度并非衡量民众对民主政治支持度的合适指标。但学界普遍认为，该指标可较好地反映民众对自身感知中的民主质量的主观评价（Cutler、Nuesser与Nyblade 2013）。近期研究发现，不同跨国调研项目报告的民主满意度水平、趋势与影响因素存在差异，这进一步引发了关于测量效度的质疑（Valgarðsson与Devine 2022）。此外，这些调研项目在民主满意度题项的措辞、量表设置以及问卷整体语境上均存在不同，进一步阻碍了比较研究的开展。<em>本研究简报（research note）聚焦调研语境如何影响民众对民主满意度问题的作答。</em>

本研究的问题设置源自欧洲社会调查（European Social Survey, ESS）的核心问卷。欧洲社会调查是民主满意度研究的热门数据来源（Christmann 2018；Harteveld et al. 2021；Magalhães 2016；Nemčok与Wass 2021；Stecker与Tausendpfund 2016；Vlachová 2019；Zilinsky 2019），其提供了关于民众社会与政治态度的高质量、大范围比较与追踪数据。在欧洲社会调查中，民主满意度测项（STFDEM）「整体而言，你对本国民主运行状况的满意度如何？」（ESS第十轮：欧洲社会调查2022）紧随多项满意度测项之后，具体包括「综合来看，你对当下整体生活的满意度如何？」（STFLIFE）、「整体而言，你对[国家]当前经济状况的满意度如何？」（STFECO），以及「现在请思考[国家]政府，你对其履职表现的满意度如何？」（STFGOV）。所有题项均采用0至10分的评分量表，两端锚点分别为「极不满意」（0分）与「极满意」（10分）。尽管这些调研题项并未以组题形式视觉呈现，但由于它们均聚焦满意度主题，受访者很可能会认为它们被集中设置存在特定缘由。因此，这一系列连续问题存在受到问题顺序偏差影响的可能性，或许问题顺序不仅会影响民主满意度的作答水平，还会影响其影响因素。在上述三项前置满意度测项中，<em>我们聚焦经济状况满意度（SWE）及其在问卷中的相对位置如何影响民主满意度的作答。</em>

本研究于2021年5月至2022年5月间在匈牙利开展了三项独立的分割选票实验（split-ballot experiments）。研究结果表明，若先询问经济状况满意度问题（即民主满意度问题被经济状况满意度问题启动），民主满意度的均值将显著且实质性低于未被启动的情况。值得注意的是，问题顺序仅会影响民主满意度，而不会影响经济状况满意度。本研究结果对四类主体具有参考价值：（1）欧洲社会调查的设计者，以助力其核心问卷的进一步优化；（2）其他尚未纳入完整满意度组题的大型比较研究项目的设计者；（3）设计包含多项满意度测项的小型研究的研究者；（4）此类研究的使用者，以帮助其获取关于数据质量的额外信息。

<b>数据提供方：</b>
- NRC Marketingkutató és Tanácsadó Kft.（数据集1、数据集2）
- Pharmaproject - Statisztika Tanácsadó és Szolgáltató Bt.（数据集3）
- 世界价值观调查（World Values Survey）第五波；Inglehart, R.、C. Haerpfer、A. Moreno、C. Welzel、K. Kizilova、J. Diez-Medrano、M. Lagos、P. Norris、E. Ponarin与B. Puranen等（编）. 2018. 《世界价值观调查：第五轮跨国合并数据集》. 西班牙马德里与奥地利维也纳：JD Systems研究所与世界价值观调查协会秘书处. doi.org/10.14281/18241.7 ; https://www.worldvaluessurvey.org/WVSDocumentationWV5.jsp

<b>复现代码（replication code）：</b>Zsófia Papp
<b>软件：</b>R 4.3.2