Replication Material to Question-order Effect in the Study of Satisfaction with Democracy. Lessons from Three Split-ballot Experiments

<b>REPLICATION MATERIAL</b><br>The study of attitudes toward democracy and democratic performance gains momentum with the gradual degradation of democracies worldwide (Bermeo 2016). During what some call the third wave of autocratization (Lührmann and Lindberg 2019; Waldner and Lust 2018), we see the halt of the liberal democracy projects in Eastern European countries such as Hungary and - formerly Poland, and the Czech Republic (Anna Vachudova 2020; Bustikova and Guasti 2017; Sitter and Bakke 2019). Arguably, citizen satisfaction with democracy (SWD) plays a role in the future of liberal democracy in that it has the potential to stabilize democracy and prevent radical changes (Bernauer and Vatter 2012). Behavioral consequences of democratic dissatisfaction include support for anti-democratic parties, participation in illegal political action, and decreasing willingness to vote (Anderson et al. 2005; André and Depauw 2017; Bengtsson and Christensen 2016).It is no surprise that SWD has been in the focus of a vast number of studies. At the same time, the SWD survey item has been criticized because of inter- and intra-country variation that prohibit the meaningful comparison of individuals and countries (Canache, Mondak, and Seligson 2001; Clarke, Dutt, and Kornberg 1993) and that it is highly sensitive to the institutional context (Linde and Ekman 2003). Thus, SWD is demonstrably not a suitable indicator of political support for democracy. However, it is generally considered a good measure of the citizens’ subjective assessment of the quality of democracy as they perceive it (Cutler, Nuesser, and Nyblade 2013). Recent evidence puts researchers on guard by demonstrating that different cross-national survey projects report varying levels, trends, and determinants of SWD raising further questions about measurement validity (Valgarðsson and Devine 2022). At the same time, these survey projects differ in terms of the wording and the scale of the SWD item, as well as the general context of the questionnaire further hampering comparative efforts. <i>This research note focuses on how the context of the survey affects responses to the SWD question.</i>Our research question is informed by the European Social Survey’s core questionnaire (ESS). ESS is a popular source for the study of SWD (among others, Christmann 2018; Harteveld et al. 2021; Magalhães 2016; Nemčok and Wass 2021; Stecker and Tausendpfund 2016; Vlachová 2019; Zilinsky 2019). It offers high-quality, wide-range comparative and longitudinal data on people’s social and political attitudes. In the ESS, the SWD measure (STFDEM) ‘… on the whole, how satisfied are you with the way democracy works in your country?’ (ESS Round 10: European Social Survey 2022) follows several satisfaction measures, namely, ‘All things considered, how satisfied are you with your life as a whole nowadays?’ (STFLIFE), ‘On the whole, how satisfied are you with the present state of the economy in [country]’ (STFECO), and ‘Now thinking about the [country] government, how satisfied are you with the way it is doing its job?’ (STFGOV). All items are measured on a 0 to 10 scale with ‘Extremely dissatisfied’ (0) and ‘Extremely satisfied’ (10) at the two extremes. While these survey items are not visually presented as a battery, due to their joint focus (i.e. satisfaction) the respondents may very well think they are clustered together for a reason. Therefore, there is a possibility that this series of connected questions may be subject to question-order bias, and perhaps the order of questions affects the level of SWD as well as its determinants. Out of the three preceding satisfaction items, <i>we zoom in on satisfaction with the state of the economy (SWE) and how its relative position in the questionnaire affects SWD responses.</i> We carried out three independent split-ballot experiments in Hungary between May 2021 and May 2022. We conclude that if the SWE question is asked first (i.e. the SWD question is primed by SWE), the mean value of SWD is significantly and meaningfully lower than in the case when the SWD question is not primed. Importantly, question order only affects SWD but not SWE. Our results are relevant to the (1) creators of ESS for the further development of the core questionnaire, (2) to creators of other large, comparative studies that have yet to include a full satisfaction battery, (3) to researchers designing smaller studies who may consider including several satisfaction items, and (4) to the users of such studies to receive additional information on data quality.<b>Data providers:</b>- NRC Marketingkutató és Tanácsadó Kft. (data1, data2)- Pharmaproject - Statisztika Tanácsadó és Szolgáltató Bt. (data3)- World Values Survey Wave 5; Inglehart, R., C. Haerpfer, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris, E. Ponarin &amp; B. Puranen et al. (eds.). 2018. World Values Survey: Round Five - Country-Pooled Datafile. Madrid, Spain &amp; Vienna, Austria: JD Systems Institute &amp; WVSA Secretariat. doi.org/10.14281/18241.7 ; https://www.worldvaluessurvey.org/WVSDocumentationWV5.jsp<b>Replication code</b>: Zsófia Papp<b>Software</b>: R 4.3.2<br>