Survey Question Wording Effects

Survey researchers have long known that the terms used in questions posed to respondents shape the answers they give. Processes underlying these differences have generally been attributed to differences in respondents’ interpretations of the questions, though it is also possible that some of this difference may stem from respondents’ ability to parse what the questions are asking about. In three online survey experiments we manipulate wordings for policy attitude questions about the DREAM Act, Trump’s trade disputes, and the Affordable Care Act. Different wordings for these issues solicited different attitudes overall as well as across partisan groups. To understand the source of these differences, we examine the sensitivity of wording effects to respondent partisanship as well as awareness of topically relevant information. A simulation approach estimating each individual’s “informed response” allows us to disentangle question wording differences due to incomplete understandings (and misunderstandings) from those attributable to partisan bias. Evidence that individuals with greater topic knowledge better recognize the similarity of different wordings implies that some of the wording effect is associated with whether respondents understand what questions mean. When we attribute the proportion of the wording difference attributable to awareness as opposed to partisan bias, we find that the influence of each of these factors varies across the issues we examine.