Measuring online survey participant engagement through qualitative response analysis

Quantitative attention checks are commonly used in online surveys including those investigating public opinions about genetic engineering (GE) in plants and animals. Measuring participant engagement via open-ended qualitative response analysis in GE surveys, however, is underexplored. We used Turnitin™ to assess the originality of open-ended responses in four online surveys about GE. Across surveys, 18-35% of participants were identified as having copied responses from online sources. Using Cronbach’s alpha, we found that participants who copied responses responded to quantitative multi-item rating scales less consistently than participants who did not copy responses. We conclude that participants who provided qualitative responses identified as copied from the internet also provided less consistent responses to quantitative questions, suggesting that they were less engaged with the survey. We encourage future research on the motivation of participants to search for information online, including what sources they find compelling, and which information they choose to present.

量化注意力检查题广泛应用于各类在线调查，其中包括针对动植物基因工程（GE）公众意见的调查。然而，在基因工程相关调查中，借助开放式质性应答分析来衡量受访者参与度的研究仍有待充分探索。我们采用Turnitin™对四项基因工程相关在线调查中的开放式应答内容的原创性进行评估。在各项调查中，18%至35%的受访者被识别出从网络来源复制应答内容。通过克隆巴赫α系数（Cronbach’s alpha）分析后发现，相较于未复制应答内容的受访者，复制应答内容的受访者在量化多条目评定量表上的作答一致性显著更低。我们由此得出结论：被识别为从互联网复制质性应答的受访者，在量化问题上的作答一致性同样较低，这表明他们对本次调查的参与度更低。我们呼吁未来开展相关研究，探索受访者在线搜索信息的动机，包括他们所青睐的信息来源，以及他们选择呈现的信息内容。