Data for Addressing fraudulent responses in online surveys: insights from a web-based participatory mapping study

Abstract:1. Web-based studies of human dimensions are increasing across environmental and socio-ecological disciplines. However, the prevalence of fraud threatens research quality. Increased fraud rates should be expected as surveys move progressively more online, motivated by expanding reach, cost savings, and/or in response to COVID-19. Web-based research must better account for fraud to maintain confidence in findings. Practical diagnostic tools and data quality protocols are required to detect fraud and ensure results quality.2. Drawing on our experience using an online participatory mapping case study, we discuss methods to detect potentially fraudulent responses – and identify some limitations. We begin by reviewing the current state of knowledge on fraudulent responses or “fraudsters” and its relative absence in environmental and socio-ecological disciplines. We then describe our research approach, the indicators and variables we used to detect and assess fraud, and our decision-making process to eliminate suspicious responses without jeopardizing research integrity.3. We found that despite several preventative measures, many fraudulent respondents could provide survey responses and effectively mimicked legitimate respondents at first glance. By assuming each response to be 'potentially fraudulent,' we determined that the complete screening of each respondent, while time-consuming, can limit the prevalence of fraud. We also determined that the most common data consistency checks (e.g., duration, trap questions, straight-liner checks) are unlikely to guarantee valid respondents.4. If not acknowledged and addressed, fraud has the potential to undermine data integrity, discredit research findings, and limit the utility of results for policy. This study contributes to environmental and socio-ecological research by reviewing existing fraudster literature and using our experience with fraud to provide recommendations for researchers to address this problem. We encourage researchers implementing online qualitative research methods to thoroughly assess and report fraud, when possible, to ensure widespread knowledge of this growing threat.

摘要：1. 环境与社会生态学科中，针对人类维度的基于网络研究日益增多。然而，造假行为的盛行严重威胁研究质量。随着调研愈发向线上迁移——这一趋势的驱动因素包括拓展覆盖范围、节约成本，或是为应对新冠疫情——造假率势必上升。基于网络的研究必须更好地应对造假问题，以维持研究结论的可信度。学界亟需实用的诊断工具与数据质量规程，以检测造假行为并保障研究成果质量。2. 依托我们开展的一项在线参与式制图案例研究经验，本文探讨了检测潜在造假问卷回复的方法，并指出了相关局限。首先，我们梳理了当前关于造假回复或"造假受访者"的研究现状，以及该议题在环境与社会生态学科中相对匮乏的现状。随后，我们阐述了自身的研究路径、用于检测与评估造假的指标与变量，以及在不损害研究严谨性的前提下剔除可疑回复的决策流程。3. 研究发现，尽管采取了多项预防措施，仍有大量造假受访者能够提交问卷回复，且初看之下可有效模仿合规受访者。通过将每份回复均视为‘潜在造假’，我们得出结论：对每位受访者进行全面筛查虽耗时，但可有效降低造假的盛行率。此外，我们还发现，最常用的数据一致性校验手段（如作答时长检测、陷阱题设置、直线作答检查等）均无法确保受访者作答合规。4. 若不加以重视并妥善应对，造假行为可能破坏数据完整性、抹黑研究成果，并限制研究成果对政策制定的应用价值。本研究通过梳理现有造假相关文献，并结合我们应对造假的实践经验，为环境与社会生态领域的研究者提供了应对该问题的建议。我们鼓励采用在线质性研究方法的研究者，在条件允许时全面评估并上报造假情况，以提升学界对这一日益严峻的威胁的认知。