Fun surveys? Developing an innovative approach to assessing learning through citizen science

Assessing the impact of citizen science participation on volunteers is increasingly important. However, traditional methods for evaluating volunteer experiences—such as quantitative surveying—have drawbacks. Response rates are often low, pre- and post-tests provide a coarse metric for assessing learning, and static evaluation techniques have failed to align with the participatory spirit of citizen science, where volunteers are involved in understanding their role in scientific inquiry. In response to these problems, we developed a more engaging type of longitudinal survey aimed at achieving higher response rates and better involving volunteers in understanding the scientific value of their survey efforts. We first reviewed the literature on “fun” and found that fun activities embody six characteristics: autonomy, social connection, playfulness, challenge, transport, and curiosity. We used this “fun heuristic” to design “SciQuest,” an exploratory survey tool that evaluated volunteers’ pro-environmental attitudes and behaviors across citizen science projects hosted on the platform SciStarter.org. SciQuest proved to be a reliable instrument that captured common learning outcome–related constructs measured by conventional, validated instruments. Although most beta-testers had no preference between SciQuest and a conventional survey, among those expressing a preference, nearly twice as many favored SciQuest. SciQuest also achieved much higher retention rates (49–67%) than past conventional surveys on SciStarter (19%). Embedded, creative approaches to citizen science evaluation are thus a viable, and perhaps preferable, complement to traditional surveys, particularly on citizen science platforms. Citizen science facilitators should extend their volunteer-centric perspectives to include participant evaluation and seek evidence-based strategies for enriching the citizen science experience.

Methods
The attached CSV contains 243 anonymized survey responses from Qualtrics "Panelists" (paid surveytakers). These surveytakers completed a survey in May, 2021 where they answered a variety of questions about their connection to nature, their pro-enviornmental behaviors, and other conservation-related constructs. Each respondent completed two different types of survey analyzing the same construct: one using existing, validated constructs, and one using a new, untested instrument called "SciQuest." 

The goal of collecting this survey data was to quasi-validate the SciQuest version of these constructs. SciQuest is designed to provide a more fun, engaging way of answering survey questions for citizen science evaluators. However, to make sure the SciQuest questions are assessing the same constructs as conventional instruments, we compared responses to the same constructs among these Panelists who took both a conventional survey and a SciQuest version of the same survey constructs. 

This dataset has already been processed to remove 197 incomplete/low-quality survey responses. 

We have also attached an R script that analzes this dataset to calculate the results presented in our manuscript. 

We have also included a supplemental video file. This file is a screen recording of the SciQuest landing page and annotated walk-throughs of each survey module on SciQuest. The video shows a participant completing the three available survey modules on SciQuest: Journey Prep, Module 1: Nature, and Module 2: Environment. The journal to which we are submitting this manuscript does not allow for video-type files. I am uploading this supplemental video file on Dryad so that we can have a stable reference to this supplemental video. 

Lastly we have included three .qsf files of each of the three Qualtrics survey modules (Supplemental Files 4 corresponds to the Journey Prep module, Supplemental File 5 corresponds to Module 1 and Supplemental File 6 corresponds to Module 2). These files can be directly loaded into Qualtrics by others interested in using the surveys. The journal to which we are submitting this manuscript does not allow for submission of the .qsf file type, so I am uploading these supplemental files on Dryad.