Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example

Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon's Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, “Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?” 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstracts. 2) Microworkers iteratively synthesized free-text answers describing the studied foods into one coherent term. Approximately 84% of foods were agreed upon, with only 4 and 8% of ratings failing manual review in different steps. 3) Microworkers were asked to rate the perceived obesogenicity of the synthesized food terms. Over 99% of responses were complete and usable, and opinions of the microworkers qualitatively matched the authors' expert expectations (e.g., sugar-sweetened beverages were thought to cause obesity and fruits and vegetables were thought to prevent obesity). 4) Microworkers extracted citation counts for each paper through Google Scholar. Microworkers reached consensus or unanimous agreement for all successful searches. To answer the example question, data were aggregated and analyzed, and showed no significant association between popular opinion and attention the paper received as measured by Scimago Journal Rank and citation counts. Direct microworker costs totaled $221.75, (estimated cost at minimum wage: $312.61). We discuss important points to consider to ensure good quality control and appropriate pay for microworkers. With good reliability and low cost, crowdsourcing has potential to evaluate published literature in a cost-effective, quick, and reliable manner using existing, easily accessible resources.

系统性评估科学文献是一项耗时费力的工作，需要耗费大量时间进行编码与评级。本文介绍了一种通过在线众包将此类任务分配给大规模群体的方法。研究借助亚马逊机械 Turk（Amazon Mechanical Turk）平台，由众包微工作者（microworkers）完成四组任务，以解答如下问题：“与大众观点一致的营养-肥胖研究，是否会比与大众观点相悖的同类研究在科学界获得更多关注？”
1) 通过资格测试的微工作者（仅19%通过测试）对摘要进行评级，以判断其是否为针对人类的营养与肥胖相关研究。前两名评级者的结论一致性中等（κ=0.586），96%的摘要达成共识。
2) 微工作者需通过迭代将描述研究食品的自由文本答案整合为一个统一术语。约84%的食品术语达成共识，不同步骤中仅4%和8%的评级未通过人工审核。
3) 微工作者需对整合后的食品术语的致肥胖性感知进行评级。超过99%的回复完整且可用，微工作者的观点与作者的专家预期定性一致（例如，认为含糖饮料会导致肥胖，而水果蔬菜则可预防肥胖）。
4) 微工作者通过谷歌学术（Google Scholar）提取每篇论文的引用量。所有成功检索的论文均由微工作者达成共识或完全一致的评级结果。

为解答上述示例问题，我们对数据进行汇总与分析，结果显示：以斯高帕斯期刊排名（Scimago Journal Rank）和引用量作为关注度衡量指标，大众观点与论文所获关注度之间并无显著关联。微工作者的直接总成本为221.75美元（按最低工资标准估算的成本为312.61美元）。我们还讨论了确保良好质量控制与为微工作者提供合理薪酬所需关注的关键要点。借助现有且易于获取的资源，众包凭借其高可靠性与低成本优势，有望以兼具经济性、高效性与可靠性的方式完成已发表文献的评估工作。