Toward a principled Bayesian workflow in cognitive science

Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan, and easily accessible front-end packages such as brms. However, the utility of Bayesian methods ultimately depends on the relevance of the Bayesian model, in particular whether or not it accurately captures the structure of the data and the data analyst’s domain expertise. Even with powerful software, the analyst is responsible for verifying the utility of their model. To accomplish this, we introduce a principled Bayesian workflow (Betancourt, 2018) to cognitive science. Using a concrete working example, we describe basic questions one should ask about the model: prior predictive checks, computational faithfulness, model sensitivity, and posterior predictive checks. The running example for demonstrating the workflow is data on reading times with a linguistic manipulation of object versus subject relative sentences. This principled Bayesian workflow also demonstrates how to use domain knowledge to inform prior distributions. It provides guidelines and checks for valid data analysis, avoiding overfitting complex models to noise, and capturing relevant data structure in a probabilistic model. Given the increasing use of Bayesian methods, we aim to discuss how these methods can be properly employed to obtain robust answers to scientific questions. All data and code accompanying this paper are available from https://osf.io/b2vx9/.

在认知科学领域，对记忆、语言及其他认知领域的研究实验正日益采用贝叶斯方法进行分析。这一转变得益于概率编程语言如Stan的进步，以及易于获取的前端软件包如brms的发展。然而，贝叶斯方法的有效性最终取决于贝叶斯模型的关联性，特别是模型是否能够准确捕捉数据结构和数据分析者的领域专业知识。即便拥有强大的软件工具，分析师仍需负责验证其模型的有效性。为此，我们引入了贝叶斯工作流程的原理性方法（Betancourt，2018），应用于认知科学。通过一个具体的实例，我们阐述了关于模型应提出的基本问题：先验预测检验、计算忠实度、模型敏感度和后验预测检验。本例所采用的运行实例是关于阅读时间的数据，其中包括对主语与宾语相对句的语用操作。这一原理性贝叶斯工作流程还展示了如何利用领域知识来指导先验分布。它提供了有效数据分析的指南和检验，旨在避免将复杂模型过度拟合至噪声，并在概率模型中捕捉相关数据结构。鉴于贝叶斯方法的使用日益广泛，我们旨在探讨如何恰当地运用这些方法以获得对科学问题的稳健答案。本文所伴随的所有数据和代码均可在https://osf.io/b2vx9/获取。