Fast Bootstrapping and Permutation Testing for Assessing Reproducibility and Interpretability of Multivariate fMRI Decoding Models

Multivariate decoding models are increasingly being applied to functional magnetic imaging (fMRI) data to interpret the distributed neural activity in the human brain. These models are typically formulated to optimize an objective function that maximizes decoding accuracy. For decoding models trained on full-brain data, this can result in multiple models that yield the same classification accuracy, though some may be more reproducible than others—i.e. small changes to the training set may result in very different voxels being selected. This issue of reproducibility can be partially controlled by regularizing the decoding model. Regularization, along with the cross-validation used to estimate decoding accuracy, typically requires retraining many (often on the order of thousands) of related decoding models. In this paper we describe an approach that uses a combination of bootstrapping and permutation testing to construct both a measure of cross-validated prediction accuracy and model reproducibility of the learned brain maps. This requires re-training our classification method on many re-sampled versions of the fMRI data. Given the size of fMRI datasets, this is normally a time-consuming process. Our approach leverages an algorithm called fast simultaneous training of generalized linear models (FaSTGLZ) to create a family of classifiers in the space of accuracy vs. reproducibility. The convex hull of this family of classifiers can be used to identify a subset of Pareto optimal classifiers, with a single-optimal classifier selectable based on the relative cost of accuracy vs. reproducibility. We demonstrate our approach using full-brain analysis of elastic-net classifiers trained to discriminate stimulus type in an auditory and visual oddball event-related fMRI design. Our approach and results argue for a computational approach to fMRI decoding models in which the value of the interpretation of the decoding model ultimately depends upon optimizing a joint space of accuracy and reproducibility.

多变量解码模型（Multivariate decoding models）正日益被应用于功能磁共振成像（functional magnetic imaging, fMRI）数据，以解析人类大脑中的分布式神经活动。这类模型通常被构建为优化目标函数以最大化解码准确率。针对在全脑数据上训练的解码模型，该过程往往会得到多个分类准确率一致的模型，但其中部分模型的可复现性更优——即对训练集进行小幅改动时，可能会选中差异极大的体素（voxels）。这类可复现性问题可通过对解码模型进行正则化（regularizing）得到部分控制。正则化结合用于估计解码准确率的交叉验证（cross-validation），通常需要重新训练大量（通常可达数千个）相关解码模型。本文提出一种结合自助法（bootstrapping）与置换检验（permutation testing）的方法，用以构建习得脑图谱（learned brain maps）的交叉验证预测准确率（cross-validated prediction accuracy）与模型可复现性指标。该方法需在fMRI数据的众多重采样版本上重新训练分类方法。鉴于fMRI数据集的规模，这一过程通常耗时良久。我们的方法借助名为广义线性模型快速同步训练（fast simultaneous training of generalized linear models, FaSTGLZ）的算法，在准确率-可复现性空间中生成一组分类器。该分类器集合的凸包（convex hull）可用于识别帕累托最优分类器（Pareto optimal classifiers）子集，进而可依据准确率与可复现性的相对成本选择单一最优分类器。我们通过弹性网络分类器（elastic-net classifiers）的全脑分析验证了所提方法：该分类器经训练后可在听觉与视觉Oddball事件相关fMRI范式（auditory and visual oddball event-related fMRI design）中区分刺激类型（stimulus type）。我们的方法与研究结果支持一种针对fMRI解码模型的计算范式，即解码模型的解释价值最终取决于对准确率与可复现性联合空间的优化。