External model overview.

Models that predict brain responses to stimuli provide one measure of understanding of a sensory system and have many potential applications in science and engineering. Deep artificial neural networks have emerged as the leading such predictive models of the visual system but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models and, thus, how to further improve models in this domain. We evaluated model-brain correspondence for publicly available audio neural network models along with in-house models trained on 4 different tasks. Most tested models outpredicted standard spectromporal filter-bank models of auditory cortex and exhibited systematic model-brain correspondence: Middle stages best predicted primary auditory cortex, while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. Models trained to recognize speech in background noise produced better brain predictions than models trained to recognize speech in quiet, potentially because hearing in noise imposes constraints on biological auditory representations. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results generally support the promise of deep neural networks as models of audition, though they also indicate that current models do not explain auditory cortical responses in their entirety.

能够预测刺激引发的大脑响应的模型，是理解感知系统的一类衡量手段，在科学与工程领域均拥有诸多潜在应用场景。深度人工神经网络已成为视觉系统这类预测模型的主流方案，但在听觉领域的相关探索仍相对匮乏。此前的研究虽已给出经音频训练的神经网络实例，这类模型可较好预测听觉皮层的功能磁共振成像（fMRI）响应，且展现出模型层级与脑区之间的对应关系，但尚未明确该结果是否可推广至其他神经网络模型，进而也未明确该领域内模型的进一步优化方向。本研究针对公开可用的音频神经网络模型，以及在4项不同任务上训练得到的自研模型，开展了模型-大脑对应性的评估工作。多数受测模型的预测性能优于针对听觉皮层的标准时频滤波器组模型，且呈现出系统性的模型-大脑对应性：模型中间层级对初级听觉皮层的预测效果最佳，而深层层级则对非初级听觉皮层的预测表现最优。但部分当前顶尖水平的模型，其大脑响应预测效果却显著较差。在背景噪声环境下训练用于语音识别的模型，其大脑响应预测效果优于在安静环境下训练的语音识别模型，这可能是因为噪声下的听觉感知过程对生物听觉表征施加了约束。训练任务会影响模型对特定皮层调谐特性的预测质量，在多任务上训练得到的模型可实现最优的整体预测效果。本研究结果总体上肯定了深度神经网络作为听觉系统模型的应用前景，但同时也表明，当前的模型尚无法完全解释听觉皮层的所有响应。