Shared spatiotemporal category representations in biological and artificial deep neural networks

Visual scene category representations emerge very rapidly, yet the computational transformations that enable such invariant categorizations remain elusive. Deep convolutional neural networks (CNNs) perform visual categorization at near human-level accuracy using a feedforward architecture, providing neuroscientists with the opportunity to assess one successful series of representational transformations that enable categorization in silico. The goal of the current study is to assess the extent to which sequential scene category representations built by a CNN map onto those built in the human brain as assessed by high-density, time-resolved event-related potentials (ERPs). We found correspondence both over time and across the scalp: earlier (0–200 ms) ERP activity was best explained by early CNN layers at all electrodes. Although later activity at most electrode sites corresponded to earlier CNN layers, activity in right occipito-temporal electrodes was best explained by the later, fully-connected layers of the CNN around 225 ms post-stimulus, along with similar patterns in frontal electrodes. Taken together, these results suggest that the emergence of scene category representations develop through a dynamic interplay between early activity over occipital electrodes as well as later activity over temporal and frontal electrodes.视觉场景类别表征的涌现速度极快，但支撑此类不变性分类的计算转换机制仍不明晰。深度卷积神经网络（Convolutional Neural Networks，简称CNN）采用前馈架构，可实现接近人类水平的视觉分类精度，为神经科学家提供了研究一套成功支撑计算机模拟分类的表征转换机制的契机。本研究的目标是，通过高密度时间分辨事件相关电位（Event-Related Potentials，简称ERP）技术采集的脑电数据，评估卷积神经网络构建的序列视觉场景类别表征，与人类大脑中形成的此类表征的匹配程度。研究发现，匹配度在时间维度与头皮电极分布维度均存在关联：在所有电极位点上，0~200ms的早期脑电活动均可由卷积神经网络的早期层结构最优解释。尽管多数电极位点的后续脑电活动与卷积神经网络的早期层结构存在对应关系，但在刺激呈现后约225ms时，右侧枕颞电极的脑电活动可由卷积神经网络后期的全连接层最优解释，额叶电极亦呈现类似模式。综合来看，上述结果表明，视觉场景类别表征的涌现是通过枕叶电极的早期脑电活动，与颞叶、额叶电极的后续脑电活动之间的动态交互实现的。