Intermediate acoustic-to-semantic representations link behavioural and neural responses to natural sounds

Recognizing sounds implicates the cerebral transformation of input waveforms into semantic representations. Although past research identified the superior temporal gyrus (STG) as a crucial cortical region, the computational fingerprint of these cerebral transformations remains poorly characterized. Here, we exploit a model-comparison framework and contrasted the ability of acoustic, semantic (continuous and categorical), and sound-to-event deep neural network (DNN) representation models to predict perceived sound dissimilarity and 7 Tesla human auditory cortex fMRI responses. We confirm that spectrotemporal modulations predict early auditory cortex (Heschl’s gyrus) responses, and that auditory dimensions (e.g., loudness, periodicity) predict STG responses and perceived dissimilarity. Sound-to-event DNNs predict HG responses similar to acoustic models but, notably, they outperform all competing models at predicting both STG responses and perceived dissimilarity. Our findings indicate that STG entails intermediate acoustic-to-semantic sound representations that neither acoustic nor semantic models can account for. These representations are compositional in nature and relevant to behaviour.

Methods
This repository includes data, analysis code and results for the following paper:

Intermediate acoustic-to-semantic representations link behavioural and neural responses to natural sounds

Bruno L. Giordano1*, Michele Esposito2, Giancarlo Valente2 and Elia Formisano2,3,4*

1 Institut des Neurosciences de La Timone, UMR 7289, CNRS and Université Aix-Marseille, Marseille, France.

2 Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands.

3 Maastricht Centre for Systems Biology (MaCSBio), Faculty of Science and Engineering, Maastricht University

4 Brightlands Institute for Smart Society (BISS), Maastricht University

*Corresponding authors.

E-mails: bruno dot giordano at univ-amu dot fr;

              e dot formisano at maastrichtuniversity dot nl

In this paper, we re-analyse behavioural data from Giordano et al. (2010; perceived natural sound and word dissimilarity), and Santoro et al. (2017; 7T fMRI responses to natural sounds).

References:

- Giordano, B. L., McDonnell, J. & McAdams, S. Hearing living symbols and nonliving icons: Category-specificities in the cognitive processing of environmental sounds. Brain Cogn 73, 7–19 (2010).

- Santoro, R. et al. Reconstructing the spectrotemporal modulations of real-life sounds from fMRI response patterns. PNAS 114, 4799–4804 (2017).