Intermediate acoustic-to-semantic representations link behavioural and neural responses to natural sounds

Recognizing sounds implicates the cerebral transformation of input waveforms into semantic representations. Although past research identified the superior temporal gyrus (STG) as a crucial cortical region, the computational fingerprint of these cerebral transformations remains poorly characterized. Here, we exploit a model-comparison framework and contrasted the ability of acoustic, semantic (continuous and categorical), and sound-to-event deep neural network (DNN) representation models to predict perceived sound dissimilarity and 7 Tesla human auditory cortex fMRI responses. We confirm that spectrotemporal modulations predict early auditory cortex (Heschl’s gyrus) responses, and that auditory dimensions (e.g., loudness, periodicity) predict STG responses and perceived dissimilarity. Sound-to-event DNNs predict HG responses similar to acoustic models but, notably, they outperform all competing models at predicting both STG responses and perceived dissimilarity. Our findings indicate tha..., This repository includes data, analysis code and results for the following paper:
Intermediate acoustic-to-semantic representations link behavioural and neural responses to natural sounds
Bruno L. Giordano1*, Michele Esposito2, Giancarlo Valente2 and Elia Formisano2,3,4*
1 Institut des Neurosciences de La Timone, UMR 7289, CNRS and Université Aix-Marseille, Marseille, France.
2 Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands.
3 Maastricht Centre for Systems Biology (MaCSBio), Faculty of Science and Engineering, Maastricht University
4 Brightlands Institute for Smart Society (BISS), Maastricht University
*Corresponding authors.
E-mails: bruno dot giordano at univ-amu dot fr;
              e dot formisano at maastrichtuniversity dot nl
In this paper, we re-analyse behavioural data from Giordano et al. (2010; perceived natural sound and word dissimilarity), and Santoro et al. (2017; 7T fMRI responses to natural so..., ##  Repo structure
* Install.m: Matlab script called inside the analysis code to install toolboxes and declare relevant paths. 
*  README_1st.txt: installation information (also included in this README.md)
* /code/: code used to fit the models to the stimuli and analyze the data. 
Main analysis code:
  * analyze_01_acoustic_models_distances.m (Matlab): fits acoustics models to sound stimuli and computes between-stimulus distances
  * analyze_02_nlp_models.py (Python): computes natural language processing embeddings for the labels describing the sound stimuli, and the categories model (data from Santoro et al. 2017, only).
  * analyze_03_semantic_distances.m : Matlab; computes semantic between-stimulus distances using the natural language processing embeddings or the categories model.
  * analyze_04a_dnns_vggish.py (Python): fits the VGGish model to the sound stimuli;
  * analyze_04b_dnns_yamnet.py (Python): fits the Yamnet model to the sound stimuli;
  * analyze_04c_dnns_kell.py (Python...