A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time

Representations from convolutional neural networks have been used as explanatory models for hierarchical sensory brain activations. Visual and auditory representations in the human brain have been studied with encoding models, RSA, decoding and reconstruction. However, none of the functional MRI data sets that are currently available has adequate amounts of data for sufficiently sampling their representations, or for training complex neural network hierarchies end-to-end on brain data for uncovering such representations. We recorded a densely sampled large fMRI dataset (TR=700 ms) in a single individual exposed to spatio-temporal visual and auditory naturalistic stimuli (30 episodes of BBC’s Doctor Who). The data consists of approximately 118,000 whole-brain volumes (approx. 23 h) of single-presentation data (full episodes, training set) and approximately 500 volumes (5 min) of repeated short clips (test set, 22 repetitions), recorded with fixation over a period of six months. This rich dataset can be used widely to study the way the brain represents audiovisual and language input across its sensory hierarchies.