hyper

# Citation

Dado, T., Güçlütürk, Y., Ambrogioni, L., Ras, G., Bosch, S., van Gerven, M., & Güçlü, U. (2022). Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space. Scientific reports, 12(1), 1-9.

# Contact person
Thirza Dado, [e-mail]

# Abstract

Neural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation thereof for HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring fMRI data as participants perceive face images synthesized by the generator network of a GAN. We show that the latent vectors used for generation effectively capture the same defining stimulus properties as the fMRI measurements. As such, these latents (conditioned on the GAN) are used as the in-between feature representations underlying the perceived images that can be predicted in neural decoding for (re-)generation of the originally perceived stimuli, leading to the most accurate reconstructions of perception to date.

# Stimuli

The stimuli folder contains the presented images (cropped, resized and superimposed fixation cross) and a Numpy file of 1086 latents of size (1086, 512) that these images are synthesised from using Progressive growing of GANs (for more information, go to https://github.com/tkarras/progressive_growing_of_gans). Note that none of the face images in this manuscript are of real people but instead computer-generated.

# Background information

You can find more information at https://doi.org/10.1038/s41598-021-03938-w.

# Restrictions on data access and reuse

The access to and use of this dataset is only allowed under the conditions listed in the data use agreement, as detailed below.

Neither the Donders Institute or Radboud University, nor the researchers that provide this dataset should be included as an author of publications or presentations if this authorship would be based solely on the use of this data.

However, we ask you to acknowledge the use of the data and data derived from the data when publicly presenting any results or algorithms that benefitted from their use:

1) Papers, book chapters, books, posters, oral presentations, and all other presentations of results derived from the data should acknowledge the origin of the data as follows: "Data were provided (in part) by the Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen".
2) Authors of publications or presentations using the data should cite relevant publications describing the methods developed and used by the Donders Institute to acquire and process the data. The specific publications that are appropriate to cite in any given study will depend on what the data were used for and for what purposes. When applicable, a list of publications will be specified on the collection overview page.

# Data use agreement for identifiable human data

Version RU-DI-HD-1.0

I request access to the data collected in the digital repository of the Donders Institute for Brain, Cognition and Behaviour, part of the Radboud University, established at Nijmegen, the Netherlands (hereinafter referred to as the Donders Institute), and I agree to the following:

1. I will comply with all relevant rules and regulations imposed by my institution and my government. This may mean that I need my research to be approved or declared exempt by a committee that oversees research on human subjects, e.g. my Institutional Review Board or Ethics Committee.

2. I will not attempt to establish the identity of or attempt to contact any of the included human subjects. I will not link this data to any other database in a way that could provide identifying information. I understand that under no circumstances will the code that would link these data to an individuals personal information be given to me, nor will any additional information about individual subjects be released to me under these Data Use Terms.

3. I will not redistribute or share the data with others, including individuals in my research group, unless they have independently applied and been granted access to this data.

4. I will acknowledge the use of the data and data derived from the data when publicly presenting any results or algorithms that benefitted from their use.
(a) Papers, book chapters, books, posters, oral presentations, and all other presentations of results derived from the data should acknowledge the origin of the data as follows: "Data were provided (in part) by the Donders Institute for Brain, Cognition and Behaviour".
(b) Authors of publications or presentations using the data should cite relevant publications describing the methods developed and used by the Donders Institute to acquire and process the data. The specific publications that are appropriate to cite in any given study will depend on what the data were used and for what purposes. When applicable, a list of publications will be included in the collection.
(c) Neither the Donders Institute or Radboud University, nor the researchers that provide this data should be included as an author of publications or presentations if this authorship would be based solely on the use of this data.

5. Failure to abide by these guidelines will result in termination of my privileges to access to these data.