Cross Modal Phonetic Encoding Data files and Trigger (Main and Auditory-Only Experiment)

Auditory Only Experiment_NH1-13_DeciFiles: In the auditory-only experiment, subjects listened to acoustic tokens while watching a still image of the speaker’s face. The acoustic tokens included 6 original (unaltered) tokens (3 /ba/ and 3 /wa/) and 6 altered tokens. All stimuli were randomized within 2 blocks of presentation. Each block lasted about 14.5 minutes and contained 182 trials. The purpose of the auditory-only experiment was to establish CV-specific amplitude and latency references for the N1 AEP, so that N1 shifts observed in the main experiment could be evaluated with respect to the auditory-only references.Main Experiment_NH1-17_DeciFiles: In the main experiment, the six altered acoustic tokens and the two silent videos (with mouth movements) were mixed and matched to create congruent and incongruent pairs of AV stimuli, in addition to visual-only (silent video) stimuli and noise-replaced stimuli. All stimuli were randomized and presented in 6 blocks. Each block lasted about 12.5 minutes and had 158 trials. There were 8 stimulus conditions and 2 possible responses (‘ba’/’bwa’ or ‘wa’/’wba’) for each stimulus condition, resulting in 16 percept types. The 8 stimulus conditions were as follows: the visual-only stimuli consisted of silent video /ba/ and silent video /wa/; the congruent stimuli consisted of video /ba/ combined with audio /ba/, and video /wa/ combined with audio /wa/; the incongruent stimuli consisted of video /ba/ combined with audio /wa/, and video /wa/ combined with audio /ba/; and the noise-replaced stimuli consisted of video /ba/ combined with noise-replaced audios (either /ba/ or /wa/) or video /wa/ combined with noise-replaced audios.We hypothesized that the N1 amplitude during illusory perception should shift to reflect the N1 characteristics ofthe phonemes conveyed visually (by mouth movements) as opposed to acoustically. Indeed, the N1 AEP became larger and occurred earlier when listeners experienced illusory ‘ba’ (video /ba/, audio /wa/, heard as ‘ba’) and vice versa when they experienced illusory ‘wa’ (video /wa/, audio /ba/, heard as ‘wa’), mirroring the N1 characteristics for /ba/ and /wa/ observed in natural acoustic situations (e.g., auditory-only setting). This visually-mediated N1 behavior was also observed for noise-replaced CVs. Taken together, the findings suggest that information relayed by the visual modality modifies phonetic representations and that similar neural mechanisms support the McGurk illusion and phonemic restoration.