The timing of an avatar’s beat gestures biases lexical stress perception in vocoded speech

This repository contains experimental data and scripts for the study "The timing of an avatar’s beat gestures biases lexical stress perception in vocoded speech" (Maran et al., 2025). For the stimuli, see the associated repository di.dcc.DSC_2023.00140_719 (https://doi.org/10.34973/b8k1-q907). Cochlear implants (CIs) are neural prostheses that restore some level of hearing capacity, albeit conveying a less fine-grained speech signal than normal hearing conditions. For example, CIs convey altered fundamental frequency (F0) information, resulting in atypical lexical stress perception (e.g., distinguishing between the noun CONtent and the adjective conTENT) in languages in which this feature rests on F0 modulations. CI users can compensate for the degraded nature of the acoustic input by exploiting the audiovisual affordances of human communication, weighing more heavily the visual information provided by speakers (e.g., lip movements and gestures). Recent studies showed that, in individuals with normal hearing, the timing of simple up-and-down movements of the hand (i.e., beat gestures) biases lexical stress perception. The present study tested if the timing of beat gestures produced by an avatar can bias Dutch lexical stress perception in vocoded speech, which limits the reliability of F0 information in a way that mimics CI-hearing conditions. The effect of gestures in vocoded speech was particularly pronounced when hearing an ambiguous or the least frequent stress pattern in Dutch. These results suggest that (even artificially generated) beat gestures can support the perception of vocoded speech, especially when processing less frequent prosodic features. Acknowledgements: Funded by an ERC Starting Grant (HearingHands, 101040276) from the European Union awarded to Hans Rutger Bosker. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.