Early integration of conceptual modality in word recognition: ERP evidence for perceptual simulation (Bernabeu, Willems, & Louwerse, in prep.)

Files: experiment overview; stimuli; EEG montage used; waveforms; difference topographies, critical statistics; fixed effects of final models; entire modeling; raw data. For a detailed view of the waveforms per group and electrode, see https://pablobernabeu.shinyapps.io/export_files/.The entire data set is available at https://osf.io/97unm/.<b>Abstract. </b>The engagement of sensory brain regions during word recognition is widely documented, yet its precise relevance is less clear. It would constitute perceptual simulation only if it has a functional role at the core of conceptual processing, as opposed to epiphenomenal processes. We investigated this in an Event-Related Potential (ERP) experiment implementing the conceptual modality switch paradigm. In each trial, participants verified the relation between a property word and a concept word. Orthogonally, we manipulated the conceptual modality of successive trials, and tested whether switching modalities incurred any processing costs at different stages of word recognition. Unlike previous studies, we time-locked ERPs to the first word of target trials, in order to measure the modality transitions from the beginning, and also to reduce confounds. Further, we included different types of switch—one from auditory to visual modality, and one from haptic to visual—, which were compared to the non-switch—visual to visual. Also, one group of participants was asked to respond quickly (<i>n</i> = 21), and another group to respond self-paced (<i>n</i> = 21), whilst a few others received no constraints (<i>n</i> = 5). We found ERP effects in four typical time windows from 160 to 750 ms after word onset. The overall effect is characterized by a negativity for modality-switching relative to not switching, and it increases over time. Further, the effect arises with both types of switch, and influences both participant groups within anterior and posterior brain regions. The emergence of this effect in the first time window particularly suggests that sensory brain systems may have a functional role in conceptual processing. The increase in the effect at later stages converges with previous studies, and invites further investigation.<b></b><b></b><b></b><b></b><b></b><b></b>ReferencesCollins, J., Pecher, D., Zeelenberg, R., &amp; Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. <i>Frontiers in Psychology, 2.</i>Hald, L. A., Marshall, J.-A., Janssen, D. P., &amp; Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. <i>Frontiers in Psychology, 2.</i> Hauk, O. (2016). Only time will tell—Why temporal information is essential for our neuroscientific understanding of semantics. <i>Psychonomic Bulletin &amp; Review, 23</i>.<i></i>Louwerse, M., &amp; Connell, L. (2011). A taste of words: linguistic context and perceptual simulation predict the modality of words. <i>Cognitive Science, 35, </i>2, 381-98.Mahon, B.Z., &amp; Hickok, G. (2016). Arguments about the nature of concepts: Symbols, embodiment, and beyond. <em>Psychonomic Bulletin &amp; Review, 23, </em>941-958.