Modality switch effects emerge early and increase throughout conceptual processing: Evidence from ERPs

This repository contains all the data from our study, including: conference paper and poster, stimuli norming, experiment set-up, behavioural pretest, main behavioural and electrophysiological data, and extensive R code for descriptives, plots, and statistical analysis. The main analysis files may be found in the folder path: "ERPs" then "Analyses of ERPs averaged across trials".

Edit and run any of the R code in RStudio: https://mybinder.org/v2/gh/pablobernabeu/Modality-switch-effects-emerge-early-and-increase-throughout-conceptual-processing/master?urlpath=rstudio

Paper: https://doi.org/10.31234/osf.io/a5pcz

Abstract
We tested whether conceptual processing is modality-specific by tracking the time course of the Conceptual Modality Switch effect. Forty-six participants verified the relation between property words and concept words. The conceptual modality of consecutive trials was manipulated in order to produce an Auditory-to-visual switch condition, a Haptic-to-visual switch condition, and a Visual-to-visual, no-switch condition. Event-Related Potentials (ERPs) were time-locked to the onset of the first word (property) in the target trials so as to measure the effect online and to avoid a within-trial confound. A switch effect was found, characterized by more negative ERP amplitudes for modality switches than no-switches. It proved significant in four typical time windows from 160 to 750 milliseconds post word onset, with greater strength in the Slow group, in posterior brain regions, and in the N400 window. The earliest switch effect was located in the language brain region, whereas later it was more prominent in the visual region. In the N400 and Late Positive windows, the Quick group presented the effect especially in the language region, whereas the Slow had it rather in the visual region. These results suggest that contextual factors such as time resources modulate the engagement of linguistic and embodied systems in conceptual processing.

Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). Modality switch effects emerge early and increase throughout conceptual processing: Evidence from ERPs. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. J. Davelaar (Eds.), Proceedings of the 39th Annual Conference of the Cognitive Science Society (pp. 1629-1634). Austin, TX: Cognitive Science Society. https://doi.org/10.31234/osf.io/a5pcz

See also further analyses available in this repository.

References
Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. Frontiers in Psychology, 2.
Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. Frontiers in Psychology, 2.
Hauk, O. (2016). Only time will tell—Why temporal information is essential for our neuroscientific understanding of semantics. Psychonomic Bulletin & Review, 23.
Louwerse, M., & Connell, L. (2011). A taste of words: linguistic context and perceptual simulation predict the modality of words. Cognitive Science, 35, 2, 381-98.
Mahon, B.Z., & Hickok, G. (2016). Arguments about the nature of concepts: Symbols, embodiment, and beyond. Psychonomic Bulletin & Review, 23, 941-958.

The modality exclusivity norms used for the stimuli are attached to this project as a linked component.

Plot waveforms within any sections of the data at https://pablobernabeu.shinyapps.io/ERP-waveform-visualization_CMS-experiment. More general visualizations at https://figshare.com/articles/EEG_study_on_conceptual_modality-switching_Bernabeu_et_al_in_prep_/4210863. See modality norms used for the stimuli of the modality switch experiment (https://pablobernabeu.shinyapps.io/dutch-modality-exclusivity-norms/; in case of downtime, please visit https://pablobernabeu.github.io/dashboards/Dutch-modality-exclusivity-norms/d.html).

本存储库包含了我们研究中的所有数据，包括：会议论文及海报、刺激物标准化、实验设置、行为预测试、主要行为和电生理数据，以及用于描述性统计、图表和统计分析的广泛R代码。主要分析文件可在我路径“ERPs”下的“试次平均ERP分析”文件夹中找到。编辑并运行RStudio中的任何R代码：https://mybinder.org/v2/gh/pablobernabeu/Modality-switch-effects-emerge-early-and-increase-throughout-conceptual-processing/master?urlpath=rstudio

论文：https://doi.org/10.31234/osf.io/a5pcz

摘要
我们通过追踪概念模式切换效应的时间进程来检验概念处理是否具有模式特异性。四十六名参与者验证了属性词与概念词之间的关系。通过操纵连续试验的概念模式，产生了听觉至视觉切换条件、触觉至视觉切换条件以及视觉至视觉、无切换条件。事件相关电位（ERP）与目标试验中第一个词（属性词）的出现时间同步，以在线测量效应并避免试验内的混杂。研究发现存在切换效应，表现为模式切换时的ERP振幅比无切换时更负。该效应在单词出现后160至750毫秒的四个典型时间窗口内显著，在慢速组、后脑区域以及N400窗口中表现更为强烈。最早的切换效应位于语言脑区，而后期则更多地在视觉区域中凸显。在N400和晚期正波窗口中，快速组在语言区域表现出效应，而缓慢组则更多地在视觉区域中表现出效应。这些结果表明，诸如时间资源等情境因素调节了语言和具身系统在概念处理中的参与程度。

Bernabeu, P., Willems, R. M., & Louwerse, M. M. (2017). 模式切换效应在概念处理中早期出现并增强：ERP的证据。见G. Gunzelmann, A. Howes, T. Tenbrink, & E. J. Davelaar (编)，第39届认知科学学会年度会议论文集（第1629-1634页）。奥斯汀，德克萨斯州：认知科学学会。https://doi.org/10.31234/osf.io/a5pcz

参见本存储库中提供的进一步分析。

参考文献
Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). 在属性验证任务中的模式切换：蜡烛在高跟鞋点击后闪烁的ERP研究。心理学前沿，2。
Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). 句子验证任务中的模式切换：ERP证据表明了具身语言处理。心理学前沿，2。
Hauk, O. (2016). 只有时间会告诉我们——为什么时间信息对于我们对语义的神经科学理解至关重要。心理科学通报与评论，23。
Louwerse, M., & Connell, L. (2011). 单词的味道：语言环境和感知模拟预测了单词的模式。认知科学，35，2，381-98。
Mahon, B.Z., & Hickok, G. (2016). 关于概念本质的论点：符号、具身以及超越。心理科学通报与评论，23，941-958。

用于刺激物的模式独特性标准已作为链接组件附加到本项目。

在https://pablobernabeu.shinyapps.io/ERP-waveform-visualization_CMS-experiment上查看数据中任何部分的波形图。更通用的可视化可在https://figshare.com/articles/EEG_study_on_conceptual_modality-switching_Bernabeu_et_al_in_prep_/4210863上找到。有关模式切换实验中刺激物的模式标准，请参阅https://pablobernabeu.shinyapps.io/dutch-modality-exclusivity-norms/（如果服务中断，请访问https://pablobernabeu.github.io/dashboards/Dutch-modality-exclusivity-norms/d.html）。