Modality switch effects emerge early and increase throughout conceptual processing: Evidence from ERPs

Conference paper in <i>Proceedings of the 39th Annual Conference of the Cognitive Science Society.</i>Files: paper, poster, design overview, stimuli, EEG montage used, waveforms, window 1 close-ups, mean differences with 95% CIs, subtraction topographies, detailed waveforms, statistical results, fixed effects of final models, entire modeling, raw data. For best resolution, download file(s). Waveforms per participant and electrode (link). <b>OSF archive with all data and code (link).</b><b>Abstract </b>We tested whether conceptual processing is modality-specific by tracking the time course of the Conceptual Modality Switch effect. Forty-six participants verified the relation between property words and concept words. The conceptual modality of consecutive trials was manipulated in order to produce an Auditory-to-visual switch condition, a Haptic-to-visual switch condition, and a Visual-to-visual, no-switch condition. Event-Related Potentials (ERPs) were time-locked to the onset of the first word (property) in the target trials so as to measure the effect online and to avoid a within-trial confound. A switch effect was found, characterized by more negative ERP amplitudes for modality switches than no-switches. It proved significant in three typical time windows from 270 to 750 milliseconds post word onset, with greater strength in the Slow group, in posterior brain regions, and in the N400 window. These results suggest that conceptual processing may be modality-specific in certain tasks, but also that the early stage of processing is relatively amodal.<b></b><b></b><b></b><b></b><b></b><b></b>ReferencesCollins, J., Pecher, D., Zeelenberg, R., &amp; Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. <i>Frontiers in Psychology, 2.</i>Hald, L. A., Marshall, J.-A., Janssen, D. P., &amp; Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. <i>Frontiers in Psychology, 2.</i> Hauk, O. (2016). Only time will tell—Why temporal information is essential for our neuroscientific understanding of semantics. <i>Psychonomic Bulletin &amp; Review, 23</i>.<i></i>Louwerse, M., &amp; Connell, L. (2011). A taste of words: linguistic context and perceptual simulation predict the modality of words. <i>Cognitive Science, 35, </i>2, 381-98.Mahon, B.Z., &amp; Hickok, G. (2016). Arguments about the nature of concepts: Symbols, embodiment, and beyond. <em>Psychonomic Bulletin &amp; Review, 23, </em>941-958.

本数据集对应发表于《第39届认知科学学会年会论文集》（*Proceedings of the 39th Annual Conference of the Cognitive Science Society*）的会议论文。包含的文件包括：论文、海报、研究设计概述、实验刺激、所用脑电图（EEG, Electroencephalogram）电极排布方案、波形图、窗口1特写图、带有95%置信区间（CI, Confidence Interval）的平均差异图、减法拓扑分布图、详细波形图、统计结果、最终模型的固定效应项、完整建模流程以及原始数据。若需获取最佳分辨率的文件，请下载对应文件。每名被试与对应电极的波形图详见链接。**包含全部数据与代码的开放科学框架（OSF, Open Science Framework）存档（链接）**。

**摘要**
本研究通过追踪概念模态转换效应的时间进程，检验概念加工是否具有模态特异性。共招募46名被试完成属性词与概念词的关系验证任务。实验操纵了连续试次的概念模态，设置了三类实验条件：听觉-视觉转换条件、触觉-视觉转换条件，以及视觉-视觉无转换对照条件。事件相关电位（ERP, Event-Related Potentials）以靶试次中首个词（属性词）的呈现起始点为时间锁定点，以实现在线效应测量并避免试次内混淆。研究发现了显著的模态转换效应：相较于无转换条件，模态转换试次的ERP振幅更负。该效应在词呈现后270至750毫秒的三个典型时窗内均达到统计显著水平，且在慢波组、后部脑区以及N400时窗中效应强度更强。上述结果表明，部分任务中的概念加工可能具有模态特异性，但加工的早期阶段相对非模态化。

参考文献
[1] Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). 属性验证任务中的模态转换：一项ERP研究探讨蜡烛闪烁与高跟鞋咔嗒声后的认知过程。《前沿心理学》（*Frontiers in Psychology*）, 2.
[2] Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). 句子验证任务中的模态转换：具身语言加工的ERP证据。《前沿心理学》, 2.
[3] Hauk, O. (2016). 唯有时间能揭晓答案——为何时间信息对语义的神经科学研究至关重要。《心理科学通报与评论》（*Psychonomic Bulletin & Review*）, 23.
[4] Louwerse, M., & Connell, L. (2011). 词语的味觉体验：语言语境与知觉模拟可预测词语的模态属性。《认知科学》（*Cognitive Science*）, 35(2), 381-398.
[5] Mahon, B.Z., & Hickok, G. (2016). 概念本质之争：符号、具身认知及其延伸。《心理科学通报与评论》, 23, 941-958.