Evidence for children’s online integration of simultaneous information from speech and iconic gestures: an ERP study

Children perceive iconic gestures, along with speech they hear. Previous studies have shown that children integrate information from both modalities. Yet it is not known whether children can integrate both types of information simultaneously as soon as they are available (as adults do) or whether they initially process them separately and integrate them later. Using electrophysiological measures, we examined the online neurocognitive processing of gesture-speech integration in 6- to 7-year-old children. We focused on the N400 event-related potential component which is modulated by semantic integration load. Children watched video clips of matching or mismatching gesture-speech combinations, which varied the semantic integration load. The ERPs showed that the amplitude of the N400 was larger in the mismatching condition than in the matching condition. This finding provides the first neural evidence that by the ages of 6 or 7, children integrate multimodal semantic information in an online fashion comparable to that of adults.

儿童在感知自身所听到的言语的同时，也会对标志性手势（iconic gestures）进行加工。既往研究表明，儿童能够整合来自两种模态的信息。但目前尚不明确，儿童究竟会如成人一般，在信息可获取的第一时间便同时整合这两类信息，还是会先分别对其进行加工，待后续再完成整合。本研究采用电生理测量手段，考察了6至7岁儿童对手势-言语整合的在线神经认知加工过程。我们重点关注了受语义整合负荷调控的N400事件相关电位（event-related potential, ERP）成分。实验中，儿童观看了匹配或不匹配的手势-言语组合视频片段，这些材料的语义整合负荷存在差异。事件相关电位结果显示，不匹配条件下的N400波幅显著高于匹配条件。该发现首次提供了神经层面的证据，表明6至7岁儿童已能够以与成人相当的在线方式完成多模态语义信息整合。