Data from: Feeling Senseless Sensations: A crossmodal study of a mismatching tactile and virtual visual experience

To create highly immersive experiences in virtual reality (VR) it is important to not only include the visual sense but also to involve multimodal sensory input. To achieve optimal results, the temporal and spatial synchronization of these multimodal inputs is critical. It is therefore necessary to find methods to objectively evaluate the synchronization of VR experiences with a continuous tracking of the user. In this study a passive touch experience was incorporated in a visual-tactile VR setup using VR glasses and tactile sensations in mid-air. Inconsistencies of multimodal perception were intentionally integrated into a discrimination task. The participants’ electroencephalogram (EEG) was recorded to obtain neural correlates of visual-tactile mismatch situations. The results showed significant differences in the event-related potentials (ERP) between match and mismatch situations. A biphasic ERP configuration consisting of a positivity at 120 ms and a later negativity at 370 ms was observed following a visual-tactile mismatch. This late negativity could be related to the N400 that is associated with semantic incongruency. These results provide a promising approach towards the objective evaluation of visual-tactile synchronization in virtual experiences.

若要打造高度沉浸的虚拟现实（Virtual Reality）体验，仅覆盖视觉感知维度仍有不足，还需纳入多模态感官输入。为达成最优效果，此类多模态输入的时空同步性至关重要。因此，亟需开发可通过对用户进行持续追踪，以客观评估VR体验同步性的方法。本研究将被动触觉体验融入一套视觉-触觉VR实验装置中，该装置依托VR眼镜与空中触觉刺激构建。研究有意将多模态感知的不一致性嵌入辨别任务之中，同时记录参与者的脑电图（Electroencephalogram, EEG），以捕捉视觉-触觉失匹配情境下的神经关联信号。研究结果显示，匹配与失匹配情境下的事件相关电位（Event-related Potentials, ERP）存在显著差异。视觉-触觉失匹配后，可观察到双相ERP波形：120ms时出现正电位，后续370ms时出现负电位。该晚期负电位或与语义不一致相关的N400存在关联。上述研究结果为虚拟体验中视觉-触觉同步性的客观评估提供了极具前景的可行路径。