Recognition of map activities using eye tracking and EEG data

Recognizing the activities being performed on a map is crucial for adaptive map design based on user context. Despite eye tracking (ET) demonstrating potential in recognizing map activities and electroencephalography (EEG) measuring map users’ cognitive load, no studies have yet combined ET and EEG for recognition of the user’s activity on maps. Our study collected participants’ ET and EEG data during four types of map activities. After feature extraction and selection, we trained LightGBM (light Gradient-Boosting Machine) to classify these activities, and achieved 88.0% accuracy when combining ET and EEG features in the entire map usage trial, which is higher than using ET (85.9%) or EEG (53.9%) alone. Acceptable recognition accuracy could also be achieved with the early time windows (73.1% when using the first 3 seconds). Saccade features of ET were the most important for differentiating map activities, indicating selective map content for different tasks. Our findings demonstrate the feasibility and advantages of combining ET and EEG for activity recognition in map use. The results not only improve our understanding of visual patterns and cognitive processes in map use, but also enable the design of adaptive maps that can automatically adapt to the activities a map user is performing.识别地图场景下用户正在执行的操作，对于基于用户情境的自适应地图设计具有关键意义。尽管眼动追踪（Eye Tracking, ET）在地图操作识别领域已展现出应用潜力，脑电图（Electroencephalography, EEG）也可用于量化地图用户的认知负荷，但目前尚无研究将ET与EEG结合，以实现地图使用场景下的用户操作识别。本研究采集了参与者在四类地图操作过程中的ET与EEG多模态数据。在完成特征提取与特征选择后，我们训练了轻量级梯度提升机（Light Gradient-Boosting Machine, LightGBM）对上述四类地图操作进行分类。在全地图使用测试周期中，结合ET与EEG特征的模型分类准确率可达88.0%，显著高于单独使用ET（85.9%）或EEG（53.9%）的表现。即使仅使用早期时间窗口的特征数据，也可获得可接受的识别准确率：如仅使用前3秒的数据时，准确率可达73.1%。ET的眼跳特征是区分不同地图操作的最关键特征，这提示可针对不同任务需求选择性设计地图内容。本研究结果证实了将ET与EEG结合用于地图使用场景下操作识别的可行性与优势。该研究不仅加深了学界对地图使用过程中视觉模式与认知过程的理解，同时也为设计可自动适应用户当前操作的自适应地图提供了技术支撑。