Multimodal SKEP dataset for attention regulation behaviors, knowledge gain, perceived learning experience, and perceived social presence in e-learning with a conversational agent

Reading on digital devices has become more commonplace, often challenging learners' attention. In this study, we hypothesized that allowing learners to reflect on their reading phases with an empathic social robot companion might enhance learners' attention in e-reading. To verify our assumption, we collected a novel SKEP dataset in an e-reading setting with social robot support.<br>We designed two interfaces: 1) a GUI-based system with a monitor, mouse, and eye tracker implemented, and 2) an HRI-based system, which has a monitor, mouse, eye tracker, and Furhat Robot as physical components. See the footnote to check the specification of the Pupil Core eye tracker and Logitech C505 HD Webcam that was implemented. For both conditions, an informative e-reading material with technicality, "Waste management and critical raw materials," has been provided through a screen-based reader, which we explicitly developed for this study. The content has been chosen, aiming for an equal baseline knowledge for general readers. The text contains 4,750 words, divided into 29 pages covering seven subtopics. The text has been implemented with 47pt on a 27-inch monitor, having 2560*1440 resolution. The setting was optimized for the eye tracker implementation, which requires a bigger font size than the usual PDF readers for high-resolution data collection.<br>We implemented four measurements that are direct and indirect attentional cues. Data features and granularity varies based on the data collection methods, collection timing, and data post-processing. Learners' self-regulatory behavior has been collected through a video feed and annotated second-by-second by human labelers as post hoc. Labels are observable behavioral cues that indicate learners' attentional shifts. Movements from the 1) eyebrow, 2) blink, 3) mumble, 4) hands, and 5) body works as good predictors of learners' self-awareness on attention loss; we annotated 60 video samples by applying six labels, including 6) neutral state as opposed to five attention regulation behavior labels. Additionally, we examined multimodal cues that are direct and indirect clues of attention: knowledge gain, perceived learning experience, and perceived social presence with interfaces (see readme.txt for descriptions of indicators).

电子设备阅读已愈发普及，却常常对学习者的注意力造成干扰。本研究提出假设：允许学习者借助具备共情能力的社交机器人伴侣（empathic social robot companion）回顾自身阅读阶段，可提升电子阅读（e-reading）过程中的注意力水平。为验证这一设想，我们在搭载社交机器人支持的电子阅读场景中采集了全新的SKEP数据集。<br>我们设计了两类交互界面：1）基于图形用户界面（Graphical User Interface，GUI）的系统，配备显示器、鼠标与眼动追踪器；2）基于人机交互（Human-Robot Interaction，HRI）的系统，除上述组件外，额外搭载Furhat机器人（Furhat Robot）作为实体交互部件。如需了解所使用的Pupil Core眼动追踪器（Pupil Core eye tracker）与罗技C505高清网络摄像头（Logitech C505 HD Webcam）的规格，请参阅脚注。两类实验条件均采用针对本研究专门开发的屏幕阅读器，推送兼具专业性与信息量的电子阅读材料《废弃物管理与关键原材料》。该内容的选取旨在为普通读者构建均衡的基线知识储备。文本共计4750词，分为29页，涵盖7个细分主题。文本在27英寸、分辨率为2560×1440的显示器上采用47pt字号呈现——这一设置是为眼动追踪器采集优化的，相较于常规PDF阅读器需要更大的字号以保障高精度数据采集。<br>我们设置了四类直接与间接注意力线索测量方案。数据特征与粒度会根据数据采集方法、采集时机以及后期处理流程的不同而有所差异。学习者的自我调节行为通过视频流采集，并由人工标注员事后逐秒进行标注。标注标签为可观测的行为线索，用于指示学习者的注意力转移情况。以下五类行为可有效预测学习者对注意力涣散的自我觉察：1）眉部动作、2）眨眼、3）喃喃自语、4）手部动作、5）身体动作。本次共标注了60份视频样本，除上述5种注意力调节行为标签外，额外设置6）中性状态作为对照标签。此外，我们还针对注意力的直接与间接多模态线索展开了分析，包括知识增益、感知学习体验以及对交互界面的感知社交存在感（各项指标的详细说明请参阅readme.txt文件）。