Multimodal WEDAR dataset for attention regulation behaviors, self-reported distractions, reaction time, and knowledge gain in e-reading

Diverse learning theories have been constructed to understand learners' internal states through various tangible predictors. We focus on self-regulatory actions that are subconscious and habitual actions triggered by behavior agents' 'awareness' of their attention loss. We hypothesize that self-regulatory behaviors (i.e., attention regulation behaviors) also occur in e-reading as 'regulators' as found in other behavior models (Ekman, P., &amp; Friesen, W. V., 1969). In this work, we try to define the types and frequencies of attention regulation behaviors in e-reading. We collected various cues that reflect learners' moment-to-moment and page-to-page cognitive states to understand the learners' attention in e-reading. <br>The text 'How to make the most of your day at Disneyland Resort Paris' has been implemented on a screen-based e-reader, which we developed in a pdf-reader format. An informative, entertaining text was adopted to capture learners' attentional shifts during knowledge acquisition. The text has 2685 words, distributed over ten pages, with one subtopic on each page. A built-in webcam on Mac Pro and a mouse have been used for the data collection, aiming for real-world implementation only with essential computational devices. A height-adjustable laptop stand has been used to compensate for participants' eye levels.<br>Thirty learners in higher education have been invited for a screen-based e-reading task (M=16.2, SD=5.2 minutes). A pre-test questionnaire with ten multiple-choice questions was given before the reading to check their prior knowledge level about the topic. There was no specific time limit to finish the questionnaire. We collected cues that reflect learners' moment-to-moment and page-to-page cognitive states to understand the learners' attention in e-reading. Learners were asked to report their distractions on two levels during the reading: 1) In-text distraction (e.g., still reading the text with low attentiveness) or 2) out-of-text distraction (e.g., thinking of something else while not reading the text anymore). We implemented two noticeably-designed buttons on the right-hand side of the screen interface to minimize possible distraction from the reporting task. After triggering a new page, we implemented blur stimuli on the text in the random range of 20 seconds. It ensures that the blur stimuli occur at least once on each page. Participants were asked to click the de-blur button on the text area of the screen to proceed with the reading. The button has been implemented in the whole text area, so participants can minimize the effort to find and click the button. Reaction time for de-blur has been measured, too, to grasp the arousal of learners during the reading. We asked participants to answer pre-test and post-test questionnaires about the reading material. Participants were given ten multiple-choice questions before the session, while the same set of questions was given after the reading session (i.e., formative questions) with added subtopic summarization questions (i.e., summative questions). It can provide insights into the quantitative and qualitative knowledge gained through the session and different learning outcomes based on individual differences. A video dataset of 931,440 frames has been annotated with the attention regulator behaviors using an annotation tool that plays the long sequence clip by clip, which contains 30 frames. Two annotators (doctoral students) have done two stages of labeling. In the first stage, the annotators were trained on the labeling criteria and annotated the attention regulator behaviors separately based on their judgments. The labels were summarized and cross-checked in the second round to address the inconsistent cases, resulting in five attention regulation behaviors and one neutral state. See WEDAR_readme.csv for detailed descriptions of features.<br>The dataset has been uploaded 1) raw data, which has formed as we collected, and 2) preprocessed, that we extracted useful features for further learning analytics based on real-time and post-hoc data.<br><br><br>ReferenceEkman, P., &amp; Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. <em style="color:rgb(34, 34, 34);">semiotica</em>, <em style="color:rgb(34, 34, 34);">1</em>(1), 49-98.

学界已构建多种学习理论，旨在通过各类可观测预测指标解析学习者的内在认知状态。本研究聚焦于由行为主体察觉到自身注意力涣散而触发的潜意识习惯性自我调节行为（self-regulatory behaviors）。我们假设，自我调节行为（即注意力调节行为（attention regulation behaviors））同样存在于电子阅读（e-reading）场景中，如同其他行为模型中所定义的“调节者”角色（Ekman, P. & Friesen, W. V., 1969）。本研究旨在明确电子阅读场景中注意力调节行为的类型与发生频率，并通过采集各类可反映学习者逐时逐页认知状态的线索，以解析其在电子阅读过程中的注意力状态。
我们以自研的PDF阅读器格式屏幕端电子阅读器，呈现文本《如何充分玩转巴黎迪士尼乐园》。为捕捉学习者在知识获取过程中的注意力转移情况，本研究选用了兼具知识性与趣味性的文本。该文本共计2685词，分为10页，每页对应一个子主题。
数据采集仅使用基础计算设备，依托Mac Pro内置摄像头与鼠标完成。同时采用高度可调节笔记本支架，以适配参与者的视线高度。
本研究招募了30名高等教育阶段学习者参与屏幕端电子阅读任务（任务时长均值M=16.2，标准差SD=5.2分钟）。阅读开始前，参与者需完成一份包含10道单选题的前测问卷，用以评估其对本次阅读主题的先验知识水平，问卷作答无明确时限。
研究采集可反映学习者逐时逐页认知状态的线索，以解析其电子阅读中的注意力状态。同时要求学习者在阅读过程中从两个维度报告自身的分心情况：1）文本内分心（例如：仍在阅读文本但注意力低下）；2）文本外分心（例如：停止阅读文本后思绪游离）。
我们在屏幕界面右侧设置了两个辨识度较高的按钮，以尽可能降低报告任务本身带来的分心干扰。在触发新页面加载后，系统会在20秒的随机区间内对文本施加模糊刺激，确保每页文本至少出现一次模糊刺激。参与者需点击屏幕文本区域内的去模糊按钮，方可继续阅读。该按钮覆盖整个文本区域，以减少参与者寻找并点击按钮的操作成本。此外，研究还记录了去模糊操作的反应时，以捕捉学习者在阅读过程中的唤醒水平。
我们要求参与者完成针对本次阅读材料的前测与后测问卷。实验开始前，参与者需完成10道单选题；阅读结束后，除相同的10道形成性测试题外，额外增设子主题总结类题目（即总结性测试题）。该问卷设计可用于量化与质性分析参与者在本次实验中获取的知识，以及基于个体差异产生的不同学习结果。
本研究的视频数据集包含931,440帧画面，我们使用逐30帧片段播放的标注工具，对数据集中的注意力调节行为进行了标注。两名标注者（均为博士生）完成了两阶段的标注工作：第一阶段，标注者先接受标注准则培训，随后基于自身判断独立完成注意力调节行为的标注；第二阶段，研究人员对标注结果进行汇总与交叉核验，以处理标注不一致的案例，最终得到5类注意力调节行为与1类中性状态。特征的详细说明请参见WEDAR_readme.csv文件。
本数据集包含两部分内容：1）原始采集数据，即实验过程中直接记录的原始数据；2）预处理数据，即基于实时与事后采集的数据，提取得到可供后续学习分析使用的有效特征。
参考文献：Ekman, P. 与 Friesen, W. V. (1969). 非言语行为总目：分类、起源、应用与编码。《符号学》（*Semiotica*），第1卷第1期，49-98页。