Table_3_Modeling items for text comprehension assessment using confirmatory factor analysis.xlsx

Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. However, only little research addresses distinct representational levels when text comprehension is assessed, analyzed or modelled. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni-or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

阅读是一项复杂的认知任务，其终极目标在于理解书面输入内容。对于更长的连贯文本，读者会构建起作为理解基础的心理表征（mental representation）。由于认知资源有限，主流的语篇表征（discourse representation）模型假设存在不同的加工层级，每个层级依赖专属的加工机制。然而，在对文本理解进行评估、分析或建模的研究中，仅有极少部分关注了不同的表征层级。此外，当前尝试将阅读的加工指标（如阅读时长、眼动轨迹）与理解表现相关联的研究，并未将理解视为多维度构念，而是将其视作单维度构念，且通常仅通过单题测试条目（one-shot items）进行评估。因此，本研究的首要目标是采用验证性因素分析（confirmatory factor analysis, CFA），检验文本理解能否被建模为单维度或多维度概念。本研究的第二项目标是探究当前广泛使用的单题测试条目，在多大程度上能够有效捕捉文本理解能力。本研究共招募400名被试，让其阅读三篇长度、语言特征及复杂度均相近的短篇故事中的任意一篇。基于每篇故事由3名独立评分员开展的评估，我们编制了16道特殊疑问句（wh-questions）题目与60道是非判断题（yes/no-statements），以提取被试总结中的微观层面（micro level）与推理层面（inference level）信息；同时提取16项核心内容，以捕捉宏观层面（macro level）的信息。然而，仅有少量测试条目表现出令人满意的心理测量学属性（psychometric properties）与因子载荷（factor loadings）——结合当前条目筛选的通用实践来看，这一结果着实引人深思。在验证性因素分析中，我们构建了两种模型：一种将文本理解视为单维度构念（即仅包含单一理解因子的单因子模型（uni-factor model）），另一种则将其视为反映三种不同表征层级的三维度构念（即三个相关的一阶因子（first-order factors））。无论针对哪篇故事或哪种题目类型，三维因子模型的拟合度（model fit）均始终更优，这为文本理解属于多维度构念提供了实证支持。本研究结果可为阅读加工相关研究中理解能力测评工具的编制提供具体的实践指导。