christiqn/EVT-items

--- language: - en license: cc-by-4.0 task_categories: - text-classification task_ids: - multi-class-classification tags: - expectancy-value-theory - psychology - psychometrics - educational-psychology - motivation - survey-instruments - content-analysis pretty_name: EVT Evaluation Set — Human-Coded Psychological Test Items size_categories: - 1K<n<10K --- EVT Evaluation Set ## Dataset Summary This dataset contains human-coded psychological test items drawn from published, peer-reviewed instruments measuring components of **Expectancy-Value Theory** (EVT; Eccles et al., 1983; Eccles & Wigfield, 2002). It was constructed as an out-of-domain evaluation benchmark for the [EVT Item Classifier](https://huggingface.co/christiqn/mpnet-EVT-classifier) and was **not used during model training**, which relied exclusively on synthetically generated items. It therefore provides an ecologically valid test of generalisation from synthetic to real, human-authored scale items. | Property | Value | |---|---| | Total items (raw) | 1,632 | | Items in final evaluation set | 1,284 | | Items excluded from evaluation | 348 | | Items used in benchmark | 1,284 | | EVT constructs | 6 (5 core + OTHER) | | Source instruments | 95 | | Language | English | | Domain | Educational psychology, motivation science, psychometrics | ## Source Data Collection ### Systematic Literature Search To gather a comprehensive database of published items measuring components of EVT, a systematic literature search was conducted in the **Web of Science** database using the following query: ``` TS=("expectancy-value theory" OR "expectancy-value model" OR "situated expectancy-value" OR SEVT OR "expectanc* for success" OR "ability belief*" OR "subjective task value" OR "attainment value" OR "intrinsic value" OR "utility value") AND TS=(measure* OR instrument* OR scale* OR questionnaire* OR item* OR inventory OR assess* OR psychometric* OR validat*) ``` During the initial title and abstract screening, studies were included only if they explicitly reported the assessment of at least one component of EVT. Inclusion was further restricted to: - English-language publications - Published, peer-reviewed papers - Papers available under Open-Access licenses - Papers accessible through the Crossref, Unpaywall, or OpenAlex API These constraints were imposed to facilitate a fully automated retrieval and extraction pipeline. ### Automated Item Extraction Following screening, full-text PDFs were retrieved automatically. Item extraction used **optical character recognition (OCR)**, which iterated through the raw text of each downloaded manuscript to identify reported scale items. Extracted strings were preprocessed to: - Remove OCR artefacts (character substitution errors, line-break artefacts) - Strip citation brackets and in-text references - Remove numerical item prefixes (e.g., "1.", "Q3:") This produced a set of clean, unlabeled text strings representing candidate scale items. ### Manual Filtering Candidate items were then manually reviewed and filtered to remove: - Items not clearly formulated as self-report items (e.g., instruction text, anchor labels) - Items not externally assessable without additional context - Items confounded by multiple response anchors that would prevent unambiguous construct assignment Items passing manual filtering were mapped to their EVT construct by a human expert coder, or flagged as `OTHER` if they did not correspond to any EVT component. ## Dataset Structure ### File Format Semicolon-delimited CSV (UTF-8 with BOM), 1,632 rows + header, 4 columns. ### Fields | Field | Type | Description | |---|---|---| | `scale` | string | Name of the source instrument or questionnaire. Set to `NOT_REPORTED` when the original publication did not name the scale. | | `item` | string | Text of the psychological test item, preprocessed to remove OCR artefacts, citation brackets, and numerical prefixes. | | `human_coder` | string | EVT construct label assigned by a human expert. One of: `ATTAINMENT_VALUE`, `COST`, `EXPECTANCY`, `INTRINSIC_VALUE`, `UTILITY_VALUE`, `OTHER`. | | `decision` | string / null | Exclusion flag. Non-null values indicate why the item was excluded from the primary evaluation: `third-person`, `multiple anchors`, or `other`. Null = item is included. | ### Class Distribution | EVT Construct | Description | N (raw) | |---|---|---| | `EXPECTANCY` | Beliefs about future success | 308 | | `OTHER` | Not classifiable as an EVT construct | 283 | | `INTRINSIC_VALUE` | Enjoyment and interest | 234 | | `UTILITY_VALUE` | Usefulness for future goals | 199 | | `COST` | Perceived negative consequences of engagement | 149 | | `ATTAINMENT_VALUE` | Personal importance of doing well | 111 | | **Total** | | **1,284** | > Items with a null `human_coder` value in the raw file correspond to rows where no EVT label was assigned (e.g., `NOT_REPORTED` scale entries or unlabelled extractions). These are not included in the evaluation set. ## Exclusion Criteria and the `decision` Column Items that passed manual filtering but were identified as methodologically problematic are retained in the dataset with a non-null `decision` value. These items were excluded from the primary classifier evaluation but are preserved for secondary analyses. | Exclusion Reason | Description | N | |---|---|---| | `third-person` | Items formulated about a third party rather than the respondent directly (e.g., "students find this interesting"), making self-report intent ambiguous. | 67 | | `multiple anchors` | Items where the response scale contains multiple distinct conceptual anchors that confound construct assignment. | 29 | | `other` | OCR artefacts, formatting fragments, instruction text, missing context, or items that were not self-contained scale items. | 252 | | **Total excluded** | | **348** | Items used in the primary model evaluation are those where `decision` is null (**N = 1,284**). ## Instrument Coverage Items were sourced from **95 distinct instruments** spanning a broad range of academic domains and populations. This diversity was intentional: the dataset tests whether EVT construct labels generalise across contexts, not just within a single instrument or domain. ### Domain Coverage - Mathematics and statistics education - Science, technology, engineering, and mathematics (STEM) - Language learning (ESL, reading, writing) - Health and clinical domains (e.g., breastfeeding motivation, occupational safety) - Music and arts education - Higher education and career motivation - General academic motivation (cross-domain instruments) ## Coding Procedure ### Construct Definitions Items were coded according to the theoretical definitions from Eccles et al. (1983) and Eccles & Wigfield (2002): - **ATTAINMENT_VALUE** — The personal importance of doing well on a task, tied to the individual's identity and self-concept. - **COST** — The perceived negative consequences of engagement, including opportunity cost, effort expenditure, and emotional cost. - **EXPECTANCY** — Beliefs about future success on a task, including ability self-concept and confidence. - **INTRINSIC_VALUE** — The enjoyment, interest, or subjective pleasure derived from engagement with a task. - **UTILITY_VALUE** — The perceived usefulness of a task for future goals, careers, or activities. - **OTHER** — Items that do not correspond to any of the five core EVT constructs. ### Coding Rules A single label was assigned per item. When an item could plausibly belong to multiple constructs, the coder assigned the label corresponding to the most salient construct, or flagged the item as `multiple anchors` if the ambiguity was considered irresolvable. Items formulated in the third person were flagged as `third-person` and excluded from the primary evaluation. ## Intended Uses - Benchmark evaluation of automated EVT construct classifiers - Annotation studies and inter-rater reliability research - Training data for future classifiers (with appropriate domain-shift caveats) - Systematic review and content analysis of EVT instrumentation - Psychometric research on construct validity and item coverage across instruments ## Limitations - **English only.** All items are in English. Cross-lingual generalisation should not be assumed. - **Synthetic-to-real gap.** The associated classifier was trained on synthetic items. This dataset documents real-world generalisation but does not eliminate the distribution gap. - **Open-access bias.** Inclusion was limited to open-access publications, which may not be fully representative of the EVT instrumentation literature. - **OCR noise.** Despite preprocessing, some items may retain minor OCR artefacts from the extraction pipeline. - **Domain imbalance.** Certain domains (mathematics, STEM) are more heavily represented than others (health, arts), reflecting the literature distribution rather than deliberate sampling. ## References - Eccles, J. S., et al. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), *Achievement and achievement motives* (pp. 75–146). W. H. Freeman. - Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. *Annual Review of Psychology, 53*(1), 109–132. - Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. *Contemporary Educational Psychology, 25*(1), 68–81.language: - 英语 许可协议: CC BY 4.0 任务类别: - 文本分类 任务子类别: - 多分类任务 标签: - 期望价值理论（Expectancy-Value Theory） - 心理学 - 心理测量学 - 教育心理学 - 动机科学 - 调查工具 - 内容分析 漂亮名称: EVT 评估集——人工编码心理测试题 规模类别: - 1000 < 样本量 < 10000 ## 数据集摘要 本数据集包含取自已发表且经同行评议的测评工具的人工编码心理测试题，这些工具用于测量**期望价值理论（Expectancy-Value Theory, EVT; Eccles et al., 1983; Eccles & Wigfield, 2002）**的相关构念。本数据集被构建为[EVT题项分类器](https://huggingface.co/christiqn/mpnet-EVT-classifier)的域外评估基准，且**未在模型训练阶段使用**——模型训练仅依赖合成生成的题项。因此，本数据集可用于检验从合成题项到真实人类编写的量表题项的泛化能力，是一项具备生态效度的验证测试。 | 属性 | 数值 | |---|---| | 原始总题数 | 1632 | | 最终评估集题数 | 1284 | | 评估集排除题数 | 348 | | 基准集使用题数 | 1284 | | EVT构念数量 | 6个（5个核心构念 + OTHER类别） | | 源工具数量 | 95个 | | 语言 | 英语 | | 研究领域 | 教育心理学、动机科学、心理测量学 | ## 源数据采集 ### 系统文献检索 为全面收集用于测量EVT相关构念的已发表题项，本研究在**Web of Science（科学网）**数据库中执行了系统文献检索，检索式如下： TS=("expectancy-value theory" OR "expectancy-value model" OR "situated expectancy-value" OR SEVT OR "expectanc* for success" OR "ability belief*" OR "subjective task value" OR "attainment value" OR "intrinsic value" OR "utility value") AND TS=(measure* OR instrument* OR scale* OR questionnaire* OR item* OR inventory OR assess* OR psychometric* OR validat*) 在初始的标题与摘要筛选阶段，仅纳入明确报告了至少一种EVT构念测评的研究。进一步的纳入标准还包括： - 英语语言出版物 - 已发表且经同行评议的论文 - 采用开放获取许可的论文 - 可通过Crossref、Unpaywall或OpenAlex API获取的论文 设置上述限制是为了支持全自动化的检索与提取流程。 ### 自动化题项提取 筛选完成后，自动获取论文的全文PDF文件。题项提取采用**光学字符识别（Optical Character Recognition, OCR）**技术，遍历每篇下载稿件的原始文本以识别报告的量表题项。对提取得到的字符串进行如下预处理： - 移除OCR伪影（字符替换错误、换行伪影） - 去除引用括号与文内参考文献标注 - 移除题项的数字前缀（如"1."、"Q3:"） 上述处理后得到一组干净的、未标注的文本字符串，作为候选量表题项。 ### 人工筛选 随后对候选题项进行人工审核与筛选，移除以下类型的题项： - 未明确设计为自陈式题项的内容（如指导语、锚定标签） - 无额外上下文则无法进行外部评估的题项 - 存在多个响应锚点导致无法明确分配构念的题项 通过人工筛选的题项将由人工专家编码员映射至对应的EVT构念，若题项不属于任何EVT构念类别，则标记为`OTHER`。 ## 数据集结构 ### 文件格式 分号分隔的CSV格式（带BOM的UTF-8编码），共1632行（含表头），包含4个字段。 ### 字段说明 | 字段 | 类型 | 描述 | |---|---|---| | `scale` | 字符串 | 源工具或问卷的名称。若原文献未标注量表名称，则设为`NOT_REPORTED`。 | | `item` | 字符串 | 经预处理的心理测试题项文本，已移除OCR伪影、引用括号与数字前缀。 | | `human_coder` | 字符串 | 人工专家分配的EVT构念标签，可选值为：`ATTAINMENT_VALUE`、`COST`、`EXPECTANCY`、`INTRINSIC_VALUE`、`UTILITY_VALUE`、`OTHER`。 | | `decision` | 字符串 / 空值 | 排除标记。非空值表示该题项被排除出主评估集的原因：`third-person`、`multiple anchors`或`other`。空值表示该题项已纳入评估集。 | ### 类别分布 | EVT构念 | 描述 | 原始样本量 | |---|---|---| | `EXPECTANCY` | 对未来成功的信念 | 308 | | `OTHER` | 无法归类为EVT构念 | 283 | | `INTRINSIC_VALUE` | 任务带来的愉悦感与兴趣 | 234 | | `UTILITY_VALUE` | 对未来目标的有用性 | 199 | | `COST` | 任务参与带来的感知负面后果 | 149 | | `ATTAINMENT_VALUE` | 表现优异的个人重要性 | 111 | | **总计** | | **1284** | > 原始文件中`human_coder`值为空的题项对应未分配EVT标签的行（如`NOT_REPORTED`量表条目或未标注的提取结果），此类题项未纳入评估集。 ## 排除标准与`decision`字段 通过人工筛选但被判定为存在方法学问题的题项会被保留在数据集中，其`decision`字段为非空值。此类题项被排除出主分类器评估集，但可用于二次分析。 | 排除原因 | 描述 | 样本量 | |---|---|---| | `third-person` | 采用第三人称表述而非直接针对被试（如"学生认为这很有趣"），导致自陈式意图模糊。 | 67 | | `multiple anchors` | 题项的响应量表包含多个不同的概念锚点，干扰构念分配。 | 29 | | `other` | OCR伪影、格式片段、指导语文本、缺失上下文或非独立量表题项。 | 252 | | **总排除题数** | | **348** | 主模型评估使用的题项为`decision`字段为空的样本（**N = 1284**）。 ## 工具覆盖范围 题项取自**95个不同的工具**，覆盖广泛的学术领域与研究人群。这种多样性是有意设计的：本数据集旨在检验EVT构念标签是否可在不同情境下泛化，而非仅局限于单一工具或领域。 ### 领域覆盖范围 - 数学与统计教育 - 科学、技术、工程与数学（STEM）领域 - 语言学习（ESL、阅读、写作） - 健康与临床领域（如母乳喂养动机、职业安全） - 音乐与艺术教育 - 高等教育与职业动机 - 通用学术动机（跨领域工具） ## 编码流程 ### 构念定义 题项编码依据Eccles等人（1983）与Eccles & Wigfield（2002）提出的理论定义： - **ATTAINMENT_VALUE** — 任务表现优异的个人重要性，与个体身份和自我概念相关。 - **COST** — 任务参与带来的感知负面后果，包括机会成本、精力投入与情绪成本。 - **EXPECTANCY** — 对任务未来成功的信念，包括能力自我概念与自信心。 - **INTRINSIC_VALUE** — 任务参与带来的愉悦感、兴趣或主观快感。 - **UTILITY_VALUE** — 任务对未来目标、职业或活动的感知有用性。 - **OTHER** — 不属于上述5种核心EVT构念的题项。 ### 编码规则 每个题项仅分配一个标签。若某题项可合理归属多个构念，编码员需选择最显著的构念对应的标签；若歧义无法解决，则将题项标记为`multiple anchors`。采用第三人称表述的题项将被标记为`third-person`，并排除出主评估集。 ## 预期用途 - 自动化EVT构念分类器的基准评估 - 标注研究与评分者信度研究 - 未来分类器的训练数据（需注意领域偏移的局限性） - EVT测评工具的系统综述与内容分析 - 跨工具的构念效度与题项覆盖范围的心理测量学研究 ## 局限性 - **仅支持英语**。所有题项均为英语，不应假设其可跨语言泛化。 - **合成题项与真实题项的分布差距**。关联的分类器仅在合成题项上训练，本数据集可检验真实场景下的泛化能力，但无法消除分布差距。 - **开放获取偏倚**。纳入标准限定为开放获取出版物，可能无法完全代表EVT测评工具的相关文献。 - **OCR噪声**。尽管经过预处理，部分题项仍可能残留提取流程中产生的轻微OCR伪影。 - **领域不平衡**。部分领域（如数学、STEM）的样本量高于其他领域（如健康、艺术），这反映了相关文献的分布而非刻意的抽样设计。 ## 参考文献 - Eccles, J. S., et al. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), *Achievement and achievement motives* (pp. 75–146). W. H. Freeman. - Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. *Annual Review of Psychology, 53*(1), 109–132. - Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. *Contemporary Educational Psychology, 25*(1), 68–81.