The Pedagogy Benchmark

该数据集由Fab Inc和AI-for-Education.org创建，旨在评估大型语言模型在跨领域教学知识（CDPK）和特殊教育需求与残疾（SEND）教学知识方面的能力。数据集包含来自智利教育部教师专业发展考试的920个多项选择题，涵盖教学策略和评估方法等教学子领域。这些基准旨在加速和改善大型语言模型在教育应用中的负责任开发，为更有效和基于证据的AI在教育中的应用铺平道路。This dataset was created by Fab Inc. and AI-for-Education.org, aiming to evaluate the capabilities of large language models (LLMs) in cross-disciplinary pedagogical knowledge (CDPK) and pedagogical knowledge for special educational needs and disabilities (SEND). It includes 920 multiple-choice questions sourced from the Chilean Ministry of Education's teacher professional development examinations, covering pedagogical subfields such as teaching strategies and assessment methods. These benchmarks are designed to accelerate and improve the responsible development of large language models for educational applications, paving the way for more effective and evidence-based AI applications in education.

搜集汇总

数据集介绍

构建方式

The Pedagogy Benchmark数据集构建基于智利教育部专业教师发展考试的精选题目，涵盖教学策略、评估方法等多个教学子领域。通过PDF提取、光学字符识别（OCR）技术处理非文本扫描文档，并采用GPT-4o-mini进行西班牙语到英语的翻译与结构化处理。经过去重、人工审核及专家标注（包括教育层级、学科类别和教学领域），最终形成包含1143道高质量多选题的基准测试集，其中920题用于跨领域教学知识（CDPK）评估，223题专用于特殊教育需求与残疾（SEND）教学知识评估。

特点

该数据集的核心特点在于其专注于评估大语言模型的教学知识而非学科内容知识，填补了现有基准测试的空白。题目源自真实教师资格考试，具有专业性和实践性；通过多轮人工校验确保翻译准确性、文化中立性及选项逻辑严谨性。此外，数据集提供细粒度元数据标注（如教育阶段、教学子领域），支持多维性能分析。SEND子集独立评估的特性进一步凸显了对特殊教育场景的针对性考量。

使用方法

使用该数据集时，需通过标准化提示模板进行少样本（few-shot）测试，要求模型直接输出选项字母以确保结果可解析。评估框架支持多种LLM接口（如OpenAI、Anthropic等），并采用bootstrap重采样计算95%置信区间。研究显示模型表现对答案位置敏感，建议通过答案轮换配置验证偏差。公开的在线排行榜（https://rebrand.ly/pedagogy）支持按模型属性筛选对比，同时提供开源代码便于复现。该基准适用于教育类AI工具开发中的模型选型及教学知识能力追踪。

背景与挑战

背景概述

The Pedagogy Benchmark, introduced in 2025 by researchers from Fab Inc and AI-for-Education.org, represents a significant advancement in evaluating large language models' (LLMs) pedagogical knowledge. This novel dataset addresses a critical gap in existing benchmarks by focusing on Cross-Domain Pedagogical Knowledge (CDPK) and Special Education Needs and Disability (SEND) pedagogical knowledge, rather than solely assessing content knowledge. The benchmark comprises 920 multiple-choice questions sourced from the Chilean Ministry of Education's professional development exams for teachers, covering various pedagogical subdomains such as teaching strategies and assessment methods. The creation of this benchmark responds to the growing need for specialized evaluations as LLMs are increasingly deployed in educational applications worldwide, particularly in addressing the global learning crisis where 90% of children in low-income countries struggle with basic literacy.

当前挑战

The Pedagogy Benchmark tackles two primary challenges in AI education evaluation. First, it addresses the domain-specific challenge of assessing pedagogical knowledge rather than content knowledge, requiring models to demonstrate understanding of teaching methodologies and educational theories. Second, the dataset construction faced significant challenges including: 1) extracting and translating questions from scanned Spanish documents while maintaining educational nuance, 2) rigorous de-duplication processes revealing over 30% duplicate questions across years, 3) expert manual annotation to ensure pedagogical relevance and quality, and 4) identifying and removing flawed questions that consistently produced incorrect responses across models. Additionally, the benchmark must contend with the rapid evolution of LLM capabilities, as evidenced by performance improvements from 50% to 82% accuracy at fixed cost points within 18 months, requiring ongoing updates to remain relevant.

常用场景

经典使用场景

在教育技术领域，The Pedagogy Benchmark数据集被广泛用于评估大型语言模型（LLMs）在跨领域教学知识（CDPK）和特殊教育需求与残疾（SEND）教学知识方面的表现。该数据集通过精心挑选的来自智利教育部教师专业发展考试的多选题，涵盖了教学策略、评估方法等多个教学子领域，为研究者和开发者提供了一个标准化的测试平台。

实际应用

在实际应用中，The Pedagogy Benchmark为教育科技产品的开发提供了重要参考。例如，在开发教师助手、个性化学习系统和学生辅导工具时，开发者可以利用该数据集选择具有优秀教学知识理解能力的模型，从而提升产品的教育效果和用户体验。此外，该数据集还为政策制定者提供了评估AI教育工具教学能力的科学依据。

衍生相关工作

围绕The Pedagogy Benchmark，已经衍生出一系列相关研究工作。例如，有研究探索如何将该数据集与MMLU等传统知识基准相结合，全面评估模型的教学能力和学科知识；另有工作专注于开发基于该数据集的细粒度评估方法，以区分模型在不同教学子领域（如课堂管理、差异化教学）的表现差异。这些工作共同推动了教育AI评估体系的发展。