fineweb-oracle-convqa-chunked
收藏资源简介：
FineWeb Oracle: Chunked ConvQA是一个用于评估语言模型在信息缺口（oracle）场景下推理能力的对话式问答数据集。其核心任务是：给定一个长文档，模型需识别自然转折点将文档分割为前缀和后缀，然后生成一个关于后缀内容的问题（该问题无法仅从前缀推导出）。接着，基线模型（仅看到前缀）尝试回答问题，其答案将与真实答案（仅看到后缀生成的答案）进行比较和评分。数据集通过三阶段自动化流程生成，全程使用Anthropic Haiku 4.5模型：第一阶段模型阅读完整文档并生成问题；第二阶段模型基于前缀生成基线答案；第三阶段模型基于后缀生成真实答案，最后由同一模型进行评分。数据集包含29,029个样本，划分为训练集（26,123条）和测试集（2,906条），原始文档来源于FineWeb数据集并经过长度处理。每个样本包含完整文档文本、生成的问题、基线答案、真实答案、基线答案正确性判断（bb_correct）和细粒度匹配分数（bb_score），以及前缀、后缀、分割点索引、文档ID等信息。后续更新增加了target_response_distractor（与真实答案相似但事实错误的干扰项）和question_category（问题分类为“元认知”或“程序性”）。该数据集适用于语言模型推理评估、对抗性干扰项研究和问答生成任务。
FineWeb Oracle: Chunked ConvQA is a conversational question-answering dataset designed to evaluate the reasoning capabilities of language models in information gap (oracle) scenarios. The core task involves: given a long document, the model must first identify a natural turning point to split the document into a prefix and a suffix, then generate a question about the suffix content (which cannot be deduced from the prefix alone). Next, a baseline model (seeing only the prefix) attempts to answer the question, and its answer is compared and scored against the ground-truth answer (generated from the suffix). The dataset is generated through a three-stage automated pipeline using the Anthropic Haiku 4.5 model throughout: Stage 1 (Round 1) involves reading the full document, selecting a split point, and generating the question; Stage 2 (Round 2) generates the baseline answer based solely on the prefix; Stage 3 (Round 3) generates the ground-truth answer based solely on the suffix. Finally, the same model judges and scores the match between the baseline and ground-truth answers. It contains 29,029 samples, split into a training set (26,123 entries) and a test set (2,906 entries). The original document text is sourced from the FineWeb dataset and processed to match specific length distributions. Each sample includes the full document text, generated question, baseline answer, ground-truth answer, model-judged baseline answer correctness (binary judgment `bb_correct`), fine-grained matching score (`bb_score` on a 0.0 to 1.0 scale), as well as prefix, suffix, split point index, document ID, and other information. Subsequent updates added two key fields: `target_response_distractor` (a distractor similar to the ground-truth answer in length, structure, and vocabulary but with key factual errors) and `question_category` (classifying generated questions as metacognitive or procedural). The dataset is suitable for language model reasoning evaluation, adversarial distractor research, and question-answering generation tasks.
