遇见数据集

CoffeeGitta/difficulty-aime_1983-2024-generations

收藏
Hugging Face2026-05-25 更新2026-05-31 收录
官方服务:

资源简介:

--- dataset_info: - config_name: Qwen--Qwen2.5-Math-1.5B-Instruct features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 10300676 num_examples: 747 - name: validation num_bytes: 2560424 num_examples: 186 - name: test num_bytes: 414719 num_examples: 30 download_size: 4396726 dataset_size: 13275819 - config_name: Qwen--Qwen2.5-Math-7B-Instruct features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 16988682 num_examples: 747 - name: validation num_bytes: 4267785 num_examples: 186 - name: test num_bytes: 685250 num_examples: 30 download_size: 12363943 dataset_size: 21941717 - config_name: Qwen--Qwen3-8B features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 139696847 num_examples: 747 - name: validation num_bytes: 34853241 num_examples: 186 - name: test num_bytes: 7636629 num_examples: 30 download_size: 69122087 dataset_size: 182186717 - config_name: deepseek-ai--DeepSeek-R1-Distill-Qwen-7B features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 103945759 num_examples: 747 - name: validation num_bytes: 25254903 num_examples: 186 - name: test num_bytes: 5949109 num_examples: 30 download_size: 51923554 dataset_size: 135149771 - config_name: openai--gpt-oss-20b_high features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 245770860 num_examples: 598 - name: validation num_bytes: 63258808 num_examples: 149 - name: test num_bytes: 17802996 num_examples: 30 download_size: 156628984 dataset_size: 326832664 - config_name: openai--gpt-oss-20b_low features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 16580971 num_examples: 747 - name: validation num_bytes: 4380677 num_examples: 186 - name: test num_bytes: 740470 num_examples: 30 download_size: 10987909 dataset_size: 21702118 - config_name: openai--gpt-oss-20b_medium features: - name: formatted_prompt dtype: string - name: generated_solutions list: - name: input_cost_usd_once dtype: float64 - name: input_tokens dtype: int64 - name: output_cost_usd dtype: float64 - name: output_tokens dtype: int64 - name: rollout_cost_usd dtype: float64 - name: score dtype: float64 - name: text dtype: string - name: ground_truth dtype: string - name: input_cost_usd_once dtype: float64 - name: k dtype: int64 - name: majority_vote_extracted_answer dtype: string - name: majority_vote_is_correct dtype: int8 - name: max_len dtype: int64 - name: model_name dtype: string - name: pass_at_k dtype: float64 - name: problem dtype: string - name: problem_id dtype: string - name: rating dtype: float64 - name: split dtype: string - name: success_rate dtype: float64 - name: task dtype: string - name: temperature dtype: float64 - name: total_cost_usd dtype: float64 - name: total_input_tokens dtype: int64 - name: total_output_cost_usd dtype: float64 - name: total_output_tokens dtype: int64 splits: - name: train num_bytes: 85607637 num_examples: 747 - name: validation num_bytes: 22787178 num_examples: 186 - name: test num_bytes: 4956684 num_examples: 30 download_size: 56277198 dataset_size: 113351499 configs: - config_name: Qwen--Qwen2.5-Math-1.5B-Instruct data_files: - split: train path: Qwen--Qwen2.5-Math-1.5B-Instruct/train-* - split: validation path: Qwen--Qwen2.5-Math-1.5B-Instruct/validation-* - split: test path: Qwen--Qwen2.5-Math-1.5B-Instruct/test-* - config_name: Qwen--Qwen2.5-Math-7B-Instruct data_files: - split: train path: Qwen--Qwen2.5-Math-7B-Instruct/train-* - split: validation path: Qwen--Qwen2.5-Math-7B-Instruct/validation-* - split: test path: Qwen--Qwen2.5-Math-7B-Instruct/test-* - config_name: Qwen--Qwen3-8B data_files: - split: train path: Qwen--Qwen3-8B/train-* - split: validation path: Qwen--Qwen3-8B/validation-* - split: test path: Qwen--Qwen3-8B/test-* - config_name: deepseek-ai--DeepSeek-R1-Distill-Qwen-7B data_files: - split: train path: deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/train-* - split: validation path: deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/validation-* - split: test path: deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/test-* - config_name: openai--gpt-oss-20b_high data_files: - split: train path: openai--gpt-oss-20b_high/train-* - split: validation path: openai--gpt-oss-20b_high/validation-* - split: test path: openai--gpt-oss-20b_high/test-* - config_name: openai--gpt-oss-20b_low data_files: - split: train path: openai--gpt-oss-20b_low/train-* - split: validation path: openai--gpt-oss-20b_low/validation-* - split: test path: openai--gpt-oss-20b_low/test-* - config_name: openai--gpt-oss-20b_medium data_files: - split: train path: openai--gpt-oss-20b_medium/train-* - split: validation path: openai--gpt-oss-20b_medium/validation-* - split: test path: openai--gpt-oss-20b_medium/test-* --- # Generations Dataset: aime_1983-2024 LLM-generated solutions across train/validation/test splits for multiple models. ## Columns | Column | Type | Description | |--------|------|-------------| | `problem` | str | Problem statement | | `generated_solutions` | list | Generated solutions with scores | | `success_rate` | float | Fraction of correct generations | | `majority_vote_is_correct` | int (0/1) | Whether majority vote is correct | | `k` | int | Number of samples generated | | `temperature` | float | Sampling temperature | | `max_len` | int | Maximum generation length | | `model_name` | str | Model used for generation | ## Usage ```python from datasets import load_dataset dataset = load_dataset("CoffeeGitta/difficulty-aime_1983-2024-generations", name="<org--model>") train = dataset["train"] ``` ## Citation ```bibtex @article{lugoloobi_llms_2026, title = {LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations}, url = {http://arxiv.org/abs/2602.09924}, author = {Lugoloobi, William and Foster, Thomas and Bankes, William and Russell, Chris}, year = {2026}, } ```

Generations Dataset: aime_1983-2024. LLM-generated solutions across train/validation/test splits for multiple models. It contains solutions generated by various models (e.g., Qwen2.5-Math, DeepSeek-R1, GPT-OSS-20B) for AIME (American Invitational Mathematics Examination, 1983-2024) problems, with features such as problem statements, generated solutions (with scores and cost information), success rates, majority vote correctness, etc. The dataset is designed for research on predicting LLM failures from pre-generation activations. Total dataset size is approximately 1.8GB, with 747 training examples, 186 validation examples, and 30 test examples (varies slightly across models).

提供机构:
CoffeeGitta
搜集汇总
数据集介绍
CoffeeGitta/difficulty-aime_1983-2024-generations 数据集图片
构建方式
该数据集立足于数学推理评测的长期需求,围绕AIME竞赛试题构建了多模型生成解答的综合性资源。其构建以1983至2024年间的AIME题目为基础,针对每一道问题,调用多个具备代表性的数学推理模型,包括Qwen系列、DeepSeek蒸馏模型以及GPT-OSS系列等,在设定采样温度与最大生成长度的条件下,为每题生成多条候选解答。生成的解答经由评分机制判定对错,并据此计算成功率、多数投票正确性以及pass@k等指标,最终按照训练、验证与测试的划分方式整理为标准化数据集,形成从问题到生成、从评分到统计的完整闭环。
特点
该数据集在结构与内容上呈现出多维度特征。一方面,它以模型为划分维度,为每个模型配置独立的数据子集,便于研究者进行跨模型的横向比较与纵向分析。另一方面,数据集不仅保留原始问题与标准答案,还细致记录了生成文本、单次推理成本、输入输出token数量以及累计开销等经济性指标,使计算效率与推理质量得以在同一框架下审视。此外,评分、成功率、多数投票与pass@k等衍生指标的纳入,为刻画模型在不同难度题目上的表现提供了丰富视角,并使数据集兼具评测基准与行为分析的双重功能。
使用方法
该数据集可借助HuggingFace的datasets库便捷加载,用户通过指定数据集名称与对应的模型配置名,即可获取特定模型的训练、验证或测试划分。使用时,研究者可依据任务需求选择相应模型子集,进而访问其问题文本、生成解答、评分结果与统计指标,用于模型能力评估、生成质量分析或推理成本核算。由于数据已按照标准划分组织,用户可直接用于训练、验证与测试流程,亦可在此基础上开展多数投票、pass@k计算或错误模式挖掘等二次分析,为数学推理模型的比较研究与改进提供数据支撑。
背景与挑战
背景概述
数学推理作为人工智能的核心能力试金石,长期依赖高难度竞赛题库进行模型评估。AIME自1983年始,以其严谨性与挑战性成为衡量大型语言模型数学推理水平的重要标尺。该数据集由William Lugoloobi等研究者构建,系统汇集了1983至2024年间AIME竞赛题目,并针对Qwen2.5-Math、DeepSeek-R1-Distill、GPT-OSS-20B等多款模型生成了大规模解答样本,涵盖训练、验证与测试三个划分。其核心研究问题在于揭示模型预生成激活状态与最终成功率之间的内在关联,为推理失败预测提供数据基础。该工作对理解LLM推理边界、推动可解释性研究具有显著影响力。
当前挑战
数据集所应对的领域问题在于高难度数学推理的精确评估与失败预测,传统方法仅依赖最终答案正确性,难以捕捉推理过程的细微偏差。构建过程中面临多重挑战:其一,AIME题目跨逾四十年,表述风格与知识覆盖差异显著,统一格式化与难度分级需精细校准;其二,多模型生成需控制温度、最大长度等采样参数,确保样本可比性并兼顾多样性;其三,为每个生成解答计算分数、多数投票正确性及pass@k等指标,涉及昂贵的计算成本与复杂的聚合逻辑;其四,预生成激活与成功率的关联分析要求对模型内部状态进行精确记录与对齐,数据工程的复杂度极高。
常用场景
经典使用场景
在大型语言模型数学推理能力评测领域,该数据集构筑了一个跨越四十年命题历史的AIME基准,涵盖1983年至2024年间的竞赛题目,并为每个问题提供多模型、多温度、多采样次数下的生成解及其正确性判定。研究者藉由pass@k与多数投票正确率等指标,得以系统刻画模型在不同难度层级上的推理边界,尤其适用于考察推理链长度、采样预算与解题成功率之间的权衡关系。
解决学术问题
该数据集直面的核心学术问题是:模型生成前的内部表征能否预示其解题成败。它通过为每个问题提供细粒度评级与大量生成解评分,使研究者得以分离题目难度、采样温度、模型规模等因素对正确率的独立影响,进而检验预生成激活预测成功率的假设。其意义在于将评估从单纯的结果统计推进至对模型失败机制的诊断层面。
衍生相关工作
基于该数据集,Lugoloobi等人在2026年发表的论文《LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations》中提出利用预生成激活预测模型成功与否的方法,构成该数据集最直接的衍生工作。此外,数据集按模型组织的多配置结构,也便于后续研究开展跨模型、跨采样参数的比较分析,推动数学推理评测范式的细化。
以上内容由遇见数据集搜集并总结生成
二维码
社区交流群
二维码
科研交流群
商业服务