sibasmarakp/Qwen2.5-Math-7B-Instruct-Qwen2.5-14B-Instruct-uPRM-T80-adapters-best_of_n-completions

--- dataset_info: - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last features: - name: problem dtype: string - name: solution dtype: string - name: answer dtype: string - name: subject dtype: string - name: level dtype: int64 - name: unique_id dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 8754857 num_examples: 500 download_size: 8075049 dataset_size: 8754857 - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2226 dataset_size: 128 - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last features: - name: problem dtype: string - name: solution dtype: string - name: answer dtype: string - name: subject dtype: string - name: level dtype: int64 - name: unique_id dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 8796685 num_examples: 500 download_size: 8112560 dataset_size: 8796685 - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2226 dataset_size: 128 - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last features: - name: problem dtype: string - name: solution dtype: string - name: answer dtype: string - name: subject dtype: string - name: level dtype: int64 - name: unique_id dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 8711849 num_examples: 500 download_size: 8048581 dataset_size: 8711849 - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2221 dataset_size: 128 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last features: - name: id dtype: int64 - name: problem dtype: string - name: solution list: string - name: answer list: string - name: context dtype: 'null' - name: image_1 dtype: 'null' - name: image_2 dtype: 'null' - name: image_3 dtype: 'null' - name: image_4 dtype: 'null' - name: image_5 dtype: 'null' - name: image_6 dtype: 'null' - name: image_7 dtype: 'null' - name: image_8 dtype: 'null' - name: image_9 dtype: 'null' - name: modality dtype: string - name: difficulty dtype: string - name: is_multiple_answer dtype: bool - name: unit dtype: string - name: answer_type dtype: string - name: error dtype: string - name: question_type dtype: string - name: subfield dtype: string - name: subject dtype: string - name: language dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 18359810 num_examples: 674 download_size: 51999570 dataset_size: 18359810 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2231 dataset_size: 128 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last features: - name: id dtype: int64 - name: problem dtype: string - name: solution list: string - name: answer list: string - name: context dtype: 'null' - name: image_1 dtype: 'null' - name: image_2 dtype: 'null' - name: image_3 dtype: 'null' - name: image_4 dtype: 'null' - name: image_5 dtype: 'null' - name: image_6 dtype: 'null' - name: image_7 dtype: 'null' - name: image_8 dtype: 'null' - name: image_9 dtype: 'null' - name: modality dtype: string - name: difficulty dtype: string - name: is_multiple_answer dtype: bool - name: unit dtype: string - name: answer_type dtype: string - name: error dtype: string - name: question_type dtype: string - name: subfield dtype: string - name: subject dtype: string - name: language dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 18281795 num_examples: 674 download_size: 51769470 dataset_size: 18281795 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2231 dataset_size: 128 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last features: - name: id dtype: int64 - name: problem dtype: string - name: solution list: string - name: answer list: string - name: context dtype: 'null' - name: image_1 dtype: 'null' - name: image_2 dtype: 'null' - name: image_3 dtype: 'null' - name: image_4 dtype: 'null' - name: image_5 dtype: 'null' - name: image_6 dtype: 'null' - name: image_7 dtype: 'null' - name: image_8 dtype: 'null' - name: image_9 dtype: 'null' - name: modality dtype: string - name: difficulty dtype: string - name: is_multiple_answer dtype: bool - name: unit dtype: string - name: answer_type dtype: string - name: error dtype: string - name: question_type dtype: string - name: subfield dtype: string - name: subject dtype: string - name: language dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 18535487 num_examples: 674 download_size: 17505392 dataset_size: 18535487 - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2231 dataset_size: 128 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last features: - name: problem dtype: string - name: answer dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 5621410 num_examples: 272 download_size: 5298756 dataset_size: 5621410 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2229 dataset_size: 128 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last features: - name: problem dtype: string - name: answer dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 5605721 num_examples: 272 download_size: 5244180 dataset_size: 5605721 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2226 dataset_size: 128 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last features: - name: problem dtype: string - name: answer dtype: string - name: completions list: string - name: scores list: list: float64 - name: pred dtype: string - name: completion_tokens list: int64 - name: agg_scores list: float64 - name: pred_weighted@1 dtype: string - name: pred_maj@1 dtype: string - name: pred_naive@1 dtype: string - name: pred_weighted@2 dtype: string - name: pred_maj@2 dtype: string - name: pred_naive@2 dtype: string - name: pred_weighted@4 dtype: string - name: pred_maj@4 dtype: string - name: pred_naive@4 dtype: string - name: pred_weighted@8 dtype: string - name: pred_maj@8 dtype: string - name: pred_naive@8 dtype: string splits: - name: train num_bytes: 5550106 num_examples: 272 download_size: 5180893 dataset_size: 5550106 - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals features: - name: n dtype: int64 - name: acc_naive dtype: float64 - name: acc_weighted dtype: float64 - name: acc_maj dtype: float64 splits: - name: train num_bytes: 128 num_examples: 4 download_size: 2236 dataset_size: 128 configs: - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last/train-* - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals/train-* - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last/train-* - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals/train-* - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last/train-* - config_name: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals data_files: - split: train path: rebuttal-MATH500--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last/train-* - config_name: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals data_files: - split: train path: rebuttal-OlympiadBench--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-0--agg_strategy-last--evals/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-1--agg_strategy-last--evals/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last/train-* - config_name: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals data_files: - split: train path: rebuttal-minervamath--T-0.7--top_p-0.8--n-8--seed-2--agg_strategy-last--evals/train-* ---

搜集汇总

数据集介绍

构建方式

在数学推理领域，数据集的构建往往依赖于大规模语言模型生成多样化的解题路径。本数据集通过Qwen2.5系列模型，在MATH500、OlympiadBench和minervamath三个数学问题基准上，采用温度参数0.7和top-p采样0.8的配置，为每个问题生成8条独立推理链。每条推理链均经过uPRM适配器评分，形成多维度分数矩阵，最终通过加权、多数投票和朴素选择三种聚合策略，构建出包含原始问题、标准答案、生成补全及其评分的结构化数据。

特点

该数据集的核心特征在于其多层次评估框架和丰富的元数据标注。每个数学问题不仅包含标准答案和解题过程，还附带了学科分类、难度等级及唯一标识符，为细粒度分析奠定基础。生成的多条补全路径配有双层评分体系，既包括每条推理链的原始分数，也涵盖聚合后的综合得分。数据集特别提供了不同选择策略下的预测结果，如加权预测、多数投票预测和朴素预测，便于研究者对比不同决策机制在数学推理任务中的表现差异。

使用方法

该数据集主要服务于数学自动推理和语言模型评估研究。使用者可加载特定配置的数据文件，直接获取问题陈述、生成补全及对应评分。通过分析不同聚合策略下的预测准确率，能够深入探究模型在复杂数学问题上的鲁棒性和一致性。数据集中的评估结果文件提供了不同n值下的准确率指标，支持快速对比实验。研究者还可利用丰富的元数据，开展跨学科、跨难度或跨问题类型的子集分析，以揭示模型在不同数学领域的性能特征。

背景与挑战

背景概述

在数学推理领域，大型语言模型的涌现能力引发了广泛关注，但如何有效评估和提升其数学问题求解的准确性与鲁棒性，成为亟待解决的核心议题。Qwen2.5-Math-7B-Instruct-Qwen2.5-14B-Instruct-uPRM-T80-adapters-best_of_n-completions数据集应运而生，它由研究团队基于Qwen模型系列构建，旨在通过集成多个数学基准如MATH500、OlympiadBench和MinervaMath，系统探索模型在复杂数学问题上的表现。该数据集聚焦于利用最佳n采样策略与适配器微调技术，以优化模型输出的选择机制，从而推动数学自动推理向更高精度与泛化能力迈进。

当前挑战

该数据集致力于解决数学自动推理中模型输出选择与评估的挑战，具体包括如何从多个候选答案中可靠地识别最优解，以及如何设计鲁棒的聚合策略以提升最终预测的准确性。在构建过程中，挑战体现在数学问题的多样性与复杂性上，例如涵盖不同学科主题、难度级别及解答形式，这要求数据集在采样、评分与标注环节保持高度一致性。同时，处理大规模生成内容并确保评估指标的公平性与可解释性，也构成了技术实施上的显著障碍。

常用场景

经典使用场景

在数学推理与大型语言模型评估领域，该数据集通过整合MATH500、OlympiadBench和MinervaMath等权威数学问题库，构建了一个多源、多难度的数学推理基准。其经典使用场景在于评估和比较不同语言模型在复杂数学问题求解中的性能，特别是针对生成式模型的推理能力进行系统性测试。数据集提供了丰富的模型生成结果及其评分，使得研究者能够深入分析模型在代数、几何、数论等子领域的表现差异，为模型优化提供数据支撑。

实际应用

在实际应用层面，该数据集为智能教育系统和自动化解题工具的研发提供了关键数据资源。教育科技公司可基于数据集中的模型生成结果，构建能够提供多步骤解题指导和错误分析的数学辅导系统。同时，数据集中对不同聚合策略（如加权投票、多数投票）的评估结果，可直接应用于优化商业数学推理引擎的答案选择机制，提升其在竞赛数学、学术研究辅助等场景下的实用性和可靠性。

衍生相关工作

围绕该数据集衍生的经典工作主要集中在数学推理的奖励建模与集成方法领域。例如，基于数据集中的评分数据，研究者发展了改进的偏好奖励模型（PRM）训练策略，用于提升模型生成结果的质量筛选。同时，数据集启发了对多生成路径集成技术的研究，如加权投票和多数投票机制在数学推理中的有效性验证，这些工作进一步推动了如Process Reward Models（PRM）和Best-of-N采样等前沿方法在复杂推理任务中的应用与优化。