amd/SAND-MATH

--- language: - en license: other license_name: license license_link: LICENSE task_categories: - question-answering tags: - mathematics - synthetic-data - question-answering - reasoning - llm pretty_name: SAND-MATH library_name: - datasets configs: - config_name: train_500 data_files: sand_math_500.json - config_name: train_1k data_files: sand_math_base_1000.json - config_name: train_dh_1k data_files: sand_math_dh_1000.json - config_name: train_gptoss_14k data_files: sand_math_gptoss_14k.json --- # SAND-Math: A Synthetic Dataset of Difficult Problems to Elevate LLM Math Performance 📃 [Paper](https://arxiv.org/abs/2507.20527) | 🤗 [Dataset](https://huggingface.co/datasets/amd/SAND-MATH) **SAND-Math** (Synthetic Augmented Novel and Difficult Mathematics) is a high-quality, high-difficulty dataset of mathematics problems and solutions. It is generated using a novel pipeline that addresses the critical bottleneck of scarce, high-difficulty training data for mathematical Large Language Models (LLMs). ## Key Features * **Novel Problem Generation**: Problems are generated from scratch using minimally constrained prompts, leveraging the latent metacognitive abilities of SOTA LLMs. * **Systematic Difficulty Hiking**: A unique methodology to increase problem complexity by synthesizing new constraints, advanced theorems, and cross-domain concepts. * **Rigorous Quality Control**: A multi-stage filtering pipeline ensures solution correctness (via self-consistency), internal diversity, decontamination against benchmark data and novelty check against web data. * **State-of-the-Art Performance**: Augmenting strong baselines with SAND-Math yields top-tier results on challenging benchmarks like AIME, AMC, and MATH. ## The SAND-Math Pipeline The dataset is created through a comprehensive end-to-end pipeline designed to maximize quality, novelty, and difficulty. <div align="center"> <img src="SAND-Math-pipeline.png" style="object-fit: contain;"/> <em><b>Figure 1:</b> The SAND-Math data generation and filtering pipeline.</em> </div> The core innovation is the **Difficulty Hiking** step, where an existing problem is transformed into a more complex one. The model is prompted with the original problem, its difficulty rating, a relevant advanced theorem, and a cross-domain concept, and instructed to create a new, harder problem that integrates these elements. ## Dataset Structure ### Data Splits This repository contains three data splits used for the experiments in the paper: 1. `train_500`: A 500-sample subset used for standalone finetuning comparisons in Table 1 2. `train_1k`: A 1000-sample subset of the base SAND-Math data. 3. `train_dh_1k`: A 1000-sample subset having difficulty hiked data, demonstrating the performance boost from the hiking process. 4. `train_gptoss_14k`: A 14k-sample subset having questions and reasoning traces generated using GPT-OSS120B as a teacher model. ### Data Fields Each instance in the dataset consists of the following fields: * `id`: problem id * `problem`: The text of the mathematics problem. * `solution`: A detailed, step-by-step solution to the problem generated using DeepSeek-R1, concluding with the final answer in `\boxed{}`. * `difficulty_rating`: A fine-grained difficulty score from 1-10 assigned by a Llama-3.3-70B-Instruct judge. * `branch`: The primary mathematical branch the question belongs to (e.g., Number Theory, Algebra). * `version`: Indicate whether a question is the normal version or the difficulty-hiked version. **stage1**: the original question, **stage2**: difficulty hiked question. ## How to Use The dataset can be easily loaded using the `datasets` library. ```python # Install the library !pip install datasets from datasets import load_dataset # Load a specific split dataset = load_dataset("amd/SAND-MATH", name="train_1k") # Access an example print(dataset['train'][0]) ``` To view the actual question and it's difficulty hiked version together: ```python # Install the datasets library (if you haven’t already) !pip install datasets from datasets import load_dataset # 1. Load both splits train1k = load_dataset("amd/SAND-MATH", name="train_1k") train_dh_1k = load_dataset("amd/SAND-MATH", name="train_dh_1k") # 2. Filter for version == 'stage2' in the difficulty-hiked split stage2_dh = train_dh_1k.filter(lambda rec: rec["version"] == "stage2") # 3. Build a lookup dict for the original problems by id orig_by_id = { rec["id"]: rec for rec in train1k } # 4. Pair up records and collect the desired fields matched_records = [] for rec in stage2_dh: orig = orig_by_id.get(rec["id"]) if orig: matched_records.append({ "id": rec["id"], "problem": orig["problem"], "difficulty_rating": orig["difficulty_rating"], "new_problem": rec["problem"], "new_difficulty_rating": rec["difficulty_rating"], }) # 5. (Optional) Inspect the first few matches for item in matched_records[:5]: print(item) ``` ## Implementation Detils In our pipeline, the core generative tasks were handled by **DeepSeek-R1**, which performed initial question generation, subsequent solution generation, and question modification during the Difficulty Hiking stage. The entire pipeline was executed on a single node equipped with 8x AMD Instinct^TM MI300X GPUs. To enhance throughput, the data generation stages of the pipeline utilized the steps mentioned in the Supercharge DeepSeek-R1 ROCm^TM [blog post](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html). ## Training Detils All models reported in our work are full-parameter finetuned using the [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) framework. Training was conducted on a single node equipped with 8x AMD Instinct^TM MI300X GPUs. Below are the key hyperparameters and a brief guide to the training setup on AMD GPUs. ### Hyperparameters The core hyperparameters used for our full-parameter finetuning experiments are detailed in Table 1. | **Hyperparameter** | **Value** | |---|---| | Learning Rate | 5.0e-6 | | LR Scheduler Type | cosine | | Warmup Ratio | 0.0 | | Number of Training Epochs | 10 | | Gradient Accumulation Steps | 1 | | Cutoff Length | 32,768 | | Flash Attention Implementation &nbsp; | fa2 | | DeepSpeed Strategy | ZeRO-3 | *Table 1: Key training hyperparameters used in our experiments.* ### Training Setup The following steps outline the process for replicating our training environment. 1. **Launch the Docker Container:** For better performance the training environment is buit using the ROCm^TM PyTorch [training container](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html). ```bash docker run -it --ipc=host --cap-add=SYS_PTRACE --network=host \ --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined \ --group-add video --privileged -w /workspace rocm/pytorch-training:v25.6 ``` 2. **Install LLaMA-Factory:** Clone the repository and install the required dependencies. A specific version of DeepSpeed was used for compatibility. ```bash git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e ".[torch,metrics]" --no-build-isolation pip install deepspeed==0.16.9 ``` 3. **Prepare Training Files:** - Download the required SAND-Math training splits from HuggingFace and format them into the Alpaca-style JSON format, as described in the [official documentation](https://github.com/hiyouga/LLaMA-Factory/tree/main/data). - Add a corresponding entry for the new dataset file in `data/dataset_info.json`. - Create a training configuration YAML file. Our configuration is adapted from the [LIMO example](https://github.com/GAIR-NLP/LIMO/blob/main/train/examples/train_limo.yaml). For experiments that augment the LIMO dataset, the `dataset` field in the configuration should be set to the names of datasets matching with `data/dataset_info.json` for example `limo,sand_math` assuming the [LIMO data](https://github.com/GAIR-NLP/LIMO/blob/main/train/data/limo.json) is also available. 4. **Launch Training:** Execute the training run using the prepared configuration file. ```bash llamafactory-cli train examples/train_full/train_sand_math.yaml ``` ## Evaluation SAND-Math demonstrates superior performance both as a standalone dataset and as an augmentation resource. ### Difficulty Distribution SAND-Math is centered at a much higher difficulty rating (mean ≈ 6) and covers a wider complexity range than other popular synthetic math datasets. <div align="center"> <img src="diffratings_violinplot_new.png" style="object-fit: contain;"/> <em><b>Figure 2:</b> Difficulty distribution of SAND-Math compared to other datasets.</em> </div> ### Finetuning Performance Augmenting the LIMO baseline with 500 samples from SAND-Math yields a top score of **73.32**, surpassing augmentation with the high-quality OpenR1 dataset and significantly outperforming other synthetic alternatives. **_Augmentation Performance (vs. Real-World Problems Datasets):_** | Training Data Configuration | Data Sample Size | AIME25 | AIME24 | AMC | MATH-500 | **Average** | | --------------------------- | ---------------- | ------ | ------ | ------ | -------- | ----------- | | **LIMO + SAND-Math (ours)** | **817+500** | **48.89** | **57.92** | **92.50** | **94.00** | **73.32** | | LIMO + openr1_math | 817+500 | 47.71 | 56.04 | 92.50 | 93.80 | 72.51 | | LIMO + llama_nemotron | 817+500 | 26.04 | 36.46 | 71.56 | 86.20 | 55.06 | **_Augmentation Performance (vs. Synthetic Datasets):_** | Training Data Configuration | Data Sample Size | AIME25 | AIME24 | AMC | MATH-500 | **Average** | | --------------------------- | ---------------- | ------ | ------ | ------ | -------- | ----------- | | **LIMO + SAND-Math (ours)** | **817+500** | **48.89** | **57.92** | **92.50** | **94.00** | **73.32** | | LIMO + MetamathQA | 817+500 | 31.04 | 46.25 | 47.24 | 56.40 | 45.23 | | LIMO + OpenmathInstruct | 817+500 | 18.13 | 38.96 | 64.53 | 72.40 | 48.50 | *Table 1: Augmentation performance comparison on Qwen-2.5-32B.* ### Impact of Difficulty Hiking The Difficulty Hiking process is a key driver of performance. It effectively shifts the data distribution towards more challenging problems, resulting in a significant boost in downstream model performance. <div align="center"> <img src="diffratings_diffhike_barplot.png" style="object-fit: contain;"/> <em><b>Figure 3:</b> Difficulty Hiking shifts questions from the mid-difficulty range (Base) to the more challenging 7.0-8.0 range (DH).</em> </div> | Dataset | Data Size | AIME25 | AIME24 | AMC24 | MATH500 | **Average** | | -------------------------- | ---------- | ------ | ------ | ------ | ------- | ----------- | | LIMO only (Baseline) | 817 | 44.50 | 56.30 | 91.41 | 93.80 | 71.50 | | LIMO + SAND-Math (Base) | 817 + 1500 | 46.38 | 59.09 | 92.71 | 93.60 | 72.94 | | **LIMO + SAND-Math (DH)** | **817 + 1500** | **49.23** | **60.55** | **93.17** | **94.60** | **74.39** | *Table 2: Ablation study showing the performance gain from Difficulty Hiked (DH) data.* # License The amd/SAND-MATH dataset is licensed for academic and research purposes under a ResearchRAIL license. For more information on license and terms, refer to our LICENSE file. # Citation ``` @misc{manem2025sandmathusingllmsgenerate, title={SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers}, author={Chaitanya Manem and Pratik Prabhanjan Brahma and Prakamya Mishra and Zicheng Liu and Emad Barsoum}, year={2025}, eprint={2507.20527}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.20527}, } ```--- 语言： - en 许可协议：其他 许可协议名称：license 许可协议链接：LICENSE 任务类别： - 问答 标签： - 数学 - 合成数据 - 问答 - 推理 - 大语言模型（LLM） 展示名称：SAND-MATH 库名称： - datasets 配置项： - 配置名称：train_500 数据文件：sand_math_500.json - 配置名称：train_1k 数据文件：sand_math_base_1000.json - 配置名称：train_dh_1k 数据文件：sand_math_dh_1000.json - 配置名称：train_gptoss_14k 数据文件：sand_math_gptoss_14k.json --- # SAND-Math：用于提升大语言模型数学性能的合成难题数据集 📃 [论文](https://arxiv.org/abs/2507.20527) | 🤗 [数据集](https://huggingface.co/datasets/amd/SAND-MATH) **SAND-Math**（Synthetic Augmented Novel and Difficult Mathematics，合成增强型新颖难题数学数据集）是一个高质量、高难度的数学问题与解答数据集。其通过全新的流水线生成，解决了数学大语言模型（LLM）面临的核心瓶颈：稀缺的高难度训练数据。 ## 核心特性 * **新颖的问题生成**：采用最小约束提示从头生成问题，利用当前顶尖大语言模型的潜在元认知能力。 * **系统性难度爬坡**：独特的方法通过合成新约束、高级定理和跨领域概念来提升问题复杂度。 * **严格的质量控制**：多阶段过滤流水线确保解答正确性（通过自一致性）、内部多样性、基准数据去污染以及网络数据新颖性检查。 * **顶尖的性能表现**：使用SAND-Math增强强基线模型后，在AIME、AMC和MATH等挑战性基准测试上取得顶级结果。 ## SAND-Math流水线 该数据集通过全面的端到端流水线创建，旨在最大化数据质量、新颖性与难度。 <div align="center"> <img src="SAND-Math-pipeline.png" style="object-fit: contain;"/> <em><b>图1：</b>SAND-Math数据生成与过滤流水线。</em> </div> 核心创新是**难度爬坡**步骤，即将现有问题转换为更复杂的问题。模型会收到原始问题、其难度评级、相关高级定理以及跨领域概念的提示，并被要求创建一个整合这些元素的全新更难问题。 ## 数据集结构 ### 数据划分 本仓库包含论文实验中使用的四类数据划分： 1. `train_500`：用于表1中独立微调对比的500样本子集 2. `train_1k`：基础SAND-Math数据的1000样本子集。 3. `train_dh_1k`：包含难度爬坡数据的1000样本子集，用于展示爬坡过程带来的性能提升。 4. `train_gptoss_14k`：包含14000个样本的子集，其问题与推理轨迹由教师模型GPT-OSS120B生成。 ### 数据字段 数据集中的每个实例包含以下字段： * `id`：问题编号 * `problem`：数学问题的文本内容。 * `solution`：使用DeepSeek-R1生成的详细分步解答，最终答案以`oxed{}`格式给出。 * `difficulty_rating`：由Llama-3.3-70B-Instruct评判模型分配的1-10级细粒度难度分数。 * `branch`：问题所属的主要数学分支（例如数论、代数）。 * `version`：标识问题是普通版本还是难度爬坡版本。**stage1**：原始问题，**stage2**：经过难度爬坡的问题。 ## 使用方法 该数据集可通过`datasets`库轻松加载。 python # 安装依赖库 !pip install datasets from datasets import load_dataset # 加载指定划分 dataset = load_dataset("amd/SAND-MATH", name="train_1k") # 查看示例数据 print(dataset["train"][0]) 若要同时查看原始问题及其难度爬坡版本： python # （若未安装则先）安装datasets库 !pip install datasets from datasets import load_dataset # 1. 加载两个划分数据集 train1k = load_dataset("amd/SAND-MATH", name="train_1k") train_dh_1k = load_dataset("amd/SAND-MATH", name="train_dh_1k") # 2. 筛选出难度爬坡划分中version == 'stage2'的样本 stage2_dh = train_dh_1k.filter(lambda rec: rec["version"] == "stage2") # 3. 构建按id索引的原始问题查找字典 orig_by_id = { rec["id"]: rec for rec in train1k } # 4. 匹配样本并收集所需字段 matched_records = [] for rec in stage2_dh: orig = orig_by_id.get(rec["id"]) if orig: matched_records.append({ "id": rec["id"], "problem": orig["problem"], "difficulty_rating": orig["difficulty_rating"], "new_problem": rec["problem"], "new_difficulty_rating": rec["difficulty_rating"], }) # 5. （可选）查看前5条匹配结果 for item in matched_records[:5]: print(item) ## 实现细节 在我们的流水线中，核心生成任务由**DeepSeek-R1**完成，包括初始问题生成、后续解答生成以及难度爬坡阶段的问题修改。整个流水线在搭载8块AMD Instinct^TM MI300X GPU的单节点上运行。为提升吞吐量，流水线的数据生成阶段采用了Supercharge DeepSeek-R1 ROCm^TM [博客文章](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)中提及的优化步骤。 ## 训练细节 本工作中提及的所有模型均使用[LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)框架进行全参数微调。训练在搭载8块AMD Instinct^TM MI300X GPU的单节点上完成。以下是关键超参数以及在AMD GPU上的训练设置简要指南。 ### 超参数 我们的全参数微调实验所用核心超参数详见表1。 | **超参数** | **取值** | |---|---| | 学习率 | 5.0e-6 | | 学习率调度器类型 | 余弦退火 | | 预热比例 | 0.0 | | 训练轮数 | 10 | | 梯度累积步数 | 1 | | 上下文最大长度 | 32,768 | | Flash Attention实现 | fa2 | | DeepSpeed策略 | ZeRO-3 | *表1：本实验所用关键训练超参数。* ### 训练设置 以下步骤可复现我们的训练环境。 1. **启动Docker容器**： 为获得更佳性能，训练环境基于ROCm^TM PyTorch [训练容器](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/training/benchmark-docker/pytorch-training.html)构建。 bash docker run -it --ipc=host --cap-add=SYS_PTRACE --network=host --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --group-add video --privileged -w /workspace rocm/pytorch-training:v25.6 2. **安装LLaMA-Factory**： 克隆仓库并安装所需依赖。为保证兼容性，需安装特定版本的DeepSpeed。 bash git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e ".[torch,metrics]" --no-build-isolation pip install deepspeed==0.16.9 3. **准备训练文件**： - 从HuggingFace下载所需的SAND-Math训练划分，并按照[官方文档](https://github.com/hiyouga/LLaMA-Factory/tree/main/data)所述格式转换为Alpaca风格的JSON文件。 - 在`data/dataset_info.json`中为新数据集文件添加对应的条目。 - 创建训练配置YAML文件。我们的配置改编自[LIMO示例](https://github.com/GAIR-NLP/LIMO/blob/main/train/examples/train_limo.yaml)。若需在实验中增强LIMO数据集，配置中的`dataset`字段应设置为`data/dataset_info.json`中匹配的数据集名称，例如`limo,sand_math`（前提是[LIMO数据](https://github.com/GAIR-NLP/LIMO/blob/main/train/data/limo.json)已准备就绪）。 4. **启动训练**： 使用准备好的配置文件执行训练。 bash llamafactory-cli train examples/train_full/train_sand_math.yaml ## 评估结果 SAND-Math无论是作为独立数据集还是增强资源，均展现出优异的性能。 ### 难度分布 与其他主流合成数学数据集相比，SAND-Math的难度分布集中在更高的分数区间（均值≈6），且覆盖的复杂度范围更广。 <div align="center"> <img src="diffratings_violinplot_new.png" style="object-fit: contain;"/> <em><b>图2：</b>SAND-Math与其他数据集的难度分布对比。</em> </div> ### 微调性能 使用500个SAND-Math样本增强LIMO基线模型后，可获得73.32的最高分，超过了使用高质量OpenR1数据集进行增强的效果，且显著优于其他合成数据集的增强表现。 **_增强性能对比（与真实问题数据集）：_** | 训练数据配置 | 数据样本量 | AIME25 | AIME24 | AMC | MATH-500 | **平均得分** | | --------------------------- | ---------------- | ------ | ------ | ------ | -------- | ----------- | | **LIMO + SAND-Math（ours）** | **817+500** | **48.89** | **57.92** | **92.50** | **94.00** | **73.32** | | LIMO + openr1_math | 817+500 | 47.71 | 56.04 | 92.50 | 93.80 | 72.51 | | LIMO + llama_nemotron | 817+500 | 26.04 | 36.46 | 71.56 | 86.20 | 55.06 | **_增强性能对比（与合成数据集）：_** | 训练数据配置 | 数据样本量 | AIME25 | AIME24 | AMC | MATH-500 | **平均得分** | | --------------------------- | ---------------- | ------ | ------ | ------ | -------- | ----------- | | **LIMO + SAND-Math（ours）** | **817+500** | **48.89** | **57.92** | **92.50** | **94.00** | **73.32** | | LIMO + MetamathQA | 817+500 | 31.04 | 46.25 | 47.24 | 56.40 | 45.23 | | LIMO + OpenmathInstruct | 817+500 | 18.13 | 38.96 | 64.53 | 72.40 | 48.50 | *表1：在Qwen-2.5-32B上的增强性能对比。* ### 难度爬坡的影响 难度爬坡过程是性能提升的关键驱动因素。它有效地将数据分布向更具挑战性的问题偏移，从而显著提升下游模型的性能。 <div align="center"> <img src="diffratings_diffhike_barplot.png" style="object-fit: contain;"/> <em><b>图3：</b>难度爬坡将问题从中等难度区间（基础版）转移到更具挑战性的7.0-8.0区间（DH版）。</em> </div> | 数据集 | 数据规模 | AIME25 | AIME24 | AMC24 | MATH500 | **平均得分** | | -------------------------- | ---------- | ------ | ------ | ------ | ------- | ----------- | | LIMO only（Baseline） | 817 | 44.50 | 56.30 | 91.41 | 93.80 | 71.50 | | LIMO + SAND-Math（Base） | 817 + 1500 | 46.38 | 59.09 | 92.71 | 93.60 | 72.94 | | **LIMO + SAND-Math（DH）** | **817 + 1500** | **49.23** | **60.55** | **93.17** | **94.60** | **74.39** | *表2：展示难度爬坡（DH）数据带来性能提升的消融实验结果。* # 许可协议 amd/SAND-MATH数据集采用ResearchRAIL许可协议，仅用于学术与研究用途。如需了解许可协议与条款的更多信息，请参阅本仓库的LICENSE文件。 # 引用信息 @misc{manem2025sandmathusingllmsgenerate, title={SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers}, author={Chaitanya Manem and Pratik Prabhanjan Brahma and Prakamya Mishra and Zicheng Liu and Emad Barsoum}, year={2025}, eprint={2507.20527}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.20527}, }