blackhao0426/PutnamGAP

--- license: cc-by-4.0 language: - en pretty_name: PutnamGAP size_categories: - 1K<n<10K task_categories: - text-generation - question-answering tags: - mathematics - benchmark - robustness - evaluation - putnam - competition-mathematics - llm-evaluation - gap-framework - equivalence-preserving - stress-test configs: - config_name: default data_files: - split: test path: dataset.parquet --- # PutnamGAP [](https://arxiv.org/abs/2508.08833) [](https://github.com/YurenHao0426/GAP) [](https://github.com/YurenHao0426/PutnamGAP) [](https://huggingface.co/datasets/blackhao0426/PutnamGAP) [](https://creativecommons.org/licenses/by/4.0/) **PutnamGAP** is a 6,306-item competition-mathematics benchmark covering every William Lowell Putnam Mathematical Competition problem from **1938 to 2024**, expanded into five mathematically equivalent variants per problem via the **GAP** (Generalization-and-Perturbation) framework. PutnamGAP is intended for stress-testing the **robustness** of large language models on advanced mathematical reasoning under semantically equivalent reformulations of the same problem. > **Paper**: *An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems* — Hao, Wan & Zhai, [arXiv:2508.08833](https://arxiv.org/abs/2508.08833) > > **GAP framework code & evaluation pipeline**: <https://github.com/YurenHao0426/GAP> — this repository hosts only the dataset; the variant generation pipeline, evaluation harness, structural-overlap analysis, repairability rescue runner, and Unicode → LaTeX cleaner all live in the GAP framework repo. > > **PutnamGAP dataset GitHub mirror** (this dataset, mirrored from Hugging Face): <https://github.com/YurenHao0426/PutnamGAP> --- ## What is in the dataset For each of the 1,051 original Putnam problems we provide: | Field | Description | |---|---| | `index` | Canonical problem id, e.g. `1987-B-2` (year-section-problem) | | `type` | Topical category (`ALG`, `ANA`, `NT`, `COMB`, `GEO`) | | `tag` | Free-form sub-tags | | `difficulty` | Difficulty proxy from problem index (1–2 Easy / 3–4 Medium / 5–6 Hard / 7–8 ExtraHard) | | `problem_type` | `proof` or `calculation` | | `question` | Original LaTeX problem statement | | `solution` | Original LaTeX canonical solution | | `vars` / `params` | Free / fixed identifiers extracted from the problem | | `variants` | Five aligned variants — see below | The `variants` object contains: | Variant key | Description | |---|---| | `descriptive_long` (`DL`) | Variables renamed to single descriptive English phrases (e.g.\ `population_density`). | | `descriptive_long_confusing` (`DLC`) | Variables renamed to 2–5 unrelated concatenated nouns. | | `descriptive_long_misleading` (`DLM`) | Variables renamed to mathematically suggestive but semantically misleading names. | | `garbled_string` (`GS`) | Variables renamed to 4–16 character random alphanumeric hashes. | | `kernel_variant` (`KV`) | Numeric / parametric slots resampled while preserving the original proof skeleton; the canonical solution is regenerated and verified by a 5-judge LLM ensemble with explicit repair loops. | Each surface variant additionally exposes a deterministic **rename map** (`variants[v].map`) from canonical variable names to variant variable names. The kernel variant carries provenance metadata in `variants.kernel_variant.metadata`. **1,051 originals × (1 + 5 variants) = 6,306 items.** ### Cleaning All text fields in this release have been processed through a Unicode → bare-LaTeX cleaner so that the contents are pure ASCII LaTeX. Greek letters, math operators, sub/superscripts, radical commands and ligatures have been converted to their LaTeX equivalents (e.g.\ `α` → `\alpha`, `≤` → `\leq`, `√{x+1}` → `\sqrt{x+1}`, `x₁₀` → `x_{10}`). The cleaner has been verified to: - produce **0 non-ASCII characters** across all 1,051 files; - introduce **0 new brace/parenthesis/bracket imbalances** beyond those already present in the source. The cleaning, audit, brace-balance, and spot-check scripts (`unicode_clean.py`, `unicode_audit.py`, `balance_diff.py`, `spotcheck_clean.py`) live in the [GAP framework repository](https://github.com/YurenHao0426/GAP) under `analysis/`, alongside the rest of the GAP pipeline. --- ## Loading The repository contains the same data in two parallel formats: 1. **`dataset.parquet`** — a flat parquet table with 35 columns. This is what the HF dataset viewer renders and what `datasets.load_dataset(...)` returns by default. To keep the schema stable across rows, the four `dict[str, str]`-with-arbitrary-keys fields (`vars`, `params`, `sci_consts`, and per-variant `map` / `metadata`) are stored as JSON-encoded strings whose names end in `_json`. Use `json.loads(...)` to recover the original dict structure. 2. **`dataset/*.json`** — 1,051 individual JSON files with the original nested structure (variants as nested dicts, rename maps as native dicts). Use this layout when running the GAP framework code directly, since the pipeline scripts expect dict access. ### Loading the parquet (default) ```python from datasets import load_dataset import json ds = load_dataset("blackhao0426/PutnamGAP", split="test") print(ds[0]["index"], ds[0]["type"]) # JSON-stringified fields print("vars:", json.loads(ds[0]["vars_json"])) print("DL rename map:", json.loads(ds[0]["variant_descriptive_long_map_json"])) print("KV question:", ds[0]["variant_kernel_variant_question"][:120]) ``` ### Loading the JSON files (preserves nested dicts) ```python import json from huggingface_hub import snapshot_download from pathlib import Path local = snapshot_download("blackhao0426/PutnamGAP", repo_type="dataset", allow_patterns="dataset/*.json") problems = [json.load(open(p)) for p in sorted(Path(local, "dataset").glob("*.json"))] print(f"{len(problems)} problems loaded; e.g. {problems[0]['index']}") print("DL map:", problems[0]["variants"]["descriptive_long"]["map"]) ``` --- ## Suggested Use - **Evaluation, not training.** PutnamGAP is an evaluation benchmark; do not include it in pre-training or fine-tuning corpora that you subsequently evaluate on. - **Paired evaluation.** The framework's value comes from comparing accuracy on `(original, variant)` pairs, not absolute accuracy on the variants in isolation. Report McNemar tests on flip cases. - **Mechanistic analyses.** The surface vs kernel decomposition supports mechanism-sensitive analyses such as the paired structural-overlap dichotomy and the repairability rescue protocol described in the paper. Per-cell trajectory tables and rescue results live in the GitHub companion repository. --- ## Important: Source Attribution > **The original Putnam Competition problem statements and the canonical solutions distributed in this dataset are reproduced from four authoritative monographs published by the Mathematical Association of America (MAA Press), under the fair-use clause printed in the front-matter of every volume:** > > *"Individual readers ... are permitted to make fair use of the material, such as to copy select pages for use in teaching or research."* > > **All original problem statements and canonical solutions remain the intellectual property of the MAA. If you use this dataset for any research output, you MUST also cite the four MAA source books in addition to citing the GAP paper. Failure to do so misrepresents the provenance of the original problems.** Problems and solutions from 2017 onward are included with the explicit permission of MAA. **Takedown notice.** If you are an author, publisher, or rights-holder and you believe any portion of this release infringes your rights, please open an issue at <https://github.com/YurenHao0426/PutnamGAP/issues> or email the maintainer. The affected items will be removed promptly. --- ## Citation If you use PutnamGAP, you **must** cite **all five** entries below: the GAP framework paper **and** the four MAA Putnam source books that the original problems and solutions are reproduced from. Citing fewer is a misrepresentation of the dataset's provenance. In-text example: > "We evaluate on PutnamGAP \cite{hao2025gap, putnamI, putnamII, putnamIII, putnamIV}." Full BibTeX (copy the entire block — all five entries are mandatory): ```bibtex @article{hao2025gap, title = {An Investigation of Robustness of {LLM}s in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems}, author = {Hao, Yuren and Wan, Xiang and Zhai, ChengXiang}, journal = {arXiv preprint arXiv:2508.08833}, year = {2025}, url = {https://arxiv.org/abs/2508.08833} } @book{putnamI, title = {The William Lowell Putnam Mathematical Competition: Problems and Solutions 1938--1964}, author = {Gleason, A. M. and Greenwood, R. E. and Kelly, L. M.}, publisher = {Mathematical Association of America}, year = {1980}, series = {MAA Problem Books}, volume = {1}, address = {Washington, DC}, note = {673\,pp; reprinted by AMS/MAA Press} } @book{putnamII, title = {The William Lowell Putnam Mathematical Competition: Problems and Solutions 1965--1984}, author = {Alexanderson, Gerald L. and Klosinski, Leonard F. and Larson, Loren C.}, publisher = {Mathematical Association of America}, year = {1985}, series = {MAA Problem Books}, volume = {30}, address = {Washington, DC}, note = {Reprinted by AMS/MAA Press} } @book{putnamIII, title = {The William Lowell Putnam Mathematical Competition 1985--2000: Problems, Solutions and Commentary}, author = {Kedlaya, Kiran S. and Poonen, Bjorn and Vakil, Ravi}, publisher = {Mathematical Association of America}, year = {2002}, series = {MAA Problem Books}, volume = {33}, address = {Washington, DC}, note = {Reprinted by AMS/MAA Press} } @book{putnamIV, title = {The William Lowell Putnam Mathematical Competition 2001--2016: Problems, Solutions and Commentary}, author = {Kedlaya, Kiran S. and Kane, Daniel M. and Kane, Jonathan M. and O'Dorney, Evan M.}, publisher = {American Mathematical Society (MAA Press)}, year = {2020}, series = {MAA Problem Books}, volume = {37}, address = {Providence, RI}, note = {Softcover and e-book versions available} } ``` > **Reminder.** The four `putnamI`–`putnamIV` entries are not optional or supplementary; the original problem statements and canonical solutions in this dataset are reproduced from those four MAA monographs under the MAA fair-use clause, and the IP belongs to the Mathematical Association of America. Any downstream use of PutnamGAP that omits the four MAA citations misrepresents the dataset's provenance. --- ## License - The **variant texts**, **rename maps**, **evaluation pipeline**, **structural-overlap analysis code**, **rescue harness**, and **Croissant metadata** are released under the [Creative Commons Attribution 4.0 International License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). - The **original Putnam Competition problem statements and canonical solutions** remain copyrighted by the Mathematical Association of America (MAA). They are redistributed here under MAA's stated fair-use clause and only for educational and research use. **Downstream users must cite the four MAA source books listed above.** - The cleaned LaTeX version of the original solutions is a derivative work whose changes (Unicode → LaTeX normalisation) are released under CC BY 4.0; the underlying text remains MAA-copyrighted. --- ## Links - **Paper (arXiv)**: <https://arxiv.org/abs/2508.08833> - **GAP framework code & evaluation pipeline (GitHub)**: <https://github.com/YurenHao0426/GAP> - **Hugging Face dataset (this release)**: <https://huggingface.co/datasets/blackhao0426/PutnamGAP> - **PutnamGAP dataset GitHub mirror**: <https://github.com/YurenHao0426/PutnamGAP> - **Issues & contact**: <https://github.com/YurenHao0426/GAP/issues>许可证：CC BY 4.0 语言： - 英语 美观名称：PutnamGAP 规模类别： - 1000 < 样本数 < 10000 任务类别： - 文本生成 - 问答 标签： - 数学 - 基准测试集 - 鲁棒性 - 评估 - 普特南竞赛 - 竞赛数学 - 大语言模型（LLM）评估 - GAP框架 - 等价保持 - 压力测试 配置： - 配置名称：default 数据文件： - 划分：测试集 路径：dataset.parquet # PutnamGAP [](https://arxiv.org/abs/2508.08833) [](https://github.com/YurenHao0426/GAP) [](https://github.com/YurenHao0426/PutnamGAP) [](https://huggingface.co/datasets/blackhao0426/PutnamGAP) [](https://creativecommons.org/licenses/by/4.0/) **PutnamGAP**是一款包含6306条数据的竞赛数学基准测试集，涵盖1938年至2024年所有威廉·洛厄尔·普特南数学竞赛（William Lowell Putnam Mathematical Competition）试题，并通过**GAP（泛化与扰动，Generalization-and-Perturbation）**框架为每道试题生成5个数学等价的变体。PutnamGAP旨在针对同一问题的语义等价重表述，压力测试大语言模型（Large Language Model，LLM）在高阶数学推理中的鲁棒性。 > **论文**：*《大语言模型数学推理鲁棒性研究：基于高阶数学问题等价变换的基准测试》* — Hao, Wan & Zhai, [arXiv:2508.08833](https://arxiv.org/abs/2508.08833) > > **GAP框架代码与评估流水线**：<https://github.com/YurenHao0426/GAP> — 本仓库仅托管数据集；变体生成流水线、评估工具、结构重叠分析、可修复性救援运行器以及Unicode→LaTeX清洗工具均存放于GAP框架仓库中。 > > **PutnamGAP数据集GitHub镜像（本数据集从Hugging Face镜像而来）**：<https://github.com/YurenHao0426/PutnamGAP> ## 数据集内容 针对1051道原始普特南试题，我们提供以下字段： | 字段 | 说明 | |---|---| | `index` | 规范试题ID，例如`1987-B-2`（年份-赛段-试题编号） | | `type` | 主题类别（`ALG`代数、`ANA`分析、`NT`数论、`COMB`组合数学、`GEO`几何） | | `tag` | 自由格式子标签 | | `difficulty` | 难度评级，基于试题编号划分：1–2 简单 / 3–4 中等 / 5–6 困难 / 7–8 极难 | | `problem_type` | 试题类型：`proof`证明题或`calculation`计算题 | | `question` | 原始LaTeX试题表述 | | `solution` | 原始LaTeX标准解答 | | `vars` / `params` | 从试题中提取的自由/固定标识符 | | `variants` | 5个对齐变体，详见下文 | `variants`对象包含以下变体键： | 变体键 | 说明 | |---|---| | `descriptive_long` (`DL`) | 长描述性命名（DL）：将变量重命名为单个描述性英文短语（例如`population_density`人口密度） | | `descriptive_long_confusing` (`DLC`) | 长描述性混淆命名（DLC）：将变量重命名为2–5个不相关的拼接名词 | | `descriptive_long_misleading` (`DLM`) | 长描述性误导命名（DLM）：将变量重命名为具有数学暗示但语义误导的名称 | | `garbled_string` (`GS`) | 乱码字符串（GS）：将变量重命名为4–16个字符的随机字母数字哈希值 | | `kernel_variant` (`KV`) | 内核变体（KV）：在保留原始证明框架的前提下，重新采样数值/参数槽；标准解答由5个大语言模型组成的评审团重新生成并验证，包含显式修复循环 | 每个表面变体额外包含一个确定性的**重命名映射**（`variants[v].map`），用于将规范变量名映射至变体变量名。内核变体在`variants.kernel_variant.metadata`中附带来源元数据。 总条目数：1051个原始试题 × (1 + 5个变体) = 6306条。 ## 数据清洗 本版本中的所有文本字段均已通过Unicode→纯LaTeX清洗工具处理，确保内容为纯ASCII格式的LaTeX。希腊字母、数学运算符、上下标、根式命令和连字均已转换为对应的LaTeX格式（例如`α`→`alpha`，`≤`→`leq`，`√{x+1}`→`sqrt{x+1}`，`x₁₀`→`x_{10}`）。该清洗工具已验证可实现： - 所有1051个文件中**无任何非ASCII字符**； - 除源文件中已存在的大括号/圆括号/方括号失衡外，**未引入任何新的失衡情况**。 清洗、审计、括号平衡检查和抽样检查脚本（`unicode_clean.py`、`unicode_audit.py`、`balance_diff.py`、`spotcheck_clean.py`）存放于[GAP框架仓库](https://github.com/YurenHao0426/GAP)的`analysis/`目录下，与GAP流水线的其余组件一同维护。 ## 加载方式 本仓库提供两种并行格式的相同数据： 1. **`dataset.parquet`**：一个包含35列的扁平化Parquet表，这是Hugging Face数据集查看器所渲染的格式，也是`datasets.load_dataset(...)`默认返回的格式。为保持行之间的Schema稳定，四个键为任意字符串的`dict[str, str]`类型字段（`vars`、`params`、`sci_consts`以及每个变体的`map`/`metadata`）均以JSON编码字符串的形式存储，字段名以`_json`结尾。可使用`json.loads(...)`恢复原始字典结构。 2. **`dataset/*.json`**：1051个独立的JSON文件，保留原始嵌套结构（变体为嵌套字典，重命名映射为原生字典）。当直接运行GAP框架代码时推荐使用此布局，因为流水线脚本期望直接以字典方式访问数据。 ### 加载Parquet格式（默认） python from datasets import load_dataset import json ds = load_dataset("blackhao0426/PutnamGAP", split="test") print(ds[0]["index"], ds[0]["type"]) # JSON编码的字段 print("vars:", json.loads(ds[0]["vars_json"])) print("DL 重命名映射:", json.loads(ds[0]["variant_descriptive_long_map_json"])) print("KV 试题:", ds[0]["variant_kernel_variant_question"][:120]) ### 加载JSON文件（保留嵌套字典结构） python import json from huggingface_hub import snapshot_download from pathlib import Path local = snapshot_download("blackhao0426/PutnamGAP", repo_type="dataset", allow_patterns="dataset/*.json") problems = [json.load(open(p)) for p in sorted(Path(local, "dataset").glob("*.json"))] print(f"{len(problems)} 道试题已加载；例如 {problems[0]['index']}") print("DL 重命名映射:", problems[0]["variants"]["descriptive_long"]["map"]) ## 推荐使用方式 - **仅用于评估，不可用于训练**。PutnamGAP是一款评估基准测试集，请勿将其纳入后续会用于评估的预训练或微调语料库。 - **配对评估**。该框架的价值在于对比`(原始试题, 变体试题)`配对的准确率，而非单独评估变体的绝对准确率。请针对翻转案例报告McNemar检验。 - **机制性分析**。表面变体与内核变体的分解支持机制敏感型分析，例如论文中描述的配对结构重叠二分法和可修复性救援协议。单元格轨迹表和救援结果存放于GitHub配套仓库中。 ## 重要说明：来源归属 > 本数据集中分发的原始普特南竞赛试题表述与标准解答，均复刻自美国数学协会（Mathematical Association of America，MAA Press）出版的四本权威专著，遵循每卷前言中注明的合理使用条款： > > *“个人读者……有权合理使用本材料，例如复制精选页面用于教学或研究。”* > > 所有原始试题表述与标准解答的知识产权仍归MAA所有。若您将本数据集用于任何研究成果，除引用GAP论文外，还必须引用这四本MAA源书籍。未遵守此要求将歪曲数据集的来源信息。 2017年及之后的试题与解答已获得MAA的明确许可。 **下架通知**。若您为作者、出版商或权利持有人，且认为本发布的任何内容侵犯了您的权利，请在<https://github.com/YurenHao0426/PutnamGAP/issues>提交Issue或联系维护者。受影响的条目将被立即移除。 ## 引用方式 若您使用PutnamGAP，**必须**引用以下全部五项条目：GAP框架论文以及复刻原始试题与解答的四本MAA普特南源书籍。引用不足将歪曲数据集的来源信息。 文中引用示例： > "We evaluate on PutnamGAP cite{hao2025gap, putnamI, putnamII, putnamIII, putnamIV}." 完整BibTeX格式（请复制整个区块——所有五项条目均为必填）： bibtex @article{hao2025gap, title = {An Investigation of Robustness of {LLM}s in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems}, author = {Hao, Yuren and Wan, Xiang and Zhai, ChengXiang}, journal = {arXiv preprint arXiv:2508.08833}, year = {2025}, url = {https://arxiv.org/abs/2508.08833} } @book{putnamI, title = {The William Lowell Putnam Mathematical Competition: Problems and Solutions 1938--1964}, author = {Gleason, A. M. and Greenwood, R. E. and Kelly, L. M.}, publisher = {Mathematical Association of America}, year = {1980}, series = {MAA Problem Books}, volume = {1}, address = {Washington, DC}, note = {673,pp; reprinted by AMS/MAA Press} } @book{putnamII, title = {The William Lowell Putnam Mathematical Competition: Problems and Solutions 1965--1984}, author = {Alexanderson, Gerald L. and Klosinski, Leonard F. and Larson, Loren C.}, publisher = {Mathematical Association of America}, year = {1985}, series = {MAA Problem Books}, volume = {30}, address = {Washington, DC}, note = {Reprinted by AMS/MAA Press} } @book{putnamIII, title = {The William Lowell Putnam Mathematical Competition 1985--2000: Problems, Solutions and Commentary}, author = {Kedlaya, Kiran S. and Poonen, Bjorn and Vakil, Ravi}, publisher = {Mathematical Association of America}, year = {2002}, series = {MAA Problem Books}, volume = {33}, address = {Washington, DC}, note = {Reprinted by AMS/MAA Press} } @book{putnamIV, title = {The William Lowell Putnam Mathematical Competition 2001--2016: Problems, Solutions and Commentary}, author = {Kedlaya, Kiran S. and Kane, Daniel M. and Kane, Jonathan M. and O'Dorney, Evan M.}, publisher = {American Mathematical Society (MAA Press)}, year = {2020}, series = {MAA Problem Books}, volume = {37}, address = {Providence, RI}, note = {Softcover and e-book versions available} } > **提醒**。四个`putnamI`–`putnamIV`条目并非可选或补充内容；本数据集中的原始试题表述与标准解答均复刻自这四本MAA专著，遵循MAA的合理使用条款，知识产权归美国数学协会所有。任何省略这四本MAA引用的PutnamGAP下游使用行为，均会歪曲数据集的来源信息。 ## 许可证 - **变体文本、重命名映射、评估流水线、结构重叠分析代码、救援测试工具与Croissant元数据**均基于[知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International License，CC BY 4.0）](https://creativecommons.org/licenses/by/4.0/)发布。 - **原始普特南竞赛试题表述与标准解答**仍受美国数学协会（MAA）版权保护。本仓库仅依据MAA的合理使用条款对其进行再分发，仅用于教育与研究用途。**下游用户必须引用上述列出的四本MAA源书籍**。 - 原始解答的清洗后LaTeX版本属于衍生作品，其修改内容（Unicode→LaTeX规范化）基于CC BY 4.0发布；但其基础文本仍受MAA版权保护。 ## 相关链接 - **论文（arXiv）**：<https://arxiv.org/abs/2508.08833> - **GAP框架代码与评估流水线（GitHub）**：<https://github.com/YurenHao0426/GAP> - **Hugging Face 数据集（本发布版本）**：<https://huggingface.co/datasets/blackhao0426/PutnamGAP> - **PutnamGAP数据集GitHub镜像**：<https://github.com/YurenHao0426/PutnamGAP> - **问题与联系方式**：<https://github.com/YurenHao0426/GAP/issues>