Perturbed evaluation datasets for "More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists"

Perturbed evaluation datasets for "More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists" This archive contains the adversarially perturbed math QA evaluation sets used in our paper (arXiv:2511.07112). It accompanies the official code repository at https://github.com/akkarimi/multi-agent-adversarial-robustness. We perturb four math benchmarks - GSM8K, MATH (the same 20-per-(level,subject) subset distributed by Li et al., 2024), MMLU-Math (five math splits: abstract algebra, college mathematics, elementary mathematics, high school mathematics, high school statistics), and MultiArith - under five noise regimes: - clean: passthrough copy of the original questions (baseline).- punctuation_10, punctuation_30, punctuation_50: AEDA-style insertion of random punctuation (. , ! ? ; :) at 10%, 30%, and 50% of the token count.- wikitypo: word-level real-world typo injection drawn from a WikiTypo correction dictionary (also included in the archive).- r2ata: human-like typos sourced from the R2-ATA corpus (only the questions that overlap our four benchmarks are included; we do not redistribute the original R2-ATA release). Only the question text is modified; ground-truth labels are preserved verbatim, so accuracy under any noise type is directly comparable to clean accuracy on the same record id. Punctuation and WikiTypo perturbations are deterministic given the seed (seed=0) used by tools/data_prep/perturb.py in the companion repository, and the bundled files are the exact ones used to produce every number and figure in the paper. Archive contents (extracts to two top-level directories): perturbed/ gsm/{gsm_dataset_clean, gsm_dataset_punctuation_10, ..., gsm_dataset_r2ata}/test.jsonl math/{math_dataset_<dtype>}/math_subset_20.json mmlu/{mmlu_dataset_<dtype>}/<subject>_test.csv multiarith/{multiarith_dataset_<dtype>}/test.json wikitypo/ wiki_typos_en_p160000_l5186.pkl wiki_typos_en_p320000_l9368.pkl typo_modification_report.csv Per-record schemas are documented in docs/DATA.md in the companion repository. How to use: git clone https://github.com/akkarimi/multi-agent-adversarial-robustness.git cd multi-agent-adversarial-robustness ZENODO_RECORD=<this-record-id> bash scripts/download_data.sh bash scripts/stage_dataset.sh stage_dataset.sh copies the perturbed sets into AgentForest/dataset/, where the AgentForest code expects to find them via its ../dataset/... relative paths. Citation: If you use these datasets, please cite the paper: @article{alavi2025more, title = {More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists}, author = {Alavi, Khashayar and Yeltay, Zhastay and Flek, Lucie and Karimi, Akbar}, journal= {arXiv preprint arXiv:2511.07112}, year = {2025} } Please also cite the upstream sources of the underlying benchmarks (GSM8K - Cobbe et al. 2021; MATH and MMLU - Hendrycks et al. 2021; MultiArith - Roy & Roth 2015) and the WikiTypo and R2-ATA datasets when relevant. License: CC-BY-4.0. The companion code repository is released under the MIT License.