ShadenA-MathNet-Retrieve

# MathNet-Retrieve The **Math-Aware Retrieval** benchmark from [MathNet](https://mathnet.mit.edu) ([paper](https://arxiv.org/abs/2604.18584)): given an Olympiad problem, retrieve a *mathematically equivalent* reformulation of it from a large corpus of hard negatives. Each equivalent is provided at three difficulty tiers — `easy`, `medium`, `hard` — that increasingly disguise the surface form while preserving the underlying problem, so retrieval gets sharply harder even though the answer is the same. - **15,000** queries, one gold-equivalent target per tier. - **3 tiers** (`easy`, `medium`, `hard`), over a shared **117,088-doc** corpus (equivalents at all tiers + LLM near-miss hard negatives + distractors). - Standard **BEIR / MTEB** layout. ## Quick start ```python from datasets import load_dataset tier = "hard" # one of "easy", "medium", "hard" corpus = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-corpus", split="test") # {_id, text} queries = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-queries", split="test") # {_id, text} qrels = load_dataset("ShadenA/MathNet-Retrieve", f"{tier}-qrels", split="test") # {query-id, corpus-id, score} ``` All three configs share the same corpus and queries; they differ only in which reformulation is the gold positive. Each query has exactly **one** positive, and the query's own equivalents at the other tiers act as negatives. ## Tiers | Tier | Queries | Corpus | What changes | |---|---|---|---| | easy | 15,000 | 117,088 | light paraphrase (renamed vars, reordered clauses) | | medium | 15,000 | 117,088 | moderate restructuring (notation, framing) | | hard | 15,000 | 117,088 | heavy disguise; minimal lexical overlap | ## Results Recall@k (%) on the shared corpus; **bold** = best. | Model | Easy R@1 | Easy R@5 | Medium R@1 | Medium R@5 | Hard R@1 | Hard R@5 | |---|---|---|---|---|---|---| | `gemini-embedding-001` | 11.4 | **90.7** | 2.5 | **77.1** | 0.0 | **9.3** | | `qwen3-embedding-4B` | **14.8** | 83.5 | **2.9** | 71.6 | 0.0 | 8.4 | | `text-embedding-3-large` | 6.8 | 78.1 | 1.6 | 68.3 | 0.0 | 2.6 | | `all-mpnet-base-v2` | 6.8 | 82.5 | 1.3 | 51.0 | 0.0 | 1.2 | | `multi-qa-mpnet-base-dot-v1` | 5.1 | 80.6 | 1.1 | 47.3 | 0.0 | 0.8 | | `text-embedding-ada-002` | 3.3 | 66.1 | 1.1 | 55.4 | 0.0 | 0.7 | | `text-embedding-3-small` | 3.6 | 59.8 | 1.4 | 51.5 | 0.0 | 0.5 | Every model degrades sharply from easy to hard: even the strongest solve `easy` but collapse on `hard` (Recall@1 ≈ 0; hard Recall@10 tops out near 54%). Retrieving equivalent problems remains unreliable once the surface form is disguised. ## Caveats - **Not contamination-clean** — source problems may appear in pretraining data; prefer relative comparisons. - Equivalents and hard negatives are **LLM-generated and machine-verified**; a small fraction may be imperfect. ## License & citation Released under **CC BY 4.0**, consistent with [`ShadenA/MathNet`](https://huggingface.co/datasets/ShadenA/MathNet). ```bibtex @inproceedings{alshammari2026mathnet, title = {MathNet: A Global Multimodal Benchmark for Mathematical Reasoning and Retrieval}, author = {Alshammari, Shaden and Wen, Kevin and Zainal, Abrar and Hamilton, Mark and Safaei, Navid and Albarakati, Sultan and Freeman, William T. and Torralba, Antonio}, booktitle = {International Conference on Learning Representations}, year = {2026}, url = {https://mathnet.mit.edu} } ``` [Project site](https://mathnet.mit.edu) · [Paper](https://arxiv.org/abs/2604.18584) · [Main dataset](https://huggingface.co/datasets/ShadenA/MathNet)