LoHoSearch

<div align=center><h1> LoHoSearch: Benchmarking Long-Horizon<br> Search Agents Beyond the Human Difficulty Ceiling </h1></div> <p align="center"> 📃 <a href="https://arxiv.org/abs/2606.12837" target="_blank">Paper</a> • 🏆 <a href="https://huggingface.co/datasets/LoHoSearch-Team/LoHoSearch/viewer/benchmark" target="_blank">Benchmark</a> • 📦 <a href="https://huggingface.co/datasets/LoHoSearch-Team/LoHoSearch/viewer/train" target="_blank">Training Data</a> </p> ## Abstract Search agent benchmarks exemplified by BrowseComp have rapidly saturated over the past year, with the strongest models surpassing 90% accuracy. Since these benchmarks are predominantly human-authored, annotators lack a global perspective on entity statistics and cannot systematically maximize search space size and structural complexity. This creates a difficulty ceiling that is hard to break. To address this, we introduce **LoHoSearch** (**Lo**ng-**Ho**rizon **Search** Agents), a challenging benchmark comprising 544 human-verified questions across 11 domains. LoHoSearch is constructed via an automated pipeline built upon a knowledge graph covering over 7 million Wikipedia entities, which selects relations with large search spaces and assembles them into structurally complex questions with KG-verified unique answers. Our evaluation demonstrates that even the strongest model achieves only **34.74%** accuracy, and existing context management strategies (best +6.8%) yield far smaller gains than on prior benchmarks. LoHoSearch provides a more demanding standard for evaluating long-horizon reasoning and context management in search agents. For more details, see our <a href="https://arxiv.org/abs/2606.12837" target="_blank"><u>paper</u></a>. ## Dataset This repository contains two subsets: | Config | File | Split | Records | Description | Language |:--|:--|:--:|:--:|:--|:--| | `benchmark` | `LoHoSearch.csv` | test | 544 | Human-verified evaluation benchmark | English | `train` | `train.csv` | train | 2000 | Training set generated by the same automated pipeline, without human verification | English <p align="center"> <img src="assets/domain_distribution.png" alt="Domain Distribution" width="50%"> </p> ## Main Results **Evaluation setup.** Each model is equipped with two tools, `search` (keyword queries via a traditional search engine) and `browse` (fetch the content of given URLs), and uses the same system prompt as BrowseComp. We set temperature to 1.0, keep each model's default thinking settings, and use a 200K context window (184K input + 16K output). The **score** is the average correct ratio over the 544 questions, computed by averaging two LLM-judge gradings: the BrowseComp grading prompt with GPT-4.1 as judge, and the SimpleQA grading prompt with Qwen2.5-32B as judge. Averaging two complementary judges avoids the over-strictness or over-leniency of any single setup. | Model | Reasoning | Source | LoHoSearch Score (%) | |:--|:--:|:--:|:--:| | GPT-5.5 | N | Closed | **34.74** | | DeepSeek-V4-Pro | Y | Open | 15.99 | | Claude-Opus-4.6 | N | Closed | 15.62 | | Kimi-K2.6 | Y | Open | 15.53 | | Gemini-3.1-Pro | Y | Closed | 13.32 | | GLM-5.1 | Y | Open | 12.77 | | Claude-Opus-4.7 | N | Closed | 10.29 | | DeepSeek-V4-Flash | Y | Open | 10.02 | | LongCat-Flash-Thinking-2601 | Y | Open | 9.74 | | MiniMax-M2.7 | Y | Open | 2.48 | | MiniMax-M2.5 | Y | Open | 2.29 | ## Construction Pipeline <p align="center"> <img src="assets/main_pipeline.png" alt="Pipeline Overview" width="70%"> </p> The benchmark is constructed through four stages: 1. **Knowledge Graph Construction**: Built from the full English Wikipedia dump with Wikidata type annotations. 2. **Subgraph Sampling**: Tree-structured and graph-structured subgraphs are sampled with constraints on search space size, structural complexity, and answer uniqueness. 3. **QA Generation and Verification**: Relations are extracted and obfuscated, then assembled into natural-language questions with automated coverage and satisfaction checks. 4. **Post Filtering and Human Review**: Multiple rounds of uniqueness verification, difficulty filtering, and professional human annotation. ## Citation ```bibtex @misc{zhao2026lohosearchbenchmarkinglonghorizonsearch, title={LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling}, author={Jiarui Zhao and Rongzhi Zhang and Lingchuan Liu and Hao Yang and Xunliang Cai and Xi Su}, year={2026}, eprint={2606.12837}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2606.12837}, } ``` ## License This dataset is released under the MIT License.