PKU-DS-LAB/ScholarSearch

---
license: apache-2.0
language:
- zh
---

# Welcome to ScholarSearch created by PKU-DS-LAB!

## Dataset Description

ScholarSearch is the first dataset specifically designed to evaluate the complex information retrieval capabilities of Large Language Models (LLMs) in academic research.

Key characteristics of ScholarSearch include:

- **Academic Practicality**: Questions are based on real academic learning and research environments, avoiding misleading the models.
- **High Difficulty**: Answers often require at least three deep searches to derive, making them challenging for single models.
- **Concise Evaluation**: Answers are unique, with clear sources and brief explanations, facilitating audit and verification.
- **Broad Coverage**: The dataset spans at least 12 different academic disciplines, including Computer Science, Literature, Biology, Political Science, Economics, Mathematics, Demography, History of Science and Technology, Chemistry, Sociology, Public Health, and Physics.

The dataset consists of 223 meticulously curated questions in Chinese, each accompanied by an answer, explanation, and domain. It was created by a team of undergraduate and graduate students from various faculties at Peking University, ensuring the questions reflect genuine academic search scenarios.

## Dataset Structure

Each entry in the dataset contains the following fields:

- **question**: The academic query or problem.
- **answer**: The correct answer to the question.
- **explanation**: A brief explanation or justification for the answer, including sources.
- **domain**: The academic discipline or field to which the question belongs.

The dataset is provided as a JSON file containing a list of entries.

## Experiment Result

| **Model** | **All (%)** | **Science & Engineering (%)** | **Social Sciences & Humanities (%)** |
|-----------|:-------------:|:-------------------------------:|:--------------------------------------:|
| gpt-4o-search-preview | 18.83 | 18.64 | 19.05 |
| gpt-4o-mini-search-preview | 10.31 | 10.17 | 10.48 |
| deepseek-r1-0528 | 8.52 | 5.08 | 12.38 |
| gpt-4.1 | 7.17 | 5.93 | 8.57 |
| gpt-4o-2024-11-20 | 3.59 | 1.69 | 5.71 |
| gpt-4o-mini | 2.24 | 0.85 | 3.81 |

*The judge model for all experiments is GPT-4o-mini.*

## Citation Information

Paper Link: https://arxiv.org/abs/2506.13784

## Additional Information

- This project was funded by Grant 624B2005.
- We would like to thank the following individuals for their contributions to problem-solving and evaluation: Xun Zhao, Zizhuo Fu, Yuqian Zhan, Xinhao Ji, Jiarui Sun, Junhao Zhang, Shengfan Wang, Ziteng Lu, Yumeng Song, Ziyan Yang, Hongjiao Wang, Shan Zhang, Huahui Lin, Junhong Liu, Zhengyang Wang, Yuchen Lu, Yanxi Xu.

## Team Members:

Leading By:

Tong Yang; Yuhan Wu;

Core Contributors:

Junting Zhou; Wang Li; Yiyan Liao; Nengyuan Zhang; Tingjia Miao; Zhihui Qi

## Dataset Card Contact

For more details, please contact: [e-mail]

ScholarSearch is the first dataset specifically designed to evaluate the complex information retrieval capabilities of Large Language Models (LLMs) in academic research. It consists of 223 meticulously curated questions in Chinese, each accompanied by an answer, explanation, and domain, covering at least 12 different academic disciplines such as Computer Science, Literature, Biology, Political Science, Economics, Mathematics, Demography, History of Science and Technology, Chemistry, Sociology, Public Health, and Physics.

搜集汇总

背景与挑战

背景概述

ScholarSearch是首个专门用于评估大型语言模型在学术研究中复杂信息检索能力的数据集，具有高难度和广泛学科覆盖的特点。该数据集包含223个中文问题，每个问题均基于真实学术场景设计，答案通常需要多次深度搜索才能得出，旨在挑战模型的检索和推理能力。