DeepDive

# DeepDive Dataset

<div align="center">

[](https://github.com/THUDM/DeepDive)
[](https://arxiv.org/pdf/2509.10446)
[](https://huggingface.co/datasets/zai-org/DeepDive)
[](#)

</div>

## Overview

This is the training dataset for [DeepDive](https://github.com/THUDM/DeepDive), an automated approach for training deep search agents with complex, multi-step reasoning capabilities. The dataset is constructed through automated knowledge graph random walks, entity obfuscation, and difficulty filtering to create challenging questions that require sophisticated search and retrieval skills.

<div align="center">
<img src="./assets/kg_data_pipeline.svg" alt="Data Synthesis Pipeline" width="75%">
<p><em></em></p>
</div>

## Dataset Statistics

| Component | Split | Size | Description |
| :--------------------- | :--------------- | :---- | :-------------------------------------------------------- |
| **Total Dataset** | qa_sft, qa_rl | 3,250 | Complete collection of QA pairs |
| **SFT Portion** | qa_sft | 1,016 | Question-answer pairs for Supervised Fine-Tuning |
| ↳ **SFT Trajectories** | trajectories_sft | 858 | Search trajectories from SFT QA pairs via reject sampling |
| **RL Portion** | qa_rl | 2,234 | Question-answer pairs for Reinforcement Learning |

## Data Structure

### `qa_sft` and `qa_rl` Split

**Fields:**

- `id`: Unique identifier for the QA pair
- `question`: Multi-hop reasoning question requiring search
- `answer`: Ground truth answer
- `conversation`: [] (empty)

### `trajectories_sft` Split

**Fields:**

- `id`: Unique identifier for the trajectory
- `question`: The original question
- `answer`: Ground truth answer
- `conversation`: List of conversation turns showing the search process (role, content)

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("zai-org/DeepDive")

# Access splits
sft_data = dataset["qa_sft"]
rl_data = dataset["qa_rl"]
trajectories = dataset["trajectories_sft"]
```

## Citation

If you find DeepDive useful for your research, please cite our paper:

```bibtex
@misc{lu2025deepdiveadvancingdeepsearch,
title={DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL},
author={Rui Lu and Zhenyu Hou and Zihan Wang and Hanchen Zhang and Xiao Liu and Yujiang Li and Shi Feng and Jie Tang and Yuxiao Dong},
year={2025},
eprint={2509.10446},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2509.10446},
}
```

# DeepDive 数据集

<div align="center">

[](https://github.com/THUDM/DeepDive)
[](https://arxiv.org/pdf/2509.10446)
[](https://huggingface.co/datasets/zai-org/DeepDive)
[](#)

</div>

## 概览
本数据集为[DeepDive](https://github.com/THUDM/DeepDive)的训练数据集，后者是一种用于训练具备复杂多步推理能力的深度搜索智能体（AI Agent）的自动化方法。本数据集通过自动化知识图谱（Knowledge Graph）随机游走、实体混淆与难度筛选构建，生成需要运用高级搜索与检索技能的挑战性问题。

<div align="center">
<img src="./assets/kg_data_pipeline.svg" alt="数据合成流程" width="75%">
<p><em></em></p>
</div>

## 数据集统计
| 组件 | 拆分集 | 规模 | 描述 |
| :--------------------- | :--------------- | :---- | :-------------------------------------------------------- |
| **总数据集** | qa_sft, qa_rl | 3,250 | 完整的问答对集合 |
| **监督微调（Supervised Fine-Tuning，SFT）子集** | qa_sft | 1,016 | 用于监督微调的问答对 |
| ↳ **SFT轨迹** | trajectories_sft | 858 | 通过拒绝采样从SFT问答对中获取的搜索轨迹 |
| **强化学习（Reinforcement Learning，RL）子集** | qa_rl | 2,234 | 用于强化学习的问答对 |

## 数据结构

### `qa_sft` 与 `qa_rl` 拆分集

**字段说明：**
- `id`：问答对的唯一标识符
- `question`：需要进行多跳推理的搜索类问题
- `answer`：标准答案
- `conversation`：[]（空列表）

### `trajectories_sft` 拆分集

**字段说明：**
- `id`：轨迹的唯一标识符
- `question`：原始问题
- `answer`：标准答案
- `conversation`：展示搜索过程的对话轮次列表（包含角色与内容）

## 使用方法
python
from datasets import load_dataset

dataset = load_dataset("zai-org/DeepDive")

# 加载拆分集
sft_data = dataset["qa_sft"]
rl_data = dataset["qa_rl"]
trajectories = dataset["trajectories_sft"]

## 引用声明
如果您的研究中用到了DeepDive数据集，请引用我们的论文：
bibtex
@misc{lu2025deepdiveadvancingdeepsearch,
title={DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL},
author={Rui Lu and Zhenyu Hou and Zihan Wang and Hanchen Zhang and Xiao Liu and Yujiang Li and Shi Feng and Jie Tang and Yuxiao Dong},
year={2025},
eprint={2509.10446},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2509.10446},
}