BlendQA

# Dataset Card for BlendQA

We introduce BlendQA, a challenging benchmark specially tailored for heterogeneous knowledge reasoning.
BlendQA assesses a RAG system's ability to conduct flexible cross-knowledge source retrieval between reasoning steps.
We construct BlendQA across three heterogeneous knowledge sources: a full Wikipedia dump as the local text corpus, Google as the web search engine, and Wikidata as the structured knowledge graph.
BlendQA is carefully constructed through manual verification, comprising 445 total questions: 132 KG-Web questions, 163 Text-KG questions, and 150 Text-Web questions.
Existing systems achieve a maximum overall F1 score of 43.32% on BlendQA, highlighting its difficulty.

For more details, please refer to:
- Paper 📖 [AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning](https://arxiv.org/abs/2411.16495)
- [Github Repository](https://github.com/THU-KEG/AtomR)

If you feel this dataset is helpful, please cite our paper:

```
@article{xin2024atomr,
title={AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning},
author={Xin, Amy and Liu, Jinxin and Yao, Zijun and Li, Zhicheng and Cao, Shulin and Hou, Lei and Li, Juanzi},
journal={arXiv preprint arXiv:2411.16495},
year={2024}
}
```

### Data Description

- **Developed by:** Amy Xin, Jinxin Liu, Zijun Yao, Zhicheng Lee, Shulin Cao, Lei Hou, Juanzi Li
- **Language(s):** English

# BlendQA 数据集卡片

我们提出BlendQA，这是一款专为异构知识推理量身打造的高挑战性基准测试集。BlendQA用于评估检索增强生成（Retrieval-Augmented Generation, RAG）系统在推理步骤间开展灵活跨知识库检索的能力。

我们基于三类异构知识库构建BlendQA：作为本地文本语料库的完整维基百科转储、作为网页搜索引擎的谷歌（Google），以及作为结构化知识图谱（Knowledge Graph, KG）的维基数据（Wikidata）。BlendQA经人工校验精心构建，总计包含445道问题：其中132道为KG-Web类问题、163道为Text-KG类问题，以及150道为Text-Web类问题。现有系统在BlendQA上的最高整体F1分数仅为43.32%，足见该基准测试集的难度。

如需了解更多细节，请参阅：
- 论文 📖 [AtomR：面向异构知识推理的原子算子赋能大语言模型（Large Language Model, LLM）](https://arxiv.org/abs/2411.16495)
- [GitHub 仓库](https://github.com/THU-KEG/AtomR)

若本数据集对您的研究有所助益，请引用我们的论文：

@article{xin2024atomr,
title={AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning},
author={Xin, Amy and Liu, Jinxin and Yao, Zijun and Li, Zhicheng and Cao, Shulin and Hou, Lei and Li, Juanzi},
journal={arXiv preprint arXiv:2411.16495},
year={2024}
}

### 数据说明

- **开发者：** Amy Xin、刘金鑫、姚梓俊、李志成、曹树林、侯磊、李涓子
- **语言：** 英语