li-lab/Omanic

---
license: mit
configs:
- config_name: default
data_files:
- split: OmanicSynth
path: OmanicSynth.jsonl
- split: OmanicBench
path: OmanicBench.jsonl
---

# Omanic

The dataset contains two splits:

- `OmanicSynth`: 10,296 machine-generated training examples.
- `OmanicBench`: 967 expert-reviewed, human-annotated evaluation examples.

Each row is a single 4-hop reasoning instance.

For more details, please refer to the paper: [Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models](https://arxiv.org/abs/2603.16654).

## Data Fields

Each example contains the following top-level keys:

- `id`: A unique example identifier.
- `single_hop`: A list of decomposed single-hop sub-questions that form the reasoning chain. In Omanic, this chain is typically 4 hops long.
- `multi_hop_question`: The final multi-hop question.
- `multi_hop_answer`: The gold final answer to the multi-hop question.
- `multiple_choice_options`: The final answer options for the multi-hop question. In the Omanic, these are the same as the options for the last single-hop question.
- `correct_answer_label`: The correct option label for the final multi-hop question, typically one of `A`, `B`, `C`, or `D`.
- `motif_type`: The reasoning graph topology of the multi_hop_question. Common values include `Bridge`, `Chain`, and `Converging`.
- `Bridge`: the answer to `Step1` must appear in the question for `Step2`, and the answers to `Step2` and `Step3` must both appear in the question for `Step4`.
- `Chain`: each step's question must contain the answer from the immediately preceding step.
- `Converging`: the answers to `Step1` and `Step2` must both appear in the question for `Step3`, and the answer to `Step3` must appear in the question for `Step4`.

Here, `StepX` refers to the X-th sub-question in `single_hop`.

### `single_hop`

`single_hop` is a list of 4 sinle-hop questions. Each element describes one intermediate reasoning step and typically contains:

- `question`: The single-hop sub-question for this reasoning step.
- `answer`: The gold answer to this sub-question.
- `multiple_choice_options`: The multiple-choice options for this hop.
- `correct_answer_label`: The correct option label for this hop.
- `domain`: The semantic domain assigned to this hop, such as history, science, economics, or culture. This element is present for every hop except the first one.

## Example Schema

```json
{
"id": "551573_631365_53191",
"single_hop": [
{
"question": "Who is the author of Candida?",
"answer": "George Bernard Shaw",
"multiple_choice_options": {
"A": "Anton Chekhov",
"B": "Henrik Ibsen",
"C": "George Bernard Shaw",
"D": "Oscar Wilde"
},
"correct_answer_label": "C"
},
{
"question": "What is the country of citizenship of George Bernard Shaw?",
"answer": "Ireland",
"multiple_choice_options": {
"A": "Scotland",
"B": "England",
"C": "Ireland",
"D": "United Kingdom"
},
"correct_answer_label": "C",
"domain": "History and Literature"
},
{
"question": "How many distinct 3-member committees can be formed from a group of 7 candidates?",
"answer": "35",
"multiple_choice_options": {
"A": "21",
"B": "35",
"C": "42",
"D": "28"
},
"correct_answer_label": "B",
"domain": "Mathematics and Logic"
},
{
"question": "In Ireland, which political party was founded 35 years before 1968?",
"answer": "Fine Gael",
"multiple_choice_options": {
"A": "Fine Gael",
"B": "Labour Party",
"C": "Fianna Fáil",
"D": "Sinn Féin"
},
"correct_answer_label": "A",
"domain": "History and Literature"
}
],
"multi_hop_question": "In the country of citizenship of the author of Candida, which political party was founded the same number of years before 1968 as the number of distinct 3-member committees that can be formed from a group of 7 candidates?",
"multi_hop_answer": "Fine Gael",
"multiple_choice_options": {
"A": "Fine Gael",
"B": "Labour Party",
"C": "Fianna Fáil",
"D": "Sinn Féin"
},
"correct_answer_label": "A",
"motif_type": "Chain"
}
```

## Contact

For any inquiries, please reach out at **[e-mail]**

## Citation

If you find Omanic useful for your research and applications, please cite:

```bibtex
@article{gu2026omanic,
title={Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models},
author={Gu, Xiaojie and Tong, Sherry T and Feng, Aosong and Han, Sophia Simeng and Lu, Jinghui and Chen, Yingjian and Iwasawa, Yusuke and Matsuo, Yutaka and Park, Chanjun and Ying, Rex and Li, Irene},
journal={arXiv preprint arXiv:2603.16654},
year={2026}
}
```

许可证：MIT许可证
配置项：
- 配置名称：default
数据文件：
- 数据集划分：OmanicSynth
文件路径：OmanicSynth.jsonl
- 数据集划分：OmanicBench
文件路径：OmanicBench.jsonl

# Omanic
本数据集包含两个数据集划分：
- `OmanicSynth`：包含10296条机器生成的训练样本。
- `OmanicBench`：包含967条经专家审核、人工标注的评估样本。
每一行均为一个单组4跳推理实例。

如需了解更多细节，请参阅论文：[Omanic：面向大语言模型（Large Language Model）多跳推理的逐步评估](https://arxiv.org/abs/2603.16654)。

## 数据字段
每个样本包含以下顶级键：
- `id`：唯一的样本标识符。
- `single_hop`：构成推理链的分解后单跳子问题列表。在Omanic数据集中，该推理链通常包含4个跳转步骤。
- `multi_hop_question`：最终的多跳问题。
- `multi_hop_answer`：该多跳问题的标准答案（金标准答案）。
- `multiple_choice_options`：该多跳问题的最终备选项。在Omanic数据集中，这些选项与最后一个单跳子问题的备选项完全一致。
- `correct_answer_label`：该多跳问题的正确选项标签，通常为`A`、`B`、`C`或`D`之一。
- `motif_type`：该多跳问题的推理图拓扑结构，常见取值包括`Bridge`（桥接型）、`Chain`（链式）和`Converging`（汇聚型）。
- `Bridge`（桥接型）：`Step2`的问题中必须包含`Step1`的答案，且`Step4`的问题中必须同时包含`Step2`和`Step3`的答案。
- `Chain`（链式）：每一步的问题都必须包含前一步骤的答案。
- `Converging`（汇聚型）：`Step3`的问题中必须同时包含`Step1`和`Step2`的答案，且`Step4`的问题中必须包含`Step3`的答案。
此处`StepX`指代`single_hop`列表中的第X个子问题。

### `single_hop`
`single_hop`是包含4个单跳问题的列表，每个元素对应一个中间推理步骤，通常包含以下字段：
- `question`：该推理步骤对应的单跳子问题。
- `answer`：该子问题的标准答案（金标准答案）。
- `multiple_choice_options`：该单跳步骤的备选项。
- `correct_answer_label`：该单跳步骤的正确选项标签。
- `domain`：该单跳步骤所属的语义领域，例如历史、科学、经济学或文化。除第一个步骤外，所有单跳步骤均包含该字段。

## 示例结构
json
{
"id": "551573_631365_53191",
"single_hop": [
{
"question": "Who is the author of Candida?",
"answer": "George Bernard Shaw",
"multiple_choice_options": {
"A": "Anton Chekhov",
"B": "Henrik Ibsen",
"C": "George Bernard Shaw",
"D": "Oscar Wilde"
},
"correct_answer_label": "C"
},
{
"question": "What is the country of citizenship of George Bernard Shaw?",
"answer": "Ireland",
"multiple_choice_options": {
"A": "Scotland",
"B": "England",
"C": "Ireland",
"D": "United Kingdom"
},
"correct_answer_label": "C",
"domain": "History and Literature"
},
{
"question": "How many distinct 3-member committees can be formed from a group of 7 candidates?",
"answer": "35",
"multiple_choice_options": {
"A": "21",
"B": "35",
"C": "42",
"D": "28"
},
"correct_answer_label": "B",
"domain": "Mathematics and Logic"
},
{
"question": "In Ireland, which political party was founded 35 years before 1968?",
"answer": "Fine Gael",
"multiple_choice_options": {
"A": "Fine Gael",
"B": "Labour Party",
"C": "Fianna Fáil",
"D": "Sinn Féin"
},
"correct_answer_label": "A",
"domain": "History and Literature"
}
],
"multi_hop_question": "In the country of citizenship of the author of Candida, which political party was founded the same number of years before 1968 as the number of distinct 3-member committees that can be formed from a group of 7 candidates?",
"multi_hop_answer": "Fine Gael",
"multiple_choice_options": {
"A": "Fine Gael",
"B": "Labour Party",
"C": "Fianna Fáil",
"D": "Sinn Féin"
},
"correct_answer_label": "A",
"motif_type": "Chain"
}

## 联系方式
如有任何疑问，请联系**[e-mail]**。

## 引用
若您的研究或应用中使用了Omanic数据集，请引用以下文献：
bibtex
@article{gu2026omanic,
title={Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models},
author={Gu, Xiaojie and Tong, Sherry T and Feng, Aosong and Han, Sophia Simeng and Lu, Jinghui and Chen, Yingjian and Iwasawa, Yusuke and Matsuo, Yutaka and Park, Chanjun and Ying, Rex and Li, Irene},
journal={arXiv preprint arXiv:2603.16654},
year={2026}
}