HYU-NLP/MAGIC

# MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation

<p align="center">
<img src="https://img.shields.io/badge/MAGIC-v1.0-green">
<img src="https://img.shields.io/badge/created-September%202025-blue">
<img src="https://img.shields.io/badge/last%20commit-October-yellowgreen">
<a href="https://aclanthology.org/2025.findings-emnlp.466.pdf">
<img src="https://img.shields.io/badge/Paper-2412.12567-b31b1b.svg">
</a>
</p>

**Official Repository for "MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation". [[📄 Paper (ACL Anthology)](https://aclanthology.org/2025.findings-emnlp.466.pdf)]**
<br/>

<!-- <small>**Jungyeon Lee**, **Kangmin Lee**, and **Taeuk Kim**. *Accepted to EMNLP 2025 Findings.*</small> -->
<small>Jungyeon Lee, Kangmin Lee and Taeuk Kim. ***Accepted to EMNLP 2025 Findings.***</small>

---

## OverView
**MAGIC** is a large-scale benchmark designed to evaluate **knowledge conflict detection and localization** in **Retrieval-Augmented Generation (RAG)** systems. It focuses on *multi-hop reasoning* and *graph-structured contexts*, where **inter-context knowledge conflicts** emerge between retrieved passages.

## Repository Structure
```
.\
├── dataset
│ ├── multi-hop/
│ │ ├── 1-multi-hop_conflict.json/
│ │ ├── 2-multi-hop_conflict.json/
│ │ ├── 3-multi-hop_conflict.json/
│ │ ├── 4-multi-hop_conflict.json/
│ ├── single-hop/
│ │ ├── 1-single-hop_conflict.json/
│ │ ├── 2-single-hop_conflict.json/
│ │ ├── 3-single-hop_conflict.json/
│ │ ├── 4-single-hop_conflict.json/
```

### Dataset Structure
- `ID`: Unique identifier for each sample.
- `rel_id`: Relation ID corresponding to the target knowledge relation (e.g. `P150` from Wikidata)
- `subgraph`: A set of surrounding triplets retrieved via DFS traversal from the source knowledge graph around the `original_triplet`.
- `original_triplet`: Randomly sampled **target triplet** from the source graph — serves as the anchor for conflict formation.
- `perturb_triplet`: Modified triplet(s) intentionally constructed to introduce a **knowledge conflict** with the `original_triplet`.
- `context1`, `context2`: Textual representations of the `original_triplet` and `perturb_triplet`, respectively.

# MAGIC：面向检索增强生成中上下文间冲突的多跳图基准数据集

<p align="center">
<img src="https://img.shields.io/badge/MAGIC-v1.0-green" alt="MAGIC: v1.0 绿色">
<img src="https://img.shields.io/badge/created-September%202025-blue" alt="创建于：2025年9月 蓝色">
<img src="https://img.shields.io/badge/last%20commit-October-yellowgreen" alt="最后提交：10月 黄绿色">
<a href="https://aclanthology.org/2025.findings-emnlp.466.pdf">
<img src="https://img.shields.io/badge/Paper-2412.12567-b31b1b.svg" alt="论文：2412.12567 深红色">
</a>
</p>

**《MAGIC：面向检索增强生成中上下文间冲突的多跳图基准数据集》官方代码仓库。[[📄 论文（ACL文集）](https://aclanthology.org/2025.findings-emnlp.466.pdf)]**
<br/>

<small>Jungyeon Lee、Kangmin Lee 与 Taeuk Kim。***本文已被 EMNLP 2025 Findings 收录。***</small>

---

## 概述
**MAGIC** 是一款大规模基准测试集，旨在评估**检索增强生成（Retrieval-Augmented Generation, RAG）**系统中的**知识冲突检测与定位**任务。该基准聚焦于*多跳推理*与*图结构上下文*场景，其中**上下文间知识冲突**会在检索得到的多篇文本段落之间自然产生。

## 仓库结构

.
├── dataset
│ ├── multi-hop/
│ │ ├── 1-multi-hop_conflict.json/
│ │ ├── 2-multi-hop_conflict.json/
│ │ ├── 3-multi-hop_conflict.json/
│ │ ├── 4-multi-hop_conflict.json/
│ ├── single-hop/
│ │ ├── 1-single-hop_conflict.json/
│ │ ├── 2-single-hop_conflict.json/
│ │ ├── 3-single-hop_conflict.json/
│ │ ├── 4-single-hop_conflict.json/

### 数据集样本结构
- `ID`：每个样本的唯一标识符。
- `rel_id`：对应目标知识关系的关系ID（例如维基数据中的`P150`）
- `subgraph`：通过深度优先搜索（Depth-First Search, DFS）遍历源知识图谱，从`original_triplet`周边检索得到的三元组集合。
- `original_triplet`：从源图谱中随机采样的**目标三元组**，作为冲突构建的锚点。
- `perturb_triplet`：为与`original_triplet`产生知识冲突而刻意构造的修改后三元组（或多组三元组）。
- `context1`、`context2`：分别为`original_triplet`与`perturb_triplet`的文本表征形式。