ConflictQA

license: apache-2.0
task_categories:
- question-answering
language:
- en
pretty_name: conflictQA
size_categories:
- 10K<n<100K
configs:
- config_name: ConflictQA-popQA-chatgpt
data_files:
- split: test
path: "./conflictQA-popQA-chatgpt.json"
---

# Dataset Card for ConflcitQA
## Dataset Description
- **Repository:** https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict
- **Paper:** https://arxiv.org/abs/2305.13300
- **Point of Contact:** Point of Contact: [Jian Xie](mailto:[e-mail])
## Citation
If our paper or related resources prove valuable to your research, we kindly ask for citation. Please feel free to contact us with any inquiries.
```bib
@inproceedings{Xie2024KnowledgeConflict,
title={Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts},
author={Xie, Jian and Zhang, Kai and Chen, Jiangjie and Lou, Renze and Su, Yu},
booktitle={Proceedings of ICLR},
year={2024},
}
```

# ConflcitQA

We provide conflictQA-[based large language model], which utilizes large language models guided parametric memory.

```json
{"question": "What is George Rankin's occupation?", "popularity": 142, "ground_truth": ["politician", "political leader", "political figure", "polit.", "pol"], "memory_answer": "George Rankin's occupation is a professional photographer.", "parametric_memory": "As a professional photographer, George Rankin...", "counter_answer": "George Rankin's occupation is political figure.", "counter_memory": "George Rankin has been actively involved in politics for over a decade...", "parametric_memory_aligned_evidence": "George Rankin has a website showcasing his photography portfolio...", "counter_memory_aligned_evidence": "George Rankin Major General George James Rankin..."}
```

```python
#loading dataset
from datasets import load_dataset
# you can choose dataset "ConflictQA-popQA-[PLACEHOLDER]", and the [PLACEHOLDER] is in ["chatgpt","gpt4","palm2","llama2-7b","llama2-70b","qwen7b","vicuna7b","vicuna33b"].
dataset = load_dataset("osunlp/ConflictQA",'ConflictQA-popQA-chatgpt')
```

# Data Fields
- "question": The question in natural language
- "popularity": The monthly page views on Wikipedia for the given question
- "ground_truth": The factual answer to the question, which may include multiple possible answers
- "memory_answer": The answer provided by the LLM to the question
- "parametric_memory": The supportive evidence from LLM's parametric memory for the answer
- "counter_answer": The answer contradicting the "memory_answer"
- "counter_memory": The generation-based evidence supporting the counter_answer
- "parametric_memory_aligned_evidence": Additional evidence supporting the "memory_answer", which could be generated or derived from Wikipedia/human annotation
- "counter_memory_aligned_evidence": Additional evidence supporting the "counter_answer", either generated or sourced from Wikipedia/human annotation

许可证：Apache-2.0
任务类别：
- 问答（question-answering）
语言：
- 英语
展示名称：ConflictQA
规模类别：
- 10000 < 样本量 < 100000
配置项：
- 配置名称：ConflictQA-popQA-chatgpt
数据文件：
拆分方式：测试集（test）
路径："./conflictQA-popQA-chatgpt.json"

# ConflictQA数据集卡片
## 数据集描述
- **仓库地址**：https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict
- **论文地址**：https://arxiv.org/abs/2305.13300
- **联系人**：[Jian Xie](mailto:[e-mail])

## 引用说明
若您的研究工作得益于本文或相关资源，恳请引用本工作。如有任何疑问，欢迎随时与我们联系。
bib
@inproceedings{Xie2024KnowledgeConflict,
title={Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts},
author={Xie, Jian and Zhang, Kai and Chen, Jiangjie and Lou, Renze and Su, Yu},
booktitle={Proceedings of ICLR},
year={2024},
}

## ConflictQA数据集
我们提供了基于大语言模型（Large Language Model）的ConflictQA数据集变体，该数据集依托受参数记忆引导的大语言模型构建。

json
{"question": "What is George Rankin's occupation?", "popularity": 142, "ground_truth": ["politician", "political leader", "political figure", "polit.", "pol"], "memory_answer": "George Rankin's occupation is a professional photographer.", "parametric_memory": "As a professional photographer, George Rankin...", "counter_answer": "George Rankin's occupation is political figure.", "counter_memory": "George Rankin has been actively involved in politics for over a decade...", "parametric_memory_aligned_evidence": "George Rankin has a website showcasing his photography portfolio...", "counter_memory_aligned_evidence": "George Rankin Major General George James Rankin..."}

python
#loading dataset
from datasets import load_dataset
# you can choose dataset "ConflictQA-popQA-[PLACEHOLDER]", and the [PLACEHOLDER] is in ["chatgpt","gpt4","palm2","llama2-7b","llama2-70b","qwen7b","vicuna7b","vicuna33b"].
dataset = load_dataset("osunlp/ConflictQA",'ConflictQA-popQA-chatgpt')

## 数据字段说明
- "question"：自然语言形式的问题
- "popularity"：对应问题相关条目在维基百科的月度页面浏览量
- "ground_truth"：该问题的事实性标准答案，可包含多个可选答案
- "memory_answer"：大语言模型针对该问题生成的回答
- "parametric_memory"：大语言模型为其生成的memory_answer提供的支撑性证据
- "counter_answer"：与memory_answer相矛盾的对立答案
- "counter_memory"：用于支撑counter_answer的生成式证据
- "parametric_memory_aligned_evidence"：用于支撑memory_answer的额外证据，可由大语言模型生成或源自维基百科/人工标注
- "counter_memory_aligned_evidence"：用于支撑counter_answer的额外证据，可由大语言模型生成或源自维基百科/人工标注