MM-Hallu/MFC-Bench

---
license: apache-2.0
task_categories:
- image-classification
- visual-question-answering
language:
- en
tags:
- fact-checking
- multimodal
- manipulation-detection
- out-of-context
- veracity
- benchmark
pretty_name: MFC-Bench
size_categories:
- 10K<n<100K
configs:
- config_name: manipulation
data_files:
- split: train
path: manipulation/train-*
- config_name: ooc
data_files:
- split: train
path: ooc/train-*
- config_name: veracity
data_files:
- split: train
path: veracity/train-*
dataset_info:
- config_name: manipulation
features:
- name: id
dtype: string
- name: manipulate
dtype:
class_label:
names:
'0': authentic
'1': manipulated
- name: manipulation_type
dtype: string
- name: caption
dtype: string
- name: image
dtype: image
splits:
- name: train
num_examples: 31000
- config_name: ooc
features:
- name: image_id
dtype: int64
- name: id
dtype: int64
- name: caption
dtype: string
- name: matched
dtype: bool
- name: image
dtype: image
splits:
- name: train
num_examples: 2000
- config_name: veracity
features:
- name: evidence_id
dtype: string
- name: topic
dtype: int64
- name: document_id
dtype: string
- name: relevancy
dtype:
class_label:
names:
'0': not_relevant
'1': relevant
- name: claim
dtype: string
- name: image
dtype: image
splits:
- name: train
num_examples: 2000
---

# MFC-Bench: Multimodal Fact-Checking Benchmark

MFC-Bench is a comprehensive Multimodal Fact-Checking testbed designed to evaluate LVLMs in terms of identifying factual inconsistencies and counterfactual scenarios.

## Dataset Description

From the paper: **"MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models"**

MFC-Bench encompasses a wide range of visual and textual queries, organized into three binary classification tasks:

### 1. Manipulation Classification (`manipulation` config)
- **31,000 samples** with images
- Targets various alterations: face swapping, face attribute editing, background changing, image generation, entity replacement, and style transfer
- `manipulate`: 0 = authentic, 1 = manipulated
- `manipulation_type`: the specific type of manipulation applied

### 2. Out-of-Context Classification (`ooc` config)
- **2,000 samples** with images
- Focuses on identifying false connections between image and text that may both be individually true
- `matched`: whether the caption correctly matches the image

### 3. Veracity Classification (`veracity` config)
- **2,000 samples** with images
- Multimodal counterpart to classifying the veracity of textual claims given visual evidence
- `relevancy`: whether the claim is supported by the visual evidence
- `claim`: the textual claim to verify

## Citation

```bibtex
@misc{wang2024mfcbenchbenchmarkingmultimodalfactchecking,
title={MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models},
author={Shengkang Wang and Hongzhan Lin and Ziyang Luo and Zhen Ye and Guang Chen and Jing Ma},
year={2024},
eprint={2406.11288},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2406.11288},
}
```

## Source

- GitHub: [https://github.com/wskbest/MFC-Bench](https://github.com/wskbest/MFC-Bench)
- Original datasets: [manipulation-mfc-bench](https://huggingface.co/datasets/Anonymous-2024/manipulation-mfc-bench), [ooc-mfc-bench](https://huggingface.co/datasets/Anonymous-2024/ooc-mfc-bench), [veracity-mfc-bench](https://huggingface.co/datasets/Anonymous-2024/veracity-mfc-bench)