yifangsm/VideoHallucer

---
license: mit
task_categories:
- question-answering
language:
- en
size_categories:
- 1K<n<10K
configs:
- config_name: external_factual
data_files:
- split: test
path: external_factual/external_factual.json
- config_name: external_nonfactual
data_files:
- split: test
path: external_nonfactual/external_nonfactual.json
- config_name: fact_detect
data_files:
- split: test
path:
- fact_detect/fact_detect.json
- fact_detect/fact_detect_yn.json
- config_name: object_relation
data_files:
- split: test
path: object_relation/object_relation.json
- config_name: semantic_detail
data_files:
- split: test
path: semantic_detail/semantic_detail.json
- config_name: temporal
data_files:
- split: test
path: temporal/temporal.json
---
# VideoHallucer

**Paper:** https://huggingface.co/papers/2406.16338

## Dataset Description

- **Repository:** [VideoHallucer](https://github.com/patrick-tssn/VideoHallucer)
- **Paper:** [2406.16338](https://arxiv.org/abs/2406.16338)
- **Point of Contact:** mailto:[Yuxuan Wang]([e-mail])


This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non-factual hallucinations. We adopt an adversarial binary VideoQA method for comprehensive evaluation, where pairs of basic and hallucinated questions are crafted strategically.

## Data Statistics
| | Object-Relation Hallucination | Temporal Hallucination | Semantic Detail Hallucination | External Factual Hallucination | External Nonfactual Hallucination |
| ---- | ---- | ---- | ---- | ---- | ---- |
|Questions | 400 | 400 | 400 | 400 | 400 |
|Videos | 183 | 165 | 400| 200 | 200 |

## Evaluation

We provide [VideoHallucerKit](https://github.com/patrick-tssn/VideoHallucer?tab=readme-ov-file#videohallucerkit) for evaluation

## Leaderboard

See our [page](https://videohallucer.github.io/)