Taming-Hallucinations

# Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation **CVPR 2026 Findings** [Project Page](https://amap-ml.github.io/Taming-Hallucinations/) | [Paper](https://arxiv.org/pdf/2512.24271) | [Code](https://github.com/AMAP-ML/Taming-Hallucinations) ## Dataset Summary This repository hosts **DualityVidQA**, the large-scale paired video–QA dataset introduced in *Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation*. Taming Hallucinations introduces **DualityForge**, a controllable diffusion-based framework that turns real videos into counterfactual ones, automatically generating paired (real / counterfactual) videos together with their question–answer data for contrastive training. Built on top of DualityVidQA and the proposed **DNA-Train** SFT–RL regime with ℓ1-normalized advantages, our approach reduces hallucinations in multimodal LLMs by **24%** and shows strong generalization across benchmarks. ## Repository Structure ``` . ├── part_001.zip ... part_008.zip # Video data (real & counterfactual videos), split into 8 parts ├── sft/ │ ├── real.json # SFT instruction data for real videos │ └── fake.json # SFT instruction data for counterfactual videos ├── RL/ │ └── RL_data.jsonl # Data for the RL stage of DNA-Train ├── benchmark/ │ └── benchmark.csv # Evaluation benchmark (QA over real & counterfactual videos) └── benchmark.zip # Videos referenced by the benchmark ``` ### Benchmark schema (`benchmark/benchmark.csv`) | Column | Description | | ---------- | ------------------------------------------------------------------ | | `id` | Unique sample id | | `type` | `real` or `fake` (counterfactual) video | | `video` | Relative path to the video file | | `question` | The question about the video | | `answer` | Ground-truth answer | ## Usage The videos are packed as `part_001.zip` … `part_008.zip`. After downloading, unzip all parts to reconstruct the video directory, then load the corresponding annotations from `sft/`, `RL/`, or `benchmark/`. ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="GD-ML/Taming-Hallucinations", repo_type="dataset", local_dir="Taming-Hallucinations", ) ``` ## Intended Uses - Training and evaluating multimodal LLMs for video understanding. - Research on reducing hallucinations via counterfactual / contrastive learning. - Video question answering and video-text-to-text tasks. ## Citation If you find this dataset useful, please consider citing: ```bibtex @article{huang2025taming, title={Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation}, author={Huang, Zhe and Wen, Hao and Hao, Aiming and Song, Bingze and Wu, Meiqi and Wu, Jiahong and Chu, Xiangxiang and Lu, Sheng and Wang, Haoqian}, journal={arXiv preprint arXiv:2512.24271}, year={2025} } ``` ## License This dataset is released under the [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) license.