OmniVideo-100K

<div align="center"> <h1>OmniVideo-100K</h1> [](https://yzlmhzz.github.io/OmniVideo-100K/) [](https://arxiv.org/abs/2606.14702) [](https://github.com/MiG-NJU/OmniVideo-100K) [](https://huggingface.co/datasets/MiG-NJU/OmniVideo-Test) </div> --- Official repository for **OmniVideo-100K**, an instruction-tuning dataset introduced in our paper: *"[OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains](https://arxiv.org/abs/2606.14702)"*. This repository includes: * `videos.tar.part_xx`: Raw video files. * `train_oe_70k.jsonl`: Original **Open-Ended (OE)** training samples. * `train_mcq_30k.jsonl`: Original **Multiple-Choice (MCQ)** training samples. * `train_oe_70k_formatted.jsonl`: Instruction-formatted OE samples (ready for fine-tuning). * `train_mcq_30k_formatted.jsonl`: Instruction-formatted MCQ samples (ready for fine-tuning). * `scripts.jsonl`: **[New]** Detailed multimodal structured scripts for all videos. **Note:** The corresponding human-verified test set is released separately as **[OmniVideo-Test](https://huggingface.co/datasets/MiG-NJU/OmniVideo-Test)**. --- ## 🔥 News * `2026.06.24` **Structured Scripts Released**: We have officially released `scripts.jsonl`. This file provides the structured text representations of the videos，which are used to generate the dataset's QA pairs. --- ## 🔨 Data Generation Pipeline  OmniVideo-100K is generated through an automated two-stage pipeline powered by Gemini series: 1. **Entity-Anchored Video Scripting**: Transforms raw videos into structured scripts comprising a video summary, a main entity list, and segment-wise audio-visual descriptions with timestamps. The global entity list ensures referential consistency across all segments and associates speech with visual entities. 2. **Clue-Guided QA Generation**: Instead of processing the entire dense text at once, this step prompts the model to first mine cross-segment, multimodal clues (evidence chains) from the script. QA pairs are subsequently generated based on these high-value clues, guaranteeing long-term temporal spans and deep cross-modal dependencies. --- ## 🩷 About the Dataset ### Dataset Statistics OmniVideo-100K contains **100K QA pairs** derived from **5,214 videos**, split into two formats: * **70K Open-Ended (OE) QA pairs** * **30K Multiple-Choice Questions (MCQ)**  ### Task Taxonomy We define **10 audio-visual QA tasks** organized into a three-level cognitive hierarchy: * 🔍 **Alignment** (Basic perception and synchronization) * Fine-Grained Perception, Scene Transformation Detection. * 🧠 **Understanding** (Cross-modal semantic understanding) * Context Understanding, Comparison, Sentiment Analysis, Event Sequence Ordering, Summarization. * ⚙️ **Reasoning** (Advanced logical inference) * Causal Reasoning, Future Prediction, Hypothetical Reasoning. ### Video Curation Videos are sourced from YouTube. To ensure high-quality data, we apply strict filtering criteria: * ✅ Resolution ≥ 480p * ✅ English speech * ✅ Sufficient visual dynamics & word density (ensuring rich audio-visual information) * ❌ Hard-coded subtitles removed (using OCR-based tools to prevent models from relying on on-screen text) ### Comparison with Existing Datasets  Compared to existing datasets, OmniVideo-100K uniquely provides **complex temporal tasks**, **evidence-based QA**, and **structured narratives** for open-domain videos, enhancing the cross-modal synergistic capabilities of Multimodal Large Language Models. --- ## 🚀 Performance To verify the effectiveness of OmniVideo-100K, we performed full-parameter fine-tuning on three open-source MLLMs: **VITA-1.5**, **Qwen2.5-Omni-7B**, and **Qwen3-Omni-30B-A3B-Instruct**. ### Performance on OmniVideo-Test Fine-tuning on OmniVideo-100K leads to substantial performance gains on the *OmniVideo-Test* benchmark.  ### Generalization on Existing Benchmarks Models trained on our dataset exhibit strong generalization transfer, achieving improved performance on audio-visual benchmarks such as Daily-Omni and FutureOmni.  --- ## 📋 Data Format (Raw Data Structure) We provide both raw `.jsonl` files and instruction-formatted `.jsonl` files. Below is the structure for the **raw data**. ### Common Fields Every sample shares these foundational metadata fields: ```json { "video_id": "...", // Unique video identifier "question_id": "...", // Unique QA identifier "search_tag": "...", // Retrieval tag (e.g., vlog, news) "language": "English", // Video language "duration": 103, // Video duration in seconds "metadata": {}, // Original video metadata "video_path": "...", // Local video path "resolution": "1280x720",// Video resolution "task": "...", // Major task category (e.g., causal_reasoning) "subtask": "..." // Subtype (only for Fine-Grained Perception and Context Understanding) } ``` ### General Tasks Open-Ended: ```json { "question": "Why is the Brunette Girl happy with her blind box?", "answer": "She is holding a bottle instead of a can.", "analysis": { "connections": "...", // Describes the evidence chains "designated_segments": "..." // Supporting evidence timestamps for cross-segment tasks } } ``` Multiple-Choice: ```json { "question": "Why is the Brunette Girl happy with her blind box?", "options": [ "She found a rare item.", "She is holding a bottle instead of a can.", "The blind box was on sale.", "She received two figures instead of one." ], "answer": "B", "analysis": { "connections": "...", "designated_segments": "..." } } ``` ### Special Task: Event Sequence Ordering Open-Ended: ```json { "question": "What is the correct chronological sequence for the following events?", "events": [ // Unordered events "Event A description...", "Event B description...", "Event C description..." ], "answer": ["B", "C", "A"] // Correct chronological order mapped to the list indices } ``` Multiple-Choice: *(To maintain data diversity, the raw data provides both indexed `(1) (2) (3)` and textual representations of the event sequences.)* ```json { "events": [ "Event A description...", "Event B description...", "Event C description..." ], // --- Indexed Style --- "question_indexed": "What is the correct chronological sequence for the following events?\n(1) Event A\n(2) Event B\n(3) Event C", "options_indexed": [ "(1) → (2) → (3)", "(3) → (2) → (1)", "(2) → (1) → (3)", "(2) → (3) → (1)" ], // --- Textual Style --- "question_textual": "What is the correct chronological sequence for the events?", "options_textual": [ "<Event A> → <Event B> → <Event C>", "<Event C> → <Event B> → <Event A>", "<Event B> → <Event A> → <Event C>", "<Event B> → <Event C> → <Event A>" ], "answer": "D" } ``` --- ## 🤖 Instruction-Formatted Data For direct supervised fine-tuning, we provide `_formatted.jsonl` files where inputs and outputs are pre-assembled. * `"question"`: The final constructed prompt fed to the model. * `"answer"`: The expected target response. ### Formatting for General Tasks For Open-Ended (OE): ```text // Input <question> // Output <answer> ``` For Multiple-Choice, we randomly append one of several instruction prompts (either before or after the options): ```text // Input <question> <prompt_prefix> A. ... B. ... C. ... D. ... <prompt_suffix> // Output <answer> // e.g., "B" ``` <details> <summary><b>Click to view the randomized MCQ instruction prompts:</b></summary> We randomly select one pair of `[prefix, suffix]` from the following list: 1. `["\nSelect from the following choices.", ""]` 2. `["\nChoose between the following options.", ""]` 3. `["\nAnswer with the option's letter from the given choices directly.", ""]` 4. `["", "\nPlease select the correct answer from the options above."]` 5. `["", "\nAnswer with the option's letter from the given choices directly."]` 6. `["", "\nAnswer with the option's letter directly (e.g., A, B, C, or D)."]` 7. `["", "\nAnswer with the option's letter (A, B, C, or D) from the given choices directly."]` 8. `["", "\nRespond with only the letter (A, B, C, or D) of the correct option."]` </details> ### Formatting for Event Sequence Ordering For Open-Ended: ```text // Input <question> (1) ... (2) ... (3) ... Please directly answer with the correct order of all events' indices, separated by commas. // Output 2,3,1 ``` For Multiple-Choice: During formatting, we randomly alternate between the **Indexed Style** and the **Textual Style** provided in the raw data to maximize model adaptability. --- ## 📄 Structured Scripts Each line in `scripts.jsonl` represents a complete structural breakdown of a single video. Below is the JSON structure showing the available fields: ```json { "video_id": "", "search_tag": "", "duration": "", "metadata": "", "main_entities": [ // Global list of entities { "entity": "<string>", // Unique identifier "description": "<string>", // Detailed description "label": "<string>" // Category (people/object/animal) } ], "non_speech": [ // Music and sounds { "start_time": "<string>", // MM:SS "end_time": "<string>", "sound": "<string>" } ], "transcribe": [ // Raw text transcription { "start_time": "<string>", "end_time": "<string>", "text": "<string>" } ], "label_speaker": [ // Speaker-attributed transcription { "start_time": "<string>", "end_time": "<string>", "text": "<string>", "speaker": "<string>" // Maps directly to elements in 'main_entities' } ], "video_summary": "<string>", // High-level narrative summary "segments": [ // Temporal chunks { "start_time": "<string>", "end_time": "<string>", "transcription": [ // Dialogue occurring within this specific chunk { "start_time": "<string>", "end_time": "<string>", "text": "<string>", "speaker": "<string>" } ], "visual": [ // Temporal visual descriptions within the chunk { "start_time": "<string>", "end_time": "<string>", "text": "<string>" // Dense description } ], "non_speech": [ // Sound events occurring within this chunk { "start_time": "<string>", "end_time": "<string>", "sound": "<string>" } ] } ] } ``` --- ## 📑 Citation If you find this dataset or pipeline useful in your research, please cite our paper: ```bibtex @article{cai2026omnivideo100k, title={OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains}, author={Cai, Xinyue and Fu, Chaoyou and Zhang, Yi-Fan and He, Ran and Shan, Caifeng}, journal={arXiv preprint arXiv:2606.14702}, year={2026} } ```