AudioVisual-Caption/ASID-1M

--- license: cc-by-2.0 language: - en pretty_name: ASID-1M tags: - caption - audiovisual - instruction-tuning - attribute-structured - quality-verified - video-understanding task_categories: - image-text-to-text configs: - config_name: all_attributes data_files: - split: train path: - annotations/0_30_s_youtube_v0_1/train/all_attributes_0_30_s_youtube_v0_1.jsonl - annotations/30_60_s_youtube_v0_1/train/all_attributes_30_60_s_youtube_v0_1.jsonl - annotations/1_2_m_youtube_v0_1/train/all_attributes_1_2_m_youtube_v0_1.jsonl - annotations/finevideo/train/all_attributes_finevideo.jsonl - config_name: single_attribute data_files: - split: train path: - annotations/0_30_s_youtube_v0_1/train/single_attribute_0_30_s_youtube_v0_1.jsonl - annotations/30_60_s_youtube_v0_1/train/single_attribute_30_60_s_youtube_v0_1.jsonl - annotations/1_2_m_youtube_v0_1/train/single_attribute_1_2_m_youtube_v0_1.jsonl - annotations/finevideo/train/single_attribute_finevideo.jsonl --- # ASID-1M: Attribute-Structured and Quality-Verified Audiovisual Instructions [[🏠 Homepage](https://asid-caption.github.io/)] [[📖 Arxiv Paper](https://arxiv.org/pdf/2602.13013)] [[🤗 Models & Datasets](https://huggingface.co/AudioVisual-Caption)] [[💻 Code](https://github.com/HVision-NKU/ASID-Caption)] ## Introduction We introduce **ASID-1M**, a large-scale audiovisual instruction dataset built to support **universal video understanding** with **fine-grained, controllable supervision**. Most existing video-instruction data represents complex audiovisual content as a single, monolithic caption. This often leads to incomplete coverage (missing audio, camera, emotions), weak controllability, and inconsistent temporal grounding. ASID-1M addresses these limitations by organizing supervision into **structured attributes** and applying a scalable **quality verification and targeted refinement** process to improve semantic completeness and temporal consistency. ASID-1M is designed for training and evaluating video MLLMs that must jointly reason about **vision, audio, and time**, while following **attribute-specific instructions**. ## Key Features - **Attribute-Structured Supervision:** Each sample provides fine-grained annotations over 8 complementary attributes, enabling controllable instruction tuning and attribute-level evaluation. - **Two Training Views:** We provide **single_attribute** (progressive / controllable learning) and **all_attributes** (full-coverage captioning) formats. - **Quality-Verified Annotations:** A multi-stage pipeline performs generation, ensembling, attribute-wise validation, and targeted refinement to improve correctness and reduce missing content. - **Large-Scale Video Storage:** Videos are released as **tar.gz shards** for efficient distribution and scalable loading. ## Attribute Design ASID-1M provides supervision over 8 attributes: - **Scene** - **Characters** - **Objects** - **Actions** - **Narrative elements** (often timestamped / temporally grounded) - **Speech** (grounded to ASR/alignment when applicable) - **Camera** - **Emotions** ## Repository Structure Top-level folders in this repo: - 0_30_s_youtube_v0_1/ (video shards) - 30_60_s_youtube_v0_1/ (video shards) - 1_2_m_youtube_v0_1/ (video shards) - finevideo/ (video shards) - annotations/ (JSONL annotations) ## Video Storage Videos are stored as .tar.gz shards inside their corresponding subset folders, for example: - 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_1.tar.gz - 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_2.tar.gz - finevideo/finevideo_videos_1.tar.gz Each shard contains multiple video files. The `videos` field in JSONL entries is intended to map to the video files stored in these shards. ## Annotation Structure Annotations are stored under: - annotations/<subset>/train/ Each subset provides: - all_attributes_*.jsonl - single_attribute_*.jsonl ## Data Statistics Valid JSONL line counts: | Subset | all_attributes | single_attribute | | --- | ---: | ---: | | 0-30s YouTube | 68081 | 68035 | | 30-60s YouTube | 18760 | 18760 | | 1-2m YouTube | 17606 | 17606 | | FineVideo | 16094 | 16094 | Total scanned JSONL files: 12 Total valid lines: 361531 ## Data Format Each JSONL line is a dictionary: ~~~json { "videos": ["path/to/video.mp4"], "messages": [ { "role": "system", "content": [{"type": "text", "text": "System prompt..."}] }, { "role": "user", "content": [{"type": "text", "text": "Instruction..."}] }, { "role": "assistant", "content": [{"type": "text", "text": "Answer..."}] } ], "meta": { "video_id": "xxx", "attr_keys": ["Scene", "Actions", "Speech"] } } ~~~ Field description: - videos: list of video paths (usually length 1) - messages: chat-style SFT format - meta: optional metadata (video id and supervised attribute keys) ## Usage ### 1) Load annotations ~~~python from datasets import load_dataset repo = "AudioVisual-Caption/ASID-1M" # all_attributes data_files_all = { "train": [ "annotations/0_30_s_youtube_v0_1/train/all_attributes_0_30_s_youtube_v0_1.jsonl", "annotations/30_60_s_youtube_v0_1/train/all_attributes_30_60_s_youtube_v0_1.jsonl", "annotations/1_2_m_youtube_v0_1/train/all_attributes_1_2_m_youtube_v0_1.jsonl", "annotations/finevideo/train/all_attributes_finevideo.jsonl", ] } ds_all = load_dataset(repo, data_files=data_files_all, split="train") # single_attribute data_files_single = { "train": [ "annotations/0_30_s_youtube_v0_1/train/single_attribute_0_30_s_youtube_v0_1.jsonl", "annotations/30_60_s_youtube_v0_1/train/single_attribute_30_60_s_youtube_v0_1.jsonl", "annotations/1_2_m_youtube_v0_1/train/single_attribute_1_2_m_youtube_v0_1.jsonl", "annotations/finevideo/train/single_attribute_finevideo.jsonl", ] } ds_single = load_dataset(repo, data_files=data_files_single, split="train") print(ds_all[0].keys()) print(ds_all[0]["videos"]) ~~~ ### 2) Download the full repository (videos + annotations) ~~~python from huggingface_hub import snapshot_download snapshot_download( repo_id="AudioVisual-Caption/ASID-1M", repo_type="dataset", local_dir="ASID-1M", local_dir_use_symlinks=False ) ~~~ ## Working with Video Shards Extract a tar shard: ~~~bash mkdir -p extracted_videos/0_30 tar -xzf 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_1.tar.gz -C extracted_videos/0_30 ~~~ Or stream from tar shards (WebDataset-style) if your dataloader supports it. ## Training Notes Recommended strategy: - Stage 1: train with single_attribute (progressive / controllable learning) - Stage 2: continue with all_attributes (full-coverage captioning) - Optional: mixed sampling with attribute balancing via meta.attr_keys ## Quality Control ASID-1M is curated through a multi-stage pipeline including generation, ensembling, attribute-wise validation, missing-content detection, targeted refinement, and temporal consistency checks. Minor noise may remain. Users are encouraged to apply task-specific filtering as needed. ## License CC-BY-2.0 ## Citation If you use our dataset in your research, please cite our paper: ~~~bibtex @article{li2026asid, title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions}, author={Li, Yunheng and Zhang, Hengrui and Guo, Meng-Hao and Gao, Wenzhao and Jia, Shaoyong and Jiao, Shaohui and Hou, Qibin and Cheng, Ming-Ming}, journal={arXiv preprint arXiv:2602.13013}, year={2026} } ~~~ ## Contact Please open a Discussion on the dataset page for format questions, viewer issues, schema conflicts, or takedown requests.> 许可协议：知识共享署名2.0（CC-BY-2.0） > 语言：英语 > 数据集名称：ASID-1M > 标签：字幕生成、视听任务、指令微调、属性结构化、质量验证、视频理解 > 任务类别：图像文本转文本 > 配置项： > 1. 配置名称：all_attributes > 数据文件： > - 拆分方式：训练集 > 路径： > - annotations/0_30_s_youtube_v0_1/train/all_attributes_0_30_s_youtube_v0_1.jsonl > - annotations/30_60_s_youtube_v0_1/train/all_attributes_30_60_s_youtube_v0_1.jsonl > - annotations/1_2_m_youtube_v0_1/train/all_attributes_1_2_m_youtube_v0_1.jsonl > - annotations/finevideo/train/all_attributes_finevideo.jsonl > 2. 配置名称：single_attribute > 数据文件： > - 拆分方式：训练集 > 路径： > - annotations/0_30_s_youtube_v0_1/train/single_attribute_0_30_s_youtube_v0_1.jsonl > - annotations/30_60_s_youtube_v0_1/train/single_attribute_30_60_s_youtube_v0_1.jsonl > - annotations/1_2_m_youtube_v0_1/train/single_attribute_1_2_m_youtube_v0_1.jsonl > - annotations/finevideo/train/single_attribute_finevideo.jsonl # ASID-1M: 结构化属性与质量验证的视听指令数据集 [[🏠 主页](https://asid-caption.github.io/)] [[📖 Arxiv论文](https://arxiv.org/pdf/2602.13013)] [[🤗 模型与数据集](https://huggingface.co/AudioVisual-Caption)] [[💻 代码](https://github.com/HVision-NKU/ASID-Caption)] ## 概述 我们推出**ASID-1M**，这是一个大规模视听指令数据集，旨在通过细粒度、可受控的监督信号支持通用视频理解任务。 现有多数视频指令数据将复杂视听内容封装为单一整体式字幕，往往存在覆盖不全（遗漏音频、镜头、情绪等信息）、可控性弱、时序锚定不一致等问题。ASID-1M通过将监督信号组织为**结构化属性**，并采用可扩展的**质量验证与定向优化**流程，弥补了上述局限，提升了语义完整性与时序一致性。 ASID-1M专为训练与评估需要同时结合视觉、音频与时序信息，并遵循属性专属指令的视频多模态大模型（Video Multi-Modality Large Language Model, MLLM）而设计。 ## 核心特性 - **属性结构化监督**：每个样本针对8项互补属性提供细粒度标注，支持可控指令微调与属性级评估。 - **两种训练视角**：提供`single_attribute`（渐进式/可控学习）与`all_attributes`（全维度字幕生成）两种格式。 - **质量验证标注**：通过多阶段流程完成生成、集成、属性级验证与定向优化，提升标注正确性并减少内容遗漏。 - **大规模视频存储**：视频以**tar.gz分片**形式发布，便于高效分发与可扩展加载。 ## 属性设计 ASID-1M覆盖以下8项属性的监督信号： - **场景（Scene）** - **角色（Characters）** - **物体（Objects）** - **动作（Actions）** - **叙事元素（Narrative elements，通常带时间戳/时序锚定）** - **语音（Speech，适用时基于自动语音识别/对齐结果锚定）** - **镜头（Camera）** - **情绪（Emotions）** ## 仓库结构 本数据集仓库的顶层文件夹包括： - 0_30_s_youtube_v0_1/（视频分片） - 30_60_s_youtube_v0_1/（视频分片） - 1_2_m_youtube_v0_1/（视频分片） - finevideo/（视频分片） - annotations/（JSONL格式标注文件） ## 视频存储 视频存储于对应子集文件夹下的.tar.gz分片中，示例如下： - 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_1.tar.gz - 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_2.tar.gz - finevideo/finevideo_videos_1.tar.gz 每个分片包含多个视频文件。JSONL条目内的`videos`字段用于映射存储于这些分片中的视频文件。 ## 标注结构 标注文件存储于`annotations/<subset>/train/`路径下。 每个子集包含两类标注文件：`all_attributes_*.jsonl`与`single_attribute_*.jsonl`。 ## 数据统计 有效JSONL行数统计： | 子集 | all_attributes | single_attribute | | --- | ---: | ---: | | 0-30秒YouTube | 68081 | 68035 | | 30-60秒YouTube | 18760 | 18760 | | 1-2分钟YouTube | 17606 | 17606 | | FineVideo | 16094 | 16094 | 扫描得到的JSONL文件总数：12；有效总行数：361531。 ## 数据格式 每个JSONL行为一个字典： json { "videos": ["path/to/video.mp4"], "messages": [ { "role": "system", "content": [{"type": "text", "text": "System prompt..."}] }, { "role": "user", "content": [{"type": "text", "text": "Instruction..."}] }, { "role": "assistant", "content": [{"type": "text", "text": "Answer..."}] } ], "meta": { "video_id": "xxx", "attr_keys": ["Scene", "Actions", "Speech"] } } 字段说明： - `videos`：视频路径列表（通常长度为1） - `messages`：聊天式监督微调（SFT）格式数据 - `meta`：可选元数据（包含视频ID与受监督的属性键列表） ## 使用方法 ### 1) 加载标注文件 python from datasets import load_dataset repo = "AudioVisual-Caption/ASID-1M" # all_attributes data_files_all = { "train": [ "annotations/0_30_s_youtube_v0_1/train/all_attributes_0_30_s_youtube_v0_1.jsonl", "annotations/30_60_s_youtube_v0_1/train/all_attributes_30_60_s_youtube_v0_1.jsonl", "annotations/1_2_m_youtube_v0_1/train/all_attributes_1_2_m_youtube_v0_1.jsonl", "annotations/finevideo/train/all_attributes_finevideo.jsonl", ] } ds_all = load_dataset(repo, data_files=data_files_all, split="train") # single_attribute data_files_single = { "train": [ "annotations/0_30_s_youtube_v0_1/train/single_attribute_0_30_s_youtube_v0_1.jsonl", "annotations/30_60_s_youtube_v0_1/train/single_attribute_30_60_s_youtube_v0_1.jsonl", "annotations/1_2_m_youtube_v0_1/train/single_attribute_1_2_m_youtube_v0_1.jsonl", "annotations/finevideo/train/single_attribute_finevideo.jsonl", ] } ds_single = load_dataset(repo, data_files=data_files_single, split="train") print(ds_all[0].keys()) print(ds_all[0]["videos"]) ### 2) 下载完整数据集（含视频与标注） python from huggingface_hub import snapshot_download snapshot_download( repo_id="AudioVisual-Caption/ASID-1M", repo_type="dataset", local_dir="ASID-1M", local_dir_use_symlinks=False ) ## 处理视频分片 解压tar分片： bash mkdir -p extracted_videos/0_30 tar -xzf 0_30_s_youtube_v0_1/0_30_s_youtube_v0_1_videos_1.tar.gz -C extracted_videos/0_30 若你的数据加载器支持，也可以采用WebDataset风格直接从tar分片中流式读取数据。 ## 训练建议 推荐训练策略： - 阶段1：使用`single_attribute`格式进行训练（渐进式/可控学习） - 阶段2：继续使用`all_attributes`格式进行训练（全维度字幕生成） - 可选：通过`meta.attr_keys`进行属性平衡的混合采样 ## 质量管控 ASID-1M通过多阶段流程构建，涵盖生成、集成、属性级验证、内容遗漏检测、定向优化及时序一致性检查等环节。 仍可能存在少量噪声，建议用户根据具体任务需求自行过滤。 ## 许可协议 知识共享署名2.0（CC-BY-2.0） ## 引用 若您在研究中使用本数据集，请引用我们的论文： bibtex @article{li2026asid, title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions}, author={Li, Yunheng and Zhang, Hengrui and Guo, Meng-Hao and Gao, Wenzhao and Jia, Shaoyong and Jiao, Shaohui and Hou, Qibin and Cheng, Ming-Ming}, journal={arXiv preprint arXiv:2602.13013}, year={2026} } ## 联系方式 若您对数据集格式、查看问题、schema冲突或下架请求有疑问，请在数据集页面开启讨论。