DenseStep2M

# DenseStep2M Dataset This repository contains the official dataset for the paper **DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation**. Paper: https://arxiv.org/abs/2604.26565 This repository provides datasets for instructional video analysis tasks, focusing on dense temporal grounding and procedural semantics. ## 📢 Data Source & ID Mapping All videos in this repository are sourced from the **[HowTo100M Dataset](https://www.di.ens.fr/willow/research/howto100m/)**. * The `<video_id>` used in all JSON annotation files corresponds directly to the original **YouTube Video ID**. * The video files are provided as compressed shard archives under `data/videos_shards/`. * The annotation files are provided under `data/annotations/`. --- ## Repository Structure The dataset is organized as follows: ```text DenseStep2M/ └── data/ ├── annotations/ │ ├── DenseStep2M.json │ ├── DenseStep2M_attr.json │ ├── DenseStep2M_title.json │ └── DenseCaption100.json └── videos_shards/ ├── DenseStep2M_videos_manifest.tsv ├── DenseStep2M_videos_0000.tar.gz ├── DenseStep2M_videos_0001.tar.gz ├── DenseStep2M_videos_0002.tar.gz ├── ... ├── DenseStep2M_videos_0026.tar.gz └── DenseStep2M_videos_sha256sum.txt ``` --- ## 1. Annotation Files All annotation files are located in: ```text data/annotations/ ``` ### `DenseStep2M.json` A large-scale dataset containing **1,885,504** detailed, temporally grounded instructional steps annotated across **99,248** high-quality instructional videos, totaling **7,212 hours**. DenseStep2M is constructed through a training-free automated pipeline leveraging multimodal foundation models, including **Qwen2.5-VL-72B** and **DeepSeek-R1-671B**, to process noisy HowTo100M videos and produce dense procedural annotations with improved visual-textual alignment. #### Structure ```json { "<video_id>": [ { "step": "string (instructional step description)", "start": "float (start time in seconds)", "end": "float (end time in seconds)" } ] } ``` --- ### `DenseStep2M_attr.json` This file records global statistics of DenseStep2M. ```json { "video_num": 99248, "step_num": 1885504, "word_per_step": 10.465, "step_per_video": 18.998 } ``` --- ### `DenseStep2M_title.json` This file contains high-quality titles generated for each video. The titles are produced by **Qwen2.5-72B** based on ASR transcripts generated by **WhisperX**. Example: ```json { "d0N6xOQCP6c": "How to Add a Ribbed Neckline to a Sweater", "BM0Bq7EBo4A": "How to Apply the Color Meshing Technique with Wool-E" } ``` --- ### `DenseCaption100.json` A manually curated benchmark containing **100** high-quality instructional videos spanning 12 categories, including cooking, home repair, crafts, and other procedural domains. DenseCaption100 is designed for evaluating dense video captioning and procedural step grounding. Each video is annotated with detailed temporal steps that incorporate both visual details and textual context from ASR transcripts. #### Structure ```json { "<video_id>": [ { "step": "string (instructional step description)", "start": "float (start time in seconds)", "end": "float (end time in seconds)" } ] } ``` --- ## 2. Video Shards The source videos are stored as compressed shard archives under: ```text data/videos_shards/ ``` The video shards are organized as: ```text DenseStep2M_videos_0000.tar.gz DenseStep2M_videos_0001.tar.gz DenseStep2M_videos_0002.tar.gz ... DenseStep2M_videos_0026.tar.gz ``` The file: ```text DenseStep2M_videos_manifest.tsv ``` records the mapping and metadata of the video shards, including the shard filename and the files contained in each shard. Each shard can be extracted independently. Example: ```bash mkdir -p DenseStep2M_videos tar -xzf DenseStep2M_videos_0000.tar.gz -C DenseStep2M_videos ``` To extract all shards: ```bash mkdir -p DenseStep2M_videos for f in DenseStep2M_videos_*.tar.gz; do echo "Extracting $f" tar -xzf "$f" -C DenseStep2M_videos done ``` --- ## Key Features * **Dense Temporal Annotations**: DenseStep2M contains 19.00 steps per video on average, compared with 10.6 in HowToStep. * **Fine-Grained Procedural Semantics**: Each step describes meaningful procedural actions, including operational purposes, durations, quantities, and intermediate states. * **Context-Aware Annotation**: The pipeline captures abstract procedural steps that require cross-modal inference from visual content, ASR transcripts, and video-level context. * **Improved Temporal Alignment**: DenseStep2M achieves a 46.25 R1-mIoU score in timestamp alignment, with 62.0% alignable and 56.1% well-aligned steps. * **Open-Domain Coverage**: The dataset covers diverse instructional domains, including cooking, home gardening, personal care, crafts, repair, and electronics assembly. --- ## Usage Example Load DenseStep2M annotations: ```python import json annotation_path = "data/annotations/DenseStep2M.json" with open(annotation_path, "r", encoding="utf-8") as f: data = json.load(f) video_id = next(iter(data)) steps = data[video_id] print("Video ID:", video_id) print("Number of steps:", len(steps)) print("First step:", steps[0]) ``` Expected format: ```python { "step": "instructional step description", "start": 12.3, "end": 25.7 } ``` --- ## Notes * The annotation IDs correspond to YouTube video IDs from HowTo100M. * The released video files are packaged into multiple shard archives for easier storage and download. * Users can extract only the required shards instead of unpacking the entire video collection. * Please refer to `DenseStep2M_videos_manifest.tsv` for shard-level organization. --- ## Communication If you have any questions, please contact us at: ```text [e-mail] ``` --- ## Citation If you find this dataset useful for your research, please cite our paper: ```bibtex @article{ge2026densestep2m, title={DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation}, author={Ge, Mingji and Chen, Qirui and Li, Zeqian and Xie, Weidi}, journal={arXiv preprint arXiv:2604.26565}, year={2026} } ```