VSI-33K

# VSI-33K <div align="center"> <a href="https://huggingface.co/datasets/nyu-visionx/VSI-590K"><img src="https://img.shields.io/badge/Source-VSI--590K-yellow?style=for-the-badge&logo=huggingface" alt="VSI-590K"></a> &nbsp; <a href="https://arxiv.org/abs/2511.04670"><img src="https://img.shields.io/badge/arXiv-2511.04670-b31b1b?style=for-the-badge&logo=arxiv" alt="Cambrian-S paper"></a> &nbsp; <a href="https://choosealicense.com/licenses/apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge" alt="License"></a> </div> **VSI-33K** is a compact, task-balanced video spatial-reasoning dataset derived from the **ScanNet branch of [VSI-590K](https://huggingface.co/datasets/nyu-visionx/VSI-590K)**. It contains 33,000 English video question-answer pairs grounded in 1,201 indoor ScanNet scene videos. The dataset is intended as a small, reproducible experimental subset of VSI-590K. It preserves ten spatial task categories while reducing the storage, training time, and evaluation cost required for controlled studies. VSI-33K is particularly suitable for rapid model-architecture validation, multimodal fusion ablations, spatial-instruction data mixtures, and iterative AutoResearch workflows. VSI-33K is a derived dataset. The original videos, spatial annotations, question templates, and answers come from VSI-590K and its ScanNet source. This release contributes a deterministic balanced split, a unified training schema, scene-disjoint validation, verified media packaging, and storage-efficient video encoding. ## Dataset Summary | Property | Value | |---|---:| | Total QA pairs | 33,000 | | Train QA pairs | 30,000 | | Validation QA pairs | 3,000 | | Spatial task categories | 10 | | QA per task in Train | 3,000 | | QA per task in Validation | 300 | | Unique ScanNet videos/scenes | 1,201 | | Train scenes | 901 | | Validation scenes | 300 | | Train/Validation scene overlap | 0 | | Prompt language | English | | Video format | MP4, H.264, yuv420p | | Resolution and frame rate | 640 x 480, 24 FPS | | Total video frames | 1,937,879 | | Packaged video bytes | 8,017,490,555 (7.47 GiB) | ## Relationship to VSI-590K [VSI-590K](https://huggingface.co/datasets/nyu-visionx/VSI-590K) is a large-scale spatial instruction-tuning dataset containing 590,667 QA pairs from ten data sources. VSI-33K selects only its ScanNet video data. No questions or answers were newly generated for this release. The original ScanNet portion contains 92,145 QA records over 1,201 scene videos. VSI-33K reorganizes that source into a smaller protocol with: - exactly ten spatial tasks and equal QA counts per task; - a deterministic split with random seed 42; - 901 training scenes and 300 validation scenes; - no scene or video shared between Train and Validation; - unique Validation questions; - Qwen-style multimodal messages plus explicit question, choices, and answer fields. The packaged videos were re-encoded from the VSI-590K ScanNet archive using H.264 CRF 24 at their original 640 x 480 resolution and 24 FPS. Audio was removed because it is not used by the spatial QA tasks. Every video was checked by decoding its first, middle, and final frames with Decord. ## Spatial Tasks VSI-33K contains ten task categories. Each task contributes 3,000 Train and 300 Validation records. | Task category | What it measures | Train | Validation | |---|---|---:|---:| | `absolute_count` | Number of instances of an object category in a room | 3,000 | 300 | | `absolute_direction_object` | Rotation or absolute direction required to face an object | 3,000 | 300 | | `absolute_distance_object` | Metric distance between two objects | 3,000 | 300 | | `absolute_size_object` | Metric length or size of an object | 3,000 | 300 | | `absolute_size_room` | Estimated room area or dimensions | 3,000 | 300 | | `appearance_order` | Temporal order in which object categories first appear | 3,000 | 300 | | `relative_count` | More, fewer, or equal object counts | 3,000 | 300 | | `relative_direction_object` | Left, right, or behind relationships between objects | 3,000 | 300 | | `relative_distance_object` | Which object is nearer or farther from a reference | 3,000 | 300 | | `relative_size_object` | Which object is larger along its longest side | 3,000 | 300 | ### Balancing and Resampling The original ScanNet task distribution is strongly imbalanced. Validation always contains 300 unique questions per task. Train uses deterministic resampling only when the remaining scene-disjoint source questions are insufficient to reach 3,000 records. | Task category | Original ScanNet QA | Unique Train QA | Resampled Train rows | Unique Validation QA | |---|---:|---:|---:|---:| | `absolute_count` | 2,934 | 2,523 | 477 | 300 | | `absolute_direction_object` | 8,526 | 3,000 | 0 | 300 | | `absolute_distance_object` | 8,505 | 3,000 | 0 | 300 | | `absolute_size_object` | 4,546 | 3,000 | 0 | 300 | | `absolute_size_room` | 1,201 | 901 | 2,099 | 300 | | `appearance_order` | 3,819 | 3,000 | 0 | 300 | | `relative_count` | 908 | 606 | 2,394 | 300 | | `relative_direction_object` | 33,888 | 3,000 | 0 | 300 | | `relative_distance_object` | 17,818 | 3,000 | 0 | 300 | | `relative_size_object` | 10,000 | 3,000 | 0 | 300 | Train therefore contains 25,030 unique source questions and 4,970 deterministic replicas. Replica IDs and provenance are explicit in `metadata`; replicas are cycled only after every available source question in that task has been used. This balancing is useful for controlled task-level comparisons, but it should not be interpreted as the natural ScanNet question distribution. ## Construction Protocol VSI-33K was prepared in four stages: 1. **Audit:** inspect all 590,667 VSI-590K annotations and isolate the 92,145 ScanNet video QA records. 2. **Split:** assign complete ScanNet scenes to 901 Train scenes and 300 Validation scenes with seed 42, preventing scene and video leakage. 3. **Balance:** select 3,000 Train and 300 Validation records for each of ten tasks, using deterministic Train-only resampling for scarce categories. 4. **Package:** convert annotations to a unified multimodal JSONL schema, compress the 1,201 referenced videos, and validate counts, hashes, media paths, task balance, split isolation, and video decoding. The exact annotation hashes, media statistics, and encoding parameters are recorded in `manifest.json`. ## Data Format ```text VSI-33K/ ├── train.jsonl ├── validation.jsonl ├── manifest.json └── videos/ └── scannet/ ├── scene0000_00.mp4 ├── scene0000_01.mp4 └── ... ``` The MP4 files are stored individually and do not require extraction. Media paths in the JSONL files are relative to `videos/`. Each JSONL line contains one QA training record: | Field | Description | |---|---| | `id` | Stable public sample ID, including replica provenance when applicable | | `split` | `train` or `validation` | | `source` | Upstream annotation source, `vsi590k` | | `dataset` | Upstream visual dataset, `scannet` | | `scene_id` | ScanNet scene identifier used for split isolation | | `task_category` | One of the ten normalized spatial tasks | | `question_type` | Original VSI-590K question type | | `media` | Media type and path relative to `videos/` | | `question` | Original English question | | `choices` | Structured answer choices; empty for free-form numeric tasks | | `prompt` | Model-ready question and response instruction | | `ground_truth` | Raw answer and, when applicable, normalized choice label/text | | `messages` | Qwen-style user video/text and assistant answer messages | | `metadata` | Original source line, replica index, and resampling flag | Minimal loading example: ```python import json from pathlib import Path dataset_root = Path("VSI-33K") with (dataset_root / "train.jsonl").open(encoding="utf-8") as handle: sample = json.loads(next(handle)) video_path = dataset_root / "videos" / sample["media"]["path"] question = sample["prompt"] answer = sample["ground_truth"]["raw"] ``` ## Intended Uses VSI-33K is designed for small-scale, controlled spatial-intelligence research. It may be useful for: - rapidly validating multimodal model architectures before committing to a full-scale training run; - controlled comparisons of visual-geometry fusion, residual addition, gating, cross-attention, projection layers, frame sampling, and token alignment strategies; - low-cost pipeline debugging for video loading, multimodal prompting, forward/backward passes, checkpointing, and evaluation; - adding balanced spatial supervision to mixtures with general vision-language instruction data; - studying data efficiency, task interference, and per-task transfer in spatial reasoning; - supporting AutoResearch-style agents that propose, train, evaluate, and iteratively improve model or fusion architectures; - fast architecture search and agent self-evolution experiments where a compact and deterministic training/validation loop is required. For AutoResearch and architecture discovery, Train should be used for parameter updates and the visible Validation split for model selection. Repeated optimization against the visible split can overfit it, so final claims should also be evaluated on an independent held-out benchmark such as ReVSI. VSI-33K is not intended to replace the full diversity or scale of VSI-590K. It is best treated as a fast experimental substrate or one component of a larger multimodal training mixture. ## Limitations - All videos come from ScanNet indoor scenes; outdoor, egocentric, synthetic, and web-video domains present in VSI-590K are not represented. - The task-balanced distribution is deliberately constructed and does not reflect the natural frequency of questions or objects in ScanNet. - Three scarce Train tasks contain deterministic replicas. QA rows that share a source question are not statistically independent. - Many QA records share the same scene video, so the number of QA pairs should not be interpreted as the number of independent visual examples. - Validation is visible and is not a hidden test set. - The videos are lossy H.264 re-encodes of the upstream VSI-590K ScanNet media. - Questions and answers inherit the templating, annotation assumptions, units, and possible errors of VSI-590K and the underlying ScanNet annotations. - The dataset is English-only and focuses on ten predefined spatial tasks. ## License and Attribution VSI-33K is distributed under the Apache License 2.0, following the [VSI-590K dataset card](https://huggingface.co/datasets/nyu-visionx/VSI-590K). This is a derivative release; users should also review and comply with the terms and attribution requirements of VSI-590K and ScanNet. ## Citation If you use VSI-33K, please cite the original VSI-590K / Cambrian-S work: ```bibtex @article{yang2025cambrian, title={Cambrian-S: Towards Spatial Supersensing in Video}, author={Yang, Shusheng and Yang, Jihan and Huang, Pinzhi and Brown, Ellis and Yang, Zihao and Yu, Yue and Tong, Shengbang and Zheng, Zihan and Xu, Yifan and Wang, Muhan and Lu, Danhao and Fergus, Rob and LeCun, Yann and Fei-Fei, Li and Xie, Saining}, journal={arXiv preprint arXiv:2511.04670}, year={2025} } ``` ## Acknowledgements We thank the authors of [VSI-590K](https://huggingface.co/datasets/nyu-visionx/VSI-590K) and [Cambrian-S](https://arxiv.org/abs/2511.04670), and the creators of ScanNet. VSI-33K would not be possible without their released data, annotations, and spatial-reasoning task design.