nyu-visionx/VSI-Train-10k
收藏资源简介：
--- license: apache-2.0 task_categories: - visual-question-answering language: - en tags: - Video - Text - egocentric - spatial-reasoning - training-data size_categories: - 1K<n<10K configs: - config_name: default data_files: - split: train path: vsi_train_10k.parquet --- # VSI-Train-10k <a href="http://arxiv.org/abs/2511.04655" target="_blank"><img alt="arXiv" src="https://img.shields.io/badge/cs.CV-arXiv:2511.04655-red?logo=arxiv" height="20" /></a> <a href="https://vision-x-nyu.github.io/test-set-training/" target="_blank"><img alt="Website" src="https://img.shields.io/badge/🌎_Web-test--set--training-blue.svg" height="20" /></a> <a href="https://hf.co/datasets/nyu-visionx/VSI-Bench" target="_blank"><img alt="HF" src="https://img.shields.io/badge/HF-VSI--Bench_(Debiased)-FED123.svg?style&logo=HuggingFace" height="20" /></a> <a href="https://github.com/vision-x-nyu/test-set-training" target="_blank"><img alt="GitHub Code" src="https://img.shields.io/badge/Code-vision--x--nyu%2Ftest--set--training-white?&logo=github&logoColor=white" /></a> **Authors:**   <a href="https://ellisbrown.github.io/" target="_blank">Ellis Brown</a>, <a href="https://jihanyang.github.io/" target="_blank">Jihan Yang</a>, <a href="https://github.com/vealocia" target="_blank">Shusheng Yang</a>, <a href="https://cs.nyu.edu/~fergus" target="_blank">Rob Fergus</a>, <a href="https://www.sainingxie.com/" target="_blank">Saining Xie</a> --- **VSI-Train-10k** is an in-distribution training dataset of 10,000 question-answer pairs for visual-spatial intelligence tasks from egocentric video. This dataset accompanies the [VSI-Bench](https://hf.co/datasets/nyu-visionx/VSI-Bench) test set and is introduced in our paper: [Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts](http://arxiv.org/abs/2511.04655). ## Dataset Description VSI-Train-10k comprises 10,000 question-answer pairs generated using the same procedural logic as VSI-Bench, sourced from the **training splits** of ScanNet, ScanNet++, and ARKitScenes datasets (non-overlapping with VSI-Bench test videos). The dataset was created to study how MLLMs exploit statistical shortcuts and biases from in-distribution data. **Key characteristics:** - 10,000 QA pairs across 7 rule-based question types (excludes route planning) - Generated using the same templates and logic as VSI-Bench - Maximum 20 questions per question type per scene for diversity - Sourced from training splits only (no overlap with VSI-Bench test data) ## Dataset Structure ### Fields | Field Name | Description | | :--------- | :---------- | | `video` | Path to video file (e.g., `scannet_videos_128f/train/scene0335_02_128f.mp4`) | | `conversations` | List of conversation turns with `from` (human/gpt) and `value` (message content) | | `type` | Question format: `mc` (multiple choice) or `oe` (open-ended) | | `question_type` | High-level task category (e.g., `relative_distance`, `object_counting`) | | `question_type_detail` | Detailed task subcategory | | `source` | Video source dataset: `scannet`, `arkitscenes`, or `scannetpp` | | `question` | Full question text including instructions and options | | `ground_truth` | Correct answer | VSI-Train-10k includes 7 question types: object counting, spatial relations (closer/farther), object appearance order, size estimation, and more. Route planning questions are excluded. ## Usage ### Loading the Dataset ```python from datasets import load_dataset # Load the training dataset vsi_train = load_dataset("nyu-visionx/VSI-Train-10k") # Access the data for example in vsi_train['train']: print(example['question']) print(example['ground_truth']) ``` ### Extracting Video Files The video files are compressed in `.tar.zst` format using [zstd](http://www.zstd.net/) (much faster than gzip). To extract all shards in parallel: ```bash # Install zstd if needed: sudo apt-get install zstd (Ubuntu/Debian) or brew install zstd (macOS) for shard in vsi_train_shard_*.tar.zst; do zstd -d "$shard" -c | tar -xf - & done; wait ``` ## Files - `vsi_train_10k.parquet`: Parquet file containing dataset annotations optimized for HuggingFace Datasets - `vsi_train_10k.jsonl`: Raw JSONL file with the same annotations - `vsi_train_shard_*.tar.zst`: Compressed video files (9 shards total) ## Generation Methodology VSI-Train-10k was generated following the VSI-Bench curation pipeline: 1. Object numbers, bounding boxes, and room sizes were extracted from the training splits of ScanNet, ScanNet++, and ARKitScenes 2. Question-answer pairs were generated using the same templates as VSI-Bench 3. We create `1430` question-answer pairs per question type, with a maximum of 20 questions per question type per scene 4. All questions maintain in-distribution consistency with VSI-Bench See the paper for more details. ## Source Data Videos from the **training splits** of [ScanNet](https://arxiv.org/abs/1702.04405), [ScanNet++](https://arxiv.org/abs/2308.11417), and [ARKitScenes](https://arxiv.org/abs/2111.08897) (non-overlapping with [VSI-Bench](https://huggingface.co/datasets/nyu-visionx/VSI-Bench) test videos). ## Citation If you use this dataset, please cite our paper: ```bibtex @article{brown2025benchmark, author = {Brown, Ellis and Yang, Jihan and Yang, Shusheng and Fergus, Rob and Xie, Saining}, title = {Benchmark Designers Should ``Train on the Test Set'' to Expose Exploitable Non-Visual Shortcuts}, journal = {arXiv preprint arXiv:2511.04655}, year = {2025}, } ```
许可证: Apache-2.0 任务类别: - 视觉问答（visual-question-answering） 语言: - 英语（en） 标签: - 视频（Video） - 文本（Text） - 第一视角（egocentric） - 空间推理（spatial-reasoning） - 训练数据（training-data） 规模类别: - 1K<n<10K 配置: - 配置名称: default 数据文件: - 拆分方式: 训练集（train） 路径: vsi_train_10k.parquet --- # VSI-Train-10k <a href="http://arxiv.org/abs/2511.04655" target="_blank"><img alt="arXiv预印本" src="https://img.shields.io/badge/cs.CV-arXiv:2511.04655-red?logo=arxiv" height="20" /></a> <a href="https://vision-x-nyu.github.io/test-set-training/" target="_blank"><img alt="官方网站" src="https://img.shields.io/badge/🌎_Web-test--set--training-blue.svg" height="20" /></a> <a href="https://hf.co/datasets/nyu-visionx/VSI-Bench" target="_blank"><img alt="Hugging Face数据集" src="https://img.shields.io/badge/HF-VSI--Bench_(Debiased)-FED123.svg?style&logo=HuggingFace" height="20" /></a> <a href="https://github.com/vision-x-nyu/test-set-training" target="_blank"><img alt="GitHub代码" src="https://img.shields.io/badge/Code-vision--x--nyu%2Ftest--set--training-white?&logo=github&logoColor=white" /></a> **作者:**   <a href="https://ellisbrown.github.io/" target="_blank">埃利斯·布朗（Ellis Brown）</a>, <a href="https://jihanyang.github.io/" target="_blank">杨吉涵（Jihan Yang）</a>, <a href="https://github.com/vealocia" target="_blank">杨树生（Shusheng Yang）</a>, <a href="https://cs.nyu.edu/~fergus" target="_blank">罗布·弗格斯（Rob Fergus）</a>, <a href="https://www.sainingxie.com/" target="_blank">谢赛宁（Saining Xie）</a> --- **VSI-Train-10k** 是一个分布内训练数据集，包含10000条来自第一视角视频的视觉空间智能任务问答对。本数据集配套[VSI-Bench](https://hf.co/datasets/nyu-visionx/VSI-Bench)测试集使用，相关介绍见于我们的论文：《基准测试设计者应“在测试集上训练”以暴露可利用的非视觉捷径》（Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts）。 ## 数据集描述 VSI-Train-10k 包含10000条问答对，其生成流程与VSI-Bench遵循相同的程序逻辑，数据来源于ScanNet、ScanNet++与ARKitScenes数据集的**训练拆分**（与VSI-Bench测试视频无重叠）。本数据集旨在研究大语言模型（Large Language Model, LLM）如何利用分布内数据中的统计捷径与偏差。 **关键特征:** - 10000条问答对，涵盖7种基于规则的问答类型（不含路径规划） - 采用与VSI-Bench完全一致的模板与逻辑生成 - 每个场景下每种问答类型最多生成20条问答对以保证多样性 - 仅来源于训练拆分（与VSI-Bench测试数据无重叠） ## 数据集结构 ### 字段 | 字段名 | 描述 | | :--------- | :---------- | | `video` | 视频文件路径（例如：`scannet_videos_128f/train/scene0335_02_128f.mp4`） | | `conversations` | 对话轮次列表，包含`from`（发言者：human/gpt）与`value`（消息内容） | | `type` | 问答格式：`mc`（多项选择）或`oe`（开放式问答） | | `question_type` | 高层任务类别（例如：`relative_distance`相对距离、`object_counting`物体计数） | | `question_type_detail` | 详细任务子类别 | | `source` | 视频来源数据集：`scannet`、`arkitscenes`或`scannetpp` | | `question` | 完整问题文本，包含指令与选项 | | `ground_truth` | 正确答案 | VSI-Train-10k包含7种问答类型：物体计数、空间关系（更近/更远）、物体出现顺序、尺寸估计等。路径规划类问题已被排除。 ## 用法 ### 加载数据集 python from datasets import load_dataset # 加载训练数据集 vsi_train = load_dataset("nyu-visionx/VSI-Train-10k") # 访问数据 for example in vsi_train['train']: print(example['question']) print(example['ground_truth']) ### 提取视频文件 视频文件采用[zstd（Zstandard）](http://www.zstd.net/)压缩为`.tar.zst`格式（压缩效率远高于gzip）。若需并行解压所有分片： bash # 若未安装zstd，请先执行：Ubuntu/Debian系统执行 sudo apt-get install zstd；macOS系统执行 brew install zstd for shard in vsi_train_shard_*.tar.zst; do zstd -d "$shard" -c | tar -xf - & done; wait ## 文件 - `vsi_train_10k.parquet`: 针对Hugging Face Datasets优化的Parquet格式数据集标注文件 - `vsi_train_10k.jsonl`: 包含相同标注的原始JSONL格式文件 - `vsi_train_shard_*.tar.zst`: 压缩后的视频文件（共9个分片） ## 生成方法论 VSI-Train-10k遵循VSI-Bench的整理流程生成： 1. 从ScanNet、ScanNet++与ARKitScenes的训练拆分中提取物体数量、边界框与房间尺寸 2. 采用与VSI-Bench完全一致的模板生成问答对 3. 每种问答类型生成1430条问答对，且每个场景下每种问答类型最多包含20条问答对 4. 所有问题均保持与VSI-Bench的分布内一致性 更多细节请参阅论文。 ## 源数据 视频来源于[ScanNet](https://arxiv.org/abs/1702.04405)、[ScanNet++](https://arxiv.org/abs/2308.11417)与[ARKitScenes](https://arxiv.org/abs/2111.08897)的**训练拆分**（与[VSI-Bench](https://huggingface.co/datasets/nyu-visionx/VSI-Bench)测试视频无重叠）。 ## 引用 若使用本数据集，请引用我们的论文： bibtex @article{brown2025benchmark, author = {Brown, Ellis and Yang, Jihan and Yang, Shusheng and Fergus, Rob and Xie, Saining}, title = {Benchmark Designers Should ``Train on the Test Set'' to Expose Exploitable Non-Visual Shortcuts}, journal = {arXiv preprint arXiv:2511.04655}, year = {2025}, }
