EPFL-VILAB/TST-Replica

--- language: - en tags: - multimodal-learning --- # Dataset Card for TST-Replica ## Dataset Description - **Homepage:** [https://tst-vision.epfl.ch](https://tst-vision.epfl.ch) - **Repository:** [TST official repository](https://github.com) - **Paper:** [Arxiv](https://arxiv.org) ### Dataset Summary This custom TST-Replica dataset is used in research work "Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality". - `pretrain/` is a multimodal pretraining dataset collected using Replica simulation environment. It contains RGB images, and 9 additional tokenized modalities. - `segmentation/train` is the associated downstream dataset used to finetune TST pretrained models on semantic segmentation tasks. - `segmentation/test` contains the test dataset used for evaluation/testing on semantic segmentation task. This data corresponds to samples obtained from the test-space itself. ## Dataset Structure Pretraining Data ```python TST-Replica/ ├── pretrain/ │ ├── test_spaces/ │ │ ├── crop_settings/ # Contains .tar shards │ │ ├── det/ # Contains .tar shards │ │ ├── rgb/ # Contains .tar shards │ │ ├── tok_canny_edge@224/ # Contains .tar shards │ │ ├── ... # More tokenized feature directories │ │ └── tok_semseg@224/ # Contains .tar shards │ └── transfer/ │ ├── crop_settings/ # Contains .tar shards │ ├── det/ # Contains .tar shards │ ├── rgb/ # Contains .tar shards │ ├── tok_canny_edge@224/ # Contains .tar shards │ ├── ... # More tokenized feature directories │ └── tok_semseg@224/ # Contains .tar shards ├── segmentation/ │ ├── train/ # Training data for segmentation │ └── test/ # Test data for segmentation └── README.md ``` ## Dataset Creation We use [Omnidata](https://omnidata.vision), to densely sample Replica meshes corresponding to the 5 scenes to build our pre-training dataset. We defer the details of the sampling procedure to Omnidata. ### Source Data Original dataset samples are collected from Omnidata framework. ### Citation Information ``` @inproceedings{singh2026tst, title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality}, author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{\u{g}}uzhan Fatih Kar and Amir Zamir}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026} } ```语言： - 英语 标签： - 多模态学习 --- # TST-Replica 数据集卡片 ## 数据集描述 - **主页:** [https://tst-vision.epfl.ch](https://tst-vision.epfl.ch) - **代码仓库:** [TST 官方代码仓库](https://github.com) - **论文:** [Arxiv](https://arxiv.org) ### 数据集概览 该定制化TST-Replica数据集被用于研究论文《多模态作为监督信号：通过多模态实现面向测试环境的自监督专业化（Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality）》的相关工作中。 - `pretrain/`：使用Replica仿真环境采集的多模态预训练数据集，包含RGB图像与9种额外的Token化模态。 - `segmentation/train`：用于在语义分割任务上微调TST预训练模型的关联下游数据集。 - `segmentation/test`：包含用于语义分割任务评估与测试的测试数据集，该数据对应从测试空间本身获取的样本。 ## 数据集结构：预训练数据 python TST-Replica/ ├── pretrain/ │ ├── test_spaces/ │ │ ├── crop_settings/ # 包含.tar分块文件 │ │ ├── det/ # 包含.tar分块文件 │ │ ├── rgb/ # 包含.tar分块文件 │ │ ├── tok_canny_edge@224/ # 包含.tar分块文件 │ │ ├── ... # 更多Token化特征目录 │ │ └── tok_semseg@224/ # 包含.tar分块文件 │ └── transfer/ │ ├── crop_settings/ # 包含.tar分块文件 │ ├── det/ # 包含.tar分块文件 │ ├── rgb/ # 包含.tar分块文件 │ ├── tok_canny_edge@224/ # 包含.tar分块文件 │ ├── ... # 更多Token化特征目录 │ └── tok_semseg@224/ # 包含.tar分块文件 ├── segmentation/ │ ├── train/ # 语义分割训练数据 │ └── test/ # 语义分割测试数据 └── README.md ## 数据集构建 我们使用[Omnidata](https://omnidata.vision)对对应5个场景的Replica网格进行密集采样，以构建预训练数据集。关于采样流程的细节，我们交由Omnidata框架进行说明。 ### 源数据 原始数据集样本采集自Omnidata框架。 ### 引用信息 @inproceedings{singh2026tst, title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality}, author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{u{g}}uzhan Fatih Kar and Amir Zamir}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026} }