EPFL-VILAB/TST-ProcTHOR

--- language: - en tags: - multimodal-learning --- # Dataset Card for TST-ProcTHOR ## Dataset Description - **Homepage:** [https://tst-vision.epfl.ch](https://tst-vision.epfl.ch) - **Repository:** [TST official repository](https://github.com) - **Paper:** [Arxiv](https://arxiv.org) ### Dataset Summary This custom TST-ProcTHOR dataset is used in research work "Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality". - `pretrain/` is a multimodal pretraining dataset collected using ProcTHOR environment. It contains RGB images, and 9 additional tokenized modalities. - `segmentation/train` is the associated downstream dataset used to finetune TST pretrained models on semantic segmentation tasks. - `segmentation/test` contains the test dataset used for evaluation/testing on semantic segmentation task. This data corresponds to samples obtained from the test-space itself. - `captioning/train` is the associated downstream dataset used to finetune TST pretrained models on captioning task. - `captioning/test` contains the test dataset used for evaluation/testing on captioning task. This data corresponds to samples obtained from the test-space itself. ## Dataset Structure ```python TST-ProcTHOR/ ├── pretrain/ │ ├── test_spaces/ │ │ ├── crop_settings/ # Contains .tar shards │ │ ├── det/ # Contains .tar shards │ │ ├── rgb/ # Contains .tar shards │ │ ├── tok_canny_edge@224/ # Contains .tar shards │ │ ├── ... # More tokenized feature directories │ │ └── tok_semseg@224/ # Contains .tar shards │ └── transfer/ │ ├── crop_settings/ # Contains .tar shards │ ├── det/ # Contains .tar shards │ ├── rgb/ # Contains .tar shards │ ├── tok_canny_edge@224/ # Contains .tar shards │ ├── ... # More tokenized feature directories │ └── tok_semseg@224/ # Contains .tar shards ├── segmentation/ │ ├── train/ # Training data for segmentation │ └── test/ # Test data for segmentation ├── captioning/ │ ├── train/ # Training data for captioning │ └── test/ # Test data for captioning └── README.md ``` ## Dataset Creation It includes procedurally generated house-like environments. We use 5 procedurally generated houses as our test space. Dataset is collected by randomly sample various agent x, y, z positions and orientations along its axis in the test space, and collect RGB-D images at these points. ### Source Data Dataset is collected from ProcTHOR simulator. ### Citation Information ``` @inproceedings{singh2026tst, title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality}, author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{\u{g}}uzhan Fatih Kar and Amir Zamir}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026} } ```--- 语言： - 英语 标签： - 多模态学习（multimodal-learning） --- # TST-ProcTHOR 数据集卡片 ## 数据集描述 - **主页：** [https://tst-vision.epfl.ch](https://tst-vision.epfl.ch) - **代码仓库：** [TST 官方仓库](https://github.com) - **论文：** [Arxiv](https://arxiv.org) ### 数据集概述 这款定制化TST-ProcTHOR数据集被用于题为《Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality》的研究工作。 - `pretrain/`：使用ProcTHOR环境采集的多模态预训练数据集，包含RGB图像与9种额外的Token化模态。 - `segmentation/train`：用于在语义分割任务上微调TST预训练模型的关联下游数据集。 - `segmentation/test`：包含用于语义分割任务评估与测试的测试数据集，该数据对应于从测试空间直接获取的样本。 - `captioning/train`：用于在图像字幕生成任务上微调TST预训练模型的关联下游数据集。 - `captioning/test`：包含用于图像字幕生成任务评估与测试的测试数据集，该数据对应于从测试空间直接获取的样本。 ## 数据集结构 python TST-ProcTHOR/ ├── pretrain/ │ ├── test_spaces/ │ │ ├── crop_settings/ # 存储 .tar 分片文件 │ │ ├── det/ # 存储 .tar 分片文件 │ │ ├── rgb/ # 存储 .tar 分片文件 │ │ ├── tok_canny_edge@224/ # 存储 .tar 分片文件 │ │ ├── ... # 更多Token化特征目录 │ │ └── tok_semseg@224/ # 存储 .tar 分片文件 │ └── transfer/ │ ├── crop_settings/ # 存储 .tar 分片文件 │ ├── det/ # 存储 .tar 分片文件 │ ├── rgb/ # 存储 .tar 分片文件 │ ├── tok_canny_edge@224/ # 存储 .tar 分片文件 │ ├── ... # 更多Token化特征目录 │ └── tok_semseg@224/ # 存储 .tar 分片文件 ├── segmentation/ │ ├── train/ # 语义分割训练数据 │ └── test/ # 语义分割测试数据 ├── captioning/ │ ├── train/ # 图像字幕生成训练数据 │ └── test/ # 图像字幕生成测试数据 └── README.md ## 数据集构建 该数据集包含程序生成的类住宅环境。我们选取5个程序生成的住宅作为测试空间，通过在测试空间内沿坐标轴随机采样智能体的x、y、z位置与朝向，并在这些采样点处采集RGB-D图像来完成数据集的采集。 ### 源数据 数据集采集自ProcTHOR模拟器。 ### 引用信息 bibtex @inproceedings{singh2026tst, title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality}, author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{u{g}}uzhan Fatih Kar and Amir Zamir}, booktitle={International Conference on Learning Representations (ICLR)}, year={2026} }