EPFL-VILAB/TST-Replica

---
language:
- en
tags:
- multimodal-learning
---
# Dataset Card for TST-Replica

## Dataset Description

- **Homepage:** [https://tst-vision.epfl.ch](https://tst-vision.epfl.ch)
- **Repository:** [TST official repository](https://github.com)
- **Paper:** [Arxiv](https://arxiv.org)

### Dataset Summary

This custom TST-Replica dataset is used in research work "Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality".

- `pretrain/` is a multimodal pretraining dataset collected using Replica simulation environment. It contains RGB images, and 9 additional tokenized modalities.

- `segmentation/train` is the associated downstream dataset used to finetune TST pretrained models on semantic segmentation tasks.

- `segmentation/test` contains the test dataset used for evaluation/testing on semantic segmentation task. This data corresponds to samples obtained from the test-space itself.

## Dataset Structure Pretraining Data

```python
TST-Replica/
├── pretrain/
│ ├── test_spaces/
│ │ ├── crop_settings/ # Contains .tar shards
│ │ ├── det/ # Contains .tar shards
│ │ ├── rgb/ # Contains .tar shards
│ │ ├── tok_canny_edge@224/ # Contains .tar shards
│ │ ├── ... # More tokenized feature directories
│ │ └── tok_semseg@224/ # Contains .tar shards
│ └── transfer/
│ ├── crop_settings/ # Contains .tar shards
│ ├── det/ # Contains .tar shards
│ ├── rgb/ # Contains .tar shards
│ ├── tok_canny_edge@224/ # Contains .tar shards
│ ├── ... # More tokenized feature directories
│ └── tok_semseg@224/ # Contains .tar shards
├── segmentation/
│ ├── train/ # Training data for segmentation
│ └── test/ # Test data for segmentation
└── README.md

```

## Dataset Creation

We use [Omnidata](https://omnidata.vision), to densely
sample Replica meshes corresponding to the 5 scenes to
build our pre-training dataset. We defer the details
of the sampling procedure to Omnidata.

### Source Data

Original dataset samples are collected from Omnidata framework.

### Citation Information

```
@inproceedings{singh2026tst,
title={Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality},
author={Kunal Pratap Singh and Ali Garjani and Rishubh Singh and Muhammad Uzair Khattak and Efe Tarhan and Jason Toskov and Andrei Atanov and O{\u{g}}uzhan Fatih Kar and Amir Zamir},
booktitle={International Conference on Learning Representations (ICLR)},
year={2026}
}
```