nyu-visionx/solaris-training-dataset

---
license: apache-2.0
task_categories:
- image-to-video
tags:
- minecraft
- world-model
- multi-agent
- action-conditioned
---

# Solaris Training Dataset

[**Project Page**](https://solaris-wm.github.io/) | [**Paper**](https://huggingface.co/papers/2602.22208) | [**Github**](https://github.com/solaris-wm/solaris)

The training dataset collected via [SolarisEngine](https://github.com/solaris-wm/solaris-engine) to train the [Solaris](https://github.com/solaris-wm/solaris) multiplayer world model for Minecraft.

# Dataset Info

The dataset contains `12.64 M` frames across two players. It was recorded at `20` FPS. The observation (video) dimensions are: `1280 × 720`.

The action space is presented below:

| Action key | Type | Description |
|--------------|------------------|-------------|
| forward | bool/sustained | Player moving forward (W). |
| back | bool/sustained | Player moving backward (S). |
| left | bool/sustained | Player strafing left (A). |
| right | bool/sustained | Player strafing right (D). |
| jump | bool/sustained | Player jumping. |
| sprint | bool/sustained | Player sprinting. |
| sneak | bool/sustained | Player sneaking. |
| camera | vec2/sustained | Change in player camera orientation (yaw, pitch). |
| attack | bool/once | Player attacks. |
| use | bool/once | Player uses / interacts with the environment. |
| mount | bool/once | Player mounts an entity/vehicle. |
| dismount | bool/once | Player dismounts. |
| place_block | bool/once | Player places a block using the currently selected item. |
| place_entity | bool/once | Player places an entity item. |
| mine | bool/sustained | Player mining a block. |
| hotbar.1 | bool/once | Player selects hotbar slot 1. |
| hotbar.2 | bool/once | Player selects hotbar slot 2. |
| hotbar.3 | bool/once | Player selects hotbar slot 3. |
| hotbar.4 | bool/once | Player selects hotbar slot 4. |
| hotbar.5 | bool/once | Player selects hotbar slot 5. |
| hotbar.6 | bool/once | Player selects hotbar slot 6. |
| hotbar.7 | bool/once | Player selects hotbar slot 7. |
| hotbar.8 | bool/once | Player selects hotbar slot 8. |
| hotbar.9 | bool/once | Player selects hotbar slot 9. |

# Usage

The multiplayer `Duet` dataset is stored in a sharded form on Hugging Face. You can download and unshard it using the following steps:

### 1. Download the shards
```bash
hf download nyu-visionx/solaris-training-dataset --repo-type dataset --local-dir YOUR_STORAGE_PATH/datasets
```

### 2. Unshard the dataset
Run the `unshard_dataset.py` script provided in the [Solaris repository](https://github.com/solaris-wm/solaris):
```bash
python unshard_dataset.py --shards YOUR_STORAGE_PATH/datasets/duet_sharded --out YOUR_STORAGE_PATH/datasets/duet
```

# Citation

```bibtex
@article{savva2025solaris,
title={Solaris: Building a Multiplayer Video World Model in Minecraft},
author={Georgy Savva and Oscar Michel and Daohan Lu and Suppakit Waiwitlikhit and Timothy Meehan and Dhairya Mishra and Srivats Poddar and Jack Lu and Saining Xie},
journal={arXiv preprint arXiv:2602.22208},
year={2025}
}
```