utaustin_mutex

This dataset was created using [LeRobot](https://github.com/huggingface/lerobot).

## Dataset Description

- **Homepage:** https://ut-austin-rpl.github.io/MUTEX/
- **Paper:** https://arxiv.org/abs/2309.14320
- **License:** mit

## Dataset Structure

[meta/info.json](meta/info.json):
```json
{
"codebase_version": "v2.0",
"robot_type": "unknown",
"total_episodes": 1500,
"total_frames": 361883,
"total_tasks": 50,
"total_videos": 3000,
"total_chunks": 2,
"chunks_size": 1000,
"fps": 20,
"splits": {
"train": "0:1500"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
"features": {
"observation.images.image": {
"dtype": "video",
"shape": [
128,
128,
3
],
"names": [
"height",
"width",
"channel"
],
"video_info": {
"video.fps": 20.0,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"has_audio": false
}
},
"observation.images.wrist_image": {
"dtype": "video",
"shape": [
128,
128,
3
],
"names": [
"height",
"width",
"channel"
],
"video_info": {
"video.fps": 20.0,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"has_audio": false
}
},
"language_instruction": {
"dtype": "string",
"shape": [
1
],
"names": null
},
"observation.state": {
"dtype": "float32",
"shape": [
8
],
"names": {
"motors": [
"motor_0",
"motor_1",
"motor_2",
"motor_3",
"motor_4",
"motor_5",
"motor_6",
"motor_7"
]
}
},
"action": {
"dtype": "float32",
"shape": [
7
],
"names": {
"motors": [
"motor_0",
"motor_1",
"motor_2",
"motor_3",
"motor_4",
"motor_5",
"motor_6"
]
}
},
"timestamp": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"next.reward": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"next.done": {
"dtype": "bool",
"shape": [
1
],
"names": null
},
"index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
}
}
}
```

## Citation

**BibTeX:**

```bibtex
@inproceedings{shah2023mutex,
title={{MUTEX}: Learning Unified Policies from Multimodal Task Specifications},
author={Rutav Shah and Roberto Mart{\'\i}n-Mart{\'\i}n and Yuke Zhu},
booktitle={7th Annual Conference on Robot Learning},
year={2023},
url={https://openreview.net/forum?id=PwqiqaaEzJ}
}
```

本数据集基于[LeRobot](https://github.com/huggingface/lerobot)开发构建。

## 数据集说明
- **主页**：https://ut-austin-rpl.github.io/MUTEX/
- **论文**：https://arxiv.org/abs/2309.14320
- **许可证**：MIT许可证

## 数据集结构
[meta/info.json](meta/info.json)：
json
{
"codebase_version": "v2.0",
"robot_type": "unknown",
"total_episodes": 1500,
"total_frames": 361883,
"total_tasks": 50,
"total_videos": 3000,
"total_chunks": 2,
"chunks_size": 1000,
"fps": 20,
"splits": {
"train": "0:1500"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
"features": {
"observation.images.image": {
"dtype": "video",
"shape": [128, 128, 3],
"names": ["height", "width", "channel"],
"video_info": {
"video.fps": 20.0,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"has_audio": false
}
},
"observation.images.wrist_image": {
"dtype": "video",
"shape": [128, 128, 3],
"names": ["height", "width", "channel"],
"video_info": {
"video.fps": 20.0,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"has_audio": false
}
},
"language_instruction": {
"dtype": "string",
"shape": [1],
"names": null
},
"observation.state": {
"dtype": "float32",
"shape": [8],
"names": {
"motors": ["motor_0", "motor_1", "motor_2", "motor_3", "motor_4", "motor_5", "motor_6", "motor_7"]
}
},
"action": {
"dtype": "float32",
"shape": [7],
"names": {
"motors": ["motor_0", "motor_1", "motor_2", "motor_3", "motor_4", "motor_5", "motor_6"]
}
},
"timestamp": {
"dtype": "float32",
"shape": [1],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [1],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [1],
"names": null
},
"next.reward": {
"dtype": "float32",
"shape": [1],
"names": null
},
"next.done": {
"dtype": "bool",
"shape": [1],
"names": null
},
"index": {
"dtype": "int64",
"shape": [1],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [1],
"names": null
}
}
}

## 引用
**BibTeX：**
bibtex
@inproceedings{shah2023mutex,
title={{MUTEX}: 从多模态任务规范学习统一策略},
author={Rutav Shah and Roberto Mart{\'in-Mart{\'in} and Yuke Zhu},
booktitle={第7届机器人学习年度会议},
year={2023},
url={https://openreview.net/forum?id=PwqiqaaEzJ}
}

搜集汇总

背景与挑战

背景概述

该数据集名为utaustin_mutex，由LeRobot创建，基于UT Austin RPL的MUTEX项目，采用MIT许可证。它包含1500个episodes和50个任务，总帧数达361,883，数据以parquet和视频格式存储，特征包括图像、状态和动作等，用于机器人策略学习。