Manggu/DexCanvas

---
license: odbl
task_categories:
- robotics
- video-classification
- image-classification
- object-detection
tags:
- dexterous-manipulation
- hand-object-interaction
- motion-capture
- physics-simulation
- rgbd
- contact-forces
- computer-vision
size_categories:
- 10K<n<100K
---

# DexCanvas: Dexterous Manipulation Dataset v0.1

**⚠️ TEST RELEASE**: This is a preview version containing 1% of the full dataset. Contact force data is not included in v0.1.

DexCanvas is a large-scale hybrid dataset for robotic hand-object interaction research, combining real human demonstrations with physics-validated simulation data.

## Dataset Statistics (v0.1 Test Release)

- **Total Frames**: ~30 million multi-view RGB-D frames
- **Total Duration**: ~70 hours of dexterous hand-object interactions
- **Real Demonstrations**: ~0.7 hours of human mocap data (1/100 of collected data)
- **Expansion Ratio**: 100× from real to simulated data
- **Manipulation Types**: 21 types based on Cutkosky taxonomy
- **Objects**: 30 objects (geometric primitives + YCB objects)
- **Capture Rate**: 100 Hz optical motion capture

## Manipulation Coverage

The dataset spans four primary grasp categories:

- **Power Grasps**: Full-hand wrapping grips
- **Intermediate Grasps**: Mixed precision-power combinations
- **Precision Grasps**: Fingertip-based manipulation
- **In-Hand Manipulation**: Object reorientation and repositioning

All 21 manipulation types follow the Cutkosky grasp taxonomy.

## Data Modalities

Each frame includes:

- **RGB-D Data**: Multi-view color and depth images
- **Hand Pose**: MANO hand parameters with high-precision tracking
- **Object State**: 6-DoF pose and object wrenches
- **Annotations**: Per-frame labels and metadata

**Note**: Contact force data is not included in v0.1. Contact forces will be available in future releases.

## Data Pipeline

The dataset is generated through three stages:

1. **Real Capture**: Optical motion capture of human demonstrations at 30 Hz
2. **Force Reconstruction**: RL-based physics simulation to infer contact forces
3. **Physics Validation**: Verification of contact points, forces, and object dynamics

This hybrid approach provides contact information impossible to observe directly in real-world scenarios while maintaining physical accuracy.

## Installation

```bash
pip install datasets huggingface_hub
```

For image processing and visualization:

```bash
pip install pillow numpy torch
```

Authenticate with HuggingFace (required for private datasets):

```bash
huggingface-cli login
```

Or set your token as an environment variable:

```bash
export HF_TOKEN="your_token_here"
```

## Quick Start

### Data Structure

```json
{
"trajectory_meta_data": {
"generated_data": "int",
"data_fps": "int",
"mocap_raw_data_source": {
"operator": "str",
"object": "str",
"gesture": "str"
},
"total_frames": "int",
"mano_hand_shape": "(10,)"
//...
},
"sequence_info": {
"timestamp": "(T,)",
"hand_joint": {
"position": "(T, 3)",
"rotation": "(T, 3)",
"finger_pose": "(T, 48)"
},
"object_info": {
"pose": "(T, 6)"
},
"mano_model_output": {
"joints": "(T, 63)"
}
}
}
```
### Visualization

Visualize trajectories using the **mocap_loader**:

```bash
# Install dependencies
pip install open3d trimesh scipy

# Visualize trajectory
python -m hand_trajectory_loader.examples.visualize_trajectory \
dataset.parquet 0 \
--mano-model assets/mano/models/MANO_RIGHT.pkl \
--object assets/objects/cube1.stl \
--show-joints
```

Controls: **SPACE** pause/resume, **M** toggle hand mesh, **O** toggle object, **Q** quit

## Version Information

**v0.1 (Test Release)** includes:
- 1% of collected real human demonstration data
- MANO hand parameters
- Object pose data
- Manipulation type annotations

**Coming in future releases**:
- Complete dataset (100× larger than v0.1)
- Contact force data with physics validation
- Additional objects and manipulation types
- Extended annotations and metadata

## Contact

**Research Collaboration**
Academic inquiries: [e-mail]

**Business Inquiries**
Business collaboration: [e-mail]

**Website**
https://www.dex-robot.com/en
https://dexcanvas.github.io/

## Citation

```bibtex
@article{dexcanvas2025,
title={DexCanvas: A Large-Scale Hybrid Dataset for Dexterous Manipulation},
author={DexRobot Team},
year={2025},
eprint={2510.15786},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2510.15786}
}
```

## License

This dataset is released under the Open Database License (ODbL).

---

**Developed by DexRobot Team**
Last Updated: October 2025