mimic-data/FrankaCafe-Alpha

---
license: mit
---

<img src="assets/FrankaCafe-Alpha.gif" alt="Image Alt Text" width="60%" style="display: block; margin-left: auto; margin-right: auto;" />
<h1 align="center"><font size="7">FrankaCafe-Alpha</font></h1>

The **FrankaCafe-Alpha dataset** offers human-collected demos of single-arm manipulation tasks in kitchen environments. This **Alpha version** is designed to gather feedback on how these demos can support your projects and to understand whether scaling up such datasets will better meet the growing demand for high-quality robot learning training data.

# **Dataset Features**
- **Expanding collection** of demos and tasks based on community feedback
- **~500 demos** across 5 manipulation tasks collected in Mujoco.
- **Photorealistic camera observations** rendered with Isaac sim for high-quality training data
- Each trajectory rendered in **10 visual styles** for increased data diversity

This project is based on **robocasa**, which consists of dozens of tasks, kitchen styles and layouts. You can check out their awesome work [here](https://robocasa.ai/docs/introduction/overview.html).

# **Dataset Updates**
For the most recent dataset updates and task details, check out the [task_info.md](info/task_info.md).

# **Coming Up**
- 50+ tasks added to FrankaCafe
- sub-task and failure recovery annotations
- Large datasets with the GR1 and other bimanual setups

# **Accessing the Dataset**
The **full dataset** is hosted on a GCP bucket. You can gain access by filling out [this Google form](https://docs.google.com/forms/d/e/1FAIpQLSfxS73YyGlAxsIh60fJ2qsfNzXcRoghoGw2fl1TMOyxx6bV_w/viewform?usp=dialog) with your GCP account details. This way you can access demos and tasks we add in the coming weeks.

A **sample** of demos is available to [download](sample.tar.gz) (~30GB when extracted). The files can be viewed with `view_safetensor.py`. For example, you can do
```
# Install required packages in your python environment
pip install numpy matplotlib opencv-python safetensors tk

# Point the script to the folder with demo files, for example
python scripts/view_safetensor.py --dir demos/tasks/DryDrinkware/
```
You will then be able to view the file like in the image below.
<img src="https://cdn-uploads.huggingface.co/production/uploads/6775ff25b97c9837608b2342/2ZWWQtEHDR1ODm09cHjA2.png" alt="Image Alt Text" width="60%" style="display: block; margin-left: auto; margin-right: auto;" />

# **File Details**

The data is organized in folders named after the task, with files named as:

- **`<trajectory_id>_<style_id>.safetensors`** files where,
- *`trajectory_id`* is in `YY-MM-DD_HH-MM-SS` format, representing the time when the trajectory was approved for rendering in all styles.
- *`style_id`* corresponds to the visual style in which the demo was rendered.

This naming structure allows you to identify demos by visual style and easily differentiate style versions of the same trajectory.

### **Safetensors File Structure**
We use [**safetensors**](https://github.com/huggingface/safetensors) format to store the trajectory and image data in a single file. Each `.safetensors` file contains task and scene information, trajectory data for the robot's actions, and camera images from 3 perspectives.

The trajectory, style, and task identifiers are available in the safetensors file metadata as well. For example, you can obtain these with the safetensors `metadata()` method
```json
{
"trajectory_id": "2025-01-09_00-08-40",
"task_name": "CerealAndBowl",
"style_id": "8"
}
```

In addition, the **safetensors** files contain the following data attributes:

| **Attribute** | **Shape** | **Description** |
|-------------------------------------------------|------------------------|---------------------------------------------------------------------------|
| `actions/gripper/base_frame_delta` | (N, 6) | End-effector action deltas in the robot base frame (dx, dy, dz, droll, dpitch, dyaw). |
| `actions/gripper/command` | (N, 1) | Gripper open/close command. |
| `actions/gripper/gripper_frame_delta` | (N, 6) | End-effector action deltas in the gripper frame. |
| `actions/joints/angles` | (N, 7) | Joint angles of the robot arm (7 DoF). |
| `actions/joints/velocities` | (N, 7) | Joint velocities. |
| `images/Camera_robot0_agentview_left` | (N, 512, 512, 3) | Camera images from the left shoulder camera mounted on the robot base. |
| `images/Camera_robot0_agentview_right` | (N, 512, 512, 3) | Camera images from the right shoulder camera mounted on the robot base. |
| `images/Camera_robot0_eye_in_hand` | (N, 512, 512, 3) | Camera images from the eye-in-hand camera mounted on the robot's gripper. |

Where `N` is the number of frames for each demo.

## **Environment and Scene Details**

The environment configuration for data collection, including robot setup, camera parameters, and layout styles, is detailed in the [env_config.json](info/env_config.json) file,

# **License Information**
All data in this dataset, including demonstrations and associated files hosted on the Hugging Face repository and the GCP bucket, is made available under the [MIT License](https://choosealicense.com/licenses/mit/).