trex-dataset-23ep

This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 23 samples. ## Installation If you haven't already, install FiftyOne: ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub # Load the dataset # Note: other available arguments include 'max_samples', etc dataset = load_from_hub("Voxel51/trex-dataset-23ep") # Launch the App session = fo.launch_app(dataset) ``` # Dataset Card for T-Rex Dataset (23-episode FiftyOne subset)  A 23-episode subset of the **T-Rex Dataset**, a large-scale, tactile-reactive bimanual manipulation dataset collected via teleoperation on a Dexmate Vega-1 robot with two Sharpa Wave dexterous hands. The full dataset (5,464 episodes, 1.5 TB) is published at [zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset); this repo holds episodes `0`–`22` re-packaged as a self-contained LeRobotDataset v3.0 export and loaded into FiftyOne for exploration. ## Dataset Details ### Dataset Description - **Curated by:** Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Fei-Fei Li, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Jim Fan, Trevor Darrell - **Shared by:** zekaiwang (original dataset); this FiftyOne subset shared by the FiftyOne community - **Language(s):** English (task captions) - **License:** MIT (© 2026 The Regents of the University of California) ### Dataset Sources - **Repository:** [zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset) · [T-Rex code](https://github.com/ZhuoyangLiu2005/T-Rex) - **Paper:** [T-Rex: Tactile-Reactive Dexterous Manipulation (arXiv:2606.17055)](https://arxiv.org/abs/2606.17055) - **Demo:** [Project Page](https://tactile-rex.github.io/) ## Uses ### Direct Use Exploring and visualizing tactile-reactive bimanual manipulation episodes in the FiftyOne App — inspecting synchronized RGB + tactile video streams alongside joint state/action trajectories and per-fingertip force readings, filtering by task/motor primitive/object, and prototyping data loaders before working with the full 5,464-episode dataset. ### Out-of-Scope Use This 23-episode subset is not a statistically representative sample of the full dataset (it is simply the first LeRobot data/video shard) and should not be used to draw conclusions about task, object, or motor-primitive distributions across the full T-Rex Dataset — load the full `zekaiwang/trex_dataset` repo for that. ## Dataset Structure This is a **multimodal** FiftyOne dataset (`dataset.media_type == "multimodal"`) with **23 samples**, one sample per episode. Each sample's media (23 video streams) is not copied into per-sample files; instead it is resolved through a `media_reference` that points into the exported LeRobotDataset v3.0 source (`data/`, `videos/`, `meta/` in this repo) at import time — this is how FiftyOne represents LeRobot episodes natively. ### Fields | Field | FiftyOne type | Description | |-------|---------------|--------------| | `id` | `ObjectIdField` | FiftyOne sample id | | `media_reference` | `MediaReferenceField` | Pointer into the LeRobot source's `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample | | `tags` | `ListField(StringField)` | FiftyOne sample tags (empty by default) | | `metadata` | `Metadata` (`size_bytes`, `mime_type`) | Standard FiftyOne sample metadata | | `created_at` / `last_modified_at` | `DateTimeField` | FiftyOne bookkeeping timestamps | | `episode_index` | `IntField` | Episode index within this subset (`0`–`22`, remapped on export from the source dataset's original indices) | | `task` | `StringField` | Primary task caption for the episode (verbatim from source `caption`) | | `tasks` | `ListField(StringField)` | Full task list for the episode (LeRobot `tasks` array; length 1 for every episode here) | | `length` | `IntField` | Number of frames in the episode (verbatim from source) | | `duration` | `FloatField` | Episode duration in seconds (`length / fps`) | | `robot_type` | `StringField` | `dexmate_vega1_and_sharpa_wave` (verbatim from source `meta/info.json`) | | `fps` | `FloatField` | Recording frame rate, 30.0 (verbatim from source) | The per-frame numeric features (`observation.state` (58,), `action` (58,), `observation.tactile_force` (60,)) and the 23 per-frame video streams are **not** flattened into sample fields — they remain in the LeRobot `data/*.parquet` and `videos/*/*.mp4` files referenced by `media_reference`, and are surfaced by the FiftyOne App's State & Action, Streams, and Statistics viewer tabs rather than as queryable sample-level fields. ### Label types and why There are no traditional detection/classification/segmentation labels in this dataset. `task` is stored as a plain `StringField` rather than `fo.Classification` because task captions here are free-form natural language instructions (5,370 unique across the full dataset, 22 unique in this subset), not a fixed closed-set taxonomy — a `Classification` label with `logits`/`confidence` semantics does not fit a free-text instruction. ### `dataset.info` contents ```python { "lerobot": { "format": "LeRobotDataset", "format_major": 3, "episode_count": 5464, # total episodes in the full source dataset "imported_episode_count": 23, # episodes actually imported into this subset "skipped_episodes": [], } } ``` ### Parsing decisions - **Which episodes, and why:** episodes `0`–`22` were selected because they are the largest contiguous, zero-gap block of episodes whose `data/chunk-000/file-000.parquet` shard **and** every one of the 23 `videos/<key>/chunk-000/file-000.mp4` shards are shared — i.e. the smallest set of source files that had to be downloaded to get a complete, non-truncated set of episodes, given a limited local disk budget. It is not a curated or stratified sample. - **Re-export, not a thin reference to the original repo:** this repo is a self-contained LeRobotDataset v3.0 export (via FiftyOne's `LeRobotDatasetExporter`), not a pointer back to `zekaiwang/trex_dataset`. Episode indices, frame `index`, and video/data chunk-file coordinates were rewritten during export so the 23 episodes are contiguous (`0`–`22`) and self-consistent; task indices were remapped to only the tasks actually present in this subset. Per-episode and global statistics (`meta/stats.json`, per-episode `stats/*` columns) were recomputed from the exported rows, not carried over from the source's global stats. - **Tactile video codec (read before decoding outside FiftyOne):** the 20 tactile streams (`observation.images.tactile_{left,right}_{raw,deform}_{finger}`) are encoded losslessly (`libx264 -qp 0`) because their pixel values are physically meaningful raw sensor/deformation readings. They are grayscale, full-range `yuvj420p`, which forces the H.264 **High 4:4:4 Predictive** profile — most browsers cannot decode this profile in a plain `<video>`/WebCodecs pipeline (no thumbnails in the HF preview or generic players), so use FiftyOne, ffmpeg, PyAV, or torchcodec to view them locally. The 3 RGB streams (`head_left`, `left_wrist`, `right_wrist`) use standard limited-range `yuv420p`/BT.709 and preview normally everywhere. - **No held-out or unlabeled split:** all 23 episodes are in a single `train` split with every episode language-annotated; nothing was withheld. ## Dataset Creation ### Curation Rationale The full T-Rex Dataset was collected to study tactile-reactive dexterous manipulation — pairing bimanual arm/hand joint trajectories with synchronized per-fingertip tactile sensing so that policies can condition on touch, not just vision and proprioception. This subset exists purely as a lightweight, disk-budget-friendly slice for exploration and tooling in FiftyOne; it was not re-curated for content. ### Source Data #### Data Collection and Processing - **Robot:** Dexmate Vega-1 dual-arm mobile robot (7 actuated joints per arm) with two Sharpa Wave dexterous hands (5 fingertip tactile sensors each). During collection the wheels, torso, and head joints are fixed; only the 14 arm joints and two hands are actuated. - **Cameras:** a head-mounted ZED X Mini stereo camera (left monocular RGB stream recorded) plus two wide-view ZED X One S monocular RGB wrist cameras, all at 640×360, 30 fps. - **Tactile sensing:** each hand's 5 fingertip sensors contribute a raw sensor image (`tactile_*_raw_*`, 320×240) and an estimated deformation map (`tactile_*_deform_*`, 240×240), both stored as lossless grayscale video, plus an estimated 6-axis net wrench per fingertip in `observation.tactile_force` (60,) = (left, right) × (thumb…pinky) × (Fx, Fy, Fz, Mx, My, Mz). - **Teleoperation:** Manus gloves capture fingertip positions retargeted to the Sharpa Wave hands via the manufacturer's differential-inverse-kinematics package (Pinocchio + CasADi). Two VIVE trackers provide SE(3) wrist poses converted to arm joint commands via differential inverse kinematics (Pink), low-pass filtered, and tracked by the manufacturer's low-level cascade PID controller. A 30 Hz high-level thread records observations and joint-space targets (`action`) while a 300 Hz low-level thread runs control; the dataset's 30 fps matches the high-level loop. - **`observation.state` / `action` layout (58,):** `[L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22]` joint positions (`observation.state`) and target joint positions (`action`). Full per-dimension joint names are in the source `meta/info.json` (`features[*].names`). #### Who are the source data producers? Collected by the T-Rex authors (UC Berkeley and collaborating institutions; see author list above) via in-person teleoperation with the hardware/software stack described above. ### Annotations #### Annotation process Each episode is labeled with a human-verified natural-language `caption` (task instruction), a `motor_primitive` category (one of 22, e.g. `reach`, `lift_and_place`), a canonical `object` name, and — only for `lift_and_place` episodes — a canonical `target`/receptacle name (null otherwise). In this subset, `caption` is surfaced as the sample-level `task` field. #### Personal and Sensitive Information None identified — the dataset contains robot joint/tactile sensor data and task captions describing tabletop manipulation of everyday objects; no human subjects data. ## Citation **BibTeX:** ```bibtex @misc{trex2026, title={T-Rex: Tactile-Reactive Dexterous Manipulation}, author={Dantong Niu and Zhuoyang Liu and Zekai Wang and Boning Shao and Zhao-Heng Yin and Anirudh Pai and Yuvan Sharma and Stefano Saravalle and Ruijie Zheng and Jing Wang and Ryan Punamiya and Mengda Xu and Yuqi Xie and Yunfan Jiang and Letian Fu and Konstantinos Kallidromitis and Matteo Gioia and Junyi Zhang and Jiaxin Ge and Haiwen Feng and Fabio Galasso and Wei Zhan and David M. Chan and Yutong Bai and Roei Herzig and Jiahui Lei and Fei-Fei Li and Ken Goldberg and Jitendra Malik and Pieter Abbeel and Yuke Zhu and Danfei Xu and Jim Fan and Trevor Darrell}, year={2026}, eprint={2606.17055}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2606.17055}, } ``` **APA:** Niu, D., Liu, Z., Wang, Z., Shao, B., Yin, Z.-H., Pai, A., Sharma, Y., Saravalle, S., Zheng, R., Wang, J., Punamiya, R., Xu, M., Xie, Y., Jiang, Y., Fu, L., Kallidromitis, K., Gioia, M., Zhang, J., Ge, J., Feng, H., Galasso, F., Zhan, W., Chan, D. M., Bai, Y., Herzig, R., Lei, J., Li, F.-F., Goldberg, K., Malik, J., Abbeel, P., Zhu, Y., Xu, D., Fan, J., & Darrell, T. (2026). *T-Rex: Tactile-Reactive Dexterous Manipulation*. arXiv:2606.17055. ## More Information This is a 23-episode subset of the full [zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset) (5,464 episodes, ~50 hours, 1.5 TB), produced for local exploration under a limited disk budget. See the source repo for the full dataset, the [T-Rex Quick Start](https://github.com/ZhuoyangLiu2005/T-Rex/tree/main/dataset_quickstart) tools, and the [Colab notebook](https://colab.research.google.com/github/ZhuoyangLiu2005/T-Rex/blob/main/dataset_quickstart/quickstart.ipynb). ## Dataset Card Authors [Harpreet Sahota](https://huggingface.co/harpreetsahota) ## Dataset Card Contact [Harpreet Sahota](https://huggingface.co/harpreetsahota)