MATRIXGlove

# MATRIXGlove MATRIXGlove is a human egocentric manipulation dataset from **Xspark AI**. It contains aligned head-camera RGB, camera and wrist poses, hand joint angles, regional normal forces, binary contact states, raw tactile channels, and Chinese action instructions. - Dataset: [ModelScope / MATRIXGlove](https://www.modelscope.cn/datasets/i0stream6657/MATRIXGlove) - Project: [HumanTouch](https://xsparkai.com/sparklab/humantouch/) - Contact: [e-mail] - Format: [Open-X-Tactile](https://open-x-tactile.github.io/) and [FTP-1](https://github.com/michaelyuancb/ftp1-policy/blob/main/data_processing/README.md) ## Release contents The `standard` release contains **11,212 episodes**, **23,613,689 frames**, and **114 Zarr batches** at **60 Hz**. Its duration is **109.322634 hours**, calculated as `frame_count / 60 / 3600`. | Source | Batch identifiers, inclusive | Batches | Episodes | Frames | Hours | | --- | --- | ---: | ---: | ---: | ---: | | SSD1 | `ssd1_episode_batch_0000`–`ssd1_episode_batch_0020` | 21 | 2,075 | 6,630,294 | 30.695806 | | SSD2 | `ssd2_episode_batch_0000`–`ssd2_episode_batch_0063` | 64 | 6,297 | 11,062,358 | 51.214620 | | SSD3 | `ssd3_episode_batch_0000`–`ssd3_episode_batch_0028` | 29 | 2,840 | 5,921,037 | 27.412208 | | **Total** | | **114** | **11,212** | **23,613,689** | **109.322634** | The data comprises **3,358 archive parts**, totaling **886,142,594,066 bytes**, excluding documentation. Each episode represents one retained session interval, ending at the final annotation's exclusive end frame. A batch contains up to 100 episodes. Within each source, sessions are ordered by date, annotation task ID, and session ID. Batch numbers are zero-based within each source. The release contains the standard subset and one head-camera view. Depth and wrist-camera streams are outside its scope. The repository supplies no predefined training, validation, or test split; applications should define splits at the episode level or a coarser grouping. In OXT metadata, `task=114` counts Zarr batches, while `trajectory=11212` counts episodes. ## Storage and arrays Each batch is a **Zarr v2** group packaged as a byte-split ZIP64 archive under `standard/`. Its `data/` arrays share a leading frame dimension `T`. `meta/episode_ends` contains `E` cumulative exclusive episode ends. In the table, `{side}` means `left` or `right`, and `{group}` means `force` or `contact`. | Path inside a Zarr group | Shape | Type | Meaning | | --- | --- | --- | --- | | `data/timestamps` | `(T,)` | float64 | Absolute source timestamps in seconds | | `data/camera_ego_rgb` | `(T,126,224,3)` | uint8 | RGB in height, width, channel order | | `data/camera_ego_pose` | `(T,6)` | float64 | Camera optical-frame pose in world coordinates | | `data/{side}_wrist_pose` | `(T,6)` | float64 | Wrist-frame pose in world coordinates | | `data/{side}_hand_joints` | `(T,21)` | float64 | FTP-1 derived joint-angle values in radians | | `data/{side}_hand_joints_idx` | `(T,21)` | int64 | FAAS IDs of the angle slots | | `data/{side}_tactile_data_force` | `(T,6,1)` | float64 | Six regional normal forces in N | | `data/{side}_tactile_data_contact` | `(T,6,1)` | bool | Six regional contact states | | `data/{side}_tactile_area_{group}` | `(T,6)` | int64 | Functional-area IDs | | `data/{side}_tactile_sensor_{group}` | `(T,)` | Unicode string | Sensor name `moxian` | | `data/{side}_tactile_type_{group}` | `(T,)` | Unicode string | `state` for force; `binary` for contact | | `data/supplementary_tactile` | `(T,2,460)` | float64 | Left/right raw tactile-channel values, without a physical unit | | `data/sub_task_instruction` | `(T,)` | Unicode string | Frame-associated Chinese instruction | | `meta/episode_ends` | `(E,)` | int64 | Exclusive episode-end indices | There are 27 arrays under `data/`. String storage widths depend on batch content. RGB chunks have shape `(32,126,224,3)` and use lossless **Blosc Zstd**, compression level **3**, with **Bitshuffle**. The outer ZIP stores these compressed Zarr chunks without an additional compression layer. ## Time, images, and poses Frames are aligned by sequence and timestamp at a nominal rate of 60 Hz. Timestamps are absolute source times; subtract an episode's first timestamp when relative time is required. Process time-dependent data within episode boundaries, since timestamps can jump between recordings. Head RGB is decoded from source H.264 and resized from 1920 × 1080 to **224 × 126** using area interpolation. The image retains the orientation encoded in the source stream. Calibration accounts for that stream's 180° image rotation and the resize. The release images are not cropped or undistorted. Pose values are `[x, y, z, rx, ry, rz]`, with translation in **metres** and an axis-angle rotation vector in **radians**. The transform maps a local point to the shared right-handed world frame within its recording: `p_world = R(rotvec) @ p_local + translation`. Camera optical axes are x right, y down, and z forward. The source-to-release world basis is `diag(1,-1,1)`. Wrist positions use hand-skeleton joint 0, which includes the tracker-to-wrist reference-point offset. Camera poses combine the head pose with its camera extrinsic calibration. Each group's `camera_calibration_by_episode` attribute is keyed by zero-based episode index strings, such as `"0"`; it contains image intrinsics, distortion coefficients, source calibration, and the optical-frame convention. Hand angles follow the FTP-1 21-angle algorithm. Their FAAS IDs, in array order, are `[1,26,2,3,6,7,8,9,11,12,13,14,16,17,18,19,21,22,23,24,27]`. These arrays describe angle slots; 3D hand-landmark coordinates and source confidence/validity flags are not included. ## Tactile representation Each hand's six force and contact channels are ordered **palm, thumb, index, middle, ring, little**, with functional-area IDs **`[5,0,1,2,3,4]`**. Force values are calibrated regional normal forces in **N**, recorded as `normal_force_n`. Contact values are the Boolean `is_contact` signal, where `true` indicates contact. `supplementary_tactile[:,0,:]` is the left hand and `[:,1,:]` the right hand. Each hand has 460 slots indexed 0–459: **360 sensing points and 100 fixed unassigned slots**. The array stores unitless raw `pressureData` readings as float64, including fractional values and values in unassigned slots. [`supplementary_tactile_note.json`](supplementary_tactile_note.json) supplies each slot's region, row, column, display coordinates, and fixed layout mask. `valid_mask` is shared across all frames: `true` identifies sensing points and `false` identifies unassigned slots. Each 24-point finger group contains 16 fingertip and 8 middle-segment points; the `*_tip` region label identifies the whole finger group. The `u/v` values are dimensionless display-layout coordinates. The functional-area diagram shows area IDs and sensing-point counts. ## Download and read Download every part of the desired batch. Concatenate that batch's four-digit part sequence in filename order, then extract the ZIP: ```bash cat standard/ssd1_episode_batch_0000.zarr.zip.[0-9][0-9][0-9][0-9] > ssd1_batch_0000.zip unzip ssd1_batch_0000.zip ``` The result is `ssd1_episode_batch_0000.zarr/` with `data/` and `meta/`. For another batch, substitute its source and batch prefix. All parts of a batch are required; parts from different batches form separate archives. The following example uses Python, NumPy, and Zarr 2.x: ```python import numpy as np import zarr root = zarr.open_group("ssd1_episode_batch_0000.zarr", mode="r") ends = root["meta/episode_ends"][:] episode = 0 start = 0 if episode == 0 else int(ends[episode - 1]) stop = int(ends[episode]) rgb = root["data/camera_ego_rgb"][start:stop] timestamps = root["data/timestamps"][start:stop] relative_seconds = timestamps - timestamps[0] left_force_n = root["data/left_tactile_data_force"][start:stop, :, 0] left_adc = root["data/supplementary_tactile"][start:stop, 0, :] calibration = root.attrs["camera_calibration_by_episode"][str(episode)] ``` ## Verification The OXT structural check covers all **114 batches** and reports **114 structure passes**. It checks array schemas, episode boundaries, timestamps, constant indices, and instruction fields. The report's aggregate metadata status is `warning` for the `type` field: the joint-dimension heuristic produces `dexhand`, while the supplied metadata identifies these human glove recordings as `human`. The NAS and ModelScope inventories contain matching data-part names and sizes. Raw-tactile array metadata matches across both destinations in all 114 batches. Numerical samples comprise batch frames `[0,512)` of `ssd3_episode_batch_0000` (471,040 tactile values per destination) and source frames `[0,32)` of session `data_raw_20260829_104210_164` (29,440 tactile values, regional forces, and timestamps). Each sample matches the corresponding local converted data. Numerical results apply to these sampled intervals. ## License and attribution MATRIXGlove is distributed under [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/). See `LICENSE`. Attribute the dataset to Xspark AI and link to the dataset repository and project website.