sparsh-x-dataset

# Dataset Details

This dataset contains the sequences used for training [Sparsh-X](https://akashsharma02.github.io/sparsh-x-ssl/), a multisensory touch encoder for the [Digit 360](https://digit.ml/) sensor. Sparsh-X allows to fuse multiple touch modalities into a single embedding, such as tactile image, audio from contact microphones, IMU and pressure data.

Our Sparsh-X training dataset is generated from two primary sources: an Allegro hand with Digit 360 sensors on the fingertips that performs random motions with objects such as dipping into a tray filled with various items; and a manual picker with the same sensor adapted to the gripping mechanism, used to execute atomic manipulation actions such as picking up, sliding, tapping, placing, and dropping objects against diverse surfaces that vary in roughness, hardness, softness, friction, and texture properties. An example of the data collection is shown below:

<div align="center">
<img src="assets/object_action.gif" height="300" alt="D360 data collection">
</div>

## Dataset Structure and Usage

Each sequence contains a `data.pickle` and `metadata.yaml`. The pickle file has the following structure, with the raw messages per Digit 360 and tactile modality:

```bash
├── data.pickle
│ ├── d360_0
│ | ├── image_raw/compressed # list of msgs with tactile image @30Hz
│ | ├── imu_quat_topic # list of msgs with IMU quaternion data @400Hz
│ | ├── imu_raw_topic # list of msgs with raw 3-axis accelerometer data @400Hz
│ | ├── mic_0 # list of time-series messages from contact microphone @48kHz
│ | ├── mic_1 # list of time-series messages from contact microphone @48kHz
│ | ├── pressure_topic # list of time-series messages from pressure @200Hz
```

You can load the pickle file as:
```python
import pandas as pd
sequence_path = <PATH TO SEQUENCE>
ros_msgs = pd.read_pickle(os.path.join(sequence, "data.pickle"))
print(ros_msgs.keys())
```
And the output should be the key name for each Digit 360 qith recorded data:
```bash
> dict_keys(['d360_0', 'd360_1', 'd360_2', 'd360_3'])
```

The `metadata.yaml` for each sequence contains useful labels for Supervised Learninf downstream tasks. It describes the object in direct contact with the sensor (e.g. tennis ball), the surface the object is making contact with (e.g. grass), and the action that was perform during the data collection (e.g. tapping). Be aware that not all sequences have the three labels, for instance may not have the surface label.

Please refer to [Sparsh-X repo](https://github.com/facebookresearch/sparsh-multisensory-touch) for further information about using the dataset for SSL training. Some useful pointers:

- [List](https://github.com/facebookresearch/sparsh-multisensory-touch/config/data/d360.yaml) of sequences for SSL training
- [D360 class](https://github.com/facebookresearch/sparsh-multisensory-touch/tactile_ssl/data/d360/d360.py) for dataloader

## Citation

If you find this dataset useful for your research, please consider citing:

```bibtex
@inproceedings{
higuera2025tactile,
title={Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation},
author={Carolina Higuera and Akash Sharma and Taosha Fan and Chaithanya Krishna Bodduluri and Byron Boots and Michael Kaess and Mike Lambeta and Tingfan Wu and Zixi Liu and Francois Robert Hogan and Mustafa Mukadam},
booktitle={9th Annual Conference on Robot Learning},
year={2025},
url={https://openreview.net/forum?id=sMs4pJYhWi}
}
```

```bibtex
@article{lambeta2024digitizing,
title={Digitizing touch with an artificial multimodal fingertip},
author={Lambeta, Mike and Wu, Tingfan and Sengul, Ali and Most, Victoria Rose and Black, Nolan and Sawyer, Kevin and Mercado, Romeo and Qi, Haozhi and Sohn, Alexander and Taylor, Byron and others},
journal={arXiv preprint arXiv:2411.02479},
year={2024}
}
```

# 数据集详情

本数据集包含用于训练[Sparsh-X（https://akashsharma02.github.io/sparsh-x-ssl/）]的序列，Sparsh-X是针对[Digit 360（https://digit.ml/）]传感器设计的多模态触觉编码器。该模型可将多种触觉模态融合为单一嵌入向量，涵盖触觉图像、接触麦克风采集的音频、惯性测量单元（IMU）数据以及压力数据。

我们的Sparsh-X训练数据集源自两类主要采集场景：一是指尖搭载Digit 360传感器的Allegro机械手，该机械手可对各类物体执行随机运动，例如将物体浸入装满多样物品的托盘；二是适配了抓取机构的同款传感器手动拾取装置，用于执行基础操控动作，包括拾取、滑动、轻叩、放置以及将物体放置在粗糙度、硬度、柔软度、摩擦力和纹理属性各异的不同表面上并掉落。数据采集示例如下：

<div align="center">
<img src="assets/object_action.gif" height="300" alt="D360数据采集">
</div>

## 数据集结构与使用方法

每个序列均包含`data.pickle`与`metadata.yaml`文件。pickle文件具有如下结构，存储了每个Digit 360传感器及触觉模态的原始消息：
bash
├── data.pickle
│ ├── d360_0
│ | ├── image_raw/compressed # 30Hz帧率的触觉图像原始消息列表
│ | ├── imu_quat_topic # 400Hz帧率的IMU四元数数据消息列表
│ | ├── imu_raw_topic # 400Hz帧率的三轴原始加速度计数据消息列表
│ | ├── mic_0 # 48kHz采样率的接触麦克风0号时序消息列表
│ | ├── mic_1 # 48kHz采样率的接触麦克风1号时序消息列表
│ | ├── pressure_topic # 200Hz帧率的压力数据时序消息列表

可通过如下方式加载pickle文件：
python
import pandas as pd
sequence_path = <SEQUENCE路径>
ros_msgs = pd.read_pickle(os.path.join(sequence, "data.pickle"))
print(ros_msgs.keys())

输出结果应为每个已采集数据的Digit 360传感器对应的键名：
bash
> dict_keys(['d360_0', 'd360_1', 'd360_2', 'd360_3'])

每个序列对应的`metadata.yaml`文件包含可用于监督学习下游任务的实用标签。该文件会标注与传感器直接接触的物体（例如网球）、物体接触的表面（例如草地）以及数据采集过程中执行的动作（例如轻叩）。请注意，并非所有序列都包含这三类标签，例如部分序列可能未提供表面标签。

如需了解将该数据集用于自监督学习（SSL）训练的更多细节，请参阅[Sparsh-X代码仓库（https://github.com/facebookresearch/sparsh-multisensory-touch）]。以下为实用参考链接：
- 自监督学习训练用序列列表（https://github.com/facebookresearch/sparsh-multisensory-touch/config/data/d360.yaml）
- 数据加载器所用的D360类（https://github.com/facebookresearch/sparsh-multisensory-touch/tactile_ssl/data/d360/d360.py）

## 引用

若您的研究中用到了本数据集，请引用如下文献：

bibtex
@inproceedings{
higuera2025tactile,
title={Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation},
author={Carolina Higuera and Akash Sharma and Taosha Fan and Chaithanya Krishna Bodduluri and Byron Boots and Michael Kaess and Mike Lambeta and Tingfan Wu and Zixi Liu and Francois Robert Hogan and Mustafa Mukadam},
booktitle={9th Annual Conference on Robot Learning},
year={2025},
url={https://openreview.net/forum?id=sMs4pJYhWi}
}

bibtex
@article{lambeta2024digitizing,
title={Digitizing touch with an artificial multimodal fingertip},
author={Lambeta, Mike and Wu, Tingfan and Sengul, Ali and Most, Victoria Rose and Black, Nolan and Sawyer, Kevin and Mercado, Romeo and Qi, Haozhi and Sohn, Alexander and Taylor, Byron and others},
journal={arXiv preprint arXiv:2411.02479},
year={2024}
}