touch-slide

# Dataset Details

Touch-Slide is a dataset inspired by [YCB-Slide](https://github.com/rpl-cmu/YCB-Slide). Its purpose is to increase the amount of data from multiple DIGIT sensors for self-supervised learning (SSL) pre-training of the Sparsh model. Touch-Slide consists of human-sliding interactions on toy kitchen objects using the DIGIT sensor. We used 9 objects, as shown below, and collected 5 trajectories for each, resulting in a total of 180k frames.

<!-- insert figure from assets/touch_slide_objs.png -->

This is a visual example of how the dataset was collected, showcasing sliding interactions that capture trajectories rich in shear forces:

 

## Uses

This dataset does not include labels and is intended for self-supervised training only. It is specifically designed for training the Sparsh models listed in the Hugging Face [Sparsh collection](https://huggingface.co/collections/facebook/sparsh-67167ce57566196a4526c328).

Please refer to the [Sparsh repo](https://github.com/facebookresearch/sparsh) for further information about usage.

## Dataset Structure

The dataset consists of 5 trajectories for each object. Each trajectory is stored as a pickle file, containing binarized tactile images. The structure is as follows:

```bash
Touch-Slide
├── object_0 # eg: bread
│ ├── dataset_0.pkl
│ ├── ...
│ ├── dataset_4.pkl
├── object_1 # eg: corn
├── ...
```

This is a sample code for loading the dataset in pickle format and extracting the images:

```python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('bread/dataset_0.pkl')
img = load_bin_image(frames[0])
```

## BibTeX entry and citation info

```bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={8th Annual Conference on Robot Learning},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}
```

# 数据集详情

Touch-Slide数据集的设计灵感源自[YCB-Slide](https://github.com/rpl-cmu/YCB-Slide)，其核心目标是为Sparsh模型的自监督学习（Self-Supervised Learning，SSL）预训练提供更多来自多台DIGIT传感器的触觉数据。该数据集包含人类使用DIGIT传感器在玩具厨房器具上开展滑动交互的采集样本：共涵盖9类物体，每类物体采集5条轨迹，总计18万帧图像。

<!-- 插入来自assets/touch_slide_objs.png的图片 -->

以下为数据集采集过程的可视化示例，展示了富含剪切力的滑动交互轨迹：
 

## 数据集用途

本数据集不包含任何标注，仅用于自监督学习训练，专为训练Hugging Face平台上[Sparsh模型集合](https://huggingface.co/collections/facebook/sparsh-67167ce57566196a4526c328)中的Sparsh模型打造。如需了解更多使用相关细节，请参阅[Sparsh代码仓库](https://github.com/facebookresearch/sparsh)。

## 数据集结构

本数据集每类物体对应5条轨迹，每条轨迹以pickle文件格式存储，内含二值化触觉图像。数据集的目录结构如下：
bash
Touch-Slide
├── object_0 # 例如：面包（bread）
│ ├── dataset_0.pkl
│ ├── ...
│ ├── dataset_4.pkl
├── object_1 # 例如：玉米（corn）
├── ...

以下为加载pickle格式数据集并提取图像的示例代码：
python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('bread/dataset_0.pkl')
img = load_bin_image(frames[0])

## BibTeX引用条目与引文信息
bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={8th Annual Conference on Robot Learning},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}

搜集汇总

背景与挑战

背景概述

Touch-Slide数据集是一个用于自监督学习预训练的数据集，包含人类在9个玩具厨房对象上滑动交互的180k帧数据，每个对象有5个轨迹，数据以pickle文件格式存储。该数据集专为训练Sparsh模型设计，不包含标签。