gelsight-force-estimation

# Dataset Details

This dataset contains paired tactile and force data, intended for use in predicting 3-axis normal and shear forces applied to the sensor's elastomer. We used three different indenter shapes to collect force-labeled data: hemisphere, sharp, and flat. To measure force ground truths, we employed the ATI nano17 force/torque sensor. The protocol consisted of applying a random normal load (up to 3N) followed by a shear load, achieved by sliding the probe 2mm on the sensor's elastomer. An illustration of the protocol is shown below:


This dataset is part of TacBench for evaluating Sparsh touch representations. For more information, please visit https://sparsh-ssl.github.io/.

## Uses

This dataset includes aligned tactile data, 3-axis force, and slip labels using the GelSight Mini sensor (without markers). It is designed to evaluate the performance of [Sparsh encoders](https://huggingface.co/collections/facebook/sparsh-67167ce57566196a4526c328) in understanding tactile properties such as force estimation and slip detection. Note that slip labels {0: no_slip, 1:slip} were obtain indirectlty via the friction cone method.

For more information on how to use this dataset and set up corresponding downstream tasks, please refer to the [Sparsh repository](https://github.com/facebookresearch/sparsh).

## Dataset Structure

The dataset consists a collection of normal/shear load trajectories for each probe. The structure is as follows:

```bash
sphere
├── batch_1
│ ├── dataset_gelsight_00.pkl
│ ├── ...
│ ├── dataset_gelsight_03.pkl
│ ├── dataset_slip_forces.pkl
├── batch_2
│ ├── ...
flat
├── batch_1
│ ├── dataset_gelsight_00.pkl
│ ├── ...
│ ├── dataset_gelsight_03.pkl
│ ├── dataset_slip_forces.pkl
│ ...
sharp
├── ....
```

For each batch:
- `dataset_gelsight_xy.pkl`: contains the binarized tactile images only.
- `dataset_slip_forces.pkl`: it's a dictionary where each key represents a sliding trajectory. Each trajectory has the corresponding force and slip labels.

```python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('sphere/batch_1/dataset_gelsight_00.pkl')
img = load_bin_image(frames[0])
```

Please refer to [Sparsh repository](https://github.com/facebookresearch/sparsh) for further information about extracting the force/slip dataset.

## BibTeX entry and citation info

```bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={8th Annual Conference on Robot Learning},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}
```

# 数据集详情
本数据集包含配对的触觉-力数据，旨在用于预测施加于传感器弹性体的三维法向力与切向力。我们采用三种不同的触觉压头（indenter）形状采集带力标注的数据：半球形、尖形与平面形。为获取力的真值标签（ground truth），我们使用了ATI nano17型力/扭矩传感器（ATI nano17 force/torque sensor）。采集流程为先施加随机法向载荷（最大3N），再施加切向载荷，具体方式为将探针在传感器弹性体上滑动2毫米。该采集流程的示意图如下所示：
本数据集是用于评估Sparsh触觉表征的TacBench基准数据集的组成部分。如需了解更多信息，请访问https://sparsh-ssl.github.io/。

## 数据集用途
本数据集包含使用无标记凝胶视觉微型传感器（GelSight Mini）采集的对齐触觉数据、三维力数据与滑动标签（slip label）。其设计目的为评估Sparsh编码器（Sparsh encoders）在理解触觉属性（如力估计与滑动检测）方面的性能。请注意，滑动标签{0：无滑动，1：有滑动}是通过摩擦锥方法（friction cone method）间接获取的。如需了解本数据集的使用方法以及如何搭建对应的下游任务（downstream task），请参考[Sparsh代码仓库](https://github.com/facebookresearch/sparsh)。

## 数据集结构
本数据集按压头类型收纳了各压头对应的法向/切向载荷轨迹集合。其结构如下所示：
bash
sphere
├── batch_1
│ ├── dataset_gelsight_00.pkl
│ ├── ...
│ ├── dataset_gelsight_03.pkl
│ ├── dataset_slip_forces.pkl
├── batch_2
│ ├── ...
flat
├── batch_1
│ ├── dataset_gelsight_00.pkl
│ ├── ...
│ ├── dataset_gelsight_03.pkl
│ ├── dataset_slip_forces.pkl
│ ...
sharp
├── ....

针对每个批次：
- `dataset_gelsight_xy.pkl`：仅包含二值化触觉图像（binarized tactile image）。
- `dataset_slip_forces.pkl`：为一个字典，其中每个键对应一条滑动轨迹，每条轨迹包含对应的力与滑动标签。

python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('sphere/batch_1/dataset_gelsight_00.pkl')
img = load_bin_image(frames[0])

请参考[Sparsh代码仓库](https://github.com/facebookresearch/sparsh)以获取关于提取力/滑动数据集的更多信息。

## BibTeX引用条目与引用信息
bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: 面向视觉触觉传感的自监督触觉表征},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={第8届机器人学习年度会议},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}

搜集汇总

背景与挑战

背景概述

gelsight-force-estimation数据集是一个包含触觉图像与3轴力数据配对的数据集，主要用于力估计和滑动检测研究。数据集采用三种不同形状的压头采集，包含二值化触觉图像和对应的力/滑动标签数据，是评估Sparsh触觉表示性能的重要基准。