digit-force-estimation

# Dataset Details

This dataset contains paired tactile and force data, intended for use in predicting 3-axis normal and shear forces applied to the sensor's elastomer. We used three different indenter shapes to collect force-labeled data: hemisphere, sharp, and flat. To measure force ground truths, we employed the ATI nano17 force/torque sensor. The protocol consisted of applying a random normal load (up to 5N) followed by a shear load, achieved by sliding the probe 2mm on the sensor's elastomer. An illustration of the protocol is shown below:


This dataset is part of TacBench for evaluating Sparsh touch representations. For more information, please visit https://sparsh-ssl.github.io/.

## Uses

This dataset includes aligned tactile data, 3-axis force, and slip labels using the DIGIT sensor. It is designed to evaluate the performance of [Sparsh encoders](https://huggingface.co/collections/facebook/sparsh-67167ce57566196a4526c328) in understanding tactile properties such as force estimation and slip detection. Note that slip labels {0: no_slip, 1:slip} were obtain indirectlty via the friction cone method.

For more information on how to use this dataset and set up corresponding downstream tasks, please refer to the [Sparsh repository](https://github.com/facebookresearch/sparsh).

## Dataset Structure

The dataset consists a collection of normal/shear load trajectories for each probe. The structure is as follows:

```bash
sphere
├── batch_1
│ ├── dataset_digit_00.pkl
│ ├── ...
│ ├── dataset_digit_03.pkl
│ ├── dataset_slip_forces.pkl
├── batch_2
│ ├── ...
flat
├── batch_1
│ ├── dataset_digit_00.pkl
│ ├── ...
│ ├── dataset_digit_03.pkl
│ ├── dataset_slip_forces.pkl
│ ...
sharp
├── ....
```

For each batch:
- `dataset_digit_xy.pkl`: contains the binarized tactile images only.
- `dataset_slip_forces.pkl`: it's a dictionary where each key represents a sliding trajectory. Each trajectory has the corresponding force and slip labels.

```python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('sphere/batch_1/dataset_digit_00.pkl')
img = load_bin_image(frames[0])
```

Please refer to [Sparsh repository](https://github.com/facebookresearch/sparsh) for further information about extracting the force/slip dataset.

## BibTeX entry and citation info

```bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={8th Annual Conference on Robot Learning},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}
```

# 数据集详情

本数据集包含成对的触觉-力数据，旨在用于预测施加于传感器弹性体的三维法向与剪切力。我们采用三种不同的压头形状采集带力标注的数据集：半球形、尖锐形与扁平形。为获取力的地面真值，我们使用了ATI nano17力扭矩传感器。实验流程为先施加随机法向载荷（最大5N），随后施加剪切载荷：通过使探针在传感器弹性体表面滑动2mm完成该流程。流程示意图如下：


本数据集是用于评估Sparsh触觉表征（Sparsh touch representations）的TacBench基准数据集的组成部分。如需了解更多详情，请访问https://sparsh-ssl.github.io/。

## 数据集用途

本数据集包含使用DIGIT传感器采集的对齐触觉数据、三维力数据与滑移标注。其设计目标为评估[Sparsh编码器（Sparsh encoders）](https://huggingface.co/collections/facebook/sparsh-67167ce57566196a4526c328)在理解触觉特性（如力估计与滑移检测）方面的性能。请注意，滑移标注{0: no_slip, 1:slip}是通过摩擦锥方法间接获取的。

如需了解更多关于本数据集的使用方法以及搭建对应下游任务的细节，请参考[Sparsh代码仓库](https://github.com/facebookresearch/sparsh)。

## 数据集结构

本数据集按压头类型整理了多组法向/剪切载荷轨迹，整体目录结构如下：
bash
sphere
├── batch_1
│ ├── dataset_digit_00.pkl
│ ├── ...
│ ├── dataset_digit_03.pkl
│ ├── dataset_slip_forces.pkl
├── batch_2
│ ├── ...
flat
├── batch_1
│ ├── dataset_digit_00.pkl
│ ├── ...
│ ├── dataset_digit_03.pkl
│ ├── dataset_slip_forces.pkl
│ ...
sharp
├── ....

其中：
- 根目录下的`sphere`、`flat`、`sharp`分别对应三种压头类型的数据集目录
- 每个压头目录下包含多个批次目录（如`batch_1`、`batch_2`等）
- 每个批次目录内的`dataset_digit_xy.pkl`文件仅包含二值化触觉图像
- `dataset_slip_forces.pkl`为一个字典，其中每个键对应一条滑动轨迹，每条轨迹包含对应的力与滑移标注。

以下为数据集加载的示例代码：
python
def load_pickle_dataset(file_dataset):
with open(file_dataset, "rb") as f:
all_frames = pickle.load(f)
return all_frames

def load_bin_image(io_buf):
img = Image.open(io.BytesIO(io_buf))
img = np.array(img)
return img

frames = load_pickle_dataset('sphere/batch_1/dataset_digit_00.pkl')
img = load_bin_image(frames[0])

如需进一步了解如何提取力与滑移数据集的详细信息，请参考[Sparsh代码仓库](https://github.com/facebookresearch/sparsh)。

## BibTeX引用信息

bibtex
@inproceedings{
higuera2024sparsh,
title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
booktitle={8th Annual Conference on Robot Learning},
year={2024},
url={https://openreview.net/forum?id=xYJn2e1uu8}
}