ABC-iRobotics/oe_dataset

---
language:
- en
license: gpl-3.0
tags:
- vision
- image segmentation
- instance segmentation
- object detection
- synthetic
- sim-to-real
annotations_creators:
- machine-generated
pretty_name: OE Dataset
size_categories:
- 1K<n<10K
task_categories:
- object-detection
- image-segmentation
- robotics
task_ids:
- instance-segmentation
- semantic-segmentation
---

# The OE Dataset!


A dataset consisting of synthetic and real images annotated with instance segmentation masks for testing sim-to-real model performance for robotic manipulation.

### Dataset Summary

The OE Dataset is a collection of synthetic and real images of 3D-printed OE logos. Each image is annotated with instance segmentation masks. The dataset explicitly marks synthetic samples based on their creation method (either photorealistic synthetic samples or domain randomized samples) to facilitate sim-to-real performance tests on different synthetic datasets.

### Supported Tasks and Leaderboards

The dataset supports tasks such as semantic segmentation, instance segmentation, object detection, image classification, and testing sim-to-real transfer.

## Dataset Structure

### Data Instances

The instances of the dataset are 1920x1080x3 images in PNG format. The annotations are 1920x1080x4 PNG images representing the instance segmentation masks, where each instance is associated with a unique color.

### Data Fields

The data fields are:

1) 'image': 1920x1080x3 PNG image
2) 'mask': 1920x1080x4 PNG image

### Data Splits

The dataset contains training and validation splits for all image collections (real images, photorealistic synthetic images, domain randomized synthetic images) to facilitate cross-domain testing.

## Dataset Creation

### Curation Rationale

The dataset was created to provide a testbed for examining the effects of fine-tuning instance segmentation models on synthetic data (using various sim-to-real approaches).

### Source Data

The data is generated using two methods:

- Real images are recorded using a robotic setup and automatically annotated using the method proposed in [[1]](https://ieeexplore.ieee.org/abstract/document/9922852)
- Synthetic samples are generated using Blender and annotated using the [Blender Annotation Tool (BAT)](https://github.com/ABC-iRobotics/blender_annotation_tool)

### Citation Information

OE Dataset:
```bibtex
@ARTICLE{10145828,
author={Károly, Artúr István and Tirczka, Sebestyén and Gao, Huijun and Rudas, Imre J. and Galambos, Péter},
journal={IEEE Transactions on Cybernetics},
title={Increasing the Robustness of Deep Learning Models for Object Segmentation: A Framework for Blending Automatically Annotated Real and Synthetic Data},
year={2023},
volume={},
number={},
pages={1-14},
doi={10.1109/TCYB.2023.3276485}}
```

Automatically annotating real images with instance segmentation masks using a robotic arm:
```bibtex
@INPROCEEDINGS{9922852,
author={Károly, Artúr I. and Károly, Ármin and Galambos, Péter},
booktitle={2022 IEEE 10th Jubilee International Conference on Computational Cybernetics and Cyber-Medical Systems (ICCC)},
title={Automatic Generation and Annotation of Object Segmentation Datasets Using Robotic Arm},
year={2022},
volume={},
number={},
pages={000063-000068},
doi={10.1109/ICCC202255925.2022.9922852}}
```

Synthetic dataset generation and annotation method:
```bibtex
@INPROCEEDINGS{9780790,
author={Károly, Artúr I. and Galambos, Péter},
booktitle={2022 IEEE 20th Jubilee World Symposium on Applied Machine Intelligence and Informatics (SAMI)},
title={Automated Dataset Generation with Blender for Deep Learning-based Object Segmentation},
year={2022},
volume={},
number={},
pages={000329-000334},
doi={10.1109/SAMI54271.2022.9780790}}
```

Other related publications:
```bibtex
@INPROCEEDINGS{10029564,
author={Károly, Artúr I. and Tirczka, Sebestyén and Piricz, Tamás and Galambos, Péter},
booktitle={2022 IEEE 22nd International Symposium on Computational Intelligence and Informatics and 8th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (CINTI-MACRo)},
title={Robotic Manipulation of Pathological Slides Powered by Deep Learning and Classical Image Processing},
year={2022},
volume={},
number={},
pages={000387-000392},
doi={10.1109/CINTI-MACRo57952.2022.10029564}}
```

```bibtex
@Article{app13010525,
AUTHOR = {Károly, Artúr István and Galambos, Péter},
TITLE = {Task-Specific Grasp Planning for Robotic Assembly by Fine-Tuning GQCNNs on Automatically Generated Synthetic Data},
JOURNAL = {Applied Sciences},
VOLUME = {13},
YEAR = {2023},
NUMBER = {1},
ARTICLE-NUMBER = {525},
URL = {https://www.mdpi.com/2076-3417/13/1/525},
ISSN = {2076-3417},
ABSTRACT = {In modern robot applications, there is often a need to manipulate previously unknown objects in an unstructured environment. The field of grasp-planning deals with the task of finding grasps for a given object that can be successfully executed with a robot. The predicted grasps can be evaluated according to certain criteria, such as analytical metrics, similarity to human-provided grasps, or the success rate of physical trials. The quality of a grasp also depends on the task which will be carried out after the grasping is completed. Current task-specific grasp planning approaches mostly use probabilistic methods, which utilize categorical task encoding. We argue that categorical task encoding may not be suitable for complex assembly tasks. This paper proposes a transfer-learning-based approach for task-specific grasp planning for robotic assembly. The proposed method is based on an automated pipeline that quickly and automatically generates a small-scale task-specific synthetic grasp dataset using Graspit! and Blender. This dataset is utilized to fine-tune pre-trained grasp quality convolutional neural networks (GQCNNs). The aim is to train GQCNNs that can predict grasps which do not result in a collision when placing the objects. Consequently, this paper focuses on the geometric feasibility of the predicted grasps and does not consider the dynamic effects. The fine-tuned GQCNNs are evaluated using the Moveit! Task Constructor motion planning framework, which enables the automated inspection of whether the motion planning for a task is feasible given a predicted grasp and, if not, which part of the task is responsible for the failure. Our results suggest that fine-tuning GQCNN models can result in superior grasp-planning performance (0.9 success rate compared to 0.65) in the context of an assembly task. Our method can be used to rapidly attain new task-specific grasp policies for flexible robotic assembly applications.},
DOI = {10.3390/app13010525}
}
```

原始信息汇总

OE Dataset 数据集概述

数据集简介

OE Dataset 是一个包含合成和真实图像的数据集，这些图像带有实例分割掩码，用于测试机器人操作中从模拟到真实的模型性能。数据集中的图像主要是3D打印的OE标志，每个图像都带有实例分割掩码。数据集明确标记了合成样本的创建方法（写实合成样本或域随机化样本），以便于在不同的合成数据集上进行从模拟到真实的性能测试。

支持的任务和排行榜

该数据集支持的任务包括语义分割、实例分割、目标检测、图像分类以及从模拟到真实的转移测试。

数据集结构

数据实例

数据集中的实例是1920x1080x3的PNG格式图像。注释是1920x1080x4的PNG图像，表示实例分割掩码，每个实例与一个唯一的颜色关联。

数据字段

数据字段包括：

1. image: 1920x1080x3的PNG图像
2. mask: 1920x1080x4的PNG图像

数据分割

数据集包含训练和验证分割，适用于所有图像集合（真实图像、写实合成图像、域随机化合成图像），以便于跨域测试。

数据集创建

创建理由

该数据集的创建旨在提供一个测试平台，用于检查在合成数据上微调实例分割模型的效果（使用不同的从模拟到真实的方法）。

源数据

数据生成使用两种方法：

• 真实图像通过机器人设置录制，并使用[1]中提出的方法自动注释。
• 合成样本使用Blender生成，并使用Blender Annotation Tool (BAT)进行注释。

引用信息

OE Dataset: bibtex @ARTICLE{10145828, author={Károly, Artúr István and Tirczka, Sebestyén and Gao, Huijun and Rudas, Imre J. and Galambos, Péter}, journal={IEEE Transactions on Cybernetics}, title={Increasing the Robustness of Deep Learning Models for Object Segmentation: A Framework for Blending Automatically Annotated Real and Synthetic Data}, year={2023}, volume={}, number={}, pages={1-14}, doi={10.1109/TCYB.2023.3276485}}

自动注释真实图像的实例分割掩码使用机器人臂： bibtex @INPROCEEDINGS{9922852, author={Károly, Artúr I. and Károly, Ármin and Galambos, Péter}, booktitle={2022 IEEE 10th Jubilee International Conference on Computational Cybernetics and Cyber-Medical Systems (ICCC)}, title={Automatic Generation and Annotation of Object Segmentation Datasets Using Robotic Arm}, year={2022}, volume={}, number={}, pages={000063-000068}, doi={10.1109/ICCC202255925.2022.9922852}}

合成数据集生成和注释方法： bibtex @INPROCEEDINGS{9780790, author={Károly, Artúr I. and Galambos, Péter}, booktitle={2022 IEEE 20th Jubilee World Symposium on Applied Machine Intelligence and Informatics (SAMI)}, title={Automated Dataset Generation with Blender for Deep Learning-based Object Segmentation}, year={2022}, volume={}, number={}, pages={000329-000334}, doi={10.1109/SAMI54271.2022.9780790}}