The Insect Hotel Dataset: A photorealistic synthetic dataset for pose estimation and panoptic segmentation

The Insect Hotel Dataset is a photorealistic synthetic dataset designed for pose estimation and panoptic segmentation tasks. It contains 20,000 synthetically generated photorealistic images of objects used in a human-robot collaborative assembly scenario. The dataset was created using NViSII. It also includes the 3D object meshes and YOLOv8 model weights.

This dataset accompanies the following upcoming publication:

Juan Carlos Saborío, Marc Vinci, Oscar Lima, Sebastian Stock, Lennart Niecksch, Martin Günther, Joachim Hertzberg, and Martin Atzmüller (2025): “Uncertainty-Resilient Active Intention Recognition for Robotic Assistants”. (submitted)

File Structure

To facilitate easier downloading, the dataset has been split into 10 parts. Each part is further divided into three archives:

RGB images + JSON annotations

Depth images (optional)

Instance segmentation images (optional)

To use the complete dataset, download all 30 archives and extract them into the same root folder, so that the depth and segmentation images are located alongside the corresponding RGB and JSON files.

The dataset format (coordinate systems, conventions, and JSON fields) follows the structure documented here.

Contents of the archives:

.├── insect_hotel_20k_00.tgz # RGB images + annotation JSON files│   └── 00 # archive index (00...09)│       ├── 0000 # scene index (0000...0099), each with 20 images in front of the same background│       │   ├── 00000.jpg # RGB image│       │   ├── 00000.json # pose, bounding boxes, etc.│       │   ├── [...]│       │   ├── 00019.jpg│       │   ├── 00019.json│       │   ├── _camera_settings.json # camera intrinsics│       │   └── _object_settings.json # object metadata│       ├── [...]│       └── 0099├── insect_hotel_20k_00.depth.tgz # Depth images (.exr)│   └── 00│       └── 0000│           ├── 00000.depth.exr│           └── [...]├── insect_hotel_20k_00.seg.tgz # Instance segmentation images (.exr)│   └── 00│       └── 0000│           ├── 00000.seg.exr│           └── [...]└── insect_hotel_20k_01.tgz   └── 01       └── 0000           ├── 00000.jpg           ├── 00000.json           └── [...]

3D Meshes

The file meshes.tgz contains all object meshes used for training.

Insect hotel parts (used in the assembly task)

bright_green_part

dark_green_part

magenta_part

purple_part

red_part

yellow_part

Other objects

klt — “Kleinladungsträger” (small load carrier / blue box)

multimeter

power_drill_with_grip

relay

screwdriver

Additionally, the images include various distractor objects from the Google Scanned Objects (GSO) dataset. The corresponding meshes are not included here but can be obtained directly from the GSO dataset.

YOLOv8 Model

The file yolov8_weights.tgz contains a YOLOv8 model that was trained on a subset of the object classes. The class index mapping is as follows:

0: bright_green_part1: dark_green_part2: magenta_part3: purple_part4: red_part5: yellow_part6: klt

Helper utilities for converting the DOPE format to YOLO format, along with scripts for training, inference, and visualization, are available via:

git clone -b insect_hotel https://github.com/DFKI-NI/yolo8_keypoint_utils.git