iis-esslingen/SearchAD

---
license: cc-by-nc-sa-4.0
task_categories:
- text-to-image
- image-to-image
- object-detection
language:
- en
size_categories:
- 100M<n<1B
viewer: false
---

# SearchAD Dataset


## Main Project Page

You can find more information about the SearchAD Dataset on its official project page: [https://iis-esslingen.github.io/searchad/](https://iis-esslingen.github.io/searchad/)

## SearchAD Benchmark

The official SearchAD rare image retrieval competition including the leaderboard can be found [here](https://huggingface.co/spaces/SearchADBenchmark/SearchADLargeScaleRareImageRetrievalDatasetforAutonomousDriving)

## Dataset Overview

The SearchAD dataset is a large-scale autonomous driving datasets, specifically targeting rare and safety-critical objects and scenes. It's designed to provide a comprehensive and challenging environment for semantic image retrieval research. Due to the dataset licenses, the dataset images have to be downloaded at the official dataset hosts (see Table below).

* **Name:** SearchAD
* **Dataset Size:** **423,798 frames (images)**.
* **Origin:** Uniquely compiled by integrating data from **11 established AD datasets**, ensuring diversity and real-world variability.
| Dataset [Download Link] and Instructions | Val. Set | #Frames | # Original Classes | # SearchAD Classes | # Objects |
| :----------------------- | :--------: | :-------: | :----------------: | :-----------------: | :---------: |
| Lost and Found [[1]](https://wwwlehre.dhbw-stuttgart.de/~sgehrig/lostAndFoundDataset/index.html#:~:text=leftImg8bit.zip%20(6GB)%20left%208%2Dbit%20images%20%2D%20train%20and%20test%20set%20(2104%20images)) - Then download **leftImg8bit/** | X | 2,239 | 42 | 18 | 2,098 |
| WildDash2 [[2]](https://www.wilddash.cc/accounts/login?next=/download) - Then download **wd_public_v2p0.zip** and **wd_both_02.zip** | ✓ | 5,068 | 26 | 80 | 5,032 |
| ACDC [[3]](https://acdc.vision.ee.ethz.ch/download#:~:text=rgb_anon_trainvaltest.zip) - Then download **rgb_anon_trainvaltest.zip** | ✓ | 8,012 | 19\* | 60 | 7,471 |
| IDD Segmentation [[4]](https://idd.insaan.iiit.ac.in/accounts/login/?next=/dataset/download/) - Then download **IDD Segmentation (IDD 20k Part I) (18.5 GB)** | ✓ | 10,003 | 30\* | 52 | 12,192 |
| KITTI [[5]](https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d#:~:text=Download%20left%20color%20images%20of%20object%20data%20set%20(12%20GB)) - Then download **left color images of object data set (12 GB)** | X | 14,999 | 8 | 47 | 9,840 |
| Cityscapes [[6]](https://www.cityscapes-dataset.com/downloads/) - Then download **leftImg8bit_trainvaltest.zip (11GB) [md5]** and **leftImg8bit_trainextra.zip (44GB) [md5]** | ✓ | 24,998 | 30\* | 75 | 31,037 |
| Mapillary Vistas [[7]](https://www.mapillary.com/dataset/vistas) - Then download **mapillary-vistas-dataset_public_v2.0.zip** | ✓ | 25,000 | 66\* | 86 | 35,093 |
| ECP [[8]](https://eurocity-dataset.tudelft.nl/eval/downloads/detection) - Then download **ECP day and night, train, val, test (6 download .zip files)**| ✓ | 47,335 | 8 | 76 | 33,081 |
| nuScenes [[9]](https://www.nuscenes.org/nuscenes#download) - Then download **Trainval and Test** | ✓ | 80,314 | 32\* | 56 | 166,152 |
| BDD100K [[10]](http://bdd-data.berkeley.edu/download.html) - Then download **100K Images** | ✓ | 100,000 | 12\* | 80 | 83,102 |
| Mapillary Sign [[11]](https://www.mapillary.com/dataset/trafficsign) - Then click on **Download dataset** | ✓ | 105,830 | 313\*\* | 90 | 128,167 |
| SearchAD [[12]](https://cdn-icons-png.flaticon.com/512/48/48639.png) | Combined | 423,798 | N/A | 90 | 513,265 |
### SearchAD Class Overview
* **Annotations:** Features more than **513,265 high-quality manual bounding box annotations** across **90 rare classes**.
* **Categories:** The 90 rare classes are grouped into broader categories:
| Category | SearchAD Classes |
|---|---|
| Animal | Real: Cat, Cow, Deer, Dog, Donkey, Horse, Sheep, Wildlife |
| | Statue: Cow, Deer, Elephant, Horse, Lion |
| Human | Construction Worker, Firefighter, Medical, On Loading Area, Police, Refuse Collector, With Sticks or Crutches |
| Marking | Bicycle Symbol, Bus Text, Stop Text, Temporarily Invalidated, Yellow Lane Arrow |
| Object | Ball, Beacon, Euro Pallet, Hand Dolly, Hydrant, Office Chair, Pallet Truck, Platform Truck, Rollator, |
| | Shopping Cart, Shopping Trolley, Suitcase Trolley, Traffic Cone, Trash Bin, Wheelbarrow |
| Rideable | Cityscooter, Police Motorcycle, Quad, Segway, Skateboard, Skates, Ski, Stroller, Three Wheeler, Toy Car, Wheelchair |
| Scene | Active Amber Lights, Active Emergency Lights, Fog, Open Door, Open Hood, Open Trunk, Snow, Tunnel |
| Sign | Animal Sign, Road Bumper Sign, Temporarily Invalidated Sign, Train Sign, Warning Triangle |
| Trailer | Agricultural Trailer, Bicycle Trailer, Boat Trailer, Car Trailer, Caravan Trailer, Carriage, Warning Trailer |
| Vehicle | Construction: Concrete Mixer, Excavator, Forklift, Harvester, Loader, Steamroller, Tractor, Truck Crane |
| | Duty: Fire, Garbage, Medical, Military, Police, Winter |
| | Special: Bicycle On Back, Bicycle On Roof, Car Truck, Recreational, Train |

### Dataset Structure

Please note the following regarding the dataset structure:

* This is the **default structure** assumed for the dataset.
* It is specifically used by the **annotation JSON files** and the **default queries vision support set image paths**.
* If you use a different dataset structure or different dataset names (e.g., bdd100k instead of bdd100k_images_100k), the datasets must be either **symlinked** or the corresponding image paths must be **modified within the annotation files and default queries vision support sets**.
* This structure is crucial for **correct submission on the benchmark server**.

```
searchad/
├── ECP/
│ ├── ...
├── IDD_Segmentation/
│ ├── ...
├── acdc/
│ ├── ...
├── bdd100k_images_100k/
│ ├── ...
├── cityscapes/
│ ├── ...
├── kitti/
│ ├── ...
├── lostandfound/
│ ├── ...
├── mapillary_sign/
│ ├── ...
├── mapillary_vistas/
│ ├── ...
├── nuscenes/
│ ├── ...
├── wd_both02/
│ ├── ...
├── wd_publicv2p0/
│ ├── ...
├── searchad_annotations_train.json
├── searchad_annotations_val.json
├── searchad_test_mapping_id_to_imagepath.json
└── default_queries/
├── ...
```

### Setup for Evaluation

* **Data Splits:** SearchAD provides distinct **training, validation, and a held-out test set**.
* The **test set** is constructed from the union of test splits of the underlying datasets and is hosted on a private benchmark server to prevent any form of test leakage and ensure unbiased evaluation.
* Training and validation splits are derived from the respective partitions of the original datasets, with all 90 SearchAD classes represented in each split.
* **Query Modalities:** The benchmark supports two primary query types:
* **Text-based Queries:** Consist of precise keywords defining the class of interest, complemented by comprehensive, extended descriptions that offer detailed characterization.
* **Image-based Queries:** Utilize a **Vision Support Set** of 5 carefully selected reference images per class. These images are chosen from the training set based on size, variance, and low occlusion to represent diverse variations.