Robust-Minisets

Abstract

We introduce Robust-Minisets, a collection of robust benchmark classification datasets in the low resolution realm based on well-established image classification benchmarks, such as CIFAR, Tiny ImageNet, EuroSAT and the MedMNIST collection. We port existing robustness and generalization benchmarks (ImageNet-C, -R, -A and v2) to the small dataset domain introducing novel benchmarks to comprehensively evaluate the robustness and generalization capabilities of image classification models on low resolution datsets. This results in an extensive collection consisting of already existing test sets (e.g. CIFAR-10.1 and Tiny ImageNet-C) as well as the novel benchmarks EuroSAT-C, MedMNIST-C, and Tiny ImageNet-A, -R and -v2 introduced in our ICPR 2024 paper "GenFormer - Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets".

Installation and Requirements

We recommend our official code to download, parse and use the Robust-Minisets datasets:

% pip install robust-minisets% python

To use a standard version utilizing the downloaded files:

>>> from robust_minisets import TinyImageNetR

>>> test_dataset = TinyImageNetR(split="test")

To enable automatic downloading by setting `download=True`:

>>> from robust_minisets import EuroSATC

>>> test_dataset = EuroSATC(split="test", download=True)

Additionally, we include training and validation datasets that are not provided in common hubs (e.g. torchvision):

>>> from robust_minisets import EuroSAT

>>> train_dataset = EuroSAT(split="train", download=True)

>>> val_dataset = EuroSAT(split="val", download=True)

>>> test_dataset = EuroSAT(split="test", download=True)

License

The code is under Apache-2.0 License.

The publication licenses of the datasets can be found within the info dictionary of our official code via robust_minisets.INFO[] or here.

Citation

If you find this work useful, please consider citing us:

Sven Oehri, Nikolas Ebert, Ahmed Abdullah, Didier Stricker, Oliver Wasenmüller. "GenFormer - Generated Images are All You Need to Improve Robustness on Small Datasets". International Conference on Pattern Recognition (ICPR), 2024.

or using bibtex:

@inproceedings{oehri2024genformer,    title = {GenFormer – Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets},    author = {Oehri, Sven and Ebert, Nikolas and Abdullah, Ahmed and Stricker, Didier and Wasenm{\"u}ller, Oliver},    booktitle = {International Conference on Pattern Recognition (ICPR)},    year = {2024},}

DISCLAIMER: Robust-Minisets is based on a wide range of existing datasets and benchmarks. Thus, please also cite source data paper(s) of the Robust-Minisets subset(s):

CIFAR-10.1

EuroSAT

ImageNet-A

ImageNet-C

ImageNet-R

ImageNetv2

MedMNIST, the respective source datasets (described here)

Release versions

v1.0.0: Robust-Minisets v1 release