NINCO (Out-Of-Distribution detection dataset for ImageNet)

The <strong>NINCO</strong> (<strong>N</strong>o <strong>I</strong>mageNet <strong>C</strong>lass <strong>O</strong>bjects) dataset is introduced in the ICML 2023 paper<strong> </strong>In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation. The images in this dataset are free from objects that belong to any of the 1000 classes of ImageNet-1K (ILSVRC2012), which makes NINCO suitable for evaluating out-of-distribution detection on ImageNet-1K . The NINCO main dataset consists of 64 OOD classes with a total of 5879 samples. These OOD classes were selected to have no categorical overlap with any classes of ImageNet-1K. Each sample was inspected individually by the authors to not contain ID objects. Besides NINCO, included are (in the same .tar.gz file) <strong>truly OOD versions of 11 popular OOD datasets</strong> with in total 2715 OOD samples. Further included are <strong>17 OOD unit-tests</strong>, with 400 samples each. Code for loading and evaluating on each of the three datasets is provided at <strong>https://github.com/j-cb/NINCO.</strong> When using <strong>NINCO</strong>, please consider citing (besides the bibtex given below) the following data sources that were used to create NINCO: Hendrycks et al.: ”Scaling out-of-distribution detection for real-world settings”, ICML, 2022. Bossard et al.: ”Food-101 – mining discriminative components with random forests”, ECCV 2014. Zhou et al.: ”Places: A 10 million image database for scene recognition”, IEEE PAMI 2017. Huang et al.: ”Mos: Towards scaling out-of-distribution detection for large semantic space”, CVPR 2021. Li et al.: ”Caltech 101 (1.0)”, 2022. Ismail et al.: ”MYNursingHome: A fully-labelled image dataset for indoor object classification.”, Data in Brief (V. 32) 2020. The iNaturalist project: https://www.inaturalist.org/ When using <strong>NINCO_popular_datasets_subsamples</strong>, additionally to the above, please consider citing: Cimpoi et al.: ”Describing textures in the wild”, CVPR 2014. Hendrycks et al.: ”Natural adversarial examples”, CVPR 2021. Wang et al.: ”Vim: Out-of-distribution with virtual-logit matching”, CVPR 2022. Bendale et al.: ”Towards Open Set Deep Networks”, CVPR 2016. Vaze et al.: ”Open-set Recognition: a Good Closed-set Classifier is All You Need?”, ICLR 2022. Wang et al.: ”Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition.” ICML, 2022. Galil et al.: “A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet”, ICLR 2023.<br> For citing our paper, we would appreciate using the following bibtex entry (this will be updated once the ICML 2023 proceedings are public): <br> @inproceedings{<br> bitterwolf2023ninco,<br> title={In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation},<br> author={Julian Bitterwolf and Maximilian Mueller and Matthias Hein},<br> booktitle={ICML},<br> year={2023},<br> url={https://proceedings.mlr.press/v202/bitterwolf23a.html}<br> }