imagenet-12k-wds

## Dataset Description

- **Homepage:** https://image-net.org/index.php
- **Repository:** https://github.com/rwightman/imagenet-12k
- **Paper:** https://arxiv.org/abs/1409.0575

### Dataset Summary

This is a filtered copy of the full [ImageNet](https://www.image-net.org/) dataset consisting of the top 11821 (of 21841) classes by number of samples. It has been used to pretrain a number of `in12k` models in `timm`.

The code and metadata for building this dataset from the original full ImageNet can be found at https://github.com/rwightman/imagenet-12k

NOTE: This subset was filtered from the original `fall11` ImageNet release which has been replaced by the `winter21` release which removes close to 3000 synsets containing people, a number of these are of an offensive or sensitive nature. There is work in progress to filter a similar dataset from `winter21`, and there is already [ImageNet-21k-P](https://github.com/Alibaba-MIIL/ImageNet21K/blob/main/dataset_preprocessing/processing_instructions.md) but with different thresholds & preprocessing steps.

### Data Splits

Unlike ImageNet-1k (ILSVRC 2012), the full ImageNet dataset has no defined splits. This subset includes a validation split consiting of 40 samples per 11821 classes.

#### Train
* `imagenet12k-train-{0000..1023}.tar`
* 12129687 samples over 1024 shards

#### Validation
* `imagenet12k-validation-{0000..0511}.tar`
* 472840 samples over 512 shards

### Processing
I performed some processing while sharding this dataset:
* All exif tags not related to color space were removed
* All images with width or height < 48 were removed.
* All images with the smallest edge > 600 were resized, maintaining aspect so that they were = 600. Improving size & decoding time uniformity for typical pretrain use cases.
* Images were pre-shuffled across the shards

## Additional Information

### Dataset Curators

Authors of [[1]](https://arxiv.org/abs/1409.0575) and [[2]](https://ieeexplore.ieee.org/abstract/document/5206848):

- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C Berg
- Li Fei-Fei

### Licensing Information

In exchange for permission to use the ImageNet database (the "Database") at Princeton University and Stanford University, Researcher hereby agrees to the following terms and conditions:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
1. Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
1. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
1. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
1. Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
1. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
1. The law of the State of New Jersey shall apply to all disputes under this agreement.

### Citation Information

```bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet Large Scale Visual Recognition Challenge} },
Year = {2015},
journal = {International Journal of Computer Vision (IJCV)},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}
```

## 数据集说明

- **主页：** https://image-net.org/index.php
- **代码仓库：** https://github.com/rwightman/imagenet-12k
- **相关论文：** https://arxiv.org/abs/1409.0575

### 数据集概述
本数据集为完整ImageNet (ImageNet)数据集的过滤版本，选取了原数据集21841个类别中样本量排名前11821的类别。该数据集已被用于`timm`库中多个`in12k`模型的预训练。

从原始完整ImageNet构建该数据集的代码与元数据可在https://github.com/rwightman/imagenet-12k获取。

注意：此子集是从原始`fall11`版ImageNet中过滤得到的，而`fall11`版已被`winter21`版替代，后者移除了近3000个包含人物的同义词集，其中部分涉及冒犯性或敏感内容。目前已有工作尝试从`winter21`版中过滤得到类似数据集，且已存在[ImageNet-21k-P](https://github.com/Alibaba-MIIL/ImageNet21K/blob/main/dataset_preprocessing/processing_instructions.md)，但其采用了不同的阈值与预处理流程。

### 数据划分
与ImageNet-1k（ILSVRC 2012）不同，完整ImageNet数据集并未定义固定划分。本子集包含验证划分，每个类别对应40个样本。

#### 训练集
* 文件名格式：`imagenet12k-train-{0000..1023}.tar`
* 共12129687个样本，分布于1024个分片

#### 验证集
* 文件名格式：`imagenet12k-validation-{0000..0511}.tar`
* 共472840个样本，分布于512个分片

### 预处理流程
本数据集在分片过程中进行了如下预处理：
* 移除所有与色彩空间无关的EXIF标签
* 移除宽度或高度小于48像素的所有图像
* 将最短边大于600像素的所有图像按比例调整至最短边为600像素，以统一典型预训练场景下的图像尺寸与解码耗时
* 对所有图像进行跨分片预洗牌

## 附加信息

### 数据集整理者
论文[[1]](https://arxiv.org/abs/1409.0575)与[[2]](https://ieeexplore.ieee.org/abstract/document/5206848)的作者：
- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C. Berg
- Li Fei-Fei

### 授权信息
为获取在普林斯顿大学与斯坦福大学使用ImageNet数据库（下称"数据库"）的权限，研究人员同意遵守以下条款与条件：
1. 研究人员仅可将数据库用于非商业性研究与教育目的。
2. 普林斯顿大学与斯坦福大学不对数据库作出任何明示或默示担保，包括但不限于不侵权担保或特定用途适用性担保。
3. 研究人员需对其使用数据库的行为承担全部责任，并需就因研究人员使用数据库（包括但不限于研究人员从数据库生成的任何受版权保护的图像副本）而引发的所有索赔，对ImageNet团队、普林斯顿大学与斯坦福大学及其雇员、受托人、管理人员与代理人进行辩护并赔偿。
4. 研究人员可向其研究助理与同事提供数据库的访问权限，但前提是这些人员需先同意受本条款与条件约束。
5. 普林斯顿大学与斯坦福大学保留随时终止研究人员访问数据库的权利。
6. 若研究人员受雇于营利性商业实体，则其雇主也需受本条款与条件约束，且研究人员在此声明其已获得充分授权，可代表雇主签署本协议。
7. 本协议项下的所有争议均适用新泽西州法律。

### 引用信息
bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet Large Scale Visual Recognition Challenge} },
Year = {2015},
journal = {International Journal of Computer Vision (IJCV)},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}