imagenet-22k-wds

## Dataset Description

- **Homepage:** https://image-net.org/index.php
- **Repository:** https://github.com/rwightman/imagenet-12k
- **Paper:** https://arxiv.org/abs/1409.0575

### Dataset Summary

This is a copy of the full [ImageNet](https://www.image-net.org/) dataset consisting of all of the original 21841 clases. It also contains labels in a separate field for the '12k' subset described at at (https://github.com/rwightman/imagenet-12k, https://huggingface.co/datasets/timm/imagenet-12k-wds)

This dataset is from the original `fall11` ImageNet release which has been replaced by the `winter21` release which removes close to 3000 synsets containing people, a number of these are of an offensive or sensitive nature. There is work in progress to filter a similar dataset from `winter21`, and there is already [ImageNet-21k-P](https://github.com/Alibaba-MIIL/ImageNet21K/blob/main/dataset_preprocessing/processing_instructions.md) but with different thresholds & preprocessing steps.

### Data Splits

Unlike ImageNet-1k (ILSVRC 2012), the full ImageNet dataset has no defined splits.

This instance does include a randomly selected validation split consiting of 40 samples for the 11821 classes in ImageNet-12k. The validation split is the exact same as https://huggingface.co/datasets/timm/imagenet-12k-wds and does not fully cover all 22k classes. Beyond the 12k classes (sorted by # samples), the remaining have very few samples per-class. ImageNet-22k is not a balanced dataset.

#### Train
* `imagenet22k-train-{0000..4095}.tar`
* 13673551 samples over 4095 shards

#### Validation
* `imagenet22k-validation-{0000..0511}.tar`
* 472840 samples over 512 shards

### Processing
I performed some processing while sharding this dataset:
* All exif tags not related to color space were removed
* All images with width or height < 48 were removed.
* All images with the smallest edge > 600 were resized, maintaining aspect so that they were = 600. Improving size & decoding time uniformity for typical pretrain use cases.
* Images were pre-shuffled across the shards

## Additional Information

### Dataset Curators

Authors of [[1]](https://arxiv.org/abs/1409.0575) and [[2]](https://ieeexplore.ieee.org/abstract/document/5206848):

- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C Berg
- Li Fei-Fei

### Licensing Information

In exchange for permission to use the ImageNet database (the "Database") at Princeton University and Stanford University, Researcher hereby agrees to the following terms and conditions:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
1. Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
1. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
1. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
1. Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
1. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
1. The law of the State of New Jersey shall apply to all disputes under this agreement.

### Citation Information

```bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet Large Scale Visual Recognition Challenge} },
Year = {2015},
journal = {International Journal of Computer Vision (IJCV)},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}
```

## 数据集描述
- **主页:** https://image-net.org/index.php
- **代码仓库:** https://github.com/rwightman/imagenet-12k
- **论文:** https://arxiv.org/abs/1409.0575

## 数据集概览
本数据集为完整的[ImageNet](https://www.image-net.org/)数据集副本，涵盖原始全部21841个类别。此外，本数据集还在单独字段中提供了'12k'子集的标签，该子集的相关说明可参见(https://github.com/rwightman/imagenet-12k, https://huggingface.co/datasets/timm/imagenet-12k-wds)。

本数据集源自最初的`fall11`版ImageNet发布包，该版本已被`winter21`版取代；`winter21`版移除了近3000个包含人物的同义词集（synset），其中部分内容具有冒犯性或敏感性。目前已有基于`winter21`版构建相似数据集的过滤工作正在进行，且已存在[ImageNet-21k-P](https://github.com/Alibaba-MIIL/ImageNet21K/blob/main/dataset_preprocessing/processing_instructions.md)数据集，但其采用了不同的阈值与预处理步骤。

## 数据划分
与ImageNet-1k（ILSVRC 2012）不同，完整ImageNet数据集并无预设的数据划分。

本数据集实例包含针对ImageNet-12k的11821个类别随机选取的验证集划分，每个类别含40个样本。该验证集与https://huggingface.co/datasets/timm/imagenet-12k-wds中的验证集完全一致，且无法覆盖全部22k个类别。除按样本量排序的12k个类别外，剩余类别的单类别样本量极少，因此ImageNet-22k并非均衡数据集。

### 训练集
* `imagenet22k-train-{0000..4095}.tar`
* 共13673551个样本，分布于4095个分片（shard）中。

### 验证集
* `imagenet22k-validation-{0000..0511}.tar`
* 共472840个样本，分布于512个分片（shard）中。

## 预处理流程
在对本数据集进行分片处理时，我执行了以下预处理操作：
1. 移除所有与色彩空间无关的EXIF标签
2. 移除宽或高小于48像素的图像
3. 对最短边大于600像素的图像进行缩放，保持宽高比不变，将其最短边调整为600像素，以统一典型预训练场景下的图像尺寸与解码耗时
4. 对所有图像在分片间进行预打乱

## 附加信息
### 数据集维护者
本数据集基于[[1]](https://arxiv.org/abs/1409.0575)与[[2]](https://ieeexplore.ieee.org/abstract/document/5206848)的作者成果，作者名单如下：
- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C. Berg
- Li Fei-Fei

### 许可协议
为获得在普林斯顿大学与斯坦福大学使用ImageNet数据库的权限，研究者在此同意遵守以下条款与条件：
1. 研究者仅可将本数据库用于非商业性研究与教育用途。
2. 普林斯顿大学与斯坦福大学不对本数据库作出任何声明或担保，包括但不限于不侵权担保或特定用途适用性担保。
3. 研究者需为其使用本数据库的行为承担全部责任，并需就因研究者使用本数据库（包括但不限于研究者通过本数据库制作的受版权保护的图像副本）而引发的任何及所有索赔，对ImageNet团队、普林斯顿大学及斯坦福大学（包括其雇员、受托人、官员与代理人）进行辩护与赔偿。
4. 研究者可向研究助理与同事开放本数据库的使用权限，但前提是他们需先同意受本条款与条件约束。
5. 普林斯顿大学与斯坦福大学保留随时终止研究者使用本数据库权限的权利。
6. 若研究者受雇于营利性商业实体，则研究者的雇主同样需受本条款与条件约束，且研究者在此声明其已获得充分授权，可代表该雇主签署本协议。
7. 本协议项下的所有争议均适用新泽西州法律。

### 引用信息
bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet Large Scale Visual Recognition Challenge} },
Year = {2015},
journal = {International Journal of Computer Vision (IJCV)},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}

搜集汇总

背景与挑战

背景概述

imagenet-22k-wds是ImageNet数据集的完整版本，包含21841个类别，经过预处理优化了图像质量和大小。数据集提供了训练和验证分割，但验证集未覆盖所有类别，且使用受非商业研究许可限制。