imagenet-w21-webp-wds

## Dataset Description

- **Homepage:** https://image-net.org/index.php
- **Paper:** https://arxiv.org/abs/1409.0575

### Dataset Summary

This is a copy of the full `Winter21` release of ImageNet in webdataset tar format with WEBP encoded images. This release consists of 19167 classes, 2674 fewer classes than the original 21841 class `Fall11` release of the full ImageNet.

The classes were removed due to these concerns: https://www.image-net.org/update-sep-17-2019.php

This is the same contents as https://huggingface.co/datasets/timm/imagenet-w21-wds but encoded in webp at ~56% of the size, shard count halved.

### Data Splits

The full ImageNet dataset has no defined splits. This release follows that and leaves everything in the train split. Shards are shuffled so validation & test splits can be made by dividing at the shard level.

#### Train
* `imagenet12k-train-{0000..1023}.tar`
* 13151276 samples over 1024 shards
* 645.65 GB

### Processing
I performed some processing while sharding this dataset:
* All exif tags not related to color space were removed
* A set of 20 partially corrupted images in the original tar file were corrected and re-encoded
* All images with width or height < 32 were removed, ~2000 images.
* All images with the smallest edge > 768 were resized, maintaining aspect so that they were = 768. Improving size & decoding time uniformity for typical pretrain use cases.
* Images were re-encoded in WEBP
* Images were pre-shuffled across the shards

## Additional Information

### Dataset Curators

Authors of [[1]](https://arxiv.org/abs/1409.0575) and [[2]](https://ieeexplore.ieee.org/abstract/document/5206848):

- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C Berg
- Li Fei-Fei

### Licensing Information

In exchange for permission to use the ImageNet database (the "Database") at Princeton University and Stanford University, Researcher hereby agrees to the following terms and conditions:

1. Researcher shall use the Database only for non-commercial research and educational purposes.
1. Princeton University and Stanford University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
1. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database.
1. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
1. Princeton University and Stanford University reserve the right to terminate Researcher's access to the Database at any time.
1. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
1. The law of the State of New Jersey shall apply to all disputes under this agreement.

### Citation Information

```bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet Large Scale Visual Recognition Challenge} },
Year = {2015},
journal = {International Journal of Computer Vision (IJCV)},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}
```

## 数据集描述
- **数据集主页:** https://image-net.org/index.php
- **相关论文:** https://arxiv.org/abs/1409.0575

### 数据集概览
本数据集为ImageNet完整`Winter21`版本的复刻版，采用webdataset tar格式封装，图像以WebP格式（WEBP）编码。本版本共包含19167个类别，相较于原版完整ImageNet的`Fall11`版本（21841个类别）减少了2674个类别。
被移除类别的相关考量可参见：https://www.image-net.org/update-sep-17-2019.php
本数据集与https://huggingface.co/datasets/timm/imagenet-w21-wds 内容完全一致，但采用WebP编码后体积仅为其约56%，且分片数减半。

### 数据划分
完整ImageNet数据集并未定义官方划分方式，本版本遵循这一设定，将全部数据归入训练集。所有分片均已打乱，因此可通过分片层面的划分来构建验证集与测试集。

#### 训练集
* `imagenet12k-train-{0000..1023}.tar`
* 共包含13151276个样本，分布于1024个分片中
* 总大小：645.65 GB

### 预处理流程
本数据集在分片过程中进行了如下预处理操作：
* 移除所有与色彩空间无关的EXIF标签
* 修复并重新编码了原始tar文件中20张部分损坏的图像
* 移除所有宽或高小于32像素的图像，共计约2000张
* 将最短边大于768像素的图像按比例调整尺寸，使最短边等于768像素，以统一图像尺寸与解码耗时，适配典型的预训练场景需求
* 图像均重新编码为WebP格式
* 所有图像已在分片间预先打乱顺序

## 补充信息
### 数据集整理者
源自文献[[1]](https://arxiv.org/abs/1409.0575)与[[2]](https://ieeexplore.ieee.org/abstract/document/5206848)的作者：
- Olga Russakovsky
- Jia Deng
- Hao Su
- Jonathan Krause
- Sanjeev Satheesh
- Wei Dong
- Richard Socher
- Li-Jia Li
- Kai Li
- Sean Ma
- Zhiheng Huang
- Andrej Karpathy
- Aditya Khosla
- Michael Bernstein
- Alexander C Berg
- Li Fei-Fei

### 许可协议信息
为获得在普林斯顿大学与斯坦福大学使用ImageNet数据库（下称“本数据库”）的权限，研究人员特此同意以下条款与条件：
1. 研究人员仅可将本数据库用于非商业研究与教育用途。
2. 普林斯顿大学与斯坦福大学不对本数据库作出任何声明或担保，包括但不限于不侵权担保或特定用途适用性担保。
3. 研究人员需对其使用本数据库的行为承担全部责任，且需为因使用本数据库（包括但不限于通过本数据库制作的受版权保护的图像副本）而产生的所有索赔，对ImageNet团队、普林斯顿大学与斯坦福大学及其雇员、受托人、管理人员与代理人进行辩护并赔偿。
4. 研究人员可向其研究助手与同事提供本数据库的访问权限，但前提是这些人员需先同意受本条款与条件约束。
5. 普林斯顿大学与斯坦福大学保留随时终止研究人员访问本数据库的权利。
6. 若研究人员受雇于营利性商业实体，则其雇主也需受本条款与条件约束，且研究人员特此声明其已获得充分授权，可代表雇主签署本协议。
7. 本协议项下的所有争议均适用新泽西州法律。

### 引用信息
bibtex
@article{imagenet15russakovsky,
Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
Title = { {ImageNet大规模视觉识别挑战赛} },
Year = {2015},
journal = {国际计算机视觉杂志（IJCV）},
doi = {10.1007/s11263-015-0816-y},
volume={115},
number={3},
pages={211-252}
}

搜集汇总

背景与挑战

背景概述

该数据集是ImageNet Winter21版本的webp压缩格式副本，包含19167个类别，比原始版本减少了2674个类。数据以1024个分片tar文件形式提供，总计约1315万张图像，经过图像尺寸调整和重新编码处理。