nyanko7/danbooru2023

---
license: mit
task_categories:
- image-classification
- image-to-image
- text-to-image
language:
- en
- ja
pretty_name: danbooru2023
size_categories:
- 1M<n<10M
viewer: false
---

<img src="https://huggingface.co/datasets/nyanko7/danbooru2023/resolve/main/cover.webp" alt="cover" width="750"/>

# Danbooru2023: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset

<!-- Provide a quick summary of the dataset. -->

Danbooru2023 is a large-scale anime image dataset with over 5 million images contributed and annotated in detail by an enthusiast community. Image tags cover aspects like characters, scenes, copyrights, artists, etc with an average of 30 tags per image.

Danbooru is a veteran anime image board with high-quality images and extensive tag metadata. The dataset can be used to train image classification, multi-label tagging, character detection, generative models, and other computer vision tasks.

- **Shared by:** Nyanko Devs
- **Language(s):** English, Japanese
- **License:** MIT

This dataset is built on the top of [danbooru2021](https://gwern.net/danbooru2021). We expands the dataset to include images up to ID #6,857,737, adding over 1.8 million additional images and total size is now approximately 8 terabytes (8,000 GB).

## Use

## Format

The goal of the dataset is to be as easy as possible to use immediately, avoiding obscure file formats, while allowing simultaneous research & seeding of the torrent, with easy updates.

Images are provided in the full original form (be that JPG, PNG, GIF or otherwise) for reference/archival purposes, and bucketed into 1000 subdirectories 0000–0999 (0-padded), which is the Danbooru ID modulo 1000 (ie. all images in 0999/ have an ID ending in ‘999’); IDs can be turned into paths by dividing & padding (eg. in Bash, BUCKET=$(printf "%04d" $(( ID % 1000 )) )) and then the file is at {original,512px}/$BUCKET/$ID.$EXT.

The reason for the bucketing is that a single directory would cause pathological filesystem performance, and modulo ID is a simple hash which spreads images evenly without requiring additional future directories to be made or a filesystem IO to check where the file is. The ID is not zero-padded and files end in the relevant extension, hence the file layout looks like this:

```bash
$ tree / | less

/
├── danbooru2023 -> /mnt/diffusionstorage/workspace/danbooru/
│ ├── metadata
│ ├── readme.md
│ ├── original
│ │ ├── 0000 -> data-0000.tar
│ │ ├── 0001 -> data-0001.tar
│ │ │ ├── 10001.jpg
│ │ │ ├── 210001.png
│ │ │ ├── 3120001.webp
│ │ │ ├── 6513001.jpg
│ │
│ ├── recent
│ │ ├── 0000 -> data-1000.tar
│ │ ├── 0001 -> data-1001.tar
│ │
│ ├── updates
│ │ ├── 20240319
│ │ │ ├── dataset-0.tar
│ │ │ ├── dataset-1.tar
│ │ │
│ │ ├── 2024xxxx
│ │ │ ├── dataset-0.tar
│ │ │ ├── dataset-1.tar

```

Where `data-{1000..1999}.tar` refer to recent update files (should be updated every few months) and `updates` refer to fast patches (should be updated every few days to few weeks).

Currently represented file extensions are: avi/bmp/gif/html/jpeg/jpg/mp3/mp4/mpg/pdf/png/rar/swf/webm/wmv/zip.

Raw original files are treacherous. Be careful if working with the original dataset. There are many odd files: truncated, non-sRGB colorspace, wrong file extensions (eg. some PNGs have .jpg extensions like original/0146/1525146.jpg or original/0558/1422558.jpg), etc.

许可证：MIT许可证
任务类别：图像分类、图像到图像、文本到图像
语言：英语、日语
数据集展示名称：danbooru2023
样本规模：100万 < 样本数量 < 1000万
数据集查看器：未启用

 封面图（宽度750像素）

# Danbooru2023：大规模众包标注动画插画数据集

Danbooru2023是一款大规模动画图像数据集，由爱好者社区贡献并详细标注了超过500万张图像。图像标签涵盖角色、场景、版权归属、创作者等多个维度，单张图像平均标注标签数达30个。

Danbooru作为老牌动画图像分享平台，拥有高质量图像与丰富的标签元数据。本数据集可用于训练图像分类、多标签标注、角色检测、生成式模型等各类计算机视觉任务。

- **分享团队：** Nyanko开发团队
- **支持语言：** 英语、日语
- **许可证：** MIT许可证

本数据集基于[danbooru2021](https://gwern.net/danbooru2021)构建，我们将数据集扩展至ID编号6,857,737的图像，新增超过180万张图像，总存储规模现已达到约8太字节（8000 GB）。

## 使用说明
## 数据集格式

本数据集旨在实现开箱即用，避免使用晦涩的文件格式，同时支持科研与种子分享同步进行，并便于后续更新。

图像以原始完整格式（包括JPG、PNG、GIF等）提供，用于参考与存档，并按照Danbooru ID对1000取模的结果，被划分至0000至0999共1000个补零子目录中（即0999/目录下的所有图像ID均以"999"结尾）。可通过对ID取模并补零生成目录路径（例如在Bash环境中，执行`BUCKET=$(printf "%04d" $(( ID % 1000 )))`），最终文件路径格式为`{original,512px}/$BUCKET/$ID.$EXT`。

采用目录分桶的原因在于，单个目录会导致文件系统性能异常低下；而对ID取模是一种简单的哈希方式，可将图像均匀分布，无需后续新增目录或通过文件系统IO查询文件位置。图像ID无需补零，文件以对应扩展名结尾，因此文件布局如下所示：

bash
$ tree / | less

/
├── danbooru2023 -> /mnt/diffusionstorage/workspace/danbooru/
│ ├── metadata
│ ├── readme.md
│ ├── original
│ │ ├── 0000 -> data-0000.tar
│ │ ├── 0001 -> data-0001.tar
│ │ │ ├── 10001.jpg
│ │ │ ├── 210001.png
│ │ │ ├── 3120001.webp
│ │ │ ├── 6513001.jpg
│ │
│ ├── recent
│ │ ├── 0000 -> data-1000.tar
│ │ ├── 0001 -> data-1001.tar
│ │
│ ├── updates
│ │ ├── 20240319
│ │ │ ├── dataset-0.tar
│ │ │ ├── dataset-1.tar
│ │
│ │ ├── 2024xxxx
│ │ │ ├── dataset-0.tar
│ │ │ ├── dataset-1.tar

其中`data-{1000..1999}.tar`为近期更新文件（建议每数月更新一次），`updates`为快速补丁包（建议每数日至数周更新一次）。

当前收录的文件扩展名包括：avi、bmp、gif、html、jpeg、jpg、mp3、mp4、mpg、pdf、png、rar、swf、webm、wmv、zip。

原始文件存在诸多异常与隐患，在使用原始数据集时需格外谨慎。其中包含大量异常文件：截断文件、非sRGB色彩空间文件、扩展名错误的文件（例如部分PNG图像被错误命名为.jpg后缀，如`original/0146/1525146.jpg`或`original/0558/1422558.jpg`）等。

原始信息汇总

Danbooru2023 数据集概述

数据集规模

• 图像数量：超过500万张

图像特征

• 标签细节：每张图像平均包含30个标签
• 标签内容：涵盖角色、场景、版权、艺术家等多个方面

社区贡献

• 贡献者：由爱好者社区贡献和详细标注