site_tags

# Site Tags Dataset Collection

## Summary

This comprehensive dataset collection provides **structured tag metadata** from 18 popular image hosting and booru websites, offering a unified repository for **multilingual tag analysis** and **content classification**. The dataset encompasses over 2.5 million unique tags across various platforms including Danbooru, Gelbooru, Pixiv, Sankaku Complex, and Wallhaven, making it one of the most extensive collections of **image annotation metadata** available. Each platform's tags are provided in multiple formats including JSON, CSV, Parquet, and SQLite databases, ensuring compatibility with diverse data processing workflows.

The dataset captures rich **semantic relationships** between tags through various metadata fields such as tag categories, post counts, aliases, and hierarchical relationships. For platforms like Danbooru and Gelbooru, the collection includes comprehensive tag alias mappings that facilitate **cross-platform tag normalization** and **synonym resolution**. The multilingual nature of the dataset is particularly valuable, with tags available in English, Japanese, and Russian across different platforms, enabling research in **cross-lingual information retrieval** and **multicultural content analysis**.

Key technical features include **structured categorization** systems where tags are classified into types such as character, copyright, artist, and general tags, providing granular control over content classification. The inclusion of **usage statistics** like post counts and view metrics allows for popularity-based analysis and trend identification. This dataset serves as a foundational resource for **machine learning applications** in content recommendation, automated tagging systems, and semantic search engines, while also supporting academic research in **digital humanities** and **web content analysis**.

## Dataset Structure

The repository is organized by source website, with each directory containing tag data in multiple formats:

- **JSON files**: Complete tag metadata with full structural information
- **CSV files**: Tabular format for easy data analysis
- **Parquet files**: Optimized columnar storage for large-scale processing
- **SQLite databases**: Relational database format for complex queries

### Supported Platforms

- **Anime-focused**: anime-pictures.net, danbooru.donmai.us, safebooru.donmai.us
- **Booru networks**: gelbooru.com, konachan.com, konachan.net, rule34.xxx, e621.net
- **Art communities**: pixiv.net, en.pixiv.net, wallhaven.cc
- **Specialized**: sankakucomplex.com, hypnohub.net, lolibooru.moe, xbooru.com, zerochan.net

## Usage

The dataset can be accessed through multiple interfaces depending on your preferred data format:

### Using JSON Format
```python
import json
import pandas as pd

# Load tags from Danbooru
with open('danbooru.donmai.us/tags.json', 'r', encoding='utf-8') as f:
danbooru_tags = json.load(f)

# Convert to DataFrame for analysis
df = pd.DataFrame(danbooru_tags)
print(f"Danbooru contains {len(df)} tags")
print(f"Most popular tags: {df.nlargest(5, 'post_count')['name'].tolist()}")
```

### Using Parquet Format
```python
import pandas as pd

# Load efficient parquet format
df = pd.read_parquet('gelbooru.com/tags.parquet')
print(f"Gelbooru tag categories: {df['type'].value_counts()}")
```

### Using SQLite Database
```python
import sqlite3

# Query the SQLite database
conn = sqlite3.connect('pixiv.net/tags.sqlite')
cursor = conn.cursor()

# Get tag statistics
cursor.execute("SELECT COUNT(*), AVG(posts) FROM tags")
count, avg_posts = cursor.fetchone()
print(f"Pixiv has {count} tags with average {avg_posts:.1f} posts")
```

### Accessing Tag Aliases
```python
import json

# Load tag aliases for synonym resolution
with open('danbooru.donmai.us/tag_aliases.json', 'r') as f:
aliases = json.load(f)

# Create alias mapping
alias_map = {item['antecedent_name']: item['consequent_name'] for item in aliases}
print(f"Found {len(alias_map)} tag aliases")
```

## Data Schema

### Common Tag Fields
- `id`: Unique identifier for the tag
- `name`: Primary tag name (often in English)
- `post_count`: Number of posts using this tag
- `category/type`: Classification (0=general, 1=artist, 3=copyright, 4=character)
- `created_at/updated_at`: Timestamps for tag lifecycle

### Platform-Specific Extensions
- **Pixiv**: `wiki_url`, `views`, `checklists`, content type flags
- **Sankaku**: `trans_en`, `trans_ja`, `trans_ru` multilingual translations
- **Wallhaven**: `category_name`, `subscriptions`, view statistics
- **e621**: `related_tags` with co-occurrence information

## Original Content

### Directory Structure
```
datasets/deepghs/site_tags@main/
├── anime-pictures.net
│ ├── tags.csv
│ ├── tags.json
│ ├── tags.parquet
│ └── tags.sqlite
├── booru.allthefallen.moe
│ ├── tag_aliases.csv
│ ├── tag_aliases.json
│ ├── tags.csv
│ ├── tags.json
│ └── tags.sqlite
├── chan.sankakucomplex.com
│ ├── tags.csv
│ ├── tags.json
│ └── tags.sqlite
├── danbooru.donmai.us
│ ├── tag_aliases.csv
│ ├── tag_aliases.json
│ ├── tag_aliases.parquet
│ ├── tags.csv
│ ├── tags.json
│ ├── tags.parquet
│ └── tags.sqlite
# ... and 14 more platforms
```

### Sample Data Examples

The dataset includes comprehensive tag information from each platform. For example, Danbooru tags include metadata such as:
- Tag categories (general, artist, copyright, character)
- Post counts indicating popularity
- Timestamp information
- Deprecation status
- Word segmentation for compound tags

Pixiv data includes additional content classification flags:
- Media type indicators (anime, manga, novel, game, etc.)
- View and engagement statistics
- Wiki integration URLs

## Applications

This dataset enables various applications including:
- **Content recommendation systems** based on tag similarity
- **Multilingual tag normalization** across platforms
- **Trend analysis** through post count temporal data
- **Semantic search** enhancement using tag relationships
- **Machine learning** training for automated tagging
- **Cultural analysis** of content preferences across regions

## Citation

```bibtex
@misc{site_tags_dataset,
title = {Site Tags Dataset Collection: Comprehensive Tag Metadata from Image Hosting Platforms},
author = {deepghs},
howpublished = {\url{https://huggingface.co/datasets/deepghs/site_tags}},
year = {2024},
note = {A unified collection of structured tag metadata from 18 popular image hosting and booru websites, enabling multilingual tag analysis and content classification research},
abstract = {This comprehensive dataset collection provides structured tag metadata from 18 popular image hosting and booru websites, offering a unified repository for multilingual tag analysis and content classification. The dataset encompasses over 2.5 million unique tags across various platforms including Danbooru, Gelbooru, Pixiv, Sankaku Complex, and Wallhaven, making it one of the most extensive collections of image annotation metadata available. Each platform's tags are provided in multiple formats including JSON, CSV, Parquet, and SQLite databases, ensuring compatibility with diverse data processing workflows. The dataset captures rich semantic relationships between tags through various metadata fields such as tag categories, post counts, aliases, and hierarchical relationships.},
keywords = {tags, metadata, image-tagging, booru, dataset, multilingual}
}
```

# 站点标签数据集合集（Site Tags Dataset Collection）

## 概述
本综合数据集合集收录了来自18个主流图床与博鲁（booru）图库网站的结构化标签元数据（structured tag metadata），为多语言标签分析与内容分类任务提供了统一的数据源仓库。本数据集涵盖了包括Danbooru、Gelbooru、Pixiv、Sankaku Complex、Wallhaven在内的多个平台的超250万个唯一标签，是当前规模最庞大的图像标注元数据（image annotation metadata）合集之一。各平台的标签数据均提供JSON、CSV、Parquet以及SQLite数据库等多种格式，可适配多样化的数据处理流程。

本数据集通过标签分类、帖子计数、别名与层级关系等多种元数据字段，捕捉了标签间丰富的语义关联（semantic relationships）。针对Danbooru与Gelbooru等平台，合集还收录了完整的标签别名映射表，可助力跨平台标签归一化（cross-platform tag normalization）与同义词消解（synonym resolution）任务。本数据集的多语言特性极具研究价值，不同平台的标签覆盖英语、日语与俄语三种语言，可支持跨语言信息检索与多元文化内容分析相关研究。

其核心技术特性包含结构化分类体系（structured categorization systems）：标签被划分为角色、版权、创作者与通用标签等类型，可实现内容分类的精细化管控。数据集还收录了帖子计数、浏览量等使用统计数据，支持基于热度的分析与趋势识别。本数据集可作为内容推荐、自动标签系统与语义搜索引擎等机器学习应用（machine learning applications）的基础资源，同时也可为数字人文（digital humanities）与网络内容分析领域的学术研究提供支撑。

## 数据集结构
本数据集仓库按来源网站进行组织，每个目录下均包含多格式的标签数据：
- **JSON文件**：包含完整结构化信息的全量标签元数据
- **CSV文件**：便于数据分析的表格格式
- **Parquet文件**：针对大规模处理优化的列式存储格式
- **SQLite数据库**：支持复杂查询的关系型数据库格式

### 支持的平台
- **动漫向图库**：anime-pictures.net、danbooru.donmai.us、safebooru.donmai.us
- **博鲁（booru）图库集群**：gelbooru.com、konachan.com、konachan.net、rule34.xxx、e621.net
- **艺术创作社区**：pixiv.net、en.pixiv.net、wallhaven.cc
- **垂直细分图库**：sankakucomplex.com、hypnohub.net、lolibooru.moe、xbooru.com、zerochan.net

## 使用方法
本数据集可根据您偏好的数据格式通过多种接口获取：

### 使用JSON格式
python
import json
import pandas as pd

# Load tags from Danbooru
with open('danbooru.donmai.us/tags.json', 'r', encoding='utf-8') as f:
danbooru_tags = json.load(f)

# Convert to DataFrame for analysis
df = pd.DataFrame(danbooru_tags)
print(f"Danbooru contains {len(df)} tags")
print(f"Most popular tags: {df.nlargest(5, 'post_count')['name'].tolist()}")

### 使用Parquet格式
python
import pandas as pd

# Load efficient parquet format
df = pd.read_parquet('gelbooru.com/tags.parquet')
print(f"Gelbooru tag categories: {df['type'].value_counts()}")

### 使用SQLite数据库
python
import sqlite3

# Query the SQLite database
conn = sqlite3.connect('pixiv.net/tags.sqlite')
cursor = conn.cursor()

# Get tag statistics
cursor.execute("SELECT COUNT(*), AVG(posts) FROM tags")
count, avg_posts = cursor.fetchone()
print(f"Pixiv has {count} tags with average {avg_posts:.1f} posts")

### 访问标签别名
python
import json

# Load tag aliases for synonym resolution
with open('danbooru.donmai.us/tag_aliases.json', 'r') as f:
aliases = json.load(f)

# Create alias mapping
alias_map = {item['antecedent_name']: item['consequent_name'] for item in aliases}
print(f"Found {len(alias_map)} tag aliases")

## 数据模式
### 通用标签字段
- `id`：标签的唯一标识符
- `name`：标签的主名称（通常为英语）
- `post_count`：使用该标签的帖子数量
- `category/type`：标签分类（0=通用标签，1=创作者标签，3=版权标签，4=角色标签）
- `created_at/updated_at`：标签生命周期的时间戳

### 平台专属扩展字段
- **Pixiv专属字段**：`wiki_url`、`views`、`checklists`以及内容类型标记
- **Sankaku专属字段**：`trans_en`、`trans_ja`、`trans_ru`等多语言翻译字段
- **Wallhaven专属字段**：`category_name`、`subscriptions`以及浏览量统计数据
- **e621专属字段**：包含共现信息的`related_tags`（相关标签）

## 原始内容
### 目录结构

datasets/deepghs/site_tags@main/
├── anime-pictures.net
│ ├── tags.csv
│ ├── tags.json
│ ├── tags.parquet
│ └── tags.sqlite
├── booru.allthefallen.moe
│ ├── tag_aliases.csv
│ ├── tag_aliases.json
│ ├── tags.csv
│ ├── tags.json
│ └── tags.sqlite
├── chan.sankakucomplex.com
│ ├── tags.csv
│ ├── tags.json
│ └── tags.sqlite
├── danbooru.donmai.us
│ ├── tag_aliases.csv
│ ├── tag_aliases.json
│ ├── tag_aliases.parquet
│ ├── tags.csv
│ ├── tags.json
│ ├── tags.parquet
│ └── tags.sqlite
# ... and 14 more platforms

### 示例数据样例
本数据集收录了各平台的全量标签信息。例如，Danbooru标签包含以下元数据：
- 标签分类（通用、创作者、版权、角色）
- 体现标签热度的帖子计数
- 时间戳信息
- 标签弃用状态
- 复合标签的分词信息

Pixiv数据包含额外的内容分类标记：
- 媒体类型标识（动漫、漫画、小说、游戏等）
- 浏览量与互动统计数据
- 维基集成URL

## 应用场景
本数据集可支持以下各类应用：
- 基于标签相似度的内容推荐系统
- 跨平台多语言标签归一化
- 基于帖子计数时序数据的趋势分析
- 利用标签关联实现语义搜索增强
- 用于自动标签系统的机器学习训练
- 跨地域内容偏好的文化分析

## 引用格式
bibtex
@misc{site_tags_dataset,
title = {Site Tags Dataset Collection: Comprehensive Tag Metadata from Image Hosting Platforms},
author = {deepghs},
howpublished = {url{https://huggingface.co/datasets/deepghs/site_tags}},
year = {2024},
note = {A unified collection of structured tag metadata from 18 popular image hosting and booru websites, enabling multilingual tag analysis and content classification research},
abstract = {This comprehensive dataset collection provides structured tag metadata from 18 popular image hosting and booru websites, offering a unified repository for multilingual tag analysis and content classification. The dataset encompasses over 2.5 million unique tags across various platforms including Danbooru, Gelbooru, Pixiv, Sankaku Complex, and Wallhaven, making it one of the most extensive collections of image annotation metadata available.},
keywords = {tags, metadata, image-tagging, booru, dataset, multilingual}
}